Responding to statistical reviewer comments

PerfectPaper’s guide to statistical reviewer comments explains how to answer the twelve most common statistical objections: read what the reviewer’s short comment actually means, reanalyse or report rather than collect new data where possible, and state plainly in the response letter what changed and why. Most objections resolve through correct reanalysis, not additional experiments.

Most statistical reviewer comments are shorter than the problem they point at. “The sample size is inadequate” can mean the study is underpowered, that the reported n counts the wrong unit, or that the reviewer does not believe the effect — the worked example below runs all three readings against the same manuscript. Each reading calls for a different response, and answering the wrong one is how a revision fails twice.

The objections

• Reviewer says my sample size is too small — usually about power, sometimes about the unit of analysis, occasionally about belief.
• Reviewer asked for a post hoc power calculation — a request that is circular as literally stated, and what to give instead.
• Reviewer says my n is not independent — pseudoreplication, and why it is the one objection that changes p-values by orders of magnitude.
• Reviewer wants multiple comparison correction — when it is required, when it is arguable, and how to answer either way.
• Reviewer questions my choice of statistical test — normality, paired structure, and the tests people reach for by habit.
• Reviewer says the effect is not clinically meaningful — significance is not magnitude, and the answer is rarely more data.

Common questions about sample size and test choice

Reviewer 2 says my sample size is too small — what do I actually say back?

Work out which of three things the reviewer means: underpowered, wrong unit of analysis, or disbelief in the effect. Report the actual power calculation if one exists, or the 95% confidence interval if it does not — that interval is the honest statement of what your data can and cannot rule out.

Is n = 3 enough for an RNA-seq or qPCR experiment?

Whether three is enough depends on what the three count: three biological replicates is the floor many differential-expression pipelines accept, while three technical replicates of one biological sample is pseudoreplication regardless of the count. Name which you have, report the effect size with its confidence interval, and let the interval’s width show whether three was really enough.

My replicates don’t agree with each other — do I average them or treat each one as a separate n?

Disagreement between replicates is exactly what pseudoreplication analysis handles: fit a random effect for the true independent unit — animal, donor, or culture — rather than averaging it away or counting each replicate as its own n. Averaging loses the variance you must report; treating each replicate as n inflates significance, the error Hurlbert’s 1984 survey found in nearly half of papers using inferential statistics.

Can I use a t-test if my data are not normally distributed?

Often yes: the t-test is robust to moderate non-normality once each group has roughly 15–20 or more observations, because the central limit theorem protects the sampling distribution of the mean even when the raw data are skewed. Below that, or with heavy skew, a Wilcoxon–Mann–Whitney test or a permutation test is safer — but it compares distributions, not means, so say which you tested.

The other six objections that complete the twelve

Six further statistical objections arrive often enough to belong in the same list, and each is answerable by reanalysis, by reporting, or by narrowing a claim rather than by new data.

“The statistical methods are not described in sufficient detail.” The missing item is almost always specific: the software version, the random-effect structure of a mixed model, how ties or missing values were handled, or which of several reported analyses was the primary one. Write against the SAMPL guidelines (Lang and Altman, International Journal of Nursing Studies 2015), which enumerate what a statistical methods paragraph must name.

“Missing data are not addressed.” A complete-case analysis is a decision, not a default, and reviewers increasingly ask you to say so. Report how many observations were lost from each variable, whether loss differed by arm, and what assumption your handling makes. Where randomised participants were dropped for non-adherence, the objection is really about intention to treat and the fix is an analysis set, not a sentence. A complete-case default is defensible only when missingness plausibly depends on variables you observed (missing at random, in the terminology statisticians have used since the 1970s); once it plausibly depends on the unobserved outcome itself (missing not at random), reviewers increasingly expect multiple imputation or a pattern-mixture sensitivity analysis, and the response letter should name which assumption you are making rather than leave it implicit.

“A significant effect in one group and a non-significant effect in the other is not an interaction.” Gelman and Stern named this in The American Statistician (2006;60:328–331): the difference between significant and not significant is not itself statistically significant. Nieuwenhuis, Forstmann and Wagenmakers reviewed 513 articles in Nature, Science, Nature Neuroscience, Neuron and The Journal of Neuroscience (Nature Neuroscience 2011;14:1105–1107); of the 157 articles where the error could occur, 78 tested the interaction correctly and 79 did not. In a separate set of 120 cellular and molecular papers they found no study that used the correct procedure. The remedy is one test of the interaction term — but note that test is itself lower-powered than either main-effect test alone, because the interaction estimate’s standard error compounds the uncertainty of both simple effects, which is a second reason published interaction claims replicate poorly even when both simple effects looked solid on their own.

“The comparison is within arms rather than between them.” Bland and Altman set out why testing baseline against follow-up separately in each randomised group, then contrasting the two verdicts, is misleading (Trials 2011;12:264). The comparison the design supports is between groups, adjusted for baseline.

“The reported statistics do not reconcile.” Nuijten and colleagues ran statcheck over more than 250,000 p-values in eight psychology journals from 1985 to 2013 and found that half of papers using null-hypothesis testing contained at least one p-value inconsistent with its own test statistic and degrees of freedom, and one in eight contained a gross inconsistency capable of changing the conclusion (Behavior Research Methods 2016;48:1205–1226). Recompute every p-value in the manuscript from the statistic and degrees of freedom you report before you resubmit.

“The analysis was not pre-specified.” The reviewer is asking whether the model, the outcome and the subgroup were chosen before or after the data were seen. Say which, per analysis. Labelling exploratory work as exploratory is a complete answer; presenting it as confirmatory is what a reviewer means by p-hacking.

Questions about statistical reporting and missing data

Why did my p-value change when I added a covariate to the model?

A p-value moves when a covariate is added for one of two reasons, and the reviewer needs to know which: the covariate absorbed confounding, changing the estimate itself, or it absorbed residual noise, tightening the standard error around an estimate that barely moved. Report both models side by side with their coefficients and confidence intervals so the reviewer can see which happened.

Do I have to account for every participant who drops out of a trial, or just report how many were lost?

State more than the count: report how many observations were lost from each variable, whether the loss differed by treatment arm, and what assumption your handling of the missing data makes. A complete-case analysis is a decision, not a default, and if participants were dropped after randomisation for non-adherence, say so as an intention-to-treat question, not a footnote.

What the reporting standard actually requires

The ICMJE Recommendations set the reporting floor most journals adopt, and the statistics clause is short enough to read in full. It states: “Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to judge its appropriateness for the study and to verify the reported results. When possible, quantify findings and present them with appropriate indicators of measurement error or uncertainty (such as confidence intervals). Avoid relying solely on statistical hypothesis testing, such as P values, which fail to convey important information about effect size and precision of estimates. References for the design of the study and statistical methods should be to standard works when possible (with pages stated). Define statistical terms, abbreviations, and most symbols. Specify the statistical software package(s) and versions used. Distinguish prespecified from exploratory analyses, including subgroup analyses.”

Three obligations in that clause generate most statistical review comments. Verifiability requires enough method that a reader could rerun the analysis. Quantification requires an effect size with a confidence interval rather than a p-value alone. The prespecified-versus-exploratory distinction requires a per-analysis label that most manuscripts never supply. A response letter that quotes the clause and shows where the revision now satisfies it is harder to argue with than one that promises to have been careful.

Sorting the comment before you answer it

Sort every statistical comment into one of four kinds before planning the revision, because the four differ in what they require and in how long they take. The sorting takes an hour and saves a round.

Reporting gaps. The analysis was right and the description was incomplete. Tells: the reviewer asks what something means, which version, which package, what n refers to. The fix is text, and it belongs in the next submission in full.

Analysis gaps. The data are adequate and the model applied to them is not. Tells: a named alternative method, a request for correction, a question about clustering or variance. The fix is reanalysis of existing data, and the honest report says whether the conclusion changed.

Design gaps. The experiment as performed cannot support the claim. Tells: requests for an independent replication, a different comparator, or a randomised comparison. The fix is new data or a narrower claim, and pretending otherwise reads as evasion. These sit closer to methods and design objections than to statistics — though one specific design gap does have a statistical answer short of new data: for an unmeasured-confounding objection in an observational study, an E-value (the sensitivity-analysis measure VanderWeele and Ding introduced in 2017) quantifies how strong an unmeasured confounder would have to be, on both the exposure and the outcome, to fully explain the observed association away, which is a bounding argument rather than a replication.

Claim gaps. The numbers are fine and the sentences describing them are not. Tells: “the authors overstate”, “causal language is used for an observational design”, “the abstract does not reflect Table 3”. The fix is the abstract and discussion, and it is the cheapest of the four to make and the most common to skip. Causal wording specifically is covered by correlative, not causal.

Answers that need no new data

Six reanalyses resolve the majority of statistical objections without a single additional sample, and naming them in the response letter is more persuasive than arguing that the original analysis was acceptable.

Reanalyse at the correct independent unit — per animal, per donor, per independently derived culture — or fit a random effect for it. Report the effect size with a 95% interval alongside every p-value, which answers precision questions that a power figure cannot. Apply, or explicitly decline with a named family, the correction the reviewer wants, and put the corrected table in supplementary material either way; FDR versus family-wise error sets out which target fits which claim. When family-wise error genuinely is the right target, name Holm’s step-down procedure rather than plain Bonferroni: it controls the same error rate at the same level while rejecting at least as many hypotheses, so it answers a reviewer’s “why so conservative” follow-up before it is asked, with no statistical downside. Refit under the alternative test or distributional model and state whether the conclusion holds. Run the analysis under a second plausible handling of missing data as a sensitivity analysis. And narrow the claim in the abstract and discussion to what survives all of the above.

The one objection that resists this list is pseudoreplication, and only in the sense that reanalysis at the true unit sometimes removes the result. That is still the correct answer; the alternative is publishing an inference the design does not support. Hurlbert’s survey of 176 experimental papers (Ecological Monographs 1984;54:187–211) found pseudoreplication in 27% overall and in 48% of those applying inferential statistics, so a reviewer raising it is working from a well-founded prior rather than pedantry.

Worked example: one comment, three readings

The comment “the sample size is inadequate” has three distinct, mutually exclusive readings when applied to a manuscript reporting a 34% reduction in a marker, p = 0.008, with figure legends stating n = 120.

Read as power. The reviewer doubts the study could detect the effect claimed. The response reports the prospective calculation if one exists — assumed effect, its source, alpha, power, resulting group size — and otherwise gives the 95% interval, which here might run from a 9% to a 52% reduction. That interval is the honest statement of what the data exclude, and it is what statistical power questions are usually reaching for.

Read as unit of analysis. The n = 120 is 120 cells from four animals. The true n is 4. The response reanalyses per animal, reports the wider interval, and says plainly whether significance survives. No new animals are involved, and the reported n falls by a factor of 30.

Read as disbelief. The other comments ask for orthogonal validation or a second model. Neither more data nor more statistics addresses this; evidence from a different direction does, and where none exists, a narrower claim is the legitimate response.

Three responses, one sentence of review. Sending the power answer to a unit-of-analysis reviewer produces a second round in which the reviewer now believes you did not understand the objection, which is a harder position than the one you started in.

When the reviewer’s request is the wrong analysis

Decline a specific analysis when it is wrong, name the reason, and supply the analysis that answers the underlying concern. Three requests recur in that category.

A post hoc power calculation computed from the observed effect is circular by construction — Hoenig and Heisey showed it is a deterministic function of the p-value (The American Statistician 2001;55:19–24), so a non-significant result must return low observed power whatever the science. Offer the confidence interval instead.

A Bonferroni correction applied across every test in the paper, including unrelated hypotheses, controls nothing meaningful and inflates type II error. Name the family each correction was applied over and explain why these tests are not one family.

A formal normality test as the basis for choosing a test is a weak instrument: underpowered at the sample sizes where the assumption matters and oversensitive where it does not. Supply a QQ plot and the sample size instead.

In all three, offer the requested output in supplementary material with a one-line note on its interpretation. That satisfies the letter of the request without giving a misleading number rhetorical weight, and it removes the editor’s reason to treat the exchange as a refusal.

Questions about power calculations and corrections

Reviewer wants a post hoc power calculation — isn’t that circular?

Yes — a power calculation run on the effect size you already observed is a deterministic function of the p-value you got, so a non-significant result always returns low observed power no matter how real the underlying effect is. Hoenig and Heisey demonstrated this formally in 2001. Offer the 95% confidence interval instead; it answers the same question about precision without the circularity.

Do I have to correct for multiple comparisons if I only report the tests that came out significant?

Reporting only the significant tests does not remove the multiplicity problem — it hides it. Correction is owed for every test that was actually run within a defined family, whether or not you report the null results, and a reviewer who suspects selective reporting will ask directly how many comparisons were performed. Disclose the full set tested, then correct or justify not correcting.

Do I have to run more experiments to satisfy a statistics reviewer?

Often not. Many statistical objections are resolved by reanalysis at the correct unit, by reporting an effect size with its interval, or by narrowing a claim to what the data support. New data is the last resort, not the first.

What to put in the response letter

Answer each statistical comment in a fixed four-part shape: restate the concern in your own words, state what you did, give the number, and say where in the revised manuscript it appears. A workable instance reads: “The reviewer is correct that the analysis treated individual cells as independent. We have reanalysed at the animal level (n = 4 per group); the reduction is 31% (95% CI 4% to 50%, p = 0.03) and Figure 2B, the abstract and the discussion have been amended accordingly.”

Three details decide how that letter lands. Give the number in the letter, not only a pointer to a line in the manuscript. Say explicitly when a reanalysis changed the result, because a reviewer who discovers it themselves reads it as concealment. And where you disagree, disagree in a sentence with a citation rather than a paragraph of justification. How to write a response to reviewers covers the letter as a whole.

Naming a statistician who performed the reanalysis, in the letter and in the author list or acknowledgements, changes how the rest of the response is read. Journals differ widely in how much dedicated statistical review they can call on, and a manuscript that arrives with the statistics already independently checked is treated differently by an editor who has none available.

Frequently asked questions about the response letter

How should I respond to a statistical reviewer comment I disagree with?

Answer the concern rather than the sentence. State what the reviewer’s objection would mean if correct, give the specific evidence bearing on it, and say plainly where you disagree and why. Editors accept reasoned disagreement; they do not accept a comment being ignored.

Should I bring in a statistician to answer the reviewer’s statistical objections?

If the objections are substantive and you are not confident answering them, yes. Name that person in the response letter, and in the author list or the acknowledgements as their contribution warrants. Reviewers read a named statistician as evidence the concern was worked through rather than argued around.

How long should a response to a statistical comment be?

Four parts and rarely more than a short paragraph: restate the concern, state what you did, give the number with its interval, and cite the location in the revised manuscript. Long justifications read as defensiveness; a number and a location read as compliance.

What has to change in the manuscript, not only the letter

Every statistical concession has to reach the abstract, the results text, the figure legends and the discussion, or the revision draws the same objection a second time. This is the single most common way a technically correct revision fails.

The specific failures are easy to enumerate and easy to check. A methods section stating false discovery rate control while a figure legend still reports nominal p. An abstract asserting the original effect after the reanalysis widened the interval to include no effect. A discussion describing an effect as “marked” that the results report as a 2-point shift on a 100-point scale, where the answer is a published minimal clinically important difference or a quieter adjective. An adjusted model whose covariate coefficients are discussed as though each were a causal effect, which is the Table 2 fallacy. And a paper that now adjusts for confounding but still uses the verb “reduces” in the title.

How to use this

Read the page matching the objection, not the closest general advice. Reviewer comments are terse because the reviewer assumes you share their vocabulary, and the most common revision failure is answering a sentence rather than the concern behind it.

If you are revising, the overclaim check is worth running before you resubmit: a revision that fixes the statistics while leaving the abstract’s original claims intact draws the same objection a second time, from a reviewer who now believes you were not listening.

If you have not submitted yet, these are the objections worth pre-empting. PerfectPaper’s standing review covers statistical models, confidence-interval coherence and causal inference, and custom reviewer agents cover the field-specific conventions a general statistical reader misses.

PerfectPaper reads the reported n, the figure legends, the methods and the abstract against each other, and reports where the stated sample size, the test applied and the strength of the claim stop agreeing. Where the manuscript is high-dimensional, the multiple testing agent checks correction, threshold consistency and enrichment background; where measurements come from a hierarchy, a dedicated pseudoreplication pass reconstructs what each n counts from the methods and the figure legends.

Review my manuscript