Reviewer asked for a post hoc power calculation

Post hoc power — a power calculation run after a study, using the effect size the study itself observed — answers nothing, because it is a strictly decreasing function of the p-value: a non-significant result must return low power and a significant one high power, regardless of sample size. What it looks like is evidence; what it is, is the p-value restated in different units.

The concern behind the reviewer’s request is usually legitimate, and you should still answer it — just not with the calculation requested.

Why observed power says nothing

Observed power is calculated by assuming the true effect equals the one you measured. But your measured effect is itself uncertain, and it is the thing under dispute. Assuming it is correct in order to evaluate whether your study could have detected it takes as given exactly what the reviewer is questioning.

The arithmetic consequence is mechanical: observed power and the p-value are one-to-one, and the mapping does not depend on how many subjects you enrolled. If p is 0.049 your observed power will be near 50%, always, regardless of the science. Reporting that number as reassurance is reporting the p-value twice.

The relationship is not an approximation. For a two-sided test at alpha = 0.05, observed power is Φ(|Z| − 1.96) + Φ(−|Z| − 1.96), where Z is the test statistic that produced your p-value. Sample size, variance and effect size enter only through Z. Two studies with the same p-value have the same observed power whether one enrolled 18 animals and the other 1,800 patients. The constant changes with the test — a one-sided test at alpha = 0.05 replaces 1.96 with 1.645, and a two-sided test at alpha = 0.01 replaces it with 2.576 — but the property that only Z, equivalently only the p-value, determines the result holds for any alpha and any one- or two-sided test, and the same closed-form collapse generalises to chi-square and F-tests, only the specific function of the test statistic changes. Hoenig and Heisey set this out in “The Abuse of Power” (The American Statistician 2001;55:19–24), the paper most reviewers are half-remembering when they ask for the calculation and the paper you should cite when you decline it.

The same result has been published repeatedly in clinical and pharmacy literature, which is useful when the reviewer’s field is not statistics. Goodman and Berlin argued in Annals of Internal Medicine (1994;121:200–206) that power is a pre-experimental concept and that confidence intervals are the post-experimental tool; Levine and Ensom titled their Pharmacotherapy paper (2001;21:405–409) “Post hoc power analysis: an idea whose time has passed?”; Senn’s BMJ letter (2002;325:1304) is headed “Power is indeed irrelevant in interpreting completed studies.” Greenland went one step further in Annals of Epidemiology (2012;22:364–368), showing that non-significance combined with high power does not imply support for the null over the alternative — so even the version of the argument that runs in the reviewer’s favour does not work.

Two things follow for the response letter. A reviewer who knows this result will read a supplied observed-power figure as a sign that the objection was not understood, and CONSORT 2010 says in as many words that the calculation has little merit — the quotation is later on this page, and it is the citation to lead with.

Common questions about post hoc power

What is post hoc power?

Statistical power calculated after a study using the effect size observed in that study, rather than an effect size specified in advance. Because it assumes the observed effect is the true one, it is determined entirely by the test statistic that produced your p-value, and carries no information independent of it.

Is post hoc power the same thing as observed power or retrospective power?

Yes. Post hoc power, observed power, and retrospective power name the same calculation — one computed from the study’s own p-value rather than from an independent effect estimate. Reviewers, editors and submission templates use all three terms interchangeably, so a request for any one of them is a request for the identical circular quantity.

Why did my post hoc power come out at almost exactly 50% when my reviewer called the result borderline?

Because observed power at alpha = 0.05 sits at exactly 50% whenever your two-sided p-value equals 0.05, whatever your sample size, effect size, or design. A p-value just under 0.05 always returns power just over 50%; a p-value just over 0.05 always returns power just under it. The figure is reporting your p-value, not your study’s sensitivity.

The arithmetic, with numbers

Observed power at alpha = 0.05, two-sided, maps onto the p-value as follows. Nothing else about the study changes these figures.

Read the table as a warning rather than a lookup. The third column contains no information the second does not, and the second contains none the first does not. A referee who asks you to compute the middle column, having already read the left one, is asking for a restatement and will treat it as a finding. That is the specific harm: the number acquires rhetorical weight it has not earned, and once it is in the manuscript an editor may act on it.

Note the floor. Observed power can never fall below alpha, which is why a p-value of 0.80 still returns 5.7% rather than zero. A “power” figure that is arithmetically incapable of dropping below 5% is not measuring anything about your design.

What the reviewer is actually worried about

The request for post hoc power is usually a proxy for one of three substantive concerns, and identifying which one you have received determines the entire response.

The result is null and the reviewer suspects the study could not have seen a real effect. This is a question about precision, and the honest answer is a confidence interval plus a statement of what the design could detect. It is a fair question and you should answer it fully.

The paper reads non-significance as absence of effect. The reviewer wants the claim rewritten, not a calculation. Handle it in the discussion and abstract, and say what the data are compatible with rather than what they failed to show; how to write a limitations section covers the paragraph that usually needs rewriting.

The result is positive on a small sample and the reviewer suspects the effect is inflated. This is the strongest version of the concern and post hoc power actively conceals it, because a significant result mechanically returns high observed power. Statistically significant estimates from low-powered designs are biased upward in magnitude — Gelman and Carlin call this a type M (magnitude) error, alongside type S (sign) errors, in “Beyond Power Calculations” (Perspectives on Psychological Science 2014;9:641–651). The remedy is a design analysis using a plausible external effect size, not a calculation fed by your own estimate. Where the concern is really about whether the finding holds up at all, reviewer says the experiment was not replicated is the adjacent comment.

What to give instead

What to give instead of post hoc power depends on which of the three concerns above prompted the request, but four replacements cover nearly every case: a confidence interval on the effect, an equivalence or non-inferiority test, a prospective calculation if one exists, and a precision or design-analysis statement.

A confidence interval on the effect. This is the direct answer to the question the reviewer is actually asking, which is what effect sizes your study can rule out. A 95% interval from a 2% decrease to a 30% decrease tells the reader the data are compatible with a trivial effect and with a large one — which is honest, informative, and impossible to extract from a power figure. Report it on the scale readers care about, and pair it with a minimal clinically important difference so the interval can be read against something. The relevant question is whether the interval excludes an effect worth having, which is a property of the interval and not of the p-value; what is a confidence interval covers the interpretation reviewers most often want corrected.

An equivalence or non-inferiority framing, if your claim is that there is no difference. A non-significant p-value is not evidence of no effect. If you want to claim equivalence, test for it against a pre-specified margin and say what margin you chose and why. The standard procedure is the two one-sided tests of Schuirmann (Journal of Pharmacokinetics and Biopharmaceutics 1987;15:657–680), operationally equivalent to checking whether the 90% confidence interval lies entirely inside the margin at alpha = 0.05. Declaring the margin after seeing the interval is the failure mode reviewers catch; what is non-inferiority sets out how margins are justified. Note the asymmetry in how the result reads: a 90% interval entirely inside the margin supports equivalence, one entirely outside it supports a real difference, and one that straddles the margin boundary is inconclusive rather than a failed test — say which of the three you have, not just whether the test passed.

A prospective calculation, if you did one. Report the assumed effect, its source, alpha, power and the resulting group size. A study powered for a pre-specified effect is defensible even when the observed effect turns out smaller — that is what prospective calculation is for. Give the source of the assumed effect explicitly: a named prior study, a pilot, a regulatory margin, or a smallest effect of interest agreed in advance. “Based on previous work” without a citation invites the follow-up question.

A precision statement. “With n=12 per group, this study could detect a difference of 1.2 standard deviations with 80% power” is a legitimate sensitivity statement, because it is computed from the design rather than from the result. Frame it as what the design could detect, not as how powerful the study turned out to be. Lenth’s “Some Practical Guidelines for Effective Sample Size Determination” (The American Statistician 2001;55:187–193) is the standard reference for stating the detectable effect rather than retrospective power, and gives you something to cite that is not a rebuttal.

A design analysis, when the finding is positive and the sample is small. Take an effect size you would consider plausible from outside your own data, and report the probability that an estimate reaching significance at your sample size would exaggerate the true effect, and the probability it would have the wrong sign. This answers the inflation concern directly and turns a defensive paragraph into an analysis.

Questions about responding to a reviewer’s request

Reviewer 2 wants a post hoc power calculation — what do I actually send back?

Decline the specific calculation, state in one sentence that it is a deterministic function of the reported p-value, and supply a confidence interval for the effect in its place. Cite CONSORT 2010 item 7a, which says post hoc power has little merit and that confidence intervals are the appropriate substitute. That combination reads as a defensible position, not a refusal.

Can I just refuse to run the post hoc power calculation in my rebuttal letter?

Yes, provided you explain why and supply a substitute analysis in the same paragraph. Editors accept a reasoned decline paired with a confidence interval, an equivalence test, or a design-based precision statement; what they do not accept is silence, or a revised manuscript with no explanation for the missing calculation.

My journal’s submission form has a required field for post hoc power — do I have to fill it in?

Fill it in rather than leaving it blank, but add a one-line interpretive note — for example, that the calculation is a deterministic function of the p-value and that a confidence interval is reported instead. A blank field usually routes to an automated technical check; a completed field with a caveat routes to the editor, who can read the context.

A worked example a reviewer can check

Consider a two-arm trial with 40 participants per group, a continuous outcome, an observed mean difference of 1.8 points and a pooled standard deviation of 5.0 points. The standard error is 1.12, t = 1.61 on 78 degrees of freedom, and p = 0.11. (With markedly unequal group variances, a Welch t-test would be the more defensible choice than this pooled-variance calculation — it does not change the conclusion about post hoc power, only the arithmetic feeding it.) The reviewer writes: “The study appears underpowered; please provide a post hoc power calculation.”

Observed power for that result is 36%. It is 36% because p is 0.11. Had the same trial enrolled 400 per group and returned p = 0.11 for a difference of 0.57 points, observed power would still be 36%. The number describes the p-value, not the trial.

Now the informative version. The 95% confidence interval is −0.43 to 4.03 points. If the minimal clinically important difference on this scale is 3.0 points, the data are compatible with a 4-point benefit and with a small harm: the trial has not excluded a clinically important effect, so it is inconclusive rather than negative, and the discussion must say so. The design-based statement is that with 40 per group and a standard deviation of 5.0 points, the trial had 80% power to detect a difference of 3.2 points — marginally larger than the effect that would matter, which is a real and reportable design limitation.

Equivalence fails too, and it is worth saying why in the response. The 90% confidence interval runs −0.06 to 3.66 points, which does not lie inside a ±3.0-point margin, so the two one-sided tests do not support equivalence at that margin. Three sentences of arithmetic have told the reviewer everything the requested calculation could not: what the trial can exclude, what it cannot, and which of those is the problem.

How to word the response

The response to a post hoc power request should decline the calculation, give the reason in one clause, and offer a substitute analysis in the same breath — direct and unapologetic, with a citation attached. A workable shape:

We have not included a post hoc power calculation. Because such a calculation uses the observed effect as the assumed true effect, it is a deterministic function of the reported p-value and provides no additional information. In its place we now report the 95% confidence interval for each primary comparison (Table 2), which conveys the range of effects compatible with our data, including the smallest effect the study can exclude.

Note what that paragraph does. It declines the calculation, gives the reason in one clause, and puts a replacement analysis on the table in the same breath — which is a defensible statistical position rather than a refusal, and is the form an editor can act on without adjudicating a methods dispute.

If you ran a prospective calculation, lead with it instead, because it converts the exchange from a disagreement into a correction:

The sample size was fixed in advance: 40 per group provides 80% power at alpha = 0.05 (two-sided) to detect a 3.0-point difference, assuming a standard deviation of 5.0 points taken from Smith et al. (2021). We have added this to the Methods (page 6). We have not added a retrospective calculation based on the observed difference, since for a completed study that quantity is a function of the p-value alone (Hoenig and Heisey, 2001); the corresponding confidence interval is now reported in Table 2.

Three phrasings to avoid. Do not write “the study was underpowered to detect the observed effect”, which is the circularity restated as an admission. Do not write “achieved power was 36%, indicating that a larger sample is needed”, which invites a rejection you did not have to accept. Do not write “no significant difference was found, indicating the treatments are equivalent” anywhere in a paper where the reviewer has already raised power — that sentence is what produced the comment. General structure for the letter is in how to write a response to reviewers.

If the reviewer insists

A reviewer who repeats the request after a clear explanation is usually asking to be shown that the concern was taken seriously, not to be persuaded on the statistics. Three options, in order of preference.

Provide the confidence interval prominently and add the requested calculation in supplementary material with a one-line note on its interpretation. This satisfies the letter of the request without giving the number rhetorical weight.

Offer the substitute the reviewer will recognise as responsive: a sensitivity analysis reporting the effect detectable at 80% power given the realised sample size and observed variance, labelled as such. A reviewer asking for “power” generally wants a statement about what the study could see, and this is that statement computed legitimately. Naming it “sensitivity analysis” or “detectable effect size” rather than “post hoc power” matters, because the labels carry different methodological commitments.

Or ask the editor to adjudicate. If your response letter has already explained the circularity clearly, editors will usually side with the statistics. Where the journal’s own submission template contains a field for post hoc power, fill it and add the interpretive footnote rather than leaving it blank — a blank field routes to a technical check, and a completed field with a caveat routes to the editor.

A request that persists after a clear rebuttal sometimes originates from a journal’s assigned statistical reviewer rather than the subject-matter reviewer who wrote the original comment. A statistical reviewer is generally more likely to accept the CONSORT citation on its methodological merits than to be argued out of the position by prose alone, so lead a second reply with the quotation rather than a restated explanation.

When a post hoc calculation is legitimate

A calculation performed after data collection is legitimate whenever the effect size entering it comes from outside the data being analysed. That single test separates the defensible cases from the circular one.

Detectable effect size at the realised sample size. Using the achieved n and the observed variance, report the smallest effect the design could detect at 80% power. Variance is a nuisance parameter rather than the quantity in dispute, so reusing it is not circular in the way reusing the effect is. State it as a design property.

Power at a pre-specified effect, computed with the realised n. If enrolment fell short of target, recomputing power for the originally specified effect at the sample actually obtained is a valid and often requested amendment. Report both figures and the shortfall.

Power for a future study. Using this study’s variance estimate to size the next experiment is exactly what pilot data are for. Say that the effect estimate is not being carried forward, and use a smallest effect of interest instead; carrying a small study’s point estimate into the next calculation is how underpowered literatures perpetuate themselves.

Design analysis under an assumed effect. Type S and type M calculations require you to name a plausible effect from prior literature or theory, which is what makes them informative about exaggeration. Unlike a standard power analysis, this framework has no single widely adopted software package and no accepted probability threshold for “too exaggerated” — report the assumed effect size and its source as prominently as the resulting probability, because the assumption is doing most of the argumentative work.

The one calculation that fails the test is the one the reviewer asked for: power evaluated at your own observed effect. Everything above changes the input; nothing above changes the timing.

Questions about prospective and legitimate power calculations

Is a prospective power calculation different from post hoc power?

Entirely. A prospective calculation uses an effect size specified before the data exist, usually from prior literature or a minimum effect of interest, and it legitimately justifies a sample size. The problem is only with calculations that take the observed effect as input.

Can I calculate power after the fact if I use an effect size from a different study, not my own data?

Yes — that is a design analysis, not post hoc power in the circular sense. Naming a plausible effect from prior literature or theory, rather than your own point estimate, is exactly what makes type S and type M error calculations informative. The distinction is the source of the effect size, not the timing of the calculation.

My enrollment fell short of the planned sample size — can I recompute power for the original effect using the n I actually got?

Yes, and journals often ask for exactly this amendment. Recompute power for the effect size specified before the study, using the sample size actually obtained, and report both the shortfall and the resulting power figure. This differs from post hoc power because the effect size still comes from outside the data; only the n changes.

How to catch the problem in your own draft

Search the manuscript for the phrases that mark it before a reviewer does. “Observed power”, “achieved power”, “post hoc power”, “retrospective power” and “the study was underpowered to detect the observed difference” all indicate a circular calculation already in the text — often added in response to an earlier reviewer at a different journal.

Then search for the claim that invites the comment. “No significant difference”, “comparable between groups”, “similar in both arms” and “there was no effect of” are the standard triggers when they appear without an interval. A null result described in words but not bounded by numbers reads to a statistical reviewer as a claim the data have not established, and the power comment is the usual way that objection arrives; the underlying issue is often reviewer says the effect size is not meaningful rather than power at all.

Check finally that your denominator is the one you think it is. Power arguments collapse when the analysed n includes technical replicates, repeated measures on the same subject or multiple cells from one animal, because the independent units are fewer than the count in the table — see what is pseudoreplication. The distinction that matters is between technical replicates (repeated measurements on the same biological unit) and biological replicates (independent units): only the count of biological replicates belongs in the denominator of any power argument, observed or prospective. A power calculation on the wrong unit is wrong whichever direction it points.

Where the reporting guidance stands

CONSORT 2010 addresses this directly. The Explanation and Elaboration document (Moher et al., BMJ 2010;340:c869), under item 7a on sample size, states: “There is little merit in a post hoc calculation of statistical power using the results of a trial; the power is then appropriately indicated by confidence intervals (see item 17).” That sentence is quotable in a response letter and carries more weight with an editor than a methodological argument in your own words; the CONSORT checklist covers what item 7a asks you to report instead.

ARRIVE 2.0 takes the same line for animal work by asking only for the prospective version. Item 2b reads: “Explain how the sample size was decided. Provide details of any a priori sample size calculation, if done.” The words are “a priori”; nothing in the ARRIVE guidelines asks for a retrospective figure.

Neither document is a journal policy. Both are voluntary reporting guidelines that a given journal may or may not have adopted, and neither extends by its own terms to work outside a randomised trial or an animal experiment — an observational study or an in vitro paper sits outside both checklists’ stated scope. Cite CONSORT and ARRIVE as the strongest available methodological consensus on the point, not as a rule that binds this specific journal or this specific design.

State the limit of this accurately rather than overstating it. There is no cross-journal prohibition on post hoc power, individual journals differ in whether their statistical guidance mentions it at all, and reviewers who request it are usually following custom rather than a stated policy. What exists is a consistent methodological literature spanning 1994 to 2014 — Goodman and Berlin, Hoenig and Heisey, Lenth, Levine and Ensom, Senn, Greenland, Gelman and Carlin — and one explicit sentence in CONSORT. That is enough to decline on, and it is more than most reviewers expect you to have.

Before you resubmit

If the underlying concern was that your study is small, reviewer says my sample size is too small covers the three things that comment can mean. If the concern was that non-significance was reported as absence of effect, check the discussion — that is a claim problem rather than a power problem, and the overclaim check is the pass for it. If the comment arrived from a trials submission alongside protocol and registration queries, clinical trial review covers the cluster it usually travels in.

More objections in this family: responding to statistical reviewer comments.

Frequently asked questions

Does low observed power mean my study was underpowered?

Not on its own, because observed power at alpha = 0.05 is 50% whenever p equals 0.05 and falls as p rises, whatever the sample size. Whether the study was underpowered is answered by the width of the confidence interval relative to the smallest effect worth detecting, not by a power figure derived from the result.

I got a significant result with only 8 animals per group — could the reviewer be right that it’s a fluke even though p is under 0.05?

Possibly, and post hoc power cannot tell you either way, because a significant result always returns high observed power regardless of the truth. What answers the concern is a type M error analysis: using a plausible external effect size, estimate the probability that a significant result at your sample size would exaggerate the true effect, and report that instead.

Can I decline to run the post hoc power calculation at all?

Declining a specific analysis is acceptable when you explain why and offer an alternative that addresses the underlying concern. What editors do not accept is silence, or a changed manuscript with no explanation. A refusal with a reason and a substitute analysis is ordinary scientific disagreement, and for this particular calculation the reason is short enough to state in two sentences.

Related

What is statistical power · What is a confidence interval · What is effect size · Reviewer says the sample size is too small · Statistical review capacity

PerfectPaper flags a power figure computed from the observed effect, checks whether each null claim in the discussion is supported by an interval that excludes a meaningful effect, and reports the smallest effect the design could detect at the sample size actually analysed.

Review my manuscript