What is an effect size?

An effect size is a number that states how large a difference, ratio or association is, on a scale a reader can interpret — 4.2 mmHg, a hazard ratio of 0.68, a Cohen’s d of 0.41. An effect size answers how much, whereas a p-value answers only how surprising the data would be if nothing were happening.

Those two questions are separable, and conflating them is among the statistical faults reviewers raise most often. A study can be overwhelmingly significant and describe a change too small for any patient to notice, or non-significant while pointing at a difference that would change practice if it held. This page covers the families of effect size and what each one is on the scale of, why standardisation both helps and misleads, why Cohen’s conventional benchmarks fail in most applications, why small significant studies systematically overstate magnitude, how the problem is detected in a real manuscript, and the phrasings reviewers use when it has been mishandled.

What an effect size measures that a p-value does not

An effect size and a p-value answer different questions, and only one of them is about magnitude. A p-value is a joint function of three things: the size of the underlying effect, the variability of the measurements, and the sample size. Because sample size enters, a p-value can be driven to any value by collecting more data, whatever the magnitude of the effect.

Consider a trial of 20,000 participants that reports p < 0.001 for a mean systolic blood pressure reduction of 0.3 mmHg. The p-value is correct and the finding is uninteresting: 0.3 mmHg is smaller than the measurement error of a cuff and smaller than the within-person variation between two consecutive readings. Now consider a 24-participant crossover study reporting p = 0.09 for a 9 mmHg reduction with a 95% confidence interval of −1.4 to 19.4 mmHg. That result is compatible with a clinically decisive effect and with nothing, and the correct summary is the interval rather than the verdict.

The effect size is the quantity that survives translation. Doubling the sample changes the p-value; it does not change what the population difference is, only how precisely you have estimated it.

Common questions about p-values and effect size

Why is my result significant but the effect size is tiny?

A result can be statistically significant with a trivial effect size whenever the sample is large enough, since a p-value is driven by sample size as well as magnitude. A 20,000-person trial can return p < 0.001 for a 0.3 mmHg blood pressure reduction, smaller than a cuff’s own measurement error. Significance says the effect probably isn’t zero; effect size says whether it’s worth anything.

If the effect didn’t change, why did my p-value get smaller with more data?

A p-value shrinks as sample size grows even when the true effect is unchanged, because it is a joint function of effect size, variability and sample size, never of magnitude alone. Doubling your sample sharpens the estimate’s precision and can push the same effect past any threshold; it does not make the difference larger. Report the effect size and its confidence interval, which do not move for this reason.

My advisor told me I have to report an effect size next to my p-value — can you explain in plain terms why the p-value alone isn’t enough?

A p-value states only how surprising your data would be if there were truly no effect; it says nothing about how large that effect is. Two studies can share an identical p-value while one describes a difference too small to matter and the other a difference large enough to change practice. The effect size and its interval, not the p-value, tell a reader whether the result matters.

The four families of effect size

Effect sizes fall into four families, and the first decision in reporting one is which family the question belongs to.

Differences on the original measurement scale. A mean difference of 4.2 mmHg, a median difference of 3 days in hospital, a risk difference of 1.8 percentage points, a difference in fluorescence intensity in arbitrary units. These are unstandardised, and where the scale is meaningful to the reader they are the most informative choice available.

Standardised differences. Cohen’s d expresses a mean difference in units of a pooled standard deviation. Hedges’ g applies a small-sample correction to d — the multiplier is approximately 1 − 3/(4·df − 1), which shrinks d by about 4% at n = 10 per group and under 1% at n = 50 per group. Glass’s Δ divides by the control group’s standard deviation alone, which is the right choice when the intervention plausibly changes the variance as well as the mean. A further distinction sits inside d itself: a within-subjects design standardised by the standard deviation of the difference scores yields a systematically larger d than the same two means standardised the between-subjects way, related by roughly d(within) ≈ d(between) / √(2(1 − r)) for a pre–post correlation r. Reporting a paired d without saying which denominator produced it is a common, quiet source of an effect that looks bigger than the raw means justify.

Ratios. Risk ratio, odds ratio, rate ratio, hazard ratio and fold change are multiplicative. A ratio of 1 means no effect, ratios are asymmetric around 1, and their confidence intervals must be computed on the log scale and back-transformed. Odds ratio and risk ratio coincide only for a rare outcome — conventionally under about 10% baseline incidence, the rare-disease approximation taught in epidemiology — and pull apart as the outcome becomes common; above roughly 20–30% baseline risk an odds ratio can overstate the corresponding risk ratio by a wide margin, which is exactly why the baseline rate has to travel with any odds ratio reported for a common outcome.

Association and variance explained. Pearson’s r, R², eta-squared, partial eta-squared, omega-squared and Cramér’s V describe how much of the variation in one variable travels with another. Squared measures are not on the same scale as unsquared ones: r = 0.30 and R² = 0.09 are the same fact stated twice.

A fifth, less used family describes overlap directly. The probability of superiority — the chance a randomly drawn treated observation exceeds a randomly drawn control observation — is Φ(d/√2) under normality, so d = 0.5 corresponds to about 64%, and d = 0.2 to about 56%. Non-parametric versions include Cliff’s δ and the Vargha–Delaney A statistic, which are appropriate for ordinal outcomes where a mean difference is not defined. The normal-theory formula also breaks down under skew or heavy tails — common in count and time-to-event outcomes — where the stated probability can be badly wrong even though d itself was computed correctly; Cliff’s δ and the Vargha–Delaney A, built directly from pairwise comparisons rather than from an assumed distribution, do not share that failure mode.

Standardisation helps comparison and hides the units

Standardising a difference divides it by a standard deviation, and the standard deviation is a property of the sample studied, not of the intervention. Two identical interventions can therefore report different effect sizes.

A 5 mmHg reduction in a general-population sample with a standard deviation of 20 mmHg gives d = 0.25. The same 5 mmHg reduction in a narrowly selected trial cohort with a standard deviation of 10 mmHg gives d = 0.50. Nothing about the drug changed. Restriction of range works in the opposite direction for correlations, attenuating r when the sample spans less of the underlying variable than the population does.

Two operational consequences follow. First, report the unstandardised effect whenever the scale carries meaning — millimetres of mercury, days, grams, counts — and add the standardised version only if a meta-analytic audience needs it. Second, treat any cross-study comparison of d as conditional on the two samples having comparable dispersion, and say so rather than assuming it.

Cohen’s benchmarks and why they mislead

Jacob Cohen proposed 0.2, 0.5 and 0.8 as small, medium and large values of d in Statistical Power Analysis for the Behavioral Sciences, and he framed them as conventions for use when no better basis for judgement exists, relative to the typical magnitudes of the behavioural sciences of his era. The benchmarks were never offered as a field-independent scale.

Applied outside that framing, they invert the substantive judgement in both directions. In a large cardiovascular prevention trial, a standardised effect near 0.1 can correspond to thousands of events prevented across a treated population and be worth the intervention. In a single-cell assay with an artificial dynamic range, d = 1.2 can reflect a change of no biological consequence, or a technical artefact of normalisation.

The alternative is not another set of thresholds. The alternative is an anchor argued in the manuscript: a minimal important difference from prior work, a comparison to the effect of an established treatment on the same outcome, the population-level implication of the difference, or a defined biological threshold. A stated anchor is arguable, and reviewers can disagree with it. “A medium effect by conventional standards” gives them nothing to engage with, which is why it reads as evasion.

Questions about Cohen’s d and interpreting standardised effects

Is a Cohen’s d of 0.5 actually a big deal, or is that just a textbook label?

Cohen offered 0.2/0.5/0.8 as small/medium/large in Statistical Power Analysis for the Behavioral Sciences, a fallback for when no better anchor exists — never a universal scale. A d of 0.5 can be trivial in a large prevention trial, irrelevant in an assay with an artificial range. Judge it against a stated anchor: a minimal important difference, a comparator treatment, a population-level implication.

Can I compare my Cohen’s d to a study with a totally different sample?

Only if the two samples have comparable variability — Cohen’s d divides the mean difference by that sample’s own standard deviation. A 5 mmHg reduction gives d = 0.25 in a broad population (SD 20) but d = 0.50 in a narrow cohort (SD 10); nothing about the effect changed, only the denominator did. Compare unstandardised differences across studies, and treat any cross-study d comparison as conditional on matched dispersion.

What effect size do I plug into a power calculation if I don’t have pilot data?

Use a minimal important difference from prior published work on the same outcome rather than a pilot estimate, because a small pilot’s effect size is drawn from the same biased, threshold-filtered distribution that inflates published results. Where no published anchor exists, a conservative choice — smaller than any effect you have seen reported for the outcome — protects the study from being powered on a number that was itself an overestimate.

Effect size estimates from small samples are biased upward

An effect size estimated from a small sample and filtered through a significance threshold is systematically too large. The mechanism is a selection effect rather than a computational error: with low power, only the larger sample estimates clear the threshold, so the published estimates are drawn from the upper tail of the sampling distribution.

Gelman and Carlin (2014) formalised this as the type M (magnitude) error — the expected factor by which a statistically significant estimate exaggerates the true effect — alongside the type S (sign) error, the probability that a significant estimate has the wrong sign. Both can be computed before data collection from a plausible effect size, and both grow sharply as power falls. Ioannidis (2008) made the same argument for the epidemiological literature under the heading of inflated discovered associations.

The empirical footprint is visible in replication work. The Open Science Collaboration’s 2015 replication of 100 psychology studies reported a mean original effect size of r = 0.403 against a mean replication effect of r = 0.197, with 97% of original studies significant and 36% of replications so. Shrinkage of that order is what an underpowered, threshold-filtered literature predicts.

Three practical responses exist. Use bias-corrected estimators where the correction is known — Hedges’ g rather than d, omega-squared rather than eta-squared. Design analyses with a defensible effect size rather than the one your pilot returned, since a pilot estimate is drawn from the same biased distribution. And treat a first, small, significant result as an upper bound rather than a point estimate, which is the honest framing when a reviewer asks whether the experiment has been replicated.

Common questions about small samples and effect size

Is n = 3 enough to trust the effect size I got?

An effect size from n = 3 per group that only cleared significance because it was unusually large is a textbook type M error candidate: with that little power, only inflated estimates pass the threshold at all. Treat it as an upper bound, not a point estimate, and say so in the discussion. A properly powered replication is the only way to know how much it actually shrinks.

Why did my effect size shrink when I reran the study with more people?

Shrinkage on replication is the expected signature of an underpowered original study, not evidence the second attempt went wrong. The Open Science Collaboration’s 2015 replication of 100 psychology studies found mean original r = 0.403 falling to mean replication r = 0.197, with 97% of originals significant against 36% of replications. The larger study is the more trustworthy estimate; the first number came from the tail of its sampling distribution.

Reviewer 2 wrote that my sample size is too small and my effect size is probably inflated — what should I actually put in my rebuttal?

Concede the mechanism rather than defend the number: with a small sample, a significant estimate is more likely to be large than an adequately powered study would produce — Gelman and Carlin’s type M error. Reframe the reported effect as an upper bound, name the direction of the bias, and commit to testing it in a properly powered follow-up rather than defending the original figure as published.

Precision belongs to the estimate

An effect size reported without an interval is half a result. The confidence interval states which population values the data are compatible with, and the width of that interval is frequently the more decisive fact.

CONSORT 2010 makes this explicit for trials. Item 17a requires, verbatim, “For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval).” The APA Publication Manual directs authors across the psychological literature to report effect sizes and, wherever possible, confidence intervals, and the APA Journal Article Reporting Standards list both among the elements a quantitative results section is expected to contain. The ICMJE Recommendations instruct authors not to rely solely on statistical hypothesis testing such as P values, which they describe as failing to convey important information about effect size and precision of estimates.

Two intervals with the same point estimate can support opposite conclusions. A hazard ratio of 0.85 with a 95% interval of 0.78 to 0.93 supports a modest, well-estimated benefit. A hazard ratio of 0.85 with an interval of 0.42 to 1.71 supports nothing at all, and describing the second as “a 15% reduction in hazard” is the sentence that draws the sharpest reviewer comment. Note also that a ratio measure is uninterpretable without its baseline: a hazard ratio of 0.68 against a 2% annual event rate and against a 40% annual event rate describe very different clinical situations, and the absolute figures should appear beside the ratio. The same applies to competing risks, where a cause-specific hazard ratio and a subdistribution hazard ratio answer different questions and are routinely reported as if interchangeable.

Statistical significance, practical significance, and the anchor in between

Practical significance is a claim about the world, and it requires a benchmark external to the dataset. The standard instrument is the minimal important difference: the smallest change on an outcome that patients or clinicians regard as worthwhile.

Named examples make the idea concrete. The 2014 European Respiratory Society and American Thoracic Society technical standard on field walking tests places the minimal important difference for six-minute walk distance in chronic respiratory disease at roughly 25 to 33 m, so a trial reporting a statistically significant 12 m gain has demonstrated a real but sub-threshold change and should say so. In event-driven work the anchor is often the number needed to treat: an intervention that moves a 4.0% event rate to 3.0% delivers a 25% relative reduction, a 1.0 percentage point absolute reduction, and a number needed to treat of 100 — three descriptions of one result, with very different rhetorical weight.

Minimal important differences are themselves contested. Estimates for the same instrument differ by anchor method, by baseline severity, and between improvement and deterioration, and some widely quoted values rest on single small studies. Cite the source and the population for any threshold you invoke rather than presenting it as a constant. Where no anchor-based minimal important difference has been published for the instrument in hand, a distribution-based shortcut — roughly half the outcome’s standard deviation — is sometimes substituted; that substitutes a statistical property of your own sample for a judgement about importance, the identical criticism levelled at Cohen’s fixed thresholds, so name it as a stand-in rather than present it as equivalent to an anchor-based estimate.

A related but distinct tool answers a different question: showing an effect is small enough to not matter, rather than merely failing to show it is large. A non-significant conventional test can only decline to reject “no effect” — it cannot support the positive claim that any true effect is below a meaningful threshold. An equivalence test (two one-sided tests, TOST, against a pre-specified smallest effect size of interest) is the tool built for that claim, and a discussion arguing “no difference was found” from a simple non-significant result, without one, is asserting more than the test can carry.

Effect sizes in laboratory and omics work

Preclinical and high-throughput studies report effect sizes under different names, and the same reasoning applies to each.

Fold change is a ratio effect size, usually reported as log2 fold change in transcriptomic and proteomic work. Raw fold changes computed from low-count features are unstable, which is why DESeq2 shrinks log2 fold changes with an empirical Bayes prior and limma moderates variance estimates across features. DESeq2 in fact offers more than one shrinkage estimator, and which one was used is worth asking: the original normal-prior shrinkage can over-shrink genuinely large fold changes, which is why the heavier-tailed apeglm and ashr estimators are now the more commonly recommended defaults for ranking and volcano-plot visualisation. Reporting unshrunken fold changes for lowly expressed genes reliably produces a multiple-testing and effect-inflation objection.

Percentage change in an image-derived measurement is an effect size on an arbitrary scale, and its interpretability depends entirely on the normalisation and the segmentation decisions upstream of it. A 40% increase in mean fluorescence intensity means little without the exposure settings, the background subtraction and the number of fields, which is the substance of most image quantification comments.

Difference between conditions in cultured cells is frequently reported with a denominator that counts wells or images rather than independent biological units. The effect size can be estimated correctly while its precision is fabricated — the defining feature of pseudoreplication, and the reason p-values shrink as more cells are counted without any change in the underlying magnitude.

Batch structure can manufacture an effect size outright. When condition is confounded with processing day, plate or sequencing run, the reported difference includes the batch difference, and no correction applied after the fact fully separates them. Batch effect scrutiny is therefore an effect size question, not only a normalisation question.

How mishandled effect sizes are detected in a manuscript

Detecting a mishandled effect size is a reading procedure rather than a statistical test, and it runs on the manuscript text as submitted.

Check that every claim in the abstract carries a magnitude. Abstract sentences of the form “treatment significantly improved X” with no number, matched against a results table where the difference is small, is a reliable signal.

Check the units of every reported effect size. A d, an r, an η² and a fold change reported in one results section without stating which is which, or a partial eta-squared labelled as eta-squared, indicates the values were taken from software output rather than chosen.

Check whether an interval accompanies every point estimate, and whether the discussion argues from the point estimate while the interval crosses the null.

Check the direction of the standardisation. If d is large while the raw difference is small, look for a restricted sample or a within-subject standard deviation used where a between-subject one belongs. Both inflate d without any change in the phenomenon.

Check that a ratio has its baseline. Relative risks, hazard ratios and fold changes presented without absolute rates or baseline levels cannot be judged, and the omission usually favours the authors’ preferred reading.

Check whether the interpretation rests on Cohen’s labels. The words “small”, “medium” and “large” appearing without an anchor from the literature indicate the practical-significance argument was never made.

Check the sample size against the claimed magnitude. A large standardised effect from a handful of units per group is a candidate type M error, and the discussion should acknowledge that rather than treat the estimate as settled. This is where an underpowered design and an overstated conclusion meet.

What a peer reviewer says when it has been mishandled

Reviewer phrasings for a mishandled effect size are stable across fields, and recognising them in advance is the point of reading them here.

“The authors report p-values throughout but no effect sizes or confidence intervals.” “The reported difference is statistically significant but its clinical importance is not established.” “The abstract states that the intervention improved outcomes without stating by how much.” “Effect sizes are described as medium and large by conventional criteria; the authors should justify these thresholds in the context of this outcome.” “Given the sample size, the observed effect is likely to be an overestimate, and the discussion does not acknowledge this.” “The odds ratio is reported for a common outcome and is being interpreted as a risk ratio.” “The confidence interval includes values consistent with no effect, and the conclusion should be tempered accordingly.”

The comment that most often decides a paper is the last one in substance rather than the first: a reviewer who accepts your numbers but rejects your reading of them. Anticipating it is a matter of aligning the discussion with the interval instead of the point estimate, which is also what an overclaim check looks for. When a reviewer has already written that the effect size is not meaningful, the response required is an anchor, not a defence of the p-value.

How to report an effect size well

Report the unstandardised effect with its units first, then the interval, then the standardised version if the audience needs it: “mean difference 4.2 mmHg (95% CI 1.1 to 7.3), d = 0.31”. Name the estimator explicitly — Hedges’ g rather than “effect size”, omega-squared rather than “variance explained” — and state the denominator used for any standardisation.

Give the baseline alongside every ratio, and the absolute difference alongside every relative one. State the anchor by which you are judging importance and cite where it comes from. Where the study was powered on an assumed effect, report that assumption and compare it to what was observed, rather than converting the observed result into retrospective power, which is a deterministic function of the p-value and adds no information.

Finally, write the discussion from the interval. If the interval spans trivial and substantial values, the sentence that follows is “the data are compatible with effects ranging from negligible to clinically important”, and reviewers of statistical objections reward that sentence far more than they reward confidence.

Contested ground and honest limits

Several points in the reporting of effect sizes are genuinely unsettled, and a manuscript that pretends otherwise reads worse than one that names them.

Whether standardised effect sizes should be used at all in clinical research is disputed: they permit meta-analysis across instruments and simultaneously invite comparisons across incomparable populations. Which variance-explained measure to prefer is disputed, with partial eta-squared dominant in practice and criticised for not being comparable across designs with different factor structures. Minimal important differences differ by derivation method for the same instrument. And the magnitude of literature-wide inflation is estimated rather than known — replication programmes measure it in particular fields under particular selection rules, and those estimates do not transfer cleanly to yours.

None of that licenses silence. Each is a bounded uncertainty that can be stated in a sentence with its source, and stating it is what distinguishes a discussion section a reviewer trusts from one they audit.

Related

When a reviewer says the effect size is not meaningful · Post-hoc power · Sample size too small · Pseudoreplication · Confounding · Statistical objections

Checked before submission by overclaim detection, which compares the magnitude in the results table against the strength of the claim in the abstract and discussion, and names the sentences that outrun the interval.

Review my manuscript

Frequently asked questions

Why does my odds ratio look way bigger than my risk difference for the same result?

An odds ratio diverges from a risk ratio once the outcome is common, since odds and risk coincide only for rare events, conventionally under a 10% baseline rate. Above that, an odds ratio overstates a risk ratio, sometimes substantially — exactly the reviewer comment “the odds ratio is being interpreted as a risk ratio.” Report the absolute rate in each group beside any ratio so the true magnitude is visible.

Do I need to report an effect size if my result wasn’t significant?

Yes — a non-significant result with no effect size is uninterpretable, since it cannot distinguish “no effect” from “not enough data to see one.” A 24-participant crossover trial reporting p = 0.09 for a 9 mmHg reduction with a 95% CI of −1.4 to 19.4 mmHg is compatible with a decisive effect and with nothing. Report the effect size and interval regardless of the p-value.

Is a t-value or an F-statistic an effect size?

No — a test statistic and a p-value both depend on sample size, while an effect size is defined not to. An absolute risk difference, a number needed to treat, a log2 fold change and a standardised mean difference are effect sizes; t, F, χ² and the p-value itself are not, because doubling the sample changes all four without changing what the population difference actually is.