What is multiple testing correction?

Multiple testing correction is the adjustment of significance thresholds or p-values when more than one hypothesis is tested, so that the chance of a false positive across the whole set stays at the level you intended to accept. Twenty independent tests at α = 0.05 yield at least one spurious result 64% of the time; correction restores control over that inflated error rate.

Correction is arithmetically simple and methodologically contested. The formulae take a line each; the difficult parts are deciding which set of tests forms the family, which error rate is the one worth controlling, and what the correction cannot repair. This page covers all three, along with how mishandled multiplicity is detected in a submitted manuscript and what reviewers write when they find it.

The arithmetic that creates the problem

A single test at α = 0.05 has a 5% chance of a false positive when the null is true. Run m independent such tests under the global null and the probability that at least one is significant is 1 − 0.95^m.

That function rises faster than most authors expect. Five tests give 22.6%. Fourteen tests cross one-half, at 51.2%. Twenty give 64.2%. One hundred give 99.4%.

Ordinary papers reach these numbers without anyone noticing. Three outcomes measured at two timepoints in four subgroups is 24 comparisons. A figure with six panels, each carrying three pairwise asterisks, is 18. A transcriptomic screen is 20,000 in one command. The number of tests a paper performed is almost always larger than the number a reader can count from the results section, which is why stating m explicitly is more informative than naming a method.

Correlation between tests reduces the inflation but does not remove it. Two strongly correlated outcomes still behave as more than one independent test, so describing a correlated family as though it were a single comparison is not a defence. Only when two tests are perfectly correlated do they collapse into one, and outcomes that perfectly correlate are the same measurement under two names.

Family-wise error rate versus false discovery rate

Multiple testing correction controls one of two different quantities, and the choice between them is a scientific decision rather than a statistical one.

The family-wise error rate (FWER) is the probability of making at least one type I error anywhere in the family. Controlling FWER at 0.05 means that if every null hypothesis in the family were true, there is a 5% chance the study produces any false positive at all. This is the standard for confirmatory claims: a regulatory approval, a primary trial endpoint, a single headline finding.

The false discovery rate (FDR), introduced by Benjamini and Hochberg in the Journal of the Royal Statistical Society Series B in 1995, is the expected proportion of false positives among the hypotheses you reject. Controlling FDR at 0.05 with 500 discoveries means roughly 25 of them are expected to be wrong, and you have decided that is an acceptable trade for the other 475.

The two coincide in exactly one situation: when every null hypothesis is true, no rejection is correct, so any rejection makes the false discovery proportion 1, and FDR control collapses to FWER control. Everywhere else FDR is the more permissive criterion, and its permissiveness increases with the number of genuinely non-null tests.

The practical rule follows from what happens next. Use FWER when a positive result ends the investigation and becomes a claim. Use FDR when a positive result begins one and will be validated — a screen whose hits go to the bench. A ranked gene list at FDR 5% is a work order, not a finding.

Methods that control the family-wise error rate

Bonferroni compares each p-value to α/m, or equivalently multiplies each p-value by m. Its guarantee follows from Boole’s inequality and therefore holds under any dependence structure whatsoever, which is why it survives despite being the least powerful option. The correction carries the name of Carlo Emilio Bonferroni, but its use for multiple comparisons traces to Olive Jean Dunn, “Multiple comparisons among means”, Journal of the American Statistical Association 1961;56:52–64.

Šidák uses 1 − (1 − α)^(1/m), which is exact under independence rather than merely conservative. The gain is negligible in practice: at m = 20 and α = 0.05, Šidák gives 0.002561 against Bonferroni’s 0.002500, a difference of 2.4%. Choosing between Bonferroni and Šidák is never the decision that changes a paper’s conclusions.

Holm’s step-down procedure (Sture Holm, Scandinavian Journal of Statistics 1979;6:65–70) sorts p-values ascending and compares the smallest to α/m, the next to α/(m − 1), and so on, stopping at the first failure. Holm controls FWER under arbitrary dependence and rejects every hypothesis Bonferroni rejects, plus sometimes more. There is no situation in which plain Bonferroni is statistically preferable. Bonferroni persists because it yields one printable threshold and because it inverts directly into simultaneous confidence intervals, which step-down procedures do not.

Hochberg’s step-up procedure (Biometrika 1988;75:800–802) works from the largest p-value downward and is uniformly more powerful than Holm, while requiring independence or positive dependence rather than none.

Permutation-based maxT (Westfall and Young, 1993) estimates the null distribution of the largest test statistic by resampling, so the correction reflects the actual correlation among tests rather than assuming the worst case. It is the least conservative valid FWER method when tests are heavily correlated, and it demands heavy computation together with observations that are exchangeable under the null.

Methods that control the false discovery rate

The Benjamini–Hochberg procedure sorts the m p-values ascending, finds the largest rank k for which p₍ₖ₎ ≤ (k/m)q, and rejects every hypothesis from rank 1 to k. The step is one line of code and the result is a set, not a threshold — which is why quoting “the BH threshold” for a study is usually a sign the procedure was misunderstood.

Benjamini–Hochberg control holds under independence and under positive regression dependency, a condition established by Benjamini and Yekutieli in the Annals of Statistics 2001;29:1165–1188. The same paper gives a procedure valid under arbitrary dependence, which divides the threshold by the harmonic sum Σ(1/j) from j = 1 to m. For m = 20,000 that sum is about 10.5, making Benjamini–Yekutieli roughly ten times stricter than Benjamini–Hochberg. This is why genomics uses Benjamini–Hochberg almost universally while rarely checking whether its dependence condition holds: the honest alternative is unusable, and correlated genes generally satisfy positive dependency well enough in practice.

Storey’s q-value refines the procedure by estimating π₀, the proportion of hypotheses that are truly null, rather than assuming it is 1. When a substantial fraction of tests are non-null — a well-powered differential expression experiment with strong biology — the gain in discoveries is real. When π₀ is close to 1, the q-value and the Benjamini–Hochberg adjusted p-value are nearly identical.

An adjusted p-value from any step procedure is monotone in the ranked order. A results table in which BH-adjusted values are not non-decreasing as raw p-values increase contains an arithmetic error, and that is checkable from the published table alone.

Defining the family does more work than choosing the method

The family, in any multiple testing correction, is the set of hypotheses over which the error rate is controlled, and it is chosen by the analyst rather than determined by the data. Nothing in any correction formula tells you what belongs in m.

Reasonable people define it differently. All tests in a paper? All tests addressing one research question? All tests on one outcome across all subgroups? Every analysis a laboratory has ever run on a dataset? Each answer produces a different threshold from identical data.

The decision has more influence on the result than any choice among Bonferroni, Holm and Hochberg, and it is almost never stated. A methods section reading “p-values were Bonferroni-corrected” is uninterpretable, because the reader cannot recover m. The defensible convention is to define the family as the set of tests that together support one claim, then to declare that set before analysis and state it in the paper.

Two corollaries follow. A primary endpoint and its supporting secondary endpoints belong to one family; unrelated outcomes reported in the same paper do not. And an exploratory analysis of 40 variables cannot be rescued by correcting the three that survived, because the family is what was tested, not what was reported. That gap is the most common form of the error in submitted manuscripts and the hardest for a reader to detect without a registered protocol.

Multiplicity in confirmatory clinical trials

Regulatory practice treats multiplicity as a design problem rather than an analysis step. The FDA’s final guidance Multiple Endpoints in Clinical Trials, issued 21 October 2022 and finalising the draft of 13 January 2017, states that as the number of endpoints analysed in a single trial increases, the likelihood of false conclusions about a drug’s effects becomes a concern without appropriate adjustment for multiplicity.

The strategies it describes are structural. Alpha splitting divides the 0.05 across co-primary endpoints, for instance 0.04 and 0.01, in proportions fixed before unblinding. Fixed-sequence (hierarchical) testing orders the endpoints and tests each at full alpha, continuing only while each preceding test succeeds — it spends no alpha at all, and in exchange a failure at step one makes every later result non-confirmatory regardless of its p-value. Gatekeeping and graphical approaches generalise this, allowing alpha to be recycled from a rejected hypothesis to others in a pre-specified graph.

The ordering in a hierarchical scheme is therefore the whole design decision, taken in advance, on scientific grounds, with no opportunity to revise it once the data are seen. Trials that put a fragile endpoint first have forfeited the sequence before enrolling anyone.

Safety endpoints are conventionally not corrected, and this asymmetry is deliberate rather than an oversight. Correction reduces false positives by accepting more false negatives; for harm, the false negative is the error that matters, so multiplicity adjustment on adverse-event analyses works against the purpose of the analysis. Reviewers who demand Bonferroni on a safety table have inverted the logic. Registered endpoint structure is checked before submission by trial registration and endpoint discipline.

Multiplicity in high-dimensional biology

Genome-wide association studies use a fixed threshold of 5 × 10⁻⁸, which is 0.05 divided by one million. That million is not the number of variants genotyped — arrays test hundreds of thousands and imputation tens of millions — but the effective number of independent common-variant tests given linkage disequilibrium in European-ancestry samples. Pe’er and colleagues derived it in 2008 by extrapolating from the resequenced HapMap ENCODE regions; Dudbridge and Gusnanto, extrapolating genotype density to the infinite limit in the same year, obtained approximately 7.2 × 10⁻⁸.

That derivation carries a consequence rarely acted upon: the threshold is ancestry-specific. African-ancestry samples have shorter linkage disequilibrium blocks and therefore more independent tests, so 5 × 10⁻⁸ is not conservative in the same degree there. Applying the European figure to a multi-ancestry study without comment is a defensible convention but not a neutral one, and saying so in the manuscript is better than letting a reviewer raise it.

Transcriptomics, proteomics and metabolomics use Benjamini–Hochberg across all features tested, and the recurring failure is not the procedure but the denominator: filtering low-expression features before correction is legitimate only when the filter is independent of the test statistic, and filtering by the outcome-related signal biases the entire FDR calculation. Correlated false positives from batch structure are a separate problem entirely, addressed under batch effects, and no multiplicity method removes them because they are not chance findings. Omics multiplicity specifically is checked by multiple testing in omics.

Neuroimaging supplies the field’s clearest cautionary result. Eklund, Nichols and Knutsson, in PNAS 2016;113:7900–7905, ran three million random task group analyses on real resting-state data and found that the parametric cluster-wise inference in SPM, FSL and AFNI produced familywise error rates reaching 70% against a nominal 5%. The correction was being applied; its distributional assumptions were wrong.

What multiple testing correction cannot fix

Multiple testing correction addresses one failure mode — chance findings among a fixed, pre-specified set of tests — and leaves several others untouched.

Optional stopping. Every correction assumes m was fixed before the data were seen. Adding subjects until p drops below 0.05 inflates type I error independently of any adjustment, and the adjustment cannot detect it.

Analytic flexibility. Gelman and Loken’s garden of forking paths, described in American Scientist in 2014, is the case where only one analysis was run but many were available. The effective number of tests is then unknown and uncountable, so no denominator exists to correct with. Pre-registration addresses this; correction does not.

Non-independent observations. Treating 300 cells from three animals as n = 300 inflates false positives through the standard error, not through multiplicity, and the corrected p-value is as wrong as the raw one. See pseudoreplication and my p-value shrinks with more cells.

Systematic bias. Correction controls the rate of chance findings under the null. Confounding, selection effects and measurement error produce associations that are not chance findings at all, and a Bonferroni-surviving confounded estimate is simply a confounded estimate with a smaller p-value attached.

The wrong test. A correction applied to a test whose assumptions are violated propagates the original error. See reviewer says I used the wrong statistical test.

Interpretation. A result that survives correction has cleared a threshold, not acquired importance. Effect size and interval estimation carry that information; see reviewer says the effect size is not meaningful.

Whether to correct at all, honestly stated

The requirement to correct is genuinely contested in observational epidemiology, and a page that presented it as settled would be misrepresenting the literature.

Kenneth Rothman argued in Epidemiology 1990;1:43–46, in a paper titled “No adjustments are needed for multiple comparisons”, that a policy of not adjusting leads to fewer errors of interpretation when the data are real observations rather than random numbers. Thomas Perneger, in the BMJ 1998;316:1236, wrote that describing which tests were performed and why is generally the better way of handling multiple comparisons than routine Bonferroni adjustment.

Their strongest argument is structural. Bonferroni’s threshold for testing hypothesis A depends on how many unrelated hypotheses happened to be tested alongside it — which is a property of the investigator’s schedule rather than of hypothesis A. Two identical analyses in two papers get different thresholds because one author bundled more work into a single submission. The universal null hypothesis being protected, that nothing in the study is real, is rarely the hypothesis anyone cares about.

The dispute concentrates precisely where the family is a judgement call. In a cohort study of 40 exposures and one outcome, no defensible m exists and correction is arguable. In a genome-wide scan, a confirmatory trial with declared co-primary endpoints, or a screen designed to feed validation, the family is fixed by the design before any data arrive, and correction there is uncontested. The practical resolution most journals have converged on is neither position but the reporting requirement: state the number of comparisons, state what was pre-specified, present unadjusted estimates with intervals, and reserve unqualified claims for pre-specified analyses. That satisfies both camps, because both objected to concealment more than to arithmetic.

How mishandled multiplicity is detected in a manuscript

Detection begins with counting rather than reading. Count every p-value in the tables, figures, supplements and text, then compare that count with the m implied by the stated correction. A mismatch is the finding.

Specific signals, in rough order of how often they appear in submitted work:

No correction and a large count. A methods section stating “p < 0.05 was considered statistically significant” beside a results section carrying 30 or more p-values. The absence of the word “correction” anywhere in the methods is a faster check than any statistical reasoning.

A denominator smaller than the analysis. The correction names m = 12 while the tables report 30 tests, or the supplementary material reveals timepoints and subgroups not counted. This is the diagnostic that most often changes an editorial decision.

Arithmetic that does not reconcile. Where both raw and adjusted p-values are printed, Bonferroni is verifiable by multiplication and Benjamini–Hochberg adjusted values must be non-decreasing in the ranked order of raw p-values. Published tables fail one or the other more often than authors expect.

Nominal significance in a high-dimensional setting. A heatmap of “significant” genes at unadjusted p < 0.05 with 20,000 features tested, or a claim resting on p = 1 × 10⁻⁵ in a genome-wide scan.

Method named without assumption. “FDR-corrected” with no procedure, no q, and no statement of m. “Adjusted for multiple comparisons” without naming what family was adjusted over.

Subgroup findings absent from the registration. A registered primary endpoint that is negative, followed by an emphasised subgroup effect. Comparing the manuscript with its registry entry is the only reliable way to see this, and it is why registration numbers belong in the abstract.

Asterisk conventions doing statistical work. Panel legends reading *p<0.05, **p<0.01, ***p<0.001 where the design has more than two groups and no correction across panels is described. Multiplicity mishandled through figures is far more common than multiplicity mishandled through tables, because figure legends are rarely read as methods. The general class is covered under statistical objections from reviewers.

What a peer reviewer says when it is mishandled

Reviewer comments on multiplicity are formulaic, which makes them easy to anticipate and easy to pre-empt. The phrasings that recur:

“No adjustment for multiple comparisons is reported, and the number of comparisons performed is not stated.”

“Given the number of tests conducted at the 0.05 level, at least one significant result would be expected by chance alone.”

“The correction appears to have been applied to a subset of the analyses actually performed. Please state the full family of hypotheses over which the error rate is controlled.”

“The subgroup effect in Figure 3 was not pre-specified and should be described as hypothesis-generating rather than as a finding.”

“Several outcomes are treated as co-primary without any allocation of alpha between them.”

“Significance is claimed at nominal p-values in a transcriptome-wide analysis. Please report FDR-adjusted values and the number of features tested.”

“The authors correct for multiplicity in the primary analysis but not in the safety analysis, without explanation.”

The subgroup comment is the one that most often decides a paper, and it has an established rhetorical anchor. The ISIS-2 trial, reporting on 17,187 patients with suspected myocardial infarction in The Lancet in 1988, divided its patients into twelve subgroups by astrological birth sign to demonstrate how unreliable subgroup analysis is; aspirin showed a non-significant adverse effect in Gemini and Libra against an overall benefit at p < 0.00001. Reviewers still invoke it, and a paper whose headline finding rests on one of twelve subgroups has walked into a comparison it cannot win. See reviewer says multiple comparisons were not accounted for for how to answer these comments rather than concede them.

How to report multiple testing correction

State the family before stating the method. A methods sentence should identify the set of hypotheses, the number of tests in it, the procedure and its assumption: “The nine secondary endpoints were treated as one family and controlled at FWER 0.05 using Holm’s step-down procedure, which requires no assumption about dependence among tests.”

Report unadjusted estimates with confidence intervals alongside adjusted p-values, so a reader can see the effect independently of the threshold decision. Distinguish pre-specified from post hoc analyses explicitly — the CONSORT 2025 statement, published in April 2025, asks trial reports to say which analyses were pre-specified, to identify any analysis added after the protocol was written, and to acknowledge in the discussion where multiplicity limits what the results can support.

For FDR analyses report q-values rather than the phrase “FDR < 0.05”, state the number of features tested after filtering, and state what the filter was and why it is independent of the test statistic. For hierarchical testing, publish the sequence and its date of specification.

Where correction was deliberately not applied, say so and give the reason, in the Rothman and Perneger terms if those are your grounds. A stated position defended in one sentence survives review far better than silence, because silence reads to a reviewer as an omission rather than a choice.

PerfectPaper reports the number of statistical comparisons a manuscript performs, compares it with the correction the methods section claims, and names the analyses that fall outside the declared family.

Related

Reviewer says multiple comparisons were not accounted for · Multiple testing in omics · Pseudoreplication · Post hoc power · Table 2 fallacy · Overclaiming

Review my manuscript

Frequently asked questions

What does multiple testing correction mean?

Multiple testing correction means adjusting significance thresholds or p-values so the error rate across a whole set of tests, rather than each test individually, stays at the intended level. Family-wise methods such as Bonferroni and Holm control the chance of any false positive; false discovery rate methods such as Benjamini–Hochberg control the expected proportion among rejected hypotheses.

What is the multiple comparisons problem?

The multiple comparisons problem is the inflation of false positives that occurs when many hypotheses are tested at the same threshold. At α = 0.05, twenty independent tests under the null produce at least one significant result 64.2% of the time, and one hundred produce one 99.4% of the time. Correction restores the intended error rate.

Why do you need to correct for multiple comparisons?

Correction is needed because an uncorrected threshold controls error for one test, not for a set. A paper reporting 30 comparisons at p < 0.05 expects roughly 1.5 false positives by chance. Without correction, a reader cannot distinguish a real finding from the arithmetic consequence of having looked thirty times.

What is a multiplicity adjustment in a clinical trial?

A multiplicity adjustment in a clinical trial is the pre-specified scheme that keeps the type I error rate at 0.05 across several endpoints. The FDA’s 2022 guidance describes alpha splitting, fixed-sequence hierarchical testing and gatekeeping approaches, all fixed before unblinding. Safety endpoints are conventionally left unadjusted, because the costly error there is a false negative.

Is multiple testing correction the same as the Bonferroni correction?

Bonferroni is one method of multiple testing correction, not the whole category. Bonferroni divides α by the number of tests and holds under any dependence. Holm’s step-down procedure controls the same error rate under the same conditions and rejects at least as many hypotheses, and false discovery rate methods control a different quantity altogether.

What does it mean to adjust p-values for multiple comparisons?

Adjusting p-values for multiple comparisons means rescaling each reported value so it can be compared with the usual 0.05 threshold while the error rate across the set is controlled. A Bonferroni-adjusted p-value is the raw value multiplied by the number of tests; a Benjamini–Hochberg adjusted value is a rank-based rescaling that must increase monotonically with the raw values.

What is a correction for multiple testing in statistics?

A correction for multiple testing is a rule that changes the decision threshold according to how many hypotheses were tested in a declared family. Bonferroni divides α by that number, Holm applies a decreasing sequence of thresholds, and Benjamini–Hochberg uses a rank-based cutoff controlling the expected proportion of false discoveries rather than any false positive.