False discovery rate or family-wise error rate?

The false discovery rate and the family-wise error rate control different quantities, so the choice follows from what you will claim. Family-wise error rate bounds the probability of even one false positive across the family. False discovery rate bounds the expected proportion of false positives among the hypotheses you reject. Individual claims need FWER; set-level claims need FDR.

That rule is easy to state and routinely misapplied, because most manuscripts pick a procedure — Bonferroni, Benjamini–Hochberg — before deciding what the reported findings are for. This page covers the two formal definitions, the claim-level test that decides between them, a worked example at 20,000 tests, the procedures available under each, what dependence does to both, why the word “family” is contested, the conventions in genomics and in confirmatory trials, and what reviewers write when the choice is wrong.

What each error rate actually controls

Family-wise error rate is the probability of making at least one Type I error across a specified family of hypotheses. Controlling the family-wise error rate at 0.05 means that if every null hypothesis in the family were true, the chance of rejecting any of them is at most 5%. The quantity is a probability, and it does not scale with the number of rejections: one false positive counts the same as forty.

False discovery rate is the expected value of the proportion of rejected hypotheses that are true nulls. Benjamini and Hochberg defined it in 1995 (Journal of the Royal Statistical Society B) as E[V/R], with the ratio set to zero when R = 0, where V is the number of false rejections and R the total number of rejections. Controlling the false discovery rate at 0.05 means that averaged over repetitions of the experiment, no more than 5% of the findings you report should be null.

Two consequences follow that are easy to miss. First, the false discovery rate is an expectation over hypothetical repetitions, not a guarantee about the list in front of you: the realised false discovery proportion in a single study can exceed the nominal level substantially, particularly when R is small. Second, when every null in the family is true, V/R is either 0 or 1, so E[V/R] equals P(V ≥ 1), and false discovery rate control coincides exactly with family-wise error rate control. Benjamini–Hochberg is not a permissive procedure under the complete null: there it carries the same family-wise guarantee as Bonferroni, bounding the probability of any false positive at 5%. It still rejects whenever Bonferroni would, and sometimes more.

The question that decides it: what will you claim?

The choice between the false discovery rate and the family-wise error rate is settled by the grammatical subject of your conclusions, not by the number of tests. Ask what sentence the reader will carry away from the abstract.

If the sentence names one hypothesis — “the drug reduced the primary endpoint”, “this variant is associated with the trait” — the claim is per-hypothesis, and it needs family-wise error rate control. Each named finding has to stand on its own, and an error rate that tolerates 5% of a list being wrong tells the reader nothing about whether this particular one is among the wrong ones.

If the sentence names a set — “these 312 transcripts were differentially expressed and were carried into pathway enrichment” — the claim is set-level, and the false discovery rate is the right currency. The reader is told, correctly, that roughly 5% of the 312 are expected to be null, and no argument in the paper depends on any individual member being real.

The most frequent error in applied work is mixing the two: controlling the false discovery rate across a genome-wide screen, then building the discussion, the headline figure and the follow-up experiment on the single gene at the top of the ranked list. A q-value of 0.03 is a property of the list, not a posterior probability for that gene. If one member will be named and defended, that member needs protection of its own — a within-family family-wise correction, an orthogonal assay, or independent replication. Reviewers reading multiple-comparisons objections raise this specific mismatch more often than they raise the choice of procedure.

A worked example at 20,000 tests

Take 20,000 gene-level tests, of which 1,000 are genuinely differential and 19,000 are null. The false-positive arithmetic below is exact given those assumptions; the true-positive counts are illustrative, because they depend on effect sizes and sample size.

With no adjustment at α = 0.05, the 19,000 null genes alone produce an expected 950 false positives. If 600 of the 1,000 real signals are also detected, roughly 61% of the reported list is noise.

Bonferroni at a family-wise error rate of 0.05 sets the per-test threshold at 0.05 / 20,000 = 2.5 × 10⁻⁶. The expected number of false positives across the whole screen is at most 0.05. Only large, well-measured effects survive.

Benjamini–Hochberg at q = 0.05 finds the largest rank k whose ranked p-value satisfies p(k) ≤ (k/m)q, and rejects every hypothesis of rank k or below. Suppose the procedure rejects 500 genes. About 25 of the 500 are expected to be null.

The instructive detail is the threshold Benjamini–Hochberg actually applied. With 500 rejections out of 20,000 at q = 0.05, the largest rejected p-value can be as large as (500/20,000) × 0.05 = 1.25 × 10⁻³ — five hundred times looser than the Bonferroni threshold on the same data. That is where the power difference comes from, and it carries an uncomfortable corollary: under Benjamini–Hochberg the fate of a gene with a fixed p-value depends on what else was on the array. Add 20,000 uninformative probes and the same gene can drop out; add a second strong signal and it can come back in.

Procedures that control the family-wise error rate

Bonferroni rejects when p ≤ α/m. It is valid under any dependence structure, assumes nothing, and yields a fixed numeric threshold that can be written down before any data exist — which is precisely why genome-wide association studies use it. Its weakness is conservatism when tests are correlated.

Šidák uses 1 − (1 − α)^(1/m) and is valid under independence. At m = 20,000 and α = 0.05, Šidák’s threshold is 2.565 × 10⁻⁶ against Bonferroni’s 2.500 × 10⁻⁶, about 2.6% less stringent. The gain rarely justifies the added independence assumption.

Holm (1979, Scandinavian Journal of Statistics) is a step-down procedure: order the p-values ascending and compare p(i) with α/(m − i + 1), stopping at the first comparison that fails. Holm controls the family-wise error rate under arbitrary dependence and is uniformly at least as powerful as Bonferroni, so Bonferroni is statistically dominated except where a fixed pre-specified threshold is itself the requirement.

Hochberg (1988, Biometrika) is a step-up procedure using the same constants, more powerful than Holm, but valid under independence or positive dependence rather than arbitrary dependence.

Westfall–Young permutation builds the null distribution of the smallest p-value by resampling, so the correlation among tests is estimated from the data rather than assumed away. It recovers substantial power when tests are strongly correlated — linkage disequilibrium blocks, cis-eQTL windows, repeated measures on the same subjects — at the expense of computation and of a permutation scheme that must preserve the dependence you care about.

Fixed-sequence (hierarchical) testing orders the hypotheses in advance and tests each at full α until one fails. Nothing is spent on multiplicity at all, which makes it the most powerful option and the most brittle: every hypothesis below the first failure goes untested. Graphical approaches generalise this: each hypothesis is allocated a share of α, the shares summing to the total, each is tested at its own share, and the α of a rejected hypothesis is propagated along weighted edges to the hypotheses still under test. The FDA’s October 2022 guidance Multiple Endpoints in Clinical Trials discusses graphical and gatekeeping approaches alongside Bonferroni, Holm and Hochberg, which is the working menu for confirmatory trial analyses.

Procedures that control the false discovery rate

Benjamini–Hochberg (1995) is the step-up procedure described above, and it controls the false discovery rate under independence and under positive regression dependency. Its adjusted p-values are enforced monotone by taking a cumulative minimum from the largest raw p-value downward and capping at 1, which is why a set of Benjamini–Hochberg q-values can never exceed 1 and can never invert the raw ordering.

Benjamini–Yekutieli (2001, Annals of Statistics) controls the false discovery rate under arbitrary dependence by dividing the level by the harmonic sum Σ(1/i) over i = 1 to m. At m = 20,000 that sum is about 10.5, so Benjamini–Yekutieli at q = 0.05 behaves roughly like Benjamini–Hochberg at q = 0.0048. Applying it reflexively, in settings where the positive-dependence condition already holds, tightens the effective level by an order of magnitude for no gain in validity.

Storey’s q-value (2002) exploits the fact that Benjamini–Hochberg controls the false discovery rate at π₀ × q, where π₀ is the proportion of true nulls. Storey estimates π₀ from the flat right-hand region of the p-value histogram and recovers that conservatism. Where π₀ = 0.95 the gain is negligible; where π₀ = 0.5, as in a well-powered cis-eQTL scan, plain Benjamini–Hochberg holds the actual false discovery rate near q/2 rather than q, and estimating π₀ recovers that conservatism at no cost to validity. How many additional discoveries that recovery yields depends on the p-value distribution of the alternatives, so it is worth reporting rather than assuming.

Local false discovery rate, developed by Efron, reports for each individual test the posterior probability that it is null given its own statistic, rather than a tail-area quantity computed over the list. This is the object most researchers already believe a q-value to be, and it is the defensible way to make a per-feature probabilistic statement inside a false discovery rate framework.

What dependence does to each

Dependence between tests changes which procedures remain valid and how variable the realised error is. Bonferroni and Holm hold under arbitrary dependence. Benjamini–Hochberg requires positive regression dependency on a subset, the condition Benjamini and Yekutieli proved sufficient in 2001; it is satisfied by multivariate normal statistics with non-negative correlations, by multivariate t, and by comparisons of many treatments against a single control.

Most genomic data satisfy that condition approximately: linkage disequilibrium between nearby variants is positive, and co-regulated transcripts are positively correlated. The awkward case is negative dependence induced by compositionality — microbiome relative abundances, deconvolved cell-type proportions, any vector constrained to sum to one — where positive regression dependency is not assured and Benjamini–Yekutieli or permutation is the honest answer rather than a reflex.

Dependence also inflates the variance of the false discovery proportion even where the false discovery rate is controlled correctly. The average across hypothetical repetitions is right; the single realised list can be far from it. Strong correlation makes the discoveries arrive in blocks, so a study either catches a correlated cluster of true signals or catches a correlated cluster of noise. That variance is an argument for reporting effect sizes and replication alongside q-values, and it is invisible to any procedure. Batch structure produces exactly this pattern, which is why batch-effect scrutiny and multiple-testing scrutiny belong in the same paragraph of a methods section.

What counts as a family, and why that is contested

Neither error rate is defined without a family, and no statistical principle fixes the family’s boundary. Rothman argued in Epidemiology (1990;1:43–46), under the title “No adjustments are needed for multiple comparisons”, that declining to adjust leads to fewer errors of interpretation when the data are observations on nature rather than random numbers. Perneger made a narrower version of the case in the BMJ (1998;316:1236), “What’s wrong with Bonferroni adjustments”: the universal null hypothesis that Bonferroni protects is rarely the hypothesis anyone cares about, and the severity of the adjustment depends on how many other tests the analyst happened to run.

The opposing position is that the family is determined by the decision the analysis supports, and that decision is knowable in advance. The FDA’s October 2022 guidance takes that position: Type I error control is expected to span the primary and secondary endpoint families together, and the boundary is drawn on the other side just as plainly, with exploratory endpoints left outside the adjustment because they are not used to support the conclusions the label rests on.

That is the workable resolution. A family is the set of tests any one of which, if significant, would be used to make the same claim. Its boundary is fixed by the prespecified protocol, not by the argument list of a software call. Where the boundary is genuinely a judgement — a secondary analysis that might or might not enter the abstract — state the judgement and report what happens under the alternative, which is the discipline registered endpoint reporting is designed to enforce.

Field conventions and where they came from

Genome-wide association studies control the family-wise error rate, at p < 5 × 10⁻⁸. That number is 0.05 Bonferroni-corrected for approximately one million effectively independent common variants, an effective count estimated by different methods, and by different groups, with broadly similar answers — Pe’er et al. (2008) and Dudbridge and Gusnanto (2008) are the two most often cited. The field chose family-wise control deliberately, because its unit of currency is the individually named, individually replicated locus, and because a threshold set before genotyping cannot be loosened by whatever else ended up on the array. It is worth being clear about what that number is: a convention, calibrated to the common-variant genotyping era, rather than a quantity derived from first principles for any particular study. Sample sizes have since grown by orders of magnitude, sequencing has pushed the effective number of independent tests up, and the appropriate threshold has been revisited repeatedly in the statistical-genetics literature — which is why genomics reviewers increasingly expect the threshold to be justified for the design at hand rather than cited as a fixed constant.

Exome-wide rare-variant burden testing generally uses about 2.5 × 10⁻⁶, which is 0.05 divided by the roughly 20,000 protein-coding genes, on the same family-wise logic.

Differential expression and eQTL mapping control the false discovery rate, conventionally at 5% via Benjamini–Hochberg. Well-constructed eQTL pipelines use both rates in sequence: permutation within each gene’s cis-window produces one family-wise-corrected p-value per gene, then Benjamini–Hochberg is applied across genes. That two-stage structure matches the claim structure exactly — the per-gene claim is protected individually, the gene list is protected as a set — and it is the pattern to imitate whenever a paper names some features and enumerates others.

Confirmatory clinical trials control the family-wise error rate, typically at 2.5% one-sided. Regulatory multiplicity guidance is framed in family-wise terms throughout, and that is not an oversight: a regulator licenses individual claims that appear on an individual label, so an error rate defined over a list has nothing to attach to. The exposure being controlled is easy to quantify. Ten independent endpoints, each tested one-sided at 2.5%, carry a probability of about 22% that at least one is declared significant when every null is true — 1 − 0.975¹⁰ — which is why an unadjusted secondary-endpoint claim is treated as unsupported rather than as marginal.

How the choice is detected in a manuscript

Reading a manuscript for multiple-testing discipline means reconciling three numbers that are usually stated in three different places, and looking for eight specific tells.

First, the number of tests corrected for does not match the number of tests the methods describe. Count the comparisons in the design, not the ones in the results table.

Second, adjusted p-values are reported that exceed 1, or that are not monotone in the raw p-values. Benjamini–Hochberg adjusted values are enforced monotone and capped at 1, so either pattern means the arithmetic was done by hand or the wrong column was pasted.

Third, q-values are labelled as p-values, and a q of 0.03 is glossed as “a 3% chance that this gene is a false positive”. That reading is a local false discovery rate statement, and the tail-area q-value does not supply it.

Fourth, the false discovery rate is applied separately within each of several subgroups and the union is then reported as one findings list. The family was silently redefined between the analysis and the abstract.

Fifth, Bonferroni is applied to the comparisons that were reported rather than the comparisons that were run. This is visible whenever the results section contains fewer contrasts than the methods promised.

Sixth, the p-value histogram is not approximately flat over [0.5, 1]. A histogram that slopes or humps near 1 indicates conservative or dependent test statistics, and every downstream error rate — both of them — is then wrong regardless of which procedure was applied.

Seventh, the paper’s central conclusion rests on the top-ranked item of a false-discovery-controlled list, with no separate protection for that item.

Eighth, and most common, a procedure is named without its family: “we corrected for multiple comparisons using Benjamini–Hochberg”, with no statement of the set the correction spanned. That sentence is not reproducible, and it is the one that most often survives to publication. Automated screening for this class, including the omics multiple-testing check, works by extracting the test count from the methods and comparing it with the correction the results claim.

What reviewers say when it is mishandled

Reviewer comments on this class are usually short and specific, and the phrasings recur across fields.

“The authors control the false discovery rate but then base the central claim on a single gene; FDR does not license a per-gene statement.”

“It is not stated over which set of tests the Benjamini–Hochberg correction was applied.”

“Secondary endpoints are described as significant, but no prespecified Type I error control strategy covering primary and secondary endpoints is presented.”

“Bonferroni appears to have been applied to the comparisons reported rather than the comparisons performed.”

“Given the correlation among the outcome measures, a permutation-based correction would be more appropriate than Bonferroni, which is badly conservative here.”

“The choice of q = 0.10 is not justified. Please state the proportion of false findings the analysis is intended to tolerate, and why.”

The comment that most often decides the outcome is the second one. Where the family is not prespecified, the correction is unfalsifiable: any threshold at all can be obtained by choosing what to count, and a reviewer who notices that will treat every number downstream as provisional. Statistical reviewers escalate this quickly, and it appears frequently among statistical objections that end in a request for reanalysis rather than revision.

How to report the choice

State the error rate, the level and the family in a single sentence, before any result: “We controlled the family-wise error rate at 5% using Holm’s procedure across the four prespecified contrasts”, or “We controlled the false discovery rate at 5% using Benjamini–Hochberg across all 18,412 genes passing the expression filter.”

Report the number of tests, and give both raw and adjusted values so a reader can recompute. Justify the level by the asymmetry of consequences in your setting: what a false positive does to the field here, against what a missed signal does. A screen feeding a further experiment tolerates q = 0.10 comfortably; a claim entering a clinical guideline does not.

If you control the false discovery rate, say in the text that the guarantee is set-level, and say what protects any individual member you name. If you control the family-wise error rate, name the procedure and its dependence assumption — Hochberg and Šidák require independence or positive dependence, Bonferroni and Holm do not.

Prespecify. A correction chosen after the p-values were seen controls nothing at all, and a reviewer asking about overclaiming will read a post hoc correction as a selection effect, not a safeguard.

Limitations of both error rates

Both the false discovery rate and the family-wise error rate are guarantees about Type I error under a correctly specified model, and neither addresses several failure modes that damage papers more often.

Neither rate protects against a wrong test. Correcting 40 p-values that each came from an inappropriate model produces 40 correctly adjusted wrong numbers, which is the situation behind most wrong statistical test objections.

Neither addresses non-independence of observations. Where the unit of analysis is wrong — cells treated as replicates, repeated measures pooled — the raw p-values are already invalid, and multiplicity correction operates downstream of the damage. That is a pseudoreplication problem, and no adjustment reaches it.

Neither says anything about the direction or magnitude of an estimate. Effects that only just clear a stringent threshold are, on average, overestimated, because clearing the threshold requires a favourable draw of sampling error. A stringent family-wise threshold therefore produces fewer false claims and more exaggerated true ones, which is why replication effect sizes in genome-wide association studies are systematically smaller than discovery estimates, and why post hoc power computed from an observed effect is uninformative.

Neither touches selective reporting. If analyses that produced nothing were never written down, the family the correction spans is not the family that was tested, and the guarantee is void in a way no reader can detect from the manuscript.

And the family-wise error rate has one further honest limitation: it says nothing about the composition of a findings list. A study that rejects 200 of 20,000 hypotheses under Bonferroni has not shown that at most 5% of the 200 are wrong; it has shown that, whatever the true state of the nulls, the chance the analysis produced even one false rejection was at most 5%. Those are different assurances, and papers frequently report the first while having established the second.

Related

Multiple comparisons objections · Statistics objections · Pseudoreplication · Independence of n · Table 2 fallacy

Checked before submission by the omics multiple-testing reviewer, which reconciles the number of tests described in the methods with the correction claimed in the results and flags per-feature claims made from a set-level guarantee.

Review my manuscript

Frequently asked questions

Should I use FDR or Bonferroni correction?

Use Bonferroni, or preferably Holm, when each finding will be named and defended individually, as with a trial endpoint or a genome-wide association locus. Use Benjamini–Hochberg when the output is a ranked list interpreted as a set, as with differential expression. The unit of your claim decides, not the number of tests.

What is the difference between the false discovery rate and the family-wise error rate?

Family-wise error rate is the probability of at least one false positive anywhere in the family. False discovery rate is the expected proportion of false positives among the hypotheses actually rejected. One is a probability of any error occurring; the other is an average contamination level of the findings list. Under the complete null they coincide.

When is the false discovery rate more appropriate than the family-wise error rate?

The false discovery rate is more appropriate when many hypotheses are tested, a moderate contamination of the results list is tolerable, and no single member of the list carries the conclusion. Screens, differential expression, and any analysis whose output feeds a further experiment fit that description. Confirmatory analyses supporting individual claims do not.

Is Benjamini–Hochberg or Bonferroni better for my analysis?

Neither is better in general. Benjamini–Hochberg is far more powerful at large numbers of tests, because its effective threshold rises with the number of rejections. Bonferroni gives a fixed threshold, valid under any dependence, that can be prespecified before data exist. Choose by whether your conclusions are about a list or about individual hypotheses.

Which multiple testing correction should I use?

Choose the error rate first, then the procedure. For family-wise control, Holm dominates Bonferroni under any dependence, Hochberg is more powerful under positive dependence, and permutation recovers power when tests are strongly correlated. For false discovery control, Benjamini–Hochberg is the default, Benjamini–Yekutieli for arbitrary dependence, and Storey q-values when the null proportion is well below one.

Does controlling FDR instead of FWER mean my analysis is less rigorous?

No. Controlling the false discovery rate is a different guarantee, not a weaker version of the same one, and under the complete null the two are identical. Rigour fails when the guarantee and the claim do not match — for example reporting a set-level false discovery rate and then defending one named feature as though it were individually protected.

How do I decide between q-values and adjusted p-values?

Decide by what the number must support. Family-wise adjusted p-values license a statement about an individual hypothesis. Tail-area q-values license a statement about the contamination of a list. If you need a per-feature probability of being null, neither is correct: report a local false discovery rate, which is the posterior probability for that feature alone.