What is publication bias?

Publication bias is the distortion that arises when whether a study reaches the literature depends on what it found. Trials with statistically significant or favourable results are published more often, sooner, and more prominently than those without, so the visible evidence base overstates effects. Meta-analyses built on that visible base inherit the overstatement and cannot detect it from the included studies alone.

Robert Rosenthal named the underlying phenomenon the “file drawer problem” in 1979: the studies that never appear are not a random sample of the studies that were run, and no amount of care in synthesising what did appear recovers what did not. This page covers the mechanism, the measured size of the effect in named literatures, funnel plots and the tests built on them, adjustment methods and their failure modes, how publication bias is detected while reading an actual manuscript, and what reviewers write when a paper handles it badly.

The mechanism: three gates, all sensitive to the result

Publication bias is produced by a chain of decisions in which the study’s result influences whether the next step happens. Three gates matter, and they are not equally important.

The author gate is the largest. Surveys of investigators with unpublished work consistently find that the dominant reason given is the authors’ own judgement that a null result was not interesting or not worth writing up — not rejection by a journal. A null trial that is never drafted cannot be rejected, and it leaves no trace in any editorial record.

The editorial gate operates on perceived importance. A negative result is rarely rejected for being negative in so many words; it is rejected for lacking novelty, impact or a clear advance, which correlates with direction of effect. See why papers get rejected and desk rejected without review for how that judgement is actually worded.

Time-lag bias is a third gate that is often missed. Cohorts of trials followed forward from ethics approval — the design used by Stern and Simes and by the Cochrane methodology reviews that followed — consistently find that trials with statistically significant results reach publication substantially sooner than those without. The size of the gap varies by cohort and by field, and it is measured in years rather than months. A meta-analysis run at any given moment therefore sees a version of the literature that is skewed even if every trial is eventually published.

What publication bias is, and what it is not

Publication bias concerns whether a study becomes available. Several related biases concern what happens to a study that is available, and conflating them leads people to apply the wrong remedy.

Outcome reporting bias occurs within a published paper: the trial appears, but the outcomes that reached statistical significance are reported fully while others are dropped, downgraded from primary to secondary, or described only as “not significant”. Chan and colleagues compared trial protocols with the resulting publications and found that roughly half of efficacy outcomes and around two-thirds of harm outcomes per trial were incompletely reported, and that significant outcomes were markedly more likely to be reported in full. Cross-checking against registered endpoints is the specific remedy, and it does not require a meta-analysis.

Small-study effects is the correct name for the pattern seen in a funnel plot: smaller studies giving systematically different results from larger ones. Publication bias is one cause. Genuine clinical heterogeneity — smaller trials recruiting higher-risk patients in whom the treatment works better — is another, as is poorer methodological quality in small studies, and chance.

Citation bias, language bias and duplicate publication bias affect which available studies a reviewer finds. Trials reporting positive results are cited more, are more likely to appear in English-language journals, and are more likely to be published more than once, so a search that stops at English-language indexed journals compounds the primary bias.

Selective analysis reporting overlaps with multiple comparisons: one analysis of many is reported, chosen after seeing which produced a small p-value.

The Cochrane Handbook files all five of the above, plus publication bias itself, under one umbrella term: reporting biases. Publication bias narrowly means the whole study never surfaces; the other four mean a study that did surface is still not the fair record of what was done. A manuscript can be clear of one and still carry another — a fully published trial can still selectively report its outcomes, and a null trial that does get published can still be cited less often than a positive one.

Common questions about publication bias and related biases

What’s the file drawer problem?

The file drawer problem is Robert Rosenthal’s 1979 name for publication bias: null and unexciting results end up sitting in a researcher’s own filing cabinet instead of being submitted anywhere. Investigator surveys find that authors’ own judgement that a result isn’t worth the effort of writing up, not journal rejection, is the dominant reason those studies never leave the drawer.

Is publication bias the same thing as p-hacking?

No. P-hacking happens inside one study — trying multiple analyses, subgroups or cutoffs until a result clears p<0.05. Publication bias happens afterward, at the level of the whole literature: whichever result a study produced, favourable findings are more likely to be written up, accepted and published than null ones. The two compound each other but are separate mechanisms with separate remedies.

Why don’t researchers just publish their null results?

Mostly because authors themselves decide a null finding isn’t worth the effort of writing up, not because a journal rejected it — investigator surveys consistently find the author’s own judgement is the largest single gate, ahead of editorial rejection. A null result that is submitted also takes measurably longer to reach print than a positive one, a separate gate called time-lag bias.

How large the distortion is, in named literatures

Publication bias has been quantified directly wherever a complete registry of studies exists to compare against the published record. Four measurements are worth knowing by name.

Antidepressant trials, Turner and colleagues, New England Journal of Medicine, 2008. Using FDA registration records for 12 antidepressants and 12,564 patients, the authors identified 74 registered trials, of which 31% were never published. Separate meta-analyses of the FDA dataset and the published dataset differed by 32% in effect size overall, with per-drug inflation ranging from 11% to 69%. This is the single most-cited demonstration because the denominator — every trial the regulator saw — was knowable.

Preclinical stroke research, Sena and colleagues, 2010. In a survey of animal stroke studies, only about 2% of publications reported no significant effect, a rate no plausible experimental programme produces. The authors estimated that publication bias accounted for roughly a third of the reported efficacy. Preclinical literatures are especially exposed because registration is rare rather than absent: dedicated registries exist — preclinicaltrials.eu and the German Animal Study Registry — but uptake remains low, so in practice there is usually no denominator against which to name the missing studies; see preclinical review and in vivo rigour.

Registered Reports in psychology, Scheel, Schijen and Lakens, 2021. Comparing 71 published Registered Reports with 152 hypothesis-testing studies sampled from the standard literature, the authors found positive results in 96% of standard articles and 44% of Registered Reports — a 52-percentage-point gap. The comparison is against a random sample of the standard literature rather than against matched studies, so the two sets differ in more than the timing of the publication decision — but no other difference plausibly accounts for a gap that size, and the authors point to reduced publication bias and Type I error inflation as the likeliest explanation.

Cross-disciplinary trend, Fanelli, 2012. Sampling papers across disciplines and countries, Fanelli found that the frequency of papers reporting support for the tested hypothesis rose from roughly 70% at the start of the 1990s to roughly 86% by 2007, with the increase present in most disciplines. The number is not a bias estimate on its own — some rise is expected as fields learn which hypotheses are worth testing — but no field has a hypothesis-generation process good enough to justify a figure that high or a trend in that direction.

Questions about how large the distortion is

How much does publication bias actually inflate a meta-analysis?

It depends on the literature, but the best-documented case is large: Turner and colleagues compared every FDA-registered antidepressant trial against the published subset and found pooled effect sizes differed by 32% overall, with per-drug inflation ranging from 11% to 69%. Smaller, better-registered literatures show smaller gaps — the size is not a fixed constant you can apply elsewhere.

Is publication bias worse in some fields than others?

Yes — it tracks how easy non-publication is to get away with. Preclinical animal research, where dedicated registries exist but uptake remains low, shows some of the most extreme documented distortion: Sena and colleagues found only about 2% of published stroke studies reported a null result. Fields with mandatory prospective registration, such as regulated drug trials, show smaller, though still real, gaps.

Funnel plots: what they show and what they do not

A funnel plot displays each study’s effect estimate on the horizontal axis against its precision — conventionally the standard error, plotted with zero at the top — on the vertical axis. Because small studies estimate the effect imprecisely, they scatter widely at the bottom; large studies cluster near the pooled estimate at the top. Under no bias the scatter is a symmetrical inverted funnel.

Asymmetry — a visible gap where small studies with small or negative effects should sit — is the classic signal. Contour-enhanced funnel plots, introduced by Peters and colleagues, add statistical-significance contours to the plot so that a reader can see whether the missing region is the non-significant region, which points at publication bias, or a region that has no particular significance status, which points at heterogeneity.

Four limitations are worth stating plainly. Funnel plot asymmetry is not a test for publication bias; it is a description of small-study effects with several possible causes. Visual assessment is unreliable, and published re-readings of the same plots by different assessors disagree. A funnel plot with fewer than about ten studies conveys almost nothing — the scatter of a handful of points is indistinguishable from any shape you care to see in it. And a symmetrical plot cannot rule out outcome reporting bias sitting inside it: a trial that was never submitted anywhere and a trial that was published but swapped its primary outcome for a friendlier one both remove the same kind of unfavourable point from the same region, so the plot cannot tell you which mechanism, if either, is at work.

A funnel plot’s working assumption — that each point is an independent study — also breaks in a specific, common way. A three-arm trial that contributes two treatment-versus-control comparisons to the same meta-analysis places two correlated points on the plot rather than two independent ones, which can manufacture the appearance of asymmetry that has nothing to do with missing studies. The Cochrane Handbook’s guidance for multi-arm trials is to either combine the arms into a single comparison or split the shared control group’s information across them before the plot is drawn — not to plot every arm as though it came from a separate trial.

Statistical tests for funnel plot asymmetry

Tests for funnel plot asymmetry ask whether effect size is associated with precision across the included studies. Four are in common use, and choosing between them is not arbitrary.

Egger’s regression test (Egger and colleagues, BMJ, 1997) regresses the standardised effect estimate on its precision and tests whether the intercept differs from zero. It is the default in most software.

Begg and Mazumdar’s rank correlation test (1994) correlates effect size with variance non-parametrically. It has lower power than Egger’s test and is now rarely preferred.

Harbord’s and Peters’ tests exist because Egger’s test misbehaves with binary outcomes: the log odds ratio and its standard error are mathematically correlated, which inflates the false-positive rate when events are rare or effects are large. For odds ratios, one of these variants should be used instead.

Two conventions govern interpretation. The Cochrane Handbook’s rule of thumb is that tests for funnel plot asymmetry should be used only when at least ten studies are included, because below that the power is too low to separate chance from real asymmetry. And the conventional threshold is p < 0.10, not 0.05, precisely because the tests are underpowered — a compensation that is often silently dropped in manuscripts reporting “no evidence of publication bias, p = 0.31”.

A non-significant asymmetry test in eight studies is not evidence of absence. It is an uninformative test that should not have been run.

Questions about funnel plots and asymmetry tests

Why does my funnel plot look asymmetric?

Asymmetry has at least four separate causes, and a funnel plot alone can’t tell you which one you’re looking at: publication bias, genuine clinical heterogeneity (smaller trials often enrol higher-risk patients who respond more), poorer methodological quality in small studies, and chance. A contour-enhanced funnel plot narrows this by showing whether the missing region specifically corresponds to non-significant results.

How many studies do I need before running Egger’s test?

The Cochrane Handbook’s rule of thumb is at least ten. Below that, an asymmetry test has too little power to distinguish real asymmetry from noise, so a non-significant result with six or eight studies is not evidence of no bias — it’s an uninformative test that shouldn’t have been run in the first place.

My funnel plot looks asymmetric but I only have 8 studies — could that just be chance, not publication bias?

Yes, plausibly. With fewer than about ten studies, the scatter of a handful of points is visually indistinguishable from any shape, including a symmetrical one — which is exactly why the Cochrane Handbook restricts formal asymmetry testing to ten or more studies. Report the plot honestly, and state that the study count was too small to test asymmetry formally, rather than running Egger’s test anyway.

Reviewer 2 says my meta-analysis needs a publication-bias test but I only pooled 6 studies — what do I say?

Say plainly that tests for funnel-plot asymmetry were not performed because six studies falls below the ten-study threshold the Cochrane Handbook sets for adequate power, and state that as a limitation rather than an omission. That sentence is stronger than reporting an underpowered Egger’s test result, and it reads to reviewers as methodological awareness rather than evasion.

Adjustment methods and their failure modes

Methods that attempt to correct a pooled estimate for publication bias exist, and every one of them rests on an assumption about the missing studies that the data cannot verify.

Trim and fill (Duval and Tweedie, 2000) removes asymmetric studies, estimates the number missing, imputes mirror-image studies and recomputes the pooled estimate. Simulation work — Terrin and colleagues among others — shows it performs poorly under between-study heterogeneity, where it can add studies that were never missing and shift an unbiased estimate. Trim and fill is best presented as a sensitivity analysis, never as a corrected result.

PET-PEESE (Stanley and Doucouliagos) uses the regression of effect on standard error to extrapolate to a hypothetical study of infinite precision. The extrapolation is far outside the observed data and is sensitive to the largest study in the set — and it is weakest exactly where it is needed most: a literature dominated by small trials, which is the same literature a real publication-bias problem tends to produce, rarely contains enough large, precise studies to anchor the extrapolation reliably.

Selection models — Vevea and Hedges’s weight-function models, and Copas’s selection model — write down an explicit model for the probability that a study with a given p-value is published, then estimate the effect under that model. The honesty of the approach is its strength: the assumption is visible and can be varied. The weakness is that the selection function is not identified by the data and must be assumed — and in practice that assumption is often set by a software default rather than chosen deliberately, so two analysts running the same package on the same dataset can report different adjusted estimates without either one having misused the method.

p-curve (Simonsohn, Nelson and Simmons, 2014) examines the distribution of significant p-values, on the reasoning that a real effect produces right-skew towards small p-values while selective reporting of a null effect produces a flat or left-skewed curve. p-curve addresses evidential value across a set of significant findings, not the pooled magnitude, and it is sensitive to which results are entered.

The excess significance test (Ioannidis and Trikalinos) compares the number of significant findings observed with the number expected given the studies’ power. A large excess implies missing null results somewhere in the chain.

Agreement between two or more of these methods is more informative than any one of them. Disagreement is itself a reportable finding.

Frequently asked questions about correcting for publication bias

Can trim and fill actually fix publication bias?

No — trim and fill produces a corrected pooled estimate, it does not fix anything. Duval and Tweedie’s method imputes hypothetical missing studies to restore funnel-plot symmetry, but simulation work shows it performs poorly under between-study heterogeneity, sometimes adding studies that were never actually missing. Present it as a sensitivity analysis, not a correction.

Should I report the trim-and-fill adjusted estimate as my main result?

No. Report your primary pooled estimate as the primary result, and present the trim-and-fill number alongside it, explicitly labelled a sensitivity analysis under a stated assumption. Presenting an adjusted number as though it were corrected, without naming the heterogeneity it assumes away, is exactly the pattern peer reviewers flag.

What’s the difference between PET-PEESE and trim and fill?

Trim and fill imputes missing studies and recomputes the pooled effect; PET-PEESE instead regresses effect size on standard error and extrapolates to a hypothetical study of infinite precision. PET-PEESE’s extrapolation is least reliable in exactly the literatures that need it most, because a literature dominated by small trials rarely has enough large, precise studies to anchor the extrapolation.

How publication bias is detected in a real manuscript

Detecting publication bias while reading a submitted manuscript is a document-comparison task before it is a statistical one. Six checks do most of the work, and four of them require no meta-analysis at all.

Compare the registry record with the paper. Retrieve the ClinicalTrials.gov, ISRCTN or PROSPERO entry named in the methods. Check the registered primary outcome against the reported primary outcome, the registered sample size against the analysed sample, and the registration date against the enrolment start date — a registration date that falls after enrolment began is retrospective registration, and it means an outcome could still have been chosen after early results were already visible, even though a registry entry exists to point to. A primary outcome that appears in the paper as a secondary outcome, or a registered outcome that has vanished, is the highest-yield single finding available.

Count the outcomes. A trial protocol listing nine outcomes and a paper reporting four, with all four significant, is selective reporting on its face regardless of what the discussion says.

Check the search strategy in a systematic review. A search restricted to PubMed, restricted to English, or with no attempt at trial registries, conference abstracts, regulatory documents or contact with authors, has a known direction of error. The review’s own inclusion flow chart usually reveals this in one line.

Look at the number of included studies before believing any asymmetry test. If the meta-analysis pools seven studies and reports Egger’s test, the test is uninformative, and the manuscript’s conclusion drawn from it is unsupported.

Read the funnel plot rather than the sentence about it. Manuscripts routinely include a plot showing visible asymmetry and a sentence stating that the plot is symmetrical.

Check whether harms are reported at the same resolution as benefits. Asymmetry between a fully specified efficacy analysis and a one-paragraph safety summary is a reporting-bias signal within a single paper.

PerfectPaper performs the registry-to-manuscript comparison and the outcome census as part of a structured review of clinical and epidemiological manuscripts, reporting what it found in the registry entry alongside the claim in the text rather than asserting misconduct. See clinical trial review and epidemiology review for how those checks are scoped.

What a peer reviewer says when it is mishandled

Reviewers rarely write the phrase “publication bias” when they mean it. The comments below are composed examples in the register these comments arrive in, grouped by what the reviewer has noticed.

On an underpowered asymmetry test: “The authors report no evidence of publication bias on the basis of Egger’s test with eight included studies. This test has negligible power at that number and cannot support the conclusion drawn.”

On an absent assessment: “No assessment of risk of bias due to missing results is presented. Please address this domain explicitly, per PRISMA 2020.”

On trim and fill presented as a correction: “The trim-and-fill adjusted estimate is presented alongside the primary estimate without qualification. Trim and fill is unreliable under the level of heterogeneity reported here (I² = 71%) and should be described as a sensitivity analysis.”

On the search: “The search was limited to English-language publications indexed in MEDLINE. Given the known association between direction of effect and language of publication, this restriction should be justified or the limitation stated.”

On a registry discrepancy: “The registered primary outcome (ClinicalTrials.gov NCT…) differs from the primary outcome reported here. Please explain when and why the outcome was changed.”

On conclusions drawn past the evidence: “Small-study effects are evident in Figure 3, yet the discussion treats the pooled estimate as unbiased.”

On the discussion section: “The limitations paragraph acknowledges publication bias in general terms without stating its likely direction or magnitude for this synthesis.”

That last comment is the most common and the most easily pre-empted, and it is the same failure mode as a generic limitations paragraph anywhere else — see overclaim checking.

What actually reduces it

Publication bias is a structural problem, and the interventions that measurably change it operate on structures rather than on individual authors’ diligence.

Prospective registration is the precondition for everything else, because it creates a denominator. The ICMJE has required registration before enrolment of the first participant as a condition of consideration since 2005, and registration is what makes the registry-versus-paper comparison above possible.

Registered Reports move the accept decision to before data collection, on the basis of the question and the method. The 96%-versus-44% gap measured by Scheel and colleagues is the clearest available evidence that much of the excess of positive findings comes from the publication process rather than from nature — the authors themselves put it as the most plausible explanation for a gap that large, not as a demonstrated cause.

Mandatory results posting. The FDA Amendments Act of 2007 requires summary results for applicable trials to be posted to ClinicalTrials.gov, generally within twelve months of primary completion. Compliance audits have repeatedly found substantial non-posting, and the published estimates vary with the cohort and the definition of an applicable trial, so quote a specific rate only alongside the cohort it came from.

Preprints and data sharing shorten the time-lag gate and make unreported outcomes recoverable; see data availability and accessions.

Formal assessment tools. ROB-ME, the Cochrane tool for risk of bias due to missing evidence, replaces the informal funnel-plot ritual with signalling questions: how many studies are known to be missing despite having measured the outcome, whether the search was comprehensive, and whether statistical or graphical methods suggest results are missing because of what they showed. GRADE treats publication bias as a domain that can lower certainty by one or two levels independently of the other domains.

What to write in your own manuscript

A publication bias assessment in your own manuscript reads as a specification, not a courtesy, when it states four things: what you did to find unpublished work, how many studies you pooled and therefore which asymmetry methods were appropriate, what the methods showed, and in which direction and by roughly how much a plausible amount of missing evidence would move your conclusion.

Report the funnel plot when you have ten or more studies and say what the contours show. Report the asymmetry test with the threshold you used and why. Present any adjusted estimate as a sensitivity analysis with its assumption named. If you pooled fewer than ten studies, write that tests for funnel plot asymmetry were not performed because the number of studies was insufficient — that sentence is stronger than an uninformative p-value, and reviewers read it that way.

For a primary study rather than a synthesis, the equivalent commitment is a registered protocol, a reported outcome list that matches it, and a results section that reports the pre-specified analysis whether or not it worked. A null primary outcome reported cleanly is publishable work; see when your effect size is called not meaningful and replication objections for how to frame it.

Honest limitations of everything above

Publication bias cannot be measured from the published literature. Every method described on this page infers something about studies that are absent from the properties of studies that are present, and that inference is not identified by the data — it is identified by an assumption about the selection process. Two analysts applying different, equally defensible assumptions to the same meta-analysis can reach different adjusted estimates.

Funnel plot asymmetry has at least four causes and the plot does not distinguish them. Trim and fill is unreliable under heterogeneity, which is present in most clinical meta-analyses. Asymmetry tests are underpowered where they are most needed. Registries close the loop only for registered study types, which excludes most preclinical, observational and qualitative research entirely.

The honest position is that publication bias is a risk to be characterised and bounded, not a quantity to be measured and subtracted. Reviewers respond well to that framing and badly to an adjusted number presented as a correction.

Related

Confounding · Collider bias · Multiple comparisons objections · Sample size objections · Journal selection

Checked before submission by trial registration and endpoint review, which compares the registered outcomes against the reported ones and names each discrepancy with its registry identifier.

Review my manuscript