What is a funnel plot?

A funnel plot is a scatter plot used in meta-analysis that displays each contributing study’s effect estimate on the horizontal axis against a measure of its precision, usually the standard error, on an inverted vertical axis. In the absence of bias and heterogeneity the points form a symmetrical inverted funnel, because small studies scatter widely and large studies cluster near the pooled effect.

Almost every practical difficulty with funnel plots comes from what happens when the funnel is not symmetrical. Asymmetry is routinely reported as evidence of publication bias, and that inference is wrong often enough that the methodological literature has spent decades trying to constrain it. This page covers the axes, the confidence contours, what asymmetry can and cannot mean, the tests and when each applies, how the problem is detected in a submitted manuscript, and what reviewers write when it has been handled badly.

Where the funnel plot came from and what it was for

Light and Pillemer introduced the funnel plot in Summing Up: The Science of Reviewing Research (Harvard University Press, 1984), as a visual device for judging whether a body of literature looked complete. The reasoning was straightforward sampling theory rather than a new statistic: the sampling variance of an effect estimate falls as the study grows, so estimates from small studies should scatter widely around the true effect and estimates from large studies should sit close to it.

That expectation gives the plot its diagnostic content. A hole in the wide, imprecise part of the funnel — the region where small studies with unremarkable results would sit — indicates that studies which should exist are absent from the synthesis. The plot says nothing about why they are absent, and Light and Pillemer’s original framing was explicitly exploratory. Three decades of use turned an exploratory display into a test, which is the source of most of the trouble described below.

The axes, and why the vertical axis matters more than it looks

The vertical axis of a funnel plot should carry the standard error of each study’s effect estimate, plotted with zero at the top so that the most precise studies sit at the apex. Sterne and Egger compared the plausible candidates — sample size, standard error, variance, precision (the reciprocal of the standard error) and inverse variance — in Journal of Clinical Epidemiology in 2001 and recommended the standard error, a recommendation carried forward into the Cochrane Handbook.

The reason is geometric rather than aesthetic. When the vertical axis is the standard error, the expected 95% region is bounded by the straight lines pooled effect ± 1.96 × standard error, so the funnel is a triangle with straight sides and the eye can judge symmetry against a ruler. Plot the same data against sample size or against precision and those bounds become curves, compressing the imprecise studies into a narrow band at the bottom and making asymmetry far harder to see. For ratio measures — odds ratios, risk ratios, hazard ratios — the horizontal axis must be on the logarithmic scale, because a ratio measure is asymmetric on the raw scale and will manufacture apparent asymmetry from symmetric data.

Reading the plot: pseudo-confidence contours

The triangle drawn on a funnel plot is a pseudo-confidence region, not a confidence interval for anything estimated from the plot. Its lines are conventionally set at the fixed-effect pooled estimate ± 1.96 standard errors, so that roughly 95% of studies would fall inside it if the pooled estimate were the single true effect and heterogeneity were absent.

Two consequences follow, and both are frequently missed in submitted manuscripts. First, the contours are drawn about the fixed-effect estimate even when the synthesis itself is random-effects, because a random-effects centre and a fixed-effect width describe two different models. Second, when between-study heterogeneity is real, points will fall outside the triangle for reasons that have nothing to do with reporting bias, so a plot in which several studies sit outside the contours is evidence about heterogeneity before it is evidence about anything else. Reading scatter outside the triangle as missing studies is a category error.

What funnel plot asymmetry can mean

Funnel plot asymmetry has at least five distinct explanations, and the 2011 BMJ recommendations by Sterne and colleagues (BMJ 2011;343:d4002) set them out as a checklist that authors are expected to work through rather than a menu from which publication bias is chosen.

Reporting biases. Publication bias in the narrow sense — whole studies unpublished because their results were unremarkable — but also selective outcome reporting, where the study is published and the outcome is not, and selective analysis reporting, where one of several analyses is presented. Selective outcome reporting is common in trials, and it is invisible to any funnel plot drawn from published effect estimates alone.

Poor methodological quality in smaller studies. Small trials are, on average, less likely to conceal allocation, to blind outcome assessment, or to analyse by intention to treat, and each of those deficits inflates effect estimates in a predictable direction. The result is a genuine association between study size and observed effect with no suppression of any kind. In observational syntheses the analogous mechanism is that smaller studies typically adjust for fewer confounders.

True heterogeneity related to size. Small studies often differ substantively from large ones: higher-risk participants, more intensive delivery of the intervention, shorter follow-up, single expert centres rather than multi-centre networks. If the effect genuinely is larger in those settings, the funnel is asymmetric and every study in it is honestly reported.

Artefact of the effect measure. For some measures the estimate and its standard error are mathematically linked — the standard error of a log odds ratio depends on the observed event counts, so a study with an extreme odds ratio necessarily has a larger standard error. That dependence bends the plot without any bias at all, and it is the reason the standard asymmetry test misbehaves on binary outcomes.

Chance. With ten studies, a funnel plot has ten points. Visual asymmetry in ten points is common under any null hypothesis you care to state.

Small-study effects: the term to use instead of publication bias

Small-study effects is the term for the phenomenon a funnel plot actually detects: a systematic tendency for smaller studies to show larger treatment effects than larger studies. Sterne, Gavaghan and Egger gave the phrase currency in Journal of Clinical Epidemiology in 2000 precisely because “publication bias” names a cause that the plot cannot identify.

The distinction is not pedantry, and it is the single most useful edit available to authors writing this section. “The funnel plot showed evidence of publication bias” asserts a mechanism from a display that cannot distinguish suppression from methodological quality, from heterogeneity, or from the arithmetic of the effect measure. “The funnel plot suggested small-study effects; publication bias, differential study quality and heterogeneity of intervention intensity are each plausible contributors, and we could not distinguish them” states exactly what the evidence supports. Reviewers in statistics-heavy fields read the first version as a claim the authors have not earned, and the objection appears often enough to sit alongside the standard statistical objections.

Contour-enhanced funnel plots

A contour-enhanced funnel plot overlays regions of statistical significance on the ordinary funnel plot, shading the areas in which a study’s result would reach p < 0.01, p < 0.05 and p < 0.10. Peters and colleagues introduced the device in Journal of Clinical Epidemiology in 2008 to separate the explanations above using the plot itself rather than an argument.

The logic is that publication bias, if it operates through statistical significance, removes studies from a specific region: the non-significant zone on the side of the null nearest the small studies. If the gap in the funnel falls in the shaded non-significant area, suppression by significance is a coherent reading. If the gap falls in an area where the missing studies would have been statistically significant, suppression by significance does not explain it, and heterogeneity or study quality moves up the list. This is the one modification to a funnel plot that changes what the plot can be used to conclude, which is why the 2011 BMJ recommendations single it out, and why a reviewer asking for one is asking for information rather than decoration.

The tests for funnel plot asymmetry

Egger’s test is the default and the most frequently misapplied. Egger, Davey Smith, Schneider and Minder published it in BMJ in 1997 (315:629–634): regress each study’s standard normal deviate — the effect estimate divided by its standard error — on its precision, and test whether the intercept differs from zero. A non-zero intercept means the effect estimate depends on precision, which is asymmetry expressed as a coefficient.

Begg and Mazumdar’s rank correlation test (Biometrics 1994;50:1088–1101) correlates standardised effect estimates with their variances and assumes no particular distribution, but it has substantially lower power than Egger’s test in most realistic configurations. Harbord’s modified test (Statistics in Medicine, 2006) replaces the effect and standard error with the efficient score and its Fisher information, which breaks the mathematical dependence between the two that inflates Egger’s false-positive rate on binary outcomes. Peters’ test (JAMA 2006;295:676–680) uses the reciprocal of the total sample size as the predictor for the same reason. Rücker’s arcsine test (Statistics in Medicine, 2008) applies a variance-stabilising transformation to proportions.

Choosing among them is not a matter of preference. For continuous outcomes with mean differences, Egger’s test is appropriate. For binary outcomes analysed as odds ratios — particularly when events are common, effects are large, or group sizes are unbalanced — Egger’s test rejects far more often than its nominal level, and Harbord’s or Peters’ test should be used instead. Submitting Egger’s test on log odds ratios is a specific, checkable wrong statistical test rather than a debatable judgement call.

The ten-study rule, and why a null result proves nothing

Ten studies is the threshold below which the Cochrane Handbook and the 2011 BMJ recommendations both advise against performing any test for funnel plot asymmetry. The reason is power: with fewer than ten studies the tests cannot distinguish real asymmetry from chance, so the result is uninformative in both directions.

Above the threshold the situation improves slowly rather than suddenly. Power depends on the number of studies, the spread of their sizes, and the magnitude of the bias; a synthesis of twelve studies of similar size has almost no ability to detect moderate small-study effects, because asymmetry tests are driven by the contrast between the precise and the imprecise studies and a set of uniformly sized studies supplies no such contrast. This is why a sentence of the form “Egger’s test was non-significant, indicating no publication bias” is among the most common defects in this part of a manuscript. The correct reading is that the analysis had insufficient power to detect asymmetry, which is the same post-hoc power reasoning error that appears when a null primary outcome is described as showing equivalence.

Trim and fill, and why it is a sensitivity analysis

Trim and fill is an algorithm published by Duval and Tweedie (Biometrics 2000;56:455–463) that estimates the number of studies apparently missing from one side of a funnel plot, imputes mirror-image studies to restore symmetry, and recomputes the pooled effect including the imputed points. The output is an adjusted estimate that is frequently, and incorrectly, presented as a corrected result.

Trim and fill assumes that asymmetry arises from suppression and that the suppressed studies are the mirror images of the observed ones. Terrin, Schmid, Lau and Olkin showed in Statistics in Medicine in 2003 that the method performs poorly when between-study heterogeneity is present, imputing studies that do not exist and moving the estimate away from the truth. Because heterogeneity is itself one of the causes of asymmetry, the conditions under which trim and fill is most likely to be invoked are the conditions under which it is least trustworthy. The defensible use is as a sensitivity analysis reported alongside the primary estimate — the pooled effect with its confidence interval, then the number of studies imputed and the adjusted estimate with its own interval, so the reader learns how fragile the conclusion is. The indefensible use is reporting the filled estimate as the finding.

What the funnel plot cannot do

A funnel plot cannot detect publication bias that is unrelated to study size. If an entire field suppresses results in one direction, if a single multi-centre trial goes unpublished, or if suppression operates through a mechanism other than significance-by-precision, the surviving studies can form a perfectly symmetrical funnel around a wrong pooled estimate. Symmetry is not evidence of completeness, and no manuscript should say that it is.

Visual inspection is also weaker than researchers believe. Terrin, Schmid and Lau reported in Journal of Clinical Epidemiology in 2005 that, in an empirical evaluation, researchers could not reliably identify publication bias from funnel plots, and Lau, Ioannidis, Terrin, Schmid and Olkin made the practical case against over-reading them in BMJ in 2006 under the title “The case of the misleading funnel plot”. The honest position — contested, and worth stating in a limitations paragraph rather than concealing — is that the funnel plot is a screening display with poor operating characteristics, retained because the alternatives are harder and because a picture of the evidence base is worth showing.

Stronger tools exist and belong in the same discussion. Selection models, including the Copas model and Vevea and Hedges’ weight-function approach, state the suppression mechanism explicitly and estimate its parameters, but they rest on assumptions that cannot be checked from the data. PET-PEESE, common in economics and psychology, extrapolates the meta-regression of effect on standard error to a standard error of zero. The most decisive evidence is not statistical at all: comparing the synthesis against trial registry records and protocols detects outcomes that were measured and not reported, which is the same discipline applied to a single study by registered endpoint checking, and it identifies suppression that no plot of published estimates can reach.

How the problem is detected in a real manuscript

Detecting a mishandled funnel plot in a submitted manuscript is a sequence of checks against the figure, its caption, the methods paragraph and the forest plot, and each check is concrete enough to run in a few minutes.

Count the points and reconcile them with the forest plot. A funnel plot with fourteen points against a forest plot of nine studies means subgroups or multiple time points from the same study have entered as independent observations, which violates the independence the plot assumes and manufactures clusters that read as asymmetry. This is the same non-independence failure as pseudoreplication in a primary experiment.

Read the vertical axis label. Sample size or precision on the vertical axis, drawn with straight-line contours, is an internal inconsistency: straight pseudo-confidence limits are only correct against standard error.

Check the scale of the horizontal axis for ratio measures. An odds ratio axis running 0 to 5 linearly, rather than logarithmically, produces asymmetry from symmetric data.

Match the test to the outcome type. Egger’s test named in the methods with log odds ratios as the effect measure is the most common mismatch; Harbord’s or Peters’ test is the expected substitution.

Count the studies against the threshold. Any asymmetry test performed on fewer than ten studies should be either removed or reported with an explicit statement that it is underpowered.

Read the interpretation sentence for the absence-of-evidence inversion. “No evidence of publication bias was found” following a non-significant test on a small number of studies is the specific claim to flag, and it recurs across syntheses often enough that it should be part of any overclaim check.

Trace asymmetry forward into the discussion and the certainty rating. A manuscript that reports asymmetry in the results and then rates the evidence without downgrading for reporting bias, or discusses the pooled effect as though the asymmetry had not been observed, has detected a problem and discarded it.

Check that reporting bias assessment was pre-specified. PRISMA 2020 item 14 asks authors to describe the methods used to assess risk of bias due to missing results; a funnel plot that appears only in the results, with no corresponding methods sentence and no protocol entry, is a post-hoc analysis presented as a prespecified one, which is the synthesis-level version of the selective analysis problem.

There is a structural reason these defects survive to publication. Editors assign statistical review unevenly, and asymmetry-test selection is exactly the kind of error a subject-matter reviewer will pass over, which is why statistical review capacity determines whether this class of problem is caught at all.

What a peer reviewer says when a funnel plot is mishandled

Reviewer comments on funnel plots are unusually formulaic, because the underlying errors are a short and well-known list. The phrasings below recur across journals and fields.

“With only seven included studies, a funnel plot and Egger’s test are uninformative and should be removed or explicitly described as exploratory.” “The authors interpret a non-significant asymmetry test as evidence of no publication bias; absence of evidence is not evidence of absence.” “Egger’s test is not appropriate for binary outcomes analysed as odds ratios; please repeat the analysis using the Harbord or Peters test.” “Funnel plot asymmetry is attributed to publication bias without consideration of heterogeneity, differential methodological quality, or the effect measure itself.” “Please provide a contour-enhanced funnel plot so that readers can assess whether the apparently missing studies fall in regions of statistical non-significance.” “The trim-and-fill adjusted estimate is presented as the primary result; it should be reported as a sensitivity analysis with the assumptions stated.” “The funnel plot appears to include more estimates than there are independent studies.” “The vertical axis is plotted against sample size; please replot against standard error.” “Reporting bias assessment does not appear in the protocol or the registration record.”

Each of these is answerable, and most are answerable without new analysis, which makes them a favourable class of comment to receive — the response usually involves restating a conclusion accurately rather than defending it, and that distinction is worth making explicit when writing the response letter.

How to report a funnel plot well

Report the number of studies contributing to the plot in the caption, alongside the effect measure, the scale of the horizontal axis, and the quantity on the vertical axis. State in the methods, before the results, that asymmetry would be assessed, by which test, and above what number of studies — and if the threshold was not reached, say that the assessment was not performed rather than performing it anyway.

Write the interpretation in terms of small-study effects and enumerate the candidate explanations for the observed asymmetry, ranking them by plausibility given what is known about the included studies. Where a contour-enhanced plot was drawn, say which significance region the gap falls in. Where trim and fill was run, report the number of imputed studies and both estimates. Where asymmetry was found, carry it into the certainty-of-evidence rating and into the discussion, and state which conclusions would change if the missing studies were as the plot suggests.

Scope note

PerfectPaper reviews original research articles rather than systematic reviews and meta-analyses. A funnel plot inside an original research article — a consortium meta-analysis pooling the study’s own cohorts, a genetic association paper combining discovery and replication samples, a trial reported alongside an updated pooled analysis — is within that scope, and the checks above apply to it unchanged.

Related

Confounding · Collider bias · Pseudoreplication · Reviewer objections about statistics · Research methods concepts

Checked before submission by the epidemiology review lane, which reports how asymmetry was assessed, whether the test matches the effect measure, and whether the interpretation is stated as small-study effects rather than as publication bias.

Review my manuscript

Frequently asked questions

What does a funnel plot show?

A funnel plot shows each study in a meta-analysis as a point, positioned by its effect estimate horizontally and its standard error vertically, with the most precise studies at the top. The expected shape is a symmetrical inverted funnel, because imprecise studies scatter widely and precise studies cluster near the pooled effect.

What is a funnel plot used for in meta-analysis?

A funnel plot in meta-analysis is used to judge whether smaller studies report systematically different effects from larger ones. That pattern, called small-study effects, is consistent with publication bias, with poorer methods in small studies, with genuine heterogeneity related to study size, and with artefacts of the effect measure.

How do you read a funnel plot?

Read a funnel plot by comparing the scatter on each side of the pooled estimate against the triangular pseudo-confidence region drawn at the pooled effect plus or minus 1.96 standard errors. A gap in the wide lower portion on one side indicates that small studies with results in that direction are absent from the synthesis.

What is an asymmetric funnel plot?

An asymmetric funnel plot means the observed effect depends on study precision. Publication bias is one explanation; selective outcome reporting, weaker methodology in small trials, true heterogeneity in intervention intensity, mathematical dependence between an odds ratio and its standard error, and chance are the others, and a plot alone cannot separate them.

Why is it called a funnel plot?

The display is called a funnel plot because of its expected shape: with standard error on an inverted vertical axis, unbiased studies form an inverted cone that is wide at the imprecise base and narrow at the precise apex. Light and Pillemer named it in Summing Up: The Science of Reviewing Research in 1984.

What is a contour-enhanced funnel plot?

A contour-enhanced funnel plot is an ordinary funnel plot with regions of statistical significance shaded, typically at p < 0.01, p < 0.05 and p < 0.10. Shading lets a reader see whether the apparently missing studies would have been non-significant, which supports suppression by significance, or significant, which does not.

What does a symmetrical funnel plot mean?

A symmetrical funnel plot does not mean there is no publication bias. It means no size-related asymmetry was visible, which is a weaker statement. Suppression unrelated to study size, one unpublished large trial, or too few studies to see a pattern all leave the funnel symmetrical, so symmetry supports no conclusion about the completeness of the evidence base.