What is heterogeneity in meta-analysis?

Heterogeneity in meta-analysis is variation in the true effect sizes underlying the pooled studies, beyond what sampling error alone would produce. When heterogeneity is present, the studies are estimating different quantities rather than one shared effect, so a pooled estimate describes the average of a distribution of effects rather than a value every study has in common.

That definition is short, and most of the practical difficulty in evidence synthesis comes from measuring the variation correctly and then saying what it implies. This page covers the three kinds of heterogeneity, what Cochran’s Q tests, what I² does and does not measure, why τ² is the absolute quantity, how prediction intervals change the reading of a pooled result, how heterogeneity is detected in a submitted manuscript, and what reviewers write when it has been handled badly.

Heterogeneity is variation in the effects, not disagreement in the results

Heterogeneity is a property of the underlying effects, not of the observed numbers. Two trials can report point estimates of 0.55 and 0.95 and be perfectly consistent with a single true effect, if both are small and their confidence intervals overlap heavily. Two other trials can report 0.78 and 0.84 and be strongly heterogeneous, if each has 40,000 participants and confidence intervals a few hundredths wide.

The question is never “do these studies look different?” It is “do these studies differ by more than their own imprecision can explain?” Every statistic on this page answers a different part of that one question: Cochran’s Q asks whether the excess variation is detectable, I² asks what proportion of the observed variation it represents, and τ² asks how large it is on the scale of the effect measure.

The three kinds: clinical, methodological, statistical

Meta-analysts distinguish three kinds of heterogeneity, and conflating them is the most common conceptual error in a synthesis discussion section.

Clinical heterogeneity is variation in participants, interventions and outcomes: different age ranges, different doses, different comparators, different definitions of the endpoint. Clinical heterogeneity is present in almost every synthesis and is assessed by reading the studies, not by computing anything.

Methodological heterogeneity is variation in study design and conduct: randomised versus non-randomised allocation, blinded versus open outcome assessment, different lengths of follow-up, different handling of missing data.

Statistical heterogeneity is the observable consequence — variation in the effect estimates greater than chance would produce. Statistical heterogeneity is what Q, I², τ² and prediction intervals quantify, and it is the only one of the three with a number attached.

Clinical and methodological heterogeneity cause statistical heterogeneity, which is why a synthesis that reports I² without discussing what differed across the studies has described a symptom and skipped the diagnosis.

Cochran’s Q and what the heterogeneity test actually tests

Cochran’s Q, introduced by William Cochran in 1954, is the weighted sum of squared deviations of each study’s effect estimate from the pooled fixed-effect estimate. Under the null hypothesis that every study estimates the same true effect, Q follows a chi-squared distribution with k − 1 degrees of freedom for k studies.

Q has one well-documented failing that shapes how it should be read. The Cochrane Handbook (section 10.10.2) states: “Care must be taken in the interpretation of the Chi2 test, since it has low power in the (common) situation of a meta-analysis when studies have small sample size or are few in number.” A non-significant Q is therefore not evidence of homogeneity; in a synthesis of five small trials it is close to uninformative. The converse also holds: with twenty large trials, Q reaches significance on differences too small to matter clinically.

Because of the power problem, many authors use a threshold of p < 0.10 rather than p < 0.05 for the Q test. That convention raises sensitivity and does nothing about the second failure mode, which is why Q is now normally reported alongside I² and τ² rather than used as a decision rule on its own.

What I² measures — and the thing it is routinely mistaken for

I², introduced by Higgins and Thompson in Statistics in Medicine in 2002, is the percentage of total variation across studies attributable to heterogeneity rather than to chance, computed as I² = (Q − df) / Q × 100%, with negative values set to zero.

A worked instance: eight trials give Q = 21.0 on 7 degrees of freedom (p ≈ 0.004), so I² = (21.0 − 7) / 21.0 = 66.7%. The Cochrane Handbook’s rough guide reads 0% to 40% as “might not be important”, 30% to 60% as “may represent moderate heterogeneity”, 50% to 90% as “may represent substantial heterogeneity”, and 75% to 100% as “considerable heterogeneity” — deliberately overlapping bands, because the Handbook intends them as orientation rather than as cut-points.

The routine mistake is reading I² as the amount of heterogeneity. I² is a ratio, and its denominator includes the within-study variance, which shrinks as studies get larger. Rücker, Schwarzer, Carpenter and Schumacher set out the consequence in BMC Medical Research Methodology in 2008, under the title “Undue reliance on I² in assessing heterogeneity may mislead”: I² rises towards 100% as precision increases even when τ² is held exactly constant, so I² should not be the basis for deciding whether pooling is defensible.

The arithmetic makes the point in one line. With a between-study variance of τ² = 0.02 on the log-odds scale and a typical within-study variance of 0.10, I² is 0.02 / 0.12 ≈ 17%. Run the same trials at ten times the sample size, so the within-study variance falls to 0.01, and I² becomes 0.02 / 0.03 ≈ 67%. The heterogeneity has not changed at all. Only the precision has. An I² of 67% in a synthesis of very large trials and an I² of 67% in a synthesis of pilot studies are not the same finding.

τ², the absolute measure

τ² is the estimated variance of the true effects across studies, expressed on the scale of the effect measure — log risk ratio, log odds ratio, standardised mean difference — and τ, its square root, is that spread in units a reader can interpret.

τ² is the quantity that does not move when precision changes, which is why it belongs in every report of a random-effects synthesis. A τ of 0.25 on the log risk ratio scale means the middle of the effect distribution spans roughly exp(±0.25), or about 0.78 to 1.28 around the average — a statement a clinician can act on, which “I² = 67%” is not.

Estimating τ² is itself contested. The DerSimonian and Laird method-of-moments estimator (1986) is the historical default and is known to underestimate τ² when studies are few. Veroniki and colleagues, in Research Synthesis Methods (2016; 7(1):55–79), identified sixteen estimators and reported that “simulation studies suggest that for both dichotomous and continuous data the estimator proposed by Paule and Mandel and for continuous data the restricted maximum likelihood estimator are better alternatives to estimate the between-study variance” — while noting that their recommendations rest on “a qualitative evaluation of the existing literature and expert consensus.” Any manuscript reporting τ² should name the estimator, because two estimators on the same data can differ enough to change the width of a prediction interval materially.

τ² also carries its own uncertainty, and with fewer than about ten studies that uncertainty is large. A confidence interval for τ², via the Q-profile method or a generalised Cochran between-study variance statistic, is the honest accompaniment to a point estimate and is reported far less often than it should be.

What a random-effects model does with heterogeneity

A random-effects model does not remove heterogeneity or correct for it. A random-effects model changes the estimand: instead of estimating one common effect, it estimates the mean of a distribution of true effects whose variance is τ².

Two consequences follow, and both are frequently misstated in manuscripts. First, the pooled estimate is no longer “the effect”; it is the average effect across the population of settings the studies represent, and it may correspond to no individual study. Second, random-effects weighting is flatter than fixed-effect weighting, so small studies gain influence relative to large ones. Where small studies are systematically different — through publication selection or lower methodological quality — a random-effects pooled estimate can be further from the truth than a fixed-effect one, not closer.

The standard random-effects confidence interval also has poor coverage when studies are few. IntHout, Ioannidis and Borm compared the two approaches in BMC Medical Research Methodology in 2014 and reported that the Hartung–Knapp–Sidik–Jonkman adjustment held type I error rates far closer to the nominal level than DerSimonian–Laird, whose intervals were too narrow under heterogeneity; re-analysing a large set of Cochrane meta-analyses, they found that a substantial share of results significant under DerSimonian–Laird were no longer significant under HKSJ. A synthesis of six trials reporting an unadjusted DerSimonian–Laird interval is therefore making a stronger claim than its data support, and this is now a routine statistical objection at review.

Prediction intervals: the interval that answers the question readers are asking

A prediction interval gives the range in which the true effect of a new study, in a new but similar setting, is expected to fall — as distinct from the confidence interval, which describes only the precision of the estimated average. Higgins, Thompson and Spiegelhalter proposed its routine use in the Journal of the Royal Statistical Society Series A in 2009, and it is computed as the pooled estimate ± t(k−2) × √(τ̂² + SE(μ̂)²).

The distinction is not academic. Take a pooled log risk ratio of −0.22 (RR 0.80) with SE 0.08, τ² = 0.06, from k = 10 studies. The 95% confidence interval is −0.22 ± 1.96 × 0.08, giving RR 0.69 to 0.94: apparently a settled benefit. The 95% prediction interval is −0.22 ± 2.306 × √(0.06 + 0.0064) = −0.22 ± 0.594, giving RR 0.44 to 1.45. The same data support a statement that the average effect is beneficial and a statement that the next trial could plausibly show harm.

IntHout, Ioannidis, Rovers and Goeman quantified how often this happens. Re-examining statistically significant random-effects Cochrane meta-analyses in BMJ Open in 2016 — a paper titled “Plea for routinely presenting prediction intervals in meta-analysis” — they found that in the large majority the 95% prediction interval still admitted a null effect, and in a sizeable minority it extended into effects opposite in direction to the summary estimate. The Cochrane Handbook (section 10.10.4.3) accordingly encourages prediction intervals “when the number of studies is reasonable (e.g. five or more)”, and warns that they “can be very problematic when the number of studies is small, in which case they can appear spuriously wide or spuriously narrow” — the Handbook presents the interval on k − 1 degrees of freedom rather than the k − 2 of the original proposal.

How heterogeneity is detected in a real manuscript

Detecting mishandled heterogeneity in a submitted manuscript is a matter of checking six specific things, in this order, and most defects surface in the first three.

Recompute I² from Q and df. Both are usually printed in the forest plot header. If I² = (Q − df) / Q does not reproduce the stated value, the numbers have come from different models or different subsets — commonly a forest plot regenerated after studies were added, with the text left unchanged.

Check that τ² is reported at all, and that the estimator is named. A results section giving I² and p-for-heterogeneity but no τ² cannot support any statement about how large the variation is, and a τ² with no named estimator cannot be reproduced.

Compare the forest plot against the narrative. A plot with non-overlapping confidence intervals and a discussion describing “consistent findings across studies” is a direct contradiction, and it is common. So is the reverse: I² = 0% in six tiny trials described as evidence of homogeneity, when Q had almost no power to detect anything.

Check whether the model was switched after the fact. A protocol or registration specifying a fixed-effect analysis, and a paper reporting random effects, means the model was chosen after the heterogeneity was seen. That is sometimes defensible and always needs stating; unstated, it becomes a model-choice objection.

Check the number of studies against the number of subgroup and meta-regression analyses. Four covariates explored across nine studies is not an investigation of heterogeneity; it is an exercise in multiplicity with a near-certain spurious finding.

Check for dependence in the data structure. Multiple effect sizes drawn from the same cohort, one trial arm used as a comparator twice, or several timepoints entered as separate studies all inflate the apparent number of independent units, which is pseudoreplication in a synthesis and distorts both the pooled estimate and τ².

Investigating the causes: subgroup analysis and meta-regression

Subgroup analysis and meta-regression are the two standard tools for explaining heterogeneity, and both are observational analyses regardless of how the included studies were designed.

Subgroup analysis splits the studies by a characteristic and tests whether the pooled effects differ. The correct statistic is a formal test for subgroup differences, not a comparison of whether each subgroup’s own confidence interval excludes the null — a subgroup reaching significance while another fails to is not evidence that the two differ, a fallacy that survives in the literature because it is superficially intuitive.

Meta-regression fits study-level covariates against effect estimates, and carries a distinctive hazard: aggregation bias. A relationship between a study’s mean baseline severity and its effect size is a relationship among study averages, and need not hold among individuals within any study. Study-level confounding also has no design protection here — the studies that used one dose also tended to be the newer, larger, better-blinded ones, and no covariate in the model separates those.

The Cochrane Handbook is blunt about the credibility of both: “Reliable conclusions can only be drawn from analyses that are truly pre-specified before inspecting the studies’ results,” and “explorations of heterogeneity that are devised after heterogeneity is identified can at best lead to the generation of hypotheses.” For grading a subgroup claim, the ICEMAN instrument (Schandelmaier et al., CMAJ 2020; 192(32):E901–E906) supplies a structured set of core questions specific to meta-analyses, covering pre-specification, whether the direction was predicted in advance, the test for interaction, and whether the effect modifier was measured within studies or only between them.

Heterogeneity is not the same as small-study effects or publication bias

Small-study effects and publication bias are distinct phenomena from heterogeneity, and merging the three in a discussion section leaves all of them unaddressed.

Small-study effects describe a systematic relationship between a study’s precision and its estimated effect — smaller studies reporting larger effects. Publication bias is one possible cause of that pattern; others include genuinely different populations in early small trials, lower methodological quality, and selective outcome reporting.

The relationship to heterogeneity runs one way. Small-study effects produce statistical heterogeneity, so a high I² is consistent with them. But heterogeneity has many other causes, and funnel plot asymmetry cannot be interpreted as publication bias when substantial heterogeneity is present, because heterogeneity itself generates asymmetry. Cochrane guidance advises against funnel plot asymmetry tests with fewer than ten studies, for the same reason the Q test is weak: too little power to distinguish anything.

Contested points and honest limitations

Several parts of heterogeneity practice are unsettled, and a manuscript that acknowledges the disagreement reads better than one that does not.

The I² thresholds have no empirical basis. The Cochrane bands are explicitly a “rough guide” whose interpretation the Handbook says depends on the magnitude of the effect and the strength of evidence for heterogeneity. Treating 50% as a decision boundary — for pooling, for downgrading, for anything — is a convention, not a result, and the Handbook does not endorse it.

There is no agreed threshold at which pooling becomes inappropriate. Some methodologists hold that if a common estimand cannot be defined, no amount of statistical machinery makes pooling meaningful; others hold that a pooled average with a prediction interval is always more informative than narrative synthesis. Neither position is settled, and the honest manuscript states which one it took and why.

τ² is badly estimated when studies are few, which is the usual case. A synthesis of five studies reports a τ² whose own confidence interval may span an order of magnitude, and every prediction interval derived from it inherits that instability.

I² of 0% does not mean the studies agree. It means Q did not exceed its degrees of freedom, which happens routinely in small syntheses through lack of power, and the resulting random-effects analysis collapses to the fixed-effect one without anyone having decided that it should.

What reviewers say when heterogeneity is mishandled

Reviewers of syntheses use a recognisable set of phrasings when heterogeneity has been reported without being addressed.

“Substantial heterogeneity (I² = 78%) is reported but not explained; the authors should identify its likely sources rather than defaulting to a random-effects model.” “Pooling does not appear appropriate given the clinical diversity of the included populations.” “The authors report I² without τ² or a prediction interval; the reader cannot judge the magnitude of between-study variation.” “A non-significant Q test is interpreted as absence of heterogeneity, which the test’s power does not support with six studies.” “The subgroup analyses were not pre-specified and no test for interaction is reported.” “The random-effects confidence interval should be presented with the Hartung–Knapp adjustment given the small number of studies.” “Please state which estimator was used for τ².”

Two of those decide papers more often than the rest. The first is the demand for a prediction interval, because supplying one frequently changes the conclusion the abstract can carry — and an abstract stating a benefit the prediction interval does not support is the precise shape of overclaim that editors return. The second is the objection that the pooled estimate has no clear interpretation, which is a request to justify the estimand rather than to add an analysis, and it cannot be answered with more statistics. When a response letter has to concede that pooling was optimistic, saying so directly and adding a stratified presentation reads far better than a defence; the structure for that concession is set out in how to write a response to reviewers.

How to report heterogeneity

Report heterogeneity as four elements together. PRISMA 2020 requires that statistical heterogeneity be described and reported, though it names none of the four measures individually.

Give Q with its degrees of freedom and p value, I², τ² with its estimator named, and a 95% prediction interval whenever five or more studies are pooled. PRISMA 2020 item 13d asks authors to “describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used”; item 13e asks for the methods “used to explore possible causes of heterogeneity among study results (e.g. subgroup analysis, meta-regression)”; item 20b asks that results present “the summary estimate and its precision (e.g. confidence/credible interval) and measures of statistical heterogeneity.”

Beyond the checklist, three things separate a strong report. State the estimand — whether the pooled figure is a common effect or the mean of a distribution — in one sentence. Distinguish pre-specified from post hoc investigations explicitly, by name, rather than describing all of them in the same neutral past tense. And write the discussion against the prediction interval rather than the confidence interval, since the prediction interval is what a reader deciding whether the result applies to their own setting actually needs — the same discipline that keeps a claim about an effect’s practical size tied to the numbers supporting it.

Scope note

PerfectPaper reviews original research articles, and a systematic review or meta-analysis manuscript falls outside that scope. Heterogeneity nevertheless appears throughout original research: pooled analyses across cohorts, multi-site randomised trials reporting site-by-treatment interaction, individual participant data analyses, and primary papers that append a meta-analysis of prior work to their own results. In each of those the statistical questions on this page apply unchanged, and they are among the objections raised most often in statistical peer review and in clinical trial reporting.

Related

Confounding · Collider bias · Pseudoreplication · Multiple comparisons objections · Research methods concepts

Review my manuscript

Frequently asked questions

What does heterogeneity mean in a meta-analysis?

Heterogeneity means the studies being pooled are estimating different true effects rather than one shared effect. Variation among the reported estimates exceeds what each study’s own sampling error can explain, so the pooled figure represents the average of a distribution of effects rather than a single value common to every study in the analysis.

What is statistical heterogeneity between studies?

Statistical heterogeneity is measurable excess variation in effect estimates across studies, beyond chance. Cochran’s Q tests whether that excess is detectable, I² expresses it as a percentage of total variation, and τ² gives its size on the scale of the effect measure. Clinical and methodological differences between the studies are its causes.

What is heterogeneity in a systematic review?

Heterogeneity in a systematic review is variation among the included studies — in participants, interventions, outcomes and design, and consequently in their effect estimates. A review reports it through Cochran’s Q, I², τ² and a prediction interval, then assesses whether the studies are similar enough that a pooled estimate answers a coherent question.

What does it mean when studies in a meta-analysis are heterogeneous?

Heterogeneous studies are estimating different underlying effects, so no single number describes all of them. A pooled estimate remains computable and reports the average across the settings those studies represent, but the range a new study would be expected to fall in is given by the prediction interval, which is typically much wider than the confidence interval.

What is between-study heterogeneity?

Between-study heterogeneity is variation in the true effects from one study to another, quantified by τ², the variance of that effect distribution. Between-study heterogeneity is distinct from within-study variation, which is each study’s own imprecision. I² is the ratio of the first to the total of both, which is why I² rises as the pooled studies grow larger.

How is heterogeneity defined in evidence synthesis?

Evidence synthesis defines heterogeneity as variability among the studies in a review, split into three kinds: clinical variability in participants, interventions and outcomes; methodological variability in design and risk of bias; and statistical variability in the effect estimates beyond chance. Only the third carries a numerical measure, and the first two cause it.

In simple terms, what is heterogeneity in meta-analysis?

Heterogeneity means the pooled studies do not all measure the same thing. Their results differ by more than measurement noise explains, usually because the populations, doses, outcome definitions or study designs differ. The consequence is that an average across them describes a range of effects, and that range belongs in the report alongside the average.