What is heterogeneity in meta-analysis?

Heterogeneity in meta-analysis is variation in the true effect sizes underlying the pooled studies, beyond what sampling error alone would produce. When heterogeneity is present, the studies are estimating different quantities rather than one shared effect, so a pooled estimate describes the average of a distribution of effects rather than a value every study has in common.

That definition is short, and most of the practical difficulty in evidence synthesis comes from measuring the variation correctly and then saying what it implies. This page covers the three kinds of heterogeneity, what Cochran’s Q tests, what I² does and does not measure, why τ² is the absolute quantity, how prediction intervals change the reading of a pooled result, how heterogeneity is detected in a submitted manuscript, and what reviewers write when it has been handled badly.

Heterogeneity is variation in the effects, not disagreement in the results

Heterogeneity is a property of the underlying effects, not of the observed numbers. Two trials can report point estimates of 0.55 and 0.95 and be perfectly consistent with a single true effect, if both are small and their confidence intervals overlap heavily. Two other trials can report 0.78 and 0.84 and be strongly heterogeneous, if each has 40,000 participants and confidence intervals a few hundredths wide.

The question is never “do these studies look different?” It is “do these studies differ by more than their own imprecision can explain?” Every statistic on this page answers a different part of that one question: Cochran’s Q asks whether the excess variation is detectable, I² asks what proportion of the observed variation it represents, and τ² asks how large it is on the scale of the effect measure.

The three kinds: clinical, methodological, statistical

Meta-analysts distinguish three kinds of heterogeneity, and conflating them is the most common conceptual error in a synthesis discussion section.

Clinical heterogeneity is variation in participants, interventions and outcomes: different age ranges, different doses, different comparators, different definitions of the endpoint. Clinical heterogeneity is present in almost every synthesis and is assessed by reading the studies, not by computing anything.

Methodological heterogeneity is variation in study design and conduct: randomised versus non-randomised allocation, blinded versus open outcome assessment, different lengths of follow-up, different handling of missing data.

Statistical heterogeneity is the observable consequence — variation in the effect estimates greater than chance would produce. Statistical heterogeneity is what Q, I², τ² and prediction intervals quantify, and it is the only one of the three with a number attached.

Clinical and methodological heterogeneity cause statistical heterogeneity, which is why a synthesis that reports I² without discussing what differed across the studies has described a symptom and skipped the diagnosis.

Cochran’s Q and what the heterogeneity test actually tests

Cochran’s Q, introduced by William Cochran in 1954, is the weighted sum of squared deviations of each study’s effect estimate from the pooled fixed-effect estimate. Under the null hypothesis that every study estimates the same true effect, Q follows a chi-squared distribution with k − 1 degrees of freedom for k studies.

Q has one well-documented failing that shapes how it should be read. The Cochrane Handbook (section 10.10.2) states: “Care must be taken in the interpretation of the Chi2 test, since it has low power in the (common) situation of a meta-analysis when studies have small sample size or are few in number.” A non-significant Q is therefore not evidence of homogeneity; in a synthesis of five small trials it is close to uninformative. The converse also holds: with twenty large trials, Q reaches significance on differences too small to matter clinically.

Because of the power problem, many authors use a threshold of p < 0.10 rather than p < 0.05 for the Q test. That convention raises sensitivity and does nothing about the second failure mode, which is why Q is now normally reported alongside I² and τ² rather than used as a decision rule on its own.

What I² measures — and the thing it is routinely mistaken for

I², introduced by Higgins and Thompson in Statistics in Medicine in 2002, is the percentage of total variation across studies attributable to heterogeneity rather than to chance, computed as I² = (Q − df) / Q × 100%, with negative values set to zero.

A worked instance: eight trials give Q = 21.0 on 7 degrees of freedom (p ≈ 0.004), so I² = (21.0 − 7) / 21.0 = 66.7%. The Cochrane Handbook’s rough guide reads 0% to 40% as “might not be important”, 30% to 60% as “may represent moderate heterogeneity”, 50% to 90% as “may represent substantial heterogeneity”, and 75% to 100% as “considerable heterogeneity” — deliberately overlapping bands, because the Handbook intends them as orientation rather than as cut-points.

The routine mistake is reading I² as the amount of heterogeneity. I² is a ratio, and its denominator includes the within-study variance, which shrinks as studies get larger. Rücker, Schwarzer, Carpenter and Schumacher set out the consequence in BMC Medical Research Methodology in 2008, under the title “Undue reliance on I² in assessing heterogeneity may mislead”: I² rises towards 100% as precision increases even when τ² is held exactly constant, so I² should not be the basis for deciding whether pooling is defensible.

The arithmetic makes the point in one line. With a between-study variance of τ² = 0.02 on the log-odds scale and a typical within-study variance of 0.10, I² is 0.02 / 0.12 ≈ 17%. Run the same trials at ten times the sample size, so the within-study variance falls to 0.01, and I² becomes 0.02 / 0.03 ≈ 67%. The heterogeneity has not changed at all. Only the precision has. An I² of 67% in a synthesis of very large trials and an I² of 67% in a synthesis of pilot studies are not the same finding.

I² also depends on which effect measure is pooled, not only on precision. Recomputing the same trials as a risk ratio versus a risk difference, or as a standardised mean difference versus a raw mean difference, changes the within-study variance term inside Q and therefore changes I² on identical data. No single “true” I² exists for a set of studies independent of the scale chosen to combine them — a reason to read I² as a property of a specific analysis, not of the studies themselves.

Common questions about I² and heterogeneity tests

Why is my I² so high when all my studies show effects in the same direction?

Direction of effect and I² measure different things. I² is the share of variance beyond chance, and studies can agree on direction while disagreeing sharply on magnitude: five trials all favouring treatment, with risk ratios anywhere from 0.85 to 0.35, still show high I² even though not one of them crossed into harm.

Is an I² of 0% proof that my studies are homogeneous?

No. An I² of 0% only means Cochran’s Q did not exceed its degrees of freedom, which happens routinely in a synthesis of four or five small trials simply because the test lacks power, not because the true effects agree. Check the confidence interval around I² before treating a zero as a finding.

What counts as a good I² value for a meta-analysis?

There is no validated cutoff. The Cochrane Handbook’s 0–40%, 30–60%, 50–90% and 75–100% bands are explicitly a rough guide, deliberately overlapping, meant for orientation rather than pass-or-fail thresholds. Treating 50% as a hard line for whether to pool is a convention some journals enforce, not a statistical result.

I have 6 trials for my meta-analysis and I² came out at 82 percent — should I still report a pooled estimate or just describe the studies separately?

High I² does not by itself rule out pooling; it means the pooled figure needs a prediction interval alongside it, not that pooling must be abandoned. Report the pooled estimate, its prediction interval, and the study characteristics that plausibly explain the 82%. Abandon pooling entirely only if no common estimand can be argued for at all.

τ², the absolute measure

τ² is the estimated variance of the true effects across studies, expressed on the scale of the effect measure — log risk ratio, log odds ratio, standardised mean difference — and τ, its square root, is that spread in units a reader can interpret.

τ² is the quantity that does not move when precision changes, which is why it belongs in every report of a random-effects synthesis. A τ of 0.25 on the log risk ratio scale means the middle of the effect distribution spans roughly exp(±0.25), or about 0.78 to 1.28 around the average — a statement a clinician can act on, which “I² = 67%” is not.

Estimating τ² is itself contested. The DerSimonian and Laird method-of-moments estimator (1986) is the historical default and is known to underestimate τ² when studies are few. Veroniki and colleagues, in Research Synthesis Methods (2016; 7(1):55–79), identified sixteen estimators and reported that “simulation studies suggest that for both dichotomous and continuous data the estimator proposed by Paule and Mandel and for continuous data the restricted maximum likelihood estimator are better alternatives to estimate the between-study variance” — while noting that their recommendations rest on “a qualitative evaluation of the existing literature and expert consensus.” Any manuscript reporting τ² should name the estimator, because two estimators on the same data can differ enough to change the width of a prediction interval materially.

τ² also carries its own uncertainty, and with fewer than about ten studies that uncertainty is large. A confidence interval for τ², via the Q-profile method or a generalised Cochran between-study variance statistic, is the honest accompaniment to a point estimate and is reported far less often than it should be.

τ² values are also not comparable across meta-analyses that use different effect measures or different outcomes. A τ² of 0.05 on the log-odds scale for a rare binary outcome and a τ² of 0.05 for a continuous outcome measured in standardised mean-difference units describe unrelated amounts of dispersion; only τ² values computed on the same scale, within the same synthesis or across closely comparable ones, support a direct comparison.

What a random-effects model does with heterogeneity

A random-effects model does not remove heterogeneity or correct for it. A random-effects model changes the estimand: instead of estimating one common effect, it estimates the mean of a distribution of true effects whose variance is τ².

Two consequences follow, and both are frequently misstated in manuscripts. First, the pooled estimate is no longer “the effect”; it is the average effect across the population of settings the studies represent, and it may correspond to no individual study. Second, random-effects weighting is flatter than fixed-effect weighting, so small studies gain influence relative to large ones. Where small studies are systematically different — through publication selection or lower methodological quality — a random-effects pooled estimate can be further from the truth than a fixed-effect one, not closer.

The standard random-effects confidence interval also has poor coverage when studies are few. IntHout, Ioannidis and Borm compared the two approaches in BMC Medical Research Methodology in 2014 and reported that the Hartung–Knapp–Sidik–Jonkman adjustment held type I error rates far closer to the nominal level than DerSimonian–Laird, whose intervals were too narrow under heterogeneity; re-analysing a large set of Cochrane meta-analyses, they found that a substantial share of results significant under DerSimonian–Laird were no longer significant under HKSJ. A synthesis of six trials reporting an unadjusted DerSimonian–Laird interval is therefore making a stronger claim than its data support, and this is now a routine statistical objection at review.

HKSJ is not unconditionally the safer choice, though. A documented edge case in the methods literature shows that when the between-study variance estimate rounds to zero, the Hartung–Knapp–Sidik–Jonkman adjustment can produce a confidence interval narrower than DerSimonian–Laird’s — the opposite of its usual conservative behaviour. A synthesis should check where its τ̂² point estimate actually falls rather than switching to HKSJ as an unconditional default.

Prediction intervals: the interval that answers the question readers are asking

A prediction interval gives the range in which the true effect of a new study, in a new but similar setting, is expected to fall — as distinct from the confidence interval, which describes only the precision of the estimated average. Higgins, Thompson and Spiegelhalter proposed its routine use in the Journal of the Royal Statistical Society Series A in 2009, and it is computed as the pooled estimate ± t(k−2) × √(τ̂² + SE(μ̂)²).

The distinction is not academic. Take a pooled log risk ratio of −0.22 (RR 0.80) with SE 0.08, τ² = 0.06, from k = 10 studies. The 95% confidence interval is −0.22 ± 1.96 × 0.08, giving RR 0.69 to 0.94: apparently a settled benefit. The 95% prediction interval is −0.22 ± 2.306 × √(0.06 + 0.0064) = −0.22 ± 0.594, giving RR 0.44 to 1.45. The same data support a statement that the average effect is beneficial and a statement that the next trial could plausibly show harm.

IntHout, Ioannidis, Rovers and Goeman quantified how often this happens. Re-examining statistically significant random-effects Cochrane meta-analyses in BMJ Open in 2016 — a paper titled “Plea for routinely presenting prediction intervals in meta-analysis” — they found that in the large majority the 95% prediction interval still admitted a null effect, and in a sizeable minority it extended into effects opposite in direction to the summary estimate. The Cochrane Handbook (section 10.10.4.3) accordingly encourages prediction intervals “when the number of studies is reasonable (e.g. five or more)”, and warns that they “can be very problematic when the number of studies is small, in which case they can appear spuriously wide or spuriously narrow” — the Handbook presents the interval on k − 1 degrees of freedom rather than the k − 2 of the original proposal.

Questions about pooling, prediction intervals and model choice

What is the difference between a confidence interval and a prediction interval in a meta-analysis?

A confidence interval describes the precision of the pooled average effect; a prediction interval describes the range in which a new study’s true effect is expected to fall, and is almost always wider. A synthesis can show a confidence interval entirely on the beneficial side while its prediction interval still crosses into harm.

Fixed effect or random effects — which model should I use?

Use random effects whenever the included studies differ in population, dose, follow-up or design, which is almost every real synthesis. Fixed-effect models assume one true effect shared by every study, an assumption clinical and methodological heterogeneity usually violate before any statistic is even computed, so random effects is the safer default.

My supervisor wants a prediction interval added to my meta-analysis chapter — what is it, and do I actually have to include one?

A prediction interval is the range a new study’s true effect would be expected to fall in, computed from the pooled estimate, its standard error and τ². The Cochrane Handbook recommends one whenever five or more studies are pooled, so your supervisor’s request matches standard guidance rather than an extra step.

How heterogeneity is detected in a real manuscript

Detecting mishandled heterogeneity in a submitted manuscript is a matter of checking six specific things, in this order, and most defects surface in the first three.

Recompute I² from Q and df. Both are usually printed in the forest plot header. If I² = (Q − df) / Q does not reproduce the stated value, the numbers have come from different models or different subsets — commonly a forest plot regenerated after studies were added, with the text left unchanged.

Check that τ² is reported at all, and that the estimator is named. A results section giving I² and p-for-heterogeneity but no τ² cannot support any statement about how large the variation is, and a τ² with no named estimator cannot be reproduced.

Compare the forest plot against the narrative. A plot with non-overlapping confidence intervals and a discussion describing “consistent findings across studies” is a direct contradiction, and it is common. So is the reverse: I² = 0% in six tiny trials described as evidence of homogeneity, when Q had almost no power to detect anything.

Check whether the model was switched after the fact. A protocol or registration specifying a fixed-effect analysis, and a paper reporting random effects, means the model was chosen after the heterogeneity was seen. That is sometimes defensible and always needs stating; unstated, it becomes a model-choice objection.

Check the number of studies against the number of subgroup and meta-regression analyses. Four covariates explored across nine studies is not an investigation of heterogeneity; it is an exercise in multiplicity with a near-certain spurious finding.

Check for dependence in the data structure. Multiple effect sizes drawn from the same cohort, one trial arm used as a comparator twice, or several timepoints entered as separate studies all inflate the apparent number of independent units, which is pseudoreplication in a synthesis and distorts both the pooled estimate and τ².

Investigating the causes: subgroup analysis and meta-regression

Subgroup analysis and meta-regression are the two standard tools for explaining heterogeneity by a pre-specified characteristic, and both are observational analyses regardless of how the included studies were designed.

Subgroup analysis splits the studies by a characteristic and tests whether the pooled effects differ. The correct statistic is a formal test for subgroup differences, not a comparison of whether each subgroup’s own confidence interval excludes the null — a subgroup reaching significance while another fails to is not evidence that the two differ, a fallacy that survives in the literature because it is superficially intuitive.

Meta-regression fits study-level covariates against effect estimates, and carries a distinctive hazard: aggregation bias. A relationship between a study’s mean baseline severity and its effect size is a relationship among study averages, and need not hold among individuals within any study. Study-level confounding also has no design protection here — the studies that used one dose also tended to be the newer, larger, better-blinded ones, and no covariate in the model separates those.

The Cochrane Handbook is blunt about the credibility of both: “Reliable conclusions can only be drawn from analyses that are truly pre-specified before inspecting the studies’ results,” and “explorations of heterogeneity that are devised after heterogeneity is identified can at best lead to the generation of hypotheses.” For grading a subgroup claim, the ICEMAN instrument (Schandelmaier et al., CMAJ 2020; 192(32):E901–E906) supplies a structured set of core questions specific to meta-analyses, covering pre-specification, whether the direction was predicted in advance, the test for interaction, and whether the effect modifier was measured within studies or only between them.

Locating which studies drive a high I², as opposed to which characteristic explains it, is a separate question, and three graphical diagnostics answer it without fitting a covariate at all. A Baujat plot places each study on axes of its contribution to Q against its influence on the pooled estimate, so a single outlying trial responsible for most of the heterogeneity is visible at a glance rather than inferred from a subgroup split. A leave-one-out (jackknife) forest plot recomputes the pooled estimate and I² with each study dropped in turn, which reveals whether heterogeneity collapses once one influential trial is removed — a finding that changes the write-up (a fragile pooled result driven by one study) more than it changes the model choice. A GOSH plot fits the model to every possible subset of the studies and shows whether the estimates cluster into two or more distinct groups, which subgroup analysis will only find if the grouping variable happens to be one the authors coded and tested.

Heterogeneity is not the same as small-study effects or publication bias

Small-study effects and publication bias are distinct phenomena from heterogeneity, and merging the three in a discussion section leaves all of them unaddressed.

Small-study effects describe a systematic relationship between a study’s precision and its estimated effect — smaller studies reporting larger effects. Publication bias is one possible cause of that pattern; others include genuinely different populations in early small trials, lower methodological quality, and selective outcome reporting.

The relationship to heterogeneity runs one way. Small-study effects produce statistical heterogeneity, so a high I² is consistent with them. But heterogeneity has many other causes, and funnel plot asymmetry cannot be interpreted as publication bias when substantial heterogeneity is present, because heterogeneity itself generates asymmetry. Cochrane guidance advises against funnel plot asymmetry tests with fewer than ten studies, for the same reason the Q test is weak: too little power to distinguish anything.

Contested points and honest limitations

Several parts of heterogeneity practice are unsettled, and a manuscript that acknowledges the disagreement reads better than one that does not.

The I² thresholds have no empirical basis. The Cochrane bands are explicitly a “rough guide” whose interpretation the Handbook says depends on the magnitude of the effect and the strength of evidence for heterogeneity. Treating 50% as a decision boundary — for pooling, for downgrading, for anything — is a convention, not a result, and the Handbook does not endorse it.

There is no agreed threshold at which pooling becomes inappropriate. Some methodologists hold that if a common estimand cannot be defined, no amount of statistical machinery makes pooling meaningful; others hold that a pooled average with a prediction interval is always more informative than narrative synthesis. Neither position is settled, and the honest manuscript states which one it took and why.

τ² is badly estimated when studies are few, which is the usual case. A synthesis of five studies reports a τ² whose own confidence interval may span an order of magnitude, and every prediction interval derived from it inherits that instability.

I² of 0% does not mean the studies agree. It means Q did not exceed its degrees of freedom, which happens routinely in small syntheses through lack of power, and the resulting random-effects analysis collapses to the fixed-effect one without anyone having decided that it should.

Rare-event syntheses strain every formula on this page. When several included trials report zero events in one or both arms, the continuity correction needed to compute an odds ratio or risk ratio at all also feeds into Q, I² and τ², so different correction choices can move the computed I² noticeably on the same underlying trials. The Cochrane Handbook’s guidance on rare and zero-event outcomes favours exact methods or Peto’s odds ratio, which avoid continuity correction, over trusting a standard I² computed from heavily corrected sparse data.

What reviewers say when heterogeneity is mishandled

Reviewers of syntheses use a recognisable set of phrasings when heterogeneity has been reported without being addressed.

“Substantial heterogeneity (I² = 78%) is reported but not explained; the authors should identify its likely sources rather than defaulting to a random-effects model.” “Pooling does not appear appropriate given the clinical diversity of the included populations.” “The authors report I² without τ² or a prediction interval; the reader cannot judge the magnitude of between-study variation.” “A non-significant Q test is interpreted as absence of heterogeneity, which the test’s power does not support with six studies.” “The subgroup analyses were not pre-specified and no test for interaction is reported.” “The random-effects confidence interval should be presented with the Hartung–Knapp adjustment given the small number of studies.” “Please state which estimator was used for τ².”

Two of those decide papers more often than the rest. The first is the demand for a prediction interval, because supplying one frequently changes the conclusion the abstract can carry — and an abstract stating a benefit the prediction interval does not support is the precise shape of overclaim that editors return. The second is the objection that the pooled estimate has no clear interpretation, which is a request to justify the estimand rather than to add an analysis, and it cannot be answered with more statistics. When a response letter has to concede that pooling was optimistic, saying so directly and adding a stratified presentation reads far better than a defence; the structure for that concession is set out in how to write a response to reviewers.

Common questions about reviewer pushback on heterogeneity

Reviewer 2 says my heterogeneity is too high to pool — how do I respond?

Address the specific objection rather than defending the average: report τ² and a prediction interval, name the estimator used, and identify which study characteristics plausibly explain the variation. If no common estimand can be argued for, concede narrative or stratified reporting directly — that reads far better than defending a pooled figure the data do not support.

Why did a reviewer ask me to name which τ² estimator I used?

Because DerSimonian-Laird, Paule-Mandel and REML can produce meaningfully different τ² estimates from the same data, especially with fewer than ten studies, which changes the width of any prediction interval built on it. Naming the estimator lets a reader judge how much of your conclusion depends on that choice.

Do I need to justify switching from a fixed-effect to a random-effects model after seeing my results?

Yes, explicitly. A protocol specifying fixed-effect and a paper reporting random effects means the model was chosen after heterogeneity was observed, which is sometimes defensible but must be stated as a deviation. Reviewers and editors read an unexplained switch as a sign the model was fitted to the result.

How many studies do I need before I can trust my heterogeneity statistics?

There is no fixed minimum, but below about ten studies both I² and τ² carry wide uncertainty, and Cochran’s Q has too little power to detect real heterogeneity reliably. With five or fewer studies, treat every heterogeneity statistic as a signal worth investigating, not a number to report as settled.

How to report heterogeneity

Report heterogeneity as four elements together. PRISMA 2020 requires that statistical heterogeneity be described and reported, though it names none of the four measures individually.

Give Q with its degrees of freedom and p value, I², τ² with its estimator named, and a 95% prediction interval whenever five or more studies are pooled. PRISMA 2020 item 13d asks authors to “describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used”; item 13e asks for the methods “used to explore possible causes of heterogeneity among study results (e.g. subgroup analysis, meta-regression)”; item 20b asks that results present “the summary estimate and its precision (e.g. confidence/credible interval) and measures of statistical heterogeneity.”

Beyond the checklist, three things separate a strong report. State the estimand — whether the pooled figure is a common effect or the mean of a distribution — in one sentence. Distinguish pre-specified from post hoc investigations explicitly, by name, rather than describing all of them in the same neutral past tense. And write the discussion against the prediction interval rather than the confidence interval, since the prediction interval is what a reader deciding whether the result applies to their own setting actually needs — the same discipline that keeps a claim about an effect’s practical size tied to the numbers supporting it.

Scope note

PerfectPaper reviews original research articles, and a systematic review or meta-analysis manuscript falls outside that scope. Heterogeneity nevertheless appears throughout original research: pooled analyses across cohorts, multi-site randomised trials reporting site-by-treatment interaction, individual participant data analyses, and primary papers that append a meta-analysis of prior work to their own results. In each of those the statistical questions on this page apply unchanged, and they are among the objections raised most often in statistical peer review and in clinical trial reporting.

Related

Confounding · Collider bias · Pseudoreplication · Multiple comparisons objections · Research methods concepts

Review my manuscript