My effect disappeared when I added a covariate

An effect that disappears when you add a covariate has three explanations, and only one of them means the original finding was wrong. The instinct is to conclude it was confounded, and sometimes that is right — but adjusting for a mediator or a collider produces the same symptom, and both mean the adjusted model is the wrong one to report.

You had a clear association. You added an obvious control variable and it vanished. Which of the three you have depends entirely on where the covariate sits causally, and no amount of model fit tells you that. It is a question about the world, answered before you fit anything.

Did the estimate move, or did the interval widen?

“My effect disappeared” describes two different events, and separating them takes one minute and decides which problem you actually have. Compare the crude and adjusted point estimates, not the p-values.

If the adjusted point estimate has moved substantially toward the null, you have a causal-structure question and the three explanations below apply. If the point estimate has barely moved and only the confidence interval has widened across the null, you have a precision problem, and no reclassification of the covariate will fix it.

Three things widen the interval without moving the estimate. The covariate strongly predicts the exposure, so the residual exposure variation that identifies the effect falls. The covariate is categorical with many levels, and each level spends a parameter. Or — most often, and most quietly — the covariate has missing values, so the adjusted model was fitted on fewer rows than the crude one.

Check that last one first, by printing N for both models. A crude model on 1,204 participants and an adjusted model on 1,061 are not two analyses of one cohort; they are two analyses of two cohorts, and the comparison confounds the adjustment with a change in who was analysed. Refit the crude model on exactly the adjusted model’s rows before interpreting the difference between them. The check takes one line of output and it can end the investigation before it starts, because an attenuation produced by differential missingness needs a missing-data strategy — complete-case reporting, multiple imputation, or a stated restriction — rather than a causal argument about the covariate.

The three explanations

Confounding, now controlled. The covariate causes both your exposure and your outcome. Adjusting removes a spurious path and the adjusted estimate is the better one. This is the case everyone assumes. It is also the only one of the three in which the crude estimate was the misleading number, which is why the assumption is expensive when it is wrong — see confounding for the identification conditions it requires.

You adjusted for a mediator. The covariate sits on the causal pathway: exposure causes it, and it causes the outcome. Adjusting for it removes exactly the mechanism you were trying to measure. The effect disappears not because it was never there but because you conditioned it away. Schisterman, Cole and Platt gave overadjustment bias its working definition in Epidemiology (2009;20:488–495), defining it as control for “an intermediate variable (or a descending proxy for an intermediate variable) on a causal path from exposure to outcome”.

This is extremely common in health research. Adjusting the relationship between socioeconomic position and cancer survival for stage at diagnosis, when late-stage presentation is how disadvantage produces worse survival, removes most of the effect and leaves an estimate answering a question nobody asked.

You adjusted for a collider. The covariate is caused by both the exposure and the outcome, or by causes of each. Conditioning on it creates an association where none existed, and can flip a real effect’s sign. Selection into your sample is the most insidious version: if enrolment depends on both variables, you have conditioned on a collider without adjusting for anything. The published demonstration is the birth weight paradox: restrict a birth cohort to low-birth-weight infants and maternal smoking appears protective against infant mortality, because birth weight is a collider on the paths from smoking and from unmeasured causes of death (Hernández-Díaz, Schisterman and Hernán, American Journal of Epidemiology 2006;164:1115–1120). See collider bias.

A worked example with the arithmetic

The disadvantage-and-stage example carries numbers well, and the arithmetic below is a constructed illustration rather than a published dataset. Suppose 5-year mortality is 38% in the most-deprived quintile and 26% in the least — a risk difference of 12 percentage points and a risk ratio of 1.46. Suppose 44% of the most-deprived group and 29% of the least present at stage III or IV.

Add stage to the model and the adjusted risk difference falls to 3 percentage points. Nine of the twelve points have gone. Those nine points are not bias that has been removed; they are the part of the deprivation effect that travels through late presentation. The adjusted 3-point estimate is a controlled direct effect — the deprivation gap that would remain if everyone presented at the same stage — and it is a legitimate quantity that answers a different question from the one the abstract asks.

Report the 3-point figure as “the effect of deprivation on survival, adjusted for stage” and you have told the reader that disadvantage barely matters, using a number that was constructed by holding constant the main way disadvantage kills people. Nothing in the output objects. R² rises, the model is better calibrated, the residuals behave, and every fit statistic improves precisely because a strong mediator was added.

The collider version of the same illustration has a sharper signature: the adjusted estimate crosses zero and reverses. A reversal is not proof of collider bias — Simpson’s paradox shows a reversal is a purely numerical fact that can occur under any of the three structures — but it should stop the analysis, because none of the three explanations makes a sign change harmless.

How to tell them apart

Confounding, mediation and collider bias are told apart by the direction of the arrows into the covariate, and by nothing the data can show you. Draw the causal structure before you fit. For each covariate, ask two questions in this order:

Does the exposure cause this variable? If yes, it is a mediator or a descendant of one. Do not adjust for it unless you specifically want the direct effect, and if you do, say so and interpret it as a direct effect rather than as the total effect.

Is this variable caused by both the exposure and the outcome? If yes, it is a collider. Do not adjust.

Only variables that are common causes of exposure and outcome — antecedent to both — belong in a confounding adjustment set.

Temporal order helps and is not sufficient: a variable measured before the outcome can still be a mediator if the exposure preceded it. The recurring failure is measurement date standing in for causal order — body mass index recorded at baseline, medication use recorded at the index visit and pack-years recorded at enrolment all sit downstream of exposures that began years earlier. Ask when the process that set the variable’s value occurred, not when the value was written down. Mediator or confounder? sets out the full decision procedure, including the case where the variable is neither.

Write the structure down as a directed acyclic graph rather than holding it in your head. Textor and colleagues’ R package dagitty and its browser version at dagitty.net enumerate the sufficient adjustment sets automatically once the arrows are drawn (International Journal of Epidemiology 2016;45:1887–1894). When a graph returns more than one sufficient set, agreement between estimates from two disjoint sets is evidence the assumed structure is not badly wrong.

Overadjustment bias and unnecessary adjustment are different problems

Overadjustment bias and unnecessary adjustment have different consequences and different remedies, and Schisterman, Cole and Platt separated them in Epidemiology (2009;20:488–495). Overadjustment bias is control for an intermediate variable on a causal path from exposure to outcome — a validity failure. Unnecessary adjustment is control for a variable that “does not affect bias of the causal relation between exposure and outcome but may affect its precision” — a penalty, not an error.

The operational difference is the one authors most often miss: overadjustment bias is not a finite-sample bias. It does not shrink as the sample grows, whereas the inefficiency caused by unnecessary adjustment is a function of sample size. A larger cohort makes an unnecessarily adjusted estimate more precise and makes an overadjusted estimate more precisely wrong.

This is why “we had 40,000 participants” is not a response to a mediator objection, and why a tight confidence interval around an attenuated coefficient is not reassurance. It also explains a pattern reviewers see repeatedly in large registry analyses: the adjustment set grew with the dataset, because everything in the extract was available, and the covariates that arrived last are systematically the post-baseline clinical variables most likely to be mediators.

When the estimate changes and nothing is biased

Three mechanisms change an estimate on adjustment without any bias being introduced or removed, and each is regularly misread as confounding.

Non-collapsibility. The odds ratio and the hazard ratio are non-collapsible: the conditional estimate differs from the marginal one even when the covariate is not a confounder at all. Greenland, Pearl and Robins separated confounding from collapsibility in “Confounding and Collapsibility in Causal Inference” (Statistical Science, 1999, volume 14). For a non-null odds ratio, adding a covariate that predicts the outcome moves the conditional estimate further from 1, so non-collapsibility rarely explains a disappearance — but it routinely explains a change in magnitude that authors then attribute to confounding control. Risk differences and risk ratios are collapsible, so if the question is whether adjustment changed anything, a collapsible scale answers it more cleanly.

Built-in selection in the hazard ratio. A hazard ratio conditions on survival to each time point, so it acquires a selection structure as follow-up lengthens even in a randomised trial. Hernán set this out in “The hazards of hazard ratios” (Epidemiology 2010;21:13–15). A hazard ratio that attenuates over time is the expected behaviour of the estimand, not evidence that a covariate was doing confounding work.

Bias amplification. Adjusting for a variable that strongly predicts the exposure but has no independent path to the outcome — a prescriber preference, a formulary rule, distance to clinic — leaves unmeasured confounding in place while shrinking the exposure variation that identifies the effect, so the existing bias grows. Myers and colleagues demonstrated this in the American Journal of Epidemiology (2011;174:1213–1222), and their own conclusion is worth quoting against the overcorrection: in most scenarios they simulated, the increases in error from conditioning on an instrument were small compared with total estimation error, so “minimizing unmeasured confounding should be the priority when selecting variables for adjustment, even at the risk of conditioning on IVs.”

A fourth case belongs to pre-post designs. Adjusting a follow-up score for its own baseline value and analysing change scores answer different questions and can disagree in direction — Lord’s paradox, set out in Psychological Bulletin (1967;68:304–305), and closely related to regression to the mean.

How to detect it in your own manuscript

Detect an adjustment problem by auditing the covariate list one variable at a time, because the aggregate model tells you nothing. Four checks find nearly all of it.

Build a sequential adjustment table and find the single variable that did the work. Report crude, then demographics, then clinical, then the full model. Attenuation is usually not spread evenly across ten covariates; it is one variable, and once you know which one, the question collapses to classifying that variable.

Date every covariate against the onset of exposure, not against the outcome. Any covariate whose value could have been altered by the exposure is a mediator candidate, and any covariate that is a proxy for such a variable is a descending proxy, which Schisterman, Cole and Platt treat identically.

Search your own Methods for the four phrases that signal a procedure standing in for a design. “Adjusted for all available covariates.” “Adjusted for potential confounders.” “Variables with p < 0.20 in univariable analysis were entered.” “Variables that changed the estimate by more than 10% were retained.” The last two select on association with the outcome, which is exactly what mediators and colliders have.

Check that the adjustment set was not built by a propensity model on the same bad list. A propensity score is an adjustment set with an extra step, and a score built from post-exposure variables carries the mediator into every matched, weighted or stratified estimate that follows.

The change-in-estimate criterion deserves its own note. Mickey and Greenland compared confounder-selection criteria by simulation in the American Journal of Epidemiology (1989;129:125–137), and found the change-in-estimate criterion tended to outperform significance testing at conventional levels. It was never a rule for deciding whether a variable is a confounder, and it cannot be: a mediator and a collider both move the coefficient, which is precisely why the criterion selects them.

What to report

Report both estimates. The unadjusted and the adjusted, with the adjustment set stated and justified by role rather than by having been available in the dataset. If a covariate is a mediator and you want the mechanism, do a mediation analysis and name it as one rather than presenting the attenuated coefficient as a corrected total effect.

The phrase to avoid is “adjusted for all available covariates”. It describes a procedure, not a design, and it guarantees that any mediators and colliders in the dataset are in the model.

Reporting both estimates is also a checklist requirement, not a preference. STROBE item 16(a) asks authors to “Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (eg, 95% confidence interval). Make clear which confounders were adjusted for and why they were included” — the second sentence is the one manuscripts skip, and “why they were included” means the causal role, not the availability. Item 7 asks you to “Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers”, which requires the classification to have been made. The checklists are at strobe-statement.org.

Label the estimand on every row. A total effect and a controlled direct effect are different quantities, and a table that presents one under the other’s name is wrong even when every number in it is correctly computed.

The related trap in a table

If your table of adjusted coefficients presents every covariate’s estimate as though each were a causal effect of interest, you have the Table 2 fallacy: a single adjustment set cannot be correct for every variable in the model simultaneously. The set chosen for your exposure is generally wrong for the covariates.

Westreich and Greenland named the error in “The table 2 fallacy: presenting and interpreting confounder and modifier coefficients” (American Journal of Epidemiology 2013;177:292–298), and their finding compounds the problem this page describes: a covariate’s coefficient in your model is a controlled direct effect with respect to the exposure, and it “may also be confounded even though the effect estimate for the main exposure is not confounded”. A paper that has already adjusted away its own mechanism and then reads the mediator’s coefficient as that mediator’s effect has made the same error twice. See the Table 2 fallacy.

What reviewers say when it is mishandled

Reviewers rarely write “you adjusted for a mediator”. They write the specific version, and these are the comments that trigger a major revision.

“Stage at diagnosis is plausibly on the causal pathway from the exposure to the outcome; please justify its inclusion or report the total effect.” “Several adjusted covariates were measured after baseline; please restrict the model to pre-exposure variables or reframe the estimand as a direct effect.” “The adjustment set is not derived from a stated causal structure; please provide a directed acyclic graph.” “The unadjusted estimate is not reported.” “The adjusted and unadjusted models appear to be fitted on different numbers of participants.” “The authors interpret the attenuation as evidence of confounding; an alternative explanation is that the covariate mediates the association, and the manuscript does not distinguish them.” “The abstract reports the fully adjusted estimate as the effect of the exposure; the discussion interprets it as a total effect.”

The last is the one that decides papers, because it is a contradiction inside the manuscript rather than a dispute about method. If the objection is that the language has outrun the design rather than that the model is wrong, see correlative, not causal.

What to do when you cannot simply drop the covariate

Dropping the covariate is often unavailable, and each version of that has a different answer.

A reviewer demanded the adjustment. Report both estimates, state the assumed causal role of the variable, and give the graph. A reviewer who asked for stage adjustment usually wants reassurance about confounding, not a direct effect, and will accept the total effect once the mediating role is made explicit — see how to write a response to reviewers. Do not silently comply and leave the interpretation unchanged.

You genuinely want the direct effect. Run a formal mediation analysis rather than reading a coefficient. Report the natural direct and natural indirect effects with confidence intervals, name the software and version, and state the identification conditions — which include no unmeasured mediator–outcome confounding, an assumption the attenuated coefficient makes silently. Bound it: mediational E-values (Smith and VanderWeele, Epidemiology 2019;30:835–837) give the minimum strength of unmeasured mediator–outcome confounding needed to explain away a direct or indirect effect, and one number outperforms a paragraph of acknowledgement in a limitations section.

The collider is your sample, not a column. Selection cannot be adjusted away, because the selection indicator has the same value for everyone in the dataset and there is no variation left to model. Inverse probability of selection weighting works when the drivers of selection were measured, and bounds work when they were not — see selection bias.

The graph admits no sufficient adjustment set. That is a design finding, and it points toward a negative-control outcome, an instrumental variable, or a different data source. It does not point toward a longer covariate list. Say it in the limitations rather than fitting the model anyway and hoping.

Catching it before a reviewer does

Causal language discipline checks how the adjustment set was chosen and flags variables that are plausibly mediators or colliders, naming each and the likely consequence. For disparities work specifically, health equity reporting reports where an access or socioeconomic variable on the pathway from structural disadvantage to outcome has been adjusted away as though it were a confounder — the single most common version of this error in that literature.

PerfectPaper reads the crude and adjusted estimates, the covariate list and the timing of each covariate together, and reports where an attenuation is better explained by a mediator on the pathway than by confounding that has been controlled.

If a reviewer has already raised it, see reviewer questions my choice of statistical test for the adjacent objection.

Related

Confounding · Mediator or confounder? · Collider bias · Directed acyclic graphs · STROBE checklist · Registry data limitations · Statistics objections · Epidemiology review

Review my manuscript

Frequently asked questions

Why did my effect disappear when I added a covariate?

Three structures produce that symptom. The covariate was a confounder and the adjusted estimate is the better one; the covariate was a mediator and adjusting deleted the mechanism you were measuring; or the covariate was a collider and adjusting induced a spurious association that cancelled a real one. Which applies depends on the causal position of that variable, not on model fit.

Does an effect vanishing after adjustment mean it was confounded?

No. Attenuation on adjustment is equally consistent with having adjusted for a mediator, in which case the crude estimate was the correct total effect and the adjusted one is a controlled direct effect. First confirm the point estimate actually moved rather than the confidence interval widening, and that both models were fitted on the same rows.

My effect disappeared in the fully adjusted model — how do I find which covariate did it?

Build a sequential adjustment table: crude, then demographics, then clinical variables, then the full model. Attenuation is rarely spread evenly across a long covariate list; usually one variable does nearly all of it. Once that variable is identified, the question narrows to classifying it as a confounder, a mediator or a collider.

My effect reversed sign after I added a covariate — what happened?

A sign reversal is consistent with all three structures, so it is a stop signal rather than a diagnosis. It is the classic signature of conditioning on a collider, which can manufacture an association running opposite to the true one, but a reversal is also a purely arithmetic possibility under Simpson’s paradox. Resolve it from the assumed causal graph, not from the output.

My effect disappeared after adjustment — which estimate should I report?

Report both, and label each estimand. If the covariate is a confounder, the adjusted figure is the total effect. If it is a mediator, the crude figure is the total effect and the adjusted one is a controlled direct effect answering a different question. Presenting an attenuated, mediator-adjusted coefficient as the effect of the exposure is the error that draws a major revision.

Can an effect shrink on adjustment without anything being biased?

Yes, in three ways. Odds ratios and hazard ratios are non-collapsible, so the conditional estimate differs from the marginal one even with no confounding. A hazard ratio conditions on survival and attenuates with follow-up by construction. And adjusting for a strong predictor of the exposure with no path to the outcome can amplify existing unmeasured confounding rather than remove any.

My effect disappeared — should I keep the covariate because it changed the estimate by more than 10%?

No. The change-in-estimate criterion is a screening device among variables already justified on causal grounds, not a test of whether a variable is a confounder. Mediators and colliders both move coefficients substantially, which is exactly why a rule that selects on movement selects them.