My effect disappeared when I added a covariate

An effect that disappears when you add a covariate has three explanations, and only one of them means the original finding was wrong. The instinct is to conclude it was confounded, and sometimes that is right — but adjusting for a mediator or a collider produces the same symptom, and both mean the adjusted model is the wrong one to report.

You had a clear association. You added an obvious control variable and it vanished. Which of the three you have depends entirely on where the covariate sits causally, and no amount of model fit tells you that. It is a question about the world, answered before you fit anything.

Did the estimate move, or did the interval widen?

“My effect disappeared” describes two different events, and separating them takes one minute and decides which problem you actually have. Compare the crude and adjusted point estimates, not the p-values.

If the adjusted point estimate has moved substantially toward the null, you have a causal-structure question and the three explanations below apply. If the point estimate has barely moved and only the confidence interval has widened across the null, you have a precision problem, and no reclassification of the covariate will fix it.

Three things widen the interval without moving the estimate. The covariate strongly predicts the exposure, so the residual exposure variation that identifies the effect falls. The covariate is categorical with many levels, and each level spends a parameter. Or — most often, and most quietly — the covariate has missing values, so the adjusted model was fitted on fewer rows than the crude one.

Check that last one first, by printing N for both models. A crude model on 1,204 participants and an adjusted model on 1,061 are not two analyses of one cohort; they are two analyses of two cohorts, and the comparison confounds the adjustment with a change in who was analysed. Refit the crude model on exactly the adjusted model’s rows before interpreting the difference between them. The check takes one line of output and it can end the investigation before it starts, because an attenuation produced by differential missingness needs a missing-data strategy — complete-case reporting, multiple imputation, or a stated restriction — rather than a causal argument about the covariate.

A variance inflation factor over the conventional cutoff of 10 is often read as proof that a covariate is absorbing the exposure’s effect, but VIF measures collinearity, not causal role. It cannot distinguish a legitimate confounder that happens to correlate with the exposure from a mediator that is collinear with it precisely because the exposure caused it. Use VIF to explain why an interval widened; never use it to decide whether a covariate belongs in the model.

Questions about diagnosing what happened first

Why did my p-value flip from significant to non-significant after I added one covariate?

A p-value crossing 0.05 happens two ways, and they call for different responses. The point estimate moved substantially toward the null — a causal-structure question, answered by classifying the covariate as a confounder, mediator or collider. Or the estimate barely moved and only the standard error grew, which is usually a precision problem, often driven by missing data on the new covariate.

My confidence interval crossed the null but the point estimate barely moved — is that still confounding?

Usually not. A widened interval with a stable point estimate is typically a precision loss: the covariate predicts your exposure strongly, it has many categories and spends extra degrees of freedom, or — most often — its missing values silently dropped rows from the adjusted model. Refit the crude model on the adjusted model’s exact rows before concluding anything about confounding.

I added a covariate to my logistic regression and now my exposure variable isn’t significant anymore — does that mean my original result was wrong?

Not necessarily. It means the covariate correlates with your exposure and predicts your outcome, which is true of confounders, mediators and colliders alike. Before trusting the adjusted model, work out where the covariate sits causally: only a variable that is a common cause of exposure and outcome, and not itself a downstream consequence of the exposure, belongs in that adjustment set.

The three explanations

Confounding, now controlled. The covariate causes both your exposure and your outcome. Adjusting removes a spurious path and the adjusted estimate is the better one. That conclusion holds only for the confounding this particular covariate carries — if a second, unmeasured variable shares causes with it, the adjusted estimate is closer to the truth than the crude one but is not thereby certified unbiased. “Better” is relative to the crude estimate, not a guarantee of exchangeability. This is the case everyone assumes. It is also the only one of the three in which the crude estimate was the misleading number, which is why the assumption is expensive when it is wrong — see confounding for the identification conditions it requires.

You adjusted for a mediator. The covariate sits on the causal pathway: exposure causes it, and it causes the outcome. Adjusting for it removes exactly the mechanism you were trying to measure. The effect disappears not because it was never there but because you conditioned it away. Schisterman, Cole and Platt gave overadjustment bias its working definition in Epidemiology (2009;20:488–495), defining it as control for “an intermediate variable (or a descending proxy for an intermediate variable) on a causal path from exposure to outcome”.

This is extremely common in health research. Adjusting the relationship between socioeconomic position and cancer survival for stage at diagnosis, when late-stage presentation is how disadvantage produces worse survival, removes most of the effect and leaves an estimate answering a question nobody asked.

You adjusted for a collider. The covariate is caused by both the exposure and the outcome, or by causes of each. Conditioning on it creates an association where none existed, and can flip a real effect’s sign. Selection into your sample is the most insidious version: if enrolment depends on both variables, you have conditioned on a collider without adjusting for anything. The published demonstration is the birth weight paradox: restrict a birth cohort to low-birth-weight infants and maternal smoking appears protective against infant mortality, because birth weight is a collider on the paths from smoking and from unmeasured causes of death (Hernández-Díaz, Schisterman and Hernán, American Journal of Epidemiology 2006;164:1115–1120). The collider does not need to be a genuine causal effect of both exposure and outcome to do this damage — epidemiologists distinguish M-bias, where the shared variable has no causal effect on either one and only traces to unmeasured upstream causes of each, from classical collider-stratification bias. Both close the same kind of path once you condition on the variable, and the same graphical check catches either. See collider bias.

A worked example with the arithmetic

The disadvantage-and-stage example carries numbers well, and the arithmetic below is a constructed illustration rather than a published dataset. Suppose 5-year mortality is 38% in the most-deprived quintile and 26% in the least — a risk difference of 12 percentage points and a risk ratio of 1.46. Suppose 44% of the most-deprived group and 29% of the least present at stage III or IV.

Add stage to the model and the adjusted risk difference falls to 3 percentage points. Nine of the twelve points have gone. Those nine points are not bias that has been removed; they are the part of the deprivation effect that travels through late presentation. The adjusted 3-point estimate is a controlled direct effect — the deprivation gap that would remain if everyone presented at the same stage — and it is a legitimate quantity that answers a different question from the one the abstract asks.

Report the 3-point figure as “the effect of deprivation on survival, adjusted for stage” and you have told the reader that disadvantage barely matters, using a number that was constructed by holding constant the main way disadvantage kills people. Nothing in the output objects. R² rises, the model is better calibrated, the residuals behave, and every fit statistic improves precisely because a strong mediator was added.

The collider version of the same illustration has a sharper signature: the adjusted estimate crosses zero and reverses. A reversal is not proof of collider bias — Simpson’s paradox shows a reversal is a purely numerical fact that can occur under any of the three structures — but it should stop the analysis, because none of the three explanations makes a sign change harmless.

How to tell them apart

Confounding, mediation and collider bias are told apart by the direction of the arrows into the covariate, and by nothing the data can show you. Draw the causal structure before you fit. For each covariate, ask two questions in this order:

Does the exposure cause this variable? If yes, it is a mediator or a descendant of one. Do not adjust for it unless you specifically want the direct effect, and if you do, say so and interpret it as a direct effect rather than as the total effect.

Is this variable caused by both the exposure and the outcome? If yes, it is a collider. Do not adjust.

Only variables that are common causes of exposure and outcome — antecedent to both — belong in a confounding adjustment set.

Temporal order helps and is not sufficient: a variable measured before the outcome can still be a mediator if the exposure preceded it. The recurring failure is measurement date standing in for causal order — body mass index recorded at baseline, medication use recorded at the index visit and pack-years recorded at enrolment all sit downstream of exposures that began years earlier. Ask when the process that set the variable’s value occurred, not when the value was written down. Mediator or confounder? sets out the full decision procedure, including the case where the variable is neither.

Write the structure down as a directed acyclic graph rather than holding it in your head. Textor and colleagues’ R package dagitty and its browser version at dagitty.net enumerate the sufficient adjustment sets automatically once the arrows are drawn (International Journal of Epidemiology 2016;45:1887–1894). When a graph returns more than one sufficient set, agreement between estimates from two disjoint sets is evidence the assumed structure is not badly wrong.

Questions about telling confounders, mediators and colliders apart

How do I know if a variable is a confounder or something I shouldn’t adjust for?

Ask two questions in order. Could the exposure have caused this variable? If yes, it is a mediator, or a descendant of one, and adjusting removes part of the effect you are measuring. Is the variable caused by both exposure and outcome, or by causes of each? If yes, it is a collider and adjusting induces bias. Only variables antecedent to both belong in the model.

Can adding a covariate make my effect reverse sign instead of just shrinking?

Yes. A sign reversal is consistent with confounding, mediation and collider bias alike, so on its own it identifies nothing — treat it as a signal to stop and check the causal structure, not as proof of any one explanation. It is also the textbook signature of conditioning on a collider, which can manufacture an association running opposite to the truth.

If my effect flips direction after I control for a variable, does that always mean it’s a collider?

No. A reversal is a purely arithmetic possibility under Simpson’s paradox and can occur under any of the three structures. Collider conditioning is the most common cause of a genuine reversal, but confirm it from the assumed causal diagram rather than reading the direction of the coefficient as a diagnosis by itself.

Does a variable measured at baseline automatically count as a confounder instead of a mediator?

No — measurement timing is not causal timing. A baseline-measured variable can still sit downstream of an exposure that began years earlier, the way body mass index or medication use at enrolment can reflect a disease process already underway. Ask when the process that set the variable’s value started, not when it was recorded on the case report form.

Overadjustment bias and unnecessary adjustment are different problems

Overadjustment bias and unnecessary adjustment have different consequences and different remedies, and Schisterman, Cole and Platt separated them in Epidemiology (2009;20:488–495). Overadjustment bias is control for an intermediate variable on a causal path from exposure to outcome — a validity failure. Unnecessary adjustment is control for a variable that “does not affect bias of the causal relation between exposure and outcome but may affect its precision” — a penalty, not an error.

The operational difference is the one authors most often miss: overadjustment bias is not a finite-sample bias. It does not shrink as the sample grows, whereas the inefficiency caused by unnecessary adjustment is a function of sample size. A larger cohort makes an unnecessarily adjusted estimate more precise and makes an overadjusted estimate more precisely wrong.

This is why “we had 40,000 participants” is not a response to a mediator objection, and why a tight confidence interval around an attenuated coefficient is not reassurance. It also explains a pattern reviewers see repeatedly in large registry analyses: the adjustment set grew with the dataset, because everything in the extract was available, and the covariates that arrived last are systematically the post-baseline clinical variables most likely to be mediators.

When the estimate changes and nothing is biased

Three mechanisms change an estimate on adjustment without any bias being introduced or removed, and each is regularly misread as confounding.

Non-collapsibility. The odds ratio and the hazard ratio are non-collapsible: the conditional estimate differs from the marginal one even when the covariate is not a confounder at all. Greenland, Pearl and Robins separated confounding from collapsibility in “Confounding and Collapsibility in Causal Inference” (Statistical Science, 1999, volume 14). For a non-null odds ratio, adding a covariate that predicts the outcome moves the conditional estimate further from 1, so non-collapsibility rarely explains a disappearance — but it routinely explains a change in magnitude that authors then attribute to confounding control. Risk differences and risk ratios are collapsible, so if the question is whether adjustment changed anything, a collapsible scale answers it more cleanly. The distortion is largest for common outcomes; when the outcome’s incidence stays under roughly 10%, the odds ratio numerically approximates the risk ratio and non-collapsibility becomes too small to explain a disappearance on its own — a useful check before invoking it as the explanation.

Built-in selection in the hazard ratio. A hazard ratio conditions on survival to each time point, so it acquires a selection structure as follow-up lengthens even in a randomised trial. Hernán set this out in “The hazards of hazard ratios” (Epidemiology 2010;21:13–15). A hazard ratio that attenuates over time is the expected behaviour of the estimand, not evidence that a covariate was doing confounding work.

Bias amplification. Adjusting for a variable that strongly predicts the exposure but has no independent path to the outcome — a prescriber preference, a formulary rule, distance to clinic — leaves unmeasured confounding in place while shrinking the exposure variation that identifies the effect, so the existing bias grows. Myers and colleagues demonstrated this in the American Journal of Epidemiology (2011;174:1213–1222), and their own conclusion is worth quoting against the overcorrection: in most scenarios they simulated, the increases in error from conditioning on an instrument were small compared with total estimation error, so “minimizing unmeasured confounding should be the priority when selecting variables for adjustment, even at the risk of conditioning on IVs.”

A fourth case belongs to pre-post designs. Adjusting a follow-up score for its own baseline value and analysing change scores answer different questions and can disagree in direction — Lord’s paradox, set out in Psychological Bulletin (1967;68:304–305), and closely related to regression to the mean.

How to detect it in your own manuscript

Detect an adjustment problem by auditing the covariate list one variable at a time, because the aggregate model tells you nothing. Four checks find nearly all of it.

Build a sequential adjustment table and find the single variable that did the work. Report crude, then demographics, then clinical, then the full model. Attenuation is usually not spread evenly across ten covariates; it is one variable, and once you know which one, the question collapses to classifying that variable.

Date every covariate against the onset of exposure, not against the outcome. Any covariate whose value could have been altered by the exposure is a mediator candidate, and any covariate that is a proxy for such a variable is a descending proxy, which Schisterman, Cole and Platt treat identically.

Search your own Methods for the four phrases that signal a procedure standing in for a design. “Adjusted for all available covariates.” “Adjusted for potential confounders.” “Variables with p < 0.20 in univariable analysis were entered.” “Variables that changed the estimate by more than 10% were retained.” The last two select on association with the outcome, which is exactly what mediators and colliders have.

Check that the adjustment set was not built by a propensity model on the same bad list. A propensity score is an adjustment set with an extra step, and a score built from post-exposure variables carries the mediator into every matched, weighted or stratified estimate that follows.

Distrust an algorithmically chosen covariate list exactly as much as a manually chosen one. LASSO, elastic net and stepwise selection choose variables by how well they predict the outcome (or, in some pipelines, the exposure); the penalty term carries no information about which direction the arrows point, so it selects mediators and colliders as eagerly as confounders whenever they happen to be the best available predictors.

The change-in-estimate criterion deserves its own note. Mickey and Greenland compared confounder-selection criteria by simulation in the American Journal of Epidemiology (1989;129:125–137), and found the change-in-estimate criterion tended to outperform significance testing at conventional levels. It was never a rule for deciding whether a variable is a confounder, and it cannot be: a mediator and a collider both move the coefficient, which is precisely why the criterion selects them.

Questions about finding the covariate responsible

My effect disappeared in the fully adjusted model — how do I figure out which variable did it?

Build a sequential adjustment table: crude, then demographic covariates, then clinical covariates, then the full model, reporting the point estimate at each step. Attenuation is rarely spread evenly across a long covariate list — usually one variable accounts for nearly all of it, and once you find it the question narrows to classifying that single variable as a confounder, mediator or collider.

Should I keep a covariate just because it changed my estimate by more than 10%?

No. The 10%-change-in-estimate rule is a screening step among variables already justified on causal grounds, not a test of confounder status, and it cannot distinguish a genuine confounder from a mediator or a collider — both of the latter also move the coefficient substantially, which is exactly why a rule based on movement selects them too.

What to report

Report both estimates. The unadjusted and the adjusted, with the adjustment set stated and justified by role rather than by having been available in the dataset. If a covariate is a mediator and you want the mechanism, do a mediation analysis and name it as one rather than presenting the attenuated coefficient as a corrected total effect.

The phrase to avoid is “adjusted for all available covariates”. It describes a procedure, not a design, and it guarantees that any mediators and colliders in the dataset are in the model.

Reporting both estimates is also a checklist requirement, not a preference. STROBE item 16(a) asks authors to “Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (eg, 95% confidence interval). Make clear which confounders were adjusted for and why they were included” — the second sentence is the one manuscripts skip, and “why they were included” means the causal role, not the availability. Item 7 asks you to “Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers”, which requires the classification to have been made. The checklists are at strobe-statement.org.

Label the estimand on every row. A total effect and a controlled direct effect are different quantities, and a table that presents one under the other’s name is wrong even when every number in it is correctly computed.

The related trap in a table

If your table of adjusted coefficients presents every covariate’s estimate as though each were a causal effect of interest, you have the Table 2 fallacy: a single adjustment set cannot be correct for every variable in the model simultaneously. The set chosen for your exposure is generally wrong for the covariates.

Westreich and Greenland named the error in “The table 2 fallacy: presenting and interpreting confounder and modifier coefficients” (American Journal of Epidemiology 2013;177:292–298), and their finding compounds the problem this page describes: a covariate’s coefficient in your model is a controlled direct effect with respect to the exposure, and it “may also be confounded even though the effect estimate for the main exposure is not confounded”. A paper that has already adjusted away its own mechanism and then reads the mediator’s coefficient as that mediator’s effect has made the same error twice. See the Table 2 fallacy.

What reviewers say when it is mishandled

Reviewers rarely write “you adjusted for a mediator”. They write the specific version, and these are the comments that trigger a major revision.

“Stage at diagnosis is plausibly on the causal pathway from the exposure to the outcome; please justify its inclusion or report the total effect.” “Several adjusted covariates were measured after baseline; please restrict the model to pre-exposure variables or reframe the estimand as a direct effect.” “The adjustment set is not derived from a stated causal structure; please provide a directed acyclic graph.” “The unadjusted estimate is not reported.” “The adjusted and unadjusted models appear to be fitted on different numbers of participants.” “The authors interpret the attenuation as evidence of confounding; an alternative explanation is that the covariate mediates the association, and the manuscript does not distinguish them.” “The abstract reports the fully adjusted estimate as the effect of the exposure; the discussion interprets it as a total effect.”

The last is the one that decides papers, because it is a contradiction inside the manuscript rather than a dispute about method. If the objection is that the language has outrun the design rather than that the model is wrong, see correlative, not causal.

What to do when you cannot simply drop the covariate

Dropping the covariate is often unavailable, and each version of that has a different answer.

A reviewer demanded the adjustment. Report both estimates, state the assumed causal role of the variable, and give the graph. A reviewer who asked for stage adjustment usually wants reassurance about confounding, not a direct effect, and will accept the total effect once the mediating role is made explicit — see how to write a response to reviewers. Do not silently comply and leave the interpretation unchanged.

You genuinely want the direct effect. Run a formal mediation analysis rather than reading a coefficient. Report the natural direct and natural indirect effects with confidence intervals, name the software and version, and state the identification conditions — which include no unmeasured mediator–outcome confounding, an assumption the attenuated coefficient makes silently. Bound it: mediational E-values (Smith and VanderWeele, Epidemiology 2019;30:835–837) give the minimum strength of unmeasured mediator–outcome confounding needed to explain away a direct or indirect effect, and one number outperforms a paragraph of acknowledgement in a limitations section. That no-unmeasured-mediator-outcome-confounding assumption fails whenever a variable exists that is itself affected by the exposure and also affects both the mediator and the outcome — a common situation in longitudinal cohorts, where a post-baseline clinical event does exactly this. When it fails, the natural direct and indirect effects are not identified by any amount of data, and the estimable substitute is an interventional effect built by randomly assigning mediator values from their exposure-specific distribution rather than fixing them — a related but distinct quantity that should be labelled as such rather than reported under the natural-effects name.

The collider is your sample, not a column. Selection cannot be adjusted away, because the selection indicator has the same value for everyone in the dataset and there is no variation left to model. Inverse probability of selection weighting works when the drivers of selection were measured, and bounds work when they were not — see selection bias.

The graph admits no sufficient adjustment set. That is a design finding, and it points toward a negative-control outcome, an instrumental variable, or a different data source. It does not point toward a longer covariate list. Say it in the limitations rather than fitting the model anyway and hoping.

Frequently asked questions about reporting and reviewers

Which estimate should I put in my abstract, the crude one or the adjusted one?

Report both in the results, with the adjustment set stated and its causal role named. If the covariate is a confounder, lead with the adjusted figure as the total effect. If it is a mediator, the crude figure is the total effect and the adjusted figure is a controlled direct effect that needs its own label, not the abstract’s headline number.

A reviewer wants me to adjust for a variable I suspect is a mediator — what do I say back?

State the variable’s assumed causal role, show both the crude and adjusted estimates, and provide the directed acyclic graph the adjustment set was built from. Most reviewers asking for the adjustment want reassurance about confounding rather than a direct effect, and will accept the unadjusted total effect once the mediating pathway is made explicit instead of silently complying and leaving the interpretation unchanged.

Is writing “adjusted for all available covariates” in my methods section a red flag to reviewers?

Increasingly, yes. It describes a procedure rather than a causal design, and it guarantees that any mediators or colliders present in the dataset entered the model alongside genuine confounders. State instead which variables were included and why, by causal role, so a reviewer can audit the adjustment set rather than take the model’s fit on faith.

Catching it before a reviewer does

Causal language discipline checks how the adjustment set was chosen and flags variables that are plausibly mediators or colliders, naming each and the likely consequence. For disparities work specifically, health equity reporting reports where an access or socioeconomic variable on the pathway from structural disadvantage to outcome has been adjusted away as though it were a confounder — the single most common version of this error in that literature.

PerfectPaper reads the crude and adjusted estimates, the covariate list and the timing of each covariate together, and reports where an attenuation is better explained by a mediator on the pathway than by confounding that has been controlled.

If a reviewer has already raised it, see reviewer questions my choice of statistical test for the adjacent objection.

Related

Confounding · Mediator or confounder? · Collider bias · Directed acyclic graphs · STROBE checklist · Registry data limitations · Statistics objections · Epidemiology review

Review my manuscript