What is selection bias?

Selection bias is a distortion of an association that arises from how people entered a study or stayed in it, rather than from anything measured within it. When inclusion, participation, survival or retention depends on both the exposure and the outcome, the association in the analysed sample differs systematically from the association in the population the study was meant to describe.

Selection bias is the bias that adjustment does not fix and that sample size does not fix. The 1936 Literary Digest poll returned some 2.4 million ballots from around 10 million mailed and called the election for Landon; Roosevelt won it in a landslide. A sample thousands of times larger than a modern opinion poll named the wrong winner because of who answered, not how many.

The structural definition

Selection bias has a single causal structure, set out by Hernán, Hernández-Díaz and Robins in “A structural approach to selection bias” (Epidemiology 2004;15:615–625): the study conditions on a common effect of two variables, one of which is the exposure or a cause of the exposure, and the other of which is the outcome or a cause of the outcome.

That variable is selection itself — a node, usually written S, that takes the value 1 for the people who ended up in the analysis. Restricting the analysis to S = 1 is a conditioning operation, exactly like putting a covariate in a regression model, and it has the same consequence when S is a collider. Selection bias is therefore a form of collider bias in which the collider is the act of being studied.

The practical value of this definition is that it unifies things taught as separate named biases — Berkson’s bias, the healthy worker effect, differential loss to follow-up, non-response, survivor bias — under one question you can ask about any study: does membership of the analysed sample depend on both the exposure side and the outcome side? If the answer is yes, the bias is present whatever covariates were chosen, and it was present before the first model was fitted.

Selection bias versus confounding

Selection bias and confounding are distinguished by the direction of the arrows, and the distinction determines whether adjustment helps. Confounding arises from a common cause of exposure and outcome that exists in the population before the study touches it. Selection bias arises from a common effect of exposure and outcome that the study conditions on by choosing whom to analyse.

The operational consequence is stark. A confounder that you measured can be adjusted away by regression, stratification, matching or weighting. A selection effect cannot be adjusted away by adding covariates, because the covariate that would fix it — S — has the same value, 1, for every person in the dataset. There is no variation left to model.

A study can carry both, one, or neither. A randomised trial removes confounding by every variable, measured and unmeasured, and removes none of the selection bias introduced afterwards by differential dropout. This is why “we adjusted for potential confounders” is not a response to a selection concern, and why a reviewer who raises selection will not be satisfied by a longer covariate list.

The four gates where selection acts

Selection acts at four distinct gates, and a manuscript can pass three of them and fail the fourth.

Recruitment and eligibility. The sampling frame, the inclusion criteria and the recruitment channel decide who could enter. Hospital-based recruitment, clinic convenience samples, volunteer registries and online panels each impose a filter correlated with health, income and health-seeking behaviour.

Participation and response. Among those invited, who agrees. Non-response is only a threat when the reasons for not responding relate to both the exposure and the outcome, which is unknowable from the responders alone.

Survival to measurement. Among those eligible, who is still alive, still enrolled, or still detectable when measurement happens. Studies of prevalent cases select on survival with the disease, which is a different thing from developing it.

Retention and analysis. Among those enrolled, who remains and who has complete data. Complete-case analysis conditions on having complete data, and missingness that depends on both exposure and outcome makes that a collider. Studies that exclude participants after randomisation on the basis of adherence, dose received or protocol deviation are conditioning on post-baseline behaviour.

The named forms

Selection bias has accumulated named variants, each of which is the same structure seen in a different setting.

Berkson’s bias. Joseph Berkson showed in Biometrics Bulletin (1946;2:47–53) that two conditions with no real relationship become associated among hospital inpatients, because either condition raises the chance of admission. His worked example, on cholecystitis and diabetes, implied when read naively that one disease protected against the other — an artefact of admission, not a finding. Any study whose sample is defined by treatment-seeking carries this risk.

The healthy worker effect. McMichael’s paper in the Journal of Occupational Medicine (1976;18:165–168) is the standard reference for the consistent tendency of the actively employed to have more favourable mortality than the general population, which makes standardised mortality ratios computed against a general-population reference understate occupational risk. Employment is a selection gate that both illness and exposure influence.

Prevalence–incidence, or Neyman’s, bias. Neyman noted in Science (1955;122:401–406) that a case-control study assembled from prevalent cases can invert an exposure’s apparent effect when the exposure alters survival with the disease, because the exposure changes who is still available to be sampled as a case.

Index event and survivor bias. Studies that begin at a disease event — first myocardial infarction, first hospitalisation, entry onto a transplant list — select on having survived to that event. Several paradoxical findings, including the obesity paradox, have been attributed to this structure rather than to biology.

Screening-related selection. Lead time bias and length time bias both operate through who is detected by a screening programme and when, and are not addressed by adjusting for tumour stage.

Time-based selection. Immortal time bias arises from how follow-up time is allocated relative to exposure definition, and is frequently created by an eligibility criterion that requires surviving long enough to receive treatment.

Worked example: control selection in a case-control study

Consider a case-control study of whether a medication increases the risk of a gastrointestinal bleed, with cases recruited from patients admitted for bleeding at a tertiary hospital.

If the controls are other inpatients at the same hospital, they were admitted for reasons of their own — and many of the common reasons for admission are associated with taking the same medication. Control prevalence of the exposure is then higher than in the population that produced the cases, and the odds ratio is pulled toward the null or below it. Nothing in the analysis reveals this; the odds ratio has an ordinary confidence interval and ordinary model diagnostics.

The correct comparison follows the source population principle: controls must be sampled from the population that would have become cases in this study had they developed the outcome. If the cases are everyone in a defined catchment admitted for bleeding, the controls belong in that catchment, not on the ward next door.

The same failure appears in registry and electronic health record analyses, where the comparator is often “everyone else in the database” — a group defined by having generated healthcare encounters, which is itself a function of illness and of the exposure.

When selection does not bias the estimate

Selection restricts a study without biasing its estimate whenever selection is not a common effect of both the exposure side and the outcome side, and this exception is larger than most manuscripts assume.

Selecting on the exposure alone is ordinary study design. A cohort restricted to a single occupational group, a trial that enrols only patients offered a particular therapy, and a study of one hospital’s patients all select on exposure or its causes; the within-sample effect estimate remains valid for those people, and what is at stake is transportability rather than internal validity. Selecting on the outcome alone is what a case-control study does by construction, and the odds ratio is unbiased provided selection is not additionally related to exposure.

The structure that bites is selection on both. Restricting to hospitalised patients when both the exposure and the outcome influence hospitalisation, requiring survival to a second visit when both influence survival, and complete-case analysis when both influence missingness are the three most common instances in submitted work.

There is also a genuinely contested case. Selection that depends on a covariate related to both exposure and outcome — recruitment skewed by education, for example — does not automatically bias a conditional estimate that adjusts for that covariate, but it does bias marginal estimates and any prevalence figure. Papers routinely treat this as an all-or-nothing property of the cohort when it is a property of each specific estimate reported.

The generalisability question is not the same question

Generalisability failure and selection bias are frequently conflated in manuscripts, and separating them changes what a reviewer expects you to fix.

UK Biobank is the clearest published case. Fry and colleagues reported in the American Journal of Epidemiology (2017;186:1026–1034) that of approximately 9.2 million people invited, 5.5% participated; participants were less deprived, less likely to be obese, smoke or drink daily, and had, at ages 70–74, all-cause mortality 46.2% lower in men and 55.5% lower in women, and total cancer incidence 11.8% and 18.1% lower respectively, than the general population of the same age. Those figures are unusable as population estimates.

Batty and colleagues then compared risk-factor–mortality associations in UK Biobank against a pooled set of 18 general-population cohort studies recruited at conventional response rates (BMJ 2020;368:m131) and found the associations directionally consistent, with some heterogeneity of magnitude. The honest summary is therefore two-part: a 5.5% response rate makes descriptive estimates invalid for the population, and does not, on the available evidence, invalidate the exposure–outcome associations studied. Do not use that finding as a general licence — it is empirical, specific to those risk factors and endpoints, and says nothing about an exposure that itself affects participation.

How selection bias is detected in a manuscript

Selection bias is detected in a manuscript by reconciling numbers and by reading the eligibility criteria for post-baseline information, not by any statistical test.

Make the denominators reconcile. Abstract N, Table 1 column totals, the primary model’s analysed N and the flow diagram frequently disagree. A paper reporting 1,204 enrolled, 1,180 in Table 1 and 1,061 in the adjusted model has 143 people whose exit is unexplained; STROBE item 13(a) asks for numbers at each stage — potentially eligible, examined for eligibility, confirmed eligible, included, completing follow-up and analysed — and item 13(b) asks for reasons for non-participation at each stage.

Read eligibility for future information. Criteria such as “patients who completed at least three cycles”, “participants with at least 24 months of follow-up” or “those who returned for the second assessment” require the participant to have survived and remained, which is outcome information used to define entry.

Check where the comparator came from. Cases and controls, or exposed and unexposed, drawn through different recruitment channels is the single most common structural defect in submitted case-control work.

Look for a response rate with its denominator. “Response rate 72%” with no statement of what sat in the denominator is not a reported response rate; AAPOR standard definitions exist precisely because one study can report several very different figures.

Check whether attrition is reported by exposure group. Overall loss to follow-up is nearly uninformative on its own. The five-and-twenty heuristic from Schulz and Grimes (Lancet 2002;359:781–785) — under 5% loss unlikely to bias, 20% or more a serious concern — is a rule of thumb rather than a threshold, and differential loss of 8% versus 3% between arms can matter more than 25% loss that falls equally.

Check the direction of any surprising result. An implausibly protective association in a group selected on illness is the classic signature, as is a result that contradicts randomised evidence. These signals have innocent explanations too; they warrant re-examining the selection structure rather than trusting a number because it is adjusted. See also my effect disappeared after adjusting.

What reviewers say when it is mishandled

Reviewers rarely write the words “selection bias” first. They write the specific version, and these comments decide papers.

“The analysed sample is not the sample described in the eligibility criteria; please account for all screened individuals in a flow diagram.” “Controls do not appear to be drawn from the source population that gave rise to the cases.” “Inclusion required survival to the second assessment, which is itself influenced by the outcome.” “Loss to follow-up is 27% and is not compared across exposure groups; please report baseline characteristics of completers and non-completers.” “The response rate is reported without its denominator.” “The findings apply to volunteers who attended an assessment centre; the abstract generalises them to the general population.” “The authors describe the cohort as representative without naming the target population.” “Selection bias is acknowledged in the limitations, but its likely direction is not stated.”

The last of those is worth anticipating, because it is the comment most often triggered by a limitations paragraph written to pre-empt it. A generic acknowledgement reads as an admission without an analysis.

What to do about it

Selection bias is addressed at design, and bounded at analysis when design has already failed.

Define the target population first, then sample from it. Write down the population the conclusion is meant to describe before recruitment begins, and state the sampling frame that stands in for it. Most selection problems are decisions made at this stage and discovered at review.

Apply eligibility using information available at baseline only. A criterion that uses post-baseline information is a selection gate. Where a treatment must be received to define exposure, use a design that handles it — a landmark analysis with a pre-specified landmark, or a time-varying exposure model.

Weight for selection when the mechanism is measured. Inverse probability of selection weighting reconstructs the source population when the variables driving selection are recorded, and it is the standard remedy for measured non-response and measured differential attrition. It cannot address selection driven by unrecorded factors, which is most of it.

Bound what you cannot remove. The selection bounds of Smith and VanderWeele (Epidemiology 2019;30:509–516) give the minimum strength of selection required to explain away an observed risk ratio, using parameters describing the relationship between the unmeasured selection factor and the measured variables, with no functional-form assumption about those factors. Implementations exist in the R package EValue. A bound makes a far stronger limitations paragraph than an acknowledgement.

Show sensitivity. Report the estimate under alternative handling of missing data and under plausible alternative selection assumptions, so readers can see how much the conclusion depends on the analysed sample. Where the outcome is time-to-event and censoring may depend on exposure, treat that as a competing risks and censoring question rather than an administrative detail.

How to report it

Report selection bias by naming the gate, the direction and the magnitude, in that order.

State the sampling frame and how it relates to the target population. Report participant flow with reasons at every stage, including a flow diagram. Report the response rate with its denominator and its definition. Compare those analysed with those not analysed on baseline characteristics, by exposure group. Name the direction the bias would push the estimate and say why, rather than writing that results “should be interpreted with caution”. Where a bound or a weighted analysis is feasible, give the number.

Keep the reported estimand consistent with the selected sample. An estimate conditional on being in a selected cohort is not the population estimate, and presenting adjusted coefficients for selection-related covariates as though they were causal effects compounds the problem — see the Table 2 fallacy. Where selection differs systematically across groups, say so explicitly rather than aggregating, which is also the substance of health equity reporting.

Related

Collider bias · Confounding · Immortal time bias · Lead time bias · Epidemiology review

PerfectPaper reads the eligibility criteria, the participant flow and the analysed denominators together, and reports where the analysed sample stops matching the population the conclusion describes. Checked before submission by causal language discipline, which flags eligibility criteria that use post-baseline information and denominators that do not reconcile.

Review my manuscript

Frequently asked questions

What does selection bias mean in research?

Selection bias means the people analysed differ systematically from the population the study intends to describe, in a way related to both the exposure and the outcome. It originates in recruitment, participation, survival or retention rather than in any measured variable, so covariate adjustment does not remove it.

What is a simple definition of selection bias?

Selection bias is error introduced by who ends up in the analysis. When being included depends on both the exposure and the outcome, the association observed among those included differs from the true association, and that difference persists however carefully the analysis is adjusted.

Can you give an example of selection bias?

Berkson’s 1946 hospital example is the standard one: two unrelated conditions appear associated among inpatients because either condition raises the chance of admission. Read naively, his worked example implied that one disease protected against the other — an artefact of who was admitted rather than a finding about disease.

How is selection bias different from confounding?

Confounding comes from a common cause of exposure and outcome existing in the population; selection bias comes from conditioning on a common effect by choosing whom to analyse. Confounding by a measured variable can be adjusted away. Selection bias cannot, because everyone analysed shares the same selection status.

How do you detect selection bias in a study?

Reconcile the denominators across abstract, table, flow diagram and model; read eligibility criteria for information only available after baseline; check whether the comparator came from the same source population; and compare attrition by exposure group. No statistical test identifies selection bias.

How do you reduce selection bias?

Define the target population before recruiting, sample from a frame that represents it, apply eligibility using baseline information only, and minimise differential attrition. Where selection has already occurred and its drivers were measured, inverse probability of selection weighting can partly correct it.

Does a larger sample fix selection bias?

No. A larger sample narrows the confidence interval around a biased estimate, making a wrong answer look more precise. The 1936 Literary Digest poll collected some 2.4 million responses and still called the election for the losing candidate, because of who responded rather than how many.