What is propensity score matching?

Propensity score matching estimates a causal effect from observational data by pairing treated and untreated subjects who had a similar predicted probability of receiving the treatment. That probability, the propensity score, was introduced by Rosenbaum and Rubin in 1983 and collapses many measured covariates into one number, so matching on that single number balances the covariates behind it — and only those.

Propensity score matching is therefore a method for handling confounding by measured variables, dressed in the vocabulary of experimental design. A matched cohort resembles a randomised trial in one specific respect: measured baseline characteristics are similar across arms. It resembles a trial in no other respect, and most disputes about the method concern how far that resemblance is allowed to be stretched.

What the propensity score is

The propensity score for a subject is the conditional probability of receiving the treatment given that subject’s measured baseline covariates, written e(X) = Pr(Z = 1 | X). A patient with a propensity score of 0.72 had, according to the fitted model, a 72% chance of being prescribed the drug given their recorded age, comorbidities, prior medications and laboratory values.

The result that makes the score useful is that e(X) is a balancing score: within a stratum of subjects who share the same propensity score, treatment assignment is independent of the covariates in X. Two patients with an identical score of 0.72 may look nothing alike — one elderly with mild disease, one young with severe disease — but across the whole set of subjects at that score, the distribution of covariates among the treated matches the distribution among the untreated.

This is what converts a high-dimensional matching problem into a one-dimensional one. Exact matching on 20 binary covariates requires filling 2^20 — about 1.05 million — cells with both a treated and an untreated subject, which no observational dataset does. Matching on the score requires filling one dimension.

Common questions about propensity scores

What is a propensity score in simple terms?

A propensity score is the probability that a subject would have received the treatment, given their recorded baseline characteristics. A score of 0.72 means the fitted model put that subject’s chance of treatment at 72%, given age, comorbidities, prior medications and laboratory values. Matching subjects on this single number balances the many covariates the model used to produce it.

How does propensity score matching work?

Propensity score matching works in three steps: fit a model predicting treatment from baseline covariates, pair each treated subject with an untreated subject whose predicted probability falls within a stated caliper, then check that covariates are balanced across the two arms before estimating the outcome effect within the matched cohort.

Why use propensity score matching instead of just adjusting for confounders in a regression?

A regression coefficient adjusts for confounders whether or not treated and untreated subjects actually resemble each other on those variables, and will extrapolate silently into covariate regions with no data. Propensity score matching separates that design question from the analysis: balance is checked and shown before any effect is estimated, and subjects with no comparable counterpart are dropped rather than extrapolated over.

The assumptions the method rests on

Propensity score matching identifies a causal effect only under conditions the data cannot verify. Rosenbaum and Rubin called the pair of them strongly ignorable treatment assignment.

Conditional exchangeability, or no unmeasured confounding. Potential outcomes must be independent of treatment given the measured covariates. Stated plainly: every common cause of treatment and outcome must be in X, and measured accurately. Nothing in the fitted model, the balance table or the matched estimate provides evidence for this assumption, and it is almost never exactly true in routinely collected health data.

Positivity, or common support. Every subject must have a non-zero probability of receiving either treatment. A patient with a propensity score of 0.998 has no plausible untreated counterpart, and where the two score distributions do not overlap, no matching algorithm can manufacture a comparison — it can only discard subjects or import a distant one and call it a match. The violation matters differently depending on where it comes from. Random non-positivity, from a modest sample that happens not to contain some rare covariate combination, shrinks as more data arrive. Structural non-positivity, where a covariate value rules out one treatment arm by design — a drug contraindicated above a certain age, a protocol that excludes a comorbidity — does not shrink with sample size, because no amount of additional data will ever put a treated observation at that covariate value.

Because these two assumptions carry the entire causal claim, and neither is testable, a manuscript that reports elaborate matching diagnostics and says nothing about either has documented the easy half of the argument.

Questions about propensity score matching assumptions

Can propensity score matching fix confounding I didn’t measure?

No. Propensity score matching balances only the covariates entered into the model that produced the score. Anything left out of that model — an unrecorded clinical impression, a lab value nobody coded — remains just as unbalanced after matching as before it. This is the same exposure ordinary regression adjustment carries, presented behind an explicit balance table rather than a bare coefficient.

What’s the difference between positivity and having enough overlap?

Positivity is the assumption itself: every subject must have a nonzero chance of receiving either treatment. Overlap is what a researcher checks to see whether positivity plausibly holds, by comparing propensity score distributions between arms. Poor overlap is evidence positivity may be violated — especially if it is structural, such as a contraindication, rather than a small-sample coincidence more data would resolve.

Which variables belong in the propensity score model

Variable selection for a propensity score model is a causal question, not a predictive one, and treating it as predictive is the most common technical error in applied work.

Include confounders and outcome predictors. Variables that cause the outcome, whether or not they strongly predict treatment, reduce variance and reduce bias when included. Brookhart and colleagues argued in 2006 that including a variable related only to the outcome carries almost no penalty and can be beneficial.

Exclude instruments. A variable that strongly predicts treatment but affects the outcome only through treatment — a prescriber’s preference, a clinic-level formulary — is an instrument. Including it in the propensity score model inflates variance and amplifies bias from whatever unmeasured confounding remains. Bias amplification is counterintuitive enough that instruments are routinely included by researchers who believe they are being thorough.

Exclude anything measured after treatment starts. A post-treatment variable is a mediator, a descendant of one, or a collider. Laboratory values drawn at the first follow-up visit, dose changes and adherence measures all fall here. A propensity score model containing a post-baseline variable is adjusting away the effect it is trying to estimate, or inducing an association that does not exist.

Do not select by stepwise regression or by model fit. A propensity score model with excellent discrimination is not a better model. A high c-statistic indicates that treated and untreated subjects are easy to tell apart, which means overlap is poor and matching will be difficult — the opposite of what a modeller usually wants. The score is a device for achieving balance, and balance, not fit, is the criterion by which it should be judged.

High-dimensional propensity score methods, developed by Schneeweiss and colleagues in 2009 for claims data, invert this logic deliberately: they screen hundreds of empirically defined code-based covariates as proxies for unmeasured confounders. That is a defensible strategy in a specific setting, and it is not the same thing as running stepwise selection over twelve variables.

A related trap runs in the opposite direction from bias amplification. Adjusting for a variable that is a common effect of two otherwise unrelated causes — one connected to treatment, the other to the outcome — can open a biasing path that did not exist before the adjustment, an error known as M-bias. M-bias needs a specific causal structure and is rarer in practice than confounding left unadjusted, but it is the reason “include everything measured at baseline” is not a safe default: what matters is the direction each variable’s arrows point, not merely whether it was recorded before treatment started.

The matching step and the choices that change the answer

Propensity score matching’s matching step involves at least five decisions, each of which moves the estimate, and each of which goes unreported often enough that a reader frequently cannot reconstruct what was done.

Distance metric. Most applied work matches on the logit of the propensity score rather than the score itself, because the logit is closer to normally distributed and avoids compressing distances near 0 and 1.

Caliper. A caliper is the maximum distance permitted within a matched pair. Rosenbaum and Rubin suggested 0.25 standard deviations of the logit of the propensity score in 1985; Austin’s simulation study in Pharmaceutical Statistics in 2011 found that a caliper of 0.2 standard deviations of the logit performed best across the scenarios examined. Matching with no caliper at all admits pairs of arbitrary dissimilarity, and is the default behaviour in some software.

Ratio and replacement. One-to-one matching without replacement is the common default. Matching with replacement lets a single untreated subject serve as the control for many treated subjects, which reduces bias where controls are scarce and requires weights in the analysis, because one control may be counted forty times. One-to-many matching increases precision and degrades average match quality, and the gain falls off quickly: the second and third control added per treated subject contribute far more than the tenth, because each further match is drawn from progressively further out in the distance ranking — so a large ratio can buy little additional power while steadily degrading every pair it adds.

Algorithm. Greedy nearest-neighbour matching processes treated subjects in the order they appear in the file. That has a consequence rarely stated in a methods section: sorting the dataset differently produces a different matched cohort and a different estimate. Optimal matching minimises total within-pair distance across the whole set and is order-independent. Full matching forms variable-sized strata and discards almost nobody.

Who gets discarded. One-to-one matching of treated subjects to controls estimates the average treatment effect in the treated — and only among treated subjects who found a match. If 1,842 treated patients yield 1,203 matched pairs, the estimand refers to those 1,203, a population defined by an algorithm rather than by a clinical question. The 639 unmatched treated patients are systematically different, usually more extreme in indication, and they are the patients a clinician is most likely to be asking about.

Questions about propensity score matching in practice

Why do I have fewer patients after matching than I started with?

One-to-one matching discards any treated subject with no untreated counterpart inside the caliper, along with unused untreated subjects. A drop from 1,842 treated patients to 1,203 matched pairs is normal, not an error, but the 639 who did not match are usually the more clinically extreme cases, and a paper should characterise them rather than stay silent about who was lost.

What caliper should I use for propensity score matching?

Austin’s 2011 simulation study in Pharmaceutical Statistics found a caliper of 0.2 standard deviations of the logit of the propensity score performed best across the scenarios tested, and it is the most commonly cited default. A wider caliper matches more subjects at the cost of pair similarity; matching with no caliper at all, still some software’s default, permits pairs of arbitrary dissimilarity.

Reviewer 2 says my sample size is too small for propensity score matching — what do I say?

Report the number of outcome events per covariate in the outcome model, not just the total sample size, since matching does not repeal the usual guidance of roughly ten events per variable. If overlap is genuinely poor, say so directly and report inverse probability weighting or full matching instead, since both use every subject rather than discarding the ones a small sample cannot pair.

Assessing balance, which is the real output of matching

Balance assessment is what a propensity score analysis actually produces, and it belongs in the paper as a primary methodological result rather than as an appendix table.

Standardised mean differences, not p-values. The standardised mean difference — the difference in means divided by a pooled standard deviation — is reported for every covariate before and after matching. A threshold of 0.1 is the working convention, and it is worth being honest that 0.1 is a convention rather than a derived quantity: nothing bad happens at 0.11 and nothing is guaranteed at 0.09.

Hypothesis tests for balance are the wrong instrument, a point argued forcefully by Imai, King and Stuart in 2008. A balance p-value is a function of sample size as well as of imbalance. Matching discards subjects, sample size falls, p-values rise, and a table of post-matching p-values above 0.05 can be produced by the loss of statistical power alone. Balance is a property of the sample in hand, not a hypothesis about a population.

Means are not the whole distribution. Two groups can share a mean and differ in variance, in skew, and in joint structure. Rubin’s guidance includes checking the ratio of variances of the propensity score between groups, with values roughly between 0.5 and 2 regarded as acceptable. Balance on interactions and on squared terms is checked far less often than it should be, and residual imbalance in an interaction is invisible in a standard baseline table.

A love plot — standardised differences for all covariates, before and after matching, on one axis — makes the result inspectable at a glance, and is the single most useful figure a propensity score paper can include. It summarises balance across the whole matched sample, though, and can still conceal imbalance within a clinically important subgroup — older patients, or a specific comorbidity stratum — that only a stratified balance check would reveal.

What propensity score matching does not do

Propensity score matching does not address unmeasured confounding, and no diagnostic within the method can detect its presence. This is the sentence most often missing from a discussion section, and it is not a technicality.

The measured-covariate limitation is identical to the limitation of ordinary regression adjustment. A propensity score model and a multivariable regression fitted to the same variables carry the same exposure to unmeasured confounding; the propensity score presents the problem more legibly, through an explicit balance table and an explicit overlap check, but does not reduce it. Investigators sometimes describe a matched cohort as “randomised-like” or “pseudo-randomised”. Randomisation balances unmeasured variables. Matching cannot, because the unmeasured variables are by definition absent from the score.

In pharmacoepidemiology the practical form of this limitation is confounding by indication and confounding by frailty. Clinicians prescribe on the basis of judgements — a patient’s trajectory, functional status, an impression formed across a consultation — recorded nowhere in a claims database. No number of coded covariates reaches an unrecorded clinical impression.

Propensity score matching also does nothing about biases that are not confounding. Immortal time bias arising from how follow-up is allocated relative to exposure definition survives matching intact, and matching at a post-baseline landmark can create it. Selection into the cohort, differential loss to follow-up and outcome misclassification are all untouched. Matching addresses one bias, and defaulting to it when the problem lies elsewhere leaves the real bias in place — the pattern described in my effect disappeared after adjusting.

The propensity score matching paradox

The propensity score matching paradox, described by King and Nielsen in Political Analysis in 2019, is the finding that pruning observations by propensity score distance can increase imbalance and model dependence beyond a certain point, rather than decreasing them monotonically as intuition suggests.

The argument runs as follows. Propensity score matching aims at the balance a completely randomised experiment would produce — balance in expectation across groups — rather than the stronger balance of a blocked experiment, where units are matched on covariates directly. Once the score distributions overlap well, further pruning removes pairs that were adequately matched on the score while doing nothing for their covariate profiles, and the remaining sample can become less balanced on the covariates themselves. King and Nielsen’s recommendation is to prefer methods that match on covariates directly, such as Mahalanobis distance matching or coarsened exact matching.

This is a genuinely contested point rather than a settled one. Methodologists have responded that the paradox arises mainly in particular configurations, that it is largely avoided by checking covariate balance rather than score balance at each step, and that the proposed alternatives carry their own difficulties as dimension grows. The honest position for an author is that the criticism exists, that the defence against it is to assess balance on covariates rather than assume the score has delivered it, and that a paper citing neither has not engaged with the current state of the method. For how methodological disputes of this kind land in review, see statistics objections.

Matching, weighting and regression compared

Propensity score matching is one of four standard uses of the score, and the choice among them has consequences for the estimand and for precision.

Matching produces an intuitive matched cohort, discards non-overlapping subjects transparently, and typically estimates the effect in the treated. What it gives up is discarded data and an estimand defined partly by an algorithm.

Inverse probability of treatment weighting retains all subjects, weighting each by the inverse of the probability of the treatment they received, and can target the average effect in the whole population. Its characteristic failure is extreme weights: a subject with a propensity score of 0.01 who received treatment carries a weight of 100 and can dominate the estimate. Weight truncation and stabilisation address this by trading a little bias for stability. Overlap weighting sidesteps the same failure by construction rather than correction: each subject is weighted by the probability of the treatment they did not receive, a quantity bounded between 0 and 1 that shrinks toward zero exactly where positivity is weakest, so no truncation rule is needed. The trade-off is the estimand — overlap weighting targets the effect in the population where covariate distributions overlap most, a different quantity from the effect in the treated or in everyone.

Stratification on quintiles of the score is simple and removes a large share of measured confounding, with residual imbalance remaining within strata.

Covariate adjustment for the score — entering the score itself as a regressor — reintroduces a functional-form assumption and loses the method’s main diagnostic advantage, because balance is never displayed.

Combining approaches is generally better than choosing one. Regression adjustment for covariates within the matched sample gives a doubly robust estimator: the effect estimate remains consistent if either the propensity score model or the outcome model is correctly specified. Reporting a matched estimate alongside a weighted estimate and a conventionally adjusted estimate lets a reader see how far the conclusion depends on the analytic route, which is more informative than any single number.

Analysing the matched sample

Analysis of a propensity-score-matched cohort raises a question with no unanimous answer, and the disagreement is better acknowledged than resolved by silence.

Matched subjects are not independent, because matching induces a dependence between members of a pair. Austin has argued that the analysis should respect the matched design, using paired methods — a paired t-test, McNemar’s test, conditional logistic regression, or a Cox model stratified on matched set. Others have argued that ignoring the matching yields standard errors that are conservative rather than anti-conservative, so the consequence is lost precision rather than a false positive.

Two points are not disputed. First, uncertainty in the estimated propensity score is ignored by naive standard errors, and Abadie and Imbens showed in 2008 that the ordinary bootstrap is not valid for nearest-neighbour matching with a fixed number of matches — a result worth knowing before bootstrapping a matched analysis by reflex. Second, matching with replacement requires weights, because a control used repeatedly contributes repeatedly.

Where the outcome is time to event, the matched structure interacts with the usual survival assumptions, and a matched cohort does not repair non-proportional hazards or crossing survival curves.

How propensity score matching is detected in a manuscript

Propensity score matching is detected in a manuscript by reconciling the numbers, not by reading the methods paragraph, because the methods paragraph is often one sentence and the numbers usually disagree with it.

The checks that find problems, roughly in order of yield:

Do the counts reconcile? Treated subjects before matching, matched pairs, and the analytic sample should be traceable through a flow diagram. A drop from 1,842 treated to 1,203 matched that is never mentioned in the text is the most common finding.

Is the caliper stated? A methods section describing “one-to-one nearest neighbour matching” with no caliper, no replacement status and no matching ratio has under-specified the analysis to the point where it cannot be reproduced.

Is balance shown before and after, as standardised differences? A post-matching table showing only p-values, or showing only the matched cohort with no pre-matching column, cannot demonstrate that matching achieved anything.

Does the covariate list contain post-baseline variables? Cross-checking the propensity score covariates against the study timeline catches mediators and colliders that a baseline table conceals by listing everything as “baseline characteristics”.

Does the discussion claim more than the design supports? Phrases such as “eliminated confounding” or “randomised-like”, and causal verbs applied to the exposure without qualification, mark the point where a defensible analysis becomes an indefensible conclusion. Related overreach appears in the table 2 fallacy, where covariate coefficients from an adjustment model are read as effects in their own right.

Is there a sensitivity analysis for unmeasured confounding? Rosenbaum bounds, an E-value, or a negative control outcome are all available; the absence of any of them, in a paper whose central limitation is unmeasured confounding, is conspicuous. These checks are routine substance in epidemiology review, and where the data source is a registry or claims database they compound with the limitations of that data source.

What a peer reviewer says when it is mishandled

“The propensity score model is not specified. Please list the covariates included and state the basis on which they were selected.”

“Balance is assessed using significance tests. Please report standardised mean differences before and after matching for all covariates, ideally as a love plot.”

“Of 1,842 treated patients, 1,203 were matched. The 639 unmatched patients are not characterised, and the population to which the estimate applies is therefore unclear.”

“Propensity score matching balances measured covariates only. The conclusion that the treatment reduced mortality is not supported by this design, and the causal phrasing throughout the discussion should be revised.”

“Please report the caliper width, whether matching was performed with replacement, and the matching ratio. As written, the analysis cannot be reproduced.”

“Several covariates in the propensity score model appear to have been measured after treatment initiation.”

“Confounding by indication is likely to persist. A quantitative bias analysis or E-value stating how strong an unmeasured confounder would need to be to nullify the result would strengthen the paper considerably.”

“Standard errors do not appear to account for the matched design.”

The fourth of these decides more papers than the rest combined, and it is the one authors can pre-empt entirely by editing their own discussion section. A matched analysis described accurately is publishable; a matched analysis described as equivalent to a trial invites a rejection the analysis itself did not deserve. The general form of that objection is set out in correlative, not causal.

How to report propensity score matching

Reporting a propensity score analysis well means letting a reader reconstruct the matched cohort and disagree with how it was built.

State the covariates in the score and the causal reason for each, not merely that they were available. State the estimation method — logistic regression, or a machine-learning alternative with its tuning specified. State the matching algorithm, the caliper in units of standard deviations of the logit, the ratio, and whether replacement was used. Give a flow diagram of subjects lost at each step, and describe the unmatched treated subjects. Report standardised mean differences before and after for every covariate. Name the estimand explicitly: the average treatment effect in the treated, in the matched treated, or in an overlap population. Report the analysis method used on the matched data, and say whether it accounted for pairing.

Then address unmeasured confounding as a quantity rather than as a caveat. State which confounders were unavailable in the data source, the direction in which each would bias the estimate, and how strong an unmeasured confounder would have to be to explain the finding. A limitation section naming frailty, functional status and prescriber preference as unmeasured, and quantifying what they would need to do, is more credible than one saying residual confounding cannot be excluded.

Related

Confounding · Collider bias · Mediator versus confounder · Immortal time bias · Table 2 fallacy

Checked before submission by causal language discipline, which reports whether the adjustment strategy is described in enough detail to be reproduced, and flags conclusions phrased more causally than the design supports.

Review my manuscript

Frequently asked questions

How do I know if my propensity score matching actually worked?

Matching only worked if the standardised mean difference for every covariate in the model, not just the ones that predicted treatment strongly, is close to zero after matching, typically under 0.1, and if the resulting matched sample is still large enough to answer the question. A model with excellent discrimination but poor post-match balance has not worked, whatever its c-statistic suggests.

Is propensity score matching the same as randomisation?

No. Propensity score matching balances measured covariates only; randomisation balances measured and unmeasured covariates alike, because treatment assignment is independent of every patient characteristic, known or not. Describing a matched cohort as randomised-like overstates what matching achieves, and it is a phrase reviewers routinely flag in a manuscript’s discussion section.

I ran a propensity score match and now half my treated patients are gone — is that a problem?

Losing half your treated patients to matching is only a problem if a manuscript does not say so. If those patients lacked any comparable untreated subject within the caliper, dropping them is exactly what matching is supposed to do — but the estimate now describes only the matched half, and the unmatched patients, usually the more extreme cases, deserve their own line in the results.

Can you explain why my p value went from significant to not significant after propensity score matching?

A p value can move after matching for reasons unrelated to the true effect: matching discards subjects, which shrinks the sample and widens standard errors, and an analysis that ignores the paired structure produces a variance estimate that can be off in either direction. Check the matched sample size and whether standard errors were stratified on matched set before concluding the effect disappeared.