What is overfitting?

Overfitting occurs when a statistical or machine-learning model captures noise specific to the sample it was fitted on, so its apparent performance overstates how well it will perform on new data. The gap between apparent and out-of-sample performance is called optimism, and it grows as model flexibility rises relative to the information the data actually contain.

That definition is uncontroversial. What causes trouble in manuscripts is that overfitting is usually introduced somewhere the authors do not think of as modelling — during variable screening, hyperparameter tuning, preprocessing, or sample construction — and that the fitted model gives no internal signal that anything has gone wrong. This page covers the mechanism, how much data is enough, the routes by which optimism enters, how overfitting is detected in a submitted manuscript, what internal validation must include, and what reviewers write when it has been handled badly.

The mechanism: signal, noise and optimism

Every dataset contains structure that would recur in a new sample from the same population, and structure that is an accident of the particular observations drawn. Estimation cannot tell the two apart. A fitting procedure with enough freedom will reproduce both, and the accidental part will not recur, so performance measured on new data falls.

The quantity that matters is optimism: apparent performance minus expected performance in new data from the same population. The bootstrap estimate of optimism is due to Bradley Efron (Journal of the American Statistical Association, 1983), and it is what bootstrap internal validation is built to estimate. Optimism rises with the number of parameters estimated, rises as the effective sample size falls, and rises further with every decision made by looking at the outcome — which is why counting only the predictors that survived into the final model understates it.

One caveat on optimism as defined above: it is conceptually related to, but not identical with, the correction implicit in information criteria such as AIC or BIC. Those criteria approximate the expected optimism averaged across repeated sampling from a model of a given complexity, not the optimism of the one specific model actually fitted to this dataset — which is why choosing a model by AIC does not substitute for reporting a bootstrap-corrected performance estimate for whichever model that selection lands on.

The classical framing is the bias–variance decomposition: a more flexible model has lower bias and higher variance, and expected prediction error is minimised somewhere between the two. The opposite failure, underfitting, is a model too rigid to capture structure that is genuinely there, and it shows a small gap between apparent and validated performance at a uniformly poor level. That framing is a useful teaching device and an incomplete account of modern overparameterised models, a limitation addressed further down this page.

Overfitting is not the same as a large number of predictors

A model can be overfitted with three predictors and well fitted with three hundred. The number of columns in the design matrix is a poor proxy for optimism, and manuscripts that defend a model by pointing at a modest predictor count are answering the wrong question.

The problem compounds for penalised models. Ridge regression, lasso and elastic net fit a coefficient for every predictor in the formula, but the quantity that actually governs overfitting risk is the effective degrees of freedom, which is typically far below the nominal coefficient count because the penalty shrinks most coefficients toward zero rather than estimating them freely. Comparing a penalised and an unpenalised model by counting terms in the two formulas compares two different things, and a manuscript that reports “we used the same twelve predictors as the unpenalised model” has not thereby matched its complexity.

What governs optimism is the ratio of estimated quantities to effective information, together with how much of the fitting was steered by the outcome. Effective information is not the row count. For binary logistic regression the binding constraint is the smaller of the two outcome classes, so a cohort of 5,000 patients containing 40 deaths is constrained by those 40 events, not by the 5,000 rows. For Cox regression the constraint is the number of events, not the number at risk. For clustered designs the constraint is the number of independent units — a point shared with pseudoreplication, where treating correlated observations as independent inflates the apparent information in exactly the same way.

State the effective sample size in the manuscript, not the row count. A reviewer who has to compute it from your Table 1 will usually conclude you did not compute it either.

How much data is enough

The rule of thumb most authors cite is ten events per variable, from the simulation study by Peduzzi and colleagues published in the Journal of Clinical Epidemiology in 1996. That work examined coverage and bias of coefficients in logistic regression, and the figure entered the literature as a general adequacy criterion it was never designed to be.

Subsequent work has been unkind to it. Van Smeden and colleagues showed that events per variable is neither necessary nor sufficient: adequacy depends on the outcome proportion, the predictor effects, and the intended use of the model, and models satisfying ten events per variable can still be badly overfitted while models below it can be adequate. Riley and colleagues then published sample-size criteria for prediction model development that replace the rule with explicit targets — an expected uniform shrinkage factor of at least 0.9, small absolute optimism in the apparent explained variation, and precise estimation of the overall outcome risk. The criteria are implemented in the pmsampsize package and require the analyst to state an anticipated model performance up front.

Riley’s criteria were derived for regression-based prediction models and are stated in terms of events, candidate predictors and anticipated shrinkage; they do not translate unchanged to tree ensembles, gradient boosting or neural networks, where parameters are not countable in the same sense and the operative constraint is closer to effective model capacity under the specific training procedure than to a fixed events-per-parameter ratio. A manuscript using those model classes needs a sample-size justification suited to them, not a borrowed logistic-regression number.

The practical consequence for a manuscript is that “we had more than ten events per variable” is no longer a sufficient answer, and reviewers working in prediction modelling now say so directly. See also reviewer says the sample size is too small for how that objection is usually phrased and what answers it.

Common questions about sample size for prediction models

Is ten events per variable actually enough?

Ten events per variable is not a reliable adequacy threshold on its own. Van Smeden and colleagues showed the rule is neither necessary nor sufficient: models meeting it can still be badly overfitted, and models below it can be adequate, because adequacy depends on outcome prevalence, predictor effects and intended use, not on a fixed ratio.

How many patients or events do I actually need for logistic regression?

The number needed for logistic regression is not the patient count but the event count in the smaller outcome class, and it depends on the anticipated model performance you plan to report. Riley and colleagues’ criteria target an expected shrinkage factor of at least 0.9 and small optimism in explained variation, implemented in the pmsampsize package, replacing fixed ratios with explicit, study-specific targets.

Can a model with only three or five predictors still be overfitted?

Yes: a model with three predictors can be badly overfitted, and one with three hundred can be well fitted, because overfitting tracks the ratio of estimated quantities to effective information, not the raw predictor count. A small cohort with few outcome events yields little effective information regardless of how few variables the final model reports.

Where optimism enters without being counted

Overfitting is most often introduced by steps the authors describe in the methods as preparation rather than as modelling. Each of the following consumes degrees of freedom against the outcome and belongs inside any resampling loop used to estimate performance.

Univariable screening. Selecting predictors by their marginal association with the outcome, at p < 0.05 or p < 0.20, is a data-dependent modelling decision. The screening looked at the outcome; the final model inherits its optimism.

Stepwise selection. Forward, backward and bidirectional selection produce coefficients biased away from zero, standard errors that are too small, and p-values that do not have their nominal meaning, because the selection is ignored when the final model is summarised.

Hyperparameter tuning. Choosing a penalty strength, a tree depth or a learning rate by comparing held-out performance uses the held-out data for fitting. Reported performance from that same split is an in-sample figure wearing an out-of-sample label.

Cutpoint optimisation. Choosing the threshold that maximises Youden’s index, then reporting sensitivity and specificity at that threshold without correction, is one of the most reliable sources of unreproducible diagnostic accuracy in the literature.

Preprocessing fitted on all the data. Centring, scaling, imputation models, principal components and class-balancing such as SMOTE all estimate quantities. Fitting them before the split leaks information from the evaluation set into the model.

Data leakage: the version that produces impossible performance

Data leakage is the case where information that would not be available at prediction time reaches the model, and it produces the reported accuracies that experienced reviewers distrust on sight — area under the curve above 0.95 for a hard clinical problem, or near-zero error in a high-dimensional classification task.

The canonical demonstration is Ambroise and McLachlan’s 2002 paper in PNAS on gene extraction from microarray data, which showed that selecting discriminating genes on the full dataset and then cross-validating the classifier gives an almost unbelievably low error rate, while performing the selection inside each fold gives a far higher and far more honest one. The same structure recurs constantly in omics work; see multiple testing in omics for the related inference problem and batch effects for the case where the leaked variable is the processing run rather than the outcome.

Kapoor and Narayanan’s 2023 paper in Patterns catalogued eight distinct types of leakage and identified affected studies across seventeen scientific fields, arguing that leakage is a principal driver of irreproducible machine-learning results in applied science. Their taxonomy is worth reading before writing a methods section, because several of the eight — no test set at all, temporal leakage, and non-independence between training and test samples — are invisible in the results and detectable only from the design description.

A leakage variant that is easy to miss even with a correctly constructed train/test split is a predictor that is itself a near-proxy for the outcome: a post-treatment laboratory value, a billing or procedure code that was only assigned because the diagnosis had already been made, or an unstructured text field that mentions the outcome directly. None of these depend on which rows sit in which split, so the leak is in the column rather than the split, and it is caught only by asking, predictor by predictor, whether that value could plausibly have been available at the moment a prediction is actually needed.

Non-independence between training and evaluation sets

Splitting at the level of the observation rather than the level of the independent unit is a leakage mechanism that survives peer review more often than it should, because the split looks correct in the code.

Multiple images from one patient, repeated measures from one animal, cells from one culture, or records from one hospital placed on both sides of a split let the model learn the unit rather than the phenomenon. Reported accuracy then reflects re-identification. The fix is grouped splitting — group-wise cross-validation, or leave-one-site-out — and the manuscript must say which grouping variable was used. This is the prediction-modelling face of the objection covered in reviewer says the n is not independent, and the correction is the same: the unit of analysis is the unit of randomisation or sampling, not the row.

Temporal structure creates the same problem in a different shape. Random splitting of longitudinal or electronic-health-record data places future observations in the training set and past observations in the evaluation set, which is not the prediction task the model claims to perform.

How overfitting is detected in a real manuscript

Overfitting cannot be detected by inspecting the fitted model, because a badly overfitted model has excellent fit statistics by construction. Detection is done from the design description and from a small number of reported quantities, in roughly this order.

Compute the effective sample size and count the estimated parameters. Events, not patients. Count every candidate predictor considered, every knot in a spline, every interaction tested, and every category level, not the parameters surviving in the final table.

Look for the sequence of the analysis. Locate where variable selection, imputation, tuning and threshold choice occurred relative to the split or the resampling loop. Methods sections that describe selection and validation in separate paragraphs, in that order, usually performed them in that order.

Read the calibration. A validation calibration slope below 1 is the direct measurement of overfitting, described below. Absence of any calibration reporting is itself a finding.

Compare apparent and validated performance. If only one performance figure appears, ask which it is. A single C-statistic quoted with no statement of how it was obtained is almost always apparent.

Check whether the accuracy is plausible for the problem. Discrimination markedly better than published models for the same outcome, in a smaller sample, is a leakage signal rather than an achievement, and warrants tracing the data flow.

Calibration slope: the quantity that measures overfitting

The calibration slope is obtained by fitting the outcome in the validation data on the linear predictor from the developed model and reading the coefficient. A slope of 1 indicates predictions of the correct spread; a slope below 1 indicates predictions that are too extreme — high risks too high, low risks too low — which is the characteristic signature of an overfitted model.

Van Calster and colleagues defined the hierarchy of calibration — mean, weak, moderate and strong — in the Journal of Clinical Epidemiology in 2016, and in BMC Medicine in 2019 called calibration the Achilles heel of predictive analytics because it is the property most often omitted and the one that determines whether a model is safe to act on. Discrimination can be preserved while calibration fails completely: a model can rank patients correctly and still tell every one of them a wrong absolute risk.

Report the calibration slope and the calibration intercept from validation, alongside a flexible calibration curve. A model whose C-statistic is quoted without a calibration slope has, in practice, not been shown to be usable, and reviewers in the prediction-model literature increasingly treat the omission as a major revision rather than a presentational one.

The intercept and the slope catch different failures, and both are needed. The calibration intercept, sometimes called calibration-in-the-large, catches a model whose average predicted risk is systematically too high or too low across the whole validation sample even when the relative ranking of patients is fine — a problem of level, not shape. The slope catches predictions that are too extreme in one direction and too weak in the other, the shape problem that overfitting specifically produces. A model can pass one of the two checks and fail the other, so reporting only one is not a partial answer, it is an answer to a different question than the one calibration is meant to settle.

Common questions about spotting overfitting in a manuscript

Why does my model do great on training data but fall apart on the test set?

A model that performs well in training and poorly on a held-out test set has fitted noise specific to the training sample rather than structure that recurs in new data, and the gap between the two is optimism. It grows with the number of quantities estimated and with every decision, such as tuning or variable choice, made by looking at the outcome.

Is a C-statistic or AUC of 0.95 or higher too good to be true?

A C-statistic above 0.95 for a genuinely hard clinical prediction problem is usually a leakage signal rather than a modelling achievement, particularly in a small sample where no comparable published model reaches that level. Reviewers reading such a figure typically trace the data flow for a variable that encodes the outcome, rather than accept it as validated discrimination.

My model has a C-statistic of 0.95 but only 60 events in the validation data — could this be overfitting?

A C-statistic of 0.95 from only 60 validation events is plausible overfitting or leakage rather than genuine performance, because 60 events supply little effective information regardless of the headline discrimination figure. Check the calibration slope from that same validation: a slope below 1, alongside an implausibly high C-statistic, is the specific pattern reviewers flag.

What’s the actual difference between overfitting and data leakage?

Overfitting is a model fitting noise in a sample it legitimately had access to; data leakage is information reaching the model that would not be available at prediction time, such as a variable derived after the outcome occurred. Leakage produces the extreme, near-perfect performance figures documented in Ambroise and McLachlan’s 2002 microarray study, while ordinary overfitting degrades performance more moderately.

Internal validation that actually estimates optimism

Internal validation estimates how much the developed model’s apparent performance is inflated for the population it was developed in. Done correctly, the entire modelling process — screening, selection, tuning, imputation, threshold choice — is repeated inside each resample.

Bootstrap optimism correction is the reference method for prediction models. Draw a bootstrap sample, run the whole modelling procedure on it, measure performance on the bootstrap sample and on the original data, and take the difference as optimism; average over a few hundred repetitions and subtract from the apparent performance. Steyerberg and colleagues compared internal validation procedures in the Journal of Clinical Epidemiology in 2001 and found bootstrapping efficient and stable where split-sample validation was neither.

Nested cross-validation is required whenever hyperparameters are tuned. Tuning occurs in an inner loop; performance is measured in an outer loop that the tuning never saw. Varma and Simon showed in BMC Bioinformatics in 2006 that a single cross-validation loop used for both tuning and evaluation gives a substantially biased error estimate.

Random split-sample validation is the weakest of the three and is still the most commonly reported. Holding out 30% of a modest dataset both wastes information that the model needed and produces a performance estimate with wide uncertainty, and it does not become external validation by being called a test set.

Shrinkage and penalisation, and their limits

Shrinkage deliberately pulls estimated coefficients towards zero to counteract the outward bias that fitting introduces, trading a little bias for a worthwhile reduction in variance. Van Houwelingen and Le Cessie’s heuristic uniform shrinkage factor, and the penalised approaches — ridge, lasso and elastic net — are the standard tools, and a uniform shrinkage factor estimated during development should be reported alongside the coefficients.

The honest limitation is that penalisation is not a repair for an inadequate sample. Riley and colleagues showed in the Journal of Clinical Epidemiology in 2021 that penalisation and shrinkage methods are themselves unreliable when the development sample is small: the tuning parameter is estimated with high variance, so the amount of shrinkage applied varies substantially across samples and can be badly wrong in either direction for any individual dataset. Penalisation improves expected performance across many datasets without guaranteeing improvement in yours.

A model that needs heavy shrinkage to behave is a model developed on too little data. Report the shrinkage factor as a diagnostic, not only as a remedy.

Common questions about fixing overfitting

Should feature selection happen before or after I split the data?

Feature selection must happen after splitting, and inside each resampling fold if performance is being estimated by cross-validation or bootstrapping, never on the full dataset beforehand. Selecting predictors by their association with the outcome before the split lets the held-out data influence which variables enter the model, the single most common source of optimistically biased results.

Do I need external validation, or is cross-validation enough on its own?

Cross-validation and external validation answer different questions, so one cannot substitute for the other: internal validation, whether bootstrap or cross-validation, estimates optimism for the population the model was developed in, while external validation in data from a different setting, period or population estimates transportability. A model can be internally sound and still lose most of its calibration elsewhere.

External validation answers a different question

External validation measures performance in data from a different setting, period or population, and it is not interchangeable with internal validation however the two are labelled in a manuscript.

Internal validation estimates optimism for the development population. External validation estimates transportability, and models routinely lose considerable discrimination and nearly all their calibration when moved between hospitals, countries or eras — not because the development was overfitted, but because case mix and baseline risk differ. Both can be true at once, and separating the two failure modes requires reporting calibration in the new setting rather than discrimination alone.

Wynants and colleagues’ living systematic review of COVID-19 diagnostic and prognostic models in The BMJ, first published in 2020, assessed the published models with PROBAST and rated all of them at high or unclear risk of bias, with small effective sample sizes, unclear handling of selection, and absent calibration among the recurring problems. The review is a useful calibration of how common these defects are in a literature written quickly and under pressure. For the related demand that a finding be confirmed by an independent method, see reviewer wants orthogonal validation.

What a peer reviewer says when overfitting is mishandled

Reviewer comments on overfitting are recognisable and repetitive. Anticipating the exact phrasing is among the most efficient revision work available.

“The reported performance is apparent performance; internal validation correcting for optimism is not presented.” “Variable selection was performed on the full dataset prior to cross-validation, so the reported error rate is optimistically biased.” “The number of candidate predictors is large relative to the number of events; the authors should report the effective sample size and a shrinkage factor.” “No calibration is reported; the C-statistic alone does not establish that the model is usable.” “The train–test split appears to place multiple observations from the same patient on both sides.” “The threshold was selected to maximise accuracy in the same data in which accuracy is reported.” “The authors describe this as external validation, but the validation set is a random partition of the development data.”

The comment that most often decides a prediction-model paper is the calibration one, because it cannot be answered by argument — either the slope was reported or the analysis has to be rerun. The overclaiming that accompanies an uncorrected performance figure attracts its own objection; see overclaim checking.

One miscommunication is worth naming directly, because it wastes an entire revision cycle when missed. A reviewer who writes “this needs external validation” sometimes means the internal validation was performed correctly but is insufficient evidence for the claimed clinical use, and sometimes means the internal validation itself looks wrong — split-sample rather than bootstrap, or selection done outside the resampling loop — and the reviewer is really asking for that to be redone properly before external data is even relevant. Reading the rest of the review before drafting a response letter, to work out which of the two is meant, is worth the extra pass.

Contested ground: where the classical account breaks down

The classical bias–variance account predicts that performance degrades monotonically once a model has more parameters than the data can support, and modern overparameterised models falsify that prediction in its simple form.

Belkin and colleagues described double descent in PNAS in 2019: as model capacity increases past the interpolation threshold — the point at which the model fits the training data exactly — test error first rises, as expected, and then falls again, sometimes below the classical optimum. Large neural networks operate in that second regime, which is why “the model has more parameters than observations, therefore it is overfitted” is not a valid argument against a deep-learning result, though it remains a valid argument against an unpenalised regression.

Two caveats keep this from being an escape route in applied biomedical work. First, the regime depends on implicit regularisation from the optimiser and on very large datasets, conditions rarely met in a clinical study with a few hundred events. Second, double descent concerns discrimination-type error and says nothing reassuring about calibration, which is the property clinical use depends on. The safe reading is that parameter counting is a weak argument in both directions, and that empirical optimism estimation and calibration reporting are what settle the question.

A further scope limit worth stating plainly: double descent is documented for models trained by gradient-based optimisation under squared-error or cross-entropy loss, primarily in the deep-learning literature it was described from. It is not a license to skip sample-size planning for a penalised logistic regression or a random forest fitted by the ordinary criteria used in applied biomedical work, where the classical bias–variance account still describes what is observed and where the sample sizes involved rarely approach the interpolation threshold in the first place.

Frequently asked questions

Can a model have more parameters than data points and still not be overfitted?

Yes, in the double-descent regime described by Belkin and colleagues in PNAS in 2019: past the point where a model exactly fits the training data, test error can fall again rather than keep rising. This depends on implicit regularisation and very large datasets, conditions rarely met in a clinical study with a few hundred events, so parameter counting stays a weak argument either way there.

A reviewer told me my sample size is too small for the number of predictors I used — what should I write back?

A sample-size-versus-predictors objection is answered with the effective sample size in events, the sample-size criteria used to justify the design, and optimism-corrected performance from bootstrap or cross-validation — not by re-asserting that ten events per variable was met. See reviewer says the sample size is too small for how this is usually phrased and answered.

How to report overfitting in a manuscript

Overfitting is reported adequately only when a manuscript states, together rather than piecemeal, the effective sample size, the number of candidate parameters considered including everything discarded, and the sample-size justification actually used. Describe the modelling steps in the order performed and state explicitly which steps were repeated inside the resampling loop. Report apparent performance and optimism-corrected performance as separate numbers. Report a calibration slope, a calibration intercept and a flexible calibration plot from validation, not discrimination alone. State the shrinkage or penalty applied and how the tuning parameter was chosen.

Follow TRIPOD for the reporting structure — the 2015 statement for regression-based prediction models, and TRIPOD+AI, published in The BMJ in 2024, for models developed with machine learning. Where the model is offered as explanatory rather than predictive, the separate question of which covariates belong in it is governed by confounding, not by predictive performance.

Related

Collider bias · Reviewer says the model is not appropriate · Registry data limitations

Checked before submission by the statistics objections lane, which reports whether performance figures are apparent or optimism-corrected, whether selection and tuning sit inside the resampling loop, and whether calibration is reported at all.

Review my manuscript