What is overfitting?

Overfitting occurs when a statistical or machine-learning model captures noise specific to the sample it was fitted on, so its apparent performance overstates how well it will perform on new data. The gap between apparent and out-of-sample performance is called optimism, and it grows as model flexibility rises relative to the information the data actually contain.

That definition is uncontroversial. What causes trouble in manuscripts is that overfitting is usually introduced somewhere the authors do not think of as modelling — during variable screening, hyperparameter tuning, preprocessing, or sample construction — and that the fitted model gives no internal signal that anything has gone wrong. This page covers the mechanism, how much data is enough, the routes by which optimism enters, how overfitting is detected in a submitted manuscript, what internal validation must include, and what reviewers write when it has been handled badly.

The mechanism: signal, noise and optimism

Every dataset contains structure that would recur in a new sample from the same population, and structure that is an accident of the particular observations drawn. Estimation cannot tell the two apart. A fitting procedure with enough freedom will reproduce both, and the accidental part will not recur, so performance measured on new data falls.

The quantity that matters is optimism: apparent performance minus expected performance in new data from the same population. The bootstrap estimate of optimism is due to Bradley Efron (Journal of the American Statistical Association, 1983), and it is what bootstrap internal validation is built to estimate. Optimism rises with the number of parameters estimated, rises as the effective sample size falls, and rises further with every decision made by looking at the outcome — which is why counting only the predictors that survived into the final model understates it.

The classical framing is the bias–variance decomposition: a more flexible model has lower bias and higher variance, and expected prediction error is minimised somewhere between the two. The opposite failure, underfitting, is a model too rigid to capture structure that is genuinely there, and it shows a small gap between apparent and validated performance at a uniformly poor level. That framing is a useful teaching device and an incomplete account of modern overparameterised models, a limitation addressed further down this page.

Overfitting is not the same as a large number of predictors

A model can be overfitted with three predictors and well fitted with three hundred. The number of columns in the design matrix is a poor proxy for optimism, and manuscripts that defend a model by pointing at a modest predictor count are answering the wrong question.

What governs optimism is the ratio of estimated quantities to effective information, together with how much of the fitting was steered by the outcome. Effective information is not the row count. For binary logistic regression the binding constraint is the smaller of the two outcome classes, so a cohort of 5,000 patients containing 40 deaths is constrained by those 40 events, not by the 5,000 rows. For Cox regression the constraint is the number of events, not the number at risk. For clustered designs the constraint is the number of independent units — a point shared with pseudoreplication, where treating correlated observations as independent inflates the apparent information in exactly the same way.

State the effective sample size in the manuscript, not the row count. A reviewer who has to compute it from your Table 1 will usually conclude you did not compute it either.

How much data is enough

The rule of thumb most authors cite is ten events per variable, from the simulation study by Peduzzi and colleagues published in the Journal of Clinical Epidemiology in 1996. That work examined coverage and bias of coefficients in logistic regression, and the figure entered the literature as a general adequacy criterion it was never designed to be.

Subsequent work has been unkind to it. Van Smeden and colleagues showed that events per variable is neither necessary nor sufficient: adequacy depends on the outcome proportion, the predictor effects, and the intended use of the model, and models satisfying ten events per variable can still be badly overfitted while models below it can be adequate. Riley and colleagues then published sample-size criteria for prediction model development that replace the rule with explicit targets — an expected uniform shrinkage factor of at least 0.9, small absolute optimism in the apparent explained variation, and precise estimation of the overall outcome risk. The criteria are implemented in the pmsampsize package and require the analyst to state an anticipated model performance up front.

The practical consequence for a manuscript is that “we had more than ten events per variable” is no longer a sufficient answer, and reviewers working in prediction modelling now say so directly. See also reviewer says the sample size is too small for how that objection is usually phrased and what answers it.

Where optimism enters without being counted

Overfitting is most often introduced by steps the authors describe in the methods as preparation rather than as modelling. Each of the following consumes degrees of freedom against the outcome and belongs inside any resampling loop used to estimate performance.

Univariable screening. Selecting predictors by their marginal association with the outcome, at p < 0.05 or p < 0.20, is a data-dependent modelling decision. The screening looked at the outcome; the final model inherits its optimism.

Stepwise selection. Forward, backward and bidirectional selection produce coefficients biased away from zero, standard errors that are too small, and p-values that do not have their nominal meaning, because the selection is ignored when the final model is summarised.

Hyperparameter tuning. Choosing a penalty strength, a tree depth or a learning rate by comparing held-out performance uses the held-out data for fitting. Reported performance from that same split is an in-sample figure wearing an out-of-sample label.

Cutpoint optimisation. Choosing the threshold that maximises Youden’s index, then reporting sensitivity and specificity at that threshold without correction, is one of the most reliable sources of unreproducible diagnostic accuracy in the literature.

Preprocessing fitted on all the data. Centring, scaling, imputation models, principal components and class-balancing such as SMOTE all estimate quantities. Fitting them before the split leaks information from the evaluation set into the model.

Data leakage: the version that produces impossible performance

Data leakage is the case where information that would not be available at prediction time reaches the model, and it produces the reported accuracies that experienced reviewers distrust on sight — area under the curve above 0.95 for a hard clinical problem, or near-zero error in a high-dimensional classification task.

The canonical demonstration is Ambroise and McLachlan’s 2002 paper in PNAS on gene extraction from microarray data, which showed that selecting discriminating genes on the full dataset and then cross-validating the classifier gives an almost unbelievably low error rate, while performing the selection inside each fold gives a far higher and far more honest one. The same structure recurs constantly in omics work; see multiple testing in omics for the related inference problem and batch effects for the case where the leaked variable is the processing run rather than the outcome.

Kapoor and Narayanan’s 2023 paper in Patterns catalogued eight distinct types of leakage and identified affected studies across seventeen scientific fields, arguing that leakage is a principal driver of irreproducible machine-learning results in applied science. Their taxonomy is worth reading before writing a methods section, because several of the eight — no test set at all, temporal leakage, and non-independence between training and test samples — are invisible in the results and detectable only from the design description.

Non-independence between training and evaluation sets

Splitting at the level of the observation rather than the level of the independent unit is a leakage mechanism that survives peer review more often than it should, because the split looks correct in the code.

Multiple images from one patient, repeated measures from one animal, cells from one culture, or records from one hospital placed on both sides of a split let the model learn the unit rather than the phenomenon. Reported accuracy then reflects re-identification. The fix is grouped splitting — group-wise cross-validation, or leave-one-site-out — and the manuscript must say which grouping variable was used. This is the prediction-modelling face of the objection covered in reviewer says the n is not independent, and the correction is the same: the unit of analysis is the unit of randomisation or sampling, not the row.

Temporal structure creates the same problem in a different shape. Random splitting of longitudinal or electronic-health-record data places future observations in the training set and past observations in the evaluation set, which is not the prediction task the model claims to perform.

How overfitting is detected in a real manuscript

Overfitting cannot be detected by inspecting the fitted model, because a badly overfitted model has excellent fit statistics by construction. Detection is done from the design description and from a small number of reported quantities, in roughly this order.

Compute the effective sample size and count the estimated parameters. Events, not patients. Count every candidate predictor considered, every knot in a spline, every interaction tested, and every category level, not the parameters surviving in the final table.

Look for the sequence of the analysis. Locate where variable selection, imputation, tuning and threshold choice occurred relative to the split or the resampling loop. Methods sections that describe selection and validation in separate paragraphs, in that order, usually performed them in that order.

Read the calibration. A validation calibration slope below 1 is the direct measurement of overfitting, described below. Absence of any calibration reporting is itself a finding.

Compare apparent and validated performance. If only one performance figure appears, ask which it is. A single C-statistic quoted with no statement of how it was obtained is almost always apparent.

Check whether the accuracy is plausible for the problem. Discrimination markedly better than published models for the same outcome, in a smaller sample, is a leakage signal rather than an achievement, and warrants tracing the data flow.

Calibration slope: the quantity that measures overfitting

The calibration slope is obtained by fitting the outcome in the validation data on the linear predictor from the developed model and reading the coefficient. A slope of 1 indicates predictions of the correct spread; a slope below 1 indicates predictions that are too extreme — high risks too high, low risks too low — which is the characteristic signature of an overfitted model.

Van Calster and colleagues defined the hierarchy of calibration — mean, weak, moderate and strong — in the Journal of Clinical Epidemiology in 2016, and in BMC Medicine in 2019 called calibration the Achilles heel of predictive analytics because it is the property most often omitted and the one that determines whether a model is safe to act on. Discrimination can be preserved while calibration fails completely: a model can rank patients correctly and still tell every one of them a wrong absolute risk.

Report the calibration slope and the calibration intercept from validation, alongside a flexible calibration curve. A model whose C-statistic is quoted without a calibration slope has, in practice, not been shown to be usable, and reviewers in the prediction-model literature increasingly treat the omission as a major revision rather than a presentational one.

Internal validation that actually estimates optimism

Internal validation estimates how much the developed model’s apparent performance is inflated for the population it was developed in. Done correctly, the entire modelling process — screening, selection, tuning, imputation, threshold choice — is repeated inside each resample.

Bootstrap optimism correction is the reference method for prediction models. Draw a bootstrap sample, run the whole modelling procedure on it, measure performance on the bootstrap sample and on the original data, and take the difference as optimism; average over a few hundred repetitions and subtract from the apparent performance. Steyerberg and colleagues compared internal validation procedures in the Journal of Clinical Epidemiology in 2001 and found bootstrapping efficient and stable where split-sample validation was neither.

Nested cross-validation is required whenever hyperparameters are tuned. Tuning occurs in an inner loop; performance is measured in an outer loop that the tuning never saw. Varma and Simon showed in BMC Bioinformatics in 2006 that a single cross-validation loop used for both tuning and evaluation gives a substantially biased error estimate.

Random split-sample validation is the weakest of the three and is still the most commonly reported. Holding out 30% of a modest dataset both wastes information that the model needed and produces a performance estimate with wide uncertainty, and it does not become external validation by being called a test set.

Shrinkage and penalisation, and their limits

Shrinkage deliberately pulls estimated coefficients towards zero to counteract the outward bias that fitting introduces, trading a little bias for a worthwhile reduction in variance. Van Houwelingen and Le Cessie’s heuristic uniform shrinkage factor, and the penalised approaches — ridge, lasso and elastic net — are the standard tools, and a uniform shrinkage factor estimated during development should be reported alongside the coefficients.

The honest limitation is that penalisation is not a repair for an inadequate sample. Riley and colleagues showed in the Journal of Clinical Epidemiology in 2021 that penalisation and shrinkage methods are themselves unreliable when the development sample is small: the tuning parameter is estimated with high variance, so the amount of shrinkage applied varies substantially across samples and can be badly wrong in either direction for any individual dataset. Penalisation improves expected performance across many datasets without guaranteeing improvement in yours.

A model that needs heavy shrinkage to behave is a model developed on too little data. Report the shrinkage factor as a diagnostic, not only as a remedy.

External validation answers a different question

External validation measures performance in data from a different setting, period or population, and it is not interchangeable with internal validation however the two are labelled in a manuscript.

Internal validation estimates optimism for the development population. External validation estimates transportability, and models routinely lose considerable discrimination and nearly all their calibration when moved between hospitals, countries or eras — not because the development was overfitted, but because case mix and baseline risk differ. Both can be true at once, and separating the two failure modes requires reporting calibration in the new setting rather than discrimination alone.

Wynants and colleagues’ living systematic review of COVID-19 diagnostic and prognostic models in The BMJ, first published in 2020, assessed the published models with PROBAST and rated all of them at high or unclear risk of bias, with small effective sample sizes, unclear handling of selection, and absent calibration among the recurring problems. The review is a useful calibration of how common these defects are in a literature written quickly and under pressure. For the related demand that a finding be confirmed by an independent method, see reviewer wants orthogonal validation.

What a peer reviewer says when overfitting is mishandled

Reviewer comments on overfitting are recognisable and repetitive. Anticipating the exact phrasing is among the most efficient revision work available.

“The reported performance is apparent performance; internal validation correcting for optimism is not presented.” “Variable selection was performed on the full dataset prior to cross-validation, so the reported error rate is optimistically biased.” “The number of candidate predictors is large relative to the number of events; the authors should report the effective sample size and a shrinkage factor.” “No calibration is reported; the C-statistic alone does not establish that the model is usable.” “The train–test split appears to place multiple observations from the same patient on both sides.” “The threshold was selected to maximise accuracy in the same data in which accuracy is reported.” “The authors describe this as external validation, but the validation set is a random partition of the development data.”

The comment that most often decides a prediction-model paper is the calibration one, because it cannot be answered by argument — either the slope was reported or the analysis has to be rerun. The overclaiming that accompanies an uncorrected performance figure attracts its own objection; see overclaim checking.

Contested ground: where the classical account breaks down

The classical bias–variance account predicts that performance degrades monotonically once a model has more parameters than the data can support, and modern overparameterised models falsify that prediction in its simple form.

Belkin and colleagues described double descent in PNAS in 2019: as model capacity increases past the interpolation threshold — the point at which the model fits the training data exactly — test error first rises, as expected, and then falls again, sometimes below the classical optimum. Large neural networks operate in that second regime, which is why “the model has more parameters than observations, therefore it is overfitted” is not a valid argument against a deep-learning result, though it remains a valid argument against an unpenalised regression.

Two caveats keep this from being an escape route in applied biomedical work. First, the regime depends on implicit regularisation from the optimiser and on very large datasets, conditions rarely met in a clinical study with a few hundred events. Second, double descent concerns discrimination-type error and says nothing reassuring about calibration, which is the property clinical use depends on. The safe reading is that parameter counting is a weak argument in both directions, and that empirical optimism estimation and calibration reporting are what settle the question.

How to report it

Report the effective sample size, the number of candidate parameters considered including everything discarded, and the sample-size justification actually used. Describe the modelling steps in the order performed and state explicitly which steps were repeated inside the resampling loop. Report apparent performance and optimism-corrected performance as separate numbers. Report a calibration slope, a calibration intercept and a flexible calibration plot from validation, not discrimination alone. State the shrinkage or penalty applied and how the tuning parameter was chosen.

Follow TRIPOD for the reporting structure — the 2015 statement for regression-based prediction models, and TRIPOD+AI, published in The BMJ in 2024, for models developed with machine learning. Where the model is offered as explanatory rather than predictive, the separate question of which covariates belong in it is governed by confounding, not by predictive performance.

Related

Collider bias · Reviewer says the model is not appropriate · Registry data limitations

Checked before submission by the statistics objections lane, which reports whether performance figures are apparent or optimism-corrected, whether selection and tuning sit inside the resampling loop, and whether calibration is reported at all.

Review my manuscript

Frequently asked questions

What does overfitting mean in statistics?

Overfitting means a model has fitted noise particular to its own sample as though it were signal, so performance measured on that sample overstates performance on new data. The excess is called optimism. It rises with the number of quantities estimated and falls as effective sample size increases.

What is overfitting in machine learning?

Overfitting in machine learning is a model learning sample-specific noise, giving training performance that does not transfer to new data. The mechanism is identical to the statistical case, but it enters more often through hyperparameter tuning, preprocessing applied before splitting, and non-independent training and test sets.

What causes a model to overfit?

A model overfits when the number of estimated quantities is large relative to the effective information in the data, or when decisions were made by looking at the outcome. Univariable screening, stepwise selection, hyperparameter tuning and threshold optimisation all consume degrees of freedom that final-model summaries do not show.

How can you tell if a model is overfitted?

Compare apparent performance with optimism-corrected performance and read the validation calibration slope. A slope below 1 means predictions are too extreme, the direct signature of overfitting. Fit statistics on the development data cannot detect it, because an overfitted model fits those data by construction.

What is an overfitted model?

An overfitted model is one whose fitted coefficients or learned structure encode variation particular to the development sample. Its predictions are too extreme, so the validation calibration slope falls below 1, and its measured accuracy drops when the model is applied to data it was not fitted on.

How do you prevent a model from overfitting?

Size the sample using explicit criteria rather than a rule of thumb, limit outcome-driven decisions, apply shrinkage or penalisation, and repeat every modelling step inside a bootstrap or nested cross-validation loop. Split by independent unit rather than by row, and report calibration alongside discrimination.

What is overfitting in simple terms?

Overfitting is a model memorising the peculiarities of the data it was built on instead of the pattern it was meant to find. Performance therefore looks excellent on those data and disappoints on new data. The size of that disappointment is what internal validation is designed to estimate in advance.