Research methods concepts, explained plainly

Research methods concepts are the named failure modes of study design, measurement and inference, and every term here names a specific way a study can produce a confident wrong answer. Each page states what the concept is, how it arises, how to detect it in your own manuscript, and what a reviewer will say if you miss it.

Fifteen concepts are grouped below into three families: causal structure, time and selection, and inference and error. The grouping is by mechanism rather than by field, because the same structure arrives in a mouse experiment, a registry analysis and a randomised trial under three different names, and the remedy follows the mechanism rather than the name it was given locally.

What every concept on this page has in common

Every concept on this page describes a study that returns a wrong number and shows no sign of it. The estimate carries an ordinary confidence interval, the model returns ordinary diagnostics, and nothing in the output separates it from a correct result. That is what makes these fifteen worth learning as a set rather than looking up one at a time.

Three properties recur. The defect is created at design and found at review, so no choice made during analysis can undo it. A larger sample makes it worse in the only sense that matters, narrowing the interval around a wrong value. And the direction of the error is usually knowable in advance from the structure alone, which is why a limitations paragraph that names the direction and the likely magnitude reads as an analysis, while one that asks readers to interpret the findings with caution reads as an admission.

The magnitude is not hypothetical. Samy Suissa followed 979 Saskatchewan residents for a year after a first hospitalisation for chronic obstructive pulmonary disease and analysed the same inhaled-corticosteroid data twice. The time-fixed analysis used by the cohort studies of the day returned an adjusted rate ratio of 0.69 (95% CI 0.55–0.86); a time-dependent analysis of the identical data returned 1.00 (95% CI 0.79–1.26) (American Journal of Respiratory and Critical Care Medicine 2003;168:49–53). No covariate was added and no patient was removed; only the allocation of person-time before the first dispensing changed. A protective effect became a null effect because of a clock.

Common questions about spotting a hidden methods problem

My statistics are clean and the model fits. Could one of these still apply?

Yes — that is the defining property of this set of concepts: a study can carry a wrong answer with no diagnostic sign of it. The interval is ordinary and the model converges, because none of these problems is a fit problem — each is a design decision made before analysis. A clean fit means the model matches the data, never that the data measure what you intended.

A reviewer named a concept I do not recognise. Where do I start?

Start with the concept’s own page, then read its detection section against your methods before drafting a reply. Research methods concepts are named structures in design, measurement and inference — confounding, selection, immortal time, multiplicity, pseudoreplication among them — and each implies a check and a remedy. The reviewer’s comment is almost always about one sentence in your manuscript, and finding that sentence matters more than the definition.

Causal structure

Causal-structure concepts decide what a variable is doing in the causal diagram, and therefore whether adjusting for it removes bias, creates bias, or deletes part of the effect being estimated. The three roles — common cause, intermediate, common effect — are not properties of the variable but of its position relative to the exposure and the outcome in the specific question being asked.

• Confounding — a common cause of exposure and outcome
• Collider bias — the bias adjustment creates rather than removes
• Mediator versus confounder — the distinction that decides your adjustment set
• Table 2 fallacy — why every coefficient in one model cannot be causal
• Simpson’s paradox — when every subgroup disagrees with the total

The practical consequence is that “we adjusted for potential confounders” is not a statement about validity until the adjustment set is justified variable by variable. A directed acyclic graph is the standard instrument for that justification, because it forces each variable’s role to be declared before the model is fitted rather than defended after a reviewer asks. Westreich and Greenland set out the reporting consequence in the American Journal of Epidemiology (2013;177:292–298): an adjustment set chosen to estimate one exposure’s effect does not license causal interpretation of the other coefficients in the same table, because each of those covariates has its own confounding structure that the model was never built to handle.

Adjusting for a mediator is not always an error — it is the definition of estimating a direct rather than a total effect, and mediation analysis treats the two as distinct estimands with their own assumptions. The catch is a second, harder condition: the direct-effect estimate is only unbiased if nothing confounds the mediator-outcome relationship while itself being affected by the exposure, because that configuration reintroduces collider bias inside the adjustment set. A manuscript that reports a “direct effect” after conditioning on a mediator, without addressing that second confounding structure, has usually not answered the question it thinks it answered. Measurement error compounds the ordinary case too: a confounder measured with noise — self-reported pack-years, a diagnosis code standing in for disease severity — leaves residual confounding behind even after it is entered in the model, because the noisy version does not fully block the backdoor path. “Controlled for” and “causally sufficient adjustment” are not the same claim.

Simpson’s paradox is the case where the structural question has a genuine answer and the textbook retelling skips it. Bickel, Hammel and O’Connell reported in Science in 1975 that roughly 44% of 8,442 male applicants and roughly 35% of 4,321 female applicants were admitted to Berkeley’s 1973 graduate cohort, with the disparity largely vanishing department by department. Whether the stratified estimate or the pooled one is correct depends entirely on whether department is a common cause of applicant sex and admission, or an intermediate on the path being studied.

Questions about confounding and adjustment

Why did my p-value change when I added a covariate?

Three explanations exist, and your causal diagram decides which applies. If the covariate is a confounder, the change is bias being removed — expected and correct. If it is a mediator, part of the true effect has just been deleted, not corrected. If it is a collider, you have manufactured an association that was not there. The p-value’s size never tells you which happened; the diagram does.

Do I need to adjust for every variable that’s correlated with my outcome?

No — a variable earns its place in the adjustment set by causal role, not by correlation with the outcome. Include a common cause of exposure and outcome; exclude a mediator, which sits on the path being estimated; exclude a common effect of the two, a collider. Draw the DAG first and classify each candidate, rather than adding anything that moves the estimate.

Time and selection

Time-and-selection concepts decide the answer through the study’s boundaries — who entered, who stayed, and when the clock started — before any variable is measured. These are the biases that adjustment does not touch, because the variable that would fix them takes the same value for everyone in the dataset.

• Immortal time bias — exposure defined by surviving long enough to receive it
• Lead-time bias — earlier diagnosis lengthening measured survival without delaying death
• Length-time bias — screening preferentially finding slow disease
• Overdiagnosis — detecting disease that would never have caused harm
• Selection bias — when who is in the study depends on the answer
• Regression to the mean — why extreme groups improve without treatment

Selection has a single structural definition, given by Hernán, Hernández-Díaz and Robins in “A structural approach to selection bias” (Epidemiology 2004;15:615–625): the study conditions on a common effect of two variables, one on the exposure side and one on the outcome side. That definition unifies Berkson’s bias, the healthy worker effect, differential loss to follow-up and non-response into one question you can ask of any manuscript, and it explains why the remedy is a design change or a bound rather than a longer covariate list.

The three screening concepts form a family that a stage-adjusted analysis cannot separate. Lead-time bias concerns when a case is found, length-time bias concerns which cases are found, and overdiagnosis concerns cases that never needed finding at all; a screening study that reports improved five-year survival without reporting disease-specific mortality has answered none of the three. Immortal time is arithmetic rather than sociology: person-time that carries zero events by construction is added to the exposed denominator, so the exposed rate falls. Registry and electronic health record analyses concentrate all of these at once, which is why registry data limitations is worth reading alongside them.

Regression to the mean has a condition that a simple before-and-after comparison usually skips: separating it from a real treatment effect needs a comparison group selected on the same extreme baseline value and left untreated, because the expected reversion depends only on how imperfectly the two measurements correlate, not on whether anything happened in between. A single extreme group, measured twice, cannot distinguish the two explanations no matter how large the observed change looks. Disease-specific mortality is not immune to bias either, which is part of why the three screening concepts resist a single fix: how a death gets classified can be influenced by a patient’s known screening or diagnosis history, so some screening trials also report all-cause mortality as a check, even though it has less power to detect a true effect.

Questions about immortal time bias, selection and sample size

Reviewer 2 says my sample size is too small — what do I actually write back?

Write back with the justification you actually used, not a bigger number. State whether the sample size came from an a priori power calculation, from the precision (confidence-interval width) you needed, or from a hard constraint such as cohort or specimen availability, and say so explicitly. A study can be adequately sized for its own question and still draw this comment when the manuscript never states how n was chosen.

The reviewer said the design is biased but did not name the bias. What is it likely to be?

Confounding, selection bias and time-related biases such as immortal time explain most misleading observational results, so an unnamed objection usually points at one of the three. Ask whether a common cause existed before the study began, whether the study conditioned on a common effect, or whether exposure status used information from after time zero. Only the first is fixed by adjustment — the other two are the more dangerous readings.

My before-and-after results improved most in the patients who started worst. Is that just regression to the mean?

Usually yes, if the worse-off group was selected because it scored at an extreme on the very measure later re-measured. Regression to the mean needs only that the two measurements correlate imperfectly, not that any treatment occurred in between, and the expected reversion grows as that correlation weakens. Ruling it out needs an untreated comparison group selected on the same extreme baseline, not a larger before-after change alone.

Inference and error

Inference-and-error concepts decide what repeated testing, non-independent observations and underpowered designs do to a p-value that still prints as significant. These are the failures that live in the statistics section rather than the methods section, and they are the most detectable of the three families from the manuscript alone.

• Multiple testing — and when correction is required
• False discovery rate — versus family-wise error control
• Statistical power — and why post hoc power is circular
• Pseudoreplication — counting measurements as independent observations

Benjamini and Hochberg defined the false discovery rate in 1995 (Journal of the Royal Statistical Society B) as E[V/R], the expected proportion of rejected hypotheses that are true nulls. Choosing between that and family-wise error control is a claim-level decision, not a software default: control the family-wise rate when any single reported finding must stand on its own, and the false discovery rate when the claim is about a list. Genomics conventions and confirmatory trial conventions differ for exactly that reason, and the choice is the substance of multiple testing in omics and of the reviewer comment collected at reviewer: multiple comparisons.

The Benjamini-Hochberg procedure carries a condition worth stating rather than assuming: it controls the false discovery rate exactly under independence or positive regression dependence among the tests, which holds for most standard genomics designs but not automatically for every correlation structure. Benjamini and Yekutieli’s extension (Annals of Statistics, 2001) controls the rate under arbitrary dependence by trading power for a more conservative threshold, and is the citation a reviewer expects when test statistics are correlated by design — repeated measures on the same samples, or overlapping gene sets.

Power fails in two directions. Button and colleagues estimated a median statistical power of 21% across 49 meta-analyses covering 730 studies in neuroscience (Nature Reviews Neuroscience, 2013), which sharply reduces the probability that any one significant finding in such a literature reflects a true effect, before flexible analysis is considered. Power computed after the fact from the observed effect is worse than uninformative: Hoenig and Heisey showed in The American Statistician (2001) that observed power is a monotone transformation of the p-value, so a non-significant result must yield low observed power by arithmetic and cannot be evidence about itself.

Low power distorts more than the chance of a false negative. Gelman and Carlin’s “Type M” and “Type S” error framework (Perspectives on Psychological Science, 2014) shows that an underpowered study which does clear the significance threshold produces an effect-size estimate inflated on average, and can even recover the wrong sign, because only the inflated draws from a noisy sampling distribution cross the threshold at all. A single significant, underpowered result is therefore not just a possible false positive: conditional on being significant, its magnitude is expected to be an overstatement.

Pseudoreplication is the easiest of the four to see. Stuart Hurlbert introduced the term in Ecological Monographs (1984;54:187–211) in a survey where pseudoreplication occurred in 27% of the studies examined, and in 48% of those that applied inferential statistics at all; Lazic later read a single issue of Nature Neuroscience and classified 12% of its papers as pseudoreplicated with a further 36% suspected (BMC Neuroscience 2010;11:5). If your n counts wells, cells, sections or images rather than animals, litters or independent cultures, the objection at reviewer: n is not independent is the one you will receive.

Frequently asked questions about multiple testing and small samples

Is n=3 enough for an RNA-seq experiment, or will reviewers reject it outright?

Three biological replicates per condition is the practical floor most differential-expression tools need to estimate per-gene variance at all — it is a minimum for the statistics to run, not evidence of adequate power. A reviewer is more likely to ask for the dispersion estimates, the effect sizes actually detectable at that n, and whether replicates are biological rather than technical, than to demand a fourth replicate outright.

Do I have to correct for multiple testing if my hypotheses were pre-specified?

Pre-specification narrows the problem but does not remove the need for correction across every pre-specified endpoint tested at the same significance level. The distinction that matters is primary versus secondary: a single pre-specified primary endpoint is usually reported unadjusted, while a family of secondary or exploratory endpoints still inflates the false-positive rate and still needs multiple-testing correction or explicit labelling as exploratory.

Everything here comes from epidemiology. Does any of it apply in my field?

Pseudoreplication was introduced in ecology by Hurlbert in 1984 and is now discussed across neuroscience and cell and molecular biology; multiplicity governs genomics; regression to the mean applies to any before-and-after comparison in a group selected on an extreme baseline. The structures are field-independent even where the vocabulary is not, which is why the family a concept belongs to matters more than the label a reviewer reached for.

Can these problems be seen in a finished manuscript, or do you need the data?

Many are visible in the text alone. Undefined units of analysis, missing correction, adjustment sets containing a mediator, and time-related exposure definitions all show up in the methods section without access to the underlying data. Two things do require more than the text: whether an unmeasured variable exists, and whether the data are what the authors say they are — a quantified bound is the right response to the first.

The three questions that locate most methods objections

Three questions locate most methods objections before a reviewer asks them, and each maps onto one family above.

What is the unit of analysis, and how many independent ones are there? Count the units that were randomised, housed, treated or sampled independently, then compare that count with the n in the statistical test. A gap between the two is pseudoreplication, and it is the reason a p-value shrinks when more cells are counted while nothing about the biology changes.

What is each covariate doing in the causal diagram? Take the adjustment set one variable at a time and say whether it causes the exposure, is caused by it, or is caused by both it and the outcome. A variable in the third category is a collider, and adjusting for it manufactures an association — which is also the usual explanation when an effect disappears after adjusting.

When did follow-up start, and was everyone eligible at that moment? Write down time zero for each arm and check that exposure status was determined using information available then. Any criterion requiring survival, adherence or a second visit uses future information to define entry, which creates immortal time on one side and selection on the other.

A fourth question applies whenever more than one hypothesis was tested: how many comparisons were made in total, including those that produced nothing? The denominator, not the correction procedure, is the part reviewers query first, and an undisclosed denominator is the operational definition of p-hacking.

Concepts that are routinely confused with each other

Eight pairs on this page are confused often enough that the confusion has its own signature in review, and each pair separates on a single question.

The last row is the one that reaches print most often, because the two look identical in the output and opposite in the design. The usual presenting symptom is replicates that disagree with each other, and the distinction between technical and biological replication is where the answer starts.

A worked example: one registry cohort, five concepts

Consider a retrospective cohort assembled from a national registry, comparing patients who started a drug after discharge from hospital with patients who did not, and reporting a 31% lower mortality rate in the treated group across 28 secondary outcomes.

Time zero is the first defect. If exposure is defined by having filled a prescription within 90 days of discharge, then every treated patient survived to fill it, and the survival between discharge and the fill has been credited to the drug. Reallocating that person-time is the standard remedy, and a landmark analysis with a pre-specified landmark is the standard implementation.

Selection is the second. Registry membership requires a healthcare encounter, and the comparator “everyone else in the database” is a group defined by having generated encounters — itself a function of both illness and the exposure. This is a structural problem with the comparator rather than a covariate problem, so propensity score matching will balance the measured variables without touching it.

Multiplicity is the third: 28 secondary outcomes tested at the conventional 0.05 threshold produce roughly 1.4 false positives under the complete null, and reporting the two that reached significance without the denominator is not a reporting omission but a change of claim. The fourth is the Table 2 fallacy — the covariate coefficients in the mortality model will be read as effects of age, comorbidity and deprivation by at least one reader, and by at least one reviewer as evidence that the authors did not distinguish an adjustment set from a set of causal estimates.

A fifth issue rides along whenever the outcome is mortality in an older or sicker cohort: if patients can die of causes unrelated to the drug before the outcome of interest occurs, treating those deaths as ordinary censoring in a Kaplan-Meier curve overestimates the cumulative incidence of the outcome under study. Fine and Gray’s subdistribution hazard model (Journal of the American Statistical Association, 1999) is the standard remedy once competing risks are structurally present, as they typically are in an older or multimorbid post-discharge cohort.

None of these five are visible in the effect estimate, the confidence interval or any goodness-of-fit statistic. All five are visible in the methods section.

What is detectable from the manuscript alone

Most research methods concepts are detectable in a finished manuscript with no access to the underlying data, because each leaves a specific sentence in the methods section — present, or conspicuously absent. Undefined units of analysis, missing correction, adjustment sets containing a mediator, and time-related exposure definitions are all visible in the text alone, because each is a sentence the authors either wrote or did not write.

The reporting checklists are the fastest route to those sentences, and each names the item to quote. STROBE, a 22-item guideline for observational studies published in October 2007, asks at item 13 for numbers at every stage from potentially eligible to analysed, which is how denominators that do not reconcile become visible — see the STROBE checklist. CONSORT 2025, published on 14 April 2025 with 30 items, up from 25 in CONSORT 2010, asks at item 21d which analyses were pre-specified rather than post hoc, and at item 21b who is included in each analysis and in which group — the two items that decide whether intention to treat was actually applied. PRISMA 2020, 27 items across seven sections, asks at item 14 for the reporting-bias assessment that publication bias requires. ARRIVE 2.0, 21 items split into an Essential 10 and a Recommended Set of 11, asks at sub-item 3c for the exact value of n in each experimental group for each analysis, which is the single most efficient pseudoreplication check available to a reader.

Two things are not detectable from the text: whether an unmeasured variable exists, and whether the data are what the authors say they are. A bound is the appropriate response to the first, and a bounded, quantified limitation reads as an analysis where an acknowledgement reads as an admission.

What reviewers write when these concepts are missed

Reviewers evaluating a manuscript rarely name the research methods concept behind an objection; they write its specific, manuscript-level version instead, and these are the sentences that decide papers.

“The number of animals is not stated per group; the n appears to be the number of images.” “Follow-up begins at diagnosis for the unexposed group and at treatment initiation for the exposed group.” “The adjustment set includes variables on the causal pathway between exposure and outcome.” “Twenty-eight outcomes are analysed and two are reported as significant, with no adjustment for multiplicity and no statement of the total number of comparisons.” “The improvement in the high-baseline group is consistent with regression to the mean.” “The abstract reports five-year survival; disease-specific mortality is not reported.” “The coefficients in Table 2 are discussed as if each were a causal effect.” “Controls do not appear to be drawn from the source population that gave rise to the cases.”

If you arrived here holding one of those sentences, the objection is already located: reviewer objections on statistics and reviewer objections on methods are organised by what the reviewer wrote rather than by what the concept is called.

What this collection covers, and what sits next to it

This collection of research methods concepts covers the design and inference structures that make an estimate wrong in an original research article, not how the result is written up. Reporting format is handled by the checklist pages above, and writing tasks such as drafting a cover letter or a response to reviewers sit outside this collection entirely.

Two adjacent families are worth naming because they are frequently filed here by mistake. Measurement and modelling artefacts — batch effects, overfitting — degrade an estimate through the data-generating and model-fitting process rather than through causal structure, and their remedies are design randomisation and held-out validation rather than adjustment. Reporting and interpretation quantities — effect size, confidence intervals — describe an estimate that is already either structurally sound or structurally wrong; a narrow interval reports precision, never validity.

How to use this

Each page is written to be read in two minutes by someone who has just encountered the term in a review, and to be precise enough to act on. Where a concept has a check that catches it before submission, the page names it.

If you arrived from a reviewer comment rather than from a term, reviewer objections is organised by what the reviewer said. If you arrived from something odd in your own results, analysis symptoms is organised by what you observed.

If you are drafting rather than responding, read the three questions above against your own methods section before the first submission, and read the family that matches your design in full: causal structure for observational comparisons, time and selection for anything with follow-up or screening, inference and error for anything with more than one test. The same structures recur across fields under different local names, so the family a concept belongs to matters more than the label a reviewer reached for.

PerfectPaper reads a manuscript for these structures directly — the unit of analysis against the stated n, the adjustment set against each variable’s causal role, time zero against the eligibility criteria, and the number of comparisons against the correction applied — and reports which of them the text does not resolve.

Review my manuscript