My p-value gets smaller every time I measure more cells

Pseudoreplication is why a p-value keeps falling as you image more fields or count more cells from the same three animals: the test scores each cell as independent, so more measurements shrink the standard error even though the biology hasn’t changed. This diagnostic signature is one of the few statistical errors detectable without running any statistics — your own degrees of freedom give it away.

The signature is named in the physiology literature. David Eisner titled his review of the problem “Pseudoreplication in physiology: More means less” (Journal of General Physiology 2021;153(2):e202012826), and his simulations show that comparing groups of animals by measuring several cells from each and running a t-test or ANOVA over the cells “can give erroneous positive results by assuming that the cells from each animal are independent of each other” — a problem he argues “may be responsible for much of the lack of reproducibility in the literature”. The underlying concept is pseudoreplication; this page is about the symptom you can watch happen on your own screen.

Why this happens

A test on 300 cells computes its standard error as though those 300 observations were independent draws. They are not: they came from three animals, and cells within an animal share that animal’s genetics, treatment, handling and environment.

Because standard error shrinks with the square root of n, doubling the cells you count shrinks it by about 30% — regardless of whether the extra cells contain any new information about the difference between groups. The p-value falls, the biology has not changed, and the only thing that increased was measurement effort.

The real number of independent observations stayed at three the whole time.

The degrees of freedom printed in your own output make the mismatch explicit. Six animals, three per group, with 50 cells each yields 300 measurements; a two-sample t-test over those 300 reports 298 degrees of freedom, while the design supports 4. Degrees of freedom are the most reliable tell available, because they are printed in the result and cannot be reconciled with a small animal count by any reading of the methods.

The quantity that governs how fast the p-value falls is the intraclass correlation coefficient, ICC or rho — the share of total variance that sits between animals rather than between cells within an animal. Kish’s design effect turns it into a penalty directly: DEFF = 1 + (m − 1)rho, where m is the number of measurements per unit. At 50 cells per animal and an ICC of 0.2, which is modest for a phenotype that differs between animals at all, DEFF is 10.8 and the naive standard error is understated by a factor of 3.3. The treatment was applied at the level ARRIVE 2.0 item 1b calls the experimental unit — “a single animal, litter, or cage of animals” — and the analysis is being run one level below it; see the ARRIVE guidelines.

This two-level formula assumes the cells within an animal are exchangeable — one shared correlation, sometimes called compound symmetry. Where the nesting runs deeper — twenty fields per section, three sections per animal — a three-level design effect applies, and it does not reduce to a single rho: only a mixed model with nested random effects, one term for animal and one for section within animal, recovers the right denominator. A single ICC estimated by pooling all cells regardless of section will understate the correction in that design.

Common questions about counting n in cells vs. animals

Why does measuring more cells make my p-value smaller?

Because a t-test or ANOVA run on individual cells treats each cell as an independent observation, so its standard error shrinks with the number of cells rather than the number of animals. Measuring more cells from the same three animals therefore lowers the p-value through measurement effort alone, with no new biological information behind it.

Is my n the cells I counted or the animals I treated?

Your n is the number of independent experimental units — usually animals, not cells, wells, or fields. ARRIVE 2.0 defines the experimental unit as the single animal, litter, or cage that received the treatment independently. Check by asking whether returning to the microscope alone could raise your n: if yes, you were counting cells.

I have 3 mice per group but 300 cells counted across them — what’s my real sample size?

Your real sample size is 3 per group, not 300. Each mouse contributes one independent data point regardless of how many cells were imaged from it; summarize each animal to a single mean (or fit a mixed model) before testing, because the 300 cells are correlated measurements of 3 biological replicates, not 300 separate ones.

The two curves: real gain plateaus, fake gain does not

More cells do improve a correct analysis, and this is the point most treatments of pseudoreplication skip. The distinction that matters is not whether the standard error falls but whether it stops falling.

In a two-level design with k animals per group and m cells per animal, write the between-animal variance as sigma-squared-b and the cell-to-cell variance within an animal as sigma-squared-w. The standard error of the group mean under the correct analysis is the square root of (sigma-squared-b + sigma-squared-w/m) divided by k. Only the second term shrinks with m. As m grows, the whole expression converges to the square root of sigma-squared-b over k — a floor set by how much the animals differ and how many of them you have. The naive cell-level standard error is the square root of (sigma-squared-b + sigma-squared-w) divided by km, and it has no floor at all: it goes to zero.

Three animals per group, ICC 0.2 (between-animal variance 1, within-animal variance 4), gives the following.

Going from 10 cells to 1,000 cells per animal buys a 15% improvement in the honest standard error and a 90% improvement in the dishonest one. The final column is the square root of Kish’s design effect, which is what makes the two calculations reconcile exactly.

The ceiling has a clean statement. As m grows without limit, one animal is worth at most 1/rho independent cells: at ICC 0.2 an animal can never be worth more than 5 independent cells however many you count, and at ICC 0.5 it can never be worth more than 2. Any statistical power calculation built on the number of cells rather than the number of animals inherits the same error and reports power that the experiment does not have.

The self-test

Ask what happens to your n if you go back to the microscope tomorrow.

If counting more cells raises your n, your n counts cells. If your n can only rise by treating another animal, running another independent culture, or repeating the experiment, then it counts independent units and you are fine.

That question takes ten seconds and settles it.

Three further checks take a few minutes each and are worth running before you decide.

Split the cells in half at random and re-run. Discard a random half of the measurements from each animal and repeat the analysis. Under a cell-level test the t statistic falls by a predictable factor of about 0.71, because halving the count multiplies the standard error by the square root of 2. Under a per-animal or mixed analysis the p-value should barely move, because the per-animal means barely move. A p-value that tracks the number of cells you happened to keep is not measuring the treatment.

Plot the per-animal means and look at the spread. If three treated animals and three control animals overlap while the cell-level test returns p < 0.001, the test is describing within-animal precision rather than between-animal difference.

Compare error bars with the visible scatter. Standard errors an order of magnitude smaller than the spread of the plotted points, or error bars narrower than the plotting symbol, almost always mean the denominator counts measurements.

What the correct analysis looks like

Summarise to the independent unit. Take a per-animal mean and test on those. Three numbers per group, not 150. The test is less impressive and it is the one that reflects your evidence. Use the median where the within-animal distribution is skewed or carries a handful of extreme cells, and say in the methods which you used.

Or fit a mixed-effects model with a random effect for animal, which uses all the cells while estimating the variance at the animal level. The interval widens compared with the naive analysis — that widening is the correction. Report enough to reproduce it: the package and version, the full model formula, the random effect structure, and how denominator degrees of freedom were obtained. In R that means lme4 with lmerTest, or nlme, and stating whether the approximation was Satterthwaite or Kenward–Roger; Kenward–Roger is the more conservative and is the usual choice when the number of animals is small. In SAS the equivalent is PROC MIXED with DDFM=KR, and in Stata mixed. “A mixed model was used” is not a reproducible method and reviewers say so.

A near-miss that does not fix it: adding “animal” as a fixed effect. Blocking on animal in a two-way ANOVA changes which sum of squares the treatment is tested against, but a fixed effect estimates only the means of the specific animals sampled — it does not treat between-animal variance as a source of error to be averaged over, the way a random effect does. The output can look identical to a properly specified mixed model while answering a narrower question. Where the scientific interest is the population-averaged treatment effect rather than the per-animal one, generalized estimating equations (GEE) are the standard marginal-model alternative to a mixed model, and they require only that the correlation structure within animal be approximately correct, not the full distributional assumptions a mixed model makes.

Show both levels in the figure. Plot the individual cells lightly with the per-animal means overlaid as larger points. It makes the structure of the data visible to a reader immediately, and it is increasingly expected in cell biology figures. The published recipe is the SuperPlot of Lord, Velle, Mullins and Fritz-Laylin (Journal of Cell Biology 2020;219:e202001064), which colour-codes each independent experiment so the reader can see which cells came from which repeat. State in the legend which level the error bars and the p-value were computed at: a SuperPlot drawn correctly and analysed at the cell level is still pseudoreplicated.

Questions about fixing the analysis

What does a “singular fit” warning from lme4 mean, and can I ignore it?

A singular fit means lme4 estimated the between-animal variance component at exactly zero, usually because too few independent units exist to identify it reliably — three animals per group is a common trigger. Report the singular fit by name in your methods rather than deleting the random effect to silence the warning, and fall back to a per-animal summary analysis instead.

My mixed model won’t converge with only 3 animals per group — what do I do?

With only 3 animals per group, lme4 frequently cannot estimate a random effect for animal and returns a singular or non-converging fit — this is expected, not a coding error. Name the singular fit in your methods, then fall back to the analysis that always works: a per-animal mean per group, tested with three numbers on each side.

Is n=3 biological replicates actually enough for a t-test?

Three independent biological replicates per group is the minimum a standard two-sample t-test can run on, and it is common in cell and mouse work. It is also usually underpowered: correctly analysed at the animal level, the design can still miss a real, modest effect. Fixing pseudoreplication corrects the p-value; it does not fix a sample size that is too small.

How many cells per animal are actually worth measuring

Stop adding cells once the within-animal term stops mattering, and the threshold is computable rather than a matter of taste. The variance of a per-animal mean comes within 10% of its floor once m reaches 10 × (1 − rho) / rho cells: 90 cells at ICC 0.1, 40 at ICC 0.2, and 10 at ICC 0.5. Beyond that point, additional cells buy precision on a quantity that is no longer the bottleneck.

When the total effort available is fixed and one more animal is a bigger undertaking than one more measurement, the two-level nested design has a closed-form optimum. The number of cells per animal that minimises the variance of the group mean for a given total effort is the square root of (effort per additional animal ÷ effort per additional cell) × (within-animal variance ÷ between-animal variance). If an extra mouse is 20 times the work of an extra cell and the ICC is 0.2, that comes to about 9 cells per animal, and every unit of effort after that is better spent on animals.

This is the arithmetic that makes “we have added more cells” the wrong answer to a reviewer. Adding six animals at 10 cells each moves the honest standard error; adding 500 cells to the animals you already have does not. It also explains why the effort feels productive: the number on the screen genuinely improves, and it improves in the one analysis that is not entitled to it.

Questions about how many cells are worth measuring

Does measuring more cells ever genuinely help, or should I stop counting entirely?

Yes, up to a computable point. Extra cells sharpen each animal’s own mean, and the variance of a per-animal mean comes within 10% of its floor at roughly 10 × (1 − ICC) / ICC cells — 40 cells at an ICC of 0.2. Beyond that point, effort is better spent on more animals, not more cells.

Does my p-value shrink the same way with patients instead of cells?

Yes. Patients within clinics, measurements within patients, and eyes within people share the identical structure as cells within animals: observations clustered inside a unit that received the treatment. The p-value falls the same way as you add clustered measurements, and the remedy is the same — analyse at the cluster level or model the cluster as a random effect.

When the shrinking p-value is also a stopping rule

If you stopped counting cells at the point the p-value crossed 0.05, you have committed a second, separate error on top of the first. Counting until significance is reached is optional stopping, and it inflates the type I error rate independently of any nesting; see p-hacking.

The two compound in a way worth stating plainly. Because the cell-level standard error has no floor, the cell-level t statistic diverges as you keep measuring — unless the animals you happened to sample have exactly equal true means, which they do not. Given enough cells, a cell-level test on nested data will reach any significance threshold you name, for any set of animals, whether or not the treatment did anything. The test has quietly stopped asking whether the treatment works and started asking whether these six particular animals differ, and the answer to that question is essentially always yes.

The practical consequence: a stopping rule based on the p-value is only recoverable if you say so. Pre-specify the number of cells per animal before you start counting, apply it to every animal in both groups, and report it. Where the count was not pre-specified, state the actual counting procedure in the methods rather than reporting a round number that was arrived at retrospectively.

This is not an argument against stopping early in general. Group-sequential clinical trial designs stop early on purpose, using a pre-registered alpha-spending function — O’Brien–Fleming or Pocock boundaries are the named examples — that raises the significance threshold at each interim look so the total type I error stays at the nominal level. What makes the cell-counting version an error is not the act of stopping; it is stopping at an unregistered, uncorrected threshold, with no boundary function and no count of how many looks were taken.

What to expect when you fix it

Frequently the effect survives with a wider interval, and the paper is fine. Sometimes it does not, and that is genuinely useful to learn before a reviewer does.

If it does not survive, the honest options are to report it with the effect size and interval as a non-significant trend, to reframe as preliminary and say what a properly powered design would look like, or to add independent units. What you cannot do is keep the 300-cell p-value.

Report the effect size with its confidence interval at the animal level, because that interval is what bounds the claim the experiment can support. Correcting the level of analysis frequently converts a significance problem into a sample size problem — three animals per group was always three animals per group, and the cell count was concealing it rather than fixing it.

Update every number that changed, in the same revision: abstract, results text, figure legends, supplementary tables. A manuscript that adds a mixed model to the supplement while the abstract retains the cell-level p-value is the specific failure reviewers check for on re-review.

Where else this appears

Pseudoreplication appears under many names beyond cells within animals: wells split from one culture, sections from one block, images from one slide, two tumours in one mouse, repeated measurements on one subject, patients within a clinic, students within a school. Any time measurements cluster inside a unit that received the treatment, the cluster — not the measurement — is the experimental unit.

Pooling replicate experiments and testing on the combined observations is the same error in a form people rarely notice, because between-experiment variation is usually the largest source of variability and pooling erases it entirely.

Image analysis is where the counts grow fastest, because the levels multiply: twenty fields per section, three sections per animal, four animals, and the legend reads n = 240 — see image quantification reporting. Pooled repeats are also the form most likely to be a disguised batch effect, since the day of the experiment carries reagent lot, passage number and operator with it; when the repeats visibly disagree rather than quietly inflating n, see my replicates disagree.

Single-cell sequencing is the modern extreme case. Thousands of cells from one donor are thousands of measurements of one unit, which is why aggregating to a per-donor pseudobulk profile before differential testing is now the standard analysis rather than a conservative option.

When the obvious remedy is unavailable

Summarising to the animal fails in four recognisable situations, and each has an accepted alternative.

Too few units to estimate a variance component. Three independent experiments will not support a well-identified random effect, and lme4 frequently returns a singular fit with the between-unit variance estimated at exactly zero. Report the singular fit rather than deleting the random effect to make the warning go away, and fall back to the summarised analysis on three values, naming the model that failed.

The outcome is a proportion or a count. Averaging binary per-cell calls within each animal and testing the proportions is transparent and usually acceptable. A generalized linear mixed model with a binomial or negative binomial family is the alternative when the per-animal denominators differ widely.

The question really is at the cell level. How a property is distributed across cells within one individual is a within-unit question, and cells are the right unit for it. What that design cannot support is a between-group claim, so keep the two analyses and their two sentences separate in the text.

The grouping was never recorded. Where the experiment of origin or the animal of origin was not tracked, the nesting cannot be reconstructed after the fact. Say so, analyse at the level you can defend, and state the limitation as a specification of what the experiment establishes rather than as an apology — see how to write a limitations section.

Too few units even to trust a singular fit. When k is very small — two or three per group — even the “report the singular fit and fall back to the summary” advice above stretches thin, because a REML variance estimate at that k is barely identified regardless of the outcome. Some groups instead fit a Bayesian hierarchical model with a weakly informative prior on the between-animal variance (in brms or Stan), which regularises the estimate away from the boundary rather than pinning it at zero; this needs the prior and its sensitivity stated in the methods, since the choice of prior is now doing some of the work the data cannot.

What a reviewer says

Reviewers rarely write “your p-value is shrinking because you measured more cells”. They write the specific version, and these are the forms it takes.

“The degrees of freedom reported in Table 2 correspond to the number of cells rather than the number of animals.” “n differs between panels of Figure 3 in a way that appears to track imaging effort rather than the number of animals.” “Please state how many cells were analysed per animal and whether that number was fixed in advance.” “The error bars are inconsistent with the visible spread of the plotted points.” “Statistics appear to have been computed on individual cells; please analyse per-animal values or fit a model with a random effect for animal.”

The first of those is the one to anticipate, because it is checkable from the printed result alone and needs no access to your data. Answering any of them takes three moves: name the experimental unit, re-run the analysis at that level, and update every downstream number. Do not argue that the effect is visually obvious at the cell level, and do not present both analyses and invite the reader to choose — see how to write a response to reviewers.

Frequently asked questions

If the cell-level p-value is wrong, should I stop showing individual cells?

No — showing them is good practice, not the error. The error is testing on them. Plot the cells lightly and overlay the per-animal means as larger points, then run the statistics on the means or on a mixed model — the SuperPlot recipe (Journal of Cell Biology 2020;219:e202001064) does exactly this and states which level the p-value used.

If I keep adding cells, will my result always become significant eventually?

Effectively yes, and that is itself evidence the analysis is wrong. The cell-level standard error has no floor, so its test statistic grows without limit as you add cells — unless the animals you happened to sample have exactly equal true means, which real animals never do. Enough cells will cross any threshold you name, regardless of the treatment.

What if my effect disappears once I analyse it correctly?

Report it honestly, with the effect size and its confidence interval at the animal level. The honest options are to present it as a non-significant trend, reframe the finding as preliminary and describe what a properly powered design would need, or add independent animals. Finding this yourself, before submission, is far better than a reviewer finding it after.

Before a reviewer finds it

The pseudoreplication agent reads every figure legend, determines what each stated n counts, reconstructs the physical structure of the experiment from the methods, and reports each place where the n exceeds the number of independent units — with the remedy named. It is the single most valuable pre-submission check for in vivo and cell biology work, because this is the error that changes p-values by orders of magnitude rather than at the margin.

If a reviewer has already raised it, see reviewer says my n is not independent.

Review my manuscript