My p-value gets smaller every time I measure more cells

If you can make a result more significant by imaging more fields from the same slides, or counting more cells from the same three animals, without collecting anything new, the p-value is not measuring what you think. You have found the diagnostic signature of pseudoreplication — one of the few statistical errors you can detect without doing any statistics.

The signature is named in the physiology literature. David Eisner titled his review of the problem “Pseudoreplication in physiology: More means less” (Journal of General Physiology 2021;153(2):e202012826), and his simulations show that comparing groups of animals by measuring several cells from each and running a t-test or ANOVA over the cells “can give erroneous positive results by assuming that the cells from each animal are independent of each other” — a problem he argues “may be responsible for much of the lack of reproducibility in the literature”. The underlying concept is pseudoreplication; this page is about the symptom you can watch happen on your own screen.

Why this happens

A test on 300 cells computes its standard error as though those 300 observations were independent draws. They are not: they came from three animals, and cells within an animal share that animal’s genetics, treatment, handling and environment.

Because standard error shrinks with the square root of n, doubling the cells you count shrinks it by about 30% — regardless of whether the extra cells contain any new information about the difference between groups. The p-value falls, the biology has not changed, and the only thing that increased was measurement effort.

The real number of independent observations stayed at three the whole time.

The degrees of freedom printed in your own output make the mismatch explicit. Six animals, three per group, with 50 cells each yields 300 measurements; a two-sample t-test over those 300 reports 298 degrees of freedom, while the design supports 4. Degrees of freedom are the most reliable tell available, because they are printed in the result and cannot be reconciled with a small animal count by any reading of the methods.

The quantity that governs how fast the p-value falls is the intraclass correlation coefficient, ICC or rho — the share of total variance that sits between animals rather than between cells within an animal. Kish’s design effect turns it into a penalty directly: DEFF = 1 + (m − 1)rho, where m is the number of measurements per unit. At 50 cells per animal and an ICC of 0.2, which is modest for a phenotype that differs between animals at all, DEFF is 10.8 and the naive standard error is understated by a factor of 3.3. The treatment was applied at the level ARRIVE 2.0 item 1b calls the experimental unit — “a single animal, litter, or cage of animals” — and the analysis is being run one level below it; see the ARRIVE guidelines.

The two curves: real gain plateaus, fake gain does not

More cells do improve a correct analysis, and this is the point most treatments of pseudoreplication skip. The distinction that matters is not whether the standard error falls but whether it stops falling.

In a two-level design with k animals per group and m cells per animal, write the between-animal variance as sigma-squared-b and the cell-to-cell variance within an animal as sigma-squared-w. The standard error of the group mean under the correct analysis is the square root of (sigma-squared-b + sigma-squared-w/m) divided by k. Only the second term shrinks with m. As m grows, the whole expression converges to the square root of sigma-squared-b over k — a floor set by how much the animals differ and how many of them you have. The naive cell-level standard error is the square root of (sigma-squared-b + sigma-squared-w) divided by km, and it has no floor at all: it goes to zero.

Three animals per group, ICC 0.2 (between-animal variance 1, within-animal variance 4), gives the following.

Going from 10 cells to 1,000 cells per animal buys a 15% improvement in the honest standard error and a 90% improvement in the dishonest one. The final column is the square root of Kish’s design effect, which is what makes the two calculations reconcile exactly.

The ceiling has a clean statement. As m grows without limit, one animal is worth at most 1/rho independent cells: at ICC 0.2 an animal can never be worth more than 5 independent cells however many you count, and at ICC 0.5 it can never be worth more than 2. Any statistical power calculation built on the number of cells rather than the number of animals inherits the same error and reports power that the experiment does not have.

The self-test

Ask what happens to your n if you go back to the microscope tomorrow.

If counting more cells raises your n, your n counts cells. If your n can only rise by treating another animal, running another independent culture, or repeating the experiment, then it counts independent units and you are fine.

That question takes ten seconds and settles it.

Three further checks take a few minutes each and are worth running before you decide.

Split the cells in half at random and re-run. Discard a random half of the measurements from each animal and repeat the analysis. Under a cell-level test the t statistic falls by a predictable factor of about 0.71, because halving the count multiplies the standard error by the square root of 2. Under a per-animal or mixed analysis the p-value should barely move, because the per-animal means barely move. A p-value that tracks the number of cells you happened to keep is not measuring the treatment.

Plot the per-animal means and look at the spread. If three treated animals and three control animals overlap while the cell-level test returns p < 0.001, the test is describing within-animal precision rather than between-animal difference.

Compare error bars with the visible scatter. Standard errors an order of magnitude smaller than the spread of the plotted points, or error bars narrower than the plotting symbol, almost always mean the denominator counts measurements.

What the correct analysis looks like

Summarise to the independent unit. Take a per-animal mean and test on those. Three numbers per group, not 150. The test is less impressive and it is the one that reflects your evidence. Use the median where the within-animal distribution is skewed or carries a handful of extreme cells, and say in the methods which you used.

Or fit a mixed-effects model with a random effect for animal, which uses all the cells while estimating the variance at the animal level. The interval widens compared with the naive analysis — that widening is the correction. Report enough to reproduce it: the package and version, the full model formula, the random effect structure, and how denominator degrees of freedom were obtained. In R that means lme4 with lmerTest, or nlme, and stating whether the approximation was Satterthwaite or Kenward–Roger; Kenward–Roger is the more conservative and is the usual choice when the number of animals is small. In SAS the equivalent is PROC MIXED with DDFM=KR, and in Stata mixed. “A mixed model was used” is not a reproducible method and reviewers say so.

Show both levels in the figure. Plot the individual cells lightly with the per-animal means overlaid as larger points. It makes the structure of the data visible to a reader immediately, and it is increasingly expected in cell biology figures. The published recipe is the SuperPlot of Lord, Velle, Mullins and Fritz-Laylin (Journal of Cell Biology 2020;219:e202001064), which colour-codes each independent experiment so the reader can see which cells came from which repeat. State in the legend which level the error bars and the p-value were computed at: a SuperPlot drawn correctly and analysed at the cell level is still pseudoreplicated.

How many cells per animal are actually worth measuring

Stop adding cells once the within-animal term stops mattering, and the threshold is computable rather than a matter of taste. The variance of a per-animal mean comes within 10% of its floor once m reaches 10 × (1 − rho) / rho cells: 90 cells at ICC 0.1, 40 at ICC 0.2, and 10 at ICC 0.5. Beyond that point, additional cells buy precision on a quantity that is no longer the bottleneck.

When the total effort available is fixed and one more animal is a bigger undertaking than one more measurement, the two-level nested design has a closed-form optimum. The number of cells per animal that minimises the variance of the group mean for a given total effort is the square root of (effort per additional animal ÷ effort per additional cell) × (within-animal variance ÷ between-animal variance). If an extra mouse is 20 times the work of an extra cell and the ICC is 0.2, that comes to about 9 cells per animal, and every unit of effort after that is better spent on animals.

This is the arithmetic that makes “we have added more cells” the wrong answer to a reviewer. Adding six animals at 10 cells each moves the honest standard error; adding 500 cells to the animals you already have does not. It also explains why the effort feels productive: the number on the screen genuinely improves, and it improves in the one analysis that is not entitled to it.

When the shrinking p-value is also a stopping rule

If you stopped counting cells at the point the p-value crossed 0.05, you have committed a second, separate error on top of the first. Counting until significance is reached is optional stopping, and it inflates the type I error rate independently of any nesting; see p-hacking.

The two compound in a way worth stating plainly. Because the cell-level standard error has no floor, the cell-level t statistic diverges as you keep measuring — unless the animals you happened to sample have exactly equal true means, which they do not. Given enough cells, a cell-level test on nested data will reach any significance threshold you name, for any set of animals, whether or not the treatment did anything. The test has quietly stopped asking whether the treatment works and started asking whether these six particular animals differ, and the answer to that question is essentially always yes.

The practical consequence: a stopping rule based on the p-value is only recoverable if you say so. Pre-specify the number of cells per animal before you start counting, apply it to every animal in both groups, and report it. Where the count was not pre-specified, state the actual counting procedure in the methods rather than reporting a round number that was arrived at retrospectively.

What to expect when you fix it

Frequently the effect survives with a wider interval, and the paper is fine. Sometimes it does not, and that is genuinely useful to learn before a reviewer does.

If it does not survive, the honest options are to report it with the effect size and interval as a non-significant trend, to reframe as preliminary and say what a properly powered design would look like, or to add independent units. What you cannot do is keep the 300-cell p-value.

Report the effect size with its confidence interval at the animal level, because that interval is what bounds the claim the experiment can support. Correcting the level of analysis frequently converts a significance problem into a sample size problem — three animals per group was always three animals per group, and the cell count was concealing it rather than fixing it.

Update every number that changed, in the same revision: abstract, results text, figure legends, supplementary tables. A manuscript that adds a mixed model to the supplement while the abstract retains the cell-level p-value is the specific failure reviewers check for on re-review.

Where else this appears

The same structure, different names: wells split from one culture, sections from one block, images from one slide, two tumours in one mouse, repeated measurements on one subject, patients within a clinic, students within a school. Any time measurements cluster inside a unit that received the treatment, the cluster is the unit.

Pooling replicate experiments and testing on the combined observations is the same error in a form people rarely notice, because between-experiment variation is usually the largest source of variability and pooling erases it entirely.

Image analysis is where the counts grow fastest, because the levels multiply: twenty fields per section, three sections per animal, four animals, and the legend reads n = 240 — see image quantification reporting. Pooled repeats are also the form most likely to be a disguised batch effect, since the day of the experiment carries reagent lot, passage number and operator with it; when the repeats visibly disagree rather than quietly inflating n, see my replicates disagree.

Single-cell sequencing is the modern extreme case. Thousands of cells from one donor are thousands of measurements of one unit, which is why aggregating to a per-donor pseudobulk profile before differential testing is now the standard analysis rather than a conservative option.

When the obvious remedy is unavailable

Summarising to the animal fails in four recognisable situations, and each has an accepted alternative.

Too few units to estimate a variance component. Three independent experiments will not support a well-identified random effect, and lme4 frequently returns a singular fit with the between-unit variance estimated at exactly zero. Report the singular fit rather than deleting the random effect to make the warning go away, and fall back to the summarised analysis on three values, naming the model that failed.

The outcome is a proportion or a count. Averaging binary per-cell calls within each animal and testing the proportions is transparent and usually acceptable. A generalized linear mixed model with a binomial or negative binomial family is the alternative when the per-animal denominators differ widely.

The question really is at the cell level. How a property is distributed across cells within one individual is a within-unit question, and cells are the right unit for it. What that design cannot support is a between-group claim, so keep the two analyses and their two sentences separate in the text.

The grouping was never recorded. Where the experiment of origin or the animal of origin was not tracked, the nesting cannot be reconstructed after the fact. Say so, analyse at the level you can defend, and state the limitation as a specification of what the experiment establishes rather than as an apology — see how to write a limitations section.

What a reviewer says

Reviewers rarely write “your p-value is shrinking because you measured more cells”. They write the specific version, and these are the forms it takes.

“The degrees of freedom reported in Table 2 correspond to the number of cells rather than the number of animals.” “n differs between panels of Figure 3 in a way that appears to track imaging effort rather than the number of animals.” “Please state how many cells were analysed per animal and whether that number was fixed in advance.” “The error bars are inconsistent with the visible spread of the plotted points.” “Statistics appear to have been computed on individual cells; please analyse per-animal values or fit a model with a random effect for animal.”

The first of those is the one to anticipate, because it is checkable from the printed result alone and needs no access to your data. Answering any of them takes three moves: name the experimental unit, re-run the analysis at that level, and update every downstream number. Do not argue that the effect is visually obvious at the cell level, and do not present both analyses and invite the reader to choose — see how to write a response to reviewers.

Before a reviewer finds it

The pseudoreplication agent reads every figure legend, determines what each stated n counts, reconstructs the physical structure of the experiment from the methods, and reports each place where the n exceeds the number of independent units — with the remedy named. It is the single most valuable pre-submission check for in vivo and cell biology work, because this is the error that changes p-values by orders of magnitude rather than at the margin.

If a reviewer has already raised it, see reviewer says my n is not independent.

Review my manuscript

Frequently asked questions

Why does measuring more cells make my p-value smaller?

Because the test treats every cell as an independent observation, so the standard error shrinks with the number of cells rather than with the number of animals. The apparent gain in significance comes from measurement effort, not evidence.

If I keep adding cells, will the result always become significant?

Effectively yes, and that is the clearest proof the analysis is wrong. The cell-level standard error has no floor, so the test statistic grows without limit unless the sampled animals have exactly equal true means. Given enough cells, a cell-level test reaches any threshold you name regardless of treatment.

Does measuring more cells ever genuinely help?

Yes, up to a computable point. Extra cells sharpen each animal’s own value, and the variance of a per-animal mean comes within 10% of its floor at roughly 10 × (1 − ICC) / ICC cells — 40 cells at an ICC of 0.2. Beyond that, effort is better spent on more animals.

How do I know whether my n is cells or animals?

Ask whether you could raise n by returning to the microscope. If yes, it counts measurements. If it can only rise by treating another animal or running another independent experiment, it counts independent units. The degrees of freedom in your output give the same answer: three animals per group supports 4, not 298.

If the cell-level p-value is wrong, should I stop showing individual cells?

No — showing them is good practice. The error is testing on them. Plot the cells and overlay the per-animal means, and run the statistics on the means or on a mixed model. The published recipe is the SuperPlot (Journal of Cell Biology 2020;219:e202001064).

What if my effect disappears at the correct n?

Then report it accurately with the effect size and interval, and either frame the work as preliminary or add independent units. Discovering this yourself is much better than having a reviewer discover it.

Does my p-value shrink the same way with patients instead of cells?

Yes. Patients within clinics, measurements within patients, and eyes within people are the same structure, and the remedies are the same: analyse at the cluster level or model the cluster as a random effect.