A custom AI reviewer for pseudoreplication and unit of analysis

PerfectPaper’s unit of analysis agent reads a preclinical manuscript for the error that inflates significance more often than any other: counting cells, tumours, wells or images as independent observations when the independent unit is the animal or the experiment. It reports every stated n, what that n counts, and whether the test applied matches the structure of the data.

Copy the brief below into a custom agent slot. A review takes up to three custom agents, and they run alongside the standing specialist team rather than replacing any part of it.

This is the most consequential item in the preclinical set. It is invisible in the statistics as reported, it is usually detectable from the methods text, and correcting it often changes a p-value by orders of magnitude.

When to use this agent

The unit of analysis agent earns its slot on any manuscript where the number of measurements exceeds the number of times the treatment was applied. Six situations make that near-certain.

• Your figure legends report n as a number of cells, tumours, fields, wells or organoids rather than a number of animals, donors or independent cultures.
• Measurements were taken repeatedly from the same animal or the same culture, including multiple sections from one block and multiple fields from one section.
• Two tumours were implanted per mouse, or the readout is bilateral — two eyes, two limbs, two flanks.
• Your experiments were repeated three times and the replicates were pooled into a single test on the combined observations.
• The treatment arrived through housing: a compound in the drinking water, a modified diet, a tank or cage temperature, or a compound given to a dam and assessed in her pups.
• A reviewer has already asked what n represents, which is a common form of this objection and the reason reviewer says my n is not independent exists as a separate page.

What the agent reads, and in what order

The agent reads three parts of the manuscript against each other, because pseudoreplication is never visible in any one of them alone. It reads the methods for how many animals, cultures or experiments existed; the figure legends for what each stated n counts; and the results text for the test applied and its degrees of freedom.

The reconciliation is the whole method. A legend reading n = 120 in a study whose methods describe eight mice is the finding, and no statistical test is required to reach it. Degrees of freedom are the most reliable single tell, because they are printed in the result: a two-group comparison of six animals has 4 degrees of freedom, so a result reported as t(148) was run on 150 measurements whatever the legend says.

Where a legend gives no n at all, the agent reports that as a finding rather than passing over it. Undefined n is among the most common results this brief returns, and it is also the cheapest to fix, since the remedy is one clause in a legend. The concept behind the check is set out in full at what is pseudoreplication; this page is the agent that applies it to your manuscript before a referee does.

The agent brief

Paste this into a custom agent. Suggested settings: work type statistics_methods, skill level expert, tools code execution.

Name: Unit of analysis and pseudoreplication

You are reviewing a preclinical manuscript for pseudoreplication: treating
non-independent observations as independent. Read every figure legend, the
methods, and the results text together.

For each reported comparison, determine and report:

1. The stated n. Quote it exactly as written, with its location.

2. What the n counts. Determine whether the unit is an animal, a tumour, a
   cell, a well, a field of view, an organoid, a culture, or an independent
   experiment. Report every n where the unit cannot be determined from the
   manuscript; this is itself a finding and is very common.

3. The independent unit. Determine the level at which the experimental
   treatment was applied and at which biological variation occurs. Treatment
   given to an animal makes the animal the independent unit, however many
   tumours, cells or sections are measured from it. Treatment applied to a
   culture makes the independent culture the unit, not the wells split from it.

4. The mismatch. Where the stated n exceeds the number of independent units,
   state this plainly, give both numbers where they can be determined, and
   explain that the reported test assumes independence the data do not have.
   Where the true number of independent units cannot be recovered from the
   manuscript, say exactly what the author must state.

5. The test applied. Determine whether a nested or hierarchical structure was
   analysed with a method that accounts for it, such as a mixed-effects model
   or an analysis on per-animal summary values. Report every nested structure
   analysed with an unpaired t-test or a one-way ANOVA over individual cells.

6. Repeated experiments. Where experiments were repeated, determine whether
   replicates were pooled into one analysis or analysed as independent
   experiments. Pooling cells across experiments and testing on the combined
   count is pseudoreplication; report it.

7. Two tumours per animal. Where more than one tumour, limb or eye per animal
   was measured, determine whether the analysis accounts for the pairing.

8. Housing, litters and cages. Where the treatment arrived through drinking
   water, diet, temperature, tank or enclosure, or where offspring of a treated
   dam were assessed, determine whether the cage, tank or litter was treated as
   the unit rather than the individual animal.

9. Degrees of freedom. Where a test statistic is reported with its degrees of
   freedom, check them against the number of independent units implied by the
   methods, and report every disagreement.

Use code execution to compute what the effective sample size would be under
the correct unit where the manuscript reports enough to determine it.

Where you cannot determine the unit from the manuscript, say exactly what the
author must state. Do not infer an animal count that is not given, and do not
assume a figure legend's n matches the methods.

Do not report the use of technical replicates as an error in itself. Running a
sample in triplicate is good practice; the error is carrying more than one
value per independent unit into the test. Do not flag a question that is
genuinely at the measurement level - how a property is distributed across
cells within one individual - provided no between-group claim rests on it.

Report each instance with its exact location, the stated n, the correct unit,
and the specific remedy: summarise to the animal level, or fit a model with a
random effect for animal.

Settings, and the tools to leave off

The work type statistics_methods and skill level expert matter more here than on most briefs, because the determination in item 3 is a design judgement rather than a calculation. Give the agent code execution so it can carry out the arithmetic in the next section on the numbers your manuscript actually reports.

Leave web search off. Everything this agent needs is inside the manuscript, and a tool that is available is a tool that gets used — the failure mode is an agent that fetches a paper about mixed models instead of counting your animals. Add vision only when a figure panel carries an n that appears nowhere in the text, which happens most often in image-heavy work; the general case for tool selection is set out in how to write a custom AI reviewer agent.

Keep the brief under the 6,000-character limit that applies to every custom agent. The version above runs to roughly 3,500 characters, which leaves room to add the specific convention your last reviewer objected to.

Questions about the agent’s settings

Do I need to tell the agent how many mice I used, or will it work that out from my methods section?

The agent works this out from the methods, the figure legends and the results together — that reconciliation is the whole method, and a legend reading n = 120 in a study whose methods describe eight mice is itself the finding. Where the manuscript never states an animal count anywhere, the agent will not guess one; it reports the count as missing and states exactly what to add.

What happens if my figure legend never says what n counts in the first place?

The agent reports the undefined n as a finding rather than passing over it — undefined n is among the most common results this brief returns, and it is also the cheapest to fix, since the remedy is one clause in the legend. It does not guess an animal count the manuscript never gives; a guessed number would be worse than an unanswered question.

What settings does an AI reviewer need to catch a unit of analysis error?

Work type statistics_methods, skill level expert, and code execution, so the agent can compute the effective sample size once it has identified the correct unit. Leave web search off: everything a unit-of-analysis check needs is inside the manuscript’s own methods, legends and results, and a search tool available to an agent tends to get used even when it adds nothing.

Can an AI reviewer find an n that only appears in a figure panel?

Yes, with the vision capability turned on — worth doing specifically when a panel states an n that appears nowhere in the manuscript text. Without vision the agent still reconciles the n given in legends, methods and the reported degrees of freedom, which covers most manuscripts; vision closes the remaining gap where the only record of a count sits inside an image.

What this agent catches

The unit of analysis agent catches seven recurring shapes of this error, matching what a manuscript reports against the units that were actually independent.

Each row is a mismatch between what was measured and what was independently treated, and the third column is what that mismatch does to the inference. The last three rows are the ones authors dispute most often, and each is settled by the same question: could you raise that n tomorrow without running a new experiment? If counting more fields, thawing another passage or weaning another pup raises n, the number counts measurements.

The arithmetic the agent runs

Given code execution, the agent computes the effective sample size rather than asserting that the reported one is wrong. Two calculations do the work.

The design effect. Kish’s formula converts the nesting into a penalty directly: DEFF = 1 + (m − 1)rho, where m is the number of measurements per unit and rho is the intraclass correlation, the share of total variance sitting between units rather than within them. Take 50 cells per animal and an ICC of 0.3, which is unremarkable for a phenotype that differs between animals: DEFF is 15.7, and 150 measurements carry the information of about 10 independent observations. ICC = 0 is the only value at which measurements are worth their face value, and it means the animals do not differ from one another at all. The formula above assumes every animal contributes the same number of measurements; where cell counts differ substantially between animals — 15 from one, 80 from another — the design effect has to be computed from the mean cluster size and its spread rather than read off a single animal’s count, and the more defensible route is to estimate rho from a random-intercept model fitted to the manuscript’s own data rather than assume a textbook value.

The standard error factor. Standard error falls with the square root of the sample size, so reporting 150 measurements from 6 animals as n = 150 makes the standard error smaller by a factor of five than the correct calculation on 6, and multiplies the test statistic by five. Aarts and colleagues measured what that does to the error rate: surveying 314 neuroscience articles they found nested data in about half of them, and their simulations put the probability of declaring a null effect significant at as much as 80% against a nominal 5% (Nature Neuroscience 2014;17:491–496). The same inflation corrupts any statistical power calculation built on the measurement count.

Where the manuscript reports enough to determine both numbers, the agent states them side by side. Where it does not, the agent says which quantity is missing rather than assuming a value for it.

A worked example: 150 cells from six mice

Consider a figure showing mean fluorescence intensity in tumour-infiltrating T cells, three treated mice and three controls, 25 cells scored per animal, legend reading “n = 150 cells, unpaired two-tailed t-test, p < 0.001”.

The agent reports four things about that legend. The stated n is 150 and it counts cells. The treatment was injected into animals, so the independent unit is the animal and there are 6 of them, 3 per group. The test assumes 150 independent observations and the design provides 6, so the degrees of freedom should be 4 rather than 148. And the remedy is specific: take each animal’s mean or median across its 25 cells, run the comparison on those six values, and report the test as t(4).

What usually happens next is the uncomfortable part. A finding reported at p < 0.001 on the inflated count can fall well short of significance once the analysis runs on six values, which is why this objection reverses results where most statistical criticisms shift them at the margin. Three responses to that are legitimate — report the non-significant difference with its effect size and interval, reframe the result as preliminary and state what a properly powered design would require, or collect more independent units. Adding six more animals at 10 cells each buys more information than adding 500 cells to the animals you already have, because the design effect scales with the number of units and not with the number of measurements.

The presentation that survives review shows both levels: plot the 150 cell values lightly, overlay the six per-animal means as larger points, colour-code by experiment, and compute the statistics on the means. Lord, Velle, Mullins and Fritz-Laylin published the recipe as the SuperPlot (Journal of Cell Biology 2020;219:e202001064). A SuperPlot drawn correctly and analysed at the cell level is still pseudoreplicated, so state in the legend which level the error bars and the p-value were computed at.

Common questions about this worked example

Is n=150 cells from 6 mice actually pseudoreplication, or is that a normal way to report results?

Yes. The 150 cells came from only 6 animals, and the treatment was applied to the animal, not the cell, so the animal is the independent unit. The 25 cells scored from each one make that animal’s value more precise but add no new independent observations. The correct n for a between-group test is 6, and the result should be reported as t(4), not t(148).

Why did my p-value stop being significant once I switched to per-animal averages?

Because the test statistic was inflated by treating 150 correlated cell measurements as independent; the correct analysis on six per-animal values has 4 degrees of freedom, not 148, and a properly sized sample often cannot detect the same effect at the same p-value. That is a result about the evidence in hand, not proof the effect is not real.

Will the agent flag it if I pooled three experiments into one test?

Yes, when the raw cell-level observations from the three experiments were pooled into a single test. Analysing the three experiments with a term that accounts for which experiment each observation came from, or summarising each experiment to one value first, is correct and is not flagged. Between-experiment variation is often the largest source of variability in cell biology, which is why the distinction matters.

Edge cases this brief is written to handle

The brief includes item 8 and the two negative instructions because five recognisable situations break a naive reading of the manuscript.

The unit is not the animal. A compound in the drinking water of a cage of five mice was applied once, to the cage. A compound given to a pregnant dam and assessed in her pups makes the litter the unit. The agent is instructed to look for the level at which the treatment was applied, not to default to the animal.

The measurement-level question is legitimate. A study of how a property is distributed across cells within one individual is a within-unit question, and the cells are the right unit for it. What that design cannot support is a between-group claim, so the agent flags only the claims that cross groups.

Technical replicates are not the error. Running a sample in triplicate detects pipetting error and instrument drift and is good practice; the error is carrying more than one value per independent unit into the test. An agent without this instruction reports every triplicate as a defect, which trains authors to ignore the whole category.

Pooling is a disguised batch effect. Three experiments pooled into one test lose between-experiment variation, which is usually the largest source of variability in cell biology, and the day of the experiment carries reagent lot, passage number and operator with it; see what is a batch effect.

Single-cell counts are the same problem at a larger scale. Thousands of cells from one animal are thousands of measurements of one unit, which is why pseudobulk aggregation to the donor before differential testing is the standard analysis, and why the agent treats a cell count in a sequencing figure exactly as it treats a cell count in a microscopy figure — see multiple testing in omics.

When the obvious remedy is unavailable

Summarising to the independent unit is correct in the great majority of designs, and the agent names it first for that reason. Four situations require something else, and the brief asks for the remedy that fits rather than the remedy that is shortest.

Too few units to fit a random effect. Three independent experiments will not support a well-identified variance component, and lme4 frequently returns a singular fit with the animal variance estimated at exactly zero. Report the singular fit rather than deleting the random effect, and fall back to the summarised analysis on three values, stating the model that failed and why. This is also the regime where a mixed model’s own p-value should be read cautiously even when it does converge: the asymptotic approximations behind it assume enough independent units to estimate the variance component precisely, which a handful of clusters cannot supply, so the summarised two-sample test on the small number of independent values is often the more defensible report, not merely the fallback.

The outcome is a proportion or a count. Averaging binary cell-level calls per animal and testing the proportions is transparent and usually acceptable; a generalized linear mixed model with a binomial or negative binomial family is the alternative when the per-animal denominators differ widely. A generalized estimating equation (GEE) is the other standard route: it models the population-averaged effect and the within-animal correlation directly without assuming a distribution for a random effect, and it answers a subtly different question — the average effect across animals rather than the effect for a typical animal. Do not reach for it because the animal count is small, which is the common misreading of the word robust in this context. GEE’s robustness is a property of its sandwich variance estimator, and that estimator is asymptotic in the number of clusters: with a handful of animals it is biased downward and understates the standard error, which is the same inflation this agent exists to catch, reintroduced by the remedy. GEE belongs in designs with dozens of independent units. Below that, summarise per animal, and if you must model, prefer a GLMM with the small-sample degrees-of-freedom correction named at the end of this section.

The nesting was never recorded. Where the experiment of origin was not tracked, the structure cannot be reconstructed after the fact. Say so, analyse at the level you can defend, and record the grouping variable next time — it is one column.

Only one independent unit exists. A single patient-derived line, a single donor, a single field site. No analysis rescues an n of 1, and the honest presentation is descriptive, with the limitation written as a specification of what the experiment can establish; see how to write a limitations section.

Where a mixed model is the right answer, report it reproducibly: name the package and the formula in full — lme4 with lmerTest, or nlme — and state whether denominator degrees of freedom came from the Satterthwaite or the Kenward–Roger approximation, Kenward–Roger being the more conservative and the usual choice when the number of units is small. In SAS the equivalent is PROC MIXED with DDFM=KR, in Stata mixed. “A mixed model was used” is not reproducible and reviewers say so.

Questions about fixing a flagged unit of analysis error

Do I have to fit the mixed-effects model myself, or will the agent do it for me?

The agent names the remedy but does not fit it: it identifies that animal or experiment belongs in the model as a random effect and states the structure required, while fitting the model in lme4 or nlme, choosing Satterthwaite or Kenward-Roger degrees of freedom, and reporting the result reproducibly remain the author’s work.

I only have three independent experiments — is a random-effects model even going to work?

Often not well: three clusters rarely give lme4 enough information to estimate a variance component, and the fit commonly comes back singular, with the animal- or experiment-level variance pinned at zero. Report the singular fit rather than deleting the random effect, then fall back to summarising each experiment to one value and testing on the three, stating plainly which model failed.

Does it matter if I measured two tumours, both eyes or both flanks on the same animal?

Yes — measuring both sides of one animal gives two correlated values, not two independent ones, because the animal’s genetics, handling and treatment are shared between them. Twelve tumours from six mice, two per mouse, is six independent units; the analysis needs to account for the pairing, for example by averaging the pair or including animal as a random effect, or the variance is understated.

What journals already require

Three published requirements make this a compliance check rather than a referee’s opinion, and the agent quotes them because an editor can act on a compliance failure without a statistical reviewer.

ARRIVE 2.0 item 1b of the Essential 10 asks authors to state “The experimental unit (e.g. a single animal, litter, or cage of animals)” — see the ARRIVE guidelines. The British Journal of Pharmacology states the rule in its guidance for authors and peer reviewers (Curtis et al., British Journal of Pharmacology 2018;175:987–993): group size is the number of independent values, so a single sample measured five times is n = 1 rather than n = 5, and the five measurements are technical replicates that should be reduced to one value per unit before any test is run. That guidance also sets a minimum of five independent values for a dataset subjected to statistical analysis, so a design of three animals with fifty cells each does not meet the minimum however many measurements it contains, which converts the finding into a sample size problem the moment it is corrected.

The Nature Portfolio Reporting Summary, submitted with every life-sciences manuscript, contains a Statistics checklist that asks authors to confirm “A statement on whether measurements were taken from distinct samples or whether the same sample was measured repeatedly”. A Reporting Summary answering that item one way while a figure legend counts cells the other way is an internal inconsistency inside your own submission.

What reviewers say when this is missed

Reviewers rarely use the word pseudoreplication. They write the specific version, and these five comments are what the agent is built to pre-empt.

“What does n refer to?” “The authors treat individual cells as independent observations.” “These appear to be technical rather than biological replicates.” “The analysis does not account for clustering within animals.” “It is unclear how many independent experiments contributed to this figure.”

Answering any of them takes three moves and nothing else: name the experimental unit, re-run the analysis at that level, and update every number that changed — abstract, results text, figures and supplement together. Do not present both analyses and let the reader choose, and do not argue that the effect is obvious from the cell-level data; reviewers who raise this check the revision specifically for a mixed model added to the supplement while the abstract retains the original p-value. See how to write a response to reviewers.

Questions about reviewer comments and journal requirements

Reviewer 2 says my n is actually the number of animals, not the number of cells — how do I respond?

Agree with the point rather than defend the cell count, since it is correct: the treatment was applied to the animal, so the animal is the independent unit whatever the cell count shows. Re-run the analysis on per-animal values or a mixed model with animal as a random effect, update every affected number — abstract, results, figures and supplement together — and do not present both analyses side by side.

My reviewer just asked “what does n refer to” — what’s the actual answer supposed to look like?

The answer names the independent unit explicitly, in one clause: animal, litter, cage or independent culture, and how many of each. ARRIVE 2.0 item 1b asks for precisely this — “the experimental unit (e.g. a single animal, litter, or cage of animals)” — stated once in the legend or methods, not implied by the cell count or left for the reviewer to infer.

How this differs from the built-in review

The standing team includes specialists in statistical models and regression, cohort accounting, and study design. They read the analysis as described. What none of them is briefed to do is reconstruct the physical structure of the experiment from the methods and compare it against the n in each figure legend, which is what finding pseudoreplication requires. It is a domain habit rather than a statistical technique.

The distinction that habit turns on is between independent units and independent repeats, which are different requirements a paper can satisfy separately: a study can run three genuine independent experiments and still analyse the cells within them as independent, and a study can name its unit correctly and never repeat the experiment at all. The second failure is covered by reviewer says the experiment was not replicated.

What this agent does not do

The unit of analysis agent reports the mismatch and names the remedy. It does not fit the model for you, it does not re-analyse your raw data, and it cannot see anything the manuscript does not contain — an animal count that exists only in a lab notebook is, to this agent, a missing value it will ask you to supply.

It also does not judge whether your effect is real. Where correcting the unit removes significance, that is a result about the evidence in hand rather than a verdict on the biology, and it is the reason to run this agent before submission rather than after a rejection. The related symptom — a p-value that keeps improving as you measure more cells from the same samples — has its own page at my p-value gets smaller with more cells.

Pairs with in vivo experimental rigour, which covers the design details this agent needs and often finds missing. Full set: AI peer review for preclinical cancer research.

Review my manuscript

Related agents

The four agents below share a review with this one; you can run three at a time. See AI peer review for preclinical cancer research for the full set of five preclinical agents.

• In vivo experimental rigour — use it for randomisation, blinding, exclusions and the tumour volume formula.
• Flow cytometry reporting — use it when a percentage has no named parent population.
• Image quantification integrity — use it when a representative image is standing in for a quantification.
• Model-to-human relevance — use it when an immune or microenvironment claim may exceed the host used.