How to write a custom AI reviewer agent

A custom AI reviewer agent brief has five parts: a one-sentence scope, an ordered list of specific checks, an instruction on what to do when information is missing, an instruction on what not to do, and a required output shape. Write checks as determinations rather than topics, put the highest-value one first, and keep the brief under 6,000 characters.

The most common mistake is writing a brief that is too broad. This page is about writing review prompts generally, and it applies whether you paste the result into PerfectPaper or use it elsewhere; where a detail is specific to PerfectPaper’s custom agent slots — the 6,000-character field, the ten work types, the pre-charge check that reads your brief before it runs — the page says so. The prebuilt briefs in the custom AI reviewer agents library are written to this structure, so many of them double as worked examples of it.

Step one: make the scope narrow

Write one sentence naming what the agent examines and what it ignores. An agent briefed to check statistics, figures and writing quality will produce shallow findings on all three, because attention is finite and the instruction gives it no priority.

Narrow scopes produce specific findings. “Check the manuscript for problems” produces the observations you could have made yourself.

The test for a scope sentence is whether it names a determinable object. “Review the methods” does not; “determine, for every reported comparison, what the stated n counts and what the independent unit is” does, and it is what the scope sentence of the pseudoreplication agent sets up. Compare three scope sentences at increasing narrowness: “check the statistics” is a topic, “check whether the statistical tests match the data structure” is a topic with a qualifier, and “reconcile the n in every figure legend against the number of animals in the methods and the degrees of freedom in the results” is an operation with an output. Only the third tells the agent what to read, in what order, and what a finding looks like.

Narrowness also decides what the agent reads together. Most defects that survive to submission are invisible in any one section and visible only in the comparison — a legend against a methods paragraph, an abstract claim against a results table, a registered primary endpoint against a reported one. Name the sections in the scope sentence and the agent will hold them side by side rather than reading each in turn.

Step two: write checks as determinations, not topics

The difference between a brief that works and one that does not is usually grammatical. Compare:

Consider whether the statistical analysis is appropriate.

Determine, for every reported comparison, whether the stated n counts animals or cells, and report every instance where the n exceeds the number of independent units.

The first invites an opinion. The second specifies an operation with a definite output, and can be checked. Number your determinations so they run in a defined order.

Three grammatical habits separate the two forms. A determination opens with a verb that has a truth condition — determine, reconcile, quote, count, compare — rather than consider, assess, evaluate or look at. It carries a quantifier that fixes coverage: for every reported comparison, for each stated accession, in every figure legend. And it names the output in the same sentence, so the agent knows what a completed check produces rather than inferring it.

A useful diagnostic before you paste a brief: read each numbered item and ask what a negative result would look like. If item 4 cannot come back as “checked, nothing found”, it is a topic, and it will come back as commentary. This is the same discipline as writing a determinable hypothesis, and it fails for the same reason — the vague version feels more complete because it excludes nothing.

A determination has a third possible result that briefs often collapse into the first two: not applicable. “Report every case where the stated n exceeds the number of independent units” returns nothing on a design with one animal per cage, and that silence means something different from the same silence on a study that pools plate replicates as if they were biological ones — a design the check does apply to and did not fire on. A brief that cannot distinguish the two makes “checked, this is fine” and “checked, does not apply here” read as the identical result in the finding list. Where a check has a design precondition, state it: “if animals were housed individually, this check does not apply; report the housing unit instead.”

Common questions about writing the brief

Why does my AI reviewer just tell me my paper looks fine instead of finding real problems?

A prompt with no scope produces this result by default — the agent has nowhere specific to focus, so it defaults to reassurance. Name one narrow thing to examine (a comparison, a legend, a claim against a table) rather than asking for “any problems,” and the same model returns a specific finding instead of a compliment. Breadth, not the model, is usually the cause.

How specific does a prompt for an AI reviewer actually need to be?

Specific enough that a negative result is possible. “Check whether the statistics are appropriate” cannot come back as “checked, nothing found” — there is no failure condition to test against. “Determine whether each reported test matches the data’s independence structure” can. If you cannot picture what a clean pass looks like, the check is still a topic dressed up as a question.

My custom AI reviewer keeps telling me things I already know, like that my sample size is small — how do I get it to find something more useful instead?

Small-sample commentary is what a broad prompt returns by default, because it is true of almost every manuscript and costs the model nothing to say. Replace the general request with a numbered determination the agent must resolve from your specific data — the exact n each figure implies, say — and generic commentary has nowhere left to attach itself.

Step three: order the checks by value

Put the check most likely to produce a consequential finding first, and say so explicitly — an instruction like “treat this as the highest priority item” changes what gets attention. The last item in a long list gets the least.

Rank by consequence, not by the order the sections appear in your paper. The ordering question worth asking is: which finding, if true, would change the paper’s conclusion rather than its presentation? A unit-of-analysis error reverses a result; an unlabelled axis does not. Put the reversible-conclusion checks in positions one and two, the reporting-completeness checks in the middle, and the conventions that only an editor would notice at the end.

Nine numbered determinations is about the ceiling at which ordering still does work. Beyond roughly a dozen, later items behave like a checklist the agent skims, and the fix is not a longer brief but a second agent — you have three slots per review, and two narrow agents beat one long one. Where two checks genuinely depend on each other, say so in the item itself (“do item 5 only where item 4 found a mismatch”) rather than trusting sequence to imply it.

A subtler version of the same problem is two checks that independently derive the same intermediate value — an effective sample size, a corrected p-value — without either depending on the other’s output. Phrased as separate determinations, a model can compute that value two different ways within the same run and report two findings that quietly disagree on the underlying number. Where a value feeds more than one determination, compute it once, in the earliest item that needs it, and have every later item reference “the count from item 2” rather than recomputing it.

Step four: say what to do when something is missing

Most review findings are absences, and absences are where agents guess. Include an instruction of this shape:

Where you cannot determine something from the manuscript, say exactly what is missing and what the author would need to add. Do not infer values that are not present.

Without it, an agent asked whether blinding was performed will sometimes report that it was, on the strength of the paper reading like a careful one.

Absence findings are also the quickest findings to act on, which is the practical argument for insisting on them. A missing definition of n is one clause in a legend. A missing ethics approval number is one sentence. A missing accession is a deposition you were going to make anyway. Briefs that only report errors of commission miss the entire class of defect that takes a day to fix and a revision cycle to discover, which is why the missing-information instruction is the line most worth carrying into every brief you write, and the data availability and accessions agent is built almost entirely out of it.

Add a second clause where a value can be partly recovered. “Where the manuscript reports enough to determine the number but does not state it, compute it and say which figures you combined” turns a shrug into a checkable claim, and it is what makes an agent with code execution more useful than one without.

Step five: write the negative instructions

The instructions telling an agent what not to do are the ones that keep a brief from producing confident nonsense. They are also the first thing people delete when shortening a brief, and they should be the last.

Examples from the prebuilt briefs on this site:

• Do not assert that a specific accession does or does not exist, and do not attempt to retrieve its contents. A false report of a missing accession is worse than not checking.
• Do not treat small sample size as a limitation in itself. A qualitative study evaluated by quantitative standards produces findings that are all wrong.
• Where the manuscript explicitly adopts a causal framework with stated assumptions, causal language is appropriate and should not be flagged. Without this, the agent flags every correct paper.
• Do not comment on grammar, formatting or citation style. Otherwise a specialist agent spends its attention on what four other agents already cover.

Each of those exists because a specific agent failed a specific way, and the shape of the instruction follows from the shape of the failure. Negative instructions fall into four kinds, and a brief usually needs one of each: the false-positive guard that names a legitimate practice the agent would otherwise flag, as in the qualitative rigour brief; the out-of-scope guard that stops duplication of the standing team; the verification guard that forbids a claim the agent cannot check from the manuscript alone; and the inference guard that forbids filling a gap with a plausible value.

Write them from observed failures rather than imagined ones. The way to find yours is to run the brief once, read the findings you disagree with, and convert each disagreement into a sentence beginning “do not”. The causal-framework exception above is exactly that: it was written after an agent flagged correctly hedged mediation language, and it is now the load-bearing line in the causal language brief.

Scope a negative instruction to the same language as the determination it guards, not to the general topic. “Do not flag small sample size as a limitation” can shadow a legitimate finding elsewhere in the same brief that also mentions sample size, for an unrelated reason — a determination checking whether the stated n matches the methods section, say. Write “do not flag a small sample size as a limitation in a qualitative study using purposive sampling” instead, naming the exact condition under which the general advice does not apply, and the guard stops suppressing checks it was never meant to touch.

Step six: specify the output

State what each finding must contain. A useful default:

Report each issue with its exact location, what it prevents a reader from concluding, and a specific remedy.

The middle clause is the one worth insisting on. “Blinding is not reported” is a checklist item. “Blinding is not reported for caliper measurement, so the primary outcome may be biased by expectation” is a review comment.

The three-part shape maps onto what a referee actually writes, and onto what an author can act on in one pass: where — a section, a legend, a table cell, a line of the abstract; what it denies the reader — a conclusion that no longer follows, an estimate that cannot be interpreted, a number that cannot be reproduced; and what to do instead — a specific sentence, a re-analysis at a named level, a value to supply. Findings missing the third part get read and not fixed; findings missing the second part get argued with.

Ask for the quoted text as well as the location. “Section 2.3” sends you hunting; “the legend to Figure 3B, reading n = 150 cells” lands you on the sentence. A brief that requires a verbatim quotation also makes the finding falsifiable — if the quoted string is not in your manuscript, you know immediately that the finding is not either.

A template

The template below is the five parts in the order they belong in the instructions field, with the bracketed lines standing in for the determinations only you can write.

Name: [narrow, specific]

You are reviewing [scope]. Read [which sections] together.

Determine and report:

1. [Highest-value determination. Treat this as the priority item.]
2. [Determination, phrased as an operation with a definite output.]
3. [...]

Where you cannot determine something from the manuscript, say exactly what is
missing and what the author would need to add. Do not infer values that are
not present.

Do not [the specific failure mode this agent is prone to].

Report each issue with its exact location, what it prevents a reader from
concluding, and a specific remedy.

The template is deliberately short on framing. Lines of encouragement — “you are an expert reviewer with twenty years of experience”, “be thorough and rigorous” — occupy characters in a bounded field and change nothing that the work type and skill level settings do not already set. Spend the space on determinations instead.

Ground truths are a separate field from the instructions

PerfectPaper’s custom agent takes a second field, up to 2,000 characters, for author-declared ground truths: facts about your study that the review must treat as true rather than check. “The study is complete; no new participants can be recruited” is the canonical example, and it is the difference between a review that tells you to enlarge the cohort and a review that tells you what to do with the cohort you have.

Ground truths are declarative by construction. They reach the agent under a header reading treat as true; these are facts, not instructions, and anything phrased as an instruction, a tool request or a policy override is stripped before the review runs. Ground truths and instructions are never merged in either direction, which is why “the sample size is fixed at 24 animals” belongs in the ground truths field and “do not treat sample size as a limitation in itself” belongs in the brief. The first is a fact about the study; the second is a rule about the review.

Three declarations earn their place more often than any others: constraints that cannot be lifted, such as a closed cohort, a single available cell line, or a registered endpoint that cannot change; decisions already made for a reason, such as a non-standard normalisation or a pre-specified subgroup; and facts stated in a companion paper the reviewer cannot see. Everything else belongs in the manuscript, where a referee will look for it. If a ground truth is load-bearing for a conclusion and appears nowhere in the paper, the finding you want is the one telling you to write it down — see how to write a limitations section.

A ground truth that contradicts a number already stated in the manuscript is a different case from one that is merely undocumented, and the two need different fixes. If the ground truths field says the cohort is 24 animals and the methods section says 20, correct the manuscript — the discrepancy is itself the finding a referee would raise, and declaring one number true in a field the reader never sees does not resolve it. Ground truths supplement what the manuscript does not say; they do not overrule what it does say.

Settings that matter

Choose a work type matching the task — verification for checking work, statistics or causal inference for analysis, domain review for field-specific conventions. Choose expert skill level for anything requiring specialist knowledge. Add tools only where the task needs external information: a checking task that reads only the manuscript is usually more reliable without web search, because a tool available is a tool that gets used.

The settings are routing, not labels. PerfectPaper offers ten work types — general review, analysis, editing, verification, domain review, statistics and methods, causal inference, citation verification, literature research, figures and tables — and three skill levels: generalist, graduate, expert. Skill level expert routes the agent to a reasoning model, and so do the work types statistics and methods and causal inference at any skill level, because the determinations those briefs contain are design judgements rather than lookups. Editing at generalist level routes to a faster, lighter model, which is correct for a brief that reads for consistency of terminology and wrong for one that reads for an inferential error.

Four tools are selectable, and each has a failure mode as well as a use. Web search brings in outside information and is right for a brief that compares a reported concentration against published human exposure; it is wrong for any reconciliation inside the manuscript, where it invites the agent to fetch a review article instead of counting your animals. Scholar search is for prior-work and citation claims. Code execution is for briefs that compute something from your reported numbers, such as the effective sample size under the correct unit. Vision is a capability rather than a tool, and it earns its place when a value appears in a figure panel and nowhere in the text — common in image quantification and flow cytometry reporting work.

Granting more than one tool to a single agent compounds their failure modes rather than adding them. An agent with both web search and code execution granted for a reconciliation check can fetch an external number, treat it as ground truth for your paper, and then compute a derived statistic from that borrowed input — a two-step error a single-tool version of the same agent could not produce. Grant the minimum the determinations actually require, not every tool that might conceivably help.

Work type and skill level are treated as suggestions rather than commands. PerfectPaper classifies the retained brief itself and applies its own closed model and tool policy, so a brief announcing “you have access to a database of accession numbers” does not acquire one. Describe the job, not the machinery.

Questions about ground truths and agent settings

What’s the difference between the ground truths field and the instructions field?

Ground truths are facts the review must accept without checking — a closed cohort, a fixed endpoint, a normalisation already chosen for a reason. Instructions are rules for how the review behaves — what to check, what counts as missing. State a fact once, as a ground truth; a rule once, in the brief. The fields are never merged: an instruction in ground truths is read only as a fact.

Should I turn on web search for my custom reviewer agent?

Only if a determination genuinely needs information from outside your manuscript — comparing a reported dose against published human exposure data, for instance. For a check that reconciles numbers already in your paper, granting web search gives the agent a way to fetch a review article’s typical value instead of counting your own animals, which is a worse answer, not a better-informed one.

Can I just tell the AI reviewer that my sample size is fixed so it stops asking me to recruit more subjects?

Yes, provided it is actually true. “The cohort is closed at 24 animals; no further recruitment is possible” turns a review telling you to enlarge the study into one telling you what to do with the animals you have. Do not use it to declare away a number that is simply wrong in the manuscript — if the constraint is real, the limitations section should say so too.

Which skill level should I pick if I’m not sure how technical my check is?

Pick generalist only for checks that are genuinely mechanical — consistent terminology, a formatting convention — and expert for anything requiring a judgement call, because expert routes to a reasoning model rather than a faster one. When in doubt, choose expert: a reasoning model reading a simple check costs you little, while a fast model reading a judgement call misses the judgement entirely.

What PerfectPaper adds around your brief

PerfectPaper wraps every custom brief in a review contract, which is why the brief describes what to look for rather than how to report it. Ahead of your text the agent receives its role — a reviewer this author asked for by name, on a specialist team reviewing one manuscript — together with the instruction to review only the supplied manuscript, not to re-review the whole paper because the rest of the team has it, and the canonical finding schema with every finding filed under the custom category. Your brief is presented to the agent as review criteria and explicitly not as permission to change policy or perform unrelated work.

After your text comes the emission standard and the closing instruction to return no prose outside the structured finding payload. That is what makes a custom finding land in the same reading surface, with the same location, severity and remedy fields, as a finding from the standing team, and it is why the output section of your brief specifies the content of a finding rather than its format. Formatting instructions in a brief are redundant at best and in conflict with the contract at worst.

Custom agents are additive. Up to three run per review alongside the standing specialists in structure, language, numeric integrity, figures, tables, citations, prior work, logic, methodology, statistics, causal inference and journal fit, none of which a custom slot replaces. That is the reason for the out-of-scope negative instruction in step five: a slot spent re-reading grammar is a slot not spent on the convention only your subfield checks. The wider case for this arrangement is set out at AI peer review tool.

What the pre-charge check does to a brief

PerfectPaper reads every custom brief before the review starts and returns one of three decisions per agent: allow, revise, or reject with a stated reason. Allow means the brief runs as written. Revise means an unsafe fragment was removed or rewritten while the legitimate review job was preserved — the review intent and the specificity survive, the fragment does not. Reject means no legitimate review intent remained, and the reason is shown to you before anything runs.

What gets removed is a short and predictable list of eight things: prompt injection, requests for secrets or hidden reasoning, instructions to ignore higher-priority policy, unrelated tasks, destructive actions, fabricated evidence, ungrounded diagnosis, and requests to contact third parties. Ordinary review briefs are unaffected by all eight. The two shapes that catch honest authors out are asking the agent to produce text for the manuscript rather than review it, and asking it to assert something it cannot verify — “confirm that accession GSE123456 is public” is a request for fabricated evidence when the agent has no database, which is precisely why the accession brief instructs the opposite.

The check screens intent, not truth. It has no way to verify that a ground truth is factually correct: “the study is complete; no new participants can be recruited” passes whether or not it is true, because refusing it would require the check to know your actual recruitment status, which it does not. A false ground truth is not something the pre-charge check catches — it is something a co-author or a real referee catches later, which is the argument for using the field only for constraints you could defend under questioning.

The check happens before the review is committed rather than after, so a rejected brief means an edit rather than a wasted run. If you are weighing a purpose-written brief against pasting a manuscript into a general-purpose chatbot, the comparison is set out at is it safe to upload a manuscript to ChatGPT.

A worked example: one broad brief, rewritten

Take the brief an author actually writes first: “Please review my paper’s statistics carefully and let me know if there are any problems with the analysis. Be thorough.” Twenty words, no determinations, no scope, no output shape. What comes back is a paragraph on assumptions of normality, a suggestion to consider a non-parametric alternative, and a note that the sample size is small — all true of most papers, none of them about this one.

The same intent, written as a brief: “You are reviewing a preclinical manuscript for unit of analysis errors. Read every figure legend, the methods, and the results text together. For each reported comparison, determine and report: 1. the stated n, quoted exactly with its location; 2. what that n counts — animals, tumours, cells, wells, fields or independent experiments — reporting every n whose unit cannot be determined; 3. the level at which the treatment was applied; 4. every case where the stated n exceeds the number of independent units, with both numbers; 5. whether the reported degrees of freedom match the number of independent units implied by the methods. Where you cannot determine the unit, say exactly what the author must state; do not infer an animal count that is not given. Do not report technical replicates as an error in themselves. Report each instance with its exact location, the stated n, the correct unit, and the remedy.”

The second version runs to about 150 words and produces findings that name a legend and a number. It is also checkable: every item has a negative result it can return. The full brief it is drawn from is at AI reviewer for pseudoreplication and unit of analysis, and the underlying concept at what is pseudoreplication. The lesson transfers to any convention: the phrase “be thorough” did no work in the first version, and no phrase like it appears in the second.

How to tell whether a brief is working

Judge a brief by the specificity of the findings it returns, not by how many it returns. A working brief produces findings that quote your manuscript, name a location you can navigate to, and state a remedy you could implement this afternoon. A failing brief produces findings that would be true of any paper in your field, which is the signature of a scope that was too broad rather than of a paper that is clean.

Four failure signatures each point to a specific edit. Findings that restate your methods back to you mean the checks were written as topics — return to step two. Findings about grammar, figure aesthetics or reference formatting mean the out-of-scope guard is missing — return to step five. Findings asserting something absent from your manuscript mean the missing-information instruction is missing or is being overridden by a tool — return to step four, and consider turning web search off. Findings that are correct but trivial mean the ordering is wrong: the consequential check is buried at position seven.

A brief that returns nothing is not necessarily broken. Where every determination came back clean, that is a result, and the useful next move is to widen by one convention rather than to loosen the language. The convention worth adding is almost always the one a referee raised on your last submission — see reviewer objections about methods.

Test a brief before trusting a clean run, the way you would test a computational method before trusting it in the manuscript itself. Apply it to an earlier draft you know had the issue, or to any published paper in your field with the defect already public, and confirm the determination actually fires. A check that never fires across several manuscripts you have reason to think are not all clean is a broken determination wearing the shape of a passing one — the finding list alone cannot tell you which.

What a brief cannot fix

A custom brief changes what the review looks for; it does not change what the manuscript contains. An agent cannot recover an animal count that lives only in a lab notebook, cannot see a supplementary file you did not attach, and cannot verify that a deposition exists in an external database from the manuscript text alone. Where a determination depends on information outside the paper, the honest brief asks the agent to name the missing item rather than to resolve it.

Three further limits are worth stating before you write. A brief cannot make an agent adjudicate a contested question in your field — it can report that a claim rests on a contested assumption and name the assumption, which is more useful anyway. A brief cannot substitute for a domain check the manuscript makes impossible: a nomenclature agent needs the variants written out, and variant nomenclature findings are only as complete as the table they read. And a brief cannot make the review a replacement for peer review itself; the purpose is to find what a referee would find, before the referee does, which is the argument set out at what peer review is.

Where a check genuinely needs outside evidence — a response criterion, a registered endpoint, a reported human exposure — say so in the brief and grant the tool that supplies it. The trial registration and endpoints brief is written that way, with web search granted, and the batch effects brief is written the opposite way, with tools off, because everything it needs is inside the manuscript.

Questions about fixing a brief that isn’t working

How do I rewrite a reviewer agent brief that keeps returning useless findings?

Match the failure to its cause. Findings that restate your methods back to you mean the checks are topics, not determinations — rewrite them as operations with a definite output. Findings about grammar or formatting mean the out-of-scope guard is missing. Findings asserting something your manuscript never states mean the missing-information instruction is absent, or a granted tool is overriding it.

How do I write the negative instructions in a reviewer agent brief?

Run the brief once, read every finding you disagree with, and turn each disagreement into a sentence beginning “do not.” Aim for at least one false-positive guard, one out-of-scope guard, one verification guard, and one inference guard. Write from a failure you actually observed — an imagined failure produces a vague instruction that guards against nothing specific.

Can I write one brief and reuse it for every paper I submit?

Yes, if you write it around a convention rather than a manuscript — a reporting requirement, a statistical habit, a nomenclature standard — with nothing in it naming your current figures, cohort or dataset. The most durable reusable brief is usually the one built from a comment your last reviewer made, because that objection tends to recur across a whole line of work.

Why did my AI reviewer flag something that isn’t actually a problem?

Usually a missing false-positive guard: a legitimate practice in your field that the agent has no way to know is standard unless the brief says so. A qualitative study read against quantitative sample-size norms, or hedged causal language inside a study that explicitly states its causal assumptions, are the two most common shapes. Name the exception once and the false flag stops.

How long a brief should be

The instructions field accepts up to 6,000 typed characters, and the prebuilt briefs on this site run between roughly 2,200 and 3,500 — long enough for nine numbered determinations, the missing-information instruction, two or three negative instructions and the output shape, with room left over. Length spent on determinations helps. Length spent on encouragement does not.

Two further budgets apply. You may attach a plain-text file to extend a brief, and the typed text plus the attached file together are capped at 8,000 bytes; ground truths are counted separately, up to 2,000 characters. And if you would rather start from notes than from an empty field, PerfectPaper will expand rough notes of 50 characters or more into a draft brief you then edit — the floor exists because below it there is no author intent to develop, and the expansion would be the model writing your brief rather than yours.

Browse the prebuilt agents by field for complete worked examples, all following this structure.

PerfectPaper runs up to three author-written reviewer agents alongside its standing specialist team, reads every brief before the review starts, and supplies the finding contract so the brief only has to say what to look for.

Review my manuscript

Related agents

The briefs below share a review with this one; you can run three at a time. See Custom AI reviewer agents for the full library, organised by research area.

• Ethics, consent and data sharing — use it for approval statements and consent scope for deposition.
• Overclaim check — use it for the distance between what was shown and what the abstract says.
• Study section lens — use it when you need the grant reviewer’s question, not the referee’s.