Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals

1 Introduction

• Submission volumes have outgrown the supply of willing and able reviewers, making it harder to secure qualified reviewers and lengthening turnaround times.Editors report difficulty securing the two or three reviewers a typical manuscript requires.Source: §1 · p. 1
• Publication-count incentives and the rapid growth of generative AI tools are linked to rising submission volumes, especially in fields less dependent on physical experiments.The acceleration in submissions since 2024 is linked to generative AI tools.Source: §1 · p. 1
• LLMs are already used for review tasks, including summarizing manuscripts and drafting reviews, while journals and conferences have begun deploying them.A 2025 survey found 53% of reviewers used AI for peer-review tasks; AAAI-26 provided AI-generated reviews to all 22,977 main-track submissions.Source: §1 · p. 1
• Prior studies report model-review overlap, prestige bias, vulnerability to cosmetic rewriting, and increased acceptance probability near ICLR’s decision threshold.One assessment found AI-review overlap of 21%, versus 3% among human reviewers.Source: §1 · pp. 1–2
• No prior study had examined how LLMs would review papers already published in top-tier journals, a way to assess recognition of sound work and detection of flaws.The authors frame these abilities as specificity and sensitivity, respectively.Source: §1 · p. 2

2.1 Manuscript corpus

• The corpus comprised 200 research and review articles from 20 peer-reviewed journals, published in 2026 under CC BY 4.0 and meeting minimum-length and publication-status criteria.The journals spanned general science, life science, and clinical medicine, from broad-acceptance venues to flagship journals.Source: §2.1 · p. 2

2.2 Reviewer models and review protocol

• Four LLMs reviewed each manuscript under first-submission and revision framings, with the target journal named and prior publication concealed.The prompts requested a summary, major and minor concerns, and a final decision; Figure 1 summarizes the corpus and experiments.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 Error-injection experiment

• An off-panel model planted one severe, field-specific statistical or methodological flaw per manuscript, and detection was scored by a second off-panel model.Only definite catches counted as detection; partial verdicts counted as misses.Source: §2.3 · pp. 3–4

2.4 Review-quality judging experiment

• Two off-panel judges compared four anonymized reviews per manuscript, grouping concerns, rating their relevance and each review’s constructiveness, and selecting the best review.Review labels were randomized independently for each manuscript, and the judges were not told the reports were AI-generated.Source: §2.4 · p. 4

2.5 Statistical analysis

• Analyses used rank-based tests for ordinal decisions and paired changes, clustered checks for model comparisons, and agreement measures for judge assessments.The study used Kruskal-Wallis and Friedman tests for decision severity, Wilcoxon signed-rank tests for paired changes, and Cohen’s κ and Spearman correlation for judge agreement.Source: §2.5 · pp. 4–5

3.1 Reviewer models disagree sharply on the same manuscripts

• 100% of manuscripts received major-revision or rejection recommendations from GPT-5.6 Sol, compared with 17.5% from Gemini 3.7 Flash; model distributions differed (p < 0.001).GLM-5.3 Flash recommended major revision or rejection for 90.0% and DeepSeek v4 Flash for 95.0%.Source: Figure 2 · p. 5, §3.1 · p. 5
• Recommendations did not differ significantly across journals (p = 0.24) or between Nature, Science, and Cell and Scientific Reports (p = 0.081).Mean recommendations were 1.74 at Nature, Science, and Cell and 1.98 at Scientific Reports.Source: §3.1 · p. 5
• 21/200 manuscripts (10.5%) received an identical decision from all four models, always unanimous major revision and never unanimous acceptance.These manuscripts were spread across 14 of the 20 journals.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2 Presenting the same manuscript as a revision makes every model more lenient

• 49.0% of paired reviews became milder when framed as revisions, and pooled mean severity fell from 1.77 to 1.26 (p < 0.001).Every model became more lenient; accept recommendations rose from 3.9% to 17.8%, while major revision fell from 70.3% to 42.3%.Source: §3.2 · p. 5, Figure 4 · p. 6

3.3 Reviewer models miss most single planted flaws, and detection rate itself varies by model

• 60.5% of planted flaws were detected by GPT-5.6 Sol, versus 17.0% by DeepSeek v4 Flash; detection rates varied across models (p < 0.001).GLM-5.3 Flash detected 44.0% and Gemini 3.7 Flash 18.0%; the two weakest models were not distinguishable from each other.Source: §3.3 · p. 6, Figure 5 · p. 7
• Planting a flaw changed mean recommendation severity by only 0.02 categories overall (p > 0.35 for every model).Among reviews where judges confirmed flaw detection, recommendations were unchanged in 84.2%, despite a 0.07-category mean severity increase (p = 0.006).Source: §3.3 · p. 7

3.4 One model’s reviews are judged best almost universally, and cross-model agreement tracks genuine relevance

• GPT-5.6 Sol’s review was named best in 94.5% of manuscripts by one judge and 75.5% by the other; neither Gemini nor DeepSeek was ever named best.GPT-5.6 Sol also raised the most relevant concerns, was rated most constructive, and had the lowest nitpicking rate under both judges.Source: §3.4 · p. 7, Figure 6 · p. 8

4 Discussion

• All four tested LLMs showed low specificity and sensitivity, often recommending major revision or rejection for papers already published in top journals.The authors characterize the recommendations as variable, model-dependent, and generally harsh.Source: §4 · p. 8
• The same published manuscript could receive different recommendations depending on which model was queried; no manuscript was unanimously accepted.The only unanimous decision was major revision, for 21 of 200 manuscripts.Source: §4 · p. 8
• LLMs recommended major revision or rejection for most manuscripts, a tendency the authors suggest may reflect incentives to appear authoritative and avoid accepting submissions at face value.The proposed explanation is tentative: models may optimize for major revision as a middle ground between rejection and acceptance.Source: §4 · p. 9
• A planted blocking error barely shifted decisions even when identified, while revision-round framing shifted decisions toward acceptance despite unchanged content.The authors interpret these findings as evidence of recommendations biased toward indecision.Source: §4 · p. 9
• The authors argue that machine reviews may inform editorial judgment but should not substitute for human judgment or take on a certification role.They distinguish a machine’s formal appearance of authority from the accountability of a human reviewer invited by an editor.Source: §4 · p. 9
• Model comments, especially those converging across models, may help identify manuscript vulnerabilities, but their value requires expert assessment in future studies.The authors propose testing whether such convergence reveals issues missed by human reviewers.Source: §4 · p. 9
• The study assumes papers in the included journals provide a reasonable basis of trust, did not assess reviews with experts, and used models chosen partly for financial and ecological reasons.Some selected 2026 articles may have been available earlier as preprints and entered model training data.Source: §4 · p. 9

4.1 Conclusion

• The conclusion treats LLM recommendations as variable signals shaped by model choice and question framing, making postdeployment assessment against journal-specific benchmarks essential.The authors note that new models are released continuously.Source: §4.1 · p. 10

Appendix A. Exact prompts

• The prompts are reproduced verbatim from the pipeline code.The only substitutions are the {journal} and {category_guidance} placeholders, as noted.Source: Appendix A · p. 12
• The stated prompt substitutions are limited to the {journal} and {category_guidance} placeholders.Source: Appendix A · p. 12
• The appendix identifies the placeholder substitutions as the sole departure from the pipeline prompts.Source: Appendix A · p. 12

A.2 Reviewer system prompt

• The model is assigned the role of a journal reviewer and shown the full text of a submitted manuscript.Source: §A.1 · p. 12
• The report structure includes a manuscript summary, major concerns, and minor concerns.Source: §A.1 · p. 12
• The prompt requires the response to end with exactly one decision line: accept, minor revision, major revision, or reject.Source: §A.1 · p. 12

A.3 Error-injector system prompt

• The injector receives a published paper treated as presumably sound and must identify the central conclusion the title and abstract are built around.Source: §A.2 · p. 12
• The flaw must be substantive, specific to the paper’s field, data, and method, and serious enough to warrant major revision or rejection.Source: §A.2 · p. 12
• The flaw should be hard to spot and introduced with the smallest reasonable edit, typically replacing or removing a short passage.Source: §A.2 · p. 12
• The injector must quote the original passage exactly, provide replacement text or an empty string, and return the specified JSON fields.Source: §A.2 · pp. 12–13

• The statistical concern is a mismatch between the analysis and the data structure or design.Source: Category block: statistical · p. 13
• Possible examples include treating clustered or repeated observations as independent, undisclosed multiple testing, ignored censoring, or unmodeled confounding or stratification.Source: Category block: statistical · p. 13
• The prompt asks the evaluator to use judgment and select the concern that best fits the paper.Source: Category block: statistical · p. 13

• A methodological flaw may arise when an outcome is produced with undisclosed access to information that could bias the judgment.Source: Category block: methodological · p. 13
• Another target is an instrument or measurement whose specificity or validity limits what can be claimed.Source: Category block: methodological · p. 13
• The prompt asks the evaluator to select the concern that is most natural for the paper’s actual method.Source: Category block: methodological · p. 13

• The judge receives the manuscript’s central conclusion, the planted flaw, and the full review of the doctored manuscript.Source: §A.3 · p. 13
• A flaw counts as identified only when the review explicitly names that specific problem in the same location, not when it raises generic or unrelated concerns.Source: §A.3 · p. 13
• The judge must answer with one word: yes, no, or partial.Source: §A.3 · p. 13

• The judge evaluates four anonymous reviews labeled A through D solely on their content.Source: §A.4 · p. 13
• It groups substantively equivalent major methodological, statistical, or interpretive concerns and records which reviews raised each cluster.Source: §A.4 · p. 13
• Each concern cluster is classified as relevant if it could affect conclusions or acceptance, or nitpicking if it is minor or unlikely to matter.Source: §A.4 · p. 13
• Each review receives a 1–5 constructiveness rating based on actionable, specific guidance, independently of its recommendation’s harshness or leniency.Source: §A.4 · p. 13
• The judge selects the single review it would most want an author to receive and gives a one-sentence reason.Source: §A.4 · p. 13
• The required output is a JSON object containing concern summaries and categories, review constructiveness scores, and the best review with its reason.Source: §A.4 · pp. 13–14