On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists

• Prior verdict-level evaluations miss criticism quality; independent NeurIPS committees disagreed on 49.5% of 2014 and 50.6% of 2021 submissions [Cortes and Lawrence, 2021, Beygelzimer et al., 2023].Verdict agreement does not establish whether individual criticisms are factually correct, important, or supported by credible evidence.Source: §1 · p. 2, Figure 1 · p. 3
• GPT-5.2’s criticism composite exceeded each paper’s top-rated human reviewer (60.0% vs. 48.2%, p = 0.009), while Claude Opus 4.5 and Gemini 3.0 Pro were statistically indistinguishable.AI reviewers raised more incorrect items than top-rated humans, but their correct items were more often significant and well-evidenced.Source: §1 · p. 2
• AI reviewers recovered 27.1% of human items versus 25.8% for another human, but AI-AI overlap was 21.0% versus 3.1% for human-human pairs.Roughly one quarter of AI items had no similar human counterpart; the higher AI-AI overlap suggests that an AI reviewer panel could reduce perspective diversity.Source: §1 · p. 2
• Researchers identified 16 recurring AI failure modes, led by subfield-convention gaps, long-context losses, and overcritical treatment of minor issues.These three weaknesses account for most incorrect items.Source: §1 · p. 2
• The study uses 45 scientists across three sciences to assess 2,960 atomic criticisms from reviews of 82 Nature-family papers over 469 hours.Scientists rated each criticism for correctness, significance, and sufficiency of evidence.Source: §1 · p. 2
• The authors release PEERREVIEW BENCH for continued evaluation and CMU PAPER REVIEWER for pre-submission manuscript feedback.On PEERREVIEW BENCH, GPT-5.4, DeepSeek-V4-Pro, and Claude-Opus-4.7 achieved 41.4%, 48.5%, and 50.5% F1, respectively. CMU PAPER REVIEWER’s items were more often correct, significant, and well-evidenced than those from Stanford Agentic Reviewer and OpenAIReview (95.5% vs. 59.8% and 57.6%, respectively).Source: §1 · p. 2

2.1 Methodology for reviewing a review

• Each atomic criticism is rated for correctness, conditional significance, and conditional evidence sufficiency, keeping distinct dimensions whose judgments depend on one another.Human reviews are manually divided into items using conventional markers such as bullets, enumeration, and paragraph transitions.Source: §2.1 · pp. 3–4

2.2 Scope of papers for the expert annotation study

• The corpus contains 82 Nature-family papers published from 2020 to 2025: 38 in Physical Sciences, 30 in Biological Sciences, and 14 in Health Sciences.Most submissions also include additional materials; AI reviewers could access supplementary materials for 83%, separately submitted figures for 76%, and source code for 74% of papers.Source: §2.2 · pp. 4–5

2.3 Reviewers: official human reviewers and frontier-LLM agents

• The comparison pairs first-round official human reviews, retaining up to three reviewers per paper, with GPT-5.2, Claude Opus 4.5, and Gemini 3.0 Pro agents producing up to five items each.The agents had access to paper files and tools including web search, with publisher domains blocked; each generated one review per paper-model pair.Source: §2.3 · pp. 5–6

2.4 Meta-reviewers: domain scientist annotators

• Forty-five domain scientists from 25 institutions produced 109 meta-reviews across 82 papers, contributing 469 hours of expert annotation.A second scientist independently annotated 27 papers, yielding 908 doubly annotated items; agreement was almost perfect for correctness and evidence sufficiency, and moderate for significance.Source: §2.4 · p. 6, Table 2 · p. 6
• Meta-reviewers also reported AI-raised criticisms others missed and completed paper-level surveys comparing AI reviewers with each paper’s top- and lowest-rated human reviewers.Results use papers, not individual review items, as the unit of analysis (n = 82), pooling item annotations while accounting for repeated ratings within papers.Source: §2.4 · p. 6

• The comparisons use paper-level means across 82 papers, and the results also hold in item-level rates and a GLMM robustness analysis.AI reviews were capped at five items per paper, unlike human reviews; the study’s takeaway concerns frontier models with tool-based access to the paper’s full source, code, and external literature.Source: §3 · pp. 6, 8
• GPT-5.2 correctness is 86.2% vs. the top human’s 92.3%; Claude Opus 4.5 is 83.7% and Gemini 3.0 Pro is 81.9%, while all three raise more significant criticisms.Among correct items, significance correlations are r = +0.49, +0.30, and +0.42, respectively (all p ≤ .028); GPT-5.2 and Claude Opus 4.5 have slightly higher evidence sufficiency, while Gemini 3.0 Pro is statistically indistinguishable. Against the lowest-rated human, all three match or exceed on every dimension, with the largest gains in significance.Source: §3 · pp. 6–7, Table 3 · p. 7
• 48.6%: experts judged GPT-5.2 to match or exceed the top-rated human on that share of papers, versus 73.4% against the lowest-rated human.Claude Opus 4.5 and Gemini 3.0 Pro matched or exceeded the top-rated human on 30.3% and 32.1% of papers, and the lowest-rated human on 59.6% and 68.8%, respectively.Source: Table 5 · p. 7, §3 · p. 7
• 60.0% vs. 48.2%: GPT-5.2’s fully positive item rate exceeds the top-rated human’s (p = .009), and all three AI reviewers exceed the lowest-rated human.Claude Opus 4.5 and Gemini 3.0 Pro are statistically indistinguishable from the top-rated human; GPT-5.2 scores significantly above both other AI reviewers.Source: §3 · p. 7, Table 4 · p. 8

4 To what extent do AI reviews overlap with human reviews?

• Review-item similarity is defined by shared target and criticism, regardless of whether the reviewers use the same evidence; the four-way taxonomy separates this from exact-evidence matches and near-paraphrases.GPT-5.4 achieved 92.7% binary and 83.5% four-way accuracy on 164 manually annotated pairs; prevalence estimates were Rogan-Gladen-corrected with 95% cluster-bootstrap confidence intervals.Source: §4 · pp. 8–9
• 26.0% of AI-raised items have no similar human counterpart, and these uncovered items are similarly correct and evidence-sufficient but less often highly significant than matched items.Uncovered versus matched items were 81.8% versus 84.4% correct (p = 0.23), 93.5% versus 93.2% evidence-sufficient (p = 0.97), and 57.6% versus 63.5% highly significant (p < 0.001).Source: §4 · p. 9, Table 7 · p. 9
• 83.0% of human-raised items are covered by a three-AI panel at the same-target level, but only 46.3% at the same-criticism level.The panel therefore identifies most of the same parts of papers as human reviewers but only about half of the specific criticisms with the same substance.Source: §4 · p. 9, Figure 4 · p. 10
• 20.9% of AI–AI item pairs share the same target and criticism, versus 3.4% of human–human pairs, while AI–human overlap is 5.1%.The H–A estimate is only slightly above the H–H baseline, with substantially overlapping confidence intervals; replacing one human reviewer with one AI reviewer leaves pairwise diversity nearly intact.Source: §4 · p. 9, Figure 4 · p. 10
• Only 3.4% of human–human item pairs share the same target and criticism; 96.6% raise different criticisms or address different targets.The remaining pairs comprise 8.4% with different criticisms about the same target and 88.3% with entirely different targets, illustrating the distinct concerns human reviewers contribute.Source: §4 · p. 10, Figure 4 · p. 10

5.1 Failure cases: Limitations of AI reviewers

• 189/260 weakness comments fell into five frequent patterns: severity miscalibration, missed information, redundant AI critiques, and vague or verbose reviews.Critiques can be technically reasonable yet overstate the severity of a concern because the reviewer lacks knowledge of what the field considers normal.Source: §5.1 · p. 11
• 37 comments described cases where AI reviewers called information missing despite its presence elsewhere in the manuscript, reflecting limited long-context management.The example review missed a calibration procedure that directly addressed its concern; the passage attributes such failures to context summarization that loses information across files.Source: §5.1 · pp. 12–13

5.2 Successful cases: Strengths of AI reviewers

• 115/132 strength comments described diligent scrutiny, including statistical and methodological rigor, code inspection, domain-specific technical depth, and cross-section consistency.Statistical and methodological rigor was the largest category (45 comments); source-code inspection (28) surfaced implementation bugs, data leakage, and mismatches not evident from manuscript text alone.Source: §5.2 · p. 13
• Code inspection exposed a wavefunction-ansatz ambiguity between the paper and implementation, a reproducibility issue that only the AI reviewer in the panel checked.The expert noted that human reviewers typically do not cross-check manuscript mathematics against code because it is too time-consuming.Source: §5.2 · pp. 13–14
• The authors identify severity calibration, long-context memory, and reduced redundancy as engineering targets, while preserving tool-enabled scrutiny.Their takeaway is that current agentic AI reviewers add value on rigor- and code-heavy aspects but fail systematically on field-context aspects.Source: §5.2 · p. 14

6.1 Can we mimic the fineness of expert validation and automatically evaluate AI reviewers?

• On 908 dual-annotated review items, Claude-Opus-4.7 achieves 87.9% / 56.7% / 85.6% across correctness, significance, and evidence sufficiency, near human-human agreement of 85.8% / 59.9% / 88.0%.The calibration tests whether AI agents can serve as meta-reviewers using the study’s three-axis judgments.Source: §6.1 · p. 14
• Claude-Opus-4.5 leads PEERREVIEW BENCH with F1 = 50.89, precision 75.49%, and recall 38.39%, leaving a substantial gap from the human ceiling on both metrics.The 78-paper benchmark defines precision as the fraction of AI-raised items judged fully positive and recall as the fraction of fully positive human review items also raised by the AI.Source: §6.1 · p. 14
• GPT-5.4 reaches the highest precision at 93.81% but 26.55% recall, while Gemini-3.0-Pro-Preview records 53.35% precision and 37.65% recall; Claude-Opus models balance the two best.Source: §6.1 · p. 14

6.2 CMU PAPER REVIEWER: An open-source reviewer platform for pre-submission feedback

• The open-source CMU PAPER REVIEWER pairs critiques with proposed manuscript edits or runnable code patches, grounds severity in stated limitations, enables debate, and can filter citations by publication date.Source: §6.2 · p. 15
• F1 = 58.64 for CMU PAPER REVIEWER with GPT-5.4 at fifteen items per paper, versus 51.65 for Stanford Agentic Reviewer and 47.88 for OpenAIReview.The platform is intended for pre-submission feedback and should not be used in official review processes that prohibit AI reviewers.Source: §6.2 · p. 15

7 Conclusion

• AI reviewers produce a significantly higher fraction of review items per paper than top-rated human reviewers on the composite, despite more factual errors; the advantage comes from significance and evidence sufficiency among correct items.The composite combines correctness, significance, and evidence sufficiency.Source: §7 · p. 15
• AI reviewers contribute thorough cross-reference checks, code-level inspection, and valid criticisms that human reviewers miss.Source: §7 · p. 15
• AI reviewers’ reviews overlap far more with one another than human reviews do, meaning an all-AI panel would substantially narrow the diversity of perspectives.Source: §7 · p. 15
• AI reviewers show recurring weaknesses in subfield-specific conventions, tracking long papers and supplements, and calibrating criticism so minor issues are not inflated.Source: §7 · p. 15
• Developers should address correctness and criticism calibration through better subfield-norm understanding, long-context management, and alignment with expert judgment; PEERREVIEW BENCH can track progress.Source: §7 · p. 15
• AI reviews should be assessed on correctness, significance, and evidence sufficiency rather than against human reviews as an implicit gold standard.Source: §7 · p. 15
• The CMU PAPER REVIEWER offers authors pre-submission feedback, while open questions include whether tradeoffs persist as models improve, findings generalize beyond Nature-family papers, and what governance is needed.Source: §7 · p. 15

A Related Work

• Rising submissions and reviewer fatigue accompany 100 to 160-day median submission-to-publication times; NeurIPS studies found 23% to 26% inconsistent decisions across committees.Other documented challenges include reviewer miscalibration, reviewer-author collusion, and undetected major manuscript errors.Source: A Related Work · p. 21
• AI assistance has moved into peer-review practice, from informal researcher use to conference pilots and AAAI-26 deployment on all 22,977 main-track submissions.Reported pilots include NeurIPS 2024 and ICLR 2025; AAAI-26 participants found AI reviews useful and preferred them on some dimensions [Goldberg et al., 2024, Thakkar et al., 2025, Biswas et al., 2026].Source: A Related Work · p. 21
• LLM-as-a-judge studies report strong correlations with human judgments but document position, verbosity, and self-preference biases; AI-review research spans review generation and process-improvement tools.The cited work includes multi-agent, fine-tuned, bias-aware, deep-thinking, and RL-trained review systems, as well as feedback agents and author checklists [D’Arcy et al., 2024, Yu et al., 2024, Tyser et al., 2024, Zhu et al., 2025, Zeng et al., 2025, Thakkar et al., 2026, Goldberg et al., 2024].Source: A Related Work · p. 21
• Prior evaluations usually compare review-level scores or text with human reviews, while ReviewEval, TreeReview, and FLAWS add finer-grained claim-, comment-, or error-level analysis.AI reviewers can also match human reviewers in predicting paper outcomes.Source: A Related Work · pp. 21–22
• The study evaluates individual AI and human review items under the same protocol, using field experts and separate correctness, significance, and evidence-sufficiency axes.Table 10 summarizes four design properties: bidirectional annotation, per-comment analysis, multi-axis assessment, and domain experts; no prior framework satisfies all four together.Source: A Related Work · p. 22, Table 10 · p. 22

B Extended: Expert annotation study design and experimental setup

• Appendix B collects procedural and statistical details supporting § 2, with subsections ordered to follow the main text.Source: B Extended: Expert annotation study… · p. 23

B.1 Subject category breakdown

• Table 11 presents the subject-category breakdown of the study’s 82 papers, categorized using the Nature Communications subject taxonomy5.Source: §B.1 · p. 23, Table 11 · p. 24

B.2 Evaluation criteria for reviewing a paper

• The six Nature Portfolio criteria are prioritized for AI review selection, with validity ahead of clarity and context; annotators instead judge each item independently.The criteria cover validity, conclusions, originality and significance, data and methodology, statistics and uncertainties, and clarity and context.Source: §B.2 · p. 24

B.3 Evaluation criteria for reviewing a review

• Review items are assessed through a cascade: correctness first, significance only if correct, then evidence sufficiency only if correct and at least marginally significant.Correctness is binary, significance has three levels, and evidence sufficiency is binary.Source: §B.3 · p. 25

B.4 Processing official peer review files

• Human reviews are restricted to first-round comments on the pre-review manuscript, capped at three reviewers, and manually segmented into atomic criticisms.Editor letters and author rebuttals are excluded; bullet-pointed reviews are segmented at bullets and paragraph reviews at explicit enumeration markers.Source: §B.4 · p. 25

B.5 AI reviewer configuration details

• The three AI reviewers share an OpenHands framework, access to paper files and tools, and Tavily search with safeguards against retrieving benchmark papers’ published versions or reviews.The configuration includes a conversation-history condenser and filters four domains at both the tool and prompt levels.Source: §B.5 · pp. 25–26

B.6 AI reviewer prompt

• The shared prompt asks agents to produce at most five factually correct, significant, evidence-supported atomic criticisms, prioritizing the paper-review criteria.It directs agents to inspect paper files, code, and relevant literature, and to verify potential criticisms before including them.Source: §B.6 · pp. 27–29

B.7 Domain scientist recruitment

• The 45 domain scientists were recruited through a three-stage process matching candidates to papers they felt qualified to meta-review, with about three hours of annotation per paper.The recruitment ran from September 2025 to April 2026, and participants had at least a month to complete annotations.Source: §B.7 · p. 30

B.8 Annotation guidelines

• Annotators evaluate atomic review items and complete a paper-level survey comparing reviewers; these meta-reviews support human–AI comparisons and PEERREVIEW BENCH.The survey asks annotators to identify the best and worst human reviews, assess AI reviews against those references, and note issues AI reviewers uniquely caught.Source: §B.8 · pp. 30–31

B.9 Inter-annotator agreement details

• Table 14 presents inter-annotator agreement across 908 doubly annotated items from 27 papers, broken down by reviewer type and evaluation dimension.The number of items decreases across dimensions because later judgments are conditional on earlier ones.Source: Table 14 · p. 31
• Correctness and evidence-sufficiency judgments showed high inter-annotator agreement, while significance agreement was moderate across human and AI items.Raw agreement on AI versus human items was 87.6% versus 84.7% for correctness, 59.3% versus 60.2% for significance, and 94.3% versus 83.9% for evidence sufficiency.Source: §B.9 · p. 31

C.1 Item-level descriptive statistics

• Item-level rates differ from paper-level means because item-level rates weight papers by item count, whereas paper-level means weight each paper equally.Significance is annotated only for items rated Correct, and evidence sufficiency only for items rated Correct and at least Marginally Significant; Table 15 reports item-level counts and rates across all three dimensions.Source: §C.1 · p. 31, Table 15 · p. 32

C.2 Complete pairwise paired comparisons

• The complete analysis compares all five reviewer groups pairwise on three metrics, using paired t-tests for binary metrics and Wilcoxon signed-rank tests for ordinal significance.Effect sizes are Cohen’s d for correctness and evidence sufficiency, and rank-biserial correlation r for significance.Source: §C.2 · p. 32, Table 16 · p. 32
• All three AI reviewers raised more significant issues than the Lowest-Rated Human, while GPT-5.2 exceeded Claude Opus 4.5 on significance and Gemini 3.0 Pro on evidence sufficiency.The AI-versus-Lowest-Rated Human significance effects were large (r = −0.43 to −0.56); GPT-5.2’s advantage over Claude was r = +0.30, p < .05, and its evidence-sufficiency advantage over Gemini was d = +0.31, p < .01. AI correctness gaps versus the Lowest-Rated Human were directionally negative but smaller than the Top-Rated Human gaps and nonsignificant; Claude Opus 4.5 and Gemini 3.0 Pro were indistinguishable on significance and correctness.Source: §C.2 · p. 32, Table 16 · p. 32

C.3 Generalized linear mixed-effects model robustness analysis

• The GLMM models item-level outcomes with a paper-level random intercept, accounting for items nested within papers and separating within-paper from between-paper variance.The paper-level ICC of 0.25 to 0.29 indicates that roughly one quarter of latent log-odds variance lies between papers.Source: §C.3 · pp. 32–33
• For significance, the model tests whether a criticism reaches the highest rating, estimating the probability of passing that bar conditional on being correct.This binomial GLMM cut point keeps the three metrics on a common log-odds scale.Source: §C.3 · p. 33
• Every AI reviewer has significantly lower correctness and higher significance than the top-rated human; GPT-5.2 and Claude Opus 4.5 also have higher evidence sufficiency, while Gemini 3.0 Pro is indistinguishable.The GLMM strengthens some borderline paired-analysis contrasts, including Top-Rated vs GPT-5.2 correctness (paired p = .046; GLMM p < .001).Source: §C.3 · p. 33, Table 17 · p. 33

D Extended: To what extent do AI reviewers overlap with human reviewers?

• Across 82 papers, the similarity pipeline compares review items pairwise across six reviews per paper, classifying 65,704 item pairs into four categories.The categories distinguish different targets, same target with different criticism, same target and criticism with different evidence, and agreement on target, criticism, and evidence; only the last two count as similar.Source: D Extended: To what extent do AI… · p. 33, Figure 10 · p. 34

D.1 Detailed similarity breakdown

• The extended appendix reports the full four-category similarity distribution for Human-Human, AI-AI, and Human-AI pairs, including within- and between-reviewer splits for same-group pairs.The added same-reviewer and same-model rows capture the diversity of items raised within one review, beyond agreement between different reviewers.Source: D Extended: To what extent do AI… · p. 33, §D.1 · p. 34
• Human within-review similarity is 6.0% versus 3.4% between reviewers, whereas AI same-model similarity is 13.5% versus 20.9% between different models.This pattern is consistent with individual AI reviewers diversifying their criticisms while different models converge on shared concerns.Source: §D.1 · p. 34
• At the between-reviewer level, same-target but different-criticism rates are H-H 8.4%, A-A 20.2%, and H-A 14.8%.This topical co-occurrence accounts for a substantial share of the H-A difference from the H-H baseline, suggesting humans and AI often flag the same paper areas without matching specific criticisms.Source: §D.1 · p. 34

D.2 Similarity judge calibration and selection

• GPT-5.4 led the similarity-judge candidates at 83.5% 4-way accuracy, while LLM judges reached 89–93% binary and 80–84% 4-way accuracy.The calibration set comprised 164 pairs across 32 papers and was treated as calibration, not an independent benchmark, because its labels came from the study’s annotation pool.Source: Table 19 · p. 35, §D.2 · p. 35
• GPT-5.4’s binary judge had 87.1% sensitivity and 96.8% specificity, but applying these rates to Human-Human pairs assumes untested transferability.The correction also assumes constant error rates across 65,704 pairs.Source: §D.2 · pp. 35–36, Table 20 · p. 36

E.1 Detailed case-study examples

• The appendix provides 24 case studies across nine categories and lists all 442 substantive expert comments underlying the strength/weakness classification.The examples preserve review items, expert commentary, and discussion; the comments are organized by category for traceability.Source: E Extended: What are the concrete… · p. 36, §E.1 · p. 36

D.3 Raw versus Rogan-Gladen-corrected comparison

• The correction lowered same-target, same-criticism estimates and raised not-similar estimates; changes were largest for H-H and H-A pairs and under one percentage point for A-A pairs.The main-text conclusions were qualitatively unchanged by using raw rather than corrected values.Source: §D.3 · p. 36, Table 21 · p. 37

E.1.1 W1: Missing community / field norms

• AI reviewers can make technically accurate criticisms but misjudge severity when they lack field-specific norms for what is standard practice or how claims are framed.Examples include treating unpublished collaboration-held fit artifacts as a publication gap, or reading a bounded ecological result as an unconditional universal claim.Source: §E.1.1 · pp. 37–38

E.1.2 W2: Over-harsh, out-of-scope, or unrealistic demands

• AI criticisms may be factually valid yet over-harsh when they demand out-of-scope or infeasible work rather than calibrating concerns to the paper’s claims and revision options.Examples include requesting a single-pulse invasion experiment and flagging healthy-donor data as a validity problem when a relevant patient-derived atlas is unavailable.Source: §E.1.2 · pp. 39–40
• A reproducibility critique identified missing fitting details, but its request for deterministic scripts and sensitivity analyses exceeded what noisy experimental s-SNOM measurements could realistically support.The study notes that the reported values already averaged multiple runs, making the requested computational-style reproducibility standard unrealistic for the experiment.Source: §E.1.2 · pp. 40–41

E.1.3 W3: Paper explicitly states X, AI says missing

• In 37 cases, AI reviewers said information was missing when it was present, often because they failed to manage or integrate long-context material.Failures ranged from overlooking imaging methods under an unrecognized heading to missing an explicit PM2.5 calibration procedure; another critique partly missed validation on other organisms.Source: §E.1.3 · pp. 41–42

E.1.4 W4: Redundancy across the three AI reviewers

• In 28 cases, the three AI reviewers raised substantially overlapping criticisms, so additional AI reviews often contributed little new coverage.For example, all three questioned the NequIP–DeepMD data-efficiency comparison, framing the shared concern as correlated data, missing learning curves, or a missing control; they also converged on the temperature–COVID mortality independence assumption.Source: §E.1.4 · pp. 42, 44, 45
• The three AI reviewers often elaborate on the same underlying issue rather than identify distinct problems, as in their shared criticism of pandemic-independent mortality claims.Their additions differed in emphasis—literature references, quantitative estimates, or formal statistical framing—but the core observation was one any reviewer could have made.Source: §E.1.4 · p. 45

E.1.5 W5: Vague, verbose, or without actionable recommendation

• 24 W5 cases involved critiques that were verbose and diffuse or identified a problem without suggesting how authors could address it.In one example, a defensible safety critique was buried in six paragraphs; in another, a precise landscape-scale modeling criticism offered no revision path.Source: §E.1.5 · pp. 45–46

E.1.6 W8: Citing evidence that appeared after the preprint

• 9 W8 cases relied on evidence or practices that postdated the preprint, so critiques could judge earlier work against knowledge unavailable at submission.Examples include later analysis of a neural-network method and fine-tuning practices applied to an early transformer-based polymer study; restricting literature searches by submission date could mitigate this failure.Source: §E.1.6 · pp. 47–48

E.1.7 S1: Statistical and methodological rigor

• AI reviewers identified statistical and methodological concerns including test-set model selection, potentially underpowered tests, and violations of independence assumptions that human reviewers had missed.In one case, code inspection showed that test metrics were used for model selection despite no validation split; another expert noted correlated regional observations violated K-S test assumptions.Source: §E.1.7 · pp. 48–49
• Single-capture imaging comparisons with qualitative claims and limited uncertainty reporting provided insufficient evidence to establish robust improvements.An expert agreed that the finding needed quantification; the deconvolution pipeline’s sensitivity to initialization and regularization further limits what a single reconstruction can establish.Source: §E.1.7 · pp. 49–50

E.1.8 S2: Inspecting the submitted source code

• AI reviewers used submitted code to expose mismatches between manuscript claims and implementation, including an internally ambiguous wavefunction description and test-edge leakage into training.Experts confirmed the leakage and noted that cross-checking code against the manuscript can reveal issues human reviewers often do not investigate because it is time-consuming.Source: §E.1.8 · p. 51
• 800 Hz claimed sampling contrasted with an implemented rate of approximately 2 Hz, making the reported 0–400 Hz frequency content impossible to reproduce with the described wireless setup.The code’s 500 ms loop delay and 9600-baud serial configuration also contradict the claimed rate; an expert said the core data could be wrong if the code was correct.Source: §E.1.8 · p. 52

E.1.9 S3: Domain-specific technical depth

• A domain-specific critique found that experiments targeting binary amplitude-pattern recovery did not substantiate claims of transmission of arbitrary optical fields.The passage distinguishes amplitude-only recovery from complex-field retrieval and cautions against extrapolating the method’s performance without direct evidence.Source: §E.1.9 · p. 53
• AI reviewers caught specialized technical claims, including unsupported stereocontrol and limits of quantitative modeling, while experts sometimes narrowed the severity of those critiques.For multimode-fiber imaging, the expert distinguishes the demonstrated amplitude-to-amplitude relation from the much harder recovery of arbitrary complex fields. For the pharmaceutical example, compound 4v’s stereogenic center makes missing diastereoselectivity evidence consequential; for s-SNOM, the PDM critique was valid, but a better model would reduce error rather than overturn the main findings.Source: §E.1.9 · pp. 53–54

E.2 Complete categorized expert comments

• The classification appendix lists 442 substantive comments and 121 paper-level descriptive comments, with specified exclusions and traceable entries in the HuggingFace dataset.Comments about human reviewers and comments with explicit item-number references are handled separately; listed comments are sorted by source, paper, and reviewer.Source: §E.2 · p. 54

Category summary

• W1 (54 comments) and W2 (46) were the largest weakness categories, while S1 statistical/methodological rigor had 45 comments and S3 specialized-field catches had 27.The category summary also reports 37 W3 comments, 28 W4, and 40 unspecified residual weaknesses; the listed S categories range from 7 to 45 comments.Source: Category summary · p. 55

W1: Missing community / field norms — n = 54

• Experts often said AI critiques overlooked field norms, practical data constraints, or a paper’s proof-of-concept and engineering scope.Comments cite limited feasible observational data, accepted community practices, and cases where engineering novelty—not clinical deployment—was the paper’s focus; some criticisms were technically valid but overemphasized in context.Source: W1: Missing community / field norms —… · pp. 56–57

W2: Over-harsh / out-of-scope / unrealistic — n = 46

• Experts described some AI criticisms as too harsh, out of scope, or marginal, even when the underlying concern was valid.One expert said an additional experiment could be requested without claiming the authors’ work was completely wrong; others called a safety-related statistical issue marginal or characterized an architectural choice as valid.Source: W2: Over-harsh / out-of-scope /… · p. 58
• AI critiques were often technically correct but judged too harsh, out of scope, or unrealistic given field complexity and what authors could reasonably address.Experts described demands for work that was beyond the paper’s scope or not realistically resolvable, and concerns that overstated minor limitations.Source: W2: Over-harsh / out-of-scope /… · pp. 58–59

W3: Paper explicitly states X, AI says missing — n = 37

• AI reviewers sometimes claimed information was missing despite its presence in the manuscript, and experts also noted failures to account for stated methods or study context.Examples included overlooking methods text and explicit descriptions of the study’s scope, calibration, or assumptions.Source: W3: Paper explicitly states X, AI says… · p. 60

W4: Redundancy across the 3 AI reviewers — n = 28

• The three AI reviewers frequently raised the same substantive critiques, so their combined reviews could still omit important points raised by human reviewers.Some reviews elaborated on an existing AI critique or added references, but experts judged the core point to be redundant.Source: W4: Redundancy across the 3 AI… · p. 61
• AI reviewers often repeated one another’s major points, and some missed field-specific nuance that human reviewers captured.In one case, an AI reviewer uniquely questioned the use of an assembly-based callset as ground truth, though another reviewer recognized the authors’ intended comparison.Source: W4: Redundancy across the 3 AI… · p. 62

W5: Vague / verbose / no actionable recommendation — n = 24

• AI reviews were often verbose or confusing, and could identify flaws without giving authors constructive, actionable ways to address them.Assessors also described diffuse argumentation and noted that some technically correct critiques obscured the significance of the results.Source: W5: Vague / verbose / no actionable… · pp. 62–63

W6: Trivial / nitpicking — n = 16

• AI reviewers sometimes treated minor matters—such as typos, technicalities, or limited-scope limitations—as more consequential than they were.Examples included a typo framed as a validity risk and a ground-truth coverage limitation judged minor and addressable through clarification.Source: W6: Trivial / nitpicking — n = 16 · p. 63

W7: Technical term-of-art confusion — n = 13

• AI reviewers sometimes misunderstood technical terms or domain-specific concepts, although some raised valid limitations that assessors said should be discussed rather than treated as grounds for rejection.Assessors cited confusion about terminology and scientific mechanisms, alongside a valid size-extensivity concern they considered insufficient to reject the work.Source: W7: Technical term-of-art confusion —… · p. 64

W8: Cites evidence from after the preprint — n = 9

• AI reviewers sometimes supported critiques with evidence published after the preprint, which assessors considered invalid or anachronistic.Assessors noted that chronology and historical context could change how claims or discoveries should be judged.Source: W8: Cites evidence from after the… · p. 64

W9: Over-inflates small code/text inconsistencies — n = 9

• AI reviewers sometimes overstated the implications of small code–text discrepancies that could reflect changes during a paper’s development.Assessors also cautioned that an apparent disagreement between estimates could not be established without a quoted uncertainty.Source: W9: Over-inflates small code/text… · p. 65
• AI reviewers sometimes overstated minor code or text inconsistencies and methodological differences unlikely to change sector rankings.The cited assessment says within-sector variation was modest relative to roughly fivefold between-sector differences, but a sensitivity analysis would be needed to confirm the ranking stays unchanged.Source: W9: Over-inflates small code/text… · p. 65

W10: Criticizes what authors already flagged as a limitation — n = 6

• AI critiques sometimes faulted authors for limitations they had already acknowledged, including a proof-of-concept scope and constrained future analyses.Evaluators noted that separate COVID-period models were flagged as future work because only two years of observations were available for regional-level estimation.Source: W10: Criticizes what authors already… · p. 65

W11: AI misreads a figure or caption — n = 6

• AI reviewers misread figures and captions, including reported barriers and the optimal threshold, and overlooked details such as error bars.One review identified the peak as t=0.4 when Figure 5c showed t=0.3; error bars were present in Figure 2c.Source: W11: AI misreads a figure or caption —… · pp. 65–66

W12: AI misquotes or fabricates a verbatim quote — n = 5

• AI reviewers sometimes misquoted authors or compared sentences describing different parts of a process.Other reported cases involved alleged discrepancies that evaluators could not observe and a term, CuTe, that could not be found in the manuscript or supplement.Source: W12: AI misquotes or fabricates a… · p. 66

W13: AI misses supplementary content — n = 3

• AI reviewers sometimes missed relevant supplementary content, including details and a typo clarified by a supplementary figure.In one case, the main text's number 281 was a typo; Supplementary Figure 3 showed 69281.Source: W13: AI misses supplementary content —… · p. 66

W14: Ignores authors’ own cited prior work — n = 2

• AI reviewers sometimes ignored authors’ cited prior work when evaluating a claim based on that work.Evaluators said the authors supported a nutrient-concentration and interaction-strength claim with previous work that the AI reviewers had not read.Source: W14: Ignores authors’ own cited prior… · p. 66

W15: AI misreads a table — n = 1

• An AI reviewer overinterpreted Table 1, apparently because a typographical error in its values led to an incorrect conclusion.Source: W15: AI misreads a table — n = 1 · p. 66

W16: Cannot analyze figures — only text — n = 1

• AI reviewers sometimes could not analyze figures, instead giving boilerplate feedback based only on text.In one case, the reviewer did not download the figures from Figshare, which would have been needed to assess the critique.Source: W16: Cannot analyze figures — only… · p. 67

W unspecified: Residual: AI judged Not Correct without specific reason — n = 10

• Some AI assessments were judged weak or incorrect, with reviewers noting unsupported concerns and cases where an AI missed the paper’s main point.One assessment explicitly said the claim had not been verified by running the code and relied on an assumption about the authors’ work.Source: W unspecified: Residual: AI judged Not… · p. 67

S1: Statistical / methodology rigor — n = 45

• AI reviewers also identified distinctive technical issues, including mathematical errors, class-imbalance effects, and collinearity, sometimes suggesting sensitivity analyses.Some assessments noted issues such as a suspicious standard-deviation entry, incorrect AUROC calculation, and an entropy equation error.Source: S1: Statistical / methodology rigor —… · p. 68
• AI reviews surfaced varied methodological flaws, including test-set model selection, data leakage, spatial autocorrelation, and uncertainty or validation gaps.Assessments also highlighted missing baselines and variance estimates, and critiques of simulation settings and theoretical assumptions.Source: S1: Statistical / methodology rigor —… · pp. 68–69

S2: Code reading — n = 28

• Experts praised AI reviewers for inspecting authors’ code, finding discrepancies, and verifying claims such as data leakage and test-metric model selection.One reviewer noted that code inspection can be time-consuming for humans; other assessments expressed uncertainty when code had not been checked in full.Source: S2: Code reading — n = 28 · p. 69
• AI reviewers’ code inspections uncovered consequential discrepancies between manuscripts and implementation, including incorrect cross-validation split counts, predictor counts, dataset ages, and computational biases.One review found 68 deterministic splits rather than the claimed 1,000, and seven implemented predictors rather than nine; another identified potentially opposing biases in temperature-related mortality estimates.Source: S2: Code reading — n = 28 · p. 70

S3: Specialized niche field catch — n = 27

• Across niche fields, AI reviewers surfaced specialized issues humans missed, including stereoselectivity, population structure, benchmark validity, and sensor calibration.Examples included a GWAS population-structure concern, a challenge to benchmark choice, and insufficient strain-sensor calibration.Source: S3: Specialized niche field catch — n… · p. 71

S4: Internal consistency across sections — n = 15

• AI reviewers caught internal inconsistencies, from mismatched event definitions and physiologically impossible reported values to contradictions between an abstract and figure caption.One uniquely identified contradiction was an abstract claiming “no passivation” while a figure caption specified parylene encapsulation.Source: S4: Internal consistency across… · p. 72

S5: Reproducibility / dependency failures — n = 10

• AI reviews identified reproducibility failures such as missing parameter specifications and modified external tools, though some issues may require only tidying rather than bear on scientific validity.One evaluator noted that such reproducibility issues are valid but likely do not reflect scientific validity.Source: S5: Reproducibility / dependency… · p. 72

S6: Big-picture / counter-narrative synthesis — n = 7

• AI reviewers also identified high-level flaws overlooked by humans, including circularity and counter-narratives that challenge a paper’s central interpretation.One counter-narrative suggested pandemic-period temperature-related mortality may be overestimated because the surviving population was less vulnerable.Source: S6: Big-picture / counter-narrative… · p. 73
• AI Reviewer 1 uniquely identified that the training objective was less novel than claimed and that state-of-the-art baseline comparisons were insufficient.Source: S6: Big-picture / counter-narrative… · p. 73
• An AI reviewer argued that developmental disorders may diverge more from healthy-adult brain maps, recommending patient-specific data to validate findings for conditions such as ASD and ADHD.The rationale was that these disorders emerge early in life, so their brain architecture may differ fundamentally from normative connectomes and molecular maps.Source: S6: Big-picture / counter-narrative… · p. 73

S generic: Residual: AI judged Correct without specific reason — n = 40

• Several expert annotations endorse AI criticisms with brief praise such as “good point,” without giving a specific reason.The examples include comments such as “This is a good point” and “Great point.”Source: S generic: Residual: AI judged Correct… · p. 73
• A highlighted example identifies an important issue missed by human reviewers, while other expert comments qualify AI criticisms as relevant but incomplete.Comments note that a critique could be relevant while lacking distinctions present in human reviews, or that a concern remained valid despite other evidence.Source: S generic: Residual: AI judged Correct… · p. 74
• One expert judged AI reviews technically strong and thorough, while noting that human reviewers bring experience-based insights.The comment also ranked that particular AI review below other AI reviews.Source: S generic: Residual: AI judged Correct… · p. 74

F Details of the AI meta-reviewer and PEERREVIEW BENCH

• The appendix documents the AI meta-reviewer, its calibration analysis, PEERREVIEW BENCH construction and leaderboard, and the prompt used for its runs.Source: F Details of the AI meta-reviewer and… · p. 75

F.1 The dual-annotated calibration set

• The dual-annotated calibration set contains 908 review items from 27 papers, with two scientists independently annotating each item across three judgment axes.The set comprises 568 human-review and 340 AI-review items.Source: §F.1 · p. 75
• The top three calibration-set classes were “both correct, both significant, but disagree on significance” (32.8%), “both correct, both significant, evidence sufficient” (30.5%), and “disagree on correctness” (14.2%).Source: §F.1 · p. 75
• The calibration evaluates the AI meta-reviewer on per-axis judgments and ten-class predictions, comparing its agreement with experts and the human–human agreement ceiling.The protocol reports percent agreement, Gwet’s AC1 for human–human, human–AI, and AI–AI pairs, and ten-class accuracy across all 908 items.Source: §F.1 · p. 75
• The meta-reviewer runs in OpenHands with tools to inspect preprints, figures, source code, and terminology, while being blocked from published versions to prevent access to referee reports.The evaluated backbones are Claude-Opus-4.7, GPT-5.4, and Gemini-3.1-Pro.Source: §F.1 · p. 75

F.2 Calibration results

• Claude-Opus-4.7 reached 87.9% correctness accuracy versus the 85.8% human baseline, while all three AI meta-reviewers scored below the 59.9% significance and 88.0% evidence baselines.Significance was the hardest axis for humans and AI. Same-family judgments shifted accuracy unevenly across axes, with every model declining on at least one axis.Source: §F.2 · pp. 75–76, Table 47 · p. 76
• Claude-Opus-4.7 achieved 44.3% ten-class accuracy, versus 30.1% for GPT-5.4 and 29.0% for Gemini-3.1-Pro, against a 10% random baseline.On significance, AI–AI agreement was AC1 0.83–0.88, versus 0.43–0.55 for AI–human pairs, suggesting a shared AI standard that does not fully match human judgments.Source: §F.2 · pp. 76–77, Table 48 · p. 76, Table 49 · p. 77

F.3.2 The partial-evidence trap

• The partial-evidence trap caused 41 of 54 correctness errors: GPT-5.4 treated partial coverage of a concern as full refutation instead of assessing its component claims.The analysis manually reviewed GPT-5.4’s disagreements with the primary annotator on 908 items; its seven bottom-up failure patterns could overlap.Source: §F.3.2 · pp. 77–78, §F.3.1 · p. 77

F.3.3 Over-leniency on technically detailed items

• In 13 cases, the AI meta-reviewer accepted a review item as “Correct” despite both experts rejecting it; 10 of these errors involved AI-generated review items.The items shared precise-sounding technical language that seemed persuasive even when the underlying claim did not hold up against the paper.Source: §F.3.3 · p. 78

F.3.4 Significance boundary miscalibration

• GPT-5.4 made 56 significance errors on valid review points, often underweighting methodological rigor and translational relevance while over-weighting technical presentation issues.Its “would addressing this change the core result?” test contributed to downgrading substantive concerns and upgrading some minor ones.Source: §F.3.4 · p. 79

F.3.5 Evidence closure demand

• In 13 cases, the AI meta-reviewer rated evidence “Requires More” despite experts’ “Sufficient” ratings, expecting self-contained proofs rather than pointers to specific, verifiable gaps readers could check in the paper.Examples concerned validation sample size, spatial autocorrelation in cross-validation, and COVID-19 mortality displacement.Source: §F.3.5 · pp. 79–80

F.3.6 Reviewer-type asymmetry

• GPT-5.4’s errors were asymmetric: 29 of 41 correctness false negatives involved human reviews, while 10 of 13 false positives involved AI reviews.It demanded literal precision from human items but credited AI items’ technical specificity, a prior that misfired in both directions.Source: §F.3.6 · p. 80

F.3.7 Expert-disagreement prediction failures; F.3.8 Context anchoring and paper-mismatch blind spots

• GPT-5.4 overpredicted expert agreement when narrow factual checks missed broader concerns, overpredicted disagreement on judgment calls, and anchored later judgments to an initial paper-mismatch hypothesis.The failure analysis also identified template reasoning across items from the same paper after the model hypothesized that a review concerned a different paper.Source: §F.3.7 · pp. 80–81, §F.3.8 · p. 81

F.3.8 Context anchoring and paper-mismatch blind spots

• Context anchoring caused the AI meta-reviewer to carry early judgments across items, including cases where experts disagreed.On Paper 81, it rated single-cell replication concerns Significant when experts rated them Marginally Significant; on Paper 33, it rated reproducibility and methodology concerns Significant when experts rated them Marginally Significant or Not Significant. Shared context can aid verification but also create a feedback loop; independent per-item evaluation or a reset could mitigate it.Source: §F.3.8 · p. 81

F.3.9 How the failure categories interact

• The failure categories interact: context anchoring propagates errors, while reviewer-type asymmetry and significance miscalibration compound other errors.The partial-evidence trap is worse when the AI meta-reviewer evaluates human reviewers’ items; technical items wrongly accepted as correct also tend to receive inflated significance ratings.Source: §F.3.9 · p. 82

F.4 PEERREVIEW BENCH: construction and evaluation protocol

• PEERREVIEW BENCH comprises 78 of 82 papers; each rubric pools fully positive items from the paper’s three human reviewers, while four papers with no such items are excluded.Rubrics ranged from 1–36 items (mean 10, median 8); overlapping items were not deduplicated, so repeatedly raised concerns receive correspondingly more credit. Metrics were averaged across papers, treating each as one independent observation to prevent papers with larger rubrics dominating; matches used GPT-5.4 at the “at least same criticism” threshold.Source: §F.4 · p. 82

F.5 Full leaderboard analysis

• F1 50.89 for Claude-Opus-4.5 leads the leaderboard, just ahead of Claude-Opus-4.7 at F1 50.46; Gemini-3.1-Pro-Preview scores F1 22.57.The Gemini model’s low score is driven primarily by recall collapsing to 13.92%.Source: §F.5 · p. 82
• Claude-Opus models balance precision and recall, GPT-5 models favor high precision and low recall, and Gemini-family models cast a wider net at lower precision.GPT-5.4 has the benchmark’s highest precision at 93.81% with 26.55% recall; GPT-5.2 has 88.92% precision and 32.28% recall. No backbone dominates both axes.Source: §F.5 · p. 82
• Newer models do not consistently improve F1: Claude-Opus-4.5 narrowly beats 4.7, GPT-5.2 beats 5.4, and Gemini-3.0-Pro-Preview beats 3.1-Pro-Preview.PEERREVIEW BENCH provides a testbed for measuring targeted model improvements.Source: §F.5 · pp. 82–83

F.6 AI meta-reviewer prompt

• The meta-reviewer judges each review item on correctness, significance, and evidence sufficiency, then predicts two independent expert meta-reviewers’ joint judgment using one of 10 collapsed labels encoding cascade outcome and agreement.Source: §F.6 · p. 84
• The protocol uses devil’s-advocate checks to predict expert disagreement, constrains predictions to match final axis labels, and requires JSON and label consistency checks before completion.It distinguishes factual uncertainty from disputes about significance and treats borderline evidence as potential expert disagreement.Source: §F.6 · pp. 85–87

G Details of the CMU PAPER REVIEWER platform

• Appendix G documents the CMU PAPER REVIEWER platform’s implementation, weakness mitigations, public-platform comparison, and intended-use policy.The supplied passages identify these as the appendix’s scope.Source: G Details of the CMU PAPER REVIEWER… · p. 88

G.1 Implementation and design

• The platform uses OpenHands and the study’s tool configuration, and lets users set item counts and criteria or provide supplementary code and data.The shared tools include a terminal, file editor, Tavily web search, and Mistral OCR; the default item-count range is 1 to 15.Source: §G.1 · p. 88
• To mitigate weak or overly harsh critiques, the platform pairs items with proposed edits or code patches, supports limitation-aware reviews and author challenges, and flags or blocks post-preprint citations.Authors may opt into grounding items in manuscript-stated limitations; they can challenge items through an interactive debate, and can forbid later citations entirely.Source: §G.1 · p. 88

G.2 Comparison with public AI reviewer platforms

• F1 reaches 58.64 for the CMU PAPER REVIEWER with GPT-5.4 and a 15-item cap, versus 51.65 for Stanford Agentic Reviewer and 47.88 for OpenAIReview.The comparison covers 78 PEERREVIEW BENCH papers and uses precision, recall, and F1; review items are defined from each platform’s output format.Source: §G.2 · p. 88
• Claude Opus 4.7 with a 5-item cap reaches F1 50.46, with precision 71.47%, recall 39.00%, and 4.73 items per paper versus competitors’ 11.08 and 18.64.This is a more balanced operating point with a smaller absolute item count than the competing platforms.Source: §G.2 · p. 89
• 95.46% precision and 42.32% recall at a 15-item cap compare with 93.81% precision and 26.55% recall at a 5-item cap, while output rises from 3.60 to 7.35 items per paper.Tripling the cap only roughly doubles output because the agent selects fully positive items rather than filling the budget; the 15-item setting has the comparison’s highest precision and approaches Stanford’s recall.Source: §G.2 · pp. 88–89

G.3 Availability and intended use

• The CMU PAPER REVIEWER is publicly accessible, offers up to three free reviews daily or unlimited use with a personal API key, and is open source for institutional deployment.It is intended for assistive pre-submission feedback, not as a substitute for sanctioned human peer review or use where AI reviewers are prohibited.Source: §G.3 · p. 89

H Recommendation for journal and conference organizers: Panel composition analysis

• The panel-composition analysis is intended to inform organizers’ deployment priorities and covers four configurations, seven review-item metrics, results, priority-indexed configurations, generalization caveats, and 95% confidence intervals.The supplied passages for this subsection describe its motivation and scope, not its results.Source: H Recommendation for journal and… · p. 89

H.1 Methodology and metric definitions

• The simulation uses 53 papers with three human and three AI reviews each, comparing 3H, 2H+1AI, 1H+2AI, and 3AI panels, with or without a GPT-5.4 filter.Each mixed-panel configuration averages nine possible reviewer combinations; the filter removes items it judges not fully positive.Source: §H.1 · p. 89
• Other metrics count total items, unique items, and items not fully positive, and measure fully positive unique items as a fraction of unique items, total items, and the noise per useful item.“Unique” means no similar panel counterpart (similarity ordinal < 2); not-fully-positive items fail at least one of correctness, significance, or evidence sufficiency. The fractions measure useful-item share among distinctive contributions and overall output; noise per Gem is not-fully-positive items per fully positive unique item.Source: §H.1 · p. 89
• The bottom-line metric, # Fully Pos. + Unique Items, counts items that are correct, significant, evidence-sufficient, and unique within the panel.It represents useful feedback authors actually receive after excluding items that are redundant, incorrect, insignificant, or insufficiently evidenced.Source: §H.1 · p. 89

Panel composition results

• Among unfiltered panels, 2H+1AI matches three humans at 3.9 fully-positive-and-unique items per paper, while all-AI falls to 1.8 (CI [1.4, 2.2]).All-AI panels produce 3.1 distinct issues per paper, against 11.5 for three humans, because AI reviewers overlap more with one another.Source: §H.2 · p. 90

AI meta-reviewer filter

• 20.0% vs 18.2% yield for 2H+1AI after filtering accompanies lower noise per gem (2.14 vs 2.95), but useful-item counts fall across every composition.Filtering reduces fully-positive-and-unique items from 3.9 to 1.9 for 3H, 3.9 to 2.2 for 2H+1AI, 3.5 to 2.1 for 1H+2AI, and 1.8 to 1.2 for all-AI panels; it can remove useful items along with less-positive ones.Source: §H.2 · p. 90

Recommendation 2 (1H+2AI + meta-reviewer filter): minimize editor and author triage time

• 20.6% of items are fully-positive-and-unique for filtered 1H+2AI, versus 15.1% for 3H; noise per gem is 1.95 vs 3.74.This configuration yields 2.1 fully-positive-and-unique items per paper, against 3.9 for 3H, and suits venues prioritizing editor or author triage time.Source: §H.2 · p. 91

Recommendation 3 (3AI + meta-reviewer filter): high-confidence shortlist instead of full reviews

• 63.2% of unique items are fully positive for filtered 3AI, versus 33.9% for 3H, but it yields only 1.2 fully-positive-and-unique items per paper.This selective configuration produces 3.5 not-fully-positive items per paper, against 14.6 for 3H, and is suited to producing a small, vetted shortlist rather than broad coverage.Source: §H.2 · p. 91

Caveats

• The results draw on one Nature submission pool, a five-item-per-AI cap, and three frontier AI reviewers; other venues and future models require revalidation.The authors expect the qualitative pattern to generalize but caution that the specific numbers may change with venue, item budget, or AI generation.Source: §H.2 · p. 91