PaperlayerView exact revision on arXiv

Source-linked AI summary

ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review

Gaurav Sahu, Hugo Larochelle, Laurent Charlin, Christopher Pal

arXiv:2510.08867v1cs.AIcs.CL

TL;DR

Peer review faces inconsistency, subjectivity, and scaling challenges, while AI-assisted review lacks systematic, reproducible evaluation. ReviewerToo structures the process around diverse agents and evaluates their performance; ensemble metareviewing performs strongest across the reported decision and review-quality measures, while rebuttal rounds pose a sycophancy risk.

  • Problem

    Peer review is inconsistent, subjective, and difficult to scale, while early AI-assisted deployments are hard to study reproducibly.

    Source: §1 · p. 1

  • Method

    ReviewerToo is a modular AI-assisted review framework that structures literature review, persona-based reviewing, rebuttal, and metareviewing.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Ensemble protocols outperform single-agent reviewers, with META (ALL) strongest across accuracy, F1, and ELO; single-agent personas reach close to 70% binary accuracy but trail humans in binary F1.

    Source: §6 · p. 10

  • Takeaways & Limitations

    AI reviewers should complement rather than replace humans, with domain experts retained for complex evaluative judgments.

    Source: §7 · p. 11

    Performance drops after rebuttal rounds, potentially reflecting sycophantic deference that can weaken independent judgment.

    Source: §6 · p. 10

Abstract

from arXiv · show

Peer review is the cornerstone of scientific publishing, yet it suffers from inconsistencies, reviewer subjectivity, and scalability challenges. We introduce ReviewerToo, a modular framework for studying and deploying AI-assisted peer review to complement human judgment with systematic and consistent assessments. ReviewerToo supports systematic experiments with specialized reviewer personas and structured evaluation criteria, and can be partially or fully integrated into real conference workflows. We validate ReviewerToo on a carefully curated dataset of 1,963 paper submissions from ICLR 2025, where our experiments with the gpt-oss-120b model achieves 81.8% accuracy for the task of categorizing a paper as accept/reject compared to 83.9% for the average human reviewer. Additionally, ReviewerToo-generated reviews are rated as higher quality than the human average by an LLM judge, though still trailing the strongest expert contributions. Our analysis highlights domains where AI reviewers excel (e.g., fact-checking, literature coverage) and where they struggle (e.g., assessing methodological novelty and theoretical contributions), underscoring the continued need for human expertise. Based on these findings, we propose guidelines for integrating AI into peer-review pipelines, showing how AI can enhance consistency, coverage, and fairness while leaving complex evaluative judgments to domain experts. Our work provides a foundation for systematic, hybrid peer-review systems that scale with the growth of scientific publishing.

1 INTRODUCTION

Peer review faces scaling pressure and inconsistent, subjective assessments, while early LLM deployments are difficult to study reproducibly. ReviewerToo provides a modular framework for controlled study and deployment, and the paper reports a large-scale ICLR 2025 evaluation and integration guidelines.

1 INTRODUCTION

  • Peer review faces tens of thousands of submissions at major conferences, while early LLM deployments offer scalability but risk superficial or misleading assessments and lack reproducible study.The authors identify controlled, transparent, repeatable experiments as necessary to determine where AI helps, harms, and can be responsibly integrated.Source: §1 · pp. 1–2
  • ReviewerToo is a modular framework for studying and deploying AI-assisted peer review with systematic, consistent assessments that complement human judgment.It supports standardized comparisons and can be partially or fully adopted in conference workflows.Source: §1 · p. 2
  • The study evaluates ReviewerToo on 1,963 curated ICLR 2025 submissions and analyzes reviewer personas, biases, and alignment with meta-review outcomes.The dataset balances acceptance and rejection decisions while preserving diversity across score ranges and decision categories.Source: §1 · p. 2
  • The paper proposes AI-integration guidelines informed by quantitative performance metrics and qualitative analyses of reviewer behavior.The authors position these contributions as a foundation for systematic and consistent AI integration into peer review.Source: §1 · p. 3

2 BACKGROUND

Prior work describes peer review as strained by workload, bias, and low agreement, while AI review assistance has produced mixed results. ReviewerToo addresses a comparatively neglected issue: representing reviewer diversity and analyzing disagreement among distinct personas.

2 BACKGROUND

  • Peer review faces reviewer fatigue, bias, low inter-reviewer agreement, and growing submission volumes at major conferences (Cortes & Lawrence, 2021; Adam, 2025).Reported acceptance decisions can vary almost randomly, and scalability is a pressing concern.Source: §2 · p. 3
  • LLMs are used for review generation, citation verification, fact-checking, and meta-review synthesis, but studies report both helpfulness and risks of hallucination and shallow judgment (Hossain et al., 2024).Prior work emphasizes AI-in-the-loop assistance rather than replacing expert judgment (Idahl & Ahmadi, 2024; Liang et al., 2024a).Source: §2 · p. 3
  • ReviewerToo models theorist, empiricist, and pedagogical personas to study predictive accuracy and the structure of inter-reviewer disagreement.The work addresses relatively limited prior attention to reviewer diversity, drawing on role-specific LLM social simulation (Sahakyan & AlShebli, 2025; Anthis et al., 2025).Source: §2 · p. 3

3 SYSTEM OVERVIEW

ReviewerToo organizes peer review as a modular sequence of literature review, reviewer reports, author rebuttal, and metareview. Its agents use retrieval, role-specific criteria, and evidence checks, while the single-turn protocol prioritizes conference realism and tractability.

3 SYSTEM OVERVIEW

  • ReviewerToo processes a manuscript through literature review, reviewer agents, an author rebuttal, and a metareview that integrates the full record.The modules allow humans and LLMs to participate at each stage, with optional feedback loops.Source: §3 · p. 3, Figure 2 · p. 4
  • The single-turn protocol gives each agent one contribution per stage, with an optional short post-rebuttal response from reviewers.The design prioritizes conference realism and tractability, while allowing that multi-turn deliberation could be supported.Source: §3 · p. 3
  • The literature agent uses LitLLM to retrieve and rank papers, then summarizes selected candidates to ground later agents.It generates search queries for Semantic Scholar and selects top-k candidates after debate-based ranking.Source: §3 · p. 4
  • Reviewer agents receive the manuscript, optional literature summary, and persona or rubric prompts, then provide structured, evidence-grounded assessments and categorical recommendations.Assessment dimensions include novelty, soundness, experimental validity, presentation, and impact; unsupported judgments trigger stricter retrieval and reruns.Source: §3 · p. 4
  • The author agent drafts a rebuttal addressing severe criticisms and misunderstandings, with clarifications tied to reviewer claims or relevant literature.It can propose concrete revisions such as releasing code or adding ablation studies.Source: §3 · p. 4
  • The metareviewer synthesizes reviewer stances, strengths and weaknesses, rebuttal effectiveness, stance shifts, and unresolved concerns.A fact-checking module verifies reviewer claims against the manuscript and literature summary, discards unsupported claims, and scores fact significance.Source: §3 · p. 5

4 EXPERIMENTAL SETUP

The experimental setup uses a stratified ICLR 2025 submission subset, compares ReviewerToo against heuristic, supervised, reviewer, and ensemble baselines, and evaluates decision alignment, reviewer consistency, review quality, and rebuttal helpfulness.

4.1 DATASETS

  • The ICLR-2k dataset contains 1,963 stratified submissions from ICLR 2025, with official decisions serving as ground truth for five-way and binary evaluations.Sampling balances acceptance and rejection while preserving score-range and decision-category diversity; withdrawn papers join Reject, while Desk Reject remains separate.Source: §4.1 · p. 6

4.2 BASELINES

  • Baselines span supervised classifiers, single-agent reviewers with progressively added conditioning, and reviewer ensembles including majority vote and metareviewer aggregation.This progression tests absolute effectiveness and which design choices narrow the gap to human decision-making.Source: §4.2 · p. 6

4.3 EVALUATION METRICS

  • Evaluation measures five-way and binary decision alignment, macro-averaged precision, recall, and F1, accuracy, false-positive rate, reviewer agreement, review quality, and rebuttal helpfulness.Review quality uses held-out LLM pairwise judgments across five axes, aggregated into system ELO ratings.Source: §4.3 · pp. 6–7

5 RESULTS

Across the reported results, diverse reviewer ensembles perform more reliably than individual personas, while single agents show distinct biases and fine-grained decision challenges. Review-quality ratings, baseline comparisons, ablations, and examples further show both the promise and limits of structured reviewer agents.

5 RESULTS

  • Removing conference instructions reduces F1 and ELO, and removing literature grounding drops ELO to ≤1332; rebuttal rounds increase false positives and reduce agent F1.The authors suggest the post-rebuttal pattern may reflect sycophantic tendencies; the cited table covers conference-instruction, literature, and rebuttal ablations.Source: Table 2 · p. 7, §5 · p. 9
  • MAJORITY and META show higher agreement with ground truth, while permissive and critical personas have near-zero or negative agreement with other reviewers.Human–LLM reviewer agreement is κ ≈0.1–0.2; MAJORITY and DEFAULT show moderate alignment at κ ≈0.5.Source: §5 · p. 7
  • META (ALL) leads review-quality ratings at ELO 1657, while average human reviews score 540 and the top 1% of human reviewers score 1316.Average and top-1% human reviewers retain binary F1 scores of 83.8 and 90.4, respectively.Source: §5 · pp. 7, 9
  • Normalized confusion matrices show permissive reviewers over-predict acceptance, critical reviewers favor rejection, and most personas struggle to distinguish oral from spotlight decisions.Empiricist and pedagogical personas have more balanced error profiles but still over-predict rejections relative to ground truth.Source: §5 · p. 7, Figure 3 · p. 8
  • ReviewerToo agents match or exceed supervised baselines in decision accuracy while producing substantive reviews with competitive or superior ELO ratings; supervised baselines have binary F1 scores of 43.2 to 65.3.The passage contrasts the agents’ decision alignment and author-facing review utility with supervised models’ inability to bridge both.Source: §5 · p. 9
  • In two qualitative examples, agents match human accept decisions for a strong paper and some align with a meta-review rejection despite human reviewers’ positive scores.For the rejected paper, agent and meta-review concerns centered on the robustness of the proposed approach.Source: §5 · p. 9
  • Ensembles reasonably approximate human-level decision making, with higher predictive accuracy and judged review quality; conference conditioning, rebuttals, and literature access each contribute to closing the human gap.The summary attributes stronger results to aggregating diverse reviewers through metareview protocols.Source: §5 · pp. 9–10

6 DISCUSSION

The discussion presents reviewer agents as aids for coarse decision-making and constructive feedback, while emphasizing limits in fine-grained calibration, review quality, and rebuttal handling. Its guidelines favor human oversight, ensemble protocols, structured conditioning, and evaluating feedback as well as decisions.

6 DISCUSSION

  • Single-agent personas approach 70% accuracy on binary accept/reject decisions, but five-way performance remains substantially lower, especially for oral-versus-spotlight distinctions.The discussion concludes that conference-level calibration still requires human expertise and that human reviewers maintain higher binary F1 scores.Source: §6 · p. 10
  • Ensemble protocols consistently outperform single agents, with META (ALL) strongest across accuracy, F1, and ELO; aggregating perspectives reduces individual biases and improves reliability.The authors identify metareviewing as a crucial design principle for AI-assisted peer review.Source: §6 · p. 10
  • Review agents can produce more constructive feedback than supervised baselines, but review-text quality is not always aligned with human expectations for actionability and helpfulness.Top-1% human reviewers achieve high ELO, while average human reviews perform poorly under that measure.Source: §6 · p. 10
  • Performance drops systematically after rebuttal rounds, potentially because agents defer excessively to author rebuttals and lose independent judgment.The discussion suggests calibration instructions or adversarial prompting as possible safeguards.Source: §6 · p. 10
  • LLM reviewer agreement varies substantially across personas, echoing disagreement in human peer review; ensembles and consensus protocols can reduce, but not eliminate, variance.The discussion notes strong divergence from permissive and critical personas.Source: §6 · p. 10

6.1 GUIDELINES FOR INTEGRATING AI INTO PEER REVIEW

  • Use AI as a complement to human judgment, reserving final decisions especially for borderline and high-stakes cases while aiming to raise the floor with consistent baseline reviews.The guidelines frame human–AI collaboration, rather than competition, as the design goal.Source: §6.1 · pp. 10–11
  • Prefer ensemble protocols and metareview synthesis to address persona bias, disagreement, and variance in outcomes.The guidelines recommend majority voting or metareviewing rather than relying on single-agent reviewers.Source: §6.1 · p. 11
  • Condition reviewers with conference guidance, literature retrieval, and rebuttal phases, but balance these inputs to avoid overfitting or sycophancy; assess actionable review quality as well as accuracy.The guidelines caution that uncritical rebuttal incorporation can degrade performance.Source: §6.1 · p. 11

7 CONCLUSION

ReviewerToo supports hybrid peer review: AI reviewers approach human-level binary decisions but remain weaker on fine-grained tiers and nuanced judgments. The authors recommend using AI as a complement to human expertise.

7 CONCLUSION

  • Single-agent personas approach 70% accuracy on binary accept/reject decisions, but perform substantially worse on five-way decisions, leaving conference-level calibration to human expertise.Human reviewers retain higher binary accuracy.Source: §7 · p. 11
  • Fine-grained calibration, rebuttal sycophancy, and variable agreement across personas remain challenges that support continued human involvement.Source: §7 · p. 11
  • The authors recommend complementary AI use, ensembles and meta-review, structured context, and optimizing for review quality alongside decision fidelity.Source: §7 · p. 11

8 ETHICS STATEMENT

The study used publicly available OpenReview data under its terms and guidelines, and frames ReviewerToo for research and controlled integration rather than autonomous publishing decisions.

8 ETHICS STATEMENT

  • The researchers report adhering to OpenReview’s terms and community guidelines when collecting and analyzing publicly accessible data.Source: §8 · p. 11
  • ReviewerToo is intended for research and controlled integration, not unsupervised or fully automated academic publishing decisions.Source: §8 · p. 11
  • The authors identify bias propagation, overreliance on model outputs, and possible reinforcement of systemic inequities as ethical risks addressed by their guidelines and limitations.Source: §8 · p. 11

A LLM USAGE

The authors disclose using ChatGPT for language editing and LitLLM to retrieve relevant works.

A LLM USAGE

  • ChatGPT was used to improve some paragraphs’ language, and LitLLM was used to retrieve relevant works.Source: A LLM USAGE · p. 13

B LLM-AS-A-JUDGE PROTOCOL FOR ELO

The protocol uses pairwise LLM-judge comparisons to rank review-writing quality with ELO, while applying anonymization, randomization, stratified sampling, and manual checks. The update incorporates five evaluation dimensions.

B LLM-AS-A-JUDGE PROTOCOL FOR ELO

C QUALITATIVE EXAMPLES

These qualitative examples summarize reviews of work on shallow safety alignment and lightweight image watermarking, describing contributions alongside concerns about clarity, evaluation, scope, rigor, reproducibility, and presentation. For the watermarking paper, the reviews also differ on whether its contribution is strong enough for acceptance.

Paper 6Mxhg9PtDE: Human Review

  • Reviewers say the safety paper’s related-work distinctions and its notion of a unified contribution need clearer explanation.They also ask why attack success rates remain nonzero after mitigation and whether incoherent augmented examples affect learning.Source: C QUALITATIVE EXAMPLES · pp. 18–19

Paper 6Mxhg9PtDE: Empiricist and Theoretical

  • The safety-alignment reviews describe evidence that alignment mainly changes early output tokens and frame several jailbreak and fine-tuning attacks as exploiting this shallow alignment.They summarize data augmentation with safety-recovery examples and token-wise constrained fine-tuning as proposed mitigations.Source: C QUALITATIVE EXAMPLES · pp. 19–20
  • The safety reviews flag reliance on GPT-4 safety judgments, limited statistical analysis, and evaluation concentrated on roughly 7B-parameter models.They note that human validation, more seeds, and tests on larger models or other alignment pipelines would strengthen the claims.Source: C QUALITATIVE EXAMPLES · pp. 19–21

Paper j7b4mm7Ec9: Human Review

  • The watermarking review questions comparison fairness because methods use different embedding capacities and says capacity and noise-training settings need fuller treatment.It also asks why the framework omits discriminative networks and whether its loss decomposition is novel.Source: C QUALITATIVE EXAMPLES · p. 22

Paper j7b4mm7Ec9: Empiricist

  • The watermarking review reports ≈16 k parameters, PSNR ≈41–42 dB, and average decoding accuracy ≈99.3% for the lightweight model, using ∼2% of baseline parameters.It says the model outperforms several larger models, including HiDDeN, MBRS, CIN, and FIN, on the reported benchmark.Source: C QUALITATIVE EXAMPLES · p. 22
  • The watermarking review notes missing statistical uncertainty, limited coverage of real-world attacks, and unquantified projection-head training costs.It also identifies hyperparameter sensitivity and the absence of a formal guarantee linking the proposed loss to lower decoding error.Source: C QUALITATIVE EXAMPLES · pp. 22–23

C QUALITATIVE EXAMPLES

  • Reviewers split on acceptance: one recommended an accept poster, while others rejected the paper over theoretical rigor, marginal gains, and communication problems.The accepting review still called for stronger statistical validation, broader attack evaluation, and training-overhead analysis.Source: C QUALITATIVE EXAMPLES · pp. 23, 25, 26
  • Reviewers praised the broad distortion evaluation, module and projection-head ablations, and plug-and-play potential of PH and DO.The reviews describe evaluation across combined noise, diffusion attacks, and geometric transforms, using metrics including PSNR, SSIM, LPIPS, and accuracy.Source: C QUALITATIVE EXAMPLES · pp. 24–25
  • Critics said the loss decomposition and safe-distance choice lacked theoretical justification, while PH and DO required extensive manual hyperparameter tuning.They also requested multiple-seed uncertainty estimates, statistical tests, training-cost measurements, stronger baselines, and broader robustness checks.Source: C QUALITATIVE EXAMPLES · pp. 23–24
  • The paper identifies loss-mismatch in MSE/BCE and proposes detachable projection heads and a decoding-oriented loss for lightweight watermarking.The projection head is removed at inference; the DO loss uses a safe-distance parameter to limit inflation, and the five-module decomposition supports fine-grained ablations.Source: C QUALITATIVE EXAMPLES · p. 25
  • Reviewers also criticized unclear exposition, missing or poor figures, formatting errors, and incomplete reproducibility guidance, and requested clearer diagrams and implementation details.One review additionally called for a broader-impact discussion and better organization of related work.Source: C QUALITATIVE EXAMPLES · pp. 24–25

D IMPLEMENTATION DETAILS

ReviewerToo uses asynchronous orchestration and standardized output storage to support parallel execution, caching, reproducibility, and downstream analysis. Its implementation uses gpt-oss-120b with conference-guideline-based prompts and includes classifier training and evaluation details.

D IMPLEMENTATION DETAILS

  • gpt-oss-120b serves all roles on 8xH100 GPUs with vLLM, using persona-specific prompts combined with a base rubric and official ICLR reviewer guidelines.No fine-tuning is performed; agents operate in instruction-following mode.Source: D IMPLEMENTATION DETAILS · p. 26
  • The system uses Python asynchronous orchestration and semaphores for parallelism, while standardized output directories enable caching, reproducibility, and downstream analysis.Source: D IMPLEMENTATION DETAILS · p. 26
  • XGBoost is trained with 5-fold cross-validation, while the BERT classifier uses a 70-15-15 split and validation-set tuning across learning rates and epoch counts.The reported XGBoost settings are 200 estimators, depth 6, and learning rate 0.1; BERT is fine-tuned for 20 epochs at learning rate 2e-5 with batch size 16.Source: D IMPLEMENTATION DETAILS · p. 26
  • Tables 4 and 5 summarize pipeline-agent responsibilities and reviewer and metareviewer persona emphases.Source: Table 4 · p. 27, Table 5 · p. 28

Related papers

Loading 2510.08867v1…