PaperlayerView exact revision on arXiv

Source-linked AI summary

From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent

Haishuo Fang, Yue Feng, Iryna Gurevych

arXiv:2606.13349v1cs.CL

TL;DR

Automated reviews can lack depth and concrete evidence, while fixed procedures limit investigation across a paper. ProReviewer uses an MDP-guided structured log to investigate proactively, and its 8B model achieves the strongest reported review-quality results against larger frontier models and fine-tuned baselines.

  • Problem

    Existing review methods can produce shallow or generic criticism, insufficiently investigate authors’ claims, and miss inconsistencies across sections.

    Source: §1 · pp. 1–2

  • Method

    ProReviewer formulates review as an MDP and uses a structured log of claims, questions, and notes to guide proactive investigation and evidence tracking.

    Source: §1 · p. 2

  • Results

    ProReviewer ranks first on average across five review-quality dimensions, beating frontier LLM systems by up to 39% relatively and the strongest fine-tuned baseline by 16%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest that proactive investigation supported by evidence tracking is a promising direction for scientific peer review.

    Source: §6 · p. 8

    The current text-only implementation cannot directly inspect figures, which may contain complementary evidence not accurately described in the text.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have shown promise in automating scientific peer review. However, existing approaches often struggle to generate in-depth reviews supported by concrete evidence. We argue that a key limitation is the lack of flexibility to proactively investigate suspicious parts of a paper based on accumulated evidence, as human reviewers do. In this paper, we explore how to enable an LLM-based review agent to perform such proactive investigation. We find that this can be naturally formulated as a Markov Decision Process (MDP), and propose ProReviewer, a scientific peer review agent that proactively reviews a paper guided by a maintained, structured review log. The structured review log serves as a workspace for the agent to track evidence and intermediate findings collected during review. Experiments show that ProReviewer with an 8B backbone, trained by supervised fine-tuning and optimized by reinforcement learning, achieves the highest average score across five quality dimensions, outperforming prompt-based methods with much larger frontier LLMs by up to 39% and the strongest fine-tuned baseline by 16% relatively. It also attains the highest win rates against baselines in human evaluation.

1 Introduction

The introduction argues that fixed review procedures limit evidence-driven investigation, then presents ProReviewer’s MDP-based, structured-log approach and reports stronger review quality than larger-model and fine-tuned baselines.

1 Introduction

  • Prior automated review approaches include direct prompting, multi-stage pipelines, and multi-agent collaboration.The cited examples are Robertson (2023), Liang et al. (2023), and Liu and Shah (2023) for prompting; Gao et al. (2024) and Zhu et al. (2025) for pipelines; and Jin et al. (2024) and Yamada et al. (2025) for multi-agent collaboration.Source: §1 · p. 1
  • Existing methods can produce shallow, generic, weakly grounded reviews and miss cross-section inconsistencies because their investigation paths are fixed rather than evidence-adaptive.The introduction illustrates a claim about domain robustness contradicted by experimental results.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • ProReviewer ranks first on average across five review-quality dimensions, outperforming frontier LLM systems by up to 39% relatively and the strongest fine-tuned baseline by 16%.Human evaluators also preferred its reviews across all pairwise comparisons.Source: §1 · p. 2
  • The contributions frame peer review as proactive investigation through an MDP-based agent and a structured log that tracks claims, questions, and notes throughout review.The log supports evidence-grounded review generation.Source: §1 · p. 2
  • The paper contributes a version-matched corpus of 5K ICLR 2025/2026 paper–review pairs, with each review aligned to the manuscript version it assessed.The corpus supports contamination-controlled training and evaluation.Source: §1 · p. 2

2 Related Work

Prior automated-review and agent-memory approaches tend to use fixed procedures or unstructured traces; ProReviewer distinguishes itself with learned investigation and a structured, revisable review log.

LLM-based Review Generation

Agentic Reasoning

  • Agent frameworks and persistent-memory methods can retain reasoning, but their unstructured traces make selective revision and evidence tracing difficult.ProReviewer’s typed log is part of a trainable MDP state and supports selective revision and evidence tracing without keeping the full reasoning trajectory in context.Source: §2 · pp. 2–3

3 Method

ProReviewer models scientific review as proactive paper investigation, with a policy that reads manuscript content and updates an evidence-linked review log. Its reward and training target valid, complete, technically grounded reviews.

3.1 Review Generation as a Markov Decision Process

  • Instead of a fixed pipeline, a learnable policy chooses what to read, what evidence to extract, and when to revisit; its state combines context, review log, and paper index.Environment actions retrieve paper information; log actions record, update, and outline, while external retrieval is excluded. Transitions update the context and log, and the episode ends at finish or the step limit.Source: §3.1 · p. 3

3.2 State Design: Review Log

  • The review log stores claims, questions, and intermediate notes; the agent updates entries as evidence emerges and requires final review points to cite supporting entry IDs.Claims carry source sections and verification status, questions have resolution status, and outline entries become the final review.Source: §3.2 · p. 4

3.3 Multi-dimensional Reward

  • The reward combines step-level action validity with trajectory-level review quality, targeting complementary capabilities at both interaction and final-review levels.The review-quality reward measures technical depth and grounding equally through rubric-based LLM-as-a-judge evaluation.Source: §3.1 · p. 3, §3.3 · p. 4, §3.3.2 · p. 5

3.3.1 Step-level Reward

  • The syntactic reward gives immediate per-step feedback on malformed schemas, invalid actions or arguments, and review points citing evidence absent from the log.Source: §3.3.1 · pp. 4–5

3.3.2 Trajectory-level Rewards

  • Trajectory-level rewards check for a summary, strength, weakness, and score; assess technical depth and grounding equally; and align predicted scores with human reviewer averages.Score alignment uses the rating-scale range κ (for example, κ = 9 on a 1–10 scale).Source: §3.3.2 · p. 5

3.4 Training

  • Training first applies SFT to teacher-distilled interaction trajectories, then GRPO in two phases: deterministic rewards first, followed by LLM-judge content-quality rewards.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate ProReviewer on version-matched ICLR papers using rubric-based automatic judgments and expert pairwise comparisons. ProReviewer leads the automatic and human evaluations across review-quality dimensions.

4.1 Experiment Setup

  • The evaluation uses 4,011 ICLR 2025 papers for training and validation and 1,000 version-matched ICLR 2026 papers for testing, alongside prompt-based, SFT, and single-turn RL baselines.The temporal split is intended to mitigate data contamination; fine-tuned methods use the same base models and training data as ProReviewer.Source: §4.1 · pp. 5–6

4.2 Evaluation Protocol

  • Three diverse LLM judges score actionability, grounding, verifiability, technical depth, and score alignment; five expert reviewers also make blind pairwise comparisons on 50 sampled papers.Content-quality scores use average and best-of-four reviews per method; score alignment measures calibration against mean human ratings.Source: §4.2 · p. 6

4.3.1 Automatic Evaluation Results

  • 0.57 for ProReviewer exceeds AI-Scientist-v2 (Qwen3.5-397B-A17B) at 0.46, while CycleReviewer and DeepReview score 0.36 and 0.35 on the same 8B backbone.ProReviewer’s score alignment remains competitive, and its Llama3.1-8B version scores 0.52 average, outperforming all baselines.Source: §4.3.1 · p. 6
  • 0.57 avg-of-4 and 0.65 best-of-4 are the highest overall scores for ProReviewer (Qwen3-8B) across the evaluated methods.Table 1 averages scores from three judges on a [0,1] scale.Source: §4.3.1 · p. 6, Table 1 · p. 7

4.3.2 Human Evaluation Results

  • In human pairwise evaluations against baselines, ProReviewer was preferred in every matchup on each dimension, with win rates of 51%–95%; Bradley–Terry ranking placed it first across all dimensions, with non-overlapping confidence intervals.Source: §4.3.2 · p. 7, Table 2 · p. 7

5 Discussion

Ablations report performance drops when the structured log or MDP formulation is removed. ProReviewer also leads on counterfactual error detection and maintains its scores across paper-length bins.

5.1 Ablation Study

  • In an ablation with ProReviewer’s multi-step agent loop unchanged, scores averaged across three judges fell from the full model by absolute 38% in grounding, 24% in technical depth, 15% in actionability, and 6% in verifiability.Source: §5.1 · p. 7, Table 3 · p. 7
  • Removing the MDP formulation and generating a review in one pass drops Actionability and Grounding by 13%, Technical Depth by 12%, and Verifiability by 2%.The paper associates the result with the value of sequential decision-making for iterative manuscript analysis and investigation.Source: §5.1 · p. 7

5.2 Counterfactual Error Detection

  • 27% is ProReviewer’s overall counterfactual error detection rate, 6 percentage points above AI-Scientist-V2; its detection rates stay between 24% and 29% across perturbation types.CycleReviewer and DeepReview each fall below 5% overall.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Robustness to Paper Length

  • 0.49→0.48 is ProReviewer’s average rubric score from the shortest to longest paper bins, while every baseline trends downward and ProReviewer leads on every bin.On the longest papers, ProReviewer exceeds AI-Scientist-v2 by 0.13 and Vanilla RL by 0.07.Source: §5.3 · p. 8, Figure 3 · p. 8

6 Conclusion

ProReviewer recasts automated peer review as proactive investigation guided by an MDP and a structured review log. Its 8B version outperforms larger prompt-based systems and fine-tuned baselines across automatic and human evaluation.

6 Conclusion

  • The MDP-guided review log tracks claims, questions, and notes so the agent can verify claims against later evidence, resolve open questions, and ground critiques in accumulated findings.The conclusion also reports robust performance on longer papers and detection of cross-section inconsistencies.Source: §6 · p. 8
  • ProReviewer with an 8B backbone outperforms larger prompt-based systems and fine-tuned baselines across automatic and human evaluation.The authors describe proactive investigation supported by evidence tracking as promising for LLM-assisted peer review and potentially for multi-step reasoning over complex documents.Source: §6 · p. 8

Limitations

The current ProReviewer implementation is limited to text-based, intra-manuscript reasoning and is trained and evaluated on ICLR papers. The authors identify multimodal perception, cross-domain adaptation, and external retrieval as possible extensions.

Limitations

  • The text-only agent cannot directly inspect figures, which may contain complementary evidence not accurately described in the paper text.The authors suggest multimodal perception could let it verify visual claims such as whether a reported trend matches a plotted curve.Source: Limitations · p. 9
  • ProReviewer is trained and evaluated on ICLR papers because other fields lack sufficient publicly available, clean manuscript–review pairs.The authors identify biomedicine and social sciences as potential adaptation targets once review data becomes available.Source: Limitations · p. 9
  • The implementation reasons within a manuscript and does not perform external novelty search.The authors distinguish novelty assessment as open-corpus retrieval, whose reliability depends on index coverage and corpus freshness.Source: Limitations · p. 9
  • The MDP formulation can accommodate multimodal perception, cross-domain adaptation, and external retrieval by adding actions without modifying the core architecture.The authors present these as future directions.Source: Limitations · p. 9

Ethical Considerations

The authors intend ProReviewer to support authors and reviewers, not replace human judgment. The supplied appendix passages also describe its action schema, review-log components, and system prompt.

Ethical Considerations

  • ProReviewer is intended to help authors identify potential issues before submission and to supplement reviewers’ issue identification, not provide a final judgment.The authors warn that misuse could produce superficial reviews or game review assignment, while overreliance could reduce human oversight and review quality.Source: Ethical Considerations · p. 9

A Action Schema

  • The action schema separates environment actions that interact with the paper from log actions that update the review log.Each turn outputs one environment action and a list of zero or more log actions.Source: A Action Schema · p. 12
  • The review log contains claims with verification status and source section, questions with resolution status, intermediate notes, and a final review outline whose entries cite evidence IDs.The appendix includes an example single-turn agent output in JSON format.Source: A Action Schema · p. 12

A.1 System Prompt

  • Figure 4 presents the complete system prompt used by the ProReviewer agent.Source: §A.1 · p. 12

B Dataset Details

The dataset uses version-matched ICLR manuscripts and reviews for training and temporally separated ICLR 2026 submissions for evaluation. Supervised traces are reconstructed from human reviews, and training uses a two-phase reward curriculum.

B Dataset Details

  • The evaluation samples 1,000 ICLR 2026 papers, separated in time from the training set to prevent data leakage.Training and validation use 4,011 ICLR 2025 submissions matched to their initial reviews and scores; the split is 90%/10%.Source: B Dataset Details · p. 12
  • The teacher reconstructs multi-turn reviewing traces from detailed human reviews with confidence ≥4, yielding 31,312 step-level instances from 1,485 papers.Traces cover reading sections, logging claims, raising questions, taking notes, verifying evidence, and building the review outline; they must cover at least all points in the reference review.Source: B Dataset Details · pp. 12–13

C Hyperparameters

  • Table 7 lists the SFT and GRPO RL training hyperparameters.Table 6 summarizes dataset statistics across splits, including average paper tokens, average human rating, and acceptance proportion.Source: B Dataset Details · p. 12, Table 7 · p. 13, Table 6 · p. 13
  • Training uses a two-phase reward curriculum: Phase 1 weights syntactic validity and completeness at 1.0 each and score alignment at 2.0; Phase 2 adds content quality at 2.0 and sets score alignment to 1.0.The content-quality reward combines technical depth and grounding.Source: C Hyperparameters · p. 13

D Complexity Analysis

ProReviewer uses repeated compact-state calls rather than processing full papers at each stage, with practical token use comparable to single-pass review. The accompanying rubric defines four review-quality dimensions.

D Complexity Analysis

  • For a typical 16K-token paper, ProReviewer uses more calls but smaller contexts (about 4K versus 12–20K tokens), with a comparable total token budget.Table 8 compares inference token consumption across paradigms.Source: Table 8 · p. 14, D Complexity Analysis · p. 14
  • ProReviewer’s total cost is O(T · c^2) ≪ O(N^2) for typical papers, while single-pass processing scales as O(N^2) and multi-stage processing as O(K · N^2).In practice, the final context is about 4–5K tokens, versus a full paper of about 12–20K tokens.Source: D Complexity Analysis · pp. 13–14

E Evaluation Rubrics

  • Grounding measures whether a review identifies the specific paper part, with structural locators scoring more strongly than an unlocated concept or method name.Structural references include section, table, figure, equation, or direct-quote locators.Source: E Evaluation Rubrics · p. 14
  • Actionability assesses whether suggested actions are explicit and concrete rather than inferred or vague.The highest level requires explicit actions with concrete implementation details.Source: E Evaluation Rubrics · p. 14
  • Technical Depth rates technical engagement and analytical reasoning, including whether an issue’s consequences are explained.Its criteria distinguish technically reasoned engagement from technical discussion without reasoning and non-technical reasoning.Source: E Evaluation Rubrics · pp. 14–15
  • Verifiability first checks whether a weakness makes a claim; claim-free observations score X, while claims are rated by how thoroughly they are supported.The claim scale ranges from fully supported to unsupported.Source: E Evaluation Rubrics · p. 15

F Human Evaluation

Human evaluators compare anonymized reviews, and Bradley–Terry analysis converts their pairwise judgments into relative strength estimates. The protocol also reports inter-annotator agreement.

F Human Evaluation

  • Five expert evaluators compare anonymized review pairs, with randomized A/B order and a rotating design covering all ten system pairs in aggregate.Fifty test papers are sampled; each evaluator reviews 30 papers, and about 20% overlap across evaluators supports agreement measurement.Source: F Human Evaluation · p. 15

F.2 Bradley-Terry Analysis

  • ProReviewer has the highest Bradley–Terry score on every dimension, with non-overlapping 95% confidence intervals against most baselines.Table 9 reports dimension-level Elo scores and 95% bootstrap confidence intervals.Source: Table 9 · p. 15, §F.2 · p. 15

F.1 Annotator Guidelines

  • Human evaluators receive complete guidelines for judging review quality.The guidelines are reproduced in Figure 5.Source: §F.1 · p. 15

F.2 Bradley-Terry Analysis

  • Pairwise judgments are fit with a Bradley–Terry model; ties count as half-wins, and scores use an Elo-like scale anchored at 1000.A 400-point difference corresponds to about a 10× strength ratio.Source: §F.2 · p. 15

F.3 Inter-Annotator Agreement

  • Agreement on the 20% overlap set is measured with Krippendorff’s α, Fleiss’ κ, and average pairwise quadratic-weighted Cohen’s κ2.Source: §F.3 · p. 15

G Counterfactual Error Detection

The appendices describe a counterfactual error-detection evaluation, supporting analyses, and a Surf3R case study illustrating ProReviewer’s investigation trajectory.

G Counterfactual Error Detection

  • The counterfactual evaluation uses 115 papers from six AI conferences, excluding ICLR 2025, with one injected logical error per paper across three perturbation types.The perturbations alter a conclusion, exaggerate a finding, or change a result to contradict its conclusion; an LLM judge checks whether reviews identify or imply the error.Source: G Counterfactual Error Detection · p. 16

H Paper Length Analysis

I Per-Judge Evaluation Results

J Case Study

  • In the Surf3R case study, a 30-step trajectory accumulates 12 claims, 6 questions, and 32 notes, with review points traced to evidence in the log.The trajectory shows suspicious claims flagged early, checked against experimental evidence, and updated as information emerges.Source: J Case Study · p. 16, Figure 6 · p. 21

Related papers

Loading 2606.13349v1…