PaperlayerView exact revision on arXiv

Source-linked AI summary

InternReviewer & InternAdvocate: Objective Reward and Evaluation for Agentic Reinforcement Learning in Peer Review and Rebuttal

Xuerui Su, Liya Guo, Qizhi Pei, Qipeng Guo, Zhongbo Tian, Lijun Wu, Kai Chen, Zun Wang

arXiv:2608.28612v1cs.AI

TL;DR

Professional peer reviews and rebuttals require both domain reasoning and factual grounding, while existing systems can produce ungrounded or formulaic criticism. The paper develops InternReviewer and InternAdvocate with agentic reinforcement learning, retrieval-based evidence gathering, and objective rewards, and reports superior performance under human preference evaluation on a held-out benchmark.

  • Problem

    Existing automated review systems can produce hallucinated claims, fabricated citations, or formulaic criticism, while subjective evaluation provides unstable training signals.

  • Method

    The paper trains InternReviewer and InternAdvocate using agentic reinforcement learning, a large-scale review dataset, real-time retrieval, decomposed objective rewards, and citation verification against interaction logs.

  • Results

    InternReviewer and InternAdvocate surpass GPT-5 and Gemini under human preference evaluation on a rigorous held-out benchmark.

  • Takeaways & Limitations

    Task-specific agentic training is reported as a more effective pathway to high-quality AI review than scaling model size under training-free approaches.

  • Takeaways & Limitations

    Training-free review systems depend on proprietary model deployments and tend toward sycophantic, generic criticism.

Abstract

from arXiv · show

Generating professional scholarly content, such as peer reviews and rebuttals, requires an intricate synergy between domain reasoning and factual grounding. This work presents a comprehensive framework for the development and evaluation of specialized scholarly agents, InternReviewer and InternAdvocate. We first establish a large-scale, high-quality scholarly dataset and integrate a high-efficiency arXiv retrieval tool to enable active evidence gathering. To optimize these agents, we implement an agentic Reinforcement Learning (RL) paradigm driven by a unified objective metric and reward system. This system avoids the biases of subjective model-based judging by employing multi-dimensional criteria, including reference-anchored semantic alignment, structural compliance, and a strict verification mechanism that cross-checks citations against real-time interaction logs to eliminate hallucinations. Experimental results demonstrate that agents trained within this closed-loop framework exhibit significant improvements in reasoning depth and citation accuracy.

1. Introduction

Automated peer review faces rising submission volumes, reviewer fatigue, inconsistent feedback, and hallucination-prone LLM assistance. The paper introduces specialized agents and objective, evidence-grounded training to address these problems.

  • Motivation: Submission growth has produced severe reviewer fatigue and inconsistent feedback, while vanilla LLMs may invent flaws or nonexistent literature.These failures motivate agentic systems that verify claims through external tools.
  • Existing Approaches: Training-free systems rely on prompting, whereas SFT can imitate reviewers’ superficial styles while inheriting bias and noise.The supplied passage truncates the remainder of the comparison.
  • Evaluation Gap: Subjective judging and static preference rubrics yield inconsistent reward signals, potentially prioritizing stylistic mimicry over factual accuracy.This instability is attributed to the subjectivity of scholarly criticism and variation across models or prompts.
  • Proposed Framework: InternReviewer and InternAdvocate use agentic reinforcement learning for peer review and rebuttal, respectively.Their framework combines semantic alignment with expert feedback, factual validation, and real-time arXiv search for verifiable citations.

2. Related Work

Prior AI-review systems use proprietary models, supervised or reinforcement learning, and sequential retrieval pipelines, but remain limited in training adaptability, grounded critique, or tool-supported novelty assessment. This work combines broader training data, intrinsic literature retrieval, and a more comprehensive evaluation protocol.

  • Training-Free Approaches: Training-free systems use proprietary deployments and tend toward formulaic, sycophantic reviews lacking incisive, literature-grounded critique.The paper therefore emphasizes substantiation and groundedness rather than review length or weakness coverage alone.
  • Training-Based Approaches: DeepReviewer and ReviewRL show benefits from supervised domain adaptation and subsequent reinforcement learning, but train without tool use.Without retrieval, these systems have limited ability to assess novelty against the actual research landscape.
  • This Work: The proposed agents are trained on paper–review–rebuttal triplets from NeurIPS, ICLR, and ICML OpenReview platforms.The dataset is presented as broader and more diverse than prior training foundations.
  • This Work: The framework makes literature retrieval intrinsic to review and rebuttal behavior and replaces BLEU-, paper-score-, and LLM-as-a-Judge evaluation with a broader protocol.This design is intended to capture substantive differences between generated and human-level scholarly communication.

3. Methods

The framework formalizes peer review and rebuttal as agentic sequential decision-making, combining retrieval and generation with curated scholarly data, objective rewards, and GSPO optimization. It grounds outputs in retrieved evidence while rewarding semantic alignment, structural compliance, trajectory-level tool use, and citation verification.

  • Problem Formulation: The tasks are modeled as an MDP where agents interleave reasoning, retrieval, and generation actions over manuscript and dialogue context.States include manuscript content, dialogue history, retrieved evidence, and prior reasoning trajectories; rewards optimize semantic alignment and factual grounding.
  • Data Collection and Curation: The pipeline parses PDFs into structured Markdown, truncates inputs to the main paper body, removes references, and anonymizes author identities.MinerU preserves sectional hierarchies, while preprocessing mirrors double-blind review constraints and keeps inputs within the effective context window.
  • Evidence-grounded Search Infrastructure: A local high-throughput arXiv search tool provides closed-loop evidence gathering, with retrieval latency treated as a constraint on stable RL training.The search logs also support citation verification against successfully retrieved paper metadata.
  • Objective Reward Function: The objective reward combines semantic alignment, factual grounding, structural format compliance, and step-wise tool-use incentives.Format rewards enforce role-specific sections and appropriate length, while tool-use rewards encourage necessary evidence gathering without redundant retrieval loops.
  • Policy Optimization via GSPO: GSPO optimizes complete sampled trajectories using a length-normalized sequence-level importance ratio and group-normalized advantages.This design is intended to stabilize policy updates while supporting multi-step reasoning, iterative tool use, and citation verification.

4. Experiments

Experiments evaluate semantic, lexical, behavioral, citation, and hallucination performance for InternReviewer and InternAdvocate. Across review and rebuttal tasks, the results favor objective, task-aligned metrics and show rapid learning of some reward components but slower improvement in hallucination suppression.

  • Lexical Overlap Metrics: Lexical overlap metrics can reward superficial vocabulary matching and penalize semantically equivalent phrasing, making them secondary diagnostics for model selection.Qwen3-30B-A3B-Thinking-2507 achieves BLEU and ROUGE scores competitive with or superior to GPT-5.2 despite the reported mismatch with substantive quality.
  • Agentic Behavioral Metrics: InternReviewer and InternAdvocate obtain near-perfect Format scores of 1.5000 and 1.4910, respectively, indicating strong structural compliance.GPT-5.2 also reaches a competitive review-task Format score of 1.4648, showing that instruction following can approximate structural norms.
  • Agentic Behavioral Metrics: Frontier models show suboptimal Tool behavior, including bypassed, undirected, or excessive retrieval that departs from the prescribed interaction budget.GPT-5.2 receives markedly negative Tool scores on both tasks because of excessive retrieval.
  • Citation Integrity and Hallucination Suppression: 0.7920 and 0.7620 Citation scores, alongside 0.8560 and 0.8620 Hallucination scores, show large gains for InternReviewer and InternAdvocate over closed-source competitors.The reported improvements exceed 1.5 absolute points on both metrics and are attributed to citation-consistency rewards and a zero-tolerance hallucination penalty.
  • Semantic Quality Metrics: InternReviewer and InternAdvocate achieve the best POLAR scores on their respective tasks, while GPT-5.2 ranks first on Reranker scores.The trained agents nevertheless rank second on Reranker, combining strong semantic performance with task-aligned POLAR superiority.
  • Training Dynamics: Format and Tool rewards saturate within the first 100 training steps, whereas Hallucination reward improves gradually across the full training horizon.InternReviewer citation reward rises from -0.9283 to 0.8241 by step 100, while InternAdvocate rises from -0.2406 to 0.6923 before plateauing.

5. Conclusion

The paper presents InternReviewer and InternAdvocate as specialized 30B MoE agents trained with Agentic Reinforcement Learning for peer review and rebuttal. It combines objective rewards, a multi-turn dataset, and retrieval infrastructure, while identifying several directions for future work.

  • InternReviewer and InternAdvocate are specialized agents trained via Agentic Reinforcement Learning for peer review and rebuttal, respectively.
  • The framework combines a large-scale multi-turn review dataset, a decomposed objective reward function, and high-throughput retrieval infrastructure.
  • The resulting 30B MoE models surpass GPT-5 and Gemini under human preference evaluation on a rigorous held-out benchmark.
  • Future work includes long-context processing, broader retrieval sources, explicit hallucination metrics, and multi-agent training for reviewer–author dialogue.
  • A trained review agent can provide a fast, consistent, literature-grounded quality signal within the broader AI Scientist workflow.

A.1. Training Details

The training setup applies agentic reinforcement learning directly to Qwen3-30B-A3B-Thinking without a supervised fine-tuning cold start. The associated prompts enforce structured review and rebuttal generation, tool use, citation integrity, and explicit handling of experiment requests.

  • Training Details: GSPO-based agentic RL training starts directly from Qwen3-30B-A3B-Thinking-2507 without supervised fine-tuning.
  • Reviewer Agent: The reviewer prompt requires a structured output containing Summary, Strengths, Weaknesses, Questions, and References.
  • Tool Use: Agents must use arxiv_search when external information is needed, set an end_date, and analyze returned papers before producing the final output.
  • Advocate Agent: The rebuttal prompt requires point-by-point responses that include the reviewer comment, response, and manuscript changes.
  • Citation Standards: Citation rules require sequential inline numbering, exact metadata from search results, and no fabricated citations.
  • Experiment Requests: For experiment requests, the prompt permits either principled arguments against additional experiments or acknowledgment that needed experiments were not run.

A.3. arxiv_search Tool

The arxiv_search infrastructure offers online, hybrid, and fully local variants for retrieval and PDF access. The fully local design avoids network instability and provides millisecond-level retrieval suited to high-frequency reinforcement-learning rollouts.

  • Tool Variants: The system provides three arxiv_search variants: fully online, hybrid Elasticsearch plus GCloud, and fully local Elasticsearch plus a local paper database.
  • Online Variant: The online variant has 6.707 s/req retrieval latency, 100–300 KB/s download bandwidth, and possible concurrency-related failures.
  • Hybrid Variant: The hybrid variant reduces retrieval latency by over 200× to 0.030 s/req but retains unstable GCloud download throughput caused by egress quotas.
  • Fully Local Variant: The fully local variant eliminates the network sources of instability and supports high-throughput training.
  • Fully Local Variant: At 0.014 s/req, fully local retrieval adds negligible overhead relative to model inference and avoids becoming the RL training bottleneck.

B. Challenges of LLM-as-a-Judge in Open-Ended Response Evaluation

The paper argues that LLM-as-a-Judge is unreliable for open-ended review evaluation because judgments depend on candidate order and vary across repeated trials. These issues motivate replacing it as a reward or evaluation signal.

  • Background: LLM-as-a-Judge evaluates generated outputs through scalar scores or pairwise preferences and is widely used because of low cost and correlation with human judgments.
  • Position Bias: Position swapping changes win rates substantially, showing that judge decisions depend on the relative placement of candidate reviews.
  • High Variance: Repeated evaluations produce coefficients of variation above 20%, exceeding the typical 1%–1.5% range for stable settings.
  • Implications: Position bias and high variance make LLM-as-a-Judge an unstable and biased preference signal for optimization and evaluation.
  • C.1. Review: Speculative Search addresses reasoning latency with small–large model collaboration and quality-preserving rejection, achieving up to 2.12× speedup with comparable reasoning quality.

C.1.3. Model Reasoning

Speculative Search addresses the inference bottleneck in tree-search reasoning by accelerating thought generation through small–large model collaboration at thought and token levels. The reviewed evidence reports substantial speedups with comparable accuracy, while noting concerns about threshold estimation, theoretical assumptions, and unexplained accuracy changes.

  • Problem: Thought generation consumes over 91% of runtime in tree-search reasoning, creating a major inference-latency bottleneck.The paper identifies the cost of generating numerous intermediate reasoning thoughts as the central efficiency problem.
  • Method: SpecSearch uses a small model to generate candidate thoughts and token-level correction to accelerate large-model reasoning.The framework combines coarse-grained thought speculation with fine-grained lossless speculative decoding for rejected thoughts.
  • Method: Quality-preserving rejection evaluates whole thoughts against a large model’s quality threshold rather than verifying only tokens.The mechanism is designed to reject low-quality thoughts while preserving reasoning quality.
  • Limitations: The review raises unresolved concerns about vague threshold estimation, strong distributional assumptions, GSM8K accuracy degradation, limited ablations, and unreported overhead.It specifically notes the unexplained threshold computation and the GSM8K difference of 87% versus 96% for SpecSearch versus AR.
  • Novelty: The approach is presented as novel because it performs speculation at the thought level, unlike standard token-level speculative decoding.The review also characterizes the bi-level formulation as matching the structure of tree-search reasoning.

C.1.5. Model Reasoning

SpecSearch is motivated by the high latency of tree-search reasoning and proposes bi-level speculation with a quality-preserving rejection mechanism. The review finds the contribution practical and novel, but identifies underexplained threshold estimation, strong assumptions, and limited discussion of accuracy degradation and overhead.

  • Motivation: Tree-search methods improve reasoning but suffer high latency because thought generation consumes over 91% of runtime.The bottleneck arises from generating many intermediate reasoning thoughts.
  • Approach: SpecSearch collaboratively generates candidate thoughts with a small and large model at thought and token levels.The thought-level stage proposes multiple candidates, while token-level correction handles thoughts that fail quality evaluation.
  • Mechanism: The quality-preserving rejection mechanism evaluates complete thoughts and rejects those below the large model’s estimated quality threshold.The threshold is estimated from historical reasoning steps using an exponential moving average of accepted reward scores.
  • Results: 2.12x speedup is reported on MATH-100 with 87% accuracy, compared with 88% for SpS and 87% for AR.Inference time is reported as 82.35s for SpecSearch, versus 141.55s for SpS and 275.78s for AR.
  • Novelty: The bi-level formulation extends prior token-level speculative decoding to the thought units used in tree-search reasoning.The review considers this alignment with reasoning structure a meaningful novelty.
  • Limitations: The review questions the practical explanation of threshold estimation and the validity of assumptions about normal and descending thought qualities.It also notes GSM8K accuracy of 96% versus 97% for the baseline, without sufficient main-text explanation.

C.1.6. Generated Review

The generated review presents SpecSearch as a method for reducing tree-search reasoning latency through small–large model collaboration and quality-based rejection. It reports up to 2.12x speedup with comparable reasoning quality, while emphasizing limited explanation of threshold estimation and domain coverage.

  • Summary: SpecSearch accelerates tree-search reasoning by addressing the latency caused by generating many intermediate thoughts.The paper identifies thought generation as the main computational bottleneck, consuming 91% of runtime.
  • Method: The method combines thought-level candidate generation by a small model with token-level correction and large-model verification.Historical reasoning-step rewards are used with an exponential moving average to estimate the large model’s quality threshold.
  • Results: 2.12x speedup is achieved across Qwen and Llama experiments while maintaining comparable reasoning quality.The reported evaluations use the MATH and GSM8K datasets.
  • Limitations: The generated review identifies threshold estimation as insufficiently explained in the main paper.It notes that the threshold symbol is presented without enough practical detail about computation or parameters.
  • Limitations: The evaluation reports GSM8K accuracy degradation and relies primarily on mathematical reasoning datasets.The review requests analysis of failure cases and evaluation beyond the math domain.
  • Mechanism: The quality-preserving rejection mechanism filters thoughts below the large model’s output quality threshold.The paper presents this mechanism as the basis for retaining comparable reasoning quality during acceleration.

C.2.2. Reviewer Input

Reviewer input characterizes SpecSearch as a novel, quality-preserving acceleration method with substantial speedups and appropriate evaluation metrics. It also calls for broader datasets, stronger baselines, error bars, hardware measurements, and analysis of failure cases.

  • Results: 2–3x speedup with minimal accuracy drop is reported on mathematical reasoning tasks.The reviewer notes that empirical evidence is limited to MATH and GSM8K.
  • Assessment: The reviewer considers the SpecSearch and quality-preserving rejection claims substantiated.The theoretical guarantee is also described as substantiated, though the reviewer did not check the proofs in detail.
  • Method: The method uses a small language model for draft thoughts and a large model for final verification with tree- or beam-based search.A process reward model scores the reasoning trajectory.
  • Evaluation: Further comparisons with newer speculative approaches and additional baselines are requested.The reviewer also asks for wall-clock measurements on real hardware and error bars across random seeds.

C.2.3. Model Reasoning

The rebuttal reasoning addresses dataset coverage, baselines, theoretical guarantees, model sensitivity, and non-mathematical evaluation. It emphasizes SpecSearch’s tree-based, bi-level speculative execution and reports consistent acceleration across settings.

  • Additional evaluation reportedly includes full GSM8K, AIME, Olympiad Bench, and a code-generation benchmark, although the main results focus on standard math datasets.
  • The response cites comparisons with AR and SpS as relevant baselines for accelerating thought generation without modifying search algorithms or prompting techniques.
  • SpecSearch is presented as a method that generalizes speculative execution to tree-based reasoning at both thought and token levels.
  • 2.18x to 2.87x speedups are reported across different small draft-model settings, alongside robust performance across a wide EMA-weight range.
  • The response identifies limited dataset diversity, missing error bars and baselines, comparison with SEED, small-model sensitivity, and non-mathematical evaluation as the main concerns.

C.2.6. Generated Rebuttal

The generated rebuttal responds to concerns about dataset diversity, baselines, theoretical support, model sensitivity, hardware timing, and domain generalization. It reports consistent speedups, quality preservation, and broader evaluations beyond the main math benchmarks.

  • The rebuttal acknowledges that the main experiments emphasize MATH and GSM8K while pointing to additional appendix evaluations across broader dataset categories.
  • 2.12x acceleration is reported on a code-generation task while preserving accuracy, supporting evaluation beyond mathematical reasoning.
  • The authors compare against AR and SpS and explain that the evaluation targets thought-generation acceleration without changing search algorithms or prompting techniques.
  • 88% to 78% or 80% on MATH-50 is the reported accuracy drop when evaluation or rejection components are removed.
  • 2.18x to 2.87x speedups are reported across multiple small draft models while maintaining comparable accuracy to the large model.
  • Up to 3.35x speedup over AR on MATH-100 with Qwen is attributed to generating complete reasoning thoughts before large-model verification.
Loading 2608.28612v1…