ReviewRL: Towards Automated Scientific Review with RL

1 Introduction

• Rising conference and journal submission volumes create pressure on peer review, including reviewer fatigue, inconsistent evaluations, and longer review cycles.The introduction notes that NeurIPS and ICLR process thousands of submissions and require tens of thousands of reviews annually.Source: §1 · p. 1
• Automated reviews often struggle with factual accuracy, rating consistency, and analytical depth, producing superficial or generic feedback.The paper frames these as three significant challenges for existing approaches.Source: §1 · p. 1
• ReviewRL combines ArXiv-MCP retrieval, context-aware fine-tuning, and composed-reward RL training for automatic paper-review generation.The introduction describes retrieval as incorporating relevant scientific literature to support factual assessments.Source: §1 · p. 2
• Its reward system combines rule-based metrics with judge-model evaluations because rule-based rewards alone miss structural coherence and content quality.The authors say the combined system mitigates limitations of purely rule-based methods.Source: §1 · p. 2
• ReviewRL reports superior performance to prior work in context-awareness, factual consistency, and review depth.The paper describes the framework as a preliminary step toward RL-driven automatic critique generation.Source: §1 · p. 2

2 Related Works

• Review-generation research spans foundational datasets and benchmarks, two-stage review systems, end-to-end research agents, and SFT-trained reviewers, but factualness, reasoning depth, and rating consistency remain challenges.The cited examples include PeerRead and NLPeer, Reviewer2, CycleResearcher, The AI Scientist, and DeepReviewer.Source: §2 · p. 2
• RLHF aligns LLMs with human preferences, while GRPO uses rule-based rewards to encourage extended reasoning, particularly for mathematical problem-solving.The passage also describes applications of GRPO in vision, search, and planning.Source: §2 · pp. 2–3
• The potential of RL methods such as GRPO for review and critique generation remains underexplored (Whitehouse et al., 2025).Source: §2 · p. 3

3.1 Task Formulation

• ReviewRL retrieves context for a target paper, reasons over the paper and retrieved literature, then generates a review covering strengths and weaknesses plus a rating.The workflow uses retrieved context and an intermediate reasoning process before producing the review and rating.Source: §3.1 · p. 3

3.2 Context Retrieval

• Three natural-language queries probe a paper’s novelty, methodology, and relationship to prior work to retrieve relevant literature through ArXiv-MCP.Qwen3-8B generates the queries; retrieved metadata, abstracts, and relevant excerpts are consolidated with the target paper for review generation.Source: §3.2 · pp. 3–4

3.3 Supervised Finetuning

• Supervised fine-tuning on long chain-of-thought reviews initializes reviewing skills and rating alignment to address cold-start collapse in reinforcement learning.The training data uses the ICLR 2024 portion of DeepReview-13k, and the model is trained for two epochs on Qwen2.5-7B-Instruct.Source: §3.3 · p. 4

3.4 Reinforcement Learning

• The composite RL reward combines rating consistency and format adherence with a generative judge’s assessment of review quality.The judge assesses factual accuracy, completeness, detail, related-work comparison, constructiveness, and clarity; the rule-based reward weights rating consistency and format completeness.Source: §3.4 · pp. 4–5
• RL uses paper, context, human-averaged rating, and reference-review tuples, while initializing its policy from the supervised model to stabilize training.The training set draws on Reviewer2 and DeepReview-13k; mid-range ratings are downsampled and extreme ratings upsampled, and ICLR 2025 data are excluded to avoid leakage.Source: §3.4 · p. 5

4.1 Evaluation Data

• 472 ICLR 2025 papers are sampled with average human ratings distributed approximately uniformly across the rating scale.Each paper’s ground-truth rating is the average of scores assigned by all human reviewers.Source: §4.1 · p. 5

4.2.1 Rule-based Quantitative Metrics

• Rule-based evaluation measures score prediction with MSE and Spearman correlation, and paper ranking with relation, absolute, confidence, and concordance metrics.The pairwise metrics assess directional agreement with human rankings, score closeness to ground truth, and ability to discriminate among papers of varying quality.Source: §4.2.1 · pp. 5–6

4.2.2 Model-based Quantitative Metrics

• Model-based evaluation uses a judge to rate reviews on seven human-alignment dimensions, including topic coverage, claim correctness, and analytical depth.Each dimension is rated on a 1–5 scale; Llama-3.3-70B-Instruct is the judge model.Source: §4.2.2 · p. 6

4.3 Baselines

• Baselines include open-source instruction and reasoning models, alongside CycleReviewer-8B as a supervised peer-review model.The comparisons are intended to assess performance against basic instruction following, enhanced reasoning, and purely supervised approaches.Source: §4.3 · p. 6

5.1 Rule-based Evaluation Results

• ReviewRL achieves the strongest rule-based performance overall and consistently outperforms baselines across seven model-based review-quality dimensions.The rule-based results also show a performance gap over the SFT-only counterpart in rating consistency.Source: §5.1 · p. 6, §5.2 · p. 6, Table 2 · p. 7

5.2 Model-based Evaluation Results

• ReviewRL improves over its supervised-only counterpart across all seven model-based quality dimensions, with particular strength in analytical depth.The reported results characterize the resulting reviews as more informative, faithful, and constructive.Source: §5.2 · pp. 6–7

5.3 RL Training Dynamics

• During RL training, reward rises, response length stabilizes after about step 10, and evaluation MSE decreases.The paper interprets the trends as policy optimization, more detailed outputs, and improved held-out rating accuracy, respectively.Source: §5.3 · p. 7
• RL from scratch collapses ratings around 6; data balancing partly mitigates this, while SFT plus balancing yields full-spectrum ratings with stronger discrimination and alignment.The balancing strategy upweights examples with extreme ground-truth ratings; alone, it remains insufficient to resolve the cold-start issue.Source: §5.3 · p. 7
• With only rule-based rewards, RL shows no significant model-based evaluation gain over SFT and slightly lowers actionable insights.The ablation identifies GenRM reward as important for detailed reviews and reliable reasoning.Source: §5.3 · p. 7

5.4 Context Retrieval

• Retrieval-augmented answers beat vanilla answers in all three judged criteria: factual accuracy (95.0%), evidence quality (83.3%), and clarity and coherence (67.4%).The largest reported win rate is for factual accuracy.Source: §5.4 · p. 8, Figure 5 · p. 8
• Removing retrieval degrades review performance across metrics, especially factualness, and also reduces analytical depth and topic coverage.The authors suggest that related-work comparison may be less effective without retrieval.Source: §5.4 · p. 8

6 Conclusion

• ReviewRL combines context retrieval, supervised fine-tuning, and reinforcement learning to automate scientific paper reviews with accurate ratings.The authors describe the intended output as high-quality, human-aligned reviews.Source: §6 · p. 8
• On ICLR 2025 papers, ReviewRL significantly outperforms existing methods on both rule-based and model-based evaluation metrics.Source: §6 · p. 8
• The conclusion attributes successful learning from composite rewards in this non-verifiable task to properly initialized policy models that avoid cold-start issues.It also identifies retrieved context as important for factuality and analytical depth.Source: §6 · p. 8

• ArXiv’s coverage may leave emerging directions and specialized domains with limited repository representation without sufficient context.Source: Limitations · p. 8
• The seven-dimension reward may omit nuanced human-review criteria, including implicit domain and conference norms, limiting adaptation to specialized venues or interdisciplinary work.Source: Limitations · p. 9

Ethical Considerations

• ReviewRL is intended to support, not replace, human reviewers, with outputs serving as drafts for expert evaluation and refinement.The authors also describe curated training data, reward priorities, open-source documentation, user disclosure, and retrieval designed to minimize citation bias.Source: Ethical Considerations · p. 9
• The authors say they will open-source the system with architecture and training documentation and require users to disclose affiliation and intended use.Source: Ethical Considerations · p. 9

A Prompts

• Prompts for Generation, Evaluation, and GenRM are presented in Tables 6–10.Source: A Prompts · p. 11

B Pairwise Metrics

• Pairwise metrics compare model output ratings s1 and s2 with corresponding ground-truth values s*1 and s*2.Source: B Pairwise Metrics · p. 11
• Relation assesses directional consistency, Absolute score proximity, and Confidence differences in discriminative strength relative to human reviewers.Source: B Pairwise Metrics · p. 11
• The Llama-3.1-8B-Instruct table compares MSE after SFT versus Reinforce++ training, with lower MSE better.Source: Table 3 · p. 11

C.1 RL Training

• RL training used dual 8×A800 GPU clusters, global batch size 128, eight-sample rollouts, and γ = 0.5, completing 15 optimization steps in 48 hours.DeepSpeed ZeRO-3 and Ray were used; six GPUs went to vLLM and two to GenRM.Source: §C.1 · p. 11

C.2 SFT Training

• SFT used DeepSpeed ZeRO-3 with batch size 8 and learning rate 5e-6.Source: §C.2 · p. 11

C.3 Model-based Evaluation

• ReviewRL achieves the highest score across all evaluated dimensions, with particularly significant gains in analytical depth and factuality.The reported dimensions are strongly positively correlated, suggesting high-quality scientific reviewing involves integrated capabilities rather than isolated skills.Source: §C.3 · p. 11

C.4 Comparison across RL Algorithms

• RL models consistently outperform the SFT baseline under Reinforce++, PPO, and GRPO, supporting robustness across the tested algorithms.Source: §C.4 · p. 11

C.5 Model Generality across Architectures

• The same training recipe reduces MSE relative to SFT on Llama-3.1-8B-Instruct, mirroring Qwen results and supporting generalization across model families.The section reports this as confirmation that the REVIEWRL training recipe generalizes across model families.Source: §C.5 · p. 11