Source-linked AI summary
REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning
Pawin Taechoyotin, Daniel Acuna
TL;DR
AI review systems can overpraise and provide shallow suggestions, and review quality is difficult to operationalize. REMOR combines reasoning, a multi-aspect reward, and reinforcement learning; it reports roughly double HPRR versus typical human reviews while approaching the best human reviews.
Problem
AI-generated reviews can overpraise, give generic comments, and overlook details, while good-review criteria have not been operationalized for AI.
Source: §1 · p. 2Method
REMOR fine-tunes a reasoning model on PeerRT and applies GRPO using a multi-aspect HPRR reward with human-aligned or uniform weights.
Source: §1 · p. 2Results
Approximately double HPRR versus typical human reviews, with REMOR feedback approaching parity with the best-quality human reviews.
Source: §7 · p. 9Takeaways & Limitations
HPRR can serve as a self-assessment tool for reviewers, encouraging more detailed feedback.
Source: §7 · p. 9Evaluation relies on a constructed human-aligned reward rather than direct human feedback; blinded expert evaluation should confirm the findings.
Source: §6 · p. 9
Abstract
from arXiv · showhide
AI-based peer review systems tend to produce shallow and overpraising suggestions compared to human feedback. Here, we evaluate how well a reasoning LLM trained with multi-objective reinforcement learning (REMOR) can overcome these limitations. We start by designing a multi-aspect reward function that aligns with human evaluation of reviews. The aspects are related to the review itself (e.g., criticisms, novelty) and the relationship between the review and the manuscript (i.e., relevance). First, we perform supervised fine-tuning of DeepSeek-R1-Distill-Qwen-7B using LoRA on PeerRT, a new dataset of high-quality top AI conference reviews enriched with reasoning traces. We then apply Group Relative Policy Optimization (GRPO) to train two models: REMOR-H (with the human-aligned reward) and REMOR-U (with a uniform reward). Interestingly, the human-aligned reward penalizes aspects typically associated with strong reviews, leading REMOR-U to produce qualitatively more substantive feedback. Our results show that REMOR-U and REMOR-H achieve more than twice the average rewards of human reviews, non-reasoning state-of-the-art agentic multi-modal AI review systems, and general commercial LLM baselines. We found that while the best AI and human reviews are comparable in quality, REMOR avoids the long tail of low-quality human reviews. We discuss how reasoning is key to achieving these improvements and release the Human-aligned Peer Review Reward (HPRR) function, the Peer Review Reasoning-enriched Traces (PeerRT) dataset, and the REMOR models, which we believe can help spur progress in the area.
1 Introduction
AI review systems can produce overpraising, generic feedback that misses details, while criteria for a good review remain unoperationalized for AI. REMOR combines reasoning with multi-aspect rewards and reports higher review rewards than human and other AI systems.
1 Introduction
- Existing AI-generated comments can overpraise work, remain generic, and overlook details, while good-review criteria are not operationalized for AI.Existing review-generation frameworks may also require human oversight.Source: §1 · p. 2
- Twice as high rewards were reported for REMOR reviews as for reviews generated by humans and other agentic and non-agentic systems.The comparison is part of the paper’s quantitative and qualitative evaluation.Source: §1 · p. 2
- REMOR uses reasoning and a multi-objective reward strategy intended to deepen reviews and operationalize review quality.The framework uses PeerRT for fine-tuning, then GRPO with HPRR; REMOR-H uses human-aligned preferences and REMOR-U uses uniform weights.Source: §1 · p. 2
- The contributions include PeerRT, HPRR, a specialized review model, comparisons with other systems, and qualitative analysis of generated reviews.PeerRT enriches peer reviews with reasoning traces; HPRR operationalizes review feedback quality for reinforcement learning.Source: §1 · p. 2
2 Related Work
Prior work explores agent-based review generation, automated review evaluation, and reasoning with reinforcement learning. This section also highlights trade-offs in agent-panel coordination and the challenge of evaluating reviews consistently.
2.1 Non-reasoning AI-based Peer Review Generation Systems
- Agent-panel systems assign review aspects to different agents, and large context windows, multimodality, and external tools have been reported to improve review quality [9].Agent panels can take 20 minutes or more and use millions of tokens, many for coordination [9].Source: §2.1 · pp. 2–3
2.2 Evaluating peer review generation systems
- Evaluation approaches include human judgments, text-similarity metrics, LLM judges, and proposed criteria covering comparison, factual accuracy, analytical depth, and actionable insights [9] [48] [51] [18].Many evaluation methods assume human reviews are available, although good-quality human reviews may be scarce as AI systems improve.Source: §2.2 · pp. 3–4
2.3 Reasoning, Reinforcement Learning, and GRPO
- Reasoning models build on chains of thought, while GRPO optimizes a policy using generation-trace reward values rather than requiring an explicit reward model [45] [8] [36].The related work describes GRPO as improving memory use and training performance for LLM reinforcement learning.Source: §2.3 · p. 4
3 Reasoning and Reinforcement Learning for Automated Peer Review Generation
The method pairs supervised fine-tuning on review data with multi-objective reinforcement learning. It constructs a review-and-manuscript reward and compares uniform weights with weights fitted from human preferences.
3 Reasoning and Reinforcement Learning for Automated Peer Review Generation
3.1.1 Data curation
- PeerRT draws on 16.8K ICLR 2017–2020 reviews across 5.5K submissions, adds full paper text, and augments reviews with Claude Sonnet 3.7 extended-thinking traces.The traces are intended to encourage the fine-tuned model to generate thinking traces.Source: §3.1.1 · p. 4
3.1.2 Models
- Claude Sonnet 3.7 and DeepSeek-R1 (7B distillation) are used for comparison, while DeepSeek-R1 is used during SFT and GRPO.The models passage identifies Claude Sonnet 3.7 as a large proprietary model.Source: §3.1.2 · pp. 4–5
3.1.3 Reward Model
- The reward combines sentence-level assessments of review quality, normalized by sentence count, with METEOR relevance between the review and manuscript.The assessed aspects include criticism, examples, importance and relevance, methods, presentation, results and discussion, and suggestions and solutions.Source: §3.1.3 · p. 5
3.2 Multi-objective reinforcement learning
- GRPO uses a weighted sum of reward aspects, with either uniform weights or human-aligned weights computed from human preferences.The human-aligned weighting boosts aspects preferred in pairwise review comparisons.Source: §3.2 · p. 5, §3.2.1 · p. 5
3.2.1 Obtaining reward weights from human preferences
- Human-preference vote pairs inform Bradley–Terry models; ABT models ties with positive weights, while CRM adds L1 regularization.The fitted weights are min-max scaled, rescaled to sum to nine, and Laplace-smoothed with α = 0.01 to keep any aspect from being discarded.Source: §3.2.1 · p. 5
4 Experimental results
REMOR variants outperform comparison systems on peer-review rewards, while reward design shapes which model scores highest. Learning-curve and qualitative analyses highlight metric difficulty and differences in review length and clarity.
4.1 Experimental Setup
- The team selected samples above the 90th percentile of uniform reward for SFT and GRPO, using DeepSpeed, LLaMA-Factory, and TRL.Training used one server with two A100 80GB GPUs; DeepSpeed handled multi-device optimizer states, while LLaMA-Factory supported SFT.Source: §4.1 · p. 5, §4.1.1 · p. 6, §4.1.3 · p. 6
4.1.2 Supervised fine-tuning
- SFT trained DeepSeek-R1-Qwen-Distilled-7B via LoRA for three epochs on 1.7k samples, taking approximately 6.5 hours.GRPO used four generations and ran one epoch on 1.7k samples, taking approximately 100 hours and 864 optimization steps.Source: §4.1.2 · p. 6, §4.1.3 · p. 6
4.2 Human-aligned reward vs. uniform rewards
- Human preference weights favored Importance and Relevance, Suggestions and Solution, and Relevance (METEOR), at 0.11, 0.16, and 8.67, respectively.Other metrics received weight 0.01, the Laplace smoothing parameter.Source: §4.2 · p. 6
4.3 Reasoning model results
- After GRPO, REMOR-U reached 3.884 uniform reward and REMOR-H reached 0.670 human-aligned reward, both significantly exceeding humans and Sonnet 3.7.SFT alone beat humans and Sonnet 3.7 on uniform reward but slightly trailed on human-aligned reward; pretrained DeepSeek-R1 scored 1.317 uniform and 0.283 human-aligned reward.Source: §4.3 · p. 6, Figure 2 · p. 7
4.4 Learning Curve For GRPO
- REMOR-U’s learning curves grouped metrics by peak reward: hard at 0.1, moderate at 0.4–0.6, and easy at 0.8.Relevance METEOR was classified as hard because a score of 1.0 would require an excessively long review; the authors suggest future metric-specific normalization.Source: §4.4 · pp. 6–7, Figure 3 · p. 7
4.5 Non-reasoning, agentic systems for peer review generation
- Both REMOR variants significantly outperformed human reviewers, non-reasoning systems, and other reasoning models; REMOR-U averaged 3.292 uniform reward and REMOR-H averaged 1.438 human-aligned reward.REMOR-U also led the reported review aspects, while REMOR-H scored higher than REMOR-U on materials and methods despite a much lower human-aligned weight for that aspect.Source: §4.5 · pp. 7–8, Figure 4 · p. 8
4.6 Variance analysis
- REMOR-U had the highest normalized mean and lowest variance across metrics, while DS SFT ranked second with half its mean score.The authors suggest the combination may lead to more consistent review outcomes.Source: Figure 5 · p. 8, §4.6 · p. 8
4.7 Qualitative Analysis
- High-reward human and REMOR-U reviews contained useful comments, and REMOR-U’s feedback was direct and concise; REMOR-H tended to be long, redundant, and difficult to understand.The authors attribute this difference to excessive focus on METEOR and say human comparison is needed to confirm the correlation between uniform reward and review quality.Source: §4.7 · pp. 8–9
5 Discussion
REMOR’s reasoning and multi-objective training improve review performance, but human preferences do not consistently align with the principles of a strong review. The specialized models outperform complex agentic systems and process reviews faster.
5 Discussion
- After fine-tuning and GRPO, the model’s multi-aspect reward performance increased dramatically and surpassed human performance on the evaluated tasks.The authors attribute this improvement to the combination of fine-tuning and GRPO.Source: §5 · p. 9
- Human preferences disfavor criticism despite guidelines encouraging it, while favoring importance, relevance, and suggestions, revealing a gap between preferences and review principles.The authors suggest criticism may be perceived as a net negative.Source: §5 · p. 9
- General reasoning models perform comparably to multi-agent review systems, but specialized REMOR models outperform MARG-S [9] and MAMORX and take about 1 minute rather than 20–30 minutes.The authors present simple-prompt reasoning models as a promising alternative for augmenting peer review.Source: §5 · p. 9
- Reasoning-based REMOR training substantially outperforms complex agentic review systems, including MAMORX, while optimizing multiple metrics simultaneously.The reported results also show better performance than systems using figures and external knowledge.Source: §5 · p. 9
6 Limitations and Future work
The study’s evaluation is limited by narrow review coverage, omitted modalities, context-window constraints, and a reward model based on relatively few votes. Its findings still need confirmation through blinded expert review.
6 Limitations and Future work
- The evaluation uses a narrow review set, omits citations and figures, and has a limited context window that may prevent complete manuscript analysis or inclusion of raw external knowledge.The authors state that future work will address these shortcomings.Source: §6 · p. 9
- The human-aligned reward model is based on a relatively small vote sample, and the evaluation uses a constructed reward rather than direct human feedback.The authors say a blinded evaluation by experts should confirm the findings.Source: §6 · p. 9
7 Conclusion
The authors conclude that explicit reasoning and multi-objective reinforcement learning improve automated review depth and quality. They report roughly double the HPRR of typical human reviews and release the dataset, metric, and models for further research.
7 Conclusion
- REMOR feedback achieves approximately double the Human-aligned Peer Review Reward of typical human reviews, approaching parity with the best human-generated reviews.The authors state that explicit reasoning combined with multi-objective reinforcement learning enhances review depth and quality.Source: §7 · p. 9
- The authors release PeerRT, the HPRR evaluation metric, and trained REMOR models, with future work planned on multimodal inputs and broader scientific disciplines.The proposed reward function may also serve as a reviewer self-assessment tool encouraging more detailed and relevant feedback.Source: §7 · pp. 9–10
A Metric names
The appendix names the review metrics and describes how human-aligned weights are calculated and used in the constrained reward model. It also provides the prompts used for supervised fine-tuning and reinforcement learning.
A Metric names
- Metric abbreviations include Cr, Ex, ImRe, MaMe, Pr, PrRe, ReDi, SuSo, and ReME for relevance METEOR.The short names are intended to make results tables easier to read.Source: A Metric names · p. 14
B Human values weights calculation
- The appendix section covers the calculation of human-aligned and intermediate weights that have not been adjusted.The supplied passages do not give the calculation details themselves.Source: B Human values weights calculation · p. 14
B.1 Constrained Reward Model (CRM)
- The Constrained Reward Model uses convex optimization to find human-aligned weights from review scores and matched-review outcomes.The constraints encode preferences for A, preferences for B, or a tie; ε = 10^-12 supports numerical stability for strict inequalities.Source: §B.1 · p. 14
C.1 User Message prompt for Supervised Fine Tuning and Reinforcement Learning
- The user-message template instructs the model to act as a scientific-community reviewer and review the supplied paper content.The template text includes the instruction to review the paper content.Source: §C.1 · p. 14
C.2 Assistant Prompts for Supervised Fine Tuning
- The supervised fine-tuning target template places Sonnet 3.7 thinking traces inside a think block.The target response template is shown as an assistant prompt.Source: §C.2 · p. 16
D Models and Dataset
The appendix lists the released REMOR model variants and PeerRT-related datasets, and points to scripts for processing data, loading datasets and models, and generating text.
D Models and Dataset
- The listed releases include two SFT models, REMOR-U and REMOR-H, and three ICLR 2017–2020 review datasets with different reasoning-trace or paper-text contents.The listed models are hosted under pawin205, as are the datasets.Source: D Models and Dataset · p. 16
E Code Repository
- The repository includes scripts for processing data, loading datasets and models, and performing text generation.The appendix says example loading scripts are included.Source: E Code Repository · p. 16
F Model performance across different metrics
The performance section presents model scores across metrics and reward schemes, alongside a comparison with prior systems. A separate figure describes the distribution of uniform reward for human and REMOR-U reviews.
F Model performance across different metrics
- Table 4 compares metric-wise average rewards for Human, Sonnet 3.7, Deepseek, and REMOR across 5.5k papers, with SEM values within 0.001.The caption does not provide the table's cell values.Source: Table 4 · p. 16
- Table 5 presents a performance comparison with previous systems over papers from ACL 2017 and NeurIPS 2019.The supplied caption does not report the comparison results.Source: Table 5 · p. 17
- REMOR-U reviews mostly have higher uniform reward than average human reviews, while only a very small number of human reviews reach uniform reward 4.Figure 6 shows the distribution of uniform reward for human and REMOR-U reviews.Source: Figure 6 · p. 17
G Sample review comparison between humans and REMOR
The appendix presents sampled reviews from cases with high contrast between human reviewers and REMOR, and describes a qualitative comparison based on aspect reward and METEOR score.
G Sample review comparison between humans and REMOR
- Table 6 compares reviews with high aspect reward but low METEOR against reviews with low aspect reward but high METEOR.The appendix says the samples come from papers with high contrast between human reviewers and REMOR.Source: G Sample review comparison between… · p. 17, Table 6 · p. 18