Source-linked AI summary
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Sihong Wu, Yiling Ma, Yilun Zhao, Tiansheng Hu, Owen Jiang, Manasi Patwardhan, Arman Cohan
TL;DR
LLM-generated reviews often lack specific, actionable guidance. RbtAct uses rebuttal-supervised, perspective-conditioned segment generation with RMR-75K and preference optimization, achieving higher actionability and specificity than strong baselines while maintaining grounding and relevance.
Problem
Many LLM-generated reviews are superficial, generic, and insufficiently actionable, leaving authors without concrete implementable guidance.
Method
RbtAct trains a perspective-conditioned segment-level feedback generator using RMR-75K and supervised fine-tuning followed by rebuttal-derived preference optimization.
Results
3.46 out of 5 in human evaluation and 3.38 out of 5 in LLM-as-a-judge evaluation were achieved for actionability, while grounding and relevance were maintained.
Takeaways & Limitations
Rebuttal signals provide a practical form of human feedback for producing targeted, implementable review guidance.
Takeaways & Limitations
Rebuttals capture short-horizon author uptake and the dataset mainly covers computer-science venues with public OpenReview rebuttals, limiting broader generalization.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used across the scientific workflow, including to draft peer-review reports. However, many AI-generated reviews are superficial and insufficiently actionable, leaving authors without concrete, implementable guidance and motivating the gap this work addresses. We propose RbtAct, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. Rebuttals show which reviewer comments led to concrete revisions or specific plans, and which were only defended. Building on this insight, we leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability. To support this objective, we propose a new task called perspective-conditioned segment-level review feedback generation, in which the model is required to produce a single focused comment based on the complete paper and a specified perspective such as experiments and writing. We also build a large dataset named RMR-75K that maps review segments to the rebuttal segments that address them, with perspective labels and impact categories that order author uptake. We then train the Llama-3.1-8B-Instruct model with supervised fine-tuning on review segments followed by preference optimization using rebuttal derived pairs. Experiments with human experts and LLM-as-a-judge show consistent gains in actionability and specificity over strong baselines while maintaining grounding and relevance.
1 Introduction
RbtAct addresses the limited actionability of many LLM-generated reviews by using rebuttals as implicit supervision and framing feedback generation around focused, perspective-specific review segments. The resulting dataset and training pipeline improve actionability and specificity while preserving grounding and relevance.
- LLM-generated reviews can be fluent but often miss specific issues, provide shallow analysis, and use generic phrasing.
- Perspective-conditioned segment-level generation produces one focused comment from the full paper and a target perspective, enabling more specific supervision.
- The pipeline fine-tunes Llama-3.1-8B-Instruct before applying rebuttal-derived preference optimization.
- 3.46 out of 5 in human evaluation and 3.38 out of 5 in LLM-as-a-judge evaluation were achieved for actionability, with groundedness and relevance maintained.
- RbtAct uses author rebuttals as implicit supervision to optimize review generation for actionability through preference optimization.
- RMR-75K contains 75,542 examples pairing review segments with perspectives, author responses, and annotated impact categories indicating actionability.
2 Related Work
Prior work provides datasets and methods for analyzing reviews and generating review text, but RbtAct builds on these lines with a more targeted feedback-generation setting.
- Peer review datasets have progressed from manuscript-review corpora to larger resources supporting reviewer-model fine-tuning.
- Review-generation research has moved from prompting full reviews toward fine-tuning, structured pipelines, and multi-agent approaches aimed at improving specificity and comment quality.
3 Review and Rebuttal Mapping
RMR-75K is constructed by decomposing reviews into actionable weakness and question segments and mapping them to the rebuttal spans that address them. The pipeline applies heuristic and semantic alignment followed by filters and human verification to improve supervision quality.
- Dataset construction: RMR-75K maps each reviewer key point to the specific rebuttal portion addressing it.
- Notation: Reviews are represented as sequences of weakness or question segments, while rebuttals are split into candidate response spans.
- Review–rebuttal mapping: The mapping enforces confidence-based one-to-one pairing, with each rebuttal span appearing in at most one pair.
- Review segmentation: The dataset targets actionable content by extracting weaknesses and questions, using existing itemization or prompted segmentation when needed.
- Review–rebuttal mapping: Alignment combines explicit-anchor heuristics with LLM-based span-level semantic linking and greedy one-to-one matching by confidence.
- Cleaning and quality control: Quality control removes unstable segmentations, unmatched or low-confidence pairs, restated questions, and non-substantive review segments before targeted human checks.
4 Methodology
RbtAct defines perspective-conditioned generation of focused review segments and builds rebuttal-supervised datasets that encode author uptake as preferences. It trains a policy with DPO to favor comments associated with stronger revision actions while preserving perspective control.
- Task Definition: Given a full paper and target perspective, the model generates one focused review segment addressing a weakness or question.
- Task Definition: The seven review perspectives cover experiments, evaluation, reproducibility, novelty, theory, writing, and presentation.Experiments concerns setup, baselines, ablations, and datasets; evaluation concerns metrics, analysis, and claims versus results.
- Dataset Construction: Rebuttal segments receive five impact categories ordered from concrete revision performed to deflection or reframing without change.The categories are CRP, SRP, VCR, DWC, and DRF.
- Dataset Construction: The SFT corpus contains 13,300 paper-perspective and gold-review pairs spanning 4,637 papers, with 1,900 instances per perspective.
- Dataset Construction: Preference triples pair review segments from the same paper and perspective, ranking them by rebuttal impact as CRP > SRP > VCR > DWC > DRF.Pairs are stratified by impact gap to expose the model to different preference margins.
- Policy Optimization: DPO uses a frozen SFT reference policy and increases likelihood for comments linked to higher-impact actions while decreasing likelihood for defended or deflected comments.A small supervised-loss mixture is optionally used to prevent perspective-control drift in long contexts, with λ set to 0.1 in practice.
5 Experiments
The experiments compare RbtAct with fine-tuned, prompted, and task-adapted review generators using human and LLM-based evaluations. RbtAct leads on actionability and specificity while maintaining groundedness and relevance, including against larger models.
- Evaluation Protocol: The evaluation compares RbtAct with an SFT-only model, prompted LLMs, and task-adapted review-generation methods under identical inputs, prompts, and decoding.Prompted models include GPT-5-chat, DeepSeek-V3.2, Llama-3.1-70B, and Qwen-3-32B.
- Evaluation Protocol: The test set contains 700 ICLR 2025 papers, stratified across seven perspectives with 100 papers per perspective and one human review segment per paper.
- Evaluation Protocol: Human evaluation uses three experienced annotators who rate anonymized outputs on actionability, specificity, groundedness, relevance, and helpfulness using a 1–5 rubric.
- Results: RBTACT attains the highest actionability in human and LLM-as-a-judge evaluations while remaining competitive on other quality dimensions.The judge also ranks RBTACT highest on specificity, while it stays close to strong LLM baselines on other dimensions.
- Results: RBTACT achieves the highest average pairwise actionability win rate and leads in most perspectives, followed by GPT-5-chat.
- Results: RBTACT’s gains concentrate on actionability and specificity, with parity on groundedness, relevance, and helpfulness, while remaining competitive with larger and proprietary models.Additional analyses report consistency across papers of different strengths and issue severities.
6 Conclusion
RBTACT places rebuttals at the center of actionable review feedback generation. Its rebuttal-informed pipeline improves actionability and specificity while maintaining grounding and relevance under human and LLM-as-a-judge evaluation.
- RBTACT uses rebuttals as implicit supervision for perspective-conditioned segment-level review feedback generation.The task maps each review segment to the rebuttal span addressing it and conditions generation on a specified perspective.
- RMR-75K provides review-to-rebuttal mappings with perspective labels and impact categories that capture actionability.The dataset connects each review segment with its addressing rebuttal span and records impact information.
- Supervised fine-tuning followed by preference optimization on impact-ordered pairs yields consistent gains in actionability and specificity.The pipeline also maintains grounding and relevance against strong baselines in both evaluation protocols.
- Rebuttal signals provide a practical form of human feedback for producing targeted, implementable review guidance.The paper releases data and code to support further research on rebuttal-driven learning and actionability evaluation.
Limitations
The approach treats rebuttals as informative but imperfect supervision. Its evidence is concentrated in public-rebuttal computer-science venues, and generated suggestions are not rigorously verified against research artifacts.
- Rebuttals reflect short-horizon author uptake rather than long-term implementation and may contain strategic promises or deferrals.Thus, rebuttal-derived actionability does not establish that suggested changes were ultimately implemented.
- The dataset mainly covers computer-science communities using OpenReview, leaving generalization to journals, non-English venues, and other fields uncertain.The limitation concerns the source venues and the availability of public rebuttals.
- The model can generate precise but infeasible suggestions because the setup lacks rigorous verification against manuscripts, code, or data artifacts.
C Additional Training Details
The appendix documents training, segmentation, baseline adaptation, and label configurations for the review-segment generation pipeline. It includes resource-oriented settings for SFT and DPO and deterministic output normalization for baselines.
- Segmentation: Review segmentation prompts decompose the Weaknesses & Questions section into independent, specific points while preserving meaning and numbering.The prompt supports numbered lists, letter prefixes, bullets, and section headers.
- Training Configuration: Training uses LLaMA-Factory, LoRA adapters, bf16 compute, NVIDIA H200 GPUs, and the Llama-3.1-8B-Instruct base model.Adapters target attention and MLP projections, with per-device batch size 1 and gradient accumulation.
- Training Configuration: DPO combines FlashAttention-2, DeepSpeed ZeRO-2 with gradient checkpointing, and a 4-bit frozen reference model to support a 32k-token context.These optimizations reduce activation and parameter memory during preference optimization.
- Training Configuration: SFT runs for 3 epochs and DPO for 2 epochs, using cosine schedules with learning rates 1.0×10−4 and 1.0×10−5, respectively.Reported H200 wall-clock times are approximately 120 hours for SFT and 203 hours for DPO.
- Baseline Adaptation: Baselines receive the same paper text and requested perspective and are deterministically reduced to one review segment when they produce longer outputs.MARG, LimGen, and DeepReviewer-14B are adapted to the single-segment format through perspective conditioning or fixed extraction rules.
D.3 Human Expert Evaluation Protocol
Human experts compare anonymized review candidates pairwise and rate each candidate from 1 to 5 on five dimensions using only evidence in the paper. The protocol defines rubrics for actionability, specificity, groundedness, relevance, and helpfulness.
- Protocol: Each comparison collects a pairwise preference and per-candidate 1–5 ratings on five review-quality dimensions.The dimensions are Actionability, Specificity, Groundedness, Relevance, and Helpfulness.
- Protocol: The rating anchors map 1 to Very poor, 2 to Poor, 3 to Fair, 4 to Good, and 5 to Excellent.
- Rubric Dimensions: Actionability measures clear next steps with parameters or acceptance criteria.
- Rubric Dimensions: Specificity measures whether feedback pinpoints exact sections, figures, metrics, or settings.
- Rubric Dimensions: Groundedness measures support from the paper through explicit references or numbers.
- Rubric Dimensions: Relevance measures alignment with the target perspective and main contributions, while helpfulness measures constructive guidance that improves the paper.
- LLM-as-a-Judge: LLM-as-a-judge results are visualized across seven perspectives as heatmaps whose cells show the row model’s win rate over the column model.
D.5 Automatic Evaluation
Automatic evaluation finds competitive overlap metrics for RBTACT, with rebuttal optimization making only small surface-level changes relative to SFT. Additional analyses examine issue severity and paper strength.
- RBTACT achieves the best ROUGE-Lsum (12.64) and METEOR (11.65), while RBTACT-SFT attains the highest BLEU@4 (14.93).
- GPT-5-chat is strongest on chrF, while RBTACT and RBTACT-SFT obtain close scores of 18.51 and 18.57.
- Rebuttal optimization slightly improves ROUGE-Lsum and METEOR over SFT but does not drastically change surface overlap.
- Major issues are defended more often than minor issues, with DWC rates of 45.4% versus 20.9%.
- Specific revisions account for 50.7% of major-issue mappings and 71.6% of minor-issue mappings.
- Weaker papers benefit more from rebuttal-anchored supervision, whereas stronger papers already elicit reasonably actionable feedback from strong baselines.
F.2 Retrieval Baseline Analysis
A perspective-matched retrieval baseline provides useful topical priors but performs worse than RBTACT across all evaluated dimensions, especially those requiring paper-specific adaptation.
- The retrieval baseline searches perspective-matched training review segments, excludes same-paper segments, and returns the nearest neighbor for the test paper.
- Retrieval is competitive on relevance and specificity but underperforms RBTACT on all five dimensions, especially Actionability, Groundedness, and Helpfulness.
- Nearest-neighbor reuse can produce plausible comments but often fails to adapt them to the paper’s method, evidence, and experimental setting.
F.3 Ordinal-Aware Analysis of Human Ratings
Ordinal-aware analyses confirm the main human-evaluation conclusions without treating Likert ratings as interval-scale measurements, while case studies illustrate actionable feedback across perspectives.
- Ordinal-aware summaries report medians, interquartile ranges, threshold rates, and ridit scores alongside descriptive means for Actionability and Helpfulness.
- Paired Wilcoxon signed-rank tests compare RBTACT with each baseline on matched evaluation instances and report rank-biserial correlations favoring RBTACT when positive.
- RBTACT has the strongest median-based Actionability and positive paired effects against most baselines, while its Helpfulness difference from RBTACT-SFT is small and nonsignificant.
- GPT-5-chat remains strongest on Helpfulness, reflected in its higher median, threshold rate, and ridit score.
- The ordinal-aware findings suggest that the main conclusions do not depend on treating Likert ratings as interval-scale measurements.
- A case study contrasts RBTACT feedback with baseline feedback across Experiment, Presentation, and Evaluation perspectives.