PaperlayerView exact revision on arXiv

Source-linked AI summary

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

Sihong Wu, Yiling Ma, Yilun Zhao, Tiansheng Hu, Owen Jiang, Manasi Patwardhan, Arman Cohan

arXiv:2603.09723v2cs.CLcs.AI

TL;DR

LLM reviews can be generic and insufficiently actionable. RBTACT uses rebuttals to optimize perspective-conditioned review feedback, with evaluations showing gains in actionability and specificity while maintaining grounding and relevance.

  • Problem

    LLM-generated reviews can miss specific issues and offer generic feedback rather than reliably actionable guidance.

    Source: §1 · p. 1

  • Method

    RBTACT uses rebuttal-derived preferences to optimize perspective-conditioned review feedback for actionability.

    Source: §1 · pp. 1–2, §6 · p. 8

  • Results

    3.46/5 was RBTACT's human actionability score; human and LLM-judge evaluations found improved actionability and specificity over strong baselines while maintaining groundedness and relevance.

    Source: §1 · p. 2, §5.6 · p. 8

  • Takeaways & Limitations

    Rebuttal signals are a practical form of human feedback for producing targeted, implementable review guidance.

    Source: §6 · p. 9

    Because the dataset focuses on venues with public rebuttals, mainly OpenReview computer-science communities, generalization to journals, non-English venues, and other fields remains uncertain.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used across the scientific workflow, including to draft peer-review reports. However, many AI-generated reviews are superficial and insufficiently actionable, leaving authors without concrete, implementable guidance and motivating the gap this work addresses. We propose RbtAct, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. Rebuttals show which reviewer comments led to concrete revisions or specific plans, and which were only defended. Building on this insight, we leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability. To support this objective, we propose a new task called perspective-conditioned segment-level review feedback generation, in which the model is required to produce a single focused comment based on the complete paper and a specified perspective such as experiments and writing. We also build a large dataset named RMR-75K that maps review segments to the rebuttal segments that address them, with perspective labels and impact categories that order author uptake. We then train the Llama-3.1-8B-Instruct model with supervised fine-tuning on review segments followed by preference optimization using rebuttal derived pairs. Experiments with human experts and LLM-as-a-judge show consistent gains in actionability and specificity over strong baselines while maintaining grounding and relevance.

1 Introduction

Although LLMs can draft fluent reviews, their feedback is often generic and insufficiently actionable. RBTACT uses rebuttal outcomes to supervise perspective-conditioned, segment-level feedback generation.

Introduction

  • RBTACT uses rebuttals as implicit preference supervision, favoring review comments that elicited author action to optimize feedback generation for actionability.This turns rebuttals from an object of analysis into a training signal.Source: §1 · pp. 1–2
  • The task generates one focused comment from the full paper conditioned on a perspective, such as experiments, addressing the fact that author responses target parts of reviews.Reviews are decomposed into key-point segments so each can be aligned with the rebuttal segment that addresses it.Source: §1 · pp. 1–2
  • The training pipeline fine-tunes Llama-3.1-8B-Instruct on perspective-conditioned review segments, then applies rebuttal-derived DPO to optimize for actionability.The authors report consistent gains over competitive baselines in human and LLM-as-a-judge evaluations.Source: §1 · p. 2
  • On ICLR 2025 test data, RBTACT scored 3.46/5 in human and 3.38/5 in LLM-as-a-judge actionability, leading comparisons while maintaining parity on groundedness and relevance.The comparisons included an SFT-only variant and larger prompted LLMs; gains held across multiple perspectives.Source: §1 · p. 2
  • RMR-75K contains 75,542 examples aligning review segments with rebuttal responses and annotating perspective and author-uptake impact.Impact categories distinguish completed revisions, planned revisions, and defenses; segments sharing a paper and perspective but differing in impact form preference pairs.Source: §1 · p. 2

2 Related Work

Related work spans peer-review datasets, review-generation systems, and actionability methods. RbtAct distinguishes itself by mapping review points to rebuttal segments and grounding actionability in authors’ responses and actions.

Peer Review Datasets

  • Earlier review–rebuttal links align sentences at small scale, whereas RbtAct maps review points to rebuttal segments and adds perspective and impact labels.DISAPERE [Kennard et al., 2022] annotates discourse in 506 review–rebuttal pairs, and JitsuPeer [Purkayastha et al., 2023] aligns sentences with rebuttal action types; RbtAct uses its mappings to supervise feedback generation.Source: §2 · p. 3

Review Generation

  • Prior actionable-review methods use reviewer-centric signals, while RbtAct grounds actionability in authors’ rebuttal responses and actions.ReAct labels actionability and intent, ARIES [D’Arcy et al., 2024a] links comments to edits, and MARG [D’Arcy et al., 2024b] generates actionable reviews; review-generation systems also progressed from prompting to fine-tuning, structured pipelines, and multi-agent approaches [Jin et al., 2024, Zhu et al., 2025b, Zou et al., 2026].Source: §2 · p. 3

3 Review and Rebuttal Mapping

Section 3 builds RMR-75K from ICLR 2024 papers, reviews, and rebuttals, aligning review segments with rebuttal spans and validating mapping quality.

3.2 Dataset Construction

  • RMR-75K maps review points to specific rebuttal segments and is about 150× larger than DISAPERE’s 506 pairs [Kennard et al., 2022].The mappings support perspective labels and impact signals for concrete revisions, specific plans, or defenses without changes.Source: §3 · p. 3, §3.2 · p. 3

3.1 Review–Rebuttal Source Data Collection

  • The corpus contains ICLR 2024 manuscripts, complete reviews and free-form comments, and author rebuttals concatenated chronologically when split across messages.The authors retrieved materials from OpenReview’s official API and converted manuscript PDFs to Markdown with MinerU [Wang et al., 2024].Source: §3.1 · p. 3

3.2 Dataset Construction

  • Review segments come from enumerated weakness and question items or GPT-5 atomic critiques, then heuristic anchors and an LLM matcher align them to rebuttal spans.The pipeline greedily enforces one-to-one alignment by confidence and discards ties.Source: §3.2 · p. 4

3.3 Cleaning and Quality Control

  • Quality control removes unstable, unmatched, low-confidence, nonresponsive, and nonsubstantive pairs before human verification.The confidence threshold is τ, and pairs whose rebuttal span merely restates a question are pruned.Source: §3.3 · p. 4
  • The automatic review–rebuttal mapping achieves span-level F1 = 0.91, with annotator agreement of κ = 0.80.Validation used 944 mapped segments from 60 sampled papers and counted a match as correct at token-level IoU ≥0.5.Source: Table 2 · p. 4, §3.3 · p. 4

4 Methodology

RbtAct uses rebuttal-linked author uptake as supervision for generating focused review segments conditioned on a requested perspective, first through supervised fine-tuning and then preference optimization.

4.1 Task Definition

  • RbtAct generates one focused review segment that raises a paper weakness or question, conditioned on the full paper and a specified perspective.Source: §4.1 · p. 4
  • Mapped review and rebuttal segments receive perspective and author-impact labels, with impact ordered from concrete revision through deflection.The seven perspective labels are Experiments, Writing, Presentation, Theory, Novelty, Reproducibility, and Evaluation; label accuracy was about 92% (κ = 0.81) for perspectives and 89% (κ = 0.79) for impact categories.Source: §4.1 · p. 5

4.2 Training Dataset Construction

  • The dataset reports 13,300 pairs across 4,637 papers, balanced at 1,900 instances per perspective; the preference set has 21,822 pairs from 4,825 papers, averaging 3,117 per perspective.Preference pairs use strictly ordered impact labels, are stratified by impact gap, and never mix papers or perspectives.Source: §4.2 · p. 5

4.3 Policy Optimization

  • DPO optimizes pairwise preferences against the frozen SFT reference, favoring comments linked to higher-impact author actions over defended or deflected comments.A small fraction of supervised fine-tuning on positive samples can be mixed in to limit drift in perspective control with long contexts; λ is set to 0.1.Source: §4.3 · p. 6

4.4 Training Details

  • Training uses Llama-3.1-8B-Instruct [Meta AI, 2024], with 3 supervised fine-tuning epochs followed by 2 DPO epochs.The reported runs use NVIDIA H200 GPUs with bf16 compute; SFT takes approximately 120 hours and DPO approximately 203 hours.Source: §4.4 · pp. 6–7

5 Experiments

Human and LLM evaluations show RBTACT leading on actionability, while automatic metrics show strong overlap performance. The experiments also assess agreement, baseline comparisons, and performance across perspectives and paper strengths.

5.1 Baselines

  • Under matched inputs, prompts, and decoding, RBTACT is compared with SFT-only, zero-shot LLMs, and adapted review-generation methods.The baselines include RBTACT-SFT; GPT-5-chat, DeepSeek-V3.2, Llama-3.1-70B [Meta AI, 2024], and Qwen-3-32B [Team, 2025]; adapted methods are MARG [D’Arcy et al., 2024b], LimGen [Faizullah et al., 2024], and DeepReviewer-14B [Zhu et al., 2025b].Source: §5.1 · p. 7

5.2 Evaluation Protocol

  • The evaluation uses 700 ICLR 2025 papers, with human ratings on 50 papers and GPT-5-chat judging on 105 papers.The dataset has 100 papers per perspective across seven perspectives; three qualified annotators rate outputs on five dimensions, while the judge evaluates 945 segments from nine models.Source: §5.2 · p. 7

5.3 Experiment Results

  • Human ratings put RBTACT highest on Actionability, and LLM pointwise ratings likewise rank it highest on Actionability and Specificity while competitive on other dimensions.Human–LLM model-level agreement on Actionability is Spearman’s ρ=0.94 and Kendall’s τb=0.87; item-level agreement is Spearman’s ρ=0.52.Source: Table 5 · p. 7, §5.3 · p. 8
  • RBTACT has the highest average pairwise Actionability win rate and leads in most perspectives, followed by GPT-5-chat.Table 6 reports model win percentages, and Figure 17 visualizes win rates across perspectives.Source: Table 6 · p. 8, §5.3 · p. 8

5.4 Automatic Evaluation

  • Across 700 instances, RBTACT achieves best ROUGE-Lsum and METEOR, while RBTACT-SFT attains highest BLEU@4; models are competitive or stronger on most overlap metrics.The evaluation uses BLEU@4, ROUGE-Lsum (F1), METEOR, and chrF.Source: §5.4 · p. 8

5.5 Case Study

  • Case studies across perspectives illustrate why RBTACT is more actionable than other baselines in most cases.The examples are reported in Appendix E.Source: §5.5 · p. 8

5.6 Summary

  • The authors report gains concentrated in actionability and specificity, parity on groundedness, relevance, and helpfulness, and competitiveness against much larger models.The 8B RBTACT model remains competitive with 32–70B and proprietary models on actionability and often specificity; appendix analyses report gains beyond minor issues and across papers of different strengths.Source: §5.6 · p. 8

6 Conclusion

RbtAct uses rebuttals as implicit supervision for perspective-conditioned, segment-level review feedback generation. Its training pipeline improves actionability and specificity while maintaining grounding and relevance, and the work releases the RMR-75K dataset and code.

6 Conclusion

  • RbtAct uses rebuttals as implicit supervision for perspective-conditioned, segment-level generation, mapping each review segment to the rebuttal span that addresses it.RMR-75K includes perspective labels and impact categories that capture actionability; training uses supervised fine-tuning followed by preference optimization on impact-ordered pairs.Source: §6 · p. 8
  • RbtAct gains in actionability and specificity over strong baselines while maintaining grounding and relevance in human and LLM-as-a-judge evaluations.The results indicate rebuttal signals can provide practical human feedback for generating targeted, implementable guidance.Source: §6 · pp. 8–9
  • The authors release RMR-75K data and code to support research on rebuttal-driven learning and evaluation of actionability.Source: §6 · p. 9

Limitations … B.4 Prompt for labeling impact categories

RbtAct’s supervision signal and evaluation scope have important limits, while the appendix documents its review–rebuttal mapping examples, labeling definitions, dataset distributions, and prompts. The supplied passages identify these materials but do not provide their detailed contents.

Limitations

  • Rebuttals provide informative but imperfect actionability supervision, reflecting short-term review uptake rather than long-term implementation and sometimes strategic promises or deferrals.The dataset centers on venues with public rebuttals, mainly OpenReview computer-science communities; generalization elsewhere is uncertain, and suggestions may be infeasible without rigorous manuscript, code, or data verification.Source: Limitations · p. 9

A.1 Example of Mapping

  • Figure 3 presents two sampled review–rebuttal mappings from the RMR-75K dataset.The passage does not describe the mappings’ contents.Source: §A.1 · p. 12

A.2 Prompts for Review Segment and Mapping

  • Figures 4 and 5 show the prompts for review segmentation and review-segment-to-rebuttal-segment mapping, respectively.The supplied passage identifies the prompts but does not state their wording.Source: §A.2 · p. 12

B Label Details

  • The paper discusses its labeling of perspectives and impact categories in the label-details section.The supplied passage does not specify the labeling procedure.Source: B Label Details · p. 12

B.1 Detailed Definition of Perspective labels and Impact Categories

  • Table 7 gives detailed definitions of the perspective labels and rebuttal impact categories.The supplied passage does not include the definitions themselves.Source: §B.1 · p. 12

B.2 Perspective and Impact Category Distribution

  • Table 8 presents the distribution of seven perspectives and impact categories in RMR-75K.The supplied passage provides no distribution values.Source: §B.2 · p. 12

B.3 Prompt for labeling perspectives

  • Figure 6 shows the prompt used to label the perspective of a review segment.The supplied passage does not state the prompt wording.Source: §B.3 · p. 12

B.4 Prompt for labeling impact categories

  • Figure 7 shows the prompt used to label the impact category of a rebuttal segment.The supplied passage does not state the prompt wording.Source: §B.4 · p. 12

C Additional Training Details … D.4 LLM-as-a-Judge Evaluation

The appendix specifies training and baseline adaptations, then describes human and LLM-judge evaluation procedures for review segments. Human raters assess candidate reviews using paper evidence and five dimensions, while LLM-judge results include pairwise win-rate heatmaps across seven perspectives.

C Additional Training Details

  • DPO combines attention, checkpointing, and reference-model quantization to support 32k-token paper context; SFT and DPO use separate training schedules.SFT runs for 3 epochs (4,989 steps) at 1.0×10−4 and takes approximately 120 hours; DPO initializes from the SFT checkpoint, runs for 2 epochs (2,728 steps) at 1.0×10−5, and takes approximately 203 hours. Training uses LoRA on Llama-3.1-8B-Instruct with bf16 compute on NVIDIA H200 GPUs.Source: C Additional Training Details · p. 13

D.1 Baseline Prompt; D.2 Baseline Adaptation Details

  • Baselines receive the same paper text and requested perspective and are adapted to output one segment, with deterministic extraction for longer outputs.MARG contributes its first synthesized bullet; LimGen contributes its first limitation item and paired suggestion when present; DeepReviewer-14B contributes the first paragraph under the perspective-matched subsection. The appendix also provides baseline-generation prompts.Source: §D.1 · p. 13, §D.2 · pp. 14–15

D.3 Human Expert Evaluation Protocol

  • The human-evaluation protocol assesses review segments and actionable rebuttal guidance using an interface and rubric.The appendix describes the evaluation interface and rubric used for these judgments.Source: §D.3 · p. 15

D.3.1 Interface

  • Raters make pairwise A/B/tie judgments and give each candidate 1–5 ratings on five dimensions, using evidence from the paper.They must avoid hallucinations and should not penalize missing information if it appears elsewhere in the paper; rating anchors range from 1 (Very poor) to 5 (Excellent).Source: §D.3.1 · p. 15

D.3.2 Dimension Rubrics (Concise)

  • The rubric defines actionability, specificity, groundedness, relevance, and helpfulness as distinct review-quality dimensions.Actionability concerns clear next steps with parameters or acceptance criteria; specificity pinpoints sections, figures, metrics, or settings; groundedness requires paper support; relevance aligns with the requested perspective and contributions; helpfulness means constructive guidance that improves the paper.Source: §D.3.2 · p. 15

D.4 LLM-as-a-Judge Evaluation

  • LLM-as-a-judge evaluation uses separate prompts for point-wise assessment and pairwise Actionability comparison, and visualizes results across seven perspectives as win-rate heatmaps.Each heatmap cell represents the row model’s win rate over the column model.Source: §D.4 · p. 16

D.5 Automatic Evaluation … F.1.2 Do gains correlate with paper strength?

RBTACT and RBTACT-SFT are competitive on automatic metrics, rebuttal-derived supervision covers major issues as well as minor ones, and weaker papers appear to benefit more from it. The analyses examine issue severity and actionability across paper-strength buckets.

D.5 Automatic Evaluation

  • Evaluation covers all 700 test instances using BLEU@4, ROUGE-Lsum (F1), METEOR, and chrF.The metrics are BLEU@4, ROUGE-Lsum, METEOR, and chrF [Popovi´c, 2015].Source: §D.5 · p. 16
  • RBTACT slightly improves over RBTACT-SFT on ROUGE-Lsum and METEOR, while their BLEU@4 scores are close, suggesting rebuttal optimization refines phrasing without drastically changing surface overlap.The comparison concerns automatic metric differences between RBTACT and its SFT variant.Source: §D.5 · p. 16
  • RBTACT leads ROUGE-Lsum (12.64) and METEOR (11.65), while RBTACT-SFT leads BLEU@4 (14.93); GPT-5-chat leads chrF, where the RBTACT variants score 18.51 vs. 18.57.Both RBTACT variants outperform large proprietary and open models on ROUGE-Lsum, METEOR, and BLEU@4.Source: Table 10 · p. 16, §D.5 · p. 16

E Case Study

  • Case studies across perspectives illustrate why RBTACT is more actionable than other baselines in most cases.The examples are shown in Figure 18.Source: E Case Study · p. 16

F.1 Severity and Paper-Strength Analyses.

  • The additional analyses test whether rebuttal-derived supervision over-emphasizes minor issues and whether actionability gains vary with paper strength.They examine issue severity using perspective labels and relate actionability improvements to OpenReview ratings.Source: §F.1 · p. 16

F.1.1 Are we biased toward minor issues? A severity proxy.

  • Major issues are defended more often than minor ones (DWC 45.4% vs. 20.9%), yet 50.7% of major-issue mappings correspond to specific revisions (CRP+SRP).Major issues comprise Experiments, Evaluation, Theory, Novelty, and Reproducibility; minor issues comprise Writing and Presentation. Preference pairs are within the same paper and perspective, and balanced across perspectives.Source: §F.1.1 · p. 16, Table 11 · p. 17

F.1.2 Do gains correlate with paper strength?

  • On 105 papers, actionability averages are compared across Weak, Medium, and Strong rating buckets; the trend suggests weaker papers benefit more from rebuttal-anchored supervision.Paper strength is the mean OpenReview reviewer rating, split into tertiles; stronger papers already elicit reasonably actionable feedback from strong baselines.Source: §F.1.2 · p. 17, Table 12 · p. 17

F.2 Retrieval Baseline Analysis

The retrieval baseline reuses a perspective-matched training review segment without rebuttals; it is competitive on relevance and specificity but trails RBTACT on all five judge dimensions, especially Actionability, Groundedness, and Helpfulness.

F.2 Retrieval Baseline Analysis

  • The baseline returns the nearest training review segment matching the target perspective, excludes segments from the test paper, and uses no rebuttals.It encodes the test paper as a query and uses the same single-segment output format as other systems.Source: §F.2 · p. 17
  • Retrieval trails RBTACT on all five judge dimensions, especially Actionability, Groundedness, and Helpfulness, because reused comments often fail to adapt to a paper’s method, evidence, and experimental setting.RBTACT instead learns paper-specific, more actionable feedback from rebuttal-derived supervision.Source: §F.2 · p. 17
  • Retrieval is competitive on relevance and specificity, suggesting perspective-matched review segments provide useful topical priors.Source: §F.2 · p. 17

F.3 Ordinal-Aware Analysis of Human Ratings

Ordinal-aware summaries and paired tests preserve the main findings while showing a distinction between systems’ actionability and helpfulness ratings.

F.3 Ordinal-Aware Analysis of Human Ratings

  • Ordinal summaries report medians, interquartile ranges, ratings at least 4, and pooled-frequency ridit scores alongside means for Actionability and Helpfulness.Ridit scores above 0.5 indicate ratings stochastically higher than the pooled average.Source: §F.3 · p. 17, Table 14 · p. 18
  • Paired Wilcoxon signed-rank tests compare RBTACT with each baseline on matched instances, reporting signed-rank statistics, two-sided p-values, and rank-biserial effect sizes.Positive effects favor RBTACT, and the tests avoid assuming equal intervals between Likert categories.Source: §F.3 · p. 17, Table 15 · p. 18
  • Ordinal summaries preserve the main pattern: RBTACT leads on actionability, while GPT-5-chat leads on helpfulness; RBTACT’s actionability gain over RBTACT-SFT is significant, but its helpfulness difference is not.RBTACT also has positive paired effects against most baselines.Source: §F.3 · p. 17, Table 14 · p. 18, Table 15 · p. 18

Related papers

Loading 2603.09723v2…