PaperlayerView exact revision on arXiv

Source-linked AI summary

ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

Yiling Ma, Yilun Zhao, Sihong Wu, Ziyu Chen, Manasi Patwardhan, Arman Cohan

arXiv:2609.09076v1cs.CL

TL;DR

Existing LLM review systems often identify weaknesses without explaining how to address them, motivating actionable feedback that separates diagnosis from revision guidance. ActReview uses rebuttal-guided training and rubric rewards to generate grounded revision plans, outperforming specialized review models on actionability and grounding while remaining competitive with strong prompt-based LLMs.

  • Problem

    Existing LLM review systems often provide descriptive feedback without concrete, grounded guidance for addressing identified weaknesses.

    Source: §1 · p. 2

  • Method

    ActReview separates diagnostic claims from revision suggestions and trains models with rebuttal-aligned data plus candidate-aware, weakness-specific rubric rewards.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    ActReview outperforms prior specialized review-generation models on actionability and grounding while remaining competitive with strong prompt-based LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ActReview produces more specific, constructive, and actionable revision-oriented feedback, with gains concentrated on revision usefulness rather than universal technical superiority.

    Source: §6.3 · p. 7

    ActReview-Bench excludes concerns such as fundamental novelty disputes, deep conceptual disagreements, and irreparable methodological flaws, and technical accuracy does not uniformly surpass strong proprietary LLMs.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As LLMs are increasingly used for pre-submission self-review, there is growing demand for feedback that not only identifies weaknesses but also guides authors toward concrete revisions. We study this as Actionable Peer-review Generation and decompose it into two subtasks: diagnostic claim generation and revision suggestion generation. We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans. Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback. From real review-rebuttal threads on OpenReview, we construct ActReview-40K by aligning reviewer weaknesses with author responses and grounding the resulting feedback in localized paper evidence. We post-train Qwen3-8B-Base with multi-task supervised fine-tuning followed by GRPO using candidate-aware, weakness-specific rubric rewards. We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness. Experiments show that ActReview outperforms prior specialized review-generation models on actionability and grounding while remaining competitive with strong prompt-based LLMs. Human evaluation confirms improved revision usefulness while revealing a remaining gap in technical accuracy, and additional analyses support generalization to held-out papers and robustness across independent judges.

1 Introduction

ActReview frames actionable peer-review generation as separate diagnosis and revision-guidance tasks, addressing the limited prescriptiveness and weak reference signals of prior systems. It introduces rebuttal-grounded data, benchmark evaluation, and rubric-guided post-training to improve revision usefulness while retaining technical-accuracy challenges.

Motivation

  • Existing LLM review systems mainly identify issues without specifying concrete remedies, while prior formulations and resources rarely separate diagnosis from grounded, implementable revision guidance.This leaves authors without clear information about what to revise, where to revise, or how to carry out the revision.Source: §1 · p. 2

Problem formulation and benchmark

  • ActReview-Bench evaluates both diagnostic quality and practical revision usefulness by using rebuttal-derived author actions as grounded reference signals.The benchmark is grounded in real review–rebuttal threads and distinguishes identifying deficiencies from explaining how to address them.Source: §1 · p. 2

Training data

  • ActReview-40K aligns atomic reviewer weaknesses with rebuttal spans and localized paper evidence to create structured initial-review-style feedback.Rebuttals provide latent supervision for inferring concrete revision paths, while retrieval focuses training on relevant evidence rather than full-paper context.Source: §1 · p. 2

Post-training framework

  • ActReview combines multi-task supervised fine-tuning with GRPO and candidate-aware, weakness-specific rubric rewards for diagnostic precision, grounding, and revision usefulness.The model is based on Qwen3-8B-Base [Yang et al., 2025], and the rubrics discourage fluent but generic comments by enforcing instance-level criteria.Source: §1 · p. 2

Evaluation

  • Experiments show improved performance over prior specialized review-generation models [Zhu et al., 2025a, Idahl and Ahmadi, 2025, Wu et al., 2026b], competitiveness with prompt-based LLMs, and stronger human-rated revision usefulness.Technical accuracy remains a challenge, while controlled analyses support the dual-task formulation, generalization to held-out 2025–2026 papers, and robustness across human and independent LLM judges.Source: §1 · p. 2

2 Related Work

Prior work applies LLMs to peer-review assistance and develops post-training methods for non-verifiable tasks. ActReview differs by separating diagnosis from revision suggestions and using weakness-specific, rebuttal-grounded rubric supervision.

Peer-review Generation and Assistance

Post-training for Non-verifiable Tasks

  • Post-training for non-verifiable tasks increasingly uses evaluators, self-rewarding, reference-based judges, structured rubrics, and ranked optimization because scalar preferences are costly or coarse [Zheng et al., 2023, Yuan et al., 2024, Liu et al., 2026, Shi et al., 2026, Zhao et al., 2025b, Gunjal et al., 2025, Viswanathan et al., 2026, Choi et al., 2026].These approaches target open-ended quality dimensions that exact-match rewards cannot measure reliably.Source: §2 · p. 3
  • ActReview builds on rubric-based rewards by defining weakness-specific criteria for individual paper–weakness instances from reference outputs and model candidate failures.The design makes reward criteria more targeted than generic scalar feedback.Source: §2 · p. 3

3 Actionable Peer-Review Generation

Section 3 formulates actionable peer-review generation as separate, weakness-conditioned tasks for diagnosing paper-specific deficiencies and producing grounded revision plans. It also introduces ActReview-Bench, constructed from review–rebuttal threads and filtered through human annotation.

3.1 Problem Formulation

  • The framework separates diagnostic claim generation from actionable suggestion generation, enabling independent training and evaluation while supporting an end-to-end review-to-revision workflow.Claims identify concrete deficiencies, whereas suggestions specify what and where to revise, how to implement the change, and its expected outcome.Source: §3.1 · p. 3
  • Weakness-conditioned inference queries one taxonomy label at a time, generates claims only when supported by paper evidence, and otherwise abstains rather than hallucinating.Full-paper auditing can run the model independently for each label; the main datasets lack explicit negative examples, while a separate held-out set evaluates zero-shot abstention.Source: §3.1 · p. 3

3.2 ActReview-Bench Construction

  • ActReview-Bench uses rebuttal-derived author actions as grounded reference signals and latent supervision, recognizing that rebuttals represent one plausible revision path rather than a unique gold answer.The benchmark addresses the limited availability of structured resources connecting reviewer weaknesses to concrete revision guidance.Source: §3.2 · p. 3
  • Human annotation retains 1,000 high-quality benchmark instances from 2,000 candidates, with adjudicated filtering showing strong agreement and high filtering quality.Two annotators assessed whether rebuttals substantively addressed concerns, contained concrete revision actions, and remained supported when follow-up discussion existed; disagreements were adjudicated.Source: §3.2 · p. 4

4 ActReview-40K

ActReview-40K converts real OpenReview review–rebuttal threads into approximately 40K weakness–response instances with structured diagnoses, revision suggestions, and localized paper evidence. Its pipeline combines taxonomy construction, rebuttal alignment, artifact filtering, and retrieval-based grounding.

4 ActReview-40K

  • ActReview-40K uses a four-stage pipeline to extract reviewer weaknesses, align rebuttal responses, generate feedback, and retrieve localized paper evidence.Each instance begins from a real review–rebuttal thread and becomes structured initial-review-style feedback.Source: §4 · p. 4

4.1 Data Sources and Weakness Taxonomy

  • The dataset contains approximately 40K weakness–response instances from 15,819 papers across ICLR, NeurIPS, and EMNLP, organized under a two-level weakness taxonomy.The taxonomy was induced through LLM-assisted category discovery and human refinement.Source: §4.1 · p. 4

4.2 Weakness Extraction and Rebuttal Alignment

  • Atomic weakness decomposition enables fine-grained alignment to substantively relevant rebuttal spans using structural, lexical, and LLM-based semantic cues.Human validation on a stratified sample confirms strong alignment quality.Source: §4.2 · pp. 4–5

4.3 Rebuttal-Guided Feedback Enhancement

  • Rebuttal spans provide latent supervision for diagnostic claims and concrete revision suggestions, while filtering removes leakage, retrospective rewrites, copying, and underspecified advice.A 500-instance human audit found κ = 0.78 agreement and 84.8% overall acceptability.Source: §4.3 · p. 5

4.4 Localized Evidence Retrieval

  • Localized retrieval selects paragraph-level paper chunks for diagnosis and revision support, using rebuttal-derived signals only offline and excluding rebuttal text from selected evidence.At evaluation time, all systems receive the same full-paper context, metadata, and target weakness label.Source: §4.4 · p. 5

5 Post-Training Framework

ActReview uses joint multi-task supervised fine-tuning followed by GRPO with frozen, candidate-aware weakness-specific rubrics. This design separates schema and reviewer-style learning from targeted optimization for diagnostic precision, grounding, and revision usefulness.

5 Post-Training Framework

  • Training splits enhanced instances into 90% SFT and 10% RL, while reference claims and suggestions construct offline rubrics rather than serving as direct RL supervision.RL training and rubric-construction instances are disjoint from ActReview-Bench evaluation instances.Source: §5 · p. 5

5.1 Multi-Task Supervised Fine-Tuning

  • ActReview jointly trains diagnostic claim generation and structured revision suggestions in a unified multi-task SFT model.Task 1 conditions on weakness labels and paper chunks, while Task 2 conditions on diagnostic claims and related context; SFT teaches the output schema but may produce generic feedback.Source: §5.1 · p. 5

5.2 Candidate-Aware Rubric Construction

  • Candidate-aware weakness-specific rubrics combine diverse SFT outputs with human-written and GPT-5.4-generated references to encode revision targets and candidate failure modes.GPT-5.4 synthesizes frozen rubrics containing hard constraints and weighted soft requirements tailored to each RL instance.Source: §5.2 · p. 5

5.3 Rubric-Based Reinforcement Learning

  • GRPO optimizes only the SFT-initialized ActReview policy using K=8 sampled responses scored by frozen rubrics, with zero reward for hard-constraint violations.Valid outputs receive weighted soft-requirement scores from deterministic format checks and GPT-5.4 semantic evaluation, while updates use group-relative rewards and KL regularization toward the SFT model.Source: §5.3 · pp. 5–6

6 Experiments

Across automatic, pairwise, human, and robustness evaluations, ActReview-RL improves revision-oriented feedback and grounding while remaining competitive with frontier prompt-based LLMs. Ablations attribute gains to rebuttal-enhanced data, two-task formulation, localized context, and candidate-aware rewards, with technical accuracy remaining a limitation.

6.1 Baselines

  • ActReview compares prompt-based GPT-5.1 and Gemini-3.1-Pro-Preview variants with specialized systems and supervised-only ACTREVIEW-SFT under identical paper context and weakness labels.Specialized baselines are evaluated out of the box without ActReview-Bench-specific fine-tuning, so gaps partly reflect transfer differences; rebuttals are never evaluation inputs.Source: §6.1 · p. 6

6.2 Evaluation Protocol

  • GPT-5.4 pairwise judging and human annotation directly compare outputs, with 93% categorical agreement among annotators and reference-based metrics used only as complementary diagnostics.The evaluation covers Technical Accuracy, Depth & Constructiveness, Overall quality, and task-specific Specificity & Grounding or Actionability.Source: §6.2 · pp. 6–7, Table 3 · p. 7

6.3 Main Results

  • ActReview-RL improves diagnostic specificity, constructive depth, revision actionability, grounding, and most reference-based metrics, while producing 1.24 claims per instance versus a 1.37-reference average.For Task 2, it achieves the strongest trained-model results on ROUGE-L, BLEU, suggestion similarity, evidence specificity, and rubric score; frontier LLMs remain most competitive on Technical Accuracy.Source: §6.3 · p. 7, Table 3 · p. 7

6.4 Ablation Study

  • Rebuttal-enhanced data provides particularly large Task 2 gains, localized context remains competitive with full-paper training, and full-paper context helps concerns requiring cross-section reasoning.Candidate-aware rubric rewards outperform direct judge rewards and fixed task-level rubrics; two-task SFT improves Task 2 Overall by 0.42 points and Claim–Suggestion Alignment by 0.35 points.Source: Table 4 · p. 7, §6.4 · p. 8, Table 5 · p. 8

6.5 Generalization and Reliability

  • On 175 independently annotated held-out-paper instances, ACTREVIEW-RL improves resolution sufficiency and reviewer usefulness while remaining comparable to GPT-5.1schema in weakness validity and technical correctness.It has the lowest severe-error rate at 6.29%, a 28.00% overall technical-error rate, and consistent independent Claude and human trends.Source: §6.5 · p. 8
  • ACTREVIEW-RL scores 1.49 versus 1.17 for localized training on cross-section reasoning, extends or rejects incomplete-rebuttal resolutions 66% of the time, and correctly abstains on 78.3% of unsupported pairs.It maintains a 95.4% Answer Rate on supported instances, substantially outperforming the strongest evaluated baseline.Source: §6.5 · p. 8

7 Conclusion

ActReview frames actionable peer-review generation as a Review2Revise process that separates diagnosis from revision guidance and uses rebuttals as latent supervision. Results show improved actionability and grounding, with independent contributions from rebuttal-guided enhancement and candidate-aware rubric rewards.

7 Conclusion

  • ActReview constructs rebuttal-guided, revision-oriented feedback by separating diagnostic claim generation from revision suggestion generation and applying candidate-aware weakness-specific rubric rewards.ActReview-40K uses real-world review–rebuttal threads to rewrite reviewer weaknesses into structured revision guidance.Source: §7 · p. 8
  • ActReview consistently outperforms prior specialized review-generation systems on actionability and grounding while remaining competitive with strong prompt-based LLMs despite an order-of-magnitude parameter gap.The conclusion attributes this performance to the combined Review2Revise formulation and rebuttal-guided training framework.Source: §7 · p. 8
  • Ablations show that rebuttal-guided enhancement and candidate-aware rubric rewards each independently improve output quality.The findings support review–rebuttal interactions as an underexploited supervision source.Source: §7 · p. 8

Limitations

ActReview-Bench evaluates rebuttal-resolvable weaknesses, while technical accuracy and judge-model independence remain important limitations despite gains in grounding and actionability.

Benchmark scope

  • ActReview-Bench excludes fundamental novelty disputes, deep conceptual disagreements, and irreparable methodological flaws because it focuses on weaknesses linked to concrete author-side revision actions.These excluded concerns may require broader literature comparison or interactive research guidance.Source: Limitations · p. 9

Technical accuracy

  • ActReview improves grounding and actionability but does not uniformly outperform strong proprietary LLMs on technical accuracy.The framework is therefore most effective at making feedback specific and useful for revision, while deeper technical judgment remains future work.Source: Limitations · p. 9

Evaluation dependence

  • GPT-5.4 use across data enhancement, rubric construction, semantic reward scoring, and pairwise evaluation creates potential judge-model coupling.Offline rubric construction, disjoint instances, matched-context comparisons, and human evaluation on 200 instances mitigate this risk, but broader independent judging and expert evaluation are still needed.Source: Limitations · p. 9

A Benchmark Annotation Details · B Data Collection and Licensing · B.1 Source Distribution

ActReview-Bench is independently annotated to retain review–rebuttal instances with specific, reliable revision signals, while ActReview-40K is built from publicly available OpenReview discussions and paper metadata. The benchmark evaluates actionable feedback within the narrower scope of reviewer concerns linked to plausible author-side revision paths.

A Benchmark Annotation Details

  • Rebuttal-derived actions serve as grounded reference signals, not unique gold answers, because multiple valid revision paths can address the same reviewer concern.This design evaluates both diagnostic relevance and practical usefulness for revision while recognizing that author responses provide realistic examples of possible actions.Source: A Benchmark Annotation Details · p. 11
  • Independent annotation uses two annotators who judge three criteria, make a keep/filter decision, and mark the rebuttal span containing the author-side revision action.Criterion judgments use a 0–2 scale, with N/A permitted for unavailable follow-up discussion and intermediate labels identifying borderline cases for adjudication.Source: A Benchmark Annotation Details · p. 11
  • ActReview-Bench retains instances only when rebuttals substantively address reviewer concerns, propose concrete revision actions, and lack evidence that the concern remains unresolved.Q1 and Q2 must receive clear-positive judgments, while Q3 must be clear-positive or unavailable; borderline cases are filtered by default and disagreements undergo third-party adjudication.Source: A Benchmark Annotation Details · p. 11
  • Retained cases provide specific and reliable revision signals rather than merely fluent or persuasive author responses, whereas filtered cases may be unresolved, nonspecific, or only partially relevant.The representative retained case addresses the concern, proposes robustness experiments and a capacity-matched ablation, and receives reviewer support; the filtered case fails these standards.Source: A Benchmark Annotation Details · pp. 11–12
  • The benchmark covers rebuttal-resolvable weaknesses linked to plausible revision paths, excluding unresolved novelty disputes, fundamental conceptual disagreements, and nonconvergent reviewer–author exchanges.Its scope supports evaluation of revision-oriented feedback but does not represent the full space of peer-review concerns.Source: A Benchmark Annotation Details · p. 12

B Data Collection and Licensing

  • ACTREVIEW-40K is constructed from publicly available OpenReview review–rebuttal discussions and paper metadata to provide structured supervision for diagnostic claims and actionable suggestions.The appendix documents the dataset’s source distribution, licensing, and example format.Source: B Data Collection and Licensing · p. 12

B.1 Source Distribution

  • Table 8 summarizes collected-paper distributions across venues and years, distinguishing all collected papers from those retained after filtering for actionable review–rebuttal interactions.Raw denotes all collected papers, Act. denotes papers retained after actionable-interaction filtering, and Ret. denotes the corresponding retention rate.Source: §B.1 · pp. 12–13, Table 8 · p. 13

B.2 Data Licensing and Attribution … C Rebuttal-Guided Data Construction

ActReview-40K is built from publicly available review–rebuttal and paper sources under licensing, attribution, privacy, and reproducibility constraints. The construction transforms reviewer weaknesses and author rebuttals into structured, actionable supervision with localized evidence.

B.2 Data Licensing and Attribution

  • ActReview-40K uses publicly available OpenReview review–rebuttal materials and papers from repositories such as arXiv, whose licenses vary by paper.OpenReview content is distributed under CC BY 4.0.Source: §B.2 · p. 13
  • The authors do not claim ownership of source content and preserve attribution while releasing derived fields and metadata subject to licensing constraints.Paper content is used for structured extraction, localized retrieval, and context construction.Source: §B.2 · p. 13

B.4 Real vs. Enhanced Data Example

  • The enhanced training instance specifies what to improve, where to revise it, and how to implement the change.The provided figure describes enhanced data as decomposing a real review–rebuttal example into a claim–evidence pair and further revision specifications.Source: Figure 4 · p. 14

B.3 Privacy and anonymization.

  • The released review–rebuttal data omits non-public identities and private metadata, and is intended for research use without deanonymization beyond public source information.Personally identifying information beyond what is already publicly available is removed or avoided before release.Source: §B.3 · p. 14

B.4 Real vs. Enhanced Data Example

  • A raw reviewer weakness and author rebuttal are rewritten as initial-review-style feedback containing a diagnostic claim, evidence, and actionable revision guidance.This conversion supports revision-oriented learning while avoiding direct imitation of rebuttal text.Source: §B.4 · p. 14

C Rebuttal-Guided Data Construction

  • ActReview-40K construction covers weakness taxonomy, weakness extraction and alignment prompts, alignment validation, rebuttal-guided enhancement, quality control, and localized evidence retrieval.These components are presented as the appendix’s additional construction details.Source: C Rebuttal-Guided Data Construction · p. 15

C.1 Data-Driven Weakness Taxonomy … C.7 Implementation Details and Compute Budget

The paper builds ActReview-40K through structured weakness taxonomy, rebuttal alignment, enhanced reviewer-style targets, and localized evidence retrieval. Audits support taxonomy consistency, target quality, and retrieval usefulness, while implementation uses joint SFT and GRPO without rebuttal text at inference.

C.1 Data-Driven Weakness Taxonomy

  • The taxonomy contains 7 Level-1 and 17 Level-2 categories, induced through LLM-assisted discovery and human consolidation, with substantial agreement at both levels.On 100 independently labeled weakness statements, Fleiss’ κ is 0.79 at Level 2 and 0.88 at Level 1.Source: §C.1 · p. 15

C.2 Weakness Extraction and Alignment

  • Reviews are decomposed into atomic weaknesses before each unit is aligned with an author rebuttal span, enabling fine-grained supervision from responses.This decomposition reduces entanglement among concerns and supports one-to-one weakness–rebuttal alignment.Source: §C.2 · p. 18

C.3 Alignment Quality Audit

  • A human audit of 794 filtered mappings finds high span-level accuracy, while residual errors mainly involve multi-concern paragraphs, implicit responses, and generic acknowledgments.The audit compares automatic spans with adjudicated gold mappings using token-level IoU and distinguishes substantive responses from NO RESPONSE cases.Source: §C.3 · p. 18

C.4 Constructing Enhanced Instances

  • Rebuttals are treated as latent supervision: GPT-5.4 converts aligned responses into an initial-review diagnostic claim plus structured revision suggestions specifying what, where, how, and expected outcome.Automatic filtering removes rebuttal leakage, retrospective phrasing, near-copying, and underspecified suggestions.Source: §C.4 · pp. 19–20

C.6 Localized Evidence Retrieval

  • Task-specific localized retrieval remains competitive with full-paper conditioning at substantially lower input cost, using top-2 chunks for diagnosis and top-5 balanced evidence for revision generation.Task 1 is relatively insensitive to chunk size, whereas Task 2 generally plateaus around k=5 because actionable suggestions require evidence and localization cues.Source: §C.6 · pp. 21–22
  • On a 200-context stratified subset, human annotators report a 78.6% pass rate for Task 2 localization support using dual-channel retrieval.Source: §C.6 · pp. 21, 23, Table 14 · p. 23

C.7 Implementation Details and Compute Budget

  • ActReview jointly fine-tunes Qwen3-8B-Base on both tasks, then applies GRPO with K = 8 sampled responses, rubric-based rewards, and KL regularization on four NVIDIA H200 GPUs.Training uses full-parameter SFT, deterministic format checks, GPT-5.4 semantic scoring, and distributed execution across four GPUs.Source: §C.7 · p. 23

D Rubric Construction and Reward Design … E.2 Human Evaluation Instructions

ActReview’s RL reward design uses frozen, candidate-aware rubrics to make feedback scoring instance-specific, while evaluation protocols combine pairwise LLM and human judgments across diagnostic and actionable-review dimensions.

D.1 Candidate-Aware Rubric Construction

  • Candidate-aware rubrics distinguish paper-specific missing evidence, revision locations, and candidate failure modes that generic quality prompts cannot identify.Rubrics are synthesized offline from sampled SFT outputs and reference-style outputs, then frozen before GRPO; they combine hard constraints with weighted soft requirements.Source: §D.1 · pp. 23–24, D Rubric Construction and Reward Design · p. 23

D.2 Rubric Scoring Prompt

  • An LLM judge scores each output from 1 to 5 on every weighted soft requirement, normalizes scores to [0, 1], and combines them using rubric weights.The scoring prompt operationalizes each frozen instance-specific rubric during GRPO.Source: §D.2 · p. 24

D.3 Direct Judge Reward Prompt

  • The direct-judge ablation replaces instance-specific rubrics with fixed task-level dimensions to test whether generic LLM judging alone explains GRPO gains.This isolates the contribution of explicit candidate-aware rubric construction.Source: §D.3 · p. 24

D.4 Fixed Task-Level Rubric Prompt

  • The fixed task-level rubric is shared across instances and evaluates technical accuracy, grounding, constructiveness, format, actionability, and overall usefulness without adapting to paper-specific weaknesses.It also penalizes rebuttal references, unsupported content, unrelated weakness categories, generic advice, and format violations; unsupported weaknesses may justify None for accuracy but not overall quality.Source: §D.4 · p. 24

D.5 Rubric Examples

  • Candidate-aware examples specify concrete missing evidence, paper settings, revision locations, and candidate failure modes, making rewards more discriminative than generic rubrics.Tables 15 and 16 illustrate this contrast for diagnostic claims and actionable suggestions.Source: §D.5 · p. 25, Table 15 · p. 26, Table 16 · p. 27

E Evaluation Details

  • The evaluation appendix documents baseline, schema-controlled, judge, human-annotation, metric, alignment, and bootstrap-confidence-interval procedures.These materials support the evaluation protocol used in Section 6.Source: E Evaluation Details · p. 25

E.1 LLM-as-a-Judge Prompt

  • GPT-5.4 pairwise judging compares ActReview-RL with each baseline under identical inputs, swaps output order to reduce position bias, and reports adjusted win rates across shared and task-specific dimensions.Technical Accuracy, Depth & Constructiveness, and Overall are shared; Specificity & Grounding applies to Task 1 and Actionability to Task 2, with ties scored as 0.5.Source: §E.1 · pp. 25–26

E.2 Human Evaluation Instructions

  • Human evaluation uses two blinded graduate annotators on 200 ActReview-Bench instances, and their judgments yield 93% identical categorical choices.Annotators compare anonymized outputs across accuracy, constructiveness, task-specific quality, and overall usefulness; averaged scores preserve disagreements and produce comparable adjusted win rates.Source: §E.2 · pp. 26–27

E.3 Automatic Metrics. … F.1 End-to-End versus Two-Task Formulation

The evaluations define specialized automatic metrics and calibrated claim matching, uncertainty-aware confidence intervals, and additional analyses of task decomposition. The two-task formulation outperforms end-to-end generation across all five evaluated dimensions, especially overall revision quality and claim–suggestion alignment.

E.3 Automatic Metrics.

  • Automatic metrics quantify claim counts, specificity, suggestion similarity, and evidence specificity using localized references, technical terms, quantitative details, and semantic alignment.Specificity is computed at the claim level for Task 1 and on extracted evidence for Task 2, while suggestion similarity aligns generated actions with reference actions.Source: §E.3 · pp. 27–28
  • The example-specific Rubric score combines deterministic hard constraints with GPT-5.4 judgments of weighted soft requirements to capture instance-specific criteria missed by global overlap metrics.Hard-constraint violations yield zero; otherwise, 1–5 soft scores are mapped to [0, 1], weighted, and averaged across instances.Source: §E.3 · p. 28

E.4 Claim Alignment for Automatic Metrics

  • Claim-level ROUGE-L, BLEU, and semantic similarity use one-to-one maximum-similarity alignment, exposing unmatched outputs as over-generation or under-generation.Average generated claim count is reported separately, and None outputs are categorized as correct abstention, false-positive over-generation, or under-generation.Source: §E.4 · p. 28

E.5 Bootstrap Confidence Intervals

  • The study reports 95% bootstrap confidence intervals for automatic metrics, adjusted judge win rates, and ablations, treating differences below two percentage points as trends unless consistently supported.Instances are resampled with replacement, and percentile intervals are computed from the resulting bootstrap distributions.Source: §E.5 · p. 29

F Additional Evaluations and Analyses

  • Additional evaluations examine generalization beyond rebuttal-derived references, independent-judge robustness, scientific validity, technical failures, localized-context limitations, and incomplete rebuttal resolutions.These analyses broaden evaluation beyond the two-task formulation comparison.Source: F Additional Evaluations and Analyses · p. 29

F.1 End-to-End versus Two-Task Formulation

  • Two-task SFT improves all five evaluation dimensions over end-to-end SFT, including +0.42 Task 2 Overall quality and +0.35 Claim–Suggestion Alignment.The comparison uses the same backbone, data, optimization, and protocol before rubric-based reinforcement learning; Table 6 reports the result.Source: §F.1 · p. 29

F.2 Independent Evaluation on 2025–2026 Papers … G.3 Metrics

This section evaluates ActReview’s generalization, judge robustness, revision usefulness, and remaining technical limitations, while characterizing validity–actionability tradeoffs and related analysis settings. It also defines a non-supported split and abstention protocol with five percentage-based metrics for assessing answer behavior and appropriate abstention.

F.2 Independent Evaluation on 2025–2026 Papers

  • Resolution Sufficiency rises from 1.23 to 1.70 and Reviewer Usefulness from 1.34 to 1.68 for ACTREVIEW-RL versus GPT-5.1schema on 175 held-out instances.ACTREVIEW-RL remains comparable in Weakness Validity (1.68 versus 1.67) and Technical Correctness (1.66 versus 1.64), supporting temporal and supervision-source generalization without establishing cross-venue or cross-discipline generalization.Source: §F.2 · p. 30, Table 17 · p. 30

F.3 Cross-Judge Robustness

  • Across both tasks and three baselines, GPT-5.4, Claude, and humans assign ACTREVIEW-RL adjusted win rates above 50, with judge–human correlations of ρ = 0.943 and ρ = 0.899.Mean absolute deviations from human win rates are 1.81 points for GPT-5.4 and 1.88 points for Claude; neither judge systematically favors ACTREVIEW-RL.Source: §F.3 · p. 31, Table 19 · p. 31, Table 20 · p. 31

F.4.1 Scientific Validity versus Revision Utility

  • ACTREVIEW-RL achieves the highest Valid + Actionable rate at 74% and the lowest Actionable but Invalid rate at 10%, despite Scientific Validity of 1.71 versus GPT-5.1’s 1.70.The results indicate stronger conversion of valid diagnoses into executable revisions rather than universally superior deep scientific judgment.Source: §F.4.1 · p. 32, Table 21 · p. 32

F.4.2 Fine-Grained Technical Failures

  • ACTREVIEW-RL matches Geminischema’s 28.00% overall technical-error rate, improves over GPT-5.1schema and ACTREVIEW-SFT at 29.14%, and has the lowest severe-error rate at 6.29%.Its primary remaining failure is Overstated Necessity, occurring in 17.14% of cases versus 5.71% for GPT-5.1schema and 10.29% for Geminischema.Source: §F.4.2 · p. 32, Table 22 · p. 33

F.5 Global-Context Analysis

  • Full-paper training increases Cross-Section Reasoning from 1.17 to 1.49 and reduces Unsupported Claim Rate from 22% to 14%, while localized training slightly improves Resolution Sufficiency.Weakness Validity and Technical Correctness remain comparable, indicating complementary benefits for localized and globally distributed evidence.Source: §F.5 · p. 33, Table 23 · p. 33

F.6 Hard Rebuttal-Incomplete Cases

  • On hard rebuttal-incomplete cases, ACTREVIEW-RL extends the rebuttal in 46% and rejects an inadequate path in 20%, improving upon or departing from the author response in 66%.It introduces an unsupported resolution in 6% of cases, while the analysis does not establish performance on fundamentally non-resolvable concerns.Source: §F.6 · p. 34, Table 24 · p. 34

G.1 Motivation and Relation to Reviewer Concerns

  • Abstention is evaluated zero-shot because training resources contain only reviewer-supported weaknesses, while prompts and grounding rewards require returning None when the queried label lacks paper support.The non-supported split contains 360 screened distractor pairs, with a human spot-check agreeing on 27 of 29 cases (93.1%).Source: §G.1 · p. 34, §G.2 · p. 35, Table 25 · p. 35

G.2 Constructing the Non-Supported Split

  • The non-supported split samples distractor Level-2 labels absent from each paper, prioritizes unraised Level-1 categories, and applies the same full-paper inference protocol as supported instances.Only pairs screened as absent are retained, including checks for conditional-label materiality and content reported in appendices.Source: §G.2 · pp. 34–35
  • The non-supported split treats “absent” as no full-paper evidence for a label, verified by LLM screening and human sampling, rather than as an exhaustive weakness inventory.Systematically characterizing every weakness a paper does not exhibit is outside the evaluation’s scope.Source: §G.2 · p. 35

G.3 Metrics

  • The evaluation reports Answer Rate, False-Abstention Rate, Abstention Rate, Over-generation Rate, and Balanced Accuracy for binary answer-versus-abstain decisions.Balanced Accuracy averages Answer Rate and Abstention Rate; the metrics reuse main-experiment Task-1 outputs and require no new inference.Source: §G.3 · p. 35
  • All models receive the same abstention-aware instruction and input context, with stripped non-empty outputs counted as answers and outputs equal to None counted as abstentions.Answer Rate and False-Abstention Rate use the 1,000 supported Task-1 instances, while Abstention Rate and Over-generation Rate use Nneg non-supported instances.Source: §G.3 · p. 35

G.4 Results · G.5 Case Study

ACTREVIEW-RL substantially improves abstention on unsupported weaknesses while preserving reliable answers on supported instances, although over-generation remains. The case study shows that full-paper evidence enables correct abstention when queried weaknesses are inapplicable.

G.4 Results

  • Balanced Accuracy rises from 68.2% for Qwen3-32B to 86.9% for ACTREVIEW-RL, an absolute improvement of 18.7 percentage points.Table 25 also shows at least 95% Answer Rate and at most 5% False-Abstention Rate for all models on supported instances.Source: §G.4 · pp. 35–36
  • ACTREVIEW-RL abstains on 78.3% of non-supported instances versus 71.4% for ACTREVIEW-SFT, while its supported-instance False-Abstention Rate is 4.6%.The 6.9-point RL gain is consistent with the rubric’s hard penalty on wrong-label and hallucinated claims.Source: §G.4 · p. 36
  • Prompt-based and specialized baselines rarely abstain on unsupported labels, with GPT-5.1schema fabricating weaknesses under 68.6% and DeepReviewer-14B under 87.8% of cases.These behaviors are consistent with a strong prior that queried labels correspond to real problems and that generic criticism can fit almost any label.Source: §G.4 · p. 36
  • ACTREVIEW-RL still over-generates on 21.7% of screened non-supported pairs, motivating explicit negative training and harder within-category distractor evaluation.This residual error is also illustrated by the representative failure in Figure 20.Source: §G.4 · p. 36, §G.5 · p. 36

G.5 Case Study

  • Case-study examples show ACTREVIEW-RL correctly returns None when full-paper context makes a queried weakness inapplicable or adequately supported elsewhere.One example concerns missing computational-cost analysis: the paper provides scalability, hardware, and runtime evidence supporting its efficiency claim.Source: §G.5 · p. 36
  • The case study’s evaluation prompts screen label absence using the full paper and appendices, distinguishing present, absent, and uncertain weakness judgments for non-supported pairs.Figure 19 describes this screening protocol as adapted from retrieval-quality screening but producing a single label decision.Source: Figure 19 · p. 49

Related papers

Loading 2609.09076v1…