Source-linked AI summary
DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttal
Peixuan Han, Yingjie Yu, Jingjun Xu, Jiaxuan You
TL;DR
Academic rebuttal automation is underexplored, while existing LLM approaches struggle with long contexts and targeted persuasion. DRPG decomposes reviews, retrieves evidence, plans supported strategies, and generates responses; it outperforms existing pipelines, exceeds average human performance with an 8B model, and works in multi-round discussions.
Problem
Academic rebuttal remains underexplored, and existing approaches struggle to retrieve evidence from long papers and produce targeted, persuasive responses.
Method
DRPG uses Decomposer, Retriever, Planner, and Executor components to identify atomic concerns, retrieve evidence, select supported rebuttal perspectives, and generate responses.
Results
DRPG consistently outperforms existing rebuttal methods on top-tier conference data, achieves strong performance with a compact model, and improves rebuttals through interpretable multi-perspective planning.
Takeaways & Limitations
DRPG provides grounded, coherent, and convincing rebuttal support with potential to improve the quality and efficiency of scholarly discussions.
Takeaways & Limitations
DRPG clarifies and defends existing contributions but cannot conduct new experiments, and its outputs require author review because of hallucination risks.
Abstract
from arXiv · showhide
Despite the growing adoption of large language models (LLMs) in scientific research workflows, automated support for academic rebuttal, a crucial step in academic communication and peer review, remains largely underexplored. Existing approaches typically rely on off-the-shelf LLMs or simple pipelines, which struggle with long-context understanding and often fail to produce targeted and persuasive responses. In this paper, we propose DRPG, an agentic framework for automatic academic rebuttal generation that operates through four steps: Decompose reviews into atomic concerns, Retrieve relevant evidence from the paper, Plan rebuttal strategies, and Generate responses accordingly. Notably, the Planner in DRPG reaches over 98% accuracy in identifying the most feasible rebuttal direction. Experiments on data from top-tier conferences demonstrate that DRPG significantly outperforms existing rebuttal pipelines and achieves performance beyond the average human level using only an 8B model. Our analysis further demonstrates the effectiveness of the planner design and its value in providing multi-perspective and explainable suggestions. We also showed that DRPG works well in a more complex multi-round setting. These results highlight the effectiveness of DRPG and its potential to provide high-quality rebuttal content and support the scaling of academic discussions. Codes for this work are available at https://github.com/ulab-uiuc/DRPG-RebuttalAgent.
1 Introduction
Academic rebuttal remains underexplored despite AI support across other research stages. DRPG addresses long-context evidence selection and weakly targeted responses through decomposition, retrieval, planning, and generation.
- Academic rebuttal remains largely underexplored despite growing AI assistance throughout scientific research workflows.
- Automated rebuttal could reduce authors’ workload as submission volumes and review demands increase.NeurIPS and ICLR received over 25,000 submissions in 2025.
- Existing LLM approaches struggle with long-context evidence retrieval and often produce generic, defensive, or insufficiently targeted responses.
- DRPG decomposes reviews, retrieves relevant paper evidence, plans rebuttal strategies, and generates responses.Its Planner identifies the most supported perspective with over 98% accuracy.
- Experiments on top-tier conference data show DRPG outperforms existing rebuttal pipelines and surpasses average human performance using an 8B model.It achieves around 40 points higher Elo score than existing rebuttal pipelines.
2 Related Work
Prior research has emphasized review simulation, reviewer training, rebuttal datasets, and zero-shot generation, while treating rebuttal automation as relatively neglected. Related debate and planning research supplies conceptual foundations for DRPG’s focus on persuasion and argument selection.
- Research on AI-assisted peer review has focused mainly on simulating reviews and training more effective reviewers.
- Academic rebuttal has received relatively little attention, with existing work collecting datasets and using zero-shot LLMs for preliminary responses.
- Rebuttal resembles debate and persuasion because authors seek to convince reviewers to change their opinions.
- Recent debate research has explored agentic pipelines and reinforcement learning, providing related approaches for structured argument generation.
- Academic rebuttal is presented as a promising debate domain because it is fact-oriented, paper-grounded, and supported by substantial public data.
- Planning and selecting argument directions matter in rebuttal because response quality may affect submission outcomes with few opportunities for revision.
Retriever
The Retriever supports DRPG’s planning-based workflow by supplying targeted paper evidence for each decomposed review point. This retrieval step is part of a broader agentic design for rebuttal generation.
- The Retriever identifies relevant paper paragraphs for review points, reducing the content passed downstream by around 75% in practice.
- DRPG trains a Planner to select rebuttal perspectives as part of its agentic framework.
3 Method
DRPG processes reviews through decomposition, evidence retrieval, perspective planning, and execution. Its Planner scores candidate perspectives against relevant paper paragraphs, while the Executor turns structured outputs into persuasive point-level responses.
- 3.1 Decomposer: The Decomposer converts multifaceted reviews into independent, fine-grained points representing reviewer weaknesses and questions.
- 3.2 Retriever: The Retriever addresses long-context difficulties by selecting relevant paragraphs for each atomic point and reducing downstream content length by around 75%.It incorporates hierarchical section headers when encoding paragraphs.
- 3.3 Planner: The Planner guides responses by explicitly planning how to address each review question rather than merely stating paper details.
- 3.3.1 Perspective Generation: An idea proposer generates candidate clarification or justification perspectives, while withholding paper content to encourage diverse proposals.
- 3.3.2 Selecting the Valid Perspective: The Planner scores each candidate perspective–paragraph pair with an encoder and MLP, then averages scores across relevant paragraphs.The encoder produces representations and the MLP computes supportive scores.
- 3.3.2 Selecting the Valid Perspective: Planner training uses five synthetic perspectives and one ground-truth perspective per review point with cross-entropy optimization.
- 3.3.2 Selecting the Valid Perspective: The paper excludes admitting weakness and conducting experiments because the text-only setting cannot conduct real experiments and may encourage shortcuts.
- 3.3.2 Selecting the Valid Perspective: During inference, a confidence threshold determines whether the selected perspective reaches the Executor or the pipeline falls back to a no-Planner setting.
4 Experiments
Experiments evaluate DRPG on a large rebuttal dataset using multiple model families, structured baselines, and LLM-based quality metrics. Across settings, DRPG performs best, while analyses support the value of its modular workflow and content-aware Planner.
- Experimental Setup: Experiments use Re2, containing over 17k papers and approximately 60k reviews and rebuttals from 45 top-tier computer science conferences.
- Experimental Setup: DRPG is evaluated across four base LLMs spanning different families and sizes, including Qwen3-8B, GPT-oss-20B, Mixtral-8x7B, and LLaMa3.3-70B.
- Experimental Setup: The evaluation compares DRPG with human-written rebuttals, component-ablated agentic variants, and Jiu-Jitsu, which replaces DRPG’s Planner with predefined templates.
- Experimental Setup: Elo scores from GPT-4o pairwise comparisons are used to assess rebuttal quality, while traditional ROUGE and BLEU metrics are omitted because they poorly capture reasoning quality and coherence.
- Main Results: All agent-based variants substantially outperform Direct prompting, while DRPG consistently beats other variants and achieves the highest post-rebuttal scores in most settings.
- Main Results: DRPG’s Decomposer and Retriever focus complex reviews, its Planner selects response directions, and its Executor generates tailored responses; content-aware planning outperforms fixed templates.
5 Analysis
The analysis shows that DRPG’s Planner benefits from paper-grounded, learned perspective selection, while combining clarification and justification improves rebuttal quality. DRPG also maintains an increasing advantage over baselines in simulated multi-round discussions, and its score analysis offers interpretable claim–evidence guidance.
- 5.1 Ablation Study on Planner Design: 98.64% accuracy was achieved by the Planner in identifying perspectives adopted in successful human rebuttals, outperforming alternative designs.The comparison evaluates different planner designs using training loss and test accuracy.
- 5.1 Ablation Study on Planner Design: Paper content is essential for planning, because scoring perspectives without it performs poorly.The no-paper variant scores each perspective independently without using paper content.
- 5.1 Ablation Study on Planner Design: Using Retriever-selected relevant paragraphs significantly improves planning over full-paper input by reducing irrelevant content in the score aggregation.Relevant paragraphs help focus the Planner on material directly related to each review point.
- 5.1 Ablation Study on Planner Design: Learned scoring is necessary because encoder-only similarity is insufficient to capture nuanced relationships between rebuttal perspectives and supporting evidence.The Planner’s MLP models perspective–paragraph relationships rather than relying solely on surface-level semantic similarity.
- 5.2 Analyse on Two Types of Rebuttal Perspectives: Restricting responses to Clarification or Justification underperforms full DRPG and the DRG baseline, supporting context-dependent use of both perspectives.Clarification addresses technical misunderstandings, whereas Justification responds to critiques based on subjective or questionable evaluation standards.
- 5.3 Interpreting Planner Scores: Planner score inspection reveals claim–evidence relationships and provides a structural guide for composing or refining rebuttals.In the illustrated example, one perspective is strongly supported by a paragraph about SinGAN’s performance on challenging tasks, with another paragraph supplying auxiliary evidence.
- 5.4 Multi-round Discussion with DRPG: Across three discussion rounds, baseline scores plateau after the first round while DRPG continues improving and gains an increasingly pronounced advantage.The multi-round workflow alternates DRPG rebuttals with judge-generated follow-up feedback.
6 Conclusion
DRPG addresses academic rebuttal automation with a four-component agentic framework that decomposes reviews, retrieves evidence, plans strategies, and generates responses. On top-tier conference data, it outperforms existing rebuttal methods, while structured planning provides interpretable, multi-perspective support.
- 6 Conclusion: DRPG comprises Decomposer, Retriever, Planner, and Executor components for generating grounded, coherent, and convincing rebuttals.The framework decomposes reviewer feedback, retrieves targeted evidence from long papers, and explicitly plans rebuttal strategies.
- 6 Conclusion: DRPG consistently outperforms existing rebuttal methods on top-tier conference data and achieves strong performance with a compact model.
- 6 Conclusion: Structured planning provides an interpretable, multi-perspective signal that meaningfully improves rebuttal quality.
Limitation
DRPG focuses on clarifying paper content and defending existing contributions, but it cannot conduct new experiments during rebuttal. The paper identifies integration with AI Scientist systems as a future direction for experimental supplementation.
- Limitation: DRPG primarily clarifies paper content and defends existing contributions rather than conducting new experiments.
- Limitation: Additional experimental results can sometimes help address reviewers’ concerns during rebuttal.
- Limitation: Integrating DRPG with AI Scientist systems is proposed as a future direction toward experimental supplementation and more complete rebuttal automation.
Ethical Considerations
The supplied passages describe implementation details for prompts, evaluation, dataset reconstruction, Planner training, and dataset statistics. They do not state an ethical analysis or ethical risk beyond these procedural details.
- Ethical Considerations: Prompts for the DRPG pipeline are shown in Figures 5 to 8, and generated responses for individual points are merged into a complete rebuttal.
- Ethical Considerations: Evaluation prompts appear in Figures 9 and 10, with rebuttal order randomly swapped to avoid position bias.
- Ethical Considerations: Missing initial review scores are reconstructed with GPT-oss-120B from rebuttal text and final scores, then validated through human analysis of sampled examples.
- Ethical Considerations: The Planner training set contains 50,000 review points, with 5,000 additional review points reserved for evaluation.
- Ethical Considerations: The Planner is an MLP trained for three epochs while freezing the BGE-M3 encoder and updating only the MLP module.
B.3 Judge Model Training
The judge model is trained with GRPO using Qwen3-4B and receives reward based on the distance between its predicted score and the ground-truth score. The supplied passages also situate Jiu-Jitsu as a planning baseline using canonical rebuttal templates.
- B.3 Judge Model Training: The judge model uses Group Relative Policy Optimization with Qwen3-4B as its base model.
- B.3 Judge Model Training: The reward is r = 0.25|s−sg|, where s is the judge’s answer and sg is the actual score.The passage states that exact score matches receive full reward and that reward decreases as the prediction diverges.
- B.3 Judge Model Training: Jiu-Jitsu is included as a planning baseline that generates perspectives by selecting a canonical rebuttal template.
- B.3 Judge Model Training: Jiu-Jitsu maps atomic review concerns to canonical templates through concern description, cluster assignment, candidate retrieval, and ranking.
D Case Study
The case study compares Decomp, DRG, and DRPG, with GPT-4o pairwise analyses highlighting the Retriever and Planner’s value for producing more concrete rebuttals. DRPG’s retrieval encodes paragraphs with hierarchical section titles, while its planner considers multiple rebuttal perspectives.
- D Case Study: The Retriever prepends parent and subsection titles to paragraph text before encoding.For a paragraph in Section 4.1, the titles of Sections 4 and 4.2 are added to support high-level understanding of paper content.
- D Case Study: Figures 15 and 16 compare Decomp, DRG, and DRPG through GPT-4o pairwise analyses.The analyses are used to examine the value of the Retriever and Planner in producing more concrete rebuttals.
- D Case Study: The Perspective Generator proposes up to 5 perspectives for addressing each reviewer point in a rebuttal.The perspectives must either challenge the reviewer’s point or establish the work’s value even when the criticism is correct.
- D Case Study: Clarification is one required perspective type for correcting factual errors or reviewer misunderstandings.The prompt gives examples involving supposedly missing content that is present and methodology criticized because of misunderstanding.
- D Case Study: The evaluation prompts ask reviewers to compare author responses comment by comment and decide whether to adjust the original score.The scoring rubric uses a 1–10 scale, including score 6 for papers in the top 25% and slightly above the acceptance threshold.