DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttal

1 Introduction

• Rebuttal is essential to fair evaluation, and preparing thoughtful responses is increasingly time-consuming as research communities expand.The introduction cites over 25,000 submissions at NeurIPS and ICLR in 2025 as an example of the growing scale and need for more efficient author-reviewer communication.Source: §1 · p. 1
• Prior off-the-shelf LLMs and ad-hoc pipelines (Jin et al., 2024b) struggle with evidence retrieval and persuasive, critique-specific argumentation.Long, information-dense papers make relevant evidence difficult to retrieve, while generated responses can be generic, overly conciliatory, or defensive.Source: §1 · p. 1
• DRPG decomposes reviews into atomic concerns, retrieves relevant paper paragraphs, plans among rebuttal perspectives, and generates responses.Retrieval reduces input length by over 75% while preserving critical evidence, and the Planner identifies the most supported perspective with over 98% accuracy.Source: §1 · p. 2
• DRPG outperforms existing rebuttal pipelines by around 40 Elo points and surpasses average human performance using an 8B model.Further analyses find that its Planner provides multi-perspective, explainable suggestions that improve rebuttal quality; DRPG also performs well in multi-round discussions, and a human study validates the evaluation metrics.Source: §1 · p. 2

LLM for academic rebuttal

• Rebuttal research remains limited, with prior work collecting real-world datasets (Zhang et al., 2025a; Kennard et al., 2022), generating zero-shot drafts, or training template-based agents.The paper positions its approach as a more systematic rebuttal agent.Source: §2 · p. 2

LLM for debate and persuasion

• Debate research equips LLMs with human persuasion strategies (Wang et al., 2019), dialogue data (Singh et al., 2024), agentic pipelines (Wang et al., 2025a), and debate reinforcement learning (Cheng and You, 2025).The authors characterize rebuttal as a promising debate domain because it is fact-oriented, grounded in high-quality papers, and supported by substantial public data.Source: §2 · p. 2

Planning in high-stakes decision making

• High-stakes rebuttal planning can prune candidate directions by simulating consequences with Monte Carlo search trees or supervised selector/verifier networks (Han et al., 2025; Wang et al., 2025b).The passage notes that consequence simulation is most common in multi-agent interactions or external environments.Source: §2 · p. 2

• The Retriever retrieves relevant paper paragraphs for each review point.Source: Retriever · p. 3

3.1 Decomposer

• The LLM-based Decomposer identifies reviewer weaknesses and questions, splitting multifaceted reviews into independent, fine-grained points for downstream modules.Source: §3.1 · p. 3

3.2 Retriever

• The Retriever uses dense similarity search to select paper paragraphs relevant to each atomic review point, encoding hierarchical section headers to capture context.Only the most relevant paragraphs go to the Executor, reducing content length by around 75% in practice.Source: §3.2 · p. 3

3.3 Planner

• The two-step Planner explicitly plans responses by proposing candidate perspectives and selecting a paper-supported perspective for each review point.This is intended to help LLMs address reviewers’ underlying reasoning and value judgments rather than merely state paper details.Source: §3.3 · p. 3, §3.3.1 · pp. 3–4

3.3.2 Selecting the Valid Perspective

• The selector scores candidate perspective–paragraph pairs with an encoder and MLP, averages scores across relevant paragraphs, and passes a perspective onward only above threshold T.Training uses five synthetic perspectives and one ground-truth perspective per review point, with cross-entropy loss; below-threshold confidence triggers fallback to generation without the Planner.Source: §3.3.2 · p. 4

3.4 Executor

• The Executor uses structured outputs from preceding modules to generate a coherent, persuasive rebuttal paragraph for each review point.It can be instantiated with a general-purpose LLM or a rebuttal-specialized model.Source: §3.4 · p. 4

3 Method

• DRPG integrates four specialized components to address lengthy paper and review texts and reduce generic, insufficiently targeted rebuttals.The framework comprises a Decomposer, Retriever, Planner, and Executor.Source: §3 · p. 3, §3.4 · p. 4

4.1 Experimental Setup

• Experiments use Re2, with over 17k papers and approximately 60k reviews and rebuttals, and test four base models spanning different families and sizes.The dataset covers 45 top-tier computer science conferences over eight years; the evaluated models are Qwen3-8B, GPT-oss-20B, Mixtral-8x7B, and LLaMa3.3-70B.Source: §4.1 · p. 4

4.2 Main Results

• DRPG consistently outperforms other variants in pairwise comparisons and achieves the highest post-rebuttal scores in most settings, while agent-based variants substantially outperform Direct prompting.Table 1 reports performance across settings; the accompanying results discussion attributes the gains to structured processing of reviews rather than prompting an LLM to answer an entire review at once.Source: Table 1 · p. 5, §4.2 · p. 6

4.1 Experimental Setup

• The comparison includes dataset human rebuttals (REAL) and four agentic baselines: Direct, Decomp, DRG, and Jiu-Jitsu.Direct, Decomp, and DRG are ablated DRPG variants; Jiu-Jitsu uses predefined question-type templates in place of DRPG’s Planner.Source: §4.1 · p. 5
• Evaluation uses GPT-4o pairwise comparisons summarized as Elo scores and a Qwen3-4B judge model that matches human reviewers’ scores on 71% of test data.Elo scores use a Bradley–Terry model with a base rating of 1000.Source: §4.1 · pp. 5–6
• Inference retrieves K = 15 paragraphs per review point and uses Planner confidence threshold T = 0.8, assigning a valid perspective to approximately 62% of review points.The remaining points are typically straightforward or depend heavily on paper-specific content, making a valid Planner perspective difficult to propose.Source: §4.1 · p. 6

4.2 Main Results

• Jiu-Jitsu performs below DRPG because its fixed canonical templates can yield generic or impractical perspectives, whereas DRPG selects perspectives based on paper content.The content-aware planning is intended to produce more specific and persuasive rebuttals.Source: §4.2 · p. 6

5.1 Ablation Study on Planner Design

• 98.64% accuracy in identifying the perspective adopted in successful human rebuttals makes the Planner outperform all tested alternatives.Scoring perspectives without paper content performs poorly; using only relevant retrieved paragraphs improves on using the full paper, while encoder similarity alone is insufficient.Source: Table 3 · p. 6, §5.1 · pp. 6–7

5.2 Analyse on Two Types of Rebuttal Perspectives

• Restricting rebuttals to Clarification or Justification alone underperforms full DRPG and even planner-free DRG, supporting context-dependent use of both strategies.Clarification addresses factual misunderstandings, whereas Justification responds to critiques grounded in subjective or questionable evaluation standards; the restricted-perspective experiment uses LLaMa3.3-70B.Source: Table 4 · p. 7, §5.2 · p. 7

5.3 Interpreting Planner Scores

• Planner scores reveal learned claim–evidence links beyond surface similarity, making perspective selection more transparent and offering human authors a structural guide for rebuttal writing.In the example, perspective 3 is strongly supported by paragraph 3’s statement that SinGAN performs better on challenging tasks, with paragraph 4 providing auxiliary evidence about richer information embedding.Source: Figure 2 · p. 7, §5.3 · pp. 7–8

5.4 Multi-round Discussion with DRPG

• Across three discussion rounds, DRPG’s advantage grows as baseline judge scores plateau after round one, indicating stronger use of earlier feedback and responses to follow-up questions.The study simulates alternating DRPG rebuttals and judge-model feedback, using the judge’s chain-of-thought as the next round’s review.Source: Figure 3 · p. 8, §5.4 · p. 8

5.5 Human Study on Pairwise Comparison

• Three computer-science experts showed highly consistent preferences, and their judgments substantially aligned with GPT-4o, supporting its validity as a rebuttal-quality proxy.The study sampled 50 reviews and rebuttal pairs from DRG and DRPG, selected to include 10 cases favoring each system under GPT-4o.Source: §5.5 · p. 8, Table 6 · p. 8

6 Conclusion

• DRPG combines a Decomposer, Retriever, Planner, and Executor to generate grounded, coherent rebuttals by decomposing feedback, retrieving targeted evidence, and planning strategies.This design addresses challenges that limit off-the-shelf LLMs in rebuttal generation.Source: §6 · p. 8
• On top-tier conference data, DRPG consistently outperforms existing rebuttal methods and achieves strong performance with a compact model.Source: §6 · p. 9
• Structured planning provides an interpretable, multi-perspective signal that meaningfully improves rebuttal quality and may support more effective scholarly discussions.The authors present agentic systems like DRPG as having potential to improve the quality and efficiency of scholarly discussions.Source: §6 · p. 9

Limitation

• DRPG focuses on clarifying paper content and defending existing contributions, but it cannot conduct new experiments.The authors identify this as a limitation despite DRPG’s strong performance.Source: Limitation · p. 9
• Additional experimental results can sometimes help address reviewers’ concerns during rebuttal.Source: Limitation · p. 9
• Integrating DRPG with AI Scientist systems is proposed to support experimental supplementation and complete rebuttal automation.Source: Limitation · p. 9

Ethical Considerations

• DRPG outputs should be carefully reviewed and verified by authors before submission or release because rebuttal agents can hallucinate.The caution applies to DRPG and other rebuttal agents.Source: Ethical Considerations · p. 9

A Prompts

• After responding to individual review points, the Executor merges them into a complete rebuttal; pipeline prompts appear in Figures 5–8.Figure 7 is identified for the Executor in Direct and other settings in the passage.Source: A Prompts · p. 12
• Evaluation prompts appear in Figures 9 and 10, and rebuttal order is randomly swapped during comparison to avoid position bias.Source: A Prompts · p. 12

B.1 Data Statistics

• GPT-oss-120B predicts missing initial review scores from rebuttal text and final scores to reconstruct full rebuttal trajectories.Only a subset of official reviews receives author responses, and some initial scores are missing.Source: §B.1 · p. 12
• The reconstructed rebuttal-score distributions met expectations, and human analysis of randomly sampled examples was used to validate the predictions.Source: §B.1 · p. 12

B.2 Planner Training

• Planner training data combines five candidate perspectives from the idea proposer with one ground-truth perspective extracted from human-authored rebuttals, yielding 50,000 training and 5,000 evaluation review points.Source: §B.2 · p. 12
• The Planner is a three-hidden-layer MLP that scores concatenated perspective–paragraph vectors; training freezes the BGE-M3 encoder and updates only the MLP.Hidden-layer sizes are 2048, 1024, and 512; training runs for 3 epochs with batch size 32 and learning rate 5 × 10−5.Source: §B.2 · p. 12

B.3 Judge Model Training

• The judge model uses GRPO training with Qwen3-4B as its base model.Figure 4 presents the judge model’s training configuration.Source: §B.3 · p. 12, Figure 4 · p. 12
• Its reward is calculated as r = 0.25|s−sg|, where s is the predicted score and sg is the actual score.The passage says exact matches receive full reward and reward decreases exponentially as the prediction gap increases.Source: §B.3 · p. 12
• The judge is expected to think carefully before producing a final score.Source: §B.3 · p. 12

C The Jiu-Jitsu Baseline

• Jiu-Jitsu (Purkayastha et al., 2023) serves as a planning baseline, generating perspectives by selecting canonical rebuttal templates rather than using DRPG’s Planner.Source: C The Jiu-Jitsu Baseline · p. 12
• For each atomic review point, Jiu-Jitsu uses a fine-tuned sequence-to-sequence model to produce a natural-language description that abstracts wording and summarizes the concern.Source: C The Jiu-Jitsu Baseline · p. 12
• It maps the concern description to the closest attitude root–theme cluster, pairing a review aspect with a target paper section.Source: C The Jiu-Jitsu Baseline · pp. 12–13
• Using the cluster and its rebuttal action label, Jiu-Jitsu retrieves candidate rebuttals and ranks them against the cluster description and action.Source: C The Jiu-Jitsu Baseline · p. 13
• The top-ranked candidate is selected as the canonical rebuttal and used as the pipeline’s rebuttal perspective.Source: C The Jiu-Jitsu Baseline · p. 13

D Case Study

• A retriever example prepends the titles of Sections 4 and 4.2 to a Section 4.1 paragraph before encoding, enabling high-level understanding of the paper.Figure 14 shows the encoded paragraph format.Source: D Case Study · p. 13, Figure 14 · p. 18
• Examples compare Decomp, DRG, and DRPG, with GPT-4o comparison comments indicating the Retriever and Planner’s value in providing more concrete rebuttals.The examples appear in Figures 15 and 16.Source: D Case Study · p. 13, Figure 15 · p. 18, Figure 16 · p. 18