PaperlayerView exact revision on arXiv

Source-linked AI summary

RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mind

Zhitao He, Zongwei Lyu, Yi R Fung

arXiv:2601.15715v3cs.CLcs.AI

TL;DR

Academic rebuttal remains underexplored because effective responses require strategic perspective-taking under information asymmetry, while existing approaches often imitate surface-level language. RebuttalAgent addresses this gap with a Theory-of-Mind-based TSR framework, specialized training and evaluation, and reports an average 18.3% improvement over the base model.

  • Problem

    Academic rebuttal requires addressing reviewer perspectives and concerns beyond surface-level politeness.

  • Method

    RebuttalAgent uses TSR to profile reviewers, formulate strategies, and generate evidence-based responses, supported by RebuttalBench and staged SFT-to-RL training.

  • Results

    18.3% average improvement over the base model is reported, with comparable performance to advanced proprietary models across automated and human evaluations.

  • Takeaways & Limitations

    The framework makes strategic reviewer modeling and evidence-grounded response generation central to academic rebuttal.

  • Takeaways & Limitations

    The system is intended as guidance rather than a replacement for scholarly engagement and may reinforce biases in its training data.

Abstract

from arXiv · show

Although artificial intelligence (AI) has become deeply integrated into various stages of the research workflow and achieved remarkable advancements, academic rebuttal remains a significant and underexplored challenge. This is because rebuttal is a complex process of strategic communication under severe information asymmetry rather than a simple technical debate. Consequently, current approaches struggle as they largely imitate surface-level linguistics, missing the essential element of perspective-taking required for effective persuasion. In this paper, we introduce RebuttalAgent, the first framework to ground academic rebuttal in Theory of Mind (ToM), operationalized through a ToM-Strategy-Response (TSR) framework that models reviewer mental state, formulates persuasion strategy, and generates evidence-based response. To train our agent, we construct RebuttalBench, a large-scale dataset synthesized via a novel critique-and-refine approach. Our training process consists of two stages, beginning with a supervised fine-tuning phase to equip the agent with ToM-based analysis and strategic planning capabilities, followed by a reinforcement learning phase leveraging the self-reward mechanism for scalable self-improvement. For reliable and efficient automated evaluation, we further develop Rebuttal-RM, a specialized evaluator trained on over 100K samples of multi-source rebuttal data, which achieves scoring consistency with human preferences surpassing powerful judge GPT-4.1. Extensive experiments show RebuttalAgent significantly outperforms the base model by an average of 18.3% on automated metrics, while also outperforming advanced proprietary models across both automated and human evaluations.

1 INTRODUCTION

RebuttalAgent frames academic rebuttal as strategic, perspective-taking communication rather than surface-level linguistic imitation. It combines Theory of Mind, the TSR framework, specialized data and training, and Rebuttal-RM evaluation to improve rebuttal quality.

  • Motivation: Current SFT-based approaches often imitate polite linguistic patterns without the strategic depth required for rebuttal.The paper attributes this limitation to insufficient perspective-taking and strategic reasoning about reviewer concerns.
  • Framework: RebuttalAgent uses Theory of Mind to model reviewer perspectives, identify underlying concerns, and formulate evidence-based responses through TSR.The framework is presented as a shift from linguistic imitation toward strategic reasoning.
  • Training: RebuttalBench contains over 70K samples with complete ToM-strategy-response chains, produced through a critique-and-refine pipeline.The dataset supports training the agent’s reasoning and rebuttal capabilities.
  • Training: RebuttalAgent is trained with SFT followed by RL using a self-reward mechanism to improve strategic analysis without an external expert-annotated reward model.The staged process first establishes foundational capabilities and then optimizes strategic policies.
  • Evaluation: 18.3% average improvement over the base model accompanies comparable performance to advanced proprietary models across automated and human evaluations.Rebuttal-RM is trained on over 100K multi-source samples and is reported to align with human expert preferences more consistently than GPT-4.1.

2 TASK FORMULATION

The paper formulates academic rebuttal as generating a convincing response to one target comment using the manuscript, a review, and that comment as inputs. Responses should be convincing, context-aware, and evidence-grounded while balancing these objectives.

  • Task objective: The rebuttal task aims to generate a convincing response to a target comment.The target comment is an individual critique, query, or identified weakness requiring a direct response.
  • Inputs: The inputs are the manuscript M, one review Ri from the review set, and a target comment ctarget within that review.The manuscript provides the evidentiary basis, while the review contains critiques and queries.
  • Generation: A model G generates the target response rtarget from these task inputs.The formulation treats response generation as a function of the manuscript, review, and selected comment.
  • Requirements: A successful response must be convincing, context-aware, and evidence-grounded, balancing these competing objectives.Claims and counter-arguments should be verifiably substantiated by the manuscript while addressing explicit and underlying reviewer concerns.

3 DATA PREPARATION

The data-preparation process addresses noisy, unstructured reviews and the difficulty of supplying an entire manuscript as context. It extracts target comments and retrieves focused manuscript content for each comment.

  • Review processing: Raw reviews mix substantive critiques with irrelevant material, adding noise and redundancy that can reduce rebuttal accuracy.Examples include greetings and summary restatements.
  • Review processing: Review comments are unstructured because reviewers use diverse writing styles and conference formats.This motivates preprocessing before addressing individual target comments.
  • Review processing: The pipeline processes reviews to align generation with the task of addressing one target comment at a time.The approach uses LLM-based information extraction to organize raw review content.
  • Context retrieval: Full manuscripts can overwhelm the model and dilute focus, so the system retrieves the most relevant content for each comment.The retrieval module segments the manuscript into discrete chunks before isolating relevant context.

4 TOM-STRATEGY-RESPONSE FRAMEWORK

The TSR framework operationalizes Theory of Mind by profiling the reviewer, converting that profile into a strategy, and synthesizing a response from strategic and retrieved evidence. Its stages connect reviewer modeling with targeted, context-aware generation.

  • Theory of Mind: Theory of Mind models differing beliefs, intentions, desires, and perspectives, which RebuttalAgent applies to reviewers in academic rebuttal.The framework targets the information-asymmetric and game-theoretic nature of rebuttal.
  • Theory of Mind: The framework begins by constructing a hierarchical reviewer profile to guide strategy and response generation.This profile explicitly models reviewer beliefs, knowledge background, and core concerns.
  • Reviewer analysis: Macro-level analysis infers overall reviewer intent, attitude, dominant concern, and expertise to guide global strategy and tone.It builds a holistic mental model beyond the literal review text.
  • Reviewer analysis: Micro-level analysis classifies each comment by significance, methodology, experimental rigor, and presentation for targeted responses.The resulting micro-profile aligns tactical responses with the global strategy.
  • Strategy: Strategy generation converts the reviewer profile and target comment into a concise, actionable plan before response drafting.This intermediate step bridges reviewer understanding and final response formulation.
  • Response generation: Final synthesis combines the reviewer profile and strategy with retrieved chunks and the original response to produce a strategically aligned, grounded response.The original response supplies context and a phrasing or structure reference during data synthesis, not final inference.

5 AGENT TRAINING FOR STRATEGIC PERSUASION

The training pipeline combines structured TSR supervision with self-reward reinforcement learning to develop strategically stronger rebuttal outputs. RebuttalBench supplies synthesized reasoning chains, while multiple reward signals guide policy optimization.

  • RebuttalBench: RebuttalBench structures generated training targets as concatenated Analysis, Strategy, and Response components.The dataset uses multiple powerful teacher models to generate a holistic TSR learning objective.
  • Supervised fine-tuning: Supervised fine-tuning teaches Qwen3-8B the structured TSR reasoning process and core rebuttal competencies.The training data combines varied reviews with synthesis from multiple LLMs to support robustness across reviewing styles.
  • Self-reward: The overall self-reward combines four weighted components: R(o) = w1Rformat(o) + w2Rthink(o) + w3Rresp(o) + w4Rdiv(o).The reward explicitly aggregates format, reasoning, response, and diversity signals.
  • Self-reward: Self-reward evaluates candidate responses across format adherence, reasoning quality, response quality, and response diversity.The model checks structure programmatically and uses its own evaluations for reasoning, persuasiveness, clarity, evidence use, and diversity.
  • Reinforcement learning: GRPO generates groups of candidates and updates the policy using clipped optimization with a frozen reference-policy regularizer.Each candidate receives an advantage based on relative rewards within its group.

6 REBUTTAL-RM AS JUDGE

Rebuttal-RM is designed to score rebuttal responses against contextual review information and align automated evaluation with human judgments. Its training data combines human, refined, and model-generated responses.

  • Evaluation: Table 1 reports consistency scores between models and human ratings across six standard statistical metrics, showing only a subset in the main paper.Additional metrics are provided in Appendix C.1 and Table 11.
  • Evaluator design: Rebuttal-RM scores responses using the manuscript context, review, target comment, and candidate response.The evaluator outputs multidimensional scores and an explanation.
  • Training data: The training set contains over 102K instances from original author responses, GPT-4.1-refined responses, and diverse model-generated replies.The three sources provide a human baseline, high-quality reference responses, and broader style coverage.
  • Training data: Rebuttal-RM uses 90% of the labeled data for training and 10% for testing, with Qwen3-8B as its base model.The final evaluator is fine-tuned on the constructed rebuttal dataset.

7 EXPERIMENT

The experiments evaluate Rebuttal-RM’s agreement with human judgments and RebuttalAgent against foundation and agent-based baselines on in-domain and out-of-domain rebuttal data. RebuttalAgent achieves strong automated and human-evaluation performance, while ablations support the contribution of its major components.

  • Rebuttal-RM Evaluation: 0.812 average score: Rebuttal-RM achieves the highest alignment with human judgments and leads on all individual metrics.It surpasses GPT-4.1 by 9.0% and DeepSeek-r1 by 15.2%.
  • Benchmarking RebuttalAgent: RebuttalAgent achieves the highest overall average score of 9.42, outperforming all evaluated baselines including GPT-4.1 and o3.It also attains 9.43 for Clarity and 9.20 for Persuasiveness.
  • Benchmarking RebuttalAgent: 18.3% average improvement over Qwen3-8B: RebuttalAgent gains most in Persuasiveness and Constructiveness, with improvements up to 34.6%.The reported results use R2-test, with out-of-domain results provided separately on Rebuttal-test.
  • Ablation Study: Removing ToM, Strategy, Thinking, SFT, or RL significantly reduces performance, while final-response-quality reward is the most impactful reward signal.Applying the framework to Llama-3.1-8B and Qwen3-4B raises scores from 7.44 to 9.20 and from 7.69 to 8.98, respectively.
  • Human Evaluation: 9.57 average human-evaluation score: RebuttalAgent leads o3 and GPT-4.1 across Attitude, Clarity, Persuasiveness, and Constructiveness.The evaluation used 100 balanced comments and three experienced annotators; Table 3 reports the four dimensions.
  • Human Evaluation: 9.34 Persuasiveness: RebuttalAgent improves 7.36% over the GPT-4.1 baseline, its largest relative human-evaluation gain.The result is reported alongside high scores on the other evaluation metrics.

8 RELATED WORK

Related work situates RebuttalAgent at the intersection of machine Theory of Mind, LLM debate, and LLM-assisted academic peer review. The paper extends these lines of research to strategic academic rebuttal.

  • Machine Theory of Mind: Machine Theory of Mind models others’ mental states, and prior systems use belief graphs, perspective-taking, symbolic checking, or opponent modeling.RebuttalAgent extends machine ToM to academic rebuttal through hierarchical reviewer-intent analysis.
  • LLM Debate: LLM debate uses multi-agent interaction and simulated collaborative or adversarial dialogue to enhance complex reasoning and fact-checking.This literature provides a related interaction-based approach to improving model reasoning.
  • LLM for Academic Peer Review: LLMs are increasingly applied to academic activities including automated research, writing assistance, review generation, and review-quality analysis.RebuttalAgent addresses academic rebuttal within this broader AI-for-research and peer-review landscape.

9 CONCLUSION

The paper presents RebuttalAgent as a Theory-of-Mind-based framework for strategic academic rebuttal, supported by RebuttalBench and Rebuttal-RM. Experiments report an 18.3% improvement over base models and comparable performance to advanced models across automated and human evaluations.

  • RebuttalAgent grounds academic rebuttal in Theory of Mind and frames it as a strategic reasoning task.
  • The framework combines a 70K-reasoning-chain RebuttalBench dataset, reinforcement learning with self-reward, and the Rebuttal-RM evaluator.
  • 18.3% improvement over base models was reported, with performance comparable to o3 across automated and human evaluations.

ETHICAL CONSIDERATION

The paper positions RebuttalAgent as an assistant for organizing and articulating rebuttals rather than replacing scholarly engagement. It warns that training-data biases may produce inappropriate persuasion strategies or rebutting evidence, so authors must critically review outputs.

  • RebuttalAgent is intended to guide scholars through rebuttal preparation, not replace genuine scholarly engagement.
  • Training-data biases may cause the system to reinforce inappropriate or unscholarly persuasion strategies and rebutting evidence.
  • Experimental-result comments were excluded during training to prevent fabricated evidence or data.
  • Authors are expected to assess generated content for accuracy, fairness, and rationality.

REPRODUCIBILITY STATEMENT

The paper reports documentation intended to support reproducibility of its framework, including dataset generation and complete training procedures. It also discloses limited use of LLMs for language-level polishing.

  • The framework includes Rebuttal-RM, RebuttalBench, and RebuttalAgent.
  • Dataset generation and complete RebuttalAgent training procedures, including hyperparameters, are provided in Section 5.
  • The manuscript used Gemini and GPT-4.1 only as general-purpose writing assistants for language-level polishing.

B DATA PREPARATION

The data-preparation evaluation measures comment extraction and context retrieval. Extracted comments were generally aligned with reviewer intent, while direct-comment retrieval with paragraph segmentation achieved the highest retrieval effectiveness among tested configurations.

  • Comment Extraction Accuracy: Over 98 percent of extracted comments were complete and aligned with reviewers’ intended points in a manual sample of 100 raw reviews.
  • Context Retrieval Effectiveness: The retrieval study compared three comment-encoding strategies and three manuscript-segmentation strategies using cosine similarity.
  • Context Retrieval Effectiveness: Directly using the original comment as the retrieval query with paragraph segmentation achieved the highest retrieval effectiveness.
  • Hierarchical Reviewer Profile: Table 4 organizes the hierarchical reviewer profile into macro-level and micro-level dimensions.

C.1 SETUP AND METRICS OF REBUTTAL-RM

Rebuttal-RM evaluates rebuttal quality through statistical correlation and score-range accuracy, while the broader pipeline structures reviewer analysis and response generation. The setup emphasizes standardized JSON outputs, multi-dimensional scoring, and reviewer-comment extraction.

  • Metrics: Rebuttal-RM uses Mean Absolute Error, Pearson, Spearman, and Kendall statistics to assess evaluator correlation.These metrics assess overall agreement with human judgments.
  • Metrics: Coarse-grained Accuracy maps scores into four quality tiers, from Unconvincing to Excellent.The tiers are Unconvincing (1-3), Acceptable (4-6), Good (7-8), and Excellent (9-10).
  • Metrics: Fine-grained Accuracy uses seven narrower score ranges for stricter evaluation.The rubric includes paired low-score ranges and single-point ranges for scores of 5 and 6.
  • Prompt setup: The rebuttal-generation prompt requires sequential analysis, strategy formation, and response drafting in structured tags.The analysis uses review context, a target comment, and a relevant paper fragment before producing the response.
  • Scoring rubric: The evaluation rubric scores Attitude, Clarity, Persuasiveness, and Constructiveness on a 0-10 scale.The output requires both dimension scores and explanations tied to the original response text.
  • Reviewer modeling: The reviewer-analysis prompt extracts global stance and comment-level concerns, categories, severity, and confidence into a single JSON object.Its taxonomy covers novelty, methodology, experimental rigor, and presentation-related issues.
Loading 2601.15715v3…