PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals

Sukannya Purkayastha, Anne Lauscher, Iryna Gurevych

arXiv:2311.03998v1cs.CL

TL;DR

Peer-review rebuttal writing is challenging, and existing computational counterargument work focuses on surface-level reasoning rather than underlying attitudes. This paper introduces attitude- and theme-guided rebuttal generation, enriches peer-review data with canonical rebuttals, and benchmarks generation strategies. In theme prediction, transformer models outperform baselines, with domain-specialized SciBERT exceeding its non-specialized counterpart by more than one percentage point.

  • Problem

    Existing computational counterargument generation focuses on surface-level reasoning, while writing effective peer-review rebuttals is challenging, particularly for younger researchers and non-native speakers.

    Source: §1 · pp. 1–2

  • Method

    The paper builds JITSUPEER by linking peer-review sentences with attitude roots, themes, and canonical rebuttals, then benchmarks models for end-to-end generation and two subtasks.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    In theme prediction, SciBERTds_neg outperforms its non-specialized counterpart by more than 1 percentage point.

    Source: §3.2.1 · p. 4

  • Takeaways & Limitations

    JITSUPEER provides review sentences linked to canonical rebuttals that can serve as templates for writing peer-review rebuttals.

    Source: §6 · p. 9

    JITSUPEER’s attitude taxonomy and canonical rebuttals derive from ICLR 2019 and 2020 reviews, so the taxonomy is specific to the machine-learning community and needs adaptation across domains.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In many domains of argumentation, people's arguments are driven by so-called attitude roots, i.e., underlying beliefs and world views, and their corresponding attitude themes. Given the strength of these latent drivers of arguments, recent work in psychology suggests that instead of directly countering surface-level reasoning (e.g., falsifying given premises), one should follow an argumentation style inspired by the Jiu-Jitsu 'soft' combat system (Hornsey and Fielding, 2017): first, identify an arguer's attitude roots and themes, and then choose a prototypical rebuttal that is aligned with those drivers instead of invalidating those. In this work, we are the first to explore Jiu-Jitsu argumentation for peer review by proposing the novel task of attitude and theme-guided rebuttal generation. To this end, we enrich an existing dataset for discourse structure in peer reviews with attitude roots, attitude themes, and canonical rebuttals. To facilitate this process, we recast established annotation concepts from the domain of peer reviews (e.g., aspects a review sentence is relating to) and train domain-specific models. We then propose strong rebuttal generation strategies, which we benchmark on our novel dataset for the task of end-to-end attitude and theme-guided rebuttal generation and two subtasks.

1 Introduction

Peer review and rebuttals are important to scientific publication, but crafting effective rebuttals is challenging. This work brings Jiu-Jitsu argumentation to peer review by building a dataset and benchmarking attitude- and theme-guided rebuttal generation.

1 Introduction

  • Peer review supports high-quality science, while rebuttals give authors a chance to persuade reviewers to raise their assessment scores.Writing effective rebuttals is challenging, particularly for younger researchers and non-native speakers.Source: §1 · p. 1
  • Existing computational counterargument generation focuses on surface-level reasoning, whereas Jiu-Jitsu argumentation uses underlying attitude roots to shape rebuttals.The paper is the first to explore this approach for peer-review rebuttals.Source: §1 · pp. 1–2
  • The proposed approach reuses reviewing aspects and paper-section targets to identify attitude roots and themes, then retrieves or generates a canonical rebuttal for a specified action.The example maps a Clarity/Overall review to a rebuttal for Reject Criticism.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper introduces attitude root- and theme-guided peer-review rebuttal generation and benchmarks it end to end and on two related subtasks.The subtasks generate abstract descriptions of reviews reflecting underlying attitudes and score sentences for suitability as canonical rebuttals.Source: §1 · p. 2
  • The work enriches peer-review data with attitude roots, themes, and canonical rebuttals, and trains models specialized for peer review through intermediate training.It also benchmarks baselines for end-to-end generation and two related subtasks.Source: §1 · p. 2

2 Jiu-Jitsu Argumentation for Peer Review Rebuttals

Jiu-Jitsu argumentation uses an arguer’s underlying beliefs to shape congenial counterarguments rather than directly opposing surface reasoning. For peer review, the paper proposes generating a reusable canonical rebuttal from a review’s attitude root and theme together with a rebuttal action.

2.1 Background: Jiu-Jitsu Argumentation

  • Jiu-Jitsu argumentation uses an opponent’s underlying beliefs to identify congenial counterarguments, rather than relying on one’s own force.Canonical rebuttals are general-purpose counterarguments usable across arguments in the same attitude root–theme cluster.Source: §2.1 · p. 2

2.2 Attitude Root and Theme-Guided Peer Review Rebuttal Generation

  • In peer review, attitude roots and themes represent reviewers’ underlying beliefs and opinions, while the appropriate rebuttal also depends on the specific rebuttal action.The same root and theme can pair with different actions, such as Reject Criticism or Task Done.Source: §2.2 · p. 2
  • A canonical rebuttal is a congenial counterargument that addresses underlying attitudes, generalizes across a root–theme review tuple, and expresses a specific rebuttal action.The proposed task generates this rebuttal from a peer-review argument and an action, based on the argument’s attitude root and theme.Source: §2.2 · pp. 2–3

3 JITSUPEER Dataset

JITSUPEER enriches peer-review data by linking review sentences to attitude roots, themes, and action-specific canonical rebuttals. It reuses existing review annotations and adds predicted themes, cluster descriptions, and ranked rebuttal candidates.

3.1 Starting Datasets

  • The starting data combine ICLR reviews and rebuttals with review aspects, review–rebuttal links, rebuttal actions, and paper-section annotations.The authors use review aspects as attitude roots and paper sections as candidate attitude themes.Source: §3.1 · p. 3

3.2 Enrichment

  • The enrichment pipeline predicts themes, describes root–theme clusters, and identifies canonical rebuttals for particular rebuttal actions.It links review sentences in each root–theme cluster to canonical rebuttals.Source: §3.2 · p. 3, Figure 2 · p. 4

3.2.1 Theme Prediction

  • SciBERTds_neg achieves the best theme-prediction score, exceeding its non-specialized counterpart by more than 1 percentage point.The authors use it for final theme prediction; Table 1 reports Precision, Recall, and Micro-F1 on the PEER-REVIEW-ANALYZE test set across five runs.Source: §3.2.1 · p. 4, Table 1 · p. 4
  • For theme prediction, transformer models outperform majority and random baselines, and domain specialization improves SciBERT’s score.The best model is selected by micro-F1 on PEER-REVIEW-ANALYZE; Table 1 presents the model results.Source: §3.2.1 · p. 4, Table 1 · p. 4

3.2.2 Root–Theme Cluster Descriptions

  • Natural-language descriptions supplement root–theme labels for interpretability, with annotators selecting between human-written summaries and extracted representative review sentences.Two annotators labeled 99 instances with Cohen’s κ of 0.8, and a third resolved disagreements.Source: §3.2.2 · p. 5

3.2.3 Canonical Rebuttal Identification

  • Candidate reduction combines a suitability classifier with specificity scores, leaving 1,845 rebuttal candidates for annotation.The classifier retains sentences with confidence above 95%; SPECIFICITELLER scores between 0.02 and 0.78 further reduce the set.Source: §3.2.3 · p. 5
  • Annotators compare candidate rebuttals pairwise for root–theme clusters and actions, after which the collected preferences are ranked to select canonical responses.The task used 4,402 pairwise judgments; Annotation Graph Ranking and PageRank rank sentences, with the lowest-ranked nodes selected as canonical rebuttals.Source: §3.2.3 · pp. 5–6

3.3 Dataset Analysis

  • The final dataset contains 2,332 review sentences labeled with 8 attitude roots and 143 themes, plus 302 canonical rebuttals mapping to 2,219 review sentences.Different root–theme descriptions connect to different canonical rebuttals.Source: §3.3 · p. 6

4 Baseline Experiments

The experiments benchmark canonical rebuttal scoring, review-description generation, and end-to-end canonical rebuttal generation. Across generation tasks, BART leads in zero- and few-shot settings, while T5 performs best with full fine-tuning on review descriptions.

4 Baseline Experiments

  • The experiments introduce three tasks on the dataset, including end-to-end canonical rebuttal generation.The other tasks score canonical rebuttals and generate review-cluster descriptions.Source: §4 · p. 7, §4.1 · p. 7, §4.2 · p. 8

4.1 Canonical Rebuttal Scoring

  • SciBERTds_all has the highest Pearson correlation, while BERTds_neg has the highest ranking scores for canonical rebuttal scoring.Most domain-specialized models outperform their non-specialized counterparts.Source: §4.1 · pp. 7–8

4.2 Review Description Generation

  • Review-description scores roughly double on most measures after one example; BART leads in zero- and one-shot settings, while T5 leads with full fine-tuning.The authors hypothesize that T5’s larger capacity (770M parameters versus BART’s 406M) relates to its full fine-tuning advantage.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 End2End Canonical Rebuttal Generation

  • BART excels in zero- and few-shot end-to-end rebuttal generation, while performance plateaus after about two shots.The authors attribute the steep gains and plateau to limited canonical-rebuttal variety and a split at the canonical-rebuttal level, which constrain generalization beyond observed templates.Source: §4.3 · p. 8

5 Related Work

Prior peer-review NLP work has developed datasets and studied review–rebuttal links, while computational argumentation has largely focused on arguments’ explicit content. This work adapts attitude-root ideas to peer reviewing.

5 Related Work

  • Peer-review datasets annotate review actions, stance, polarity, aspects, and links between review sentences and paper sections; fewer studies examine review–rebuttal dynamics.Cheng et al. (2020) and Bao et al. (2021) study sentence-wise links, while Kennard et al. (2022) unify review labels and study interaction.Source: §5 · pp. 8–9

Computational Argumentation and Attitude

  • Computational argumentation has mostly analyzed explicit argument content, whereas this work adapts underlying-attitude theories to scientific peer reviewing.The related work describes prior study of attitude-driven arguments, including climate-change skepticism and anti-vaccination attitudes.Source: Computational Argumentation and Attitude · p. 9

6 Conclusion

The paper explores attitude-driven Jiu-Jitsu argumentation for peer reviews through JITSUPEER, a dataset linking review sentences to canonical rebuttals. It proposes tasks and baselines for studying this approach.

6 Conclusion

  • JITSUPEER links review sentences to canonical rebuttals that can serve as templates for peer-review responses.The authors make the annotations publicly available and propose the resource as support for further computational argumentation research on rebuttal writing.Source: §6 · p. 9

Limitations

JITSUPEER’s taxonomy and mappings are constrained by its source data and domain.

Limitations

  • JITSUPEER derives its roots, themes, and canonical rebuttals from ICLR 2019 and 2020; its ML-specific taxonomy needs adaptation for other fields, and its review-to-rebuttal mapping is sparse.The authors therefore frame model-generated rebuttals as templates for human refinement rather than a replacement for human-in-the-loop writing.Source: Limitations · p. 9

7 Appendix

The appendix reports dataset details, probing results, annotation procedures, and results or evaluation setups for rebuttal ranking and generation. It also points readers to dataset-license information and full generation-task results.

7.1 Information on dataset licenses

7.2 Dataset details

  • For PEER-REVIEW-ANALYZE, the Table 6 labels cover 60.24% of review sentences; Task Done and Answer are the most common annotated canonical-rebuttal actions.The appendix also reports review-sentence and rebuttal-action distributions and plots canonical-rebuttal counts by action.Source: Table 6 · p. 13, §7.2 · p. 13, Figure 9 · p. 17

7.4 Annotator Details and Interface Design

  • Two CS Ph.D. students annotated the dataset after one hour of guideline training; the team adapted INCEpTION for both human-intervention tasks.Figures show the interfaces for Root-Theme Cluster Description and Manual Evaluation for Canonical Rebuttal Identification.Source: §7.4 · p. 13, Figure 7 · p. 14, Figure 8 · p. 14

7.3 Probing of the Domain Specialized Models

  • Pretrained RoBERTa, SciBERT, SciBERTds_neg, and SciBERTds_all were the best-performing models in probing.The probing uses frozen representations for Aspect, Review Action, and Fine-grained Review Action classification, evaluated with Accuracy and Macro-F1.Source: §7.3 · p. 13, Table 9 · p. 16

7.5 Canonical Rebuttal Ranking Task

  • BM25 was competitive, but the fine-tuned BERT sentence transformer trained on negative reviews (BERTds_neg) was the best ranking model.The task retrieves a canonical rebuttal from a root–theme cluster using its description and rebuttal action.Source: §7.5 · pp. 14, 16, Table 10 · p. 16

7.6 Review Description Generation Task

  • Review-description results compare BART, Pegasus, and T5 under zero-shot, few-shot, and full fine-tuning using ROUGE-1, ROUGE-2, ROUGE-L, and BERTscore.Source: Table 11 · p. 16, §7.6 · p. 16

7.7 End2End Canonical Rebuttal Generation Task

  • End-to-end canonical-rebuttal generation results compare BART, Pegasus, and T5 under zero-shot, few-shot, and full fine-tuning using ROUGE-1, ROUGE-2, ROUGE-L, and BERTscore.The appendix also plots canonical-rebuttal counts by rebuttal action.Source: §7.7 · p. 16, Table 12 · p. 17, Figure 9 · p. 17

Related papers

Loading 2311.03998v1…