Source-linked AI summary
DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Neha Kennard, Tim O'Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, Andrew McCallum
TL;DR
Peer-review decision makers must process extensive technical text, while prior work leaves discourse relations between reviews and rebuttals underexamined. DISAPERE provides a sentence-annotated dataset and models for predicting labels and contexts. The paper reports that rebuttal discourse cues can inform review interpretation and that models could help decision makers identify patterns, while noting limits in domain coverage and context modeling.
Problem
Peer-review discourse research has limited coverage of relations between reviews and rebuttals, despite the burden of extensive technical text on decision makers.
Source: §1 · p. 1Method
DISAPERE labels every sentence in 506 review–rebuttal pairs with review properties and rebuttal actions, stance, and review context.
Source: §1 · pp. 1–2Results
Baseline models predict DISAPERE labels and contexts, with macro-averaged F1 generally lower on tasks with larger label spaces and particularly low for ASPECT.
Source: §1 · p. 2, §6.1 · p. 8Takeaways & Limitations
The dataset and its annotations can support analysis of argumentative patterns that may inform decision makers and future conference policies.
Source: §1 · pp. 1–2DISAPERE’s AI-domain data may not generalize to other fields, and its context-alignment models predict each rebuttal sentence’s context independently.
Source: §7 · p. 9, §6.2 · p. 9
Abstract
from arXiv · showhide
At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined. We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors' stance towards review arguments. Further, we annotate every review and rebuttal sentence. We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.
1 Introduction
Peer review requires decision makers to process extensive technical discussions, while existing datasets and research leave gaps in sentence-level coverage and review–rebuttal discourse. DISAPERE addresses these gaps with a labeled dataset and a scheme for analyzing both sides of the exchange.
1 Introduction
- Peer review’s quality-control role and the growth of academic output burden area chairs and editors with extensive technical text.The paper proposes using machine-learning tools to identify patterns in argumentative strategies, goals, and intentions rather than generate condensed accounts, which risk amplifying biases.Source: §1 · p. 1
- Prior work focused on arguments, used disparate taxonomies, and left discourse relations between reviews and rebuttals—and rebuttal structure itself—understudied.Qualified annotation is needed because interpreting technical discourse requires domain understanding.Source: §1 · p. 1
- Review sentences receive four properties, while each rebuttal sentence gets an intention-and-commitment label and links to the review sentences forming its context.The review properties synthesize prior taxonomies; annotating every sentence also avoids propagating errors from argument detection.Source: §1 · pp. 1–2
- The paper reports baseline model performance, annotation guidelines and software, and examples of analyses that could inform conference decision makers.Its annotations involved over 850 person-hours from trained graduate students, with much of the test data double-annotated.Source: §1 · p. 2
- DISAPERE labels 506 English review–rebuttal pairs comprising over 20k sentences, covering every review and rebuttal sentence.The dataset gives reviews and rebuttals equal importance and includes fine-grained labels for non-argumentative types.Source: §1 · pp. 1–2
2 Related work
Prior peer-review datasets address different argument phenomena and offer taxonomies that are not directly compatible. DISAPERE unifies and extends these schemas while labeling sentences across reviews and rebuttals.
2 Related work
- Prior peer-review datasets use diverse label sets: AMPERE annotates argumentative propositions with domain-tailored labels, AMSR emphasizes argument stance toward acceptance or rejection, while other work adds polarity and aspect labels.Source: §2 · p. 2
- Cheng et al. (2020, 2021) annotated discourse relations between review and rebuttal sentences; DISAPERE instead emphasizes comprehensive dataset creation and less specialized baselines.Gao et al. (2019) found that specific, convincing, and explicit responses are more likely to elicit score changes, observations formalized in DISAPERE’s rebuttal-action labels.Source: §2 · p. 3
- DISAPERE unifies prior schemas hierarchically, adds fine-grained implicit and explicit request classes, and labels sentences rather than argument spans.Sentence-level units provide broader text coverage and avoid errors propagated from upstream argument-detection models, though other unit sizes may capture some discourse information more precisely.Source: §2 · p. 3
3 Dataset
DISAPERE annotates review–rebuttal pairs at the sentence level, combining review-function and aspect labels with rebuttal actions, stances, and review contexts. The dataset uses initial ICLR responses, and its reported annotator agreement ranges from moderate to substantial.
3 Dataset
- Each dataset example pairs a review with a rebuttal; DISAPERE uniquely labels every sentence in both and applies discourse labels to authors’ rebuttal actions.Pairs are drawn from English scientific discussions, with multiple reviews per manuscript where possible.Source: Table 1 · p. 3, §3 · p. 3
3.1.1 Review actions
- Review-action labels classify each sentence as evaluative, factual, a request, social, or structuring.Questions count as requests for information.Source: §3.1.1 · p. 3
3.1.2 Fine-grained review actions
- Fine-grained review labels subdivide structuring sentences and distinguish request types such as explanations, experiments, edits, and typo fixes.Request subtypes also include clarification of factual information.Source: §3.1.2 · pp. 3–4
3.1.3 Aspect and polarity
- For evaluative and request sentences, aspect labels cover manuscript qualities and polarity marks comments as positive or negative.The aspects include clarity, originality, soundness/correctness, replicability, substance, impact/motivation, and meaningful comparison.Source: §3.1.3 · p. 4
3.2.1 Rebuttal actions
- Each rebuttal sentence receives one of 14 action labels, grouped by whether the author concurs with, disputes, or takes a non-argumentative stance toward review context.For request responses, annotations also record completion or promise when the author concurs, and whether a requested task is out of scope when disputed.Source: §3.2 · p. 4, §3.2.1 · p. 4
3.2.2 Rebuttal context
- A rebuttal sentence’s context is the set of review sentences it responds to, allowing many-to-many links and global-context or no-context labels.This avoids fixed discourse chunking, accommodating replies to either broad sections or specific sentences.Source: §3.2.2 · pp. 4–5, Table 3 · p. 5
3.3 Data Source and Annotation
- The dataset uses ICLR 2019 and 2020 discussions, splits manuscripts 3:1:2 across train, development, and test, and focuses on initial author responses.All texts associated with a manuscript stay in one split; all test-set pairs are double-annotated.Source: §3.3 · p. 5, Table 4 · p. 5
3.4 Agreement
- Annotators label review sentences before linking rebuttal sentences to review contexts; reported agreement is moderate to substantial for rebuttal-action and rebuttal-stance labels.Cohen’s κ is calculated for multiply annotated sentences, with all pairs of annotators used when more than two annotations exist.Source: §3.3 · p. 5, §3.4 · p. 6, Table 5 · p. 6
4 Analysis
The analyses show that rebuttal contexts are not captured by sentence order alone, authors sometimes interpret review actions in noncanonical ways, and discourse-action patterns vary with ratings.
4.1 Context types
- 84.81% of rebuttal sentences link to some review context, while a small number refer to other rebuttal sentences instead.Table 6 presents context types linked to review subsets and cases without a particular review subset.Source: §4.1 · p. 6, Table 6 · p. 6
4.2 Alignment
- 0.794 is the median Spearman’s ρ between rebuttal and aligned review sentence indices; exact in-order responses are rare, so order alone misses alignments.A linear inductive bias may still benefit alignment models.Source: Figure 2 · p. 6, §4.2 · p. 6
4.3 Author interpretations of criticism
- Authors often treat evaluative statements as requests and respond as if they were asked to act, sometimes to appease a reviewer despite no explicit request.The context distributions also show that authors sometimes interpret requests as criticisms and evaluative sentences as questions.Source: §4.3 · p. 7, Figure 3 · p. 7
4.4 Relating discourse features to rating
- High-scoring manuscripts rarely receive requests for experiments or typo fixes; low-scoring ones also have few typo requests, a pattern flagged for further examination.Figure 4 presents REVIEW-ACTION distributions by rating.Source: §4.4 · p. 7, Figure 4 · p. 7
5 Application: Agreeability
The paper proposes agreeability as a discourse-based cue for area-chair attention alongside reviewer-score variance. Some low-agreeability manuscripts may merit attention despite not having high score variance.
5 Application: Agreeability
- The authors propose using discourse information as an interpretable, data-driven heuristic that draws on review content rather than numerical scores alone.They suggest it could support area chairs’ and other decision makers’ judgments.Source: §5 · p. 7
- Agreeability is the ratio of CONCUR sentences to argumentative rebuttal sentences.It is only weakly correlated with rating (Pearson’s r = 0.347).Source: §5 · p. 7
- 18% (28/159) of manuscripts outside the high-variance top quartile nevertheless fall in the low-agreeability bottom quartile and may merit closer attention.Figure 5 plots mean agreeability against reviewer variance, with dot color indicating mean reviewer rating.Source: §5 · p. 7, Figure 5 · p. 8
6 Baselines
The baselines assess sentence-label prediction and rebuttal-to-review context alignment, with results showing both ambiguous classification cases and substantial room to improve alignment.
6 Baselines
- The baselines are intended to measure off-the-shelf model performance and expose uncaptured phenomena, rather than introduce discourse-modeling innovations.Source: §6 · p. 8
6.2 Rebuttal context alignment
- Sentence classification uses BERT sentence embeddings and a feedforward classifier, while alignment compares BM25 with a Siamese-BERT model using cosine similarity.Alignment is evaluated with mean reciprocal rank and mean average precision.Source: §6.1 · p. 8, §6.2 · pp. 8–9
6.1 Sentence classification
- Sentence-classification F1 is generally lower for tasks with larger label spaces, and ASPECT has particularly low F1.ASPECT’s κ remains within moderate agreement bounds, which the authors attribute to task difficulty rather than deficient data quality.Source: §6.1 · p. 8, Table 7 · p. 8
- Classification errors largely mirror annotator disagreements on ambiguous utterances, including rhetorical questions and sentences that can fit multiple labels.Additional context can disambiguate some rhetorical questions, while reviewer-specific conventions and subtle lexical cues create other challenges.Source: §6.1 · p. 8, Table 8 · p. 9
6.2 Rebuttal context alignment
- BM25 outperforms the neural alignment model, though both leave significant room for improvement and lexical overlap alone is insufficient.Both models predict each rebuttal sentence’s context independently, without accounting for that sentence’s broader rebuttal context.Source: §6.2 · p. 9, Table 9 · p. 9
7 Conclusion
The paper presents DISAPERE as a resource for data-driven insight into peer-review interactions amid growing review workloads. Its scope is constrained by the AI domain of most publicly available peer-review data, and the authors call for robustness across fields and time.
7 Conclusion
- As peer-review workloads grow, models trained on DISAPERE are intended to help decision makers gain insight into interactions during review.Source: §7 · p. 9
- Because almost all publicly available peer-review data comes from AI, DISAPERE may not generalize to all new domains.Source: §7 · p. 9
- The authors provide detailed annotation guidelines and software to support follow-on work aimed at robust insights across fields and over time.Source: §7 · p. 9
8 Ethics
The authors caution against using machine-learning outputs for individual peer-review decisions and call for bias evaluation of interventions.
8 Ethics
- Because peer-review outcomes can strongly affect scholars’ careers, do not use machine-learning outputs to decide individual cases.Interventions based on participant-behavior information should be studied for whether they introduce or exacerbate bias.Source: §8 · p. 9
A Rationale for taxonomy construction
The taxonomy combines prior peer-review discourse labels with additional distinctions for requests and rebuttal behavior. The appendix also describes two annotation modes and shows an example annotated pair.
A Rationale for taxonomy construction
- The taxonomy draws on prior peer-review schemes, grouping summary and quote-like sentences under structuring and adding fine-grained request and rebuttal labels.It formalizes rebuttal behaviors such as expressing gratitude, promising revisions, and disagreeing with criticism.Source: A Rationale for taxonomy construction · p. 11
B Annotation tool
- The annotation tool supports sentence-by-sentence labeling with constrained dropdowns and a separate mode for linking rebuttal sentences to review context.In the review interface, fine-review-action options depend on the selected review-action label.Source: B Annotation tool · p. 11, Figure 6 · p. 12
C Annotated review-rebuttal pair
- Figure 9 presents a truncated annotated review-rebuttal pair from the DISAPERE training set.Source: C Annotated review-rebuttal pair · p. 12
D Context overlap analysis
The analysis uses 81 double-annotated examples as a proxy for agreement on rebuttal-context spans. It distinguishes exact matches and shared decisions that no context applies from cases where annotators disagree about whether a context subset exists.
D Context overlap analysis
- 81 examples annotated by two annotators serve as a proxy for agreement on rebuttal-context spans.The analysis categorizes the types of overlap between their spans.Source: D Context overlap analysis · p. 12
- Exact matches and shared decisions that no review sentences form the context count as full agreement.These are the top rows of Table 10.Source: Table 10 · p. 12
- “Disagree none” occurs when one annotator marks a subset of review sentences as context and the other does not.Source: Table 10 · p. 12
E Additional Agreement Analysis
The agreement analysis links some low IAA scores to the complexity of the labels and offers guidance for reducing the label set. It identifies nuanced disagreements among rebuttal-action labels, including cases that are difficult to distinguish from answers.
E Additional Agreement Analysis
- Some IAA scores are low, which the authors relate to labels that characterize complex relationships in text.They provide guidance on potentially acceptable label merges.Source: E Additional Agreement Analysis · p. 12
- ANSWER, MITIGATE CRITICISM, and REJECT CRITICISM are described as labels that may lie on a continuum.The authors provide guidance on which label merges may be acceptable.Source: E Additional Agreement Analysis · p. 12
- Merging MITIGATE CRITICISM into REJECT CRITICISM is suggested as a first step to reduce IAA disagreement.The distinction is nuanced: authors may accept a reviewer’s point but dispute its relevance, or reject the point itself.Source: E Additional Agreement Analysis · p. 12
- A wide range of disagreements are hard to distinguish from ANSWER labels because authors often frame disagreements as simple answers to questions.The confusion matrix illustrates this pattern.Source: E Additional Agreement Analysis · p. 14, Figure 10 · p. 16