Argument Mining Driven Analysis of Peer-Reviews

Introduction

• Argumentative content gives evidence or reasoning supporting or opposing a point, unlike informative content that describes background or how entities act.Argumentation organizes reasons to persuade a reasonable critic to accept or reject a standpoint.Source: Introduction · p. 1
• Arguments in peer reviews support or oppose acceptance, and identifying them could help editors and meta-reviewers focus on material relevant to publication decisions.Reviewers may also receive support in identifying missing argumentation; a detailed account of evaluation criteria may not make their satisfaction clear.Source: Introduction · p. 1
• The paper applies Argument Mining to peer reviewing, collects and annotates a new dataset from computer-science conferences, and evaluates state-of-the-art techniques.The evaluation examines transfer between domains and generalization across conferences.Source: Introduction · p. 1

Argument Mining

• Argument Mining identifies argument components and their relations; schemes range from Toulmin’s components to simpler distinctions between supporting, opposing, and non-argumentative text.Simpler schemes can suit text types where complex components are implicit, and heterogeneous sources can pose challenges for generalization and annotation agreement.Source: Argument Mining · p. 2
• Earlier Argument Mining methods moved from feature engineering to neural networks, with pre-trained transformers achieving state-of-the-art performance.Argument search engines use extracted arguments to support retrieval by user-defined topic.Source: Argument Mining · pp. 2–3

Application of NLP for Peer-Reviewing Process

• Prior peer-review work introduced review propositions or detected problem descriptions, but this application needs stance and both positive and negative arguments.Hua et al. (2019) lacks stance information, while Xiao et al. (2020) focuses on problem descriptions.Source: Application of NLP for Peer-Reviewing… · p. 3
• Earlier work also studied citation-impact prediction from reviews, author replies in rebuttals, and argumentative structure in scientific papers.These works address different aspects of the peer-review process.Source: Application of NLP for Peer-Reviewing… · p. 3

Dataset

• Reviews were retrieved from OpenReview for six computer-science conferences, and the annotated dataset and code are available.The paper also provides basic statistics about conferences and collected reviews.Source: Dataset · p. 3

Preprocessing

• The authors sampled 77 of 12,135 collected reviews using stratification by rating, decision, and review length within conferences.Conference and review were sampled in sequence, with conferences selected uniformly at random.Source: Preprocessing · p. 3

Annotation

• The annotation scheme labels non-arguments, supporting arguments, and attacking arguments as NON, PRO, and CON, respectively.Seven graduate-level computer-science students annotated argumentative spans token by token; each review went to three annotators, with majority vote and a fourth annotator resolving three-way disagreements.Source: Annotation · p. 3
• Sentence-level labels are derived from token annotations by assigning the majority argumentative-segment label, with token-label frequency breaking ties.The dataset contains 28,502 annotated tokens and 1,401 sentences.Source: Annotation · pp. 3–4
• The reported class distribution is skewed toward NON at token level and toward CON at sentence level.Source: Table 2 · p. 4

Agreement

• uα = 0.568 and cuα = 0.861, and annotator Macro F1 was better than in comparable datasets; the authors conclude the annotations are reliable for further experiments.The agreement measures include label agreement, with uα additionally accounting for overlap length.Source: Agreement · p. 4

Experimental Setup

• The setup description applies to both token-level and sentence-level evaluation unless otherwise specified.Source: Experimental Setup · p. 4

Problem Setting

• The study separates argumentation detection and stance detection before jointly classifying spans as PRO, CON, or NON.Argumentation detection distinguishes ARG from NON, with ARG combining PRO and CON; stance detection classifies argumentative spans by support or opposition to acceptance.Source: Problem Setting · p. 4

Evaluation

• Evaluation uses a class-stratified sentence-wise 7:1:2 train-validation-test split and reports mean Macro F1 across ten runs.The validation set supports hyperparameter optimization and early stopping; the test set is reserved for final evaluation, with significance assessed by a two-sided t-test at 1%.Source: Evaluation · pp. 4–5

Methods

• The model comparisons include zero-shot transfer, transfer initialization followed by peer-review fine-tuning, and BERT models fine-tuned on the new dataset.The transferred model is trained on another argument-mining dataset; the locally fine-tuned variants use bert-base-cased or bert-large-cased.Source: Methods · p. 5
• Human performance is estimated by scoring each annotator against the final annotations, while the majority baseline predicts the most frequent class.The human score is the mean Macro F1 across annotators and is an upper bound because the same annotations serve as ground truth.Source: Methods · p. 5, Training · p. 5

Training

• Training uses class-weighted cross-entropy, with each class weighted by the reciprocal of its sample count.The models use AdamW, a learning rate of 10^-5, and early stopping with patience 3; batch sizes differ by model size.Source: Training · p. 5

Automatic Mining of Arguments

• Automatic argument extraction performs close to human performance, while stance detection is easier than argument identification in peer reviews, unlike the inverse pattern reported for other datasets.The comparison with other datasets on token-level results is about 10 percentage points higher for this dataset.Source: Automatic Mining of Arguments · p. 5, Table 3 · p. 6
• 22% of argument-detection sentences and 23% of stance-detection sentences contain tokens with both classes, contributing to a more evident token-versus-sentence performance gap.Sentence-level performance remains acceptable, but the difference from token-level performance is more evident in this dataset.Source: Automatic Mining of Arguments · p. 5
• Zero-shot transfer beats majority vote only for stance detection, while direct training on peer-review data outperforms transferred models; performance saturates at about 60% of the training set.Intermediate fine-tuning on another argument-mining dataset brings no significant improvement over direct fine-tuning on the peer-review data.Source: Automatic Mining of Arguments · p. 5, Figure 2 · p. 6

Generalization across conferences

• About two-point and three-point drops on argument and joint detection, respectively, accompany the GI’20 evaluation, and the authors conclude the model generalizes well across subdomains.The authors note that the drop may reflect the more challenging task, rather than worse generalization.Source: Generalization across conferences · p. 6, Table 5 · p. 6

Relevance for Decision-Making

• Selecting sentences by argumentativeness likelihood improves F1 over random selection when at least a third of review text is used; 50% yields almost full-review performance.The experiment uses sentence-level selection to make comparison with random sampling meaningful.Source: Relevance for Decision-Making · p. 7, Figure 3 · p. 7

• Peer reviews contain arguments identifiable with stances, differ from other AM domains, and editorial decision-making is empirically driven by argumentation.Source: Conclusion · p. 7
• Future work will rank arguments by strength and analyze how single arguments influence the final decision.The authors describe argument-strength ranking as a useful feature for potential application.Source: Conclusion · p. 7
• Identifying similar arguments across reviews of the same paper could also help the editorial team.Source: Conclusion · p. 7