Source-linked AI summary
Argument Mining for Understanding Peer Reviews
Xinyu Hua, Mitko Nikolov, Nikhil Badugu, Lu Wang
TL;DR
Peer-review is important, but reviews’ content and structure have received little analysis. This paper collects and annotates reviews for proposition segmentation and classification, then finds model challenges and venue variation in proposition use.
Problem
Reviews matter to scientific discovery, but little work has analyzed their content and structure or evaluated their quality.
Source: §1 · p. 1Method
The authors collect peer reviews, annotate propositions and their types, and apply argument-mining models to segment and classify them.
Source: §1 · pp. 1–2, §2 · p. 2Results
Proposition usage varies across venues, while segmentation models perform worse on AMPERE, where 25% of sentences contain multiple propositions versus 8% in essays.
Source: §1 · p. 2Takeaways & Limitations
The benchmark exposes challenges for argument mining in peer reviews and motivates new solutions for segmentation and classification.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Peer-review plays a critical role in the scientific writing and publication ecosystem. To assess the efficiency and efficacy of the reviewing process, one essential element is to understand and evaluate the reviews themselves. In this work, we study the content and structure of peer reviews under the argument mining framework, through automatically detecting (1) argumentative propositions put forward by reviewers, and (2) their types (e.g., evaluating the work or making suggestions for improvement). We first collect 14.2K reviews from major machine learning and natural language processing venues. 400 reviews are annotated with 10,386 propositions and corresponding types of Evaluation, Request, Fact, Reference, or Quote. We then train state-of-the-art proposition segmentation and classification models on the data to evaluate their utilities and identify new challenges for this new domain, motivating future directions for argument mining. Further experiments show that proposition usage varies across venues in amount, type, and topic.
1 Introduction
Peer reviews are important to scientific discovery, yet their content and structure have received little analysis. This work applies argument mining to identify review propositions and their types, then examines model performance and proposition-use patterns.
1 Introduction
- The approach segments elementary argumentative propositions and classifies them by type to help characterize review composition.The motivation is that reviews can contain evaluation and concrete suggestions, and their propositions convey reviewers’ interpretation of research.Source: §1 · pp. 1–2
- 14.2K reviews were collected, and 400 were annotated with 10,386 propositions labeled as Evaluation, Request, Fact, Reference, Quote, or Non-Argument.Source: §1 · p. 2
- 25% of review sentences contain multiple propositions, versus 8% in essays, and existing segmentation models perform worse on AMPERE, mainly because review argument structure differs.The essay comparison is reported by Stab and Gurevych (2017).Source: §1 · p. 2
- Proposition usage varies across venues: ACL reviews tend to have more propositions, especially requests, but fewer facts than ML-venue reviews.Reviews with strong reject or accept ratings tend to be shorter and make fewer requests; salient vocabulary also differs by venue.Source: §1 · p. 2
2 AMPERE Dataset
AMPERE provides annotated peer-review propositions for argument-mining research. Its annotation process achieved strong segmentation agreement, while evaluation and fact were the most frequent proposition types.
2 AMPERE Dataset
- The collected corpus contains 14,202 reviews: 4,057 from ICLR, 718 from UAI, 275 from ACL, and 9,152 from NeurIPS.All venues except NeurIPS have paper rating scores attached to reviews.Source: §2 · p. 2
- Annotators segmented propositions on sentence or subsentence level by discourse function and topic, then assigned one of five types plus a non-argument category.The 400 annotated reviews were sampled from ICLR 2018; two annotators labeled them and a third resolved disagreements.Source: §2 · p. 2
- Segmentation agreement was Cohen’s κ = 0.93; classification agreement was Krippendorff’s αU = 0.61, and agreement on exactly matched segments was Cohen’s κ = 0.64.Source: §2 · p. 2
- Evaluation (38.3%) and fact (36.5%) are the most frequent proposition types in AMPERE.Source: §2 · p. 3, Table 3 · p. 3
3 Experiments with Existing Models
The experiments benchmark existing models on AMPERE for proposition segmentation and classification. BiLSTM-CRF leads segmentation F1, while joint models perform best across classification categories when gold segmentation is unavailable.
3 Experiments with Existing Models
- The benchmark uses 320 reviews for training and 80 for testing, with five-fold cross-validation on the training set for hyperparameter tuning.Formulas, variables, URLs, and formatted citations are replaced with special tokens to improve tokenization accuracy.Source: §3 · p. 3
3.1 Task I: Proposition Segmentation
- Segmentation baselines range from sentence-level units to discourse-connective and RST-parser segmentation, alongside CRF and BiLSTM-CRF sequence taggers.Joint variants append proposition types to BIO tags to model segmentation and classification together.Source: §3.1 · p. 3
- BiLSTM-CRF outperforms the other segmentation methods in F1, though review performance is below the essay CRF F1 of 86.7.The essay comparison is attributed mainly to essays’ better structure and frequent discourse connectives.Source: §3.1 · p. 3, Table 4 · p. 3
3.2 Task II: Proposition Classification
- Classification models include majority and lexicon baselines, supervised SVM and CNN models, and the joint CRF and BiLSTM-CRF models.Classification is evaluated with given proposition segments, predicted or gold-standard.Source: §3.2 · pp. 3–4
- Joint models yield the best classification F1 across all categories when gold-standard segmentation is unavailable.CNN performs better for evaluation and fact, the types with significantly more training samples.Source: Table 5 · p. 3, §3.2 · p. 4
4 Proposition Analysis by Venues
The analysis finds that proposition counts and type usage vary across venues and ratings. It also identifies recurring transitions between proposition types and venue-specific differences in salient words.
Proposition Usage by Venue and Rating
- ACL and NeurIPS reviews contain significantly more propositions than ICLR and UAI reviews, while reviews with ratings 1 and 5 tend to contain fewer propositions.Figure 2 reports average propositions per review grouped by venue and rating; UAI–ICLR and ACL–NeurIPS are the exceptions to significant venue differences.Source: §4 · p. 4, Figure 2 · p. 4
- ACL reviews contain more requests but fewer facts than other venues, and 94.6% include at least one request versus 81.5% for ICLR and 84.7% for UAI.Highest-rated reviews tend to use fewer evaluation and reference propositions; ratings 3–4 contain more requests, rating group 4 has a sharp quote decrease, and group 5 a non-argument surge.Source: §4 · p. 4, Figure 3 · p. 4, Figure 4 · p. 4
Proposition Structure
- Propositions of the same type often occur consecutively, while a quote is more likely to be followed by an evaluation.The transition matrix is used to show argument structure in AMPERE.Source: §4 · p. 4, Figure 5 · p. 5
Proposition Type and Content
- Across venues, evaluation focuses on clarity and contribution; ICLR discusses “network” more, NeurIPS often mentions equations, and ACL frequently requests “examples”.The salient words are identified by log-likelihood ratio tests; venue-unique frequent words are shown in Table 6.Source: §4 · p. 5, Table 6 · p. 5
5 Related Work
Prior work has assessed review quality through author feedback, expert judgments, and shallow linguistic features. This paper positions argument usage as a way to study review content and structure in a domain where existing argument-mining models face new challenges.
5 Related Work
- Author feedback such as satisfaction and helpfulness is subjective and often influenced by acceptance decisions, while expert or editor evaluation requires substantial work and field knowledge.Prior work also studied sentiment words as shallow linguistic features in peer reviews.Source: §5 · p. 5
- The work claims to be the first to understand peer-review content and structure via argument usage.Source: §5 · p. 5
- Most argument-mining research has focused on social media, online debates, and student essays; peer reviews introduce a new domain and challenges for existing models.The paper describes its approach as studying peer-review content and structure through argument usage.Source: §5 · p. 5
6 Conclusion
The paper studies peer-review content and structure through argument mining, benchmarks segmentation and classification models, and analyzes proposition usage across venues. Its annotation appendix describes how the 400-review AMPERE sample was selected and how revision histories were handled.
6 Conclusion
- The study collects and annotates peer reviews with propositions and types, benchmarks argument-mining models, and uses classifiers to analyze proposition usage across ML and NLP venues.The conclusion highlights patterns in proposition types and content.Source: §6 · p. 5
A Annotation Details
- The 400-review AMPERE sample has review-length and rating distributions consistent with the full ICLR 2018 dataset.The appendix compares the sample and full dataset distributions in Tables 7 and 8.Source: A Annotation Details · p. 8, Table 7 · p. 8, Table 8 · p. 8
- The annotation subset intentionally includes revised and unrevised reviews at a 3:1 ratio, rather than the full dataset’s 9:1 ratio; only initial versions are considered.Revision history is noted as a possible proxy for opinion change and review quality in future work.Source: A Annotation Details · p. 8
B.1 Data Preprocessing
Reviews are tokenized and split into sentences, with special tokens substituted for equations, URLs, and citations or references.
B.1 Data Preprocessing
- The preprocessing replaces equations, URLs, and citations or references with special tokens after tokenizing reviews and splitting them into sentences.Across 400 reviews, the authors identified 302 variables, 125 equations, 62 URL links, and 97 citations.Source: §B.1 · p. 8
B.2 Training Details
The authors use cross-validation for hyperparameter selection and configure separate baseline and neural models for segmentation and joint prediction.
B.2 Training Details
- Five-fold cross-validation on the training set selects hyperparameters for every model except CNN.Source: §B.2 · p. 8
- For CRF, the selected regularizers are C1 = 0.0 and C2 = 1.0 for segmentation, and C1 = 1.0 and C2 = 0.01 for joint prediction.Source: §B.2 · p. 8
- The selected BiLSTM-CRF uses two 100-dimensional hidden-state layers with 0.5 dropout and Komninos and Manandhar embeddings.Those embeddings outperformed the tested GloVe alternatives.Source: §B.2 · p. 9
- The SVM is linear and uses Coordinate Descent, with its group Lasso coefficient set to 0.001 by cross-validation.Source: §B.2 · p. 9
- The CNN-non-static configuration uses filter windows of 3, 4, and 5, 128 feature maps each, 0.5 dropout, and 300-dimensional word2vec embeddings.Source: §B.2 · p. 9
C Further Analysis
Review length differs across venues and ratings, while proposition-type transitions generally favor sequences of the same type, with a notable NeurIPS anomaly.
C Further Analysis
- ACL and NeurIPS reviews are significantly longer than UAI and ICLR reviews (p < 10−6), while reviews with extreme ratings tend to be shorter.Source: C Further Analysis · p. 9
- Propositions are more likely to be followed by the same type, but NeurIPS shows an unusually prominent REFERENCE-to-NON-ARG transition linked to formatted headers.Headers such as “For detailed reviewing guidelines, see <URL>” may be predicted as REFERENCE before “Comments to the author” is predicted as NON-ARG.Source: C Further Analysis · pp. 9–10, Figure 7 · p. 10