Source-linked AI summary
Aspect-based Sentiment Analysis of Scientific Reviews
Souvic Chakraborty, Pawan Goyal, Animesh Mukherjee
TL;DR
Peer reviews contain information relevant to paper decisions, but public review datasets have been scarce. This study annotates ICLR reviews for aspects and sentiment, then analyzes how those signals relate to decisions and reviewer–chair agreement. Aspect-sentiment distributions differ between accepted and rejected papers, and recommendation alignment with the chair is associated with consistency between scores and aspect sentiment.
Problem
Public peer-review datasets were scarce, limiting studies of peer-review problems and potential feedback improvements.
Source: §1 · p. 1Method
The authors use active learning to build aspect-sentiment annotations for ICLR reviews, train a classifier to label the remaining corpus, and analyze review signals and disagreement.
Source: §1 · pp. 1–2, §6.1 · p. 7Results
Accepted and rejected papers have different aspect-sentiment polarities, and chair disagreement is greater when recommendation scores correlate less with aspect sentiments.
Source: §1 · p. 2Takeaways & Limitations
Clarity, empirical/theoretical soundness, and impact sentiments are identified as strongly determining a review’s intended recommendation.
Source: §1 · p. 2The analysis assumes reviews follow a well-defined aspect structure and that their sentiments reflect paper qualities.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Scientific papers are complex and understanding the usefulness of these papers requires prior knowledge. Peer reviews are comments on a paper provided by designated experts on that field and hold a substantial amount of information, not only for the editors and chairs to make the final decision, but also to judge the potential impact of the paper. In this paper, we propose to use aspect-based sentiment analysis of scientific reviews to be able to extract useful information, which correlates well with the accept/reject decision. While working on a dataset of close to 8k reviews from ICLR, one of the top conferences in the field of machine learning, we use an active learning framework to build a training dataset for aspect prediction, which is further used to obtain the aspects and sentiments for the entire dataset. We show that the distribution of aspect-based sentiments obtained from a review is significantly different for accepted and rejected papers. We use the aspect sentiments from these reviews to make an intriguing observation, certain aspects present in a paper and discussed in the review strongly determine the final recommendation. As a second objective, we quantify the extent of disagreement among the reviewers refereeing a paper. We also investigate the extent of disagreement between the reviewers and the chair and find that the inter-reviewer disagreement may have a link to the disagreement with the chair. One of the most interesting observations from this study is that reviews, where the reviewer score and the aspect sentiments extracted from the review text written by the reviewer are consistent, are also more likely to be concurrent with the chair's decision.
1 INTRODUCTION
The paper studies whether aspect-level sentiment analysis can extract useful signals from peer reviews, a time-consuming and imperfect process with limited public datasets. It applies this analysis to ICLR reviews to examine accept/reject differences and reviewer–chair disagreement.
1 INTRODUCTION
- Public peer-review datasets were scarce, limiting research on peer-review problems and possible feedback improvements.The paper cites a 2015 estimate of approximately 63.4 million researcher hours spent on peer reviews.Source: §1 · p. 1
- The study analyzes public ICLR reviews, including reviews for accepted and rejected papers, to address that data need.The dataset also includes rebuttal comments.Source: §1 · p. 1
- The approach assumes reviews follow a well-defined aspect structure and express sentiments that reflect paper qualities.The authors postulate that reviewers discuss paper pros and cons across aspects.Source: §1 · p. 1
- Accepted and rejected papers differ in aspect-sentiment polarity, and clarity, soundness, and impact sentiments are identified as strongly determining intended recommendations.The paper also reports that reviewers’ recommendation scores align more with the chair when those scores correlate with the review’s aspect sentiments.Source: §1 · p. 2
- The authors manually annotated review sentences for aspects and associated sentiments, then trained a supervised algorithm to label the rest of the corpus.Active learning helped select sentences for annotation; the study adopted eight aspects from ACL reviewing guidelines.Source: §1 · pp. 1–2
2 RELATED WORK
Related work covers sentiment-analysis methods and studies of peer-review data. The authors position their contribution as an aspect-sentiment annotated peer-review dataset and analysis using openly available reviews.
2 RELATED WORK
- The review divides related work into sentiment-analysis techniques and studies of peer-review text.Source: §2 · p. 2
2.1 Sentiment prediction
- Prior sentiment work includes aspect-level prediction, but the authors note that the specific peer-review task lacked an aspect-sentiment annotated dataset.They also cite evidence that sentiment classifiers may not generalize well across domains.Source: §2.1 · p. 2
2.2 Analysis of peer review dataset
- Before 2017, peer-review studies mostly used private datasets, while later work analyzed openly available reviews and related tasks.Prior studies examined reviewer characteristics, bias, recommendation prediction, and rebuttals.Source: §2.2 · p. 2
3 DATASET
The study uses ICLR review data from 2017–2019 and builds aspect-sentiment annotations through active learning and expert review. It then uses the resulting annotated data to train models for the larger corpus.
3.1 Description of the raw dataset
- The dataset includes 3,343 ICLR papers from 2017–2019; earlier data were excluded because they were sparse, noisy, and difficult to separate from rebuttals.The authors recollected the data to avoid mixing rebuttals with reviews in the source framework.Source: §3.1 · p. 3, §3 · p. 3
3.2 Data annotation process
- The annotation process uses batch-mode active learning because technical review sentences are often sentiment-free and labeled training data are limited.Source: §3.2 · p. 3
3.2.1 Avoiding cold start.
- To avoid a cold start, the authors seed selection with 50 sentiment-bearing sentences, expand via embedding similarity, then cluster sentences and sample across clusters.The resulting set contains 1,321 sentences after label selection.Source: §3.2.1 · p. 3
3.2.2 Collecting more data for annotation.
- For a later annotation batch, a random forest selects 15,000 candidates, with 70% from high-entropy predictions and 30% from the remainder, followed by cluster-balanced sampling.The process selects another 1,500 sentences for annotation.Source: §3.2.2 · p. 3
3.2.3 Annotation guidelines.
- Three ML/NLP researchers annotated 3,091 sentences for aspects and associated positive, negative, neutral, or absent sentiment labels.The first two annotators agreed on 63.1% of sentences; 502 were discarded as ambiguous, leaving 2,589 annotated sentences.Source: §3.2.3 · p. 3
- The annotation scheme adopts eight aspects from ACL reviews, allowing a sentence to have multiple aspects or none.Source: §3.2.3 · p. 3
4 ASPECT-LEVEL SENTIMENT CLASSIFICATION
The authors compare traditional and neural classifiers for aspect detection and sentiment classification. FFNN-uni is reported as the strongest overall approach, and its labels are used for full-dataset analysis.
4 ASPECT-LEVEL SENTIMENT CLASSIFICATION
- FFNN-uni outperforms the other tested classifiers for aspect detection, while FFNN-sci is a close competitor that takes substantially longer to train.The compared approaches include traditional classifiers and neural models using universal sentence encoder or SciBERT embeddings.Source: §4 · p. 4, Table 3 · p. 4
- For sentiment detection, FFNN-uni has lower macro-averaged F1-scores for Substance and Recommendation than for the other aspects.The authors suggest fewer training instances, particularly for neutral sentiment, as a possible reason.Source: §4 · p. 4, Table 4 · p. 4
- The authors report that the classifier performs better than a four-class majority classifier and captures useful information across the aspects.Source: §4 · p. 4
- The authors judge classifier fidelity sufficient to label the full dataset and use those labels for exploratory analysis.Source: §4 · p. 4
5 EXPLORATORY DATA ANALYSIS
The analysis aggregates positive and negative aspect-sentiment scores across each paper’s reviews, then examines how those sentiments relate to acceptance, reviewer recommendations, and chair decisions.
5.1 Aspect sentiment distribution
- Accepted papers have higher positive sentiment across the listed aspects, while rejected papers have more negative sentiment in all listed aspects except originality.The accepted/rejected sentiment distributions are normalized separately for each aspect, so heights across aspects are not comparable.Source: §5.1 · pp. 4–5, Figure 1 · p. 6
- Review statistics for accepted and rejected papers have similar distributions in sentence count, average sentence length, and number of unique words.The dataset contains 1,035 accepted and 1,646 rejected papers from 2017–2019.Source: §5.1 · p. 4, Table 7 · p. 5
5.2 Individual recommendation scores vs the final recommendation decision
- Individual reviewers disagree with the chair’s final recommendation in 24.9% of cases, falling to 14.8% when disagreement is measured by majority vote.Recommendation-score correlations with the chair’s decision never exceed 0.78.Source: §5.2 · p. 5, Table 8 · p. 7
5.3 Why review texts are important when scores are already present?
- Reviewers whose aspect-sentiment correlations with their decisions are higher in most aspects are more likely to agree with the chair; chair-agreeing reviewers also report higher confidence.Mean confidence was 3.88 for reviewers agreeing with the chair and 3.67 for those who did not; the difference had t = 6.83 and p = 1.07 x 10^-11.Source: §5.3 · p. 6, Figure 3 · p. 6, Figure 4 · p. 7
6 DISAGREEMENT AMONG THE REVIEWERS
The section defines disagreement among reviewers and examines how it varies with average review scores, aspect-level disagreement, and chair decisions.
6.1 Definition
- Aggregate disagreement is the standard deviation of recommendation scores normalized by the maximum possible for the number of reviewers.For three reviewers, the maximum-disagreement examples are scores 1, 1, 10 or 10, 10, 1.Source: §6.1 · p. 7
6.2 Disagreement distribution
- Average aggregate disagreement is non-decreasing through average review score 5 and strictly decreases thereafter.The authors suggest boundary scores or greater room to disagree near a mean score of 5 as possible explanations.Source: §6.2 · p. 7, Figure 5 · p. 7
6.3 Disagreement: Aspect level vs aggregate
- Higher aspect-level disagreement corresponds on average to higher aggregate disagreement across all aspects.Papers are grouped into low, mid, and high aspect-disagreement bins before their average aggregate disagreement is compared.Source: §6.3 · p. 7
6.4 When will the chair have to intervene?
- 0.229 aggregate disagreement was about 30% higher when the chair’s decision differed from the reviewers’ mean than when it matched (0.175).The authors report that reviewer disagreement is larger in cases needing chair intervention.Source: §6.4 · p. 8
7 DETERMINING THE IMPORTANCE OF THE ASPECTS IN RECOMMENDATION
The paper evaluates how review-text aspect sentiments relate to recommendation and predict reviewers’ intended decisions. Clarity, empirical/theoretical soundness, and impact of ideas emerge as especially informative aspects.
7.1 The correlation study
- Clarity, empirical/theoretical soundness, and impact of ideas have higher correlations with final recommendation than the other aspects.The study correlates aspect-level sentiment values with the final recommendation.Source: §7.1 · p. 8, Table 9 · p. 8
7.2 Deep neural model
- 65.3% accuracy was achieved in 10-fold cross-validation using aspect-sentiment features to predict the intended recommendation in a review.Dropping empirical/theoretical soundness, impact of ideas, or clarity features most affected performance.Source: §7.2 · p. 9, Table 12 · p. 9
- The authors identify the most predictive aspects as potential guidance for authors on what to focus on when submitting papers.This conclusion follows their feature-importance analysis of aspect sentiments.Source: §7.2 · p. 9
8 CONCLUSION
The paper analyzes latent aspect sentiments in ICLR reviews from 2017–19, combining an annotated dataset and models with analyses of recommendations and disagreement.
8 CONCLUSION
- Around 2,500 review sentences were annotated for aspects and their associated sentiments using an active-learning framework.The collected ICLR review corpus includes metadata such as author names, rebuttal comments, and accept/reject/withdrawn information.Source: §8 · p. 9
- The authors report that certain aspects are more relevant to predicting final recommendations and may guide authors’ paper-submission focus.Their analyses also connect aspect-level disagreement with aggregate reviewer disagreement and report low score–text alignment for reviewers whose recommendations conflict with the chair.Source: §8 · p. 9
- The authors identify citation-count prediction, author-specific aspect sentiments, and temporal changes in author reviews as future research directions.Source: §8 · p. 9