Source-linked AI summary
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz
TL;DR
PeerRead addresses the limited public availability of peer reviews by compiling a large dataset with papers, decisions, and textual reviews. The authors construct it from opt-in, public, and arXiv-linked sources, then demonstrate two prediction tasks. Acceptance models reduce error by 6–21% versus a majority reject-all baseline, while the paper is intended to motivate stronger models and further analysis.
Problem
Quantitative peer-review studies were limited because few people had access to reviews from a given venue.
Method
The authors compile PeerRead from opt-in reviews, publicly available reviews, and arXiv submissions matched to conference publications, then define two prediction tasks.
Results
6–21% error reduction over the majority reject-all baseline is achieved for accept/reject prediction across four PeerRead sections.
Takeaways & Limitations
PeerRead provides a public basis for analyzing peer reviews and developing stronger acceptance and aspect-score prediction models.
Takeaways & Limitations
The paper’s goal is to demonstrate potential uses of PeerRead rather than develop the best-performing models.
Abstract
from arXiv · showhide
Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.
1 Introduction
PeerRead addresses limited public access to peer reviews by assembling a research dataset and demonstrating analyses and prediction tasks enabled by it.
- More than a quarter of NIPS 2014 papers received conflicting accept/reject decisions from two reviewer committees.
- PeerRead lowers the barrier to quantitative peer-review research by providing the first public dataset of peer reviews.
- The dataset combines opt-in submissions and reviews, crawled public reviews with aspect annotations, and arXiv submissions matched to later conference publications.
- PeerRead supports analyses of recommendation and aspect-score correlations, presentation recommendations, review biases, and educational examples of peer reviews.
- The paper introduces acceptance prediction and review-aspect score prediction as two NLP tasks based on PeerRead.
- Acceptance prediction achieves 6–21% error reduction over the majority reject-all baseline across four PeerRead sections.
2 Peer-Review Dataset (PeerRead)
PeerRead v1 compiles peer reviews, papers, decisions, and annotated aspect scores from conference-management systems, public review platforms, and arXiv matching.
- Opted-in reviews from CoNLL 2016 and ACL 2017 require author paper-draft consent and at least one reviewer’s consent for anonymous reviews.
- Public review sources include 9,152 NIPS reviews for 2,420 accepted papers and 1,304 official ICLR reviews for 427 papers.
- ArXiv papers are labeled accepted or probably-rejected by matching titles and authors to target-venue publications and applying submission-timing criteria.
- The arXiv construction yields 2,891 accepted papers and 8,887 probably-rejected papers, with manual inspection identifying one false negative among 12 cases.
- PeerRead v1 organizes data into five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013–2017, and arXiv 2007–2017.
- The dataset includes extracted paper text, standard train/development/test splits, and 1.3K reviews annotated with aspect scores.
- Annotation quality checks found 77% consistent annotations, 2% inconsistent annotations, and 21% differing on whether an aspect was discussed.
3 Data-Driven Analysis of Peer Reviews
PeerRead supports data-driven analysis of review scores, presentation recommendations, and cross-venue review characteristics. The analysis finds differing relationships among review aspects and highlights sampling and annotation biases that complicate comparisons.
- Overall recommendation vs. aspect scores: The strongest correlations with overall recommendation are substance and clarity, while soundness/correctness and originality are least correlated.These correlations were measured in the ACL 2017 section of PeerRead.
- Oral vs. poster: Reviews recommending oral presentations have a 0.9 higher average overall recommendation score than reviews recommending posters.The comparison uses mean review scores by presentation format in ACL 2017.
- ACL 2017 vs. ICLR 2017: ACL 2017 review comments are about 50% longer than ICLR 2017 comments, raising questions about cross-venue review quality.The comparison concerns reviews in the ACL 2017 and ICLR 2017 sections of PeerRead.
- ACL 2017 vs. ICLR 2017: The ACL–ICLR review-length comparison may be positively biased because ACL reviews were opted in whereas ICLR included all official reviews.Different inclusion procedures likely affect the quality profile of the reviews represented in each venue.
- ACL 2017 vs. ICLR 2017: Lower ICLR 2017 appropriateness scores likely reflect annotation bias because reviewers often mention appropriateness only when a paper is inappropriate.This selective discussion can produce a strong negative bias for that category in the ICLR dataset.
4 NLP Tasks
The paper introduces two NLP tasks built on PeerRead: predicting paper acceptance and predicting numerical review-aspect scores. Simple models outperform majority or mean baselines, while aspect prediction is strongest for higher-variance aspects.
- Task overview: The two tasks predict paper acceptance and numerical scores for seven review aspects from PeerRead data.The aspects are impact, substance, appropriateness, comparison, soundness, originality and clarity.
- 4.1 Paper Acceptance Classification: Acceptance classification estimates P(accept=True | paper) using interpretable hand-engineered features and standard classifiers.Features include paper length, lexical properties, citation information and abstract terms; experiments use ICLR 2017 and arXiv sections.
- 4.1 Paper Acceptance Classification: Up to 22% error reduction over the majority classifier was achieved, with the best model outperforming the majority classifier in every case.The authors note that simple features may correlate with strong papers or influence reviewer judgments.
- 4.2 Review Aspect Score Prediction: Aspect prediction uses regression models conditioned on paper text, review text, or both, compared with a training-set mean baseline.CNN, LSTM and DAN architectures use mean squared error; the task uses ACL 2017 and ICLR 2017 sections because they include aspect scores.
- 4.2 Review Aspect Score Prediction: Higher mean-baseline RMSE indicates greater score variance, and Paper;Review outperforms the Mean baseline on all four highest-variance aspects.Review alone outperforms Mean on three of four, while all models slightly improve over Mean on average.
- Scope: The experiments demonstrate potential uses of PeerRead rather than aiming to develop the best-performing models.This stated goal explains the simplicity of the models used.
5 Related Work
Prior work examined peer-review consistency, bias, quality, and related review infrastructure, but most reviews remained inaccessible. PeerRead addresses this gap with a large-scale public dataset for reproducible research and further study.
- Publons and Crossref supported peer-review aggregation or metadata registration, yet most reviews were not publicly available.PeerRead was organized to support research use, experiment replication, and fair comparisons.
- Existing studies examined review consistency, bias, author response, courtesy, quality, and relationships between review scores and later citation impact.Examples include disagreement between NIPS 2014 reviewer committees and low correlations between review scores and future citations.
- PeerRead provides a large-scale dataset enabling the wider scientific community to study peer-review properties and potentially improve the review model.The dataset complements initiatives focused mainly on author response and general review quality.
- The paper positions acceptance prediction and review-aspect prediction as potentially useful tools for writing, reviewing, and deciding paper acceptance.These tasks are presented alongside existing tools for reviewer assignment, grammar checking, and finding relevant uncited articles.
6 Conclusion
The conclusion presents PeerRead as a public dataset containing papers and reviews, uses it to define two prediction tasks, and reports correlations and acceptance-related patterns. It frames the dataset as a basis for future analyses and improved models.
- 14.7K papers and 10.7K reviews make PeerRead the first publicly available peer-review dataset for research purposes.
- PeerRead defines acceptance prediction from textual features and review-aspect score prediction from paper and review contents.
- The dataset analysis finds a high correlation between overall recommendation and recommending an oral presentation.
- Paper properties such as having an appendix are correlated with higher acceptance rate, while the authors call for stronger models and further demographic-bias studies.
A Acceptance Classification Features
The acceptance classification model uses a set of features whose learned coefficients are visualized for ICLR and arXiv.
- Figure 2 visualizes learned coefficient values for coarse features used in paper acceptance classification on ICLR and arXiv.
A.1 Hyperparameters
The acceptance-classification experiments use mostly default sklearn hyperparameters, with specified settings for several model families; Figure 2 shows coefficients for coarse features in ICLR and arXiv.
- A.1 Hyperparameters: Unless stated otherwise, the experiments use sklearn default hyperparameters.
- A.1 Hyperparameters: Decision trees and random forests use maximum depth=5, while random forests also use max_features=1.
- A.1 Hyperparameters: The MLP uses α = 1, k-nearest neighbors uses k = 3, and logistic regression considers both l1 and l2 penalties.
- Figure 2: Figure 2 reports coefficient values for coarse acceptance-classification features for ICLR and arXiv.
B Reviewer Instructions
The section introduces ACL 2016 reviewer instructions for assigning aspect scores and presents the feature inventory used for acceptance classification.
- ACL 2016 reviewers were instructed on how to assign aspect scores to reviewed papers.
- The acceptance classification task used coarse and lexical features, including features extracted from science-parse.The table notes that “sp” refers to features extracted from science-parse.
APPROPRIATENESS (1-5)
The reviewer instructions define multiple 1–5 dimensions for evaluating papers, including fit, correctness, comparison, substance, impact, clarity, originality, recommendation, and reviewer confidence. They also specify qualitative anchors for interpreting these scores and the final accept/reject decision.
- Appropriateness: Reviewers assess whether a paper fits ACL 2016 while considering the goal of broadening represented research areas.
- Other evaluation dimensions: Clarity, originality, empirical correctness, recommendation, and reviewer confidence capture readability, novelty, evidential trustworthiness, acceptance preference, and evaluation certainty.The final recommendation ranges from fighting for acceptance to fighting for rejection, while confidence reflects familiarity and checking of important points.
- Theoretical soundness / correctness: Theoretical soundness evaluates whether the mathematical approach is sound and whether the arguments are cogent and well-supported.Scores range from 1 to 5, with 5 indicating an apt approach and convincingly supported claims.
- Meaningful comparison: Meaningful comparison measures positioning against existing literature, reference adequacy, and comparison with prior approaches for empirical work.A score of 5 denotes a precise and complete comparison with related work.
- Substance: Substance measures whether the paper contains enough ideas or results for a conference publication, separately from quality in other categories.Scores distinguish work that goes the extra mile from work that is thin or remains in progress.
- Impact: Impact dimensions evaluate the significance of the ideas or results, the value of accompanying software, and the value or unmet need addressed by an accompanying dataset.The software and dataset scales distinguish enabling, useful, potentially useful, documentary, and unusable or absent resources.