NLPeer: A Unified Resource for the Computational Study of Peer Review

1 Introduction

• Existing peer-review NLP research lacks licensed reusable data, multi-domain coverage, and a unified data model, limiting reuse and leaving cross-domain transfer unclear.Existing datasets often use task-specific formats, making cross-dataset and cross-domain development and evaluation difficult.Source: §1 · pp. 1–2
• NLPEER gathers ethically and legally compliant data from NLP and computational linguistics, F1000 Research, and the openly licensed portion of PeerRead.The paper describes it as the most comprehensive clearly licensed open peer-review collection available to NLP at the time.Source: §1 · p. 2
• NLPEER includes papers, peer reviews, drafts, and revisions from diverse fields and reviewing systems spanning 2012–2022.The revisions also make the resource suitable for studying collaborative text work.Source: §1 · p. 2
• A unified data model preserves document structure and non-textual elements for cross-document analysis, and the paper evaluates three peer-review assistance tasks.The tasks are review score prediction, pragmatic labeling, and guided skimming for peer review.Source: §1 · p. 2
• Experiments indicate performance varies across venues and communities, identify synergies between review-structure approaches, and point toward using cross-document links for language-model benchmarking.The paper presents these experiments as analysis of the resource’s possibilities, alongside two novel NLP/CL review datasets.Source: §1 · p. 2

2.1 Peer Reviewing Terminology

• Reviewing systems vary in identity visibility, standards, publication types, and form structure; these differences shape the data produced.The paper distinguishes unstructured forms with one main text from structured forms with predefined sections.Source: §2.1 · p. 2

2.2 Existing Data

• Peer-review NLP studies have largely relied on a few OpenReview machine-learning conferences, whose materials lacked explicit licenses, while cross-domain applicability remains open.The cited F1000RD corpus also lacks NLP and computer-science data and blind-review data.Source: §2.2 · pp. 2–3

2.3 NLP Approaches to Aiding Peer Review

• NLPEER aims to supply diverse, clearly licensed source data and examines cross-domain score prediction, synergies between structured and free-form pragmatic labeling, and novel guided skimming.Guided skimming builds on cross-document modeling for peer review.Source: §2.3 · p. 3

3.1 Datasets

• NLPEER combines five datasets: two from prior work, two new NLP-domain datasets, and an updated F1000Research crawl.The collections include double-blind conference reviews and papers, open post-publication F1000 reviews, and donation-based COLING-20 and ARR-22 data.Source: §3.1 · pp. 3–4

3.2 Unification

• Each paper is represented by its submission and revised camera-ready version and linked to one or more reviews; papers are converted to intertextual graphs.The workflow extends an existing parser and combines it with GROBID; reviews use a standardized format accommodating structured and free-text reports and arbitrary score sets.Source: §3.2 · p. 4

3.3 Ethics, Licensing and Personal Data

• All datasets are distributed under open Creative Commons licenses and were collected through explicit consent or an open source-data license.F1000-22 retains reviewer and author identities, while other drafts and reviews are stripped of personal metadata; the included texts are English-only.Source: §3.3 · pp. 4–5

Documents and Texts

• NLPEER contains more than 4M peer-review tokens, including more than 400K from NLP/CL, and every paper has at least one review and revision.F1000-22 papers average 158 sentences (±90.3), compared with roughly 200 sentences and lower variance in NLP-based datasets.Source: §4 · p. 5

Scores and Acceptance

• 100% of ARR-22 papers and 34% of F1000-22 papers are accepted, but acceptance alone does not indicate review stance.The paper attributes this variation to differences in data-collection workflows.Source: §4 · p. 5
• Scores near the borderline to acceptance are well represented, with distributions otherwise skewed toward positive scores across datasets.NLP/CL conferences use 5- and 9-point scales, while F1000 Research uses a 3-point scale; Figure 3 plots normalized dataset score distributions with the borderline marked.Source: §4 · p. 5, Figure 3 · p. 5

Domain Structure

• NLP/CL review datasets have the highest vocabulary overlap (0.37–0.53); F1000-22 is closest to ARR-22 but about 0.25 lower.Figure 4 reports Jaccard overlap on review lemmas.Source: §4 · pp. 5–6, Figure 4 · p. 6

5.1 General Setup

• The experiments fine-tune general- and scientific-domain language models with task-specific heads, using 70/10/20 train-development-test splits and ten random-seed runs.The reported metrics are medians and standard deviations across runs.Source: §5.1 · p. 6

5.2 Review Score Prediction

• Review score prediction uses review text and the paper abstract to predict a score, evaluating regression error and interval-based F1 without modeling score semantics.The setup uses reviews of initial drafts and normalizes scores to each dataset’s scale.Source: §5.2 · pp. 6–7
• Neural score predictors outperform the mean-score baseline on ARR-22 and F1000-22, but are on par with or worse than it on ACL-17, CONLL-16, and COLING-20.The authors associate the weaker results with these datasets having the fewest samples; a 30% ARR-22 training subset yields 0.44 MRSE and 0.24 F1-macro, on par with the mean baseline.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Pragmatic Labeling

• Pragmatic labeling predicts a review sentence’s label; ARR sections are aligned to F1000RD classes for experiments across the two review systems.F1000Research supplies manually labeled free-form sentences, while ARR uses structured review sections aligned to Strength, Weakness, Request, and Neutral.Source: §5.3 · p. 7
• Cross-dataset pragmatic labeling substantially exceeds the in-dataset majority baseline, but remains below the in-distribution supervised model.The authors suggest structured review-form data can train free-text labeling models and reduce annotation costs.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Guided Skimming for Peer Review

• Guided skimming ranks paper paragraphs by relevance to critical reading, using explicit review-to-paper links as training signals and evaluating on unseen papers.The authors describe this as a simplification of actual peer-review skimming and a first step toward modeling reviewer focus.Source: §5.4 · p. 8
• SciBERT reaches MRR 0.41 ± 0.05 on ACL-17 and 0.34 ± 0.03 on ARR-22, beating random by 0.23 and 0.18, respectively.On ACL-17 at k ≈ 3, about 50% of relevant passages are retrieved, while about 21% of retrieved paragraphs are reviewer-linked.Source: §5.4 · pp. 8–9, Figure 6 · p. 9

• Review reports can provide supervision for summarization and for aligning arguments across reviews using aspect-score disagreements.Review summaries can ground a challenging document-summarization task; disagreements can act as weak supervision.Source: §6 · p. 9
• Automatic review generation could serve as a paper-writing aid or analytical tool, but using it to avoid reviewing effort or replace reviewers carries risks to academic quality assurance.The authors consider meaningful reviews of novel papers implausible for current state-of-the-art NLP models.Source: §6 · p. 9
• The authors urge future work using NLPeer to consider the real-world impact of developed models.Source: §6 · p. 9

• NLPeer is presented as a clearly licensed, large-scale reference corpus for studying NLP and peer review.Source: §7 · p. 9
• The resource supports cross-domain and cross-temporal NLP studies, including domain transfer and analysis of links among papers, revisions, and reviews.Source: §7 · p. 9
• For meta-science, NLPeer enables comparisons of reviewing practices across research communities and study of how researchers review and revise scientific texts.Source: §7 · p. 9

Limitations

• Author profiling violates the intended use of NLPeer, while full automation of review carries risks and dual uses.The authors urge careful reflection on risks and benefits of new tasks built on peer-review data.Source: Limitations · p. 10
• NLPeer covers only a fraction of peer review, is English-only, under-represents arts and humanities, and requires care when comparing communities’ practices and donation-based samples.The authors call for more data to support fully representative study.Source: Limitations · p. 10
• The task implementations demonstrate feasibility, but noisy ARR pragmatic labels, limited explicit-link recall, and absent extensive hyperparameter tuning constrain interpretation of the results.The results are not claims that any model, approach, or reviewing system is superior.Source: Limitations · p. 10
• High intrinsic task performance does not necessarily translate into extrinsic utility in real-world reviewing.Source: Limitations · p. 10
• Review assistance warrants targeted study: guided skimming could distract less experienced reviewers, and real-system writing assistance should have carefully designed guidelines and policies.The authors also identify the possibility of reviewers reading only suggested paragraphs.Source: Limitations · p. 11

A.1 Overview

• The overview augments some datasets with camera-ready versions matched through titles and abstracts, parses papers into a shared structure, and screens reviews for privacy-sensitive sentences.For non-exact title-and-abstract matches, the authors checked plausible alignments and omitted uncertain matches; parsing errors remain possible.Source: §A.1 · pp. 13–14

A.2 COLING-2020 Collection

• COLING-20 data came from post-review author and reviewer donations; only materials with both parties’ agreement were included to avoid leaking confidential research ideas.Source: §A.2 · p. 14

A.3 ARR-2022 Collection

• The ARR-22 collection covers reviews of final drafts for papers accepted at ACL 2022 and NAACL 2022, not the papers’ earlier revisions.Some included reviews may themselves be revisions or refer to earlier versions.Source: §A.3 · p. 14

A.5 Extended Datasets Analysis

• Abstract vocabulary overlap is overall lower than review-text overlap, indicating wider cross-domain shared vocabulary in reviews and a more specialized register in papers.The abstract comparison uses Jaccard overlap on the top 10% of lemmas.Source: §A.5 · p. 15

B.1 Fine-tuning Setup

• The experiments target task difficulty and domain differences, using established language models with a base set of near-recommended hyperparameters rather than extensive tuning.Pilot training was relatively unstable, making extensive random hyperparameter search infeasible within the study’s scope.Source: §B.1 · p. 15
• RoBERTa, BioBERT, and SciBERT were fine-tuned for up to 20 epochs with 10 random-seed runs, validation-based model selection, and early stopping.Training stopped if no improvement was observed after at most eight epochs.Source: §B.1 · p. 15
• Score prediction splits preserve the distribution of reviews per paper, while guided-skimming splits preserve relevant-paragraph distributions per paper.The study uses repeated binary stratified splits.Source: §B.1 · p. 16

B.2 Pragmatic Labeling

• Pragmatic labeling maps ARR review-form sections and F1000RD labels to shared labels, but ARR sentence labels may be noisy because forms group text by section rather than enforcing sentence-level consistency.The authors present this as a scalable supervision approach whose label quality warrants further analysis.Source: §B.2 · p. 16

B.3 Guided Skimming for Peer Review

• Guided skimming is framed as ranking paper paragraphs by relevance to review writing, using explicit review-to-paper links as relevance signals.Source: §B.3 · p. 16
• Explicit-link labels omit implicit discussions and have comparatively low recall, so they represent only a subset of regions discussed in reviews.The extraction uses anchors such as structural mentions and maps links to specific paragraphs, including quotes and line references.Source: §B.3 · p. 16
• ARR-22 has 1,100 linked paragraphs among 12,636 (roughly 9%); ACL-17 has 600 among 4,642 (13%), with line mentions accounting for 64% and 57% of relevant paragraphs, respectively.Source: §B.3 · p. 16
• Linked paragraphs tend to be longer, potentially because imperfect PDF parsing creates size variation and longer passages offer more content to discuss.The authors identify length-normalized paragraphs as a direction for further work and examine paragraph length as a spurious feature.Source: §B.3 · p. 17, Figure 8 · p. 17, Figure 9 · p. 19
• Linked paragraphs span document hierarchy levels: roughly half are section-level in ARR-22 and about two thirds in ACL-17, suggesting no substantial level bias.Source: §B.3 · p. 17
• In both datasets, introductions receive the most explicit links when excluding non-canonical sections; many section types are covered, and ARR-22 references results slightly more often than ACL-17.Source: §B.3 · p. 17

C.1 Review Score Prediction

• The mean-score baseline has the highest KL divergence from human score distributions, and even the best models show notably different rating behavior across datasets.The model with best MRSE does not consistently have one of the lowest KL-divergence scores.Source: §C.1 · p. 19

Error Analysis

• The error analysis reports confusion matrices for the best-performing RoBERTa model at its median-performing run.The model is identified as best-performing among the LLMs in the reported experiments.Source: §C.2 · p. 19

Within-dataset Errors

• On F1000-RD, neutral is the hardest class: strengths, weaknesses, and requests are most commonly confused with neutral and vice versa.The neutral label subsumes finer F1000-RD labels such as summary or structure, which may contribute to difficulty distinguishing classes.Source: §C.2 · pp. 19–20, Table 8 · p. 21
• On ARR-22, neutral is most often confused with other classes, and weaknesses and requests are commonly confused, matching the pair with highest human disagreement.The authors link neutral confusion partly to unstrict forms and factual sentences in strengths and weaknesses sections.Source: §C.2 · p. 20, Table 9 · p. 21

Out-of-dataset Errors

• Across transfers, weaknesses and requests are frequently confused; F1000-RD-to-ARR-22 transfer often labels weakness sentences as requests.The latter pattern supports the authors’ hypothesis that many sentences in ARR weakness sections are requests.Source: §C.2 · pp. 20–21
• Cross-domain transfer reaches a promising range, with neutral the hardest class to predict.The authors suggest further work on few-shot and cross-domain transfer.Source: §C.2 · p. 21

Ranking Measures

• The evaluation uses MRR, AUROC, and at-k measures; a character-count baseline ranks paragraphs by length because linked and unlinked paragraphs may differ in length.The LLM inputs are truncated to 512 tokens, making sequence-length exploitation less likely; full-text or full-paragraph methods might be more vulnerable.Source: §C.3 · p. 21
• SciBERT leads ACL-17 on MRR, AUROC, and Precision@3 but is nearly level with the length baseline overall; on ARR-22, all models beat random, yet length leads overall despite top-rank gains.Figures 12 and 13 show precision and recall by rank for ACL-17 and ARR-22, respectively.Source: §C.3 · p. 21, Figure 12 · p. 22, Figure 13 · p. 22

Conclusion

• Paragraph length may be a spurious decision cue; the authors suggest structural and contextual information, listwise losses, and length normalization as future directions.Length normalization across segmentation techniques is proposed to reduce the risk of learning paragraph length as a spurious pattern.Source: §C.3 · p. 22