Source-linked AI summary
FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes
Prabhjot Singh, Somnath Luitel, Manmeet Singh, Josh Durkee
TL;DR
Prior peer-review datasets are limited to Computer Science and Machine Learning and omit iterative author-reviewer validation and editorial-outcome ground truth. FIRSTPASS constructs a 3,668-record, five-domain dataset of multi-round Nature Communications reviews and trains tasks for critique, updating, and outcome prediction. Fine-tuned Qwen2.5-7B-Instruct reaches 80.5% accuracy and F1-macro 78.2% on outcome prediction, while the release supports reproducible benchmarking.
Problem
Prior datasets are confined to Computer Science and Machine Learning, omit multi-round dialogue, and evaluate review style rather than alignment with editorial decisions.
Method
FIRSTPASS curates complete multi-round Nature Communications reviews across five domains, assigns round-based outcome labels, and organizes three tasks for generation, updating, and prediction.
Results
80.5% accuracy and F1-macro 78.2% were achieved by fine-tuned Qwen2.5-7B-Instruct on Task 3, outperforming Gemini-3.1-flashlite-preview zero-shot by 10.4 percentage points.
Takeaways & Limitations
FIRSTPASS provides outcome-grounded, multidisciplinary training and evaluation data for benchmarking AI scientific judgment across disciplines.
Takeaways & Limitations
The dataset is sourced from a single journal and covers accepted papers only, creating documented scope and bias risks.
Abstract
from arXiv · showhide
Scientific peer review datasets have trained AI systems exclusively on Computer Science and Machine Learning venues, producing models that critique ablation studies yet have never seen a biology reviewer demand contamination controls or a chemist question Nuclear Magnetic Resonance (NMR) spectral assignments. We introduce FIRSTPASS, the first large-scale peer review dataset built on complete multi-round editorial dialogues from a multidisciplinary high-impact journal. Curated from Nature Communications mandatory transparent peer review (instituted November 2022), FIRSTPASS comprises 3,668 records spanning five scientific domains (biology, chemistry, neuroscience, physics, and earth science), capturing the full iterative structure of scientific validation: initial referee reports, author point-by-point responses, and updated reviewer assessments. Each record carries an outcome label derived directly from editorial decisions (STANDARD for two-round review; EXTENDED for three or more rounds), providing ground truth absent in all prior corpora. An automated audit confirms 100% content integrity. Expert reviews average 2,155 words, substantially denser than conference venue reviews. All data, parsing pipelines, and evaluation scripts are released to enable reproducible benchmarking of AI scientific judgment across disciplines.
1. Introduction
FIRSTPASS addresses the domain and structural limits of prior peer-review datasets by combining five scientific disciplines with complete multi-round dialogues and editorial-outcome labels.
- Prior peer-review datasets draw exclusively from Computer Science and Machine Learning venues, limiting the scientific norms represented in training data.
- Existing systems treat peer review as a one-shot event, omitting author responses, new data, and updated reviewer positions during validation.
- Evaluation commonly rewards review-shaped text rather than agreement with changes editors actually demanded, leaving scientific judgment untested.
- FIRSTPASS comprises 3,668 multi-round dialogues from Nature Communications, spanning five disciplines and labeling outcomes from real editorial decisions.
2. Dataset Construction
FIRSTPASS is constructed from transparent, multidisciplinary Nature Communications reviews using completeness filters, round-based outcome labels, and audited structured records.
- Nature Communications provides mandatory transparent peer review, five-discipline coverage, CC BY 4.0 licensing, and reviews averaging 2,155 words.
- Papers published from January 2023 to December 2025 were collected through the Springer Nature OpenAccess API and parsed from article and review PDFs.
- Records were retained only when abstracts, introductions, methods, and results each exceeded 20 words and at least two complete review rounds were present.
- Outcome labels assign STANDARD to two rounds and EXTENDED to three or more, reflecting unresolved concerns without relying on decision letters.
- An automated audit found zero hollow files and 100% content integrity across 3,668 retained records, with domain-stratified paper-level 80/10/10 splits.
3. Statistics and Tasks
FIRSTPASS combines a three-task curriculum with multidisciplinary, multi-round review data and evaluates outcome prediction against real editorial labels.
- Statistics: At 2,155 words average, FIRSTPASS reviews are more than five times longer than ICLR reviews of ∼400 words and structurally multi-round.
- Three-task curriculum: Task 2 generates Round 2 reviews from paper content, initial reviews, and author responses, targeting dialogue understanding and unresolved methodological debt.
- Three-task curriculum: Task 3 predicts STANDARD or EXTENDED from the full dialogue, using real editorial decisions instead of stylistic proxies as ground truth.
- Results: 80.5% accuracy and F1-macro 78.2% were achieved by fine-tuned Qwen2.5-7B-Instruct, outperforming Gemini-3.1-flashlite-preview zero-shot by 10.4 percentage points.
4. Release and Governance
FIRSTPASS is released as an auditable, reproducible benchmark while documenting privacy protections and governance risks associated with its source and intended use.
- Release: The dataset, parsing pipeline, extraction prompts, audit scripts, evaluation suite, and fine-tuned weights are publicly available under inherited CC BY 4.0 access.
- Release: Versioned JSON records and a unified evaluation script support reproducible benchmarking and annual extension as new eligible papers appear.
- Privacy: Reviewer identities remain anonymized, and no personally identifiable information beyond the published record is included.
- Governance: The authors identify bias replication from a single journal source and misuse for synthetic review generation as responsible-use risks.
- Governance: The release includes a datasheet documenting accepted-papers-only scope, known limitations, and recommended mitigations.