Source-linked AI summary
More Than Mimicking Reviewers: Evaluating LLMs for Pre-Submission Peer Review
Pouya Parsa, Amin Rezaei
TL;DR
Peer-review feedback often arrives too late for meaningful revision, motivating an author-facing LLM system that anticipates concerns before submission. The system generates broad atomic-issue pools, compresses them into short reports, and evaluates both historical-review agreement and the possible validity of omitted concerns; broad generation achieves substantial coverage, but paper-only compression remains difficult.
Problem
Peer-review feedback often arrives too late for authors to add experiments, repair claims, or redesign evaluations, motivating earlier author-facing stress testing.
Method
The system generates broad pools of atomic concerns, deduplicates and refills them for semantic breadth, then selects at most 32 concerns using paper-only methods without historical-review access.
Results
Broad generation reaches 78.7% strict and 84.9% weighted historical coverage, while representative selectors retain only .402–.442 weighted coverage compared with .755 for the cluster Oracle.
Takeaways & Limitations
LLM review can complement formal peer review by providing broad, actionable feedback before submission, without being limited to imitating one finite panel.
Takeaways & Limitations
Trustworthy compression remains unresolved because representative selectors lose roughly 35 weighted-coverage points, while matcher and atomization noise obscure true overlap.
Abstract
from arXiv · showhide
Peer-review feedback often arrives too late for authors to make meaningful revisions. We study an author-facing LLM system that moves part of this stress test before submission: it generates a broad pool of atomic concerns and compresses them into a short report. We evaluate agreement with historical reviews and, separately, the possible validity of concerns they omit. From 10,000 ICLR 2026 submissions, we use 3,398 manuscripts with accessible versions that predate review. On a ten-paper diagnostic, independent sampling covers 44.9% of historical issues; deduplication and refill reaches 78.7% strict and 84.9% seriousness-weighted coverage, at 3.6$\times$ more requests and 5.2$\times$ more tokens. A hidden Top-32 Oracle preserves the full 79.3% weighted coverage of a 256-candidate pool, but paper-only selectors retain only 40--44%. LLM review therefore provides broad coverage with a large candidate pool but compresses poorly; ablations identify representative selection and matcher sensitivity as the main sources of this gap.
1 Introduction
The paper asks whether an author-facing LLM can surface reviewer-like and additional concrete concerns early enough for authors to revise. It evaluates broad concern generation, compression, and historical-review agreement while separating reviewer anticipation from independent validity.
- Motivation: The system moves part of peer review before submission by inspecting drafts and surfacing plausible concerns while revision remains possible.The motivation is that formal feedback often arrives after there is little time to add experiments, repair claims, or redesign evaluations.
- Evaluation framing: Historical reviews are incomplete references, so unmatched AI concerns may be invalid or useful feedback that a panel missed.The study therefore separates agreement with historical reviews from independent validity.
- Benchmark: The benchmark uses 10,000 submissions and a strict cohort of 3,398 pre-review manuscripts.Generation is frozen before human reviews are opened, and reviews are converted into corrected atomic issues for comparison.
- Main findings: 84.9% weighted historical coverage is achieved by deduplication and refill, up from 49.5%, at 5.2× more tokens.The pipeline comparison treats semantic breadth and resource cost jointly.
- Compression analysis: The Top-32 study separates semantic clustering, representative selection, and matching or atomization as distinct error sources.These components determine which concerns survive compression and how overlap is measured.
- Conclusion: AI and human concern sets overlap substantially but are not interchangeable, motivating separate agreement and validity evaluation.The paper frames broad AI feedback as complementary to, rather than identical with, one finite reviewer panel.
2 Related work
Prior work studies review data, simulated reviewers, LLM-generated feedback, and LLM judges. This paper instead targets author-facing discovery of atomic concerns, semantic breadth, compact selection, and judge dependence.
- Review data and simulation: PeerRead enabled data-driven study of 14.7K drafts and 10.7K expert reviews, while later work simulated reviewer behavior and decisions.The paper positions its task as distinct from both review-data analysis and reviewer simulation.
- LLM-generated scientific feedback: Existing LLM-feedback systems include GPT-4 feedback, specialist-agent systems, an 8B reviewer fine-tuned on 79K reviews, and dynamically expanding review questions.The present work is complementary in focusing on breadth, generation cost, Top-k compression, and atomic agreement with hidden issues.
- LLM-as-a-judge: LLM judges can capture semantic equivalence better than lexical metrics but exhibit position and same-family biases.This paper studies those failures when judge scores control both evaluation and selection.
3 Methods
The system broadly generates evidence-grounded concerns, removes semantic repeats, and compresses the pool to at most 32 paper-only concerns. Historical agreement uses semantic matching at τ = 0.70, while validity and saturation are evaluated separately.
- Task and system design: The pipeline has two stages: broad candidate generation followed by compression into an author-facing report.The first stage seeks coverage; the second selects a compact subset.
- Broad generation: Deduplication and refill checks schema and paper evidence, rejects semantically near-duplicate concerns, and continues until the candidate quota is filled.Its objective is marginal semantic breadth rather than more nominal outputs.
- Compact selection: The paper-only selector chooses at most k = 32 concerns without historical reviews, using direct ranking or cluster-then-select methods.Oracle selectors use hidden review matches only to provide infeasible upper bounds.
- Historical-review agreement: Historical reviews are deduplicated into H(P), and a semantic judge estimates whether each candidate and issue express the same substantive concern.Equivalence is defined by substantive concern rather than shared category.
- Historical-review agreement: Issue recall is measured at threshold τ = 0.70 using candidate–issue semantic matching.The method operationalizes agreement through a thresholded match matrix.
- Metrics and diagnostics: Weighted coverage uses seriousness weights, while the Top-k and cluster Oracles quantify partition loss and representative-selection loss.The framework also reports Major-Issue Recall, candidate match rate, and semantic uniqueness.
- Independent validity and saturation: The match matrix measures reviewer anticipation rather than validity, so unmatched concerns require separate assessment for grounding, soundness, specificity, and importance.Semantic saturation models declining acceptance probability as more concerns are generated.
4 Experiments
The experiments use reviewer-visible, pre-review ICLR manuscripts and hidden historical-review targets to test generation breadth, cost, and Top-32 compression. Deduplication improves coverage substantially, but paper-only compression loses much of the available coverage.
- 4.1 Dataset and manuscript recovery: 3,398 reviewer-visible manuscripts were recovered from 10,000 ICLR 2026 submissions using pre-submission versions.The corpus separates pre-review inputs from current post-review files and uses a frozen paper-family split.
- 4.2 Targets and evaluation: Reviews were converted into 178 manually corrected, deduplicated atomic issues, with seriousness weights distinguishing quick fixes from major changes.Generation and matching hide reviews, ratings, rebuttals, decisions, later versions, and match matrices.
- 4.4 Candidate generation: Deduplication and refill covered 140/178 issues versus 80/178 for independent sampling, despite a lower candidate hit rate.The gain reflects semantic breadth rather than more nominal positives, but required 1,905 requests and 96.61M tokens versus 525 requests and 18.66M tokens.
- 4.4 Candidate generation: Weighted coverage rose from .692 to .849 when expanding from 256 to 512 candidates, requiring 593 additional requests and 55.61M tokens.The final 128 candidates added only nine issues, showing diminishing returns.
- 4.4.2 Compression: The Top-32 Oracle preserved the full 256-candidate pool’s .793 weighted coverage, whereas paper-only selectors retained only .402–.442.The Oracle uses hidden review matches and therefore provides an upper bound rather than a deployable system.
- 4.5 Historical matching: Unmatched candidates may be invalid or useful omitted feedback, so the experiments evaluate historical agreement separately from independent validity.Expert adjudication of independent validity remained incomplete.
5 Ablation study
Ablations locate the main compression bottleneck in representative selection rather than clustering, while matcher deployments can produce sharply different overlap judgments. Generation policies also show that local novelty alone is insufficient.
- 5.1 Generation policies: Fine-grained persona quotas raised duplicate rejection from 37.1% to 74.5%, while rigid section–category–persona slots returned no_issue 56.7% of the time.Together with adaptive stopping, these results indicate that exploration must value marginal pool coverage rather than local novelty alone.
- 5.2 Partitioning: The cluster Oracle retained .755 weighted coverage versus .793 for the unrestricted Oracle, indicating modest aggregate partition loss.Among 160 clusters, 70 had no historical edge and 41 linked to multiple issues, reflecting splitting and merging errors.
- 5.2 Representative selection: Representative selection reduced weighted coverage to .402–.442 versus .755 for the cluster Oracle, making selection the dominant compression bottleneck.Retaining four candidates per cluster raised the hidden ceiling to .709; Qwen score correlated only .158 with hidden utility.
- 5.3 Matcher sensitivity: At the same .70 threshold, Sol High marked 660/45,568 candidate–issue pairs positive while DeepSeek V4 Flash marked eight, with κ = .015.Matcher and atomization noise therefore affect measured misses alongside pipeline errors.
- 5.4 Generator and judge interaction: A Sol selector chose 319/320 Sol candidates from mixed pools even without provider labels, suggesting style recognition or same-family preference may confound ranking.Judge identity, prompt, threshold, and generator-conditional error should accompany judge-guided results.
6 Conclusion
The system anticipates much of historical reviewer feedback while also surfacing concerns outside recorded reviews, but trustworthy compression into 32 concerns remains unresolved.
- 78.7% strict and 84.9% weighted coverage show that deduplication-and-refill can anticipate much of a historical reviewer panel.The AI pool also contains concerns outside the recorded reviews.
- Representative selectors lose roughly 35 weighted-coverage points even though a hidden Top-32 setting can preserve broad coverage.Semantic clustering helps, but representative selection remains a central compression gap.
- LLM review can complement formal peer review without replacing it when evaluation reports breadth, Top-k loss, cost, judge robustness, anticipation, and independent validity.
A Additional experimental details
The appendix preserves experimental records and specifies the bases for recall and cost–coverage comparisons.
- Frozen machine-readable tables retain per-paper values, model and prompt versions, candidate IDs, provider usage, and artifact hashes.
- Table 4 reports candidate-generation experiments using frozen atomic human issues and a same-issue threshold of 0.70 unless noted.
- Table 5 reports cost–coverage consequences of novelty-based early stopping over the same ten-paper development cohort.Wall time was directly observed only for the adaptive run; n/r means not recorded.
B Judge and annotation disclosure
The study discloses its frozen headline matcher and clarifies that the pilot does not support estimates of validity or inter-annotator agreement.
- The headline matcher is a frozen GPT-5.6 Sol High prompt at temperature zero with threshold .70.
- The pilot is not complete enough to estimate validity or inter-annotator agreement and instead motivates a preregistered independent expert audit.
C Candidate-generation experiment inventory
The appendix inventories the data and scripts behind the candidate-generation and clustering analyses.
- Table 4’s broader generation sequence is stored in candidate_generation_metrics.json and regenerated with plot_candidate_generation.py.
- The clustering graph and illustrative unmatched concerns are stored in clustering_analysis.json and regenerated with plot_clustering_analysis.py.
E Secondary acceptance-judging diagnostic
This secondary diagnostic compares paper-only with sanitized paper-plus-review acceptance judgments on 15 training papers. Its small cohort, differing schemas, and possible post-rebuttal edits make the result exploratory rather than predictive.
- On 15 training papers, the diagnostic compares binary Accept/Reject judgments using paper-only versus sanitized paper-plus-review inputs.The cohort contains 10 Accept and five Reject papers.
- Sol and Kimi improved with review prose, whereas DeepSeek and Inkling did not improve hard accuracy.
- The result is exploratory context sensitivity rather than an acceptance predictor because the cohort is small, schemas differ, and review notes may contain post-rebuttal edits.