PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

Ngoc Phan Phuoc Loc, Toan Huynh La Viet, Thanh Tran Khanh, Duy A Nguyen, Tuan Anh Nguyen Pham, Thanh Nguyen, Nitesh V. Chawla, Wray Buntine, Kok-Seng Wong, Khoa D. Doan, Binh T. Nguyen

arXiv:2605.26730v2cs.CL

TL;DR

Growing machine-learning submission volumes have strained peer review, while evidence about whether LLM reviewers can identify scientific gaps as well as humans remains limited. PRISM benchmarks automated and human reviewers across four structured dimensions using verifiable evaluation pipelines. The results show dimension-specific LLM strengths but no system with balanced performance across all dimensions, supporting specialist use within human-assisted review rather than autonomous replacement.

  • Problem

    The central gap is limited evidence about whether LLM reviewers can reliably evaluate scientific work and identify paper gaps relative to human reviewers under review overload.

  • Method

    PRISM benchmarks five automated reviewer systems and human reviewers across depth, novelty assessment, flaw identification and prioritization, and constructiveness.

  • Results

    LLM systems match or exceed humans on selected dimensions, including analytical depth, grounded novelty verification, and critique prioritization, but exhibit distinct specialization profiles.

  • Takeaways & Limitations

    LLM reviewers are best deployed as targeted specialist co-pilots within a human-assisted pipeline rather than standalone reviewers.

  • Takeaways & Limitations

    PRISM relies primarily on Gemini 2.5 Flash Lite as its judge model, and broader multi-judge evaluation across LLM families remains necessary; its corpus covers ML/AI venues only.

Abstract

from arXiv · show

The rapid growth in submissions to machine learning venues has strained the scientific peer-review system and intensified interest in LLM-based automated peer reviewers. However, how good these systems are actually, especially compared to human reviewers at catching scientific gaps, remains poorly understood. In this work, we introduce PRISM (Peer Review Intelligence via Structured Multi-dimensional assessment), a benchmarking framework that evaluates review quality across four dimensions: Depth of Analysis, Novelty Assessment,Flaw Identification & Major Issues Prioritization, and Multi-dimensional Constructiveness. Unlike most existing evaluations based on surface-level metrics like ROUGE and BLEU, or unconstrained LLM-as-a-judge prompting that conflates fluency with rigor, PRISM grounds each dimension in argument mining, retrieval-augmented verification, and consensus-based scoring. We apply PRISM to benchmark five leading automated reviewer systems and human reviewers on a stratified corpus of reviews from ICLR, ICML, and NeurIPS. The results reveal that LLMs can match or beat human reviewers on individual dimensions: comparable depth of analysis, stronger novelty verification, and highly accurate critique prioritization. However, no single system consistently matches the balanced performance of the human baseline across all dimensions at once. Each exhibits a distinct specialization profile with characteristic blind spots -- failure modes that aggregate metrics miss entirely. The implication is that LLM reviewers are best understood as targeted supplements to human review, effective within specific dimensions, but unreliable as standalone replacements. Our demo and key results can be found at https://khanhthanhdev.github.io/prism-page/.

1 Introduction

PRISM addresses growing pressure on machine-learning peer review by benchmarking whether LLM reviewers can perform core scientific-review duties relative to humans. Across four dimensions, the results show distinct LLM specializations but no universal replacement for human reviewers.

  • Motivation: Rapid submission growth and review overload have intensified interest in automated LLM reviewers and raised concerns about review quality and reliability.NeurIPS submissions rose from 15,671 in 2024 to 21,575 in 2025, while 17–21% of recent top-tier reviews already involved LLM assistance.
  • Benchmark: PRISM evaluates technical depth, novelty grounding, flaw identification and prioritization, and constructive feedback through dedicated benchmark dimensions.The framework is grounded in official reviewer guidelines and targets duties beyond summarization.
  • Findings: CycleReviewer and DeepReview match human analytical depth, while SEA-E exceeds humans on grounded novelty verification.TreeReview instead over-indexes on presentation anomalies, and other systems show measurable novelty hallucination.
  • Findings: Reviewer2 leads in flaw recall, and LLMs broadly achieve near-perfect prioritization of critical issues over minor issues.These results indicate cognitive alignment with human reviewers for critique prioritization.
  • Findings: DeepReview provides the most actionable feedback, but every system retains a constructiveness gap relative to human reviewers.The systems therefore show strengths on individual dimensions without uniformly matching human performance.
  • Implication: No single system dominates all four dimensions, positioning LLM reviewers as task-matched specialists rather than general-purpose replacements.PRISM motivates targeted deployment within human-assisted review pipelines.

2 Related work

Prior evaluation methods often reward surface similarity or rely on holistic LLM judgments that can conflate fluency with scientific rigor. PRISM instead uses dedicated, verifiable pipelines and a larger, stratified multi-venue benchmark to expose specialization and failure modes.

  • Prior evaluation: ROUGE and BLEU reward lexical similarity but are blind to scientific reasoning quality and factual correctness.Point-level overlap can measure coverage while still missing methodological depth.
  • Prior evaluation: Holistic LLM-as-judge evaluation introduces position, verbosity, and self-enhancement biases when scientific rigor is the target.ReviewEval is more structured but covers only 16 papers and three reviewer systems.
  • PRISM distinction: PRISM assigns dedicated verifiable procedures to depth, novelty, flaw identification, prioritization, and constructiveness instead of relying on rubric-prompted judging.Its procedures include argument mining, retrieval-augmented verification, consensus-weighted scoring, severity atomization, and semantic rule matching.
  • PRISM distinction: PRISM benchmarks five automated reviewer systems on a diverse, stratified corpus of 1,000 papers spanning multiple venues.This design supports analysis of distinct specialization profiles and structured failure modes.

3 The PRISM Framework

PRISM evaluates peer reviews through four independent pipelines that extract structured evidence and compute transparent metrics rather than assigning holistic LLM ratings. The framework measures evidential depth, literature-grounded novelty, consensus-based flaw coverage and prioritization, and multidimensional constructiveness.

  • Framework overview: Each PRISM pipeline decomposes review evaluation into structured evidence-extraction tasks, with final scores computed analytically for traceability.This avoids conflating stylistic fluency with scientific rigor in holistic LLM judging.
  • Depth of Analysis: Depth of Analysis measures both the proportion of argumentative premises and their grounding quality, combining them with a harmonic mean.A high score requires balanced evidence coverage and rigorous support.
  • Novelty Assessment: Novelty assessment extracts review claims, retrieves and diversifies prior literature, then verifies each claim against candidate publications and paper context.A relevance-weighted top-3 aggregation reduces optimistic inflation from a single spuriously favorable match.
  • Flaw Identification & Prioritization: Flaw evaluation extracts criticisms, filters them through paper-grounded consensus verification, and measures severity-stratified diagnostic coverage and prioritization.Critical flaws receive greater ranking weight than minor flaws, and nCPS approaches 1 for optimal ordering.
  • Multi-Dimensional Constructiveness: Constructiveness decomposes reviews into atomic comments scored from 0 to 2 for actionability, specificity, justification, solution provision, and tone.Mean Constructiveness averages normalized scores across comments, requiring consistently helpful feedback for a perfect score.

4 Experiment and analysis

PRISM evaluates human and automated reviews across a stratified multi-conference benchmark using structured, dimension-specific metrics. Results show distinct strengths: some systems approach human analytical depth, SEA leads novelty evidence grounding, Reviewer2 finds more flaws, and DeepReview provides the strongest constructiveness.

  • 4.1 Evaluation Setting: The benchmark samples 200 manuscripts per venue-year across five conference splits, stratified by decision category and topic while preserving score distributions.
  • 4.1 Evaluation Setting: PRISM compares five automated reviewer systems and human reviewers, using four dedicated pipelines rather than a single holistic judge.
  • Depth of Analysis: 0.494 is the human macro-averaged Depth of Analysis score, while CycleReviewer (0.484) and DeepReview (0.483) closely match it.
  • Novelty Assessment: 0.833 is SEA’s novelty evidence-grounding score, above the human baseline of 0.787, although grounding does not establish objective novelty or human-level agreement.
  • Flaw Identification & Major Issues Prioritization: Reviewer2 achieves the highest recall for Critical (0.591) and Minor (0.459) issues, exceeding the human baseline values of 0.343 and 0.281.
  • Multi-dimensional Constructiveness: DeepReview reaches the highest constructiveness score (0.634) versus 0.566 for humans, leading especially in Actionability (1.414) and Solution provision (0.784).

5 Conclusion & Future Work

PRISM concludes that LLM reviewers are specialized tools rather than general-purpose replacements for human expertise. The paper recommends targeted specialist deployment while identifying judge-family and domain coverage as important boundaries for interpretation.

  • Conclusion: Each evaluated LLM reviewer excels in a specific niche but exhibits distinct blind spots across other review-quality dimensions.
  • Actionable deployment recommendations: The paper recommends a targeted ensemble: Reviewer2 for flaw scanning, DeepReview for constructive drafting, and SEA for novelty-grounding checks.
  • Limitations: PRISM’s primary evaluation pipeline relies on Gemini 2.5 Flash Lite as its core judge model, despite preliminary robustness checks with another model on a subset.
  • Limitations: A comprehensive multi-judge study across diverse LLM families remains necessary to eliminate judge-specific biases, and the corpus covers ML/AI venues only.
  • Future Work: Future work targets cross-domain recalibration, inter-judge agreement across LLM families and human raters, and validation against author satisfaction or acceptance outcomes.

B.2.2 Baseline Implementation and Configuration

The baseline implementations combine supervised reviewer models and prompting-based systems under specified model, context-window, hardware, and generation configurations. Reviews are generated from complete textual manuscripts before PRISM assessment.

  • SEA-E: SEA-E uses a 70,000-token context window, batch size 4, and an 8,000-token maximum output length for full-length manuscripts.
  • CycleReviewer: CycleReviewer uses an 8B model with a 24,000-token context window and 2–3 iterative refinement passes on a single NVIDIA RTX A5000 GPU.
  • DeepReview: DeepReview combines a 14B reasoning engine with OpenScholar retrieval, using 70,000-token evidence synthesis and 10,000-token query-processing contexts.
  • Reviewer2: Reviewer2 replaces its native generation checkpoints with Qwen-3.5-14B while retaining the official two-stage prompt templates.
  • TreeReview: TreeReview standardizes its prompting backbone to Qwen3-14B and uses an 80,000-token context window for full papers and expanding question trees.
  • Review Generation Process: Each system receives complete manuscript text, including sections and tables represented textually, while figures and visual elements are excluded.

C.1 The PRISM Evaluation Pipeline

PRISM evaluates peer-review quality through dedicated pipelines that mine arguments, verify novelty claims against retrieved literature, and aggregate evidence with structured scoring. Its Depth of Analysis pipeline progresses from ADU segmentation to argument classification and premise-grounding assessment, while Novelty Assessment uses structured extraction, retrieval, and claim-level verification.

  • Depth of Analysis: Depth of Analysis segments reviews into ADUs, classifies argument roles and aspect topics, and assigns categorical grounding scores to premises.The three phases are illustrated by Figures 10–12.
  • Depth of Analysis: Overall Depth of Analysis is the harmonic mean of Premise Ratio and Normalized Grounding Score, requiring both supporting-argument volume and evidentiary grounding.
  • Depth of Analysis: The running example extracts one claim and three premises, assigning grounding scores of 0, 1, and 2 to the premises.Score 0 is generic or vague, Score 1 is internally anchored, and Score 2 is externally comparative.
  • Novelty Assessment: Novelty Assessment extracts structured paper and review information, retrieves related work deterministically, and verifies each novelty-claim/evidence pair with a five-level verdict from −2 to +2.The retrieval stage uses Semantic Scholar candidates, deduplication, filtering, and temporal constraints; scores use a relevance-weighted top-3 rule rather than max.
  • Novelty Assessment: For C1, one off-topic work receives Unsupported (−2), while two relevant works receive Supported (+2), illustrating evidence-sensitive claim scoring.The example also shows that retrieval quality can affect opposing novelty claims similarly.

D.3 Results and Discussion

PRISM’s metrics are empirically near-orthogonal, with weak pairwise associations and negligible shared variance. This supports using multiple metrics jointly rather than treating one dimension as a proxy for overall review quality.

  • |r|max = 0.193 across six metrics, indicating consistently very weak inter-metric associations.
  • Cross-dimension independence: Novelty shows no significant association with Critical Recall, Minor Recall, or nCPS, while DoAHM shows no meaningful linear relationship with Critical Recall.Reported coefficients include Critical Recall r = −0.015, Minor Recall r = +0.021, nCPS r = +0.007, and DoAHM–Critical Recall r = −0.006.
  • Cross-dimension independence: DoAHM and MCS have r = +0.094 with p < 0.001, but r2 < 0.01, so argumentative depth and constructiveness remain distinct dimensions.
  • Overall assessment: Seven of fifteen metric pairs are non-significant, and every significant pair has |r| < 0.20 with shared variance below 4%.
  • Statistical testing: The study compares LLM baselines with humans using paired Wilcoxon signed-rank tests and Holm-Bonferroni correction across model-metric families.Correction is applied independently within each LLM and evaluation dimension across five conferences.

E.2 Depth of Analysis

Depth of Analysis varies with both justification volume and grounding quality, producing substantial differences among systems. CycleReviewer and DeepReview approach human performance, while unsupported verbosity and misplaced attention reduce depth.

  • DoA depends on the interaction between Premise Ratio and Grounding Score rather than either evidentiary quantity alone.
  • Reviewer2 produces many vague Score 0 premises despite generating more externally grounded Score 2 premises than other LLM baselines, and its low Premise Ratio dilutes overall depth.
  • Aspect focus: Human reviewers allocate the largest review shares to Methodology (∼50%) and Experimental Design (∼29%), and advanced baselines broadly mirror these priorities.
  • Aspect focus: Reviewer2 most closely matches human premise-level aspect distribution with JSD = 0.071, whereas TreeReview assigns 22.9% to Clarity versus 9.4% for humans.
  • Per-aspect depth: Human DoA is highest for Methodology at 0.510±0.156 and Experiment at 0.456±0.207, ahead of Novelty at 0.357±0.322 and Clarity at 0.266 ± 0.268.
  • CycleReviewer and DeepReview achieve DoA scores closest to, and in several cases statistically indistinguishable from, human reviewers across all four aspects.Their parity is associated with the highest LLM Premise Ratios across aspects.
  • Novelty interpretation: Novelty scores measure whether review claims are groundable under PRISM retrieval and verification, not whether manuscripts are objectively novel.The scalar metric also does not establish full claim-level agreement with human novelty judgments.

E.4 Flaw Identification & Prioritization of Major Issue

LLM reviewers differ in flaw-detection breadth, but all evaluated baselines achieve near-perfect prioritization scores. Their severity-stratified focus also broadly adapts from technical failures in critical flaws to presentation issues in minor flaws.

  • Flaw identification: Reviewer2 achieves the highest Critical and Minor flaw Recall across every evaluated venue, with Critical Recall ranging from 0.506 to 0.649.Its broader diagnostic coverage is associated with unusually high issue-generation volume.
  • nCPS > 0.96 for every baseline across all conferences, making automated systems statistically indistinguishable from humans in ranking major flaws first.
  • Severity-stratified focus: For Critical flaws, humans allocate 92.3% of critiques to Methodology and Experimental Design, while DeepReview shows a similar 91.7% allocation.Clarity accounts for 1.7% of human critical flaws and 1.6% for DeepReview.
  • Severity-stratified focus: TreeReview assigns 6.0% of critical flaws to presentation errors, broadening critical evaluation beyond the technical concentration seen in humans and DeepReview.
  • Severity-stratified focus: For Minor flaws, humans and LLMs reduce methodological emphasis and increase attention to Clarity, Presentation, and Reproducibility.
  • Interpretation: The severity-dependent shift indicates that LLMs can categorize flaws in ways that broadly resemble human review behavior.

E.5 Multi-Dimensional Constructiveness

The constructiveness analysis shows distinct reviewing profiles: humans diagnose flaws but often lack actionable remedies, while LLM systems trade off specificity, justification, solutions, and tone. DeepReview most consistently leads on overall constructiveness across venues, whereas Reviewer2 and CycleReviewer exhibit different practical weaknesses.

  • Cross-Venue Performance: DeepReview achieves the highest MCS across all five conferences, ranging from 0.629 at ICLR 2024 to 0.635 at NeurIPS 2025.Holm’s post-hoc tests assess consistency across venues.
  • Summary of Constructiveness: The multi-dimensional results distinguish human diagnosticians, verbose or superficial LLM reviewers, and DeepReview’s collaborator-like combination of flaw identification, solutions, and professional tone.The five core constructiveness dimensions are Actionability, Specificity, Justification, Constructive Suggestion, and Tone & Respect.
  • Human Reviewers: Humans show high specificity but low solution density, with D2 ≈1.72 and SD ≈0.102, meaning only about 10% of comments contain explicit fixes.The solution bottleneck separates identifying concrete flaws from proposing actionable remedies.
  • Reviewer2: Reviewer2 combines strong justification (D3 > 0.90) and CD ≈0.58 with very low solution density (SD ≈0.05), producing verbose critiques that rarely explain how to fix problems.Its high overall scores can therefore conceal limited practical usefulness.
  • CycleReviewer: CycleReviewer reaches D2 ≈1.89 and AR > 0.90 but drops to D3 ≈0.32 and SD ≈0.04, indicating specific yet shallow checklist-style critiques.Most comments reference concrete paper elements and offer general directions, but lack evidentiary backing or explicit fixes.
  • Tone and Professionalism: DeepReview scores higher than humans on Tone, with D5 ≈1.72 versus ≈1.58, indicating more professional, neutral, and encouraging feedback.The comparison frames tone as a distinct constructiveness dimension rather than a substitute for technical critique.

E.6 Review Sensitivity to Paper Quality: Accept vs. Reject Analysis

The accept-versus-reject analysis tests whether review metrics vary with eventual paper decisions and whether evaluator choice changes the benchmark’s conclusions. Human reviews show stronger decision sensitivity, while LLM results are more invariant and remain qualitatively stable across evaluator backends.

  • Interpretation: The comparison characterizes humans as outcome-sensitive reviewers and LLMs as invariant diagnostic scanners applying stable metric distributions across papers.This is the paper’s summary interpretation of Table 10.
  • Accept vs. Reject: Human reviews assign significantly higher Novelty Scores to papers ultimately accepted than to rejected papers, despite reviewers being blind to final decisions.The passage describes this association as evidence of predictive validity.
  • Accept vs. Reject: Across all five automated systems, metric differences between accepted and rejected papers are predominantly uniform; only DeepReview’s Prioritization Score is significant, at ∆= +0.011∗∗∗.This contrasts with the stronger outcome sensitivity observed in human reviews.
  • Evaluator Robustness: Replacing Gemini 2.5 Flash Lite with Mimo v2.5 Pro preserves the relative ordering of reviewer types across the evaluated metrics.The full pipeline was rerun across six metrics and four aspects on 250 papers.
  • Evaluator Robustness: Prioritization shows the smallest evaluator divergence, while SEA receives the highest Novelty Assessment scores from both evaluators: Gemini 0.841 and Mimo 0.734.The reported backend differences do not reverse any qualitative conclusion drawn from Gemini.

F.1 Depth of Analysis

The Depth of Analysis analysis distinguishes reviews that support judgments with grounded, paper-specific evidence from reviews dominated by generic claims or boilerplate. Human reviewers provide dense technical grounding, DeepReview achieves higher DoA efficiently on VLAP, and Reviewer2 and TreeReview reveal distinct grounding failures.

  • VLAP: Human reviewers produce 37 premises from 55 arguments on VLAP, with Rpremise = 0.673 and GS = 0.500, grounding observations in concrete technical components.Their critiques also question whether the paper contains a single innovative algorithmic contribution beyond two engineering contributions.
  • VLAP: DeepReview produces 11 premises from 15 arguments on VLAP, with Rpremise = 0.733 and GS = 0.546, exceeding human premise ratio and achieving the highest average grounding score.Its premises are described as architecturally precise.
  • NV-Embed: Reviewer2’s NV-Embed review has Rpremise = 0.152 and GS = 0.215, yielding DoAHM = 0.178, a 70% drop from Human’s 0.581.It generates 46 arguments but only 7 premises, and many premises restate the paper’s own assertions rather than independently evaluating them.
  • NV-Embed: The DoA metric penalizes Reviewer2 because its low premise ratio and low grounding quality jointly reduce the harmonic-mean score.Both Rpremise and GS must be high for human-level analytical depth.
  • VLAP: On VLAP, four reviewers allocate 0% of their premise budget to Clarity, while TreeReview’s generic reproducibility premises consume analytical capacity despite the paper’s simple design.The case identifies an internal reviewing heuristic, rather than genuine manuscript ambiguity, as the source of the surface-level trap.
  • VLAP: TreeReview’s VLAP review yields only 7 premises from 36 arguments, with Rpremise = 0.194 and GS = 0.357, producing DoAHM = 0.252, a 55% drop from Human’s 0.567.Its non-Clarity premises engage the mechanism but remain internally grounded without external literature anchoring.

F.2 Novelty Assessment

PRISM’s novelty assessment distinguishes evidence-grounded novelty claims from speculative concerns and compares how human and automated reviewers cover and calibrate such claims. DeepReview expands novelty coverage while SEA emphasizes directly verifiable claims, whereas human reviews can be penalized for unsupported speculation.

  • Cross-reviewer comparison: Both human and automated reviewers correctly identify core contributions, but score differences reflect calibration of uncertain assessments rather than disagreement about novelty.Human claims about contrastive learning and CONPAIR are recognized as the paper’s core contributions, while divergence arises from handling uncertain claims.
  • Human reviewers: Human novelty reviews may contain valid but unverifiable skepticism, which can substantially lower evidence-grounded scores.A speculative training/test-overlap concern averaged −0.18 because most related-paper comparisons were unsupported or insufficient, while verified non-novel claims could still earn +2.0.
  • SEA: SEA produces five directly verifiable claims and achieves a higher raw mean claim score than the human comparison despite generating half as many claims.SEA’s claims target curriculum structure, fine-tuning strategy, and VQA-based alignment; its raw mean is 1.73 versus 1.20.
  • SEA: SEA’s evidence-anchored claim style avoids speculative concerns, but its higher evidence-calibrated novelty score does not establish that it is a better reviewer.The metric rewards claims grounded in described methods and can penalize richer human reviews when their concerns cannot be corroborated through the retrieved paper pool.
  • DeepReview: DeepReview generates 2.4× more claims than the human comparison while maintaining nearly identical raw mean claim scores.It produces 12 claims with a raw mean of 1.500 versus 1.467, and adds well-evidenced analysis of scope limitations.
  • DeepReview: DeepReview can improve precision and coverage by naming novelty distinctions more specifically than human claims, including positional-encoding and precision regimes.DR2 earns 2.0 versus H-C4’s 0.667 by specifying upgraded language classes, while DR5 explicitly distinguishes constant from logarithmic precision.

G Limitations

PRISM’s current evaluation is limited by its machine-learning-only scope and reliance on frontier LLMs for several complex evaluation tasks.

  • Domain Generalization: PRISM evaluates 1,000 manuscripts exclusively from ICLR, ICML, and NeurIPS, so its current instantiation may require recalibration for non-ML domains.The authors note that venue-specific structural norms, citation densities, and evaluation criteria differ across disciplines.
  • LLM Dependency: PRISM delegates text atomization, fact-finding, and scoring to frontier LLMs, leaving residual risks from context loss, retrieval coverage, and judge priors.The framework mitigates hallucination and prompt sensitivity through constrained multi-phase pipelines and deterministic decoding, but does not eliminate these bottlenecks.
Loading 2605.26730v2…