PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

Ngoc Phan Phuoc Loc, Toan Huynh La Viet, Thanh Tran Khanh, Duy A Nguyen, Tuan Anh Nguyen Pham, Thanh Nguyen, Nitesh V. Chawla, Wray Buntine, Kok-Seng Wong, Khoa D. Doan, Binh T. Nguyen

arXiv:2605.26730v2cs.CL

TL;DR

Rapidly growing machine-learning submissions have strained peer review, while the quality of LLM reviewers relative to humans remains unclear. PRISM benchmarks automated and human reviewers across four dimensions and finds specialized strengths but no system matching humans’ balanced performance across all dimensions.

  • Problem

    Rapidly growing machine-learning submissions and emerging LLM reviewers leave their comparative ability to identify scientific gaps insufficiently understood.

    Source: §1 · pp. 1–2

  • Method

    PRISM benchmarks five automated reviewer systems and human reviewers across four dimensions using dedicated pipelines grounded in argument mining, retrieval-augmented verification, and consensus-based scoring.

    Source: §1 · p. 2

  • Results

    No single system dominates all four dimensions, with systems instead matching human analytical depth, outperforming humans on grounded novelty assessment, or specializing in distinct capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LLM reviewers are best deployed as task-matched specialist co-pilots within human-assisted review rather than autonomous replacements for human expertise.

    Source: §5 · p. 9

    The evaluation relies primarily on one judge model and covers ML/AI venues, requiring broader multi-judge validation and recalibration for other scientific domains.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

The rapid growth in submissions to machine learning venues has strained the scientific peer-review system and intensified interest in LLM-based automated peer reviewers. However, how good these systems are actually, especially compared to human reviewers at catching scientific gaps, remains poorly understood. In this work, we introduce PRISM (Peer Review Intelligence via Structured Multi-dimensional assessment), a benchmarking framework that evaluates review quality across four dimensions: Depth of Analysis, Novelty Assessment,Flaw Identification & Major Issues Prioritization, and Multi-dimensional Constructiveness. Unlike most existing evaluations based on surface-level metrics like ROUGE and BLEU, or unconstrained LLM-as-a-judge prompting that conflates fluency with rigor, PRISM grounds each dimension in argument mining, retrieval-augmented verification, and consensus-based scoring. We apply PRISM to benchmark five leading automated reviewer systems and human reviewers on a stratified corpus of reviews from ICLR, ICML, and NeurIPS. The results reveal that LLMs can match or beat human reviewers on individual dimensions: comparable depth of analysis, stronger novelty verification, and highly accurate critique prioritization. However, no single system consistently matches the balanced performance of the human baseline across all dimensions at once. Each exhibits a distinct specialization profile with characteristic blind spots -- failure modes that aggregate metrics miss entirely. The implication is that LLM reviewers are best understood as targeted supplements to human review, effective within specific dimensions, but unreliable as standalone replacements. Our demo and key results can be found at https://khanhthanhdev.github.io/prism-page/.

1 Introduction

PRISM benchmarks LLM and human peer reviewers across four dimensions of scientific review quality using dedicated, evidence-grounded evaluation pipelines. Results show specialized LLM strengths but no system consistently matches the balanced performance of human reviewers across all dimensions.

Motivation

  • Rising submissions and reviewer workloads motivate evaluating whether LLMs can identify scientific gaps as reliably as increasingly time-constrained human reviewers.NeurIPS submissions rose from 15,671 in 2024 to 21,575 in 2025, while one consistency experiment suggested that up to 23% of acceptance decisions could change with reviewer assignment [16].Source: §1 · pp. 1–2

Benchmark framework

  • PRISM evaluates Depth of Analysis, Novelty Assessment, Flaw Identification & Major Issues Prioritization, and Multi-dimensional Constructiveness.These dimensions target technical soundness, originality, critical-error diagnosis, and actionable feedback rather than mere summarization.Source: §1 · p. 2
  • Each PRISM dimension uses a dedicated pipeline grounded in argument mining, retrieval-augmented verification, and consensus-based scoring.The framework is designed to assess reviewer competence beyond surface-level prose.Source: §1 · pp. 2–3

Evaluation design

  • The benchmark compares five automated systems—TreeReview [7], Reviewer2 [8], SEA-E [9], DeepReview [10], and CycleReviewer [11]—with human reviewers on papers from ICLR, ICML, and NeurIPS.Its corpus includes manuscripts and expert human reviews, providing a consensus-driven reference for comparison.Source: §1 · pp. 2–3

Benchmark findings

  • LLM reviewers show distinct strengths: SEA-E leads grounded novelty verification, Reviewer2 leads flaw recall, and DeepReview provides the most actionable feedback.CycleReviewer and DeepReview match human analytical depth, while TreeReview tends toward surface-level assessment and other systems show novelty hallucination.Source: §1 · p. 2
  • No single system dominates all four dimensions, because each LLM reviewer specializes in a niche while retaining structured gaps that aggregate metrics miss.LLM reviewers are therefore positioned as task-matched supplements rather than general-purpose replacements for human reviewers.Source: §1 · p. 2

2 Related work

Prior automated reviewer systems use structured reasoning, optimized generation, or multi-agent collaboration, but their evaluations largely miss the grounded logic of individual critiques. PRISM addresses this gap with dedicated verifiable pipelines and a broad benchmark spanning five automated systems and 1,000 papers.

LLM-based Reviewer Systems

  • Existing reviewer systems improve generation through structured reasoning, specialized pipelines, or multi-agent collaboration, but evaluation mainly relies on generic n-gram metrics or whole-review LLM judging.TreeReview [7], DeepReview [10], Reviewer2 [8], SEA [9], and CycleReviewer [11] exemplify these approaches, whose macro-level assessments cannot verify grounded premises or novelty claims.Source: §2 · p. 3

Evaluation of AI-Generated Reviews

  • Prior evaluation methods range from surface-overlap metrics to LLM judging, yet remain limited by reasoning blindness, judge bias, small or narrow benchmarks, or lack of reusable per-review scoring.ROUGE and BLEU miss scientific reasoning; LLM judges face position [22], verbosity [23], and self-enhancement [24] biases, while ReviewEval [25] covers 16 papers and three systems and DeepReview-Bench focuses largely on ICLR.Source: §2 · p. 3

PRISM’s Distinction

  • PRISM evaluates each dimension with dedicated verifiable pipelines and benchmarks five automated systems on a stratified corpus of 1,000 papers across five venue-years.Its pipelines use argument mining, retrieval-augmented claim verification, consensus-weighted scoring, severity atomization, and semantic rule matching rather than rubric-prompted LLM judging.Source: §2 · p. 3

3 The PRISM Framework

PRISM evaluates peer reviews through four modular, independently scored pipelines that replace holistic judging with traceable evidence extraction and analytical scoring. The framework measures analysis depth, novelty assessment, flaw identification and prioritization, and constructiveness.

3 The PRISM Framework

  • PRISM distributes review evaluation across four modular LLM-driven pipelines that produce distinct quantitative metrics from structured evidence extraction rather than holistic judgments.This design targets specific LLM failure modes, separates evidence identification from score computation, and enables traceable metric formulation.Source: Figure 2 · p. 4, §3 · p. 4

3.1 Depth of Analysis

  • Depth of Analysis combines evidence coverage and grounding quality, requiring reviews to support judgments with objective premises rather than generic assertions.Argumentative Discourse Units are classified as claims or premises and assessed as vague, manuscript-internal, or externally grounded; a harmonic mean penalizes imbalance between coverage and rigor.Source: §3.1 · p. 4

3.2 Novelty Assessment

  • Novelty Assessment verifies review claims against retrieved prior literature, using relevance-weighted top-3 aggregation to reduce optimistic inflation from spurious matches.The pipeline extracts contribution anchors and novelty claims, retrieves diversified candidates, and assigns support scores ranging from contradicted to fully supported.Source: §3.2 · p. 5

3.3 Flaw Identification & Major Issues Prioritization

  • Flaw Identification and Major Issues Prioritization uses consensus-verified human and LLM findings to distinguish critical from minor flaws and evaluate their ordering.Invalid critiques are discarded, valid flaws are mapped to their original positions, and nCPS rewards reviews that place critical issues before minor ones.Source: §3.3 · p. 5

3.4 Multi-Dimensional Constructiveness

  • Multi-Dimensional Constructiveness rates atomic review comments on actionability, specificity, justification, solution quality, and professional tone.The mean score averages normalized comment-level ratings, so perfect constructiveness requires consistently specific, justified, actionable, solution-oriented, and professionally toned feedback.Source: §3.4 · p. 6

4 Experiment and analysis

PRISM evaluates five automated reviewer systems against human reviewers on a stratified multi-venue benchmark across four dimensions, revealing distinct strengths and failure modes in analytical depth, novelty evidence, flaw diagnosis, and constructiveness.

4.1 Evaluation Setting

  • The benchmark samples 200 manuscripts per venue-year across five conference splits, stratified by decision category and topic while preserving each venue’s original score distribution.The evaluation covers ICLR 2024–2026, ICML 2025, and NeurIPS 2025.Source: §4.1 · p. 6, Table 1 · p. 6, Figure 3 · p. 6

4.2 Result Analysis: LLMs vs Human-Reviewer Baselines

  • Across all four PRISM dimensions, five automated reviewer systems are compared with a human baseline using macro-averaged scores, with detailed analyses and qualitative examples provided separately.The benchmark reports aggregate results in Table 2 and expands each dimension in subsequent analyses.Source: §4.2 · p. 6, Table 2 · p. 7

4.2.1 Depth of Analysis

  • Human reviewers achieve the highest overall Depth of Analysis score at 0.494, while CycleReviewer (0.484) and DeepReview (0.483) closely match them through strong Premise Ratios.Their premise ratios are approximately 0.60, compensating for a slight gap in absolute Grounding scores; TreeReview instead devotes about 24% of effort to formatting issues.Source: Table 3 · p. 7, §4.2.1 · p. 7

4.2.2 Novelty Assessment

  • SEA attains the highest Novelty Assessment evidence-grounding score at 0.833 versus 0.787 for humans, although this measures retrievable support for selected claims rather than objective novelty judgment.SEA endorses novelty in 79% of claims versus 59% for humans, while negative novelty judgments consistently receive stronger literature grounding across systems.Source: §4.2.2 · pp. 7–8, Figure 5 · p. 8, Figure 6 · p. 8

4.2.4 Multi-dimensional Constructiveness

  • DeepReview leads Multi-Dimensional Constructiveness at 0.634 versus 0.566 for humans, particularly through Actionability (1.414) and Solution provision (0.784).Humans lead Specificity at 1.725 but score 0.470 on Solution provision, while Reviewer2’s 0.266 Solution score leaves many critiques unactionable.Source: Figure 8 · p. 8, §4.2.4 · p. 9

4.2.3 Flaw Identification & Major Issues Prioritization

  • Reviewer2 achieves the highest flaw recall for Critical (0.591) and Minor (0.459) issues, exceeding humans at 0.343 and 0.281, while all systems prioritize critical flaws similarly at approximately nCPS 0.97.Reviewer2 maintains about a 3.3% hallucination rate, whereas CycleReviewer reaches about 18.5%; hallucinated flaws occur only among minor issues.Source: §4.2.3 · pp. 8–9, Figure 7 · p. 8

5 Conclusion & Future Work … B.2.1 Taxonomy of Baseline LLM Reviewers

PRISM frames automated peer reviewers as specialized tools that complement rather than replace human expertise, while identifying judge, domain, and validation limitations. The paper formalizes and evaluates this framework across structured benchmarks, reviewer baselines, evaluation procedures, quantitative results, and qualitative analyses.

5 Conclusion & Future Work

  • PRISM recommends specialist ensembles with human oversight because no single reviewer dominates all four dimensions.Reviewer2 targets exhaustive flaw scanning, DeepReview drafts actionable feedback, and SEA performs novelty-grounding checks.Source: §5 · p. 9
  • The framework requires multi-judge studies, broader scientific domains, and human-outcome validation to address judge bias, domain limits, and metric validity.The current pipeline primarily uses Gemini 2.5 Flash Lite, with preliminary checks using Xiaomi MiMo V2.5 Pro.Source: §5 · pp. 9–10

B Experimental Details

E Full Cross-Dataset Quantitative Results

  • PRISM’s broader evaluation agenda includes metric-independence analysis, evaluator robustness across LLM backends, and explicit limitations beyond the ML/AI benchmark corpus.The supplied framework materials identify Pearson-correlation analysis and evaluator-robustness testing as complementary analyses.Source: §5 · p. 10, E Full Cross-Dataset Quantitative… · p. 15

F Qualitative Analysis & Case Studies

  • The study supplements quantitative evaluation with qualitative case studies spanning depth, novelty, flaw prioritization, and multidimensional constructiveness.These analyses examine how reviewer behavior varies across PRISM’s evaluation dimensions.Source: F Qualitative Analysis & Case Studies · p. 15

A Formal problem definition

  • PRISM defines review benchmarking as measuring cognitive depth and analytical gaps rather than linguistic fluency or surface n-gram overlap.The evaluation uses the generated review, human reference, and original paper to produce multidimensional scores while penalizing superficial summaries, hallucinated flaws, ungrounded novelty claims, and un-actionable feedback.Source: A Formal problem definition · p. 16

B.1 Dataset selection.

  • The benchmark samples exactly 200 manuscripts from each of five conference-year splits, stratified by decision category and topic while preserving venue-level score distributions.The corpus covers ICLR 2024, ICLR 2025, ICLR 2026, ICML 2025, and NeurIPS 2025.Source: §B.1 · p. 16

B.2.1 Taxonomy of Baseline LLM Reviewers

  • PRISM compares human reviews with five automated systems: SEA-E, DeepReview, Reviewer2, CycleReviewer, and TreeReview.The baselines span supervised fine-tuning and prompting-based paradigms, with prompting systems standardized on a common backbone to isolate prompting effects.Source: §B.2.1 · p. 16

B.2.2 Baseline Implementation and Configuration · B.2.3 Review generation process. · C PRISM Evaluation Framework: Pipeline Details and Experimental Setup

The study standardizes five LLM reviewer baselines with detailed configurations, then generates one complete review per model for each of 1,000 papers, producing 5,000 reviews for PRISM evaluation. PRISM also documents the hyperparameters and prompt templates used for reproducibility.

B.2.2 Baseline Implementation and Configuration

  • SEA-E uses ECNU-SEA/SEA-E to generate summaries, strengths, weaknesses, and numerical ratings within a 70,000-token context window.It processes four papers per batch and caps final critiques at 8,000 tokens, using temperature 0.7 and top-p 0.9.Source: §B.2.2 · p. 17
  • CycleReviewer applies 2 to 3 iterative refinement passes with WestlakeNLP/CycleReviewer-ML-Llama-3.1-8B on a single NVIDIA RTX A5000 GPU.It uses a 24,000-token context window and limits critiques to 3,000 tokens.Source: §B.2.2 · p. 17
  • DeepReview combines a 14B reasoning engine with retrieval augmentation, retrieving approximately 30 candidate papers and reranking them to the top 10 sources for grounding.Its retrieval subsystem uses Llama-3.1-OpenScholar-8B for evidence synthesis and Qwen-2.5-3B-Instruct for query processing, while the core model runs across two GPUs.Source: §B.2.2 · p. 17
  • Reviewer2 replaces its native generation models with Qwen-3.5-14B while retaining the official prompt templates and two-stage question-generation and critique-synthesis pipeline.The engine uses an 80,000-token context window, batch size 4, and a 7,000-token maximum generation length.Source: §B.2.2 · p. 17
  • TreeReview standardizes its backbone to Qwen3-14B and recursively decomposes review objectives into sub-questions, aggregating leaf answers to synthesize critiques.The pipeline uses an 80,000-token context window, batch size 4, and a 7,000-token output cap.Source: §B.2.2 · p. 17

B.2.3 Review generation process.

  • 5,000 automated reviews were generated by providing complete text from 1,000 papers to five LLM baselines, excluding figures and other visual elements.Each model independently produced a complete peer review, creating the corpus used for subsequent PRISM evaluations.Source: §B.2.3 · p. 18

C PRISM Evaluation Framework: Pipeline Details and Experimental Setup

  • The PRISM evaluation setup specifies the hyperparameters for all baselines and the exact prompt templates used for multi-dimensional assessment.These details are provided to support reproducibility and methodological transparency.Source: C PRISM Evaluation Framework: Pipeline… · p. 18

C.1 The PRISM Evaluation Pipeline … D.2 Statistical Method

This section presents the PRISM evaluation pipeline, which uses dimension-specific, structured judging to assess peer-review quality across argument depth, novelty verification, flaw detection and prioritization, and constructiveness. It also describes the decomposition of reviews into Atomic Review Comments and the statistical analyses used to examine whether the resulting metrics capture distinct aspects of review quality.

C.2 PRISM Judge Setup.

  • PRISM uses Gemini 2.5 Flash Lite with temperature 0.0, top-p 0.95, and standardized rubrics to score human and automated reviews consistently.The judge extracts arguments, verifies flaw validity against ground truth, and computes component scores for both reviewer types.Source: §C.2 · p. 18

C.3 Prompt Templates by Dimension

  • Depth of Analysis is computed through ADU segmentation, argument-role and aspect classification, and premise grounding evaluation.The overall score combines Premise Ratio and Normalized Grounding Score via their harmonic mean, requiring both sufficient supporting arguments and strong grounding.Source: §C.3 · pp. 18–19, Phase 3: Premise Grounding Score… · p. 21

CLASSIFICATION

  • The running examples demonstrate that PRISM converts raw review text into structured claims, premises, aspects, grounding scores, and novelty-verification outputs.For Depth of Analysis, an example yields 1 Claim and 3 Premises with grounding scores of 0, 1, and 2; for Novelty Assessment, the example’s normalized score is NS(R) = 0.778 and reflects retrieval-pool sensitivity.Source: Phase 3: Premise Grounding Score… · pp. 21–22, CLASSIFICATION · p. 27

Phase 3: LLM Judge Verification

  • Novelty Assessment retrieves prior work and verifies each novelty claim with a five-level verdict from SUPPORTED (+2) to UNSUPPORTED (−2).The default aggregates each claim over its three highest-relevance candidates using relevance-weighted averaging, reducing the influence of spuriously favorable matches; the final Novelty Verification Score averages claims.Source: Phase 3: Premise Grounding Score… · p. 22

Phase 2: Meta-reviewer Validity and Severity Judgement

  • The flaw pipeline atomizes and groups reviewer arguments, then independently verifies each micro-flaw and labels it Critical or Minor.It derives Critical Recall, Minor Recall, and normalized Critique Prioritization Score, which adapts NDCG to assess whether severe critiques are placed earlier within review sections.Source: CLASSIFICATION · p. 27

Phase 1: Atomic Review Comment (ARC) Extraction

Phase 2: Multi-dimensional Constructiveness Scoring

  • Each ARC is scored on five dimensions of constructiveness and aggregated into a normalized Multi-dimensional Constructiveness Score.The dimensions are represented as D1 through D5 and are averaged across extracted comments.Source: Phase 2: Multi-dimensional… · pp. 30–31

D.1 Motivation and Objective

  • The statistical analysis tests whether Depth, Novelty, Flaw Identification, Issue Prioritization, and Constructiveness metrics measure distinct, non-overlapping aspects of review quality.Highly correlated metrics would provide redundant information and reduce the evaluation’s effective dimensionality.Source: §D.1 · p. 31

D.2 Statistical Method

  • Metric relationships are evaluated with Pearson correlations, two-tailed t-tests for ρ = 0, and thresholds distinguishing negligible, small, moderate, and large practical effects.Only correlations that are both statistically significant and moderate-to-large in magnitude are considered substantively meaningful.Source: §D.2 · pp. 31–32

D.3 Results and Discussion · E Full Cross-Dataset Quantitative Results · E.1 Statistical Significance Testing Protocol

The six review-quality metrics are empirically near-orthogonal, supporting their joint use as a comprehensive multidimensional benchmark. Statistical comparisons use paired non-parametric tests with Holm-Bonferroni correction across venues to distinguish robust model–human differences from isolated effects.

D.3 Results and Discussion

  • The full correlation matrix shows very weak associations across all six metrics, with a maximum absolute coefficient of |r|max = 0.193.Figure 21 reports Pearson coefficients with significance annotations for every metric pair.Source: Figure 21 · p. 32, §D.3 · p. 32
  • Novelty assessment is decoupled from flaw detection: NS correlates nonsignificantly with Critical Recall, Minor Recall, and nCPS.The reported coefficients are r = −0.015, +0.021, and +0.007, respectively, with p-values above 0.05.Source: §D.3 · p. 32
  • Argumentation depth is independent of critical-flaw identification, while its marginal association with constructiveness remains negligible.DoAHM and Critical Recall have r = −0.006; DoAHM and MCS have r = +0.094 with r2 < 0.01.Source: §D.3 · pp. 32–33
  • Seven of fifteen metric pairs are nonsignificant, while all significant associations have |r| < 0.20 and shared variance below 4%, confirming distinct quality dimensions.These results justify evaluating peer reviews jointly across six complementary metrics.Source: §D.3 · p. 33

E.1 Statistical Significance Testing Protocol

  • Performance differences between LLM baselines and humans are tested with paired Wilcoxon signed-rank tests because evaluation scores are non-normally distributed.The procedure first computes uncorrected p-values across all metrics.Source: §E.1 · p. 33
  • Holm-Bonferroni correction is applied separately for each model–metric pair across five venues, controlling FWER and requiring cross-venue consistency.Each hypothesis family contains N = 5 venue comparisons, reducing the chance that significance reflects a single-venue success.Source: §E.1 · p. 33
  • Results report rank-biserial effect sizes and Holm-adjusted significance, with ns indicating human-level indistinguishability and starred results indicating corrected differences.The notation uses ∗ for pholm < 0.05, ∗∗ for pholm < 0.01, and ∗∗∗ for pholm < 0.001.Source: §E.1 · p. 33

E.2 Depth of Analysis · E.3 Novelty Assessment

Depth of Analysis varies primarily with evidentiary grounding and premise coverage: CycleReviewer and DeepReview reach human-comparable performance, while unsupported verbosity and surface-level focus reduce scores. Novelty claims are often retrievable and well grounded, with SEA leading several venues, but scalar scores do not establish objective novelty or human-level claim agreement.

E.2 Depth of Analysis

  • CycleReviewer and DeepReview achieve Depth of Analysis comparable to human reviewers across venues, whereas TreeReview, SEA, and Reviewer2 score significantly lower (p < 0.005).The parity is attributed to robust internal grounding mechanisms and high premise ratios.Source: §E.2 · pp. 33–34, Table 4 · p. 33
  • DoA differences reflect the interaction between Premise Ratio and Grounding Score, with CycleReviewer and DeepReview consistently closest to humans across aspects.Their higher evidentiary density drives cross-aspect parity, while lower premise ratios produce globally weaker performance for Reviewer2, SEA, and TreeReview.Source: §E.2 · pp. 34, 36, Figure 24 · p. 36
  • Reviewer2’s externally grounded premises do not offset its many vague statements and severely low Premise Ratio, producing an illusion of analytical depth.TreeReview instead illustrates a surface-level trap by assigning 22.9% of premise-level effort to Clarity, yielding the highest human-distribution JSD of 0.111 and only 44.7% Methodology coverage.Source: §E.2 · pp. 34–35, Figure 22 · p. 34, Figure 23 · p. 34
  • Human and LLM reviewers prioritize Methodology and Experimental Design, which receive substantially higher DoA than Novelty and Clarity across all evaluated baselines.Human DoA scores are 0.510±0.156 for Methodology, 0.456±0.207 for Experiment, 0.357±0.322 for Novelty, and 0.266 ± 0.268 for Clarity.Source: §E.2 · p. 36, Figure 24 · p. 36

E.3 Novelty Assessment

  • Novelty Assessment scores generally range from 0.730 to 0.870, indicating that human and automated novelty claims are frequently retrievable under PRISM but not objectively validated.Non-significant differences for DeepReview, Reviewer2, and TreeReview indicate similar evidence-grounding performance on this scalar metric, not claim-by-claim agreement with humans.Source: §E.3 · pp. 37–38, Table 6 · p. 37
  • SEA achieves the highest Novelty Assessment score in multiple venues and significantly exceeds humans in ICLR 2025 (pHolm < 0.01), ICLR 2026 (pHolm < 0.005), and NeurIPS 2025 (pHolm < 0.01).This suggests SEA generates claims especially easy for PRISM’s retrieval-and-verification pipeline to ground, not necessarily claims better aligned with human novelty judgments.Source: §E.3 · p. 37, Table 6 · p. 37
  • Novelty-claim volume varies substantially: DeepReview produces 8.3 claims per review, compared with SEA’s 3.0 and Reviewer2’s 3.5.The concatenated human bundle averages 5.1 claims per paper before normalization by the number of individual human reviews.Source: Figure 25 · p. 37, §E.3 · p. 38
  • Scalar novelty grounding should not be interpreted as certification of objective manuscript novelty or full replication of human novelty conclusions.Claim granularity, extraction boundaries, and downstream agreement remain distinct from how individual claims score under retrieval and verification.Source: §E.3 · p. 38

E.4 Flaw Identification & Prioritization of Major Issue

Reviewer2 leads flaw detection across venues through high-volume extraction, while all automated systems nearly match humans at prioritizing major issues. LLMs also adapt flaw categorization by severity, focusing critical critiques on technical components and minor critiques on presentation-related anomalies.

Cross-venue performance

  • Reviewer2 achieves the highest Critical and Minor flaw Recall across every evaluated venue, with Critical Recall ranging from 0.506 to 0.649 and pHolm < 0.005 versus humans.Its higher recall is associated with extracting more valid issues per review, whereas CycleReviewer and SEA generally underperform humans; TreeReview favors Minor over Critical flaws, and DeepReview often matches humans statistically.Source: Table 7 · p. 38, §E.4 · pp. 38–39
  • nCPS exceeds 0.96 for every baseline across conferences, making flaw prioritization statistically indistinguishable from humans despite differences in diagnostic capability.Automated systems consistently surface major flaws first, indicating that strategic critique ranking is a broadly solved capability.Source: Table 7 · p. 38, §E.4 · p. 39

Severity-stratified flaw topics

  • Humans allocate 92.3% of Critical critiques to Methodology and Experimental Design, closely matched by DeepReview at 91.7%, while TreeReview assigns 6.0% to critical presentation errors.Clarity accounts for 1.7% of human Critical flaws and 1.6% for DeepReview, indicating that DeepReview closely reproduces human focus on fatal scientific errors.Source: Figure 26 · p. 39, §E.4 · pp. 39–40
  • For Minor flaws, human attention to Clarity rises to 29.3%, while TreeReview reaches 38.3% and Reviewer2 17.2%, reflecting a shift toward presentation issues.Automated baselines reduce their emphasis on Methodology and increase attention to Clarity, Presentation, and Reproducibility relative to Critical flaws.Source: Figure 26 · p. 39, §E.4 · pp. 39–40
  • Across severity levels, LLMs contextually adapt flaw categorization by linking Critical issues to technical breakdowns and Minor issues to presentation anomalies, mirroring human review patterns.Models still identify minor methodological nitpicks, but generally associate a large share of Minor flaws with presentation errors rather than applying one rigid evaluation template.Source: §E.4 · p. 40

E.5 Multi-Dimensional Constructiveness

DeepReview leads constructiveness across all five venues, significantly exceeding human reviewers, while sub-dimensions reveal distinct weaknesses in human and LLM reviewing styles. Its advantage comes from combining actionable solutions with professional tone rather than merely identifying flaws or explaining them.

Cross-venue constructiveness

  • DeepReview achieves the highest MCS across all five venues, scoring 0.629–0.635 versus humans’ 0.576–0.546, with Holm-adjusted significance in every venue.Reviewer2 slightly matches the human baseline at ICML 2025 (0.584) and NeurIPS 2025 (0.586), while SEA and TreeReview remain below humans throughout.Source: §E.5 · p. 40, Table 8 · p. 40

Solution Bottleneck

  • DeepReview resolves the solution bottleneck, raising Solution Density to nearly 0.29 and D4 to near 0.80, compared with humans’ SD ≈0.102 and D2 ≈1.72.Humans identify concrete flaws effectively but only about 10% of comments contain explicit fixes, whereas DeepReview provides implementable remedies across venues.Source: §E.5 · p. 42

Verbosity Paradox of Reviewer2

  • Reviewer2’s verbosity produces D3 > 0.90 and CD ≈0.58 but leaves SD ≈0.05 and D4 ≈0.26, explaining why its critiques rarely guide repairs.Its two-stage rubric-driven generation extensively justifies claims while seldom telling authors how to fix problems.Source: §E.5 · p. 42

Actionability without Depth in CycleReviewer

  • CycleReviewer attains D2 ≈1.89 and AR > 0.90 but collapses to D3 ≈0.32 and SD ≈0.04, revealing checklist-style specificity without evidentiary or solution depth.It references concrete paper elements and offers general directions, yet rarely supplies supporting reasoning or explicit fixes.Source: §E.5 · p. 42

Tone and Professionalism

  • DeepReview also improves review tone, reaching D5 ≈1.72 versus the human baseline’s D5 ≈1.58, with more professional, neutral, and encouraging feedback.Overall, it bridges flaw identification with explicit solutions and respectful delivery, unlike baselines dominated by verbosity or superficial demands.Source: §E.5 · pp. 42–43

E.6 Review Sensitivity to Paper Quality: Accept vs. Reject Analysis · F Qualitative Analysis & Case Studies

Human reviews significantly distinguish accepted from rejected papers through novelty, prioritization, and flaw severity, whereas LLM reviews remain largely invariant across paper-quality tiers. The evaluation framework is also robust to evaluator-LLM substitution, preserving reviewer rankings and qualitative conclusions across Gemini and Mimo.

E.6 Review Sensitivity to Paper Quality: Accept vs. Reject Analysis

  • The accept–reject comparison uses two-sided unpaired Mann–Whitney U tests with Holm–Bonferroni correction, reporting accept-minus-reject differences and significance levels across metrics.Figure 29 stratifies human and LLM scores by paper decision, while Table 10 details metric differences and significance.Source: §E.6 · p. 43, Figure 29 · p. 43, Table 10 · p. 43
  • Human reviews correlate with editorial outcomes: accepted papers receive higher Novelty (+0.051*) and Prioritization (+0.006***), while rejected papers show worse Critical (−0.049***) and Minor (−0.018**) flaw scores.The Critical difference is 2.7× larger than the Minor difference, indicating severity-sensitive critique calibration.Source: §E.6 · pp. 43–44
  • Across five automated systems, review metrics are largely invariant between accepted and rejected papers, with only DeepReview’s Prioritization difference significant (∆= +0.011***).Critical scores are especially stable: Reviewer2 shows −0.058ns, TreeReview +0.043ns, and the remaining systems stay near zero.Source: §E.6 · p. 44
  • The contrasting patterns define two review paradigms: humans adapt critique severity to scientific merit, whereas LLMs act as consistent diagnostic scanners with reduced adaptive precision for fatal flaws.LLM invariance may avoid halo effects but limits organic discrimination of manuscript quality.Source: §E.6 · p. 44
  • Prioritization is evaluator-agnostic: Gemini assigns slightly higher Depth, Novelty, and Constructiveness, Mimo scores slightly higher Minor Recall, and Prioritization shows the smallest evaluator divergence.The full pipeline was rerun with Mimo v2.5 Pro and compared with Gemini 2.5 Flash Lite across six metrics and four aspects.Source: §E.6 · p. 44
  • Reviewer rankings remain consistent across evaluator backends: Reviewer2 and DeepReview lead Constructiveness and Critical Recall, CycleReviewer and SEA rank lowest overall, and SEA leads Novelty Assessment.Constructiveness is 0.629/0.570 for Gemini versus 0.587/0.586 for Mimo; Critical Recall is 0.613/0.332 versus 0.510/0.279; SEA Novelty is 0.841 versus 0.734.Source: §E.6 · pp. 44–45, Figure 30 · p. 44

F.1 Depth of Analysis

Depth of Analysis rewards reviews that combine technically grounded premises with independent analytical judgment, rather than sheer argument volume. Across NV-Embed and VLAP, DeepReview can exceed human grounding efficiency, while Reviewer2 and TreeReview expose distinct failures from self-referential or boilerplate criticism.

Case 1: The Evidentiary Collapse

  • Human NV-Embed reviewers establish the grounding standard with Rpremise = 0.673 and GS = 0.500, combining component-specific evidence with critical evaluation.They identify the latent attention layer, two-stage contrastive instruction tuning, MTEB performance, and whether the paper offers more than engineering contributions.Source: §F.1 · p. 45
  • DeepReview achieves the highest grounding quality with GS = 0.546 and exceeds Human’s premise ratio, 0.733 versus 0.673, using only 15 arguments.Its premises precisely identify architectural components and supporting ablations, illustrating precision-to-volume economy rather than exhaustive summary.Source: §F.1 · p. 45
  • DoAHM jointly penalizes low premise ratios and weak grounding, so Reviewer2’s NV-Embed review scores 0.178, 70% below Human’s 0.581.Reviewer2 produced 46 arguments but only 7 premises, with GS = 0.215; the harmonic mean requires both Rpremise and GS to be high.Source: §F.1 · p. 46

Case 2: The Surface-Level Trap in Practice

  • Reviewer2 achieves DoAHM = 0.551 on simple VLAP with 29 premises from 51 arguments (Rpremise = 0.569, GS = 0.534), all directed at Methodology and Experimental Design.Human, DeepReview, CycleReviewer, and Reviewer2 each allocate 0% of their premise budget to Clarity. CycleReviewer produces 9 premises from 17 arguments with average GS = 0.444.Source: §F.1 · pp. 46–47
  • TreeReview’s generic reproducibility premises consume 2 of 7 available premise slots and contribute nothing to analytical depth, despite four of six reviewers finding no Clarity criticism warranted.SEA’s two Clarity premises are positive quality affirmations rather than complaints; although generic and GS = 0, its three substantive Experimental premises maintain DoAHM = 0.417.Source: §F.1 · p. 47

F.2 Novelty Assessment · F.3 Flaw Identification & Major Issues Prioritization · F.4 Multi-dimensional Constructiveness

This section examines how novelty assessment, flaw identification, and constructiveness combine evidence-grounded evaluation with actionable critique. Across the cases, LLMs and humans provide complementary diagnostic coverage, while effective constructiveness depends on translating technically grounded findings into specific remedies.

F.2 Novelty Assessment

  • SEA achieved a higher raw novelty mean than Human (1.73 vs. 1.20) by focusing on specific, verifiable contributions and avoiding speculative unclear claims.Human’s plausible but unsupported dataset-overlap concern received a per-claim average of −0.18, illustrating the metric’s speculative-assessment penalty rather than overall reviewer superiority.Source: §F.2 · pp. 48–49
  • DeepReview’s limitation claims earned full novelty credit when verifiable, including gaps concerning relative positional encodings and missing empirical validation, while redundant or unsupported claims scored 0.0.This demonstrates that the metric rewards calibrated criticism regardless of whether the stance is novel or not_novel.Source: §F.2 · pp. 49–50
  • DeepReview produced 2.4× more novelty claims than Human, with nearly identical raw mean scores (1.500 vs. 1.467), by adding precise elaboration and verified scope gaps.Its additional claims identified relative positional encodings and missing empirical validation, while maintaining calibration against the related-work pool.Source: §F.2 · pp. 49–50

F.3 Flaw Identification & Major Issues Prioritization

  • On Tv2JDGw920, LLM and Human reviewers each identified approximately 9 unique valid flaws with zero overlap, revealing complementary diagnostic profiles.The LLM found equation-level issues involving data-dependent PAC-Bayes priors and gradient-independence assumptions, while humans flagged missing quantitative domain-invariance evidence and conjecture–theorem calibration.Source: Table 16 · p. 51, §F.3 · p. 51
  • LLM flaw analysis also hallucinated absent deficiencies, overlooking reported training times such as 4,292 s versus 5,443 s and existing manuscript visualizations.This failure arose when flaw claims were evaluated without cross-referencing the paper’s actual content.Source: §F.3 · p. 52
  • For aapUBU9U0D, Humans and Reviewer2 each contributed 14 valid flaws, but their union covered all four macro-categories: Methodology (7), Experimental Design (11), Clarity (3), and Applicability (7).Humans concentrated 7 of 14 flaws on Experimental Design, whereas Reviewer2 assigned 5 of 14 to Applicability and Limitations, probing scalability, generalization, and checker formalization.Source: §F.3 · pp. 52–53, Table 17 · p. 52
  • Reviewer2 identifies flaws in diagnostic categories that humans deprioritize, so combining human and LLM flaw sets broadens coverage of scope and methodological formalization gaps.The union is presented as a diagnostic-broadening mechanism that supplements rather than replaces human expertise.Source: §F.3 · p. 53

F.4 Multi-dimensional Constructiveness

  • Human reviewers achieve D2 = 1.767 but average only D4 = 0.326, typically identifying real limitations without prescribing concrete remedies.The examples concern determinism, efficiency, and dataset scope, illustrating diagnostic gatekeeping rather than guidance.Source: §F.4 · pp. 53–54
  • DeepReview reaches D1 = 1.588 and D4 = 1.059, with six ARCs scoring D4 = 2 by specifying concrete artifacts or revisions.Its recommendations include component ablations, computational-cost analysis, and clearer novelty justification.Source: §F.4 · p. 54
  • Reviewer2 has the highest raw MCS at 0.800, but D3 = 2.000 masks D1 = 1.000 and D4 = 1.000 because its observations lack implementable fixes.The D4 sub-score exposes the asymmetry between detailed justification and actionable solutions.Source: §F.4 · pp. 53–54
  • CycleReviewer and TreeReview remain weakly constructive: CycleReviewer has D4 = 0.167, while TreeReview has 29 ARCs but D4 = 0.207 and emphasizes reproducibility details.CycleReviewer’s comments are vague and unsubstantiated, whereas TreeReview’s strongest feedback avoids the paper’s core design choices.Source: §F.4 · p. 54

G Limitations

PRISM’s limitations center on limited domain coverage and residual dependence on frontier LLMs despite constrained, deterministic evaluation pipelines.

Domain Generalization

  • PRISM’s 1,000-manuscript dataset covers only premier machine learning and representation learning venues, so deployment in other disciplines may require recalibration.Different domains may have distinct structural norms, citation densities, and evaluation criteria.Source: G Limitations · p. 55

LLM Dependency: Hallucination, Prompt Sensitivity, and Judge Bias

  • Despite constrained multi-phase pipelines and deterministic decoding, PRISM remains vulnerable to context loss, retrieval-limited fact-finding, rhetorical-style priors, and model-specific judge bias.The primary pipeline uses Gemini 2.5 Flash Lite; preliminary Xiaomi MiMo V2.5 Pro checks on a subsample confirmed baseline metric stability but cannot eliminate single-judge bias.Source: G Limitations · p. 55
  • Future research should develop uncertainty quantification and comprehensive multi-judge studies across diverse LLM families to address residual extraction errors and judge-specific priors.These limitations motivate broader validation beyond the current single-judge evaluation setup.Source: G Limitations · pp. 54–55

Related papers

Loading 2605.26730v2…