PaperlayerView exact revision on arXiv

Source-linked AI summary

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2609.11234v1cs.AI

TL;DR

LLM novelty assessment lacks fine-grained, faithful diagnosis beyond holistic scores. NOVGAUGE supplies expert-anchored labels across task, problem, and method and evaluates correctness, evidence grounding, and logical support. Across 18 LLMs, faithfulness filtering sharply reduces performance, with method-level novelty especially fragile.

  • Problem

    Existing novelty benchmarks use holistic scores, making it difficult to identify which dimension an LLM misjudges or whether its evidence is faithful.

  • Method

    NOVGAUGE uses 619 paper pairs and 50 multi-paper sets with expert-anchored task, problem, and method labels and a three-stage faithfulness cascade.

  • Results

    Only 5 of 54 model-dimension pairs retain more than half of raw F1 after faithfulness filtering, with the bottleneck most severe for method-level novelty.

  • Takeaways & Limitations

    Reliable novelty assessment requires faithful links between verdicts, reasons, and evidence rather than accuracy alone.

  • Takeaways & Limitations

    NOVGAUGE evaluates judgment after candidate papers are retrieved and leaves the full retrieval process to future work.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.

1 Introduction

NOVGAUGE addresses unreliable LLM novelty assessment by diagnosing overlap independently across task, problem, and method while checking whether supporting evidence is faithful. It combines expert-anchored benchmark data with a cascading evaluation protocol to distinguish correct judgments from grounded reasoning.

  • LLM novelty assessment is unreliable, and holistic accuracy cannot reveal which novelty aspect a model misjudges or whether its evidence is faithful.
  • NOVGAUGE evaluates novelty independently across task, problem, and method, reflecting cases where papers overlap on one dimension but remain novel on others.
  • The benchmark contains 619 paper pairs and 50 multi-paper sets with fixed three-dimension labels anchored in reviewer overlap claims and verified survey co-citations.
  • Its cascading protocol checks correctness, evidence hallucination, and reasoning mismatch, counting only predictions that reach the right answer for the right reason.
  • Across 18 LLMs, correct judgments are frequently supported by fabricated or logically disconnected evidence, with failure patterns varying sharply across dimensions.

2 Related Work

Prior work spans document similarity, novelty scoring, literature-grounded assessment, and idea-level evaluation, but these benchmarks generally lack fixed per-instance diagnosis of novelty dimensions. NOVGAUGE targets this gap with independently labeled task, problem, and method overlap.

  • LLM-based peer review has progressed from single-model critique generation to multi-agent and structured-reasoning systems, motivating targeted evaluation of novelty judgment.
  • Existing similarity benchmarks measure holistic retrieval relevance, while newer systems assess generated novelty claims or manuscript and idea novelty against related work.
  • NOVGAUGE addresses these gaps with fixed per-instance ground truth across three novelty dimensions, enabling diagnosis of where and why LLM judgments fail.

3 Benchmark Construction

NOVGAUGE constructs expert-anchored pair and multi-paper datasets with fixed task, problem, and method labels. It evaluates model predictions through correctness, hallucination, and logical-mismatch stages using explicit evidence requirements.

  • Design principles: NOVGAUGE decomposes novelty into task, problem, and method, corresponding to application domain, research challenge, and technical approach.
  • Data sources: Labels come from ICLR reviewer overlap claims and annotator-verified survey co-citation groupings, with cross-group sampling supplying dimension-specific negatives.
  • Benchmark composition: The benchmark contains 619 paper-pair records, producing 875 dimension-specific evaluation instances after independently evaluating available labels.
  • Evaluation tasks: Pairwise inputs contain two papers and a target dimension, and models must output a similarity judgment, verbatim evidence from both papers, and a brief reason.
  • Evaluation tasks: Multi-paper grouping asks models to identify subsets sharing a target dimension and scores predicted and ground-truth partitions through within-group paper pairs.
  • Evaluation protocol: The cascading protocol first scores correctness, then checks whether positive outputs contain source-verifiable evidence, and finally tests whether reasons are entailed by that evidence.

4 Experimental Setup

The experiments evaluate pairwise novelty judgment across 18 LLMs, vary the amount of paper context, and test multi-paper grouping. All settings use the cascading protocol, with repeated decoding and judge-based mismatch assessment.

  • Experiments: The study runs pairwise judgment, context-granularity, and multi-paper grouping experiments, all using the cascading evaluation protocol.
  • Pairwise judgment: Pairwise evaluation spans 18 frontier, open-weight, and specialized peer-review LLMs using binary labels, reasons, evidence spans, raw F1, Hallucination Rate, Mismatch Rate, and Verified F1.
  • Granularity ablation: Context ablation compares abstract-only, abstract-plus-introduction, and full-paper inputs capped at 1k, 4k, and 16k tokens.
  • Grouping: Multi-paper grouping evaluates 12 frontier models on 85 grouping instances using full-paper inputs.
  • Evaluation procedure: Each input and dimension is decoded three times, while an external LLM judge assesses logical mismatch among non-hallucinated positive outputs.

5 Experimental Results

Across pairwise and multi-paper novelty judgments, faithfulness verification sharply reduces apparent performance, with mismatch the dominant failure mode and method-level grounding especially fragile.

  • 5.1 Main Results: The average model retains only 26% of its raw F1 after faithfulness filtering, with just 5 of 54 model-dimension cells retaining at least half.Mismatch flags 72.5% of non-hallucinated correct-positive predictions as unsupported.
  • 5.1 Main Results: Method-level novelty is the least faithful dimension: GPT-5.5 retains 84% of task raw F1 but only 55% on method.For GLM-5.1, method hallucination reaches 27.2% versus 0.0% on task.
  • 5.1 Main Results: False positives are less faithful than true positives, with hallucination rising from 6.4% to 24.9% and mismatch from 72.5% to 85.1%.Only 11% of false positives survive the full cascade, compared with 26% of true positives.
  • 5.1 Main Results: Models achieve 90–97% negative accuracy but only 47–59% positive accuracy, indicating that they more often miss genuine similarity than fabricate overlap.Within the survey source, positive accuracy is 44–59% versus 90–97% for negatives.
  • 5.2 Granularity Results: More context raises raw F1 by 3.9 points for 13 of 18 models but lowers Verified F1 by 5.0 points for 16, primarily through increased mismatch.Mismatch rises from 63.6% on abstracts to 72.6% on full papers, while hallucination rises from 0.5% to 6.3%.
  • 5.3 Grouping Results: Multi-paper grouping further reduces panel-mean Verified F1 to 17%, 8%, and 7% for task, problem, and method, while GPT-5.5 leads with 41.7%, 25.2%, and 16.7%.Grouping retains only 38–62% of corresponding pairwise values; mismatch remains high at 75%, 87%, and 86%.

6 Conclusion

NOVGAUGE evaluates novelty judgment with per-dimension labels and cascading faithfulness checks. Across 18 LLMs, faithfulness filtering sharply reduces performance, especially for method-level novelty, motivating stronger links between verdicts, reasons, and evidence.

  • Only 5 of 54 model-dimension pairs retain more than half of their raw F1 after faithfulness filtering.The faithfulness bottleneck is most severe for method-level novelty and worsens with longer paper context.
  • Method-level novelty is the most fragile setting, reflecting a gap between contribution-level reasoning and implementation-level evidence.
  • LLMs show a conservative similarity bias, more often missing genuine overlap than inventing spurious similarity.
  • Reliable novelty assessment requires faithful links between verdicts, reasons, and evidence.

Limitations

The benchmark’s conclusions are bounded by its domain coverage, evaluation dependencies, sample imbalance, possible reviewer-LLM contamination, and retrieval-stage scope.

  • Cross-disciplinary generalization is untested because all data derives from ICLR and computer-science arXiv surveys.
  • The cascading metric relies on LLM judges, and substituting another capable model can reduce comparability.
  • Uneven negative samples across dimensions, including only 33 problem negatives, yield wider variance in F1 estimates.
  • Some ICLR 2025–2026 reviews may have used LLM assistance, although their underlying novelty judgments still reflect human expertise.
  • NOVGAUGE evaluates judgment after candidate retrieval, leaving the full retrieval process for future work.

Ethics Statement

The study uses publicly available research data and frames NOVGAUGE as an evaluation-methodology resource rather than a replacement for human peer review.

  • All data comes from publicly available OpenReview submissions and arXiv papers, with no personally identifiable information beyond public-paper author names.
  • The benchmark is intended for evaluation-methodology research and does not endorse replacing human peer review with automation.
  • The evaluation covers 18 LLMs, with multi-paper grouping restricted to 12 frontier models because it requires longer context windows.
  • Papers are evaluated at abstract-only, abstract-plus-introduction, and full-paper granularities with token caps of 1k, 4k, and 16k per paper.

C Metric Definitions

The benchmark defines per-dimension novelty metrics that progressively filter predictions for evidence grounding and reasoning support. It also documents how expert-sourced pairs are extracted, verified, tiered, and audited.

  • Metric definitions: Hallucination Rate measures unsupported evidence among true positives, using exact substring and 4-gram soft matching against paper text.A span is hallucinated only when it fails both matching criteria.
  • Metric definitions: Mismatch Rate measures unsupported reasoning among non-hallucinated true positives, with an external LLM judge assessing evidential support.False-positive verification is reported separately from the true-positive rates in Tables 3 and 4.
  • Metric definitions: Verified F1 counts a true positive only when its evidence is grounded and its stated reason is supported.The cascade separates hallucinated, mismatched, and verified true positives, with TP = TPv + TPh + TPm.
  • Benchmark construction: The benchmark labels task, problem, and method overlap from reviewer claims and survey co-citations, then verifies dimensions, identities, concessions, and tier assignments.Three PhD-level annotators partition, label, and independently audit every candidate pair.
  • Benchmark construction: Pairs enter Tier-1 when multiple reviewers flag the same prior work and authors concede, while Tier-2 requires only one of those signals.Pairs with neither signal are dropped from the benchmark draft.

D.8 Year Distribution and Temporal Skew

The ICLR subset is temporally concentrated and deliberately filtered, while the survey subset uses precision-oriented extraction and verified dimension-specific groupings. These construction choices define important scope boundaries for interpreting the benchmark.

  • ICLR temporal skew: 189 of 236 ICLR pairs come from 2026, so the year distribution is not an estimate of yearly novelty-overlap prevalence.The concentration reflects the larger 2026 forum pool and prioritization of its test split.
  • ICLR temporal skew: Spearman correlations of 0.87–0.98 indicate stable model rankings after removing 2026 data.The comparison covers input settings and metrics, while Table 7 reports positive-pair metrics by year.
  • ICLR temporal skew: The released 236 ICLR pairs are a selected, content-available evaluation subset rather than an exhaustive collection of overlap concerns.The construction funnel prioritizes high-precision evaluation over prevalence estimation.
  • Survey construction: Survey candidates are filtered for explicit paper-level similarity language, excluding technical uses such as cosine or semantic similarity.Gemini 3.1 Pro proposes task, problem, and method groupings, which annotators accept or revise and then audit.
  • Survey construction: Cross-group sampling creates negative pairs only from verified multi-paper groupings, excluding pairs that appear as positives elsewhere.A single pair can receive negative labels in multiple dimensions; 56 of 156 negative pairs do so.
  • Survey construction: The survey source is skewed toward method papers and computer science, which the authors identify as a known limitation of precision-first sampling.Method papers constitute 73.3%, and computer science accounts for 86.7% overall.

H Detailed Per-Sample Results

Per-sample variation is generally small, supporting the stability of the benchmark’s conclusions. Source-level accuracy nevertheless differs substantially between positive and negative labels.

  • Per-sample variability: 0.5–3.0 percentage points of standard deviation across runs indicates robustness to sampling variation.Estimates use three independent temperature-0.6 inference runs on the same full-content test set.
  • Dataset composition: The benchmark contains 875 dimension-specific instances from 619 paper-pair records, including 649 positive and 226 negative labels.The released records include 463 positive pairs and 156 negative pairs.
  • Source-level accuracy: 44–59% positive accuracy contrasts with 90–97% negative accuracy in the survey source.ICLR positive accuracy is similar on the dimensions shared with the survey source.

I Content Granularity Ablation (Full Results)

Providing more paper content generally improves raw prediction accuracy but makes reasoning support harder to maintain. Multi-paper grouping exposes ranking changes that pairwise evaluation can hide.

  • Content granularity: Mismatch Rate rises from 38–76% with abstracts to 24–90% with full papers, despite generally improving F1.Most frontier models maintain hallucination rates below 2% across content granularities, suggesting the main degradation is reasoning mismatch.
  • Pairwise versus grouping: The top four models retain their leaderboard positions across pairwise and grouping evaluations, while the middle ranks reorder substantially.GLM-5.1 falls from fifth to eleventh, Claude Opus 4.7 reaches last place at VF1 = 2.6%, and two DeepSeek-V4 variants rise four ranks.
  • Pairwise versus grouping: Grouping separates models that sustain grounded reasoning across documents from models that rely on pairwise shortcuts.The two evaluations use the same 12 Frontier models and directly comparable Verified F1 metrics.
  • Judge validation: Human validation finds 87.0% agreement between the mismatch judge and annotators, with Cohen’s κ = 0.74.The judge achieves 78.8% precision, 95.3% recall, and 86.3% F1 for unsupported reasoning detection.

K.2 Evidence of GPT Self-Preference

Judge comparisons reveal self-preference bias concentrated in method judgments, while structured-output failures exclude two specialised peer-review models and expose limitations of task-specific fine-tuning.

  • Judge routing and self-preference: 7.3 to 20.6 percentage points higher Verified F1 on method judgments indicates GPT-5.2 self-preference relative to Gemini-3.1-Pro.Task and problem differences stayed within ±2 percentage points.
  • Judge dependence: 79–83% pairwise judge agreement and 66–76% agreement among all three judges were lowest for method judgments.The comparison used Claude Sonnet 4.6 outputs across three input settings.
  • Judge dependence: On 60 contested records, GPT-5.2 reached 80% accuracy, versus 55% for majority vote, 43% for Gemini-3.1-Pro, and 32% for DeepSeek-V4-Pro.These contested-case accuracies are not comparable to the separate 87.0% balanced human-validation result.
  • Failed specialised models: DeepReviewer-7B could not be evaluated because none of its outputs matched the required JSON structure after peer-review fine-tuning.Manual inspection found review-template prose instead of the required fields.
  • Failed specialised models: LLaMA-OpenReviewer-8B predicted false for 647 of 649 positive-pair instances and copied the prompt schema into evidence fields.Its negative-pair run contained only 60 records and was incomplete.
  • Failed specialised models: Task-specific supervised fine-tuning may erode the instruction-following and structured-output discipline required by cascading evaluation.The conclusion is drawn from both specialised-model failures.

N Illustrative Examples

The case studies show how the cascading pipeline distinguishes verified reasoning from correct labels supported by contaminated, transferred, or dimension-confused evidence. They include both a fully verified prediction and failures at hallucination and reasoning-verification stages.

  • Case A: Verified Prediction: A verified method-similar prediction passed correctness, hallucination, and reasoning checks because both excerpts described the same reduction mechanism.The example concerned non-adaptive deletion guarantees, differential privacy, and adaptive deletion guarantees.
  • Case B: Mismatched Prediction: In the IndicSuperTokenizer–SuperBPE case, the correct similar label failed reasoning verification because Paper-B implementation details were transferred onto Paper A.The mismatch involved enforcing and then lifting whitespace restrictions and related BPE details.
  • Stage 2 failures: A hallucination case copied Paper A’s evidence into Paper B’s field, causing Stage 2 to fail despite a correct similar prediction.The failure was classified as cross-paper evidence contamination.
  • Stage 3 failures: A separate method judgment confused a shared problem formulation with methodological similarity, so the reason was unsupported even though the prediction and evidence passed earlier checks.The judge identified probabilistic action corruption as a problem rather than a specific algorithmic mechanism.
Loading 2609.11234v1…