PaperlayerView exact revision on arXiv

Source-linked AI summary

Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope

arXiv:2610.02022v1cs.CL

TL;DR

Novelty judges lack standardized evaluation and are often validated on human-authored papers rather than the generated ideas they score. This study builds benchmark setups and finds that verdicts are brittle to evaluation configuration, while common fixes offer limited help.

  • Problem

    Novelty judges lack a standard benchmark and are often validated on small, outdated, or human-authored sets that differ from generated ideas.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The authors mine explicit reviewer novelty signals to build human-only and human-plus-generated benchmarks, then systematically vary judge and evaluation design choices.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Small setup changes alter verdicts on more than half of identical pairs and shift accuracy by more than 50 points; retrieval and increased reasoning offer limited help.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Ad hoc novelty judgments make findings difficult to compare and their reliability questionable.

    Source: §2 · p. 3

    The Human+Generated setup assumes every generated idea is less novel than every validated novel paper.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Automated ideation systems are often evaluated on the novelty of the ideas they produce, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We first build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.

1 Introduction

Novelty judges are often ad hoc and validated on human-authored papers, despite their use on generated ideas. This study builds a review-based benchmark and finds that judge verdicts are highly sensitive to evaluation setup, especially when generated ideas are included.

1 Introduction

  • Novelty judges are commonly built ad hoc, with design choices rarely tested or justified, and validation often uses human-authored papers rather than generated ideas.The latter are a different distribution from the ideas judges are applied to.Source: §1 · pp. 1–2
  • The benchmark mines explicit reviewer judgments of originality, retaining papers with unanimous agreement at the top or bottom of their venue and adding generated ideas.The released pipeline and data are reusable for future conference data.Source: §1 · p. 2
  • A reviewer-cue prompt changed verdicts on more than half of identical pairs and swung accuracy by more than 50 points; judges grew more brittle on generated ideas.No single fix helped across judges, and reasoning effort or retrieval had limited effects and could degrade some models.Source: §1 · p. 2
  • Greater complexity and monetary investment in dedicated novelty evaluators do not necessarily produce better novelty verdicts.Retrieval and increased reasoning effort have limited effects and can degrade some models.Source: §1 · p. 2
  • The controlled study spans six judge backbones, pointwise and pairwise judgments, prompt wording, retrieval, reasoning effort, and idea format.It also compares human-authored ideas with evaluations that include LLM-generated ideas.Source: §1 · p. 2

2 Related Work

Prior work uses novelty to evaluate ideation systems but lacks a shared benchmark or standard, and existing judges are assembled with ad hoc choices. This paper targets that gap with a systematic study and an automatically collected evaluation set.

2 Related Work

  • Prior ideation studies lack a standard novelty benchmark, evaluation set, or metric, and ad hoc judges complicate comparison between systems (Si et al., 2024; Radensky et al., 2026).Their design choices are often unjustified and may not account for bias.Source: §2 · p. 3
  • The paper addresses the gap with a systematic study of automated novelty evaluation and reports how design choices affect different judges and ideation models.Its data collection process mines OpenReview novelty labels and pairs them with weakly labeled ideas from an LLM ideator.Source: §2 · p. 3, Figure 2 · p. 3
  • Prior evaluations of generated ideas rely on small, highly specific human-annotated sets, and no prior work systematically studies their automatic novelty evaluation.Examples include a student-annotated set focused on one generation system and seven topics (Si et al., 2024), and a checker evaluated on ideas from one system (Shahid et al., 2025).Source: §2 · p. 3
  • LLM judges are known to shift with presentation order, judgment format, and surface wording, but these effects have received little examination in novelty judgment.The paper evaluates those variations in the novelty setting.Source: §2 · p. 3

3 An Automated Pipeline for Creating Novelty Evaluation Benchmarks

The paper builds a reusable benchmark pipeline that mines novelty judgments from peer reviews and organizes human and generated ideas into pools for pointwise and pairwise evaluation. Its two data setups share validated novel human ideas but use different sources for lower-novelty ideas.

3 An Automated Pipeline for Creating Novelty Evaluation Benchmarks

  • The automated collection pipeline is designed to scale and can be rerun on future conference reviews, keeping the benchmark evergreen.It is not tied to ICLR and applies to venues that publish free-text reviews.Source: §3 · p. 4

3.1 Labeled Ideas Collection

  • Novelty labels come from review passages where reviewers affirm or dispute originality, rather than from acceptance alone or outdated venue scores.The pipeline extracts novelty signals from free-form review text.Source: §3.1 · p. 4
  • Validated high-novelty ideas are abstracts of accepted papers in the top decile whose reviewers mostly affirm novelty and none dispute it.The papers must also clear a contribution-score floor.Source: §3.1 · p. 4
  • The generated pool assumes every generated idea is less novel than every validated novel idea, using a deliberately non-frontier model prompted once without tools, scaffolding, feedback, or literature access.The generated ideas are paired with high-novelty human ideas from the validated pool.Source: §3.1 · p. 4
  • A blind expert check of 30 difficult Human+Generated pairs sided with the human idea in every case, but the authors characterize it as a sanity check, not validation of every label.Twenty-eight pairs were errors by the strongest pairwise judge under the reference configuration.Source: §3.1 · p. 5

3.2 Benchmark Construction

  • The benchmark’s Human-Only and Human+Generated setups share the novel pool and differ in whether the lower-novelty pool contains human papers or generated ideas.Each setup pairs its pools for evaluation.Source: Table 1 · p. 4, §3.2 · p. 5
  • Pointwise evaluation labels each idea by its pool, while pairwise evaluation matches a novel idea with a lower-novelty idea from the same research area.In Human-Only, a lower-novelty idea can be reused in multiple matches.Source: §3.2 · p. 5
  • The benchmark presents each idea as a single abstract, removing human abstracts’ experimental outcomes and superiority claims while generated ideas already use that format.This normalization is intended to remove potential evaluation artifacts.Source: §3.2 · p. 5

4 Experimental Setup

The study varies one novelty-judgment design choice at a time across two judgment formats, six judge backbones, and both data setups. It compares prompted judges with dedicated evaluators and an agentic retrieval judge, using specified classification, accuracy, and significance measures.

4 Experimental Setup

  • Each run changes one design choice relative to a reference configuration while holding the rest fixed, across both data setups and the study’s judgment formats and judge backbones.The controlled dimensions include prompts, retrieval, reasoning effort, idea format, verdict aggregation, and the source of lower-novelty ideas.Source: §4 · p. 6, Table 2 · p. 7

4.1 Reference Configuration

  • The reference configuration asks for a novelty verdict and reasoning, uses high reasoning effort when available, and provides no literature search.Pointwise decisions aggregate three calls; pairwise decisions use three calls in each presentation order and select the idea with the higher score.Source: §4.1 · p. 6

4.2 Explored Evaluation Dimensions

  • Prompt variants test small changes to novelty definitions and whether pairwise judgments invoke reviewers’ novelty assessments or ask directly which idea is novel.One variant frames the choice as which idea reviewers judged more novel.Source: §4.2 · p. 6
  • Other controlled changes add five retrieved abstracts, lower reasoning effort, reformat ideas as purpose/mechanism plans, or reduce the number of aggregated calls by three.The idea-format change preserves idea content while changing presentation.Source: §4.2 · pp. 6–7
  • The study also varies the generated lower-novelty pool across three additional generator backbones and uses judges whose training cutoffs predate the evaluated submissions.The novelty labels come from reviews released after the submission deadline.Source: §4.2 · p. 7

4.3 Dedicated Novelty Judges

  • The dedicated-judge comparison includes a literature-grounded novelty checker and an AI-Scientist ReAct agent that searches Semantic Scholar iteratively.The agent can decide based on retrieved papers or continue searching.Source: §4.3 · pp. 7–8

4.4 Evaluation Metrics

  • Pointwise performance is reported as per-class and macro F1, while pairwise performance uses soft-accuracy and strict-accuracy to handle ties differently.Significance is assessed with paired non-parametric bootstrap confidence intervals over per-metric differences from the reference configuration.Source: §4.4 · p. 8

5 Results

Novelty judgments change sharply with prompts, data distribution, and judge backbone, even when the ideas are unchanged or reformatted. Reasoning effort and retrieval offer limited, model-dependent gains, while judges often struggle even on highly separable human-authored ideas.

5 Results

  • Judges underperform even in Human-Only comparisons, where ideas come from opposite ends of the ICLR rating distribution and should be maximally separable.Source: §5 · p. 8
  • A 52.6-point accuracy gap separates two prompts on identical Human+Generated pairs, with one prompt pushing gpt-5.4 below chance.The reviewer-informed prompt improves every judge, by more than 20 points for some; the alternate prompt asks which idea is novel without referring to prior human judgment.Source: §5 · p. 8
  • After all ideas are rewritten into a shared plan format, the direction of major shifts persists and some changes flip over two thirds of verdicts.This tests whether stylistic differences between human-written and LLM-generated ideas explain the shifts.Source: §5 · p. 8
  • A prompt change can be neutral or beneficial for one judge and destructive for another, so validation on one judge offers no guarantee for the next.Retrieval, for example, is a large gain for claude-opus-4-5 on Human+Generated but neutral for other judges.Source: §5 · p. 8
  • Lowering reasoning effort from high to low costs at most 7 performance points, while retrieval improves performance by at most 16 points for some models.Reasoning changes are usually neutral, and retrieval never brings absolute performance near 100; with other ideation backbones, retrieval can significantly degrade performance.Source: §5 · p. 8
  • For gpt-5.4, switching from P3 to P4 changes strict accuracy by approximately 7 points on Human-Only versus 52.6 points on Human+Generated.The prompt change therefore has much larger consequences when judging generated ideas than when validating on human-authored pairs.Source: §5 · p. 8
  • Removing instructions against trivial or additive novelty harms every judge on AI-generated pairs but is mostly neutral on human-authored pairs.The authors suggest this might indicate that LLM-generated ideas lean more toward this style of ideation.Source: §5 · p. 8
  • Judges generally degrade when lower-novelty ideas are LLM-generated rather than human-authored, with pairwise accuracy falling below chance against some ideation backbones.Source: §5 · p. 8

Appendix C.2.1).

Novelty judges remain sensitive to evaluation setup, and stronger retrieval or more expensive dedicated judges do not reliably improve their verdicts. The paper also finds greater uncertainty on generated ideas, making validation on human-written papers alone an unreliable guide to behavior on generated ideas.

Appendix C.2.1).

  • Pairwise judges tie on up to 35% of pairs for claude-opus-4-6 and more than half for claude-sonnet-4-5, with tie rates higher on Human+Generated.Ties can substantially change system rankings; disabling verdict aggregation more than doubles tie rates in some cases and contributes to the Human+Generated performance drop.Source: Appendix C.2.1 · p. 9
  • Pairwise judging is generally more stable than pointwise judging, especially on Human-Only data.The appendix also reports results without verdict aggregation and analyzes tie rates.Source: Appendix C.2.1 · p. 10

5.1 Retrieval’s Limitations

  • Retrieved papers can lead judges to dismiss contributions reviewers considered novel or to label known compositions novel, even when the retrieved work does not contradict the novelty assessment.The authors’ qualitative analysis examined 10 cases where retrieval flipped an otherwise consistently correct pointwise verdict to an incorrect one.Source: §5.1 · p. 10
  • 0.78 macro-F1 on Human+Generated leaves the agentic judge 11 points behind claude-opus-4-6 and within noise of claude-opus-4-5 and gpt-5.4.The agentic judge costs 10× more per decision; all judges are called once per idea without verdict aggregation.Source: Figure 5 · p. 10, §5.1 · p. 10

5.2 Dedicated Novelty Judges

  • 8–29 macro-F1 points separate the cheapest prompted configuration from both dedicated judges across settings and backbones, while it costs up to 32× less.The Pareto front consists entirely of simple prompted judges.Source: Figure 6 · p. 11, §5.2 · p. 11

Conclusions

  • Small prompt edits can swing pairwise accuracy by more than 50 points, while the same edit helps one backbone and pushes another below chance.The authors conclude that retrieval and higher reasoning effort buy little, and the tested dedicated judges are outscored by the cheapest prompted configuration at a fraction of the cost.Source: Conclusions · p. 11
  • Human-paper or single-system validation says little about pairwise judges’ behavior on generated ideas, where small setup changes can shift performance by tens of points.The authors recommend interpreting automated novelty assessments with significant caution until evaluation protocols account for these sensitivities.Source: Conclusions · p. 11

A Additional Benchmark Details

The appendices outline supplementary benchmark and judge details, including human-data collection, benchmark construction, expert validation, and judge implementations.

A Additional Benchmark Details

B Additional Judge Details

  • Appendix B lists additional details on prompted judges, the strong retrieval judge, and dedicated novelty judges.The listed subsections are B.1, B.2, and B.3.Source: B Additional Judge Details · p. 14

C Controlled Evaluation Study Details

Appendix C is organized around controlled-study details and a list of prompts used in the study.

C Controlled Evaluation Study Details

  • Appendix C.1 covers implementation details, C.2 additional results, and C.3 cases where retrieval hurts.The contents list gives their starting pages as 19, 23, and 31, respectively.Source: C Controlled Evaluation Study Details · p. 14

List of Prompts

  • The prompt list includes items for novelty-signal extraction, removing evaluation information, idea generation, and reference judges.It also lists prompts for retrieval contribution and retrieval-query generation.Source: List of Prompts · p. 14

A.1 Human Data Collection

The appendix details how the authors select human ideas, form Human-Only and Human+Generated benchmarks, and check generated-pair labels. It also specifies judge prompts and the strong retrieval judge’s search constraints.

A.1 Human Data Collection

  • High- and low-novelty human papers are selected from ICLR 2026 using area-relative rating and contribution thresholds plus majority-consistent explicit novelty signals.The pipeline extracts review signals, excludes the heterogeneous “other topics” area, and removes evaluation information from retained ideas.Source: §A.1 · p. 15

A.2 Benchmark Data Setups Construction

  • Human-Only pairs human ideas within their primary ICLR area, while Human+Generated pairs each high-novelty human idea with an LLM-generated counterpart seeded by its area.The Human-Only pool has more high-novelty than low-novelty ideas, so a low-novelty idea may be paired more than once.Source: §A.2 · p. 15

A.3 Expert Validation

  • A blind expert study matched the benchmark labels on all 30 Human+Generated pairs, including 28 pairs misclassified by the top-performing pairwise judge.The expert did not know which two pairs were controls.Source: §A.3 · p. 15

B.1 Prompted Judges

  • Prompted judges use majority voting for pointwise evaluations and reverse-order presentations for pairwise evaluations to mitigate position bias.The listed procedure calls pairwise judges three times in each presentation order.Source: §B.1 · pp. 15, 18
  • The extraction prompt retains verbatim, explicit novelty judgments and excludes generic praise, descriptions of a paper’s actions, and reviewer uncertainty.Positive and negative snippets are classified by meaning regardless of whether they appear in strengths or weaknesses.Source: §B.1 · p. 16

B.2 Strong Retrieval Judge

  • The strong retrieval judge searches arXiv without a search or result cap, but excludes papers after 2025-03-01 and the idea’s own preprint.It uses gpt-5.6-sol at xhigh reasoning effort and is called once per idea without verdict aggregation.Source: §B.2 · p. 18

B.3 Dedicated Novelty Judges

This section describes the configurations used to benchmark two dedicated novelty judges. Across both tested backbones and settings, these judges are the most expensive and least accurate configurations.

B.3 Dedicated Novelty Judges

  • Both dedicated judges use claude-opus-4-6 and gpt-5.4 at high reasoning effort, without verdict aggregation, and conduct their own Semantic Scholar retrieval.Searches are limited to papers before 2025-03-01 and exclude titles matching the source paper.Source: §B.3 · p. 18
  • The Idea Novelty Checker uses its defaults, including Specter2 filtering and RankGPT reranking, while AI-Scientist can run up to 10 search rounds.RankGPT reranking uses gpt-5.4-mini under both backbones.Source: §B.3 · p. 18
  • Across both backbones and settings, dedicated judges were the most expensive and least accurate configurations; for gpt-5.4, every prompted configuration outscored both dedicated judges at roughly 6× lower cost.Source: §B.3 · p. 18, Figure 7 · p. 19
  • The generated idea prompt requests a fluent, unstructured abstract capped at 200 words and explicitly excludes evaluation results, benchmark comparisons, metrics, and dataset sizes.The prompt frames the generator as producing a novel idea as a research-paper abstract.Source: §B.3 · p. 19

C.1 Implementation Details

The implementation details specify prompt variants, idea formatting, and retrieval procedures for the controlled evaluations. They also describe how changes from the reference configuration are tested for statistical significance.

C.1 Implementation Details

  • Five judge-prompt variants modify the reference prompts or use a separate lineage, including prompts that presume reviewer novelty judgments and prompts that do not.P1 removes novelty guardrails; P3–P5 adapt a prompt lineage from Si et al. (2024), with P3 and P5 presuming prior judgments and P4 not doing so.Source: §C.1 · pp. 19, 23
  • The idea-format variant uses claude-opus-4-6 to convert each free-form idea into a two-field structured research plan before judging.The resulting plans are treated as the ideas given to judges.Source: §C.1 · p. 23
  • For retrieval, the system extracts up to three contributions, generates five targeted queries, and retrieves five eligible abstracts after filtering by the 2025-03-01 cutoff.For each query, the highest-relevance remaining candidate is retained.Source: §C.1 · p. 23
  • Changes from the reference configuration are assessed with a two-sided paired bootstrap over evaluation instances and corpus-level metric differences.The procedure uses 100,000 resamples and BCa 95% confidence intervals; an asterisk marks intervals excluding zero.Source: §C.1 · p. 23

C.2 Additional Results

Additional results examine how novelty-judgment patterns vary across metrics, class-wise F1, verdict aggregation, prompt formats, generator choice, and retrieval. They report preserved or changed trends across these analyses, including more frequent ties with LLM-generated D− ideas.

C.2 Additional Results

  • Under soft accuracy, judges score higher, but ablation patterns remain similar, with high variance and lower perturbation sensitivity for Human-Only data.Soft accuracy counts ties as half-correct.Source: §C.2 · p. 23
  • Per-class pointwise ablations are reported using F1 for both the novel and not-novel classes.Figure 9a reports novel-class F1 and Figure 9b reports not-novel-class F1.Source: §C.2 · p. 23
  • Judges tie more on average once D− is LLM-generated.Table 6 reports exact tie rates by configuration and model; Figure 10 summarizes tie rates across the Figure 4b configurations.Source: §C.2 · p. 23
  • claude-sonnet-4-5 tied on more than half of the pairs in some configurations, and verdict aggregation substantially reduced ties.Source: §C.2 · p. 30
  • Without verdict aggregation, GPT-5.4’s Human+Generated P3 gain fell from +15.6 to +12.3 points, while its P4 loss deepened from −37.0 to −46.8 points.Pointwise scores mostly dropped by a few points, though some rose slightly.Source: §C.2 · p. 30
  • Research-plan prompts preserved the variant trends: P3 improved every judge on Human+Generated, P5 degraded every judge in both settings, and P4 dropped GPT-5.4 by −55.8 points.Scores were generally a bit lower than with abstracts.Source: §C.2 · p. 31

C.2.1 Different Ideation Models

  • LLM-generated negatives hurt pairwise accuracy most, falling below chance in several cases, while pointwise macro-F1 degraded more mildly and sometimes improved.Most pointwise judges rated ideas from claude-sonnet-4-5 no less novel than ideas from the stronger claude-opus-4-5, so testing against human ideas or one generator alone may be insufficient.Source: §C.2.1 · p. 31
  • Retrieval gains across four ideation backbones were small and inconsistent, and retrieval sometimes significantly degraded performance.This test used one judge from each model family, claude-opus-4-6 and gpt-5.4, rather than the full judge-generator cross-product.Source: §C.2.1 · p. 31

C.3 When Retrieval Hurts

The retrieval analysis examines cases where adding related work reverses a correct verdict. It identifies distinct ways retrieved papers can shift judges toward rejecting human-judged novel ideas or accepting generated ideas.

C.3 When Retrieval Hurts

  • The authors inspect five flips from each class among cases where gpt-5.4’s pointwise Human+Generated verdict was correct in all three reference calls and incorrect in all three retrieval calls.They selected these cases to reduce the chance that aggregated flips reflected sampling noise.Source: §C.3 · p. 31
  • Retrieved work can hurt in opposite directions: a close precursor or component-wise matching can make a human-judged novel idea seem incremental, while retrieval can make a generated idea seem novel when no retrieved paper covers it fully.The retrieval prompt defines novelty relative to supplied related work, which the authors say likely encourages over-reliance on retrieved papers.Source: §C.3 · p. 33
  • When retrieval harms a novel idea, a judge may treat one paper sharing its headline framing as evidence that the remaining contribution is incremental.Alternatively, it may map separate components to different papers and dismiss their combination, despite reviewers describing it as novel.Source: §C.3 · p. 33
  • For generated ideas judged correctly without retrieval, retrieved papers can displace the judge’s internal knowledge and lead it to infer novelty from the absence of a full match.This pattern underlies retrieval-induced flips from non-novel to novel judgments.Source: §C.3 · p. 33
Loading 2610.02022v1…