PaperlayerView exact revision on arXiv

Source-linked AI summary

Do AI Scientists Know What's New?

Miri Liu, Xiayu Zhao, Encheng Xie, ChengXiang Zhai

arXiv:2610.36511v1cs.CLcs.AI

TL;DR

Whole-proposal novelty checks can make it difficult to identify established contributions in proposals that also contain novel work. The paper evaluates checks with temporal ground truth at proposal and claim levels, then introduces CLAIMNOV, a claim-level novelty module. It reports high detection of disguised papers alongside low false alarms on later-realized ideas, and retention of proposals containing a spliced prior contribution.

  • Problem

    Whole-proposal checks make novelty judgments difficult and obscure whether they identify the right prior work, despite many papers combining novel and established contributions.

    Source: §1 · p. 2

  • Method

    The paper audits seven existing checks with temporal ground truth at proposal and claim levels and introduces CLAIMNOV, which bases its verdict on claim-level evidence.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    CLAIMNOV catches 97% of disguised papers while rejecting 1.5% of later-realized ideas, and keeps 91 of 100 proposals with one spliced prior contribution versus 76 for the strongest existing check.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Claim-level evidence lets CLAIMNOV retain proposals with established contributions, and the authors suggest a specific claim-level check could serve as a reward when training AI-scientist agents.

    Source: §6 · pp. 8–9

    The evaluation uses heuristic labels rather than a true dataset, fixed prior-work pools that limit retrieval evaluation, and LLM-generated proposals judged by the same model, creating possible bias.

    Source: Limitations · p. 9

Abstract

from arXiv · show

A major challenge facing AI scientific discovery pipelines is how to determine when a generated idea is meaningfully novel. Ideally, novelty checks in these pipelines must be able to reject non-novel ideas while not rejecting too many novel ideas, and they should also be able to distinguish which parts of a proposal are non-novel, as opposed to opaquely discarding entire proposals. We design an analytical framework for evaluation of novelty checks with temporal ground truth at both the proposal- and claim-level, using four datasets of 1,060 proposals built from published papers, reviewer disputes, and generated ideas matching ones the field went on to publish. Auditing existing novelty checks with this framework, we find that checks with high rejection rates on non-novel work also tend to have high false alarms on ideas that are later realized by human researchers, and observe that checks built to evaluate proposals as a whole can point to claim-level prior art only by rejecting the whole proposal. We therefore introduce ClaimNov, a fine-grained, interpretable claim-level novelty module. ClaimNov catches 97% of disguised papers at a 1.5% false-alarm rate, and its claim-level novelty profile cuts the LLM judge's false alarms from 11.8% to 1.5%. Because its verdict rests on claim-level evidence, ClaimNov also keeps proposals that build on established contributions, rejecting only 3% of ICLR-accepted proposals with a reviewer-disputed contribution and keeping 91 of 100 novel proposals with one spliced prior contribution, against 76 for the strongest existing check. Code and data are available at https://github.com/emmatliu/claimnov.

1 Introduction

AI scientific discovery systems increasingly generate and filter research ideas, but judging novelty at the whole-proposal level can obscure which contributions are already established. The paper argues for claim-level evaluation and introduces CLAIMNOV to identify prior art while retaining novel work.

1 Introduction

  • CLAIMNOV rejects 367 of 380 disguised proposals, names the source for 366, and rejects only 1.5% of 400 later-realized ideas.Figure 1 compares CLAIMNOV with ScholarEval at its default rule.Source: Figure 1 · p. 1
  • Whole-proposal novelty checks face difficult, costly, and disagreement-prone judgments, can conflict with incremental science, and obscure whether cited prior work is the right match.The paper proposes claims, rather than proposals, as the unit of novelty because papers may contain both novel and established contributions.Source: §1 · p. 2
  • The authors introduce CLAIMNOV, an interpretable claim-level novelty module, and an evaluation framework using temporal ground truth at proposal and claim levels across four datasets totaling 1,060 proposals.They audit seven existing checks; those rejecting more non-novel work also tend to reject more ideas later realized by human researchers.Source: §1 · p. 2
  • CLAIMNOV keeps 91 of 100 novel proposals with one spliced prior contribution, compared with 76 for the strongest existing check.Given the same attribution ranking, it also keeps more correctly attributed proposals.Source: §1 · p. 2

2 Related Work

Prior work measures novelty through bibliometric, text-based, and combined signals, while literature-grounded systems increasingly assess claims against retrieved prior work. This paper focuses on novelty checks deployed in AI-scientist pipelines and evaluates counterfactual changes rather than alignment with explicit human novelty labels.

2 Related Work

  • Traditional novelty measures use citation combinations, disruptiveness, semantic distance, new concepts, or new question–method combinations; some combine citation and text data.Such measures are often validated against expert judgments and later impact.Source: §2 · p. 2
  • Literature-grounded approaches range from sentence- and document-level detection to LLM methods comparing ideas, facets, or contribution claims with retrieved prior work.Benchmarks use expert annotations, review text, future papers, or controlled counterfactual changes.Source: §2 · p. 2
  • AI-scientist pipelines use hard retrieval filters, advisory or score-based judgments, and rankings, whereas this work focuses specifically on novelty rather than full-paper peer review.The evaluation targets deployed pipeline checks and counterfactual changes rather than agreement with explicit human novelty labels.Source: §2 · p. 3

3 Novelty Module

CLAIMNOV assesses novelty by extracting contribution claims, matching them with same-dimension prior claims, and supplying claim-level evidence to a verdict judge. It aggregates claim coverage into a proposal-level novelty score while retaining the detailed profile.

3 Novelty Module

  • CLAIMNOV is a drop-in, interpretable claim-level module that extracts typed claims, retrieves same-dimension prior claims, scores entailment, aggregates novelty, and gives a full profile to a verdict judge.The judge returns a binary decision and, for a failed proposal, the prior-art paper, subset of claims, and a reason.Source: §3 · p. 3, §3.5 · p. 4

3.1 Claim Extraction

  • Proposal titles and descriptions are converted into 1–4 contribution claims typed by dimension and attribute-value pair; proposals average 2.4–2.7 claims.Dimensions are Problem, Methodology, Resource, and Result; pool papers are processed once using their abstracts.Source: §3.1 · p. 3

3.2 Claim Retrieval

  • For each claim, retrieval embeds its text with the paper title and selects nearest pool claims of the same dimension, keeping at most one per paper.k is at least 10 and scales as 0.2 times the number of pool papers with a claim in that dimension, capped at 40.Source: §3.2 · p. 3

3.3 Claim Entailment

  • An LLM rates how strongly each retrieved prior claim entails or covers a proposal claim on a five-level scale, mapped to the unit interval.The scale runs from unrelated to full coverage of the claim’s substantive content.Source: §3.3 · p. 3

3.4 Novelty Aggregation

  • Rank-discounted entailment scores yield per-claim coverage, then dimension and claim-importance weights combine claims into proposal coverage and a normalized novelty score.The rank discount balances the strongest match against weaker matches, while the number of retrieved claims is capped to avoid inflated coverage when few are available.Source: §3.4 · pp. 3–4

3.5 Novelty Verdict

  • A stronger judge receives a profile listing each proposal claim, matched prior claims, their sources and abstracts, and entailment levels, then returns pass or fail.A failed verdict also identifies prior art, the affected claim subset, and a brief reason.Source: §3.5 · p. 4

4 Experimental Setup

The experimental setup standardizes inputs and outputs across novelty checks, evaluates selected systems under a shared temporal cutoff, and tests proposal- and claim-level judgments using four constructed datasets. Its metrics assess rejection, false alarms, pool sensitivity, and attribution.

4.1 Evaluation Contract

  • Checks receive a title, a 150–400-word proposal description, and a prior-art pool; outputs are compared at verdict, novelty-score, and claim-profile levels.Existing checks emit verdicts only, while CLAIMNOV’s claim profile determines its score and verdict.Source: §4.1 · p. 4

4.2 Evaluated Systems

  • The evaluation covers seven existing checks, including pipeline components, extracted pipeline judgments, and a claim-level reference baseline, alongside CLAIMNOV and its ablations.Embedded searches are replaced by dense retrieval over the controlled pool.Source: §4.2 · pp. 4–5

4.3 Judge Models and Temporal Validity

  • Checks using LLM judges are evaluated with models whose training cutoffs precede March 2026 to mitigate recognition of post-cutoff papers from training data.Sonnet 4.6 is the common final verdict judge; gpt-4o-mini handles CLAIMNOV’s earlier lightweight stages.Source: §4.3 · p. 5

4.4 Data Collection

  • Four datasets cover 380 disguised papers, 160 reviewer-disputed claims, 400 later-realized ideas, and 100 hybrid proposals with a spliced prior contribution.Disguises use four levels; later-realized ideas have contributions established by post-cutoff citing papers but not by the pool.Source: §4.4 · pp. 5–6

4.5 Metrics

  • Whole-proposal metrics measure catching non-novel ideas, false alarms on later-realized or source-removed disguises, informedness, and the change in rejection when the source is removed.Claim-level evaluation measures whether the top-ranked paper is the disputed prior art and reports rejection rates by ICLR decision.Source: §4.5 · p. 6

5 Results

CLAIMNOV balances detection of disguised work with retention of later-realized ideas, while claim-level evidence helps distinguish established contributions from proposals that should be rejected. Results also compare its attribution and hybrid-proposal performance with existing checks.

5.1 Whole-Proposal Evaluation

  • Si et al. catches all 380 disguises but rejects 359 after source removal; Si et al., ResearchAgent, AI Scientist v1, and AI-Scientist-v2 reject 96%, 98.5%, 52%, and 39% of later-realized ideas.ARC and ScholarEval reject 0.0% but reject almost nothing.Source: §5.1 · p. 6
  • CLAIMNOV catches 367 of 380 disguises while rejecting 1.5% of later-realized ideas, with the highest J (+0.95) and ∆(+0.92).CoI-Agent catches 368 disguises and rejects 3.8% of later-realized ideas.Source: §5.1 · p. 6, Table 1 · p. 7
  • Removing CLAIMNOV’s claim-level novelty profile raises the verdict judge’s false-alarm rate from 1.5% to 11.8%; a rule failing any entailed claim raises it to 56.8%.The per-claim rule catches all 380 disguises but wrongly rejects 197 of 380 proposals after source removal.Source: §5.1 · p. 7

5.2 Claim-Level Prior Art

  • CLAIMNOV rejects 3% of accepted ICLR proposals, the lowest rate among checks that catch disguises; existing whole-proposal checks name prior art only alongside rejection.CoI-Agent passes 89%, so it can name reviewer-disputed paper B for only the remaining 11%.Source: §5.2 · p. 7
  • With the same dense-ranking attribution, CLAIMNOV passes and names reviewer-disputed paper B for 36% of proposals, versus 31% for CoI-Agent.CLAIMNOV’s profile alone ranks B first for 21% of proposals, compared with 37% for whole-proposal dense embedding.Source: §5.2 · p. 7

5.3 Hybrid Proposals

  • On hybrid proposals, CLAIMNOV passes 91 of 100, compared with 76 for CoI-Agent; with shared ranking, it passes 67 with B named, versus 53 for CoI-Agent.The hybrids splice a contribution from known paper B into a later-realized novel idea.Source: Table 3 · p. 8, §5.3 · p. 8
  • When CLAIMNOV extracts the spliced contribution as its own claim, its entailment judge rates B at level 4 or 5, and the verdict treats it as one established claim among novel ones.Thirteen of the 15 papers rated level 4 or above are B; only 9 of 400 later-realized ideas reach that entailment level.Source: §5.3 · p. 8

6 Conclusion

The paper establishes a temporal-ground-truth framework for auditing novelty checks at proposal and claim levels, then introduces CLAIMNOV as a claim-level module. The authors conclude that it judges novelty against specific prior papers and can keep novel and partially novel proposals.

6 Conclusion

  • The evaluation framework uses temporal ground truth at both proposal and claim levels to audit seven existing novelty checks.The checks that reject more non-novel work also tend to reject more ideas later published by the field.Source: §6 · p. 8
  • CLAIMNOV is a drop-in claim-level novelty module that rejects non-novel ideas, keeps novel and partially novel ones, and retains more correctly attributed proposals given the same attribution ranking.The authors suggest its claim-level profile could serve as a reward for training AI-scientist agents.Source: §6 · p. 8
  • CLAIMNOV judges scientific novelty claim by claim against specific prior papers, potentially making AI scientists better novelty judges.Source: §6 · p. 9

Limitations

The evaluation relies on heuristic novelty labels rather than a true dataset and is limited in field and language coverage. Fixed prior-work pools also constrain retrieval assessment, while evaluating rejection-and-regeneration effects is left largely to future work.

Limitations

  • The proposal groups are defined with heuristics rather than a “true” dataset, making the ground truth less clean.The authors also note that novelty judgments about proposal sets are not as clean as a true dataset.Source: Limitations · p. 9
  • The study covers AI research and English-language work; its contribution-dimension weights may not generalize to other fields.The authors say broader adoption could require tuning assumptions such as valuing problem contributions more highly.Source: Limitations · p. 9
  • Fixed pools of 20 papers for disguises and disputed pairs and up to 400 for generated ideas limit evaluation of novelty-check retrieval.The pools are fixed lists rather than open search.Source: Limitations · p. 9
  • The study leaves in-the-loop effects largely to future work because evaluation is computationally expensive and ground truth for replacement ideas may require extensive annotation.A small pilot reports rejection and cost dynamics, but does not resolve the broader evaluation challenge.Source: Limitations · p. 9

Ethical Considerations

This section addresses how novelty modules should align with human understandings of novelty and how automated judges require careful use. It also documents the audited systems and the procedures, criteria, and annotator agreement involved in constructing and labeling the study’s proposals and prior-art cases.

Ethical Considerations

  • The authors say novelty modules should better align with human understandings of novelty and caution that ClaimNov, like any automated judge, should be used carefully.They release no direct paper text or ICLR review text, follow arXiv and OpenReview terms, and report collecting no survey responses or personal information; their annotation uses labels on published papers and generated ideas and is not human-subjects research.Source: Ethical Considerations · p. 9

A Pipeline Survey

  • The audit includes systems only when code is available, the proposal-level judgment is separable, the reference pool can be controlled or retrieval sandboxed, and the judge can be replaced with one whose training cutoff predates T.The survey covers novelty judgments embedded in AI-scientist pipelines and standalone judges, and examines what each pipeline passes downstream after its check.Source: A Pipeline Survey · p. 12, Table 4 · p. 13, Table 5 · p. 14

B Adapter Details

  • For the adapter comparison, checks receive the same proposal-pool pair; search is sandboxed by cosine-similarity ranking, and final judges use Sonnet 4.6 at temperature 0.The OpenNovelty-style baseline uses BM25 retrieval to avoid sharing a retriever with CLAIMNOV.Source: B Adapter Details · p. 15, Table 10 · p. 15

C CLAIMNOV Prompts

  • The appendix provides the exact CLAIMNOV prompts.The supplied passage identifies the prompts but does not include their content.Source: C CLAIMNOV Prompts · p. 15

Claim Extraction (system prompt)

  • Claim extraction conservatively identifies atomic contribution claims, usually 1–4, and classifies them as methodology, result, resource, or problem.The prompt distinguishes proposed techniques from findings, released resources, and newly introduced tasks or settings.Source: Claim Extraction (system prompt) · pp. 15–16

Support Counting (Stage B)

  • Support counting selects every paragraph with direct evidence for a claim, without a cap, and orders the selected paragraphs by relevance.Source: Support Counting (Stage B) · p. 17

Graded Entailment Judge

  • The entailment judge rates how far a prior claim covers a target on a 1–5 scale, distinguishing unrelated claims from full coverage.For numeric claims, anticipation requires comparable conditions; a different or larger number alone does not entail a finding in another setting or population.Source: Graded Entailment Judge · p. 17

Verdict (profile judge)

  • The final profile judge asks whether prior work already does the idea’s core problem and approach, treating an anticipated core contribution as non-novel but not an anticipated peripheral claim.Prior work that the idea builds on, or superficial topic overlap, does not by itself make the idea non-novel.Source: Verdict (profile judge) · p. 17

Rung 1: voice swap

  • The first two disguise prompts retain the original contribution while changing proposal framing, first by voice swap and then by deep paraphrase.The voice-swap prompt keeps technical content close to the paper; the deep-paraphrase prompt preserves technical claims while avoiding phrase overlap beyond four words, with technical proper nouns and dataset names allowed to remain.Source: Rung 1: voice swap · p. 18, Rung 2: deep paraphrase · p. 18

Rung 3: reframe

  • The reframe prompt changes motivation, order, method naming, examples, and title while retaining the same substantive technical claims, method, mechanism, and expected findings.It asks for a proposal in future tense of about 200 words and allows approximate numbers.Source: Rung 3: reframe · p. 18

Rung 4: adversarial restructuring

  • The most adversarial disguise restructures the proposal to appear unrelated while preserving the original problem, core mechanism, and expected outcome.It changes the starting point and emphasis, renames or omits methods, generalizes numbers, and permits added framing but not substantive changes to the contribution.Source: Rung 4: adversarial restructuring · p. 18

Verifier (rungs 3 and 4)

  • For rungs 3 and 4, a verifier judges whether the paper and proposal describe the same core contribution, defined by the same problem and mechanism.It returns yes or no followed by one sentence.Source: Verifier (rungs 3 and 4) · p. 18

E Later-Realized Labeling Prompts

F Label Validation

  • Annotators found 59/100 disputes genuine and 67/100 had correctly identified prior art; 40 pairs met both criteria, with 36 usable pools.For the first two annotators, raw agreement and Cohen’s κ were 81/100 and 0.62 for genuineness, 73/100 and 0.47 for identifying prior art, and 52/100 and 0.40 for dispute type; dispute-type labels were dropped for low κ. A third annotator adjudicated 63 disagreements, and the label-clean subset comprises the 40 pairs with a genuine dispute and correct prior art.Source: F Label Validation · p. 19, Table 11 · p. 19

Claim-dispute annotation (annotators 1 and 2)

Later-realized audit

  • In the later-realized audit, annotators compare each idea with 20 pre-cutoff papers and do not count post-cutoff papers as prior art.They judge whether a paper already contains the idea’s main contribution, and, if not, whether it establishes a single contribution; they also rate confidence.Source: Later-realized audit · pp. 19–20

F.1 Audit of generated ideas

A hand audit assessed generated ideas against a pool of 20 abstracts, confirming some later-realized ideas as novel and some sorter-matched ideas as already done. On the audited already-done group, CLAIMNOV rejected fewer ideas than the no-profile judge.

F.1 Audit of generated ideas

  • CLAIMNOV rejected 1 of 9 sorter-matched ideas judged already done, while the no-profile judge rejected 7 and also rejected 3 of 6 judged novel.The audit covered 60 ideas; annotators confirmed 28 of 45 later-realized ideas as novel relative to the pool, a stratum-weighted 60% of all 400.Source: §F.1 · p. 20
  • The hand audit compared each idea against 20 abstracts, a strenuous judgment because experts may disagree about whether a paper does substantially the same thing.The audited set included 45 later-realized ideas and 15 sorter-matched ideas.Source: §F.1 · p. 20
  • Table 13 reports each check’s rejections for later-realized ideas judged novel and for ideas judged already done.Source: Table 13 · p. 20, §F.1 · p. 20

G Attribution by Dimension and Relation Type

The attribution analysis breaks down claim-level results by the disputed claim’s dimension and by the relation type in the review. It also distinguishes whether a claim was extracted in a dimension.

G Attribution by Dimension and Relation Type

H CLAIMNOV Ablations

The ablations remove one CLAIMNOV stage at a time, while additional analyses examine disguise difficulty and the uncertainty and paired-test evidence for the main comparisons.

H CLAIMNOV Ablations

I Disguises Caught by Rung

  • Disguise catch does not fluctuate significantly across rungs for checks other than ARC, whose lexical design is expected to behave differently.Table 16 covers 100 disguises per rung across all 400 disguises.Source: I Disguises Caught by Rung · p. 20, Table 16 · p. 21

J Intervals and Paired Tests

K Robustness

Robustness checks test alternative labels and judges, larger pools, and simpler baselines; the appendix also examines natural already-done ideas, a pilot loop, case studies, and component validations.

K Robustness

  • CLAIMNOV’s false-alarm rate stays between 1.3% and 1.5% under three alternative labeling paradigms, and its profile still cuts false alarms and source-out rejections with two non-Anthropic judges.Catch drops with the weaker judges; tested baselines are not competitive with CLAIMNOV’s J.Source: K Robustness · p. 22, Table 23 · p. 22, Table 24 · p. 22
  • With disguise pools enlarged from 20 to about 200 papers, catch falls on the hardest rung while source-out rejections stay near zero.Smaller checks include a no-pool probe in which Sonnet 4.6 named B first for at most 1 of 160 claim-disputed proposals.Source: K Robustness · p. 22, Table 22 · p. 22, Table 26 · p. 23

L Natural Already-Done Ideas

  • A whole-idea sorter matched 837 of 3,000 generated ideas to papers that already did the idea; claim-level labeling overruled 101, and 407 of the remaining 736 had the shipped profile.The set is noisier than disguises because the sorter read the same abstracts as the no-profile ablation.Source: L Natural Already-Done Ideas · p. 23, Table 27 · p. 23

M In-the-Loop Pilot

  • The in-the-loop pilot used two anchors, pools of 178–185 papers, 20 ideas per arm, and up to three regenerations, but its review-score proxy yielded too few seeds for arm comparisons.The authors leave in-the-loop evaluation to future work.Source: M In-the-Loop Pilot · p. 23, Table 28 · p. 23

N Case Study

  • The case study traces a reviewer-disputed ICLR 2022 Spotlight paper and a rung-4 disguise with its source paper in the pool.Table 29 contrasts a claim-level dispute that CLAIMNOV passes with a rung-4 disguise it rejects.Source: N Case Study · p. 23, Table 29 · p. 24

O.1 Claim extraction

  • Claim extraction uses abstracts because F1 falls from 0.577 to 0.490 when extracting from introductions instead.The shipped extractor is gpt-4o-mini; gpt-5-mini performs equivalently on extraction but poorly as the entailment judge.Source: §O.1 · p. 23, Table 30 · p. 25

O.2 Same-dimension retrieval

  • Same-dimension retrieval is retained because only 3 of 159 disputes are not in the same dimension; the selected depth retrieves 832 of 837 sorter-matched papers.The depth rule is k = max(10, 0.2 nd), capped at 40.Source: §O.2 · pp. 23, 25, Table 31 · p. 25, Table 32 · p. 25

O.3 Graded entailment judge

The graded entailment analysis finds that scores can order claims but do not provide a reliable pass/fail threshold. The verdict stage therefore evaluates novelty over the full profile, and the paper discloses how AI agents assisted its work.

O.3 Graded entailment judge

  • The graded score orders reviewer-praised claims above sibling claims 59% of the time, but the authors find no good pass/fail novelty threshold from the score.Every judge tier agrees with the human majority more than annotators agree with one another; embedding-only similarity is a strong baseline for separating novel from established claims.Source: §O.3 · p. 25, Table 35 · p. 25, Table 37 · p. 26

O.4 Verdict

  • The adopted verdict stage evaluates novelty once over the whole claim profile rather than using a per-candidate question.Calibration tested knockout entailment levels, relation labels, and per-candidate questions.Source: §O.4 · p. 26, Table 38 · p. 26

P AI Disclosure Statement

  • LLM agents assisted with experiment and analysis code, paper structure, tables, appendix text, and release preparation; the authors devised the scientific contributions.Source: P AI Disclosure Statement · p. 26

Related papers

Loading 2610.36511v1…