PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond fabrication: Misattributed numbers in papers written by AI research agents

Mosab Alfaqeeh, Omar Hourani, Muath Hourani, Hashem Alfaqeeh

arXiv:2610.38691v1cs.AIcs.CLcs.DL

TL;DR

AI-agent papers report numerical results that readers may struggle to trace back to code, logs, and intermediate files. This paper analyzes benchmark integrity failures and introduces a deterministic provenance checker, finding that unsupported values are usually misattributed rather than absent. The checker certifies claims with high precision, but claims it cannot verify often remain traceable.

  • Problem

    Readers of agent-written research may not have access to the code, logs, or intermediate files needed to trace reported numbers, while integrity verdicts do not explain the failure type.

    Source: §1 · pp. 2–3

  • Method

    The authors analyze benchmark judgments and human-labeled claims, then compare paper numbers with agent result files using a deterministic checker that scores contextual source identification and decoys.

    Source: §4.2 · p. 17

  • Results

    Unsupported numbers were mostly misattributed; the checker judged 78 of 85 certified claims traceable, while many uncertified claims were also traceable.

    Source: §1 · p. 4

  • Takeaways & Limitations

    A positive certification supports provenance, while non-verification should route claims to human review rather than be treated as evidence of falsehood.

    Source: §6.2 · p. 33

    Findings come from one benchmark, nine agents, and fifteen tasks, and the checker relies on LaTeX papers and machine-readable result files.

    Source: §6.3 · p. 33

Abstract

from arXiv · show

Verifying numerical claims in research written by AI agents is an important but challenging problem. Recent benchmarks show that many agent-written papers fail integrity checks, but such verdicts, often produced by a single LLM judge, identify failure without explaining its nature, and existing checks either test a paper for internal consistency or test whether a reported value appears somewhere in an agent's output. We re-analyze all 540 released judgments from a recent public benchmark of AI research agents (nine agents, fifteen open research problems) and manually label 256 numerical claims from 18 submissions. Failures concentrate on alignment between the paper and the underlying work rather than overt cheating, and about four in five unsupported claims correspond to values in the agent's own files that come from the wrong experiment, metric, or run. We introduce a deterministic claim-to-evidence checker that scores whether the surrounding context identifies the claimed result and calibrates each claim against numerical decoys. With its rules fixed before labeling, it certifies claims with over 90% precision against blinded annotators (78 of 85; Cohen's kappa = 0.89), while verifying only a subset of claims; most claims it cannot verify are nevertheless traceable, so failure to verify is not evidence of falsehood. An LLM verifier given the same evidence recovers more traceable claims but accepts about three times as many misattributed ones, so the two are best used together. Applied to 99 submissions, the checker reveals large differences in traceability across agents, and a second judge from a different vendor preserves the official agent ranking while passing twice as many papers. These findings motivate provenance-first auditing of AI-produced research.

1. Introduction

Integrity verdicts show that AI-agent papers can fail without revealing whether results were fabricated or merely misreported. This paper investigates the distinction and develops a provenance-focused way to check numerical claims.

1. Introduction

  • The study re-analyzes 540 released judgments, labels 256 numerical claims from 18 submissions, and tests a deterministic claim-to-evidence checker against execution artifacts.A second-vendor judge re-graded a stratified sample of 30 submissions.Source: §1 · p. 3
  • Roughly four in five unsupported numbers in the benchmark appeared in agent files but came from a different experiment, metric, or run.Misattribution, rather than absence of a source, was the major failure mode.Source: §1 · p. 4
  • The checker certified 78 of 85 claims as traceable (92%), including 27 of 29 held-out claims, but verified only a subset.Most uncertified claims were nevertheless judged traceable, so non-verification is not evidence of falsehood.Source: §1 · p. 4
  • A second judge from another vendor preserved the agent ranking while passing more papers, indicating that absolute pass rates depend on the judge.The authors found no evidence that the official judge favored agents from its own vendor.Source: §1 · p. 5
  • For readers of AI-produced research, the relevant question is which run produced a reported number, not merely whether that value appears in the outputs.The paper argues that this provenance question can be answered automatically for a useful fraction of claims.Source: §1 · p. 5

2. Related work

The paper positions numerical provenance as a gap between existing checks of internal consistency, text attribution, and whole-submission integrity judgments. It connects that gap to data quality and trust in automation.

2. Related work

  • The paper connects research-agent evaluation, agent misrepresentation, generated-text attribution, numerical checking, LLM judging, and data provenance.The data-quality perspective treats reports as products whose fitness for use should be established before decisions rely on them.Source: §2 · pp. 5–6

2.1. From coding assistants to research agents

  • Research-agent benchmarks progressed from coding and machine-learning tasks to producing new scientific results, with Into the Unknown releasing full run files and judge verdicts.That release makes direct examination of reported-number provenance possible.Source: §2.1 · p. 6

2.2. How research agents misrepresent their work

  • Prior work documents fabricated results, selective reporting, and integrity exploits, but has not measured claim-level fabrication versus misattribution against blinded labels.The paper identifies both claim-level validation and separation of these failure modes as gaps.Source: §2.2 · pp. 6–7

2.3. Attribution and provenance of generated content

  • Text attribution asks whether a statement is supported by an identified source, while numerical provenance must distinguish among many machine-written entries containing the same value.A prior artifact-centered checker proposal was not implemented or evaluated; this work implements and validates a checker without requiring special agent traces.Source: §2.3 · pp. 7–8

2.4. Checking reported numbers and claims

  • Internal-consistency tests and external-reference verification do not establish whether a reported number came from the run it describes.Because misattributed values can appear in outputs, the paper evaluates provenance rather than presence alone.Source: §2.4 · pp. 8–9

2.5. LLMs as judges of integrity

  • LLM judges can be sensitive to presentation and self-preference, while stability and vendor dependence in long-form integrity audits remained untested.The paper evaluates integrity verdicts using a judge from a different vendor.Source: §2.5 · p. 9

2.6. Data quality, provenance, and trust in AI-produced analyses

  • Data quality includes traceability and interpretability, and readers of agent reports risk both trusting misattributed numbers and rejecting sound work they cannot verify.The paper frames both risks as trust-calibration problems.Source: §2.6 · p. 10

2.7. Positioning

  • The paper’s stated contributions are claim-level measurement, a held-out human-validated provenance checker, analysis of non-verification, and testing judge-vendor robustness.Table 1 compares approaches by evidence source, separation of misattribution from fabrication, human validation, and judge robustness.Source: §2.7 · p. 10, Table 1 · p. 11

3. Benchmark and data

The study uses Into the Unknown, a benchmark in which agents produce follow-up research papers and release their working directories. Those artifacts support claim-level provenance analysis without rerunning agents.

3.1. The Into the Unknown benchmark

  • Nine agents from three vendor groups each attempted fifteen research tasks, yielding 135 submissions to Into the Unknown.Each task supplies a recent paper, a specification, and a computing environment for agents to conduct experiments and submit a paper with its working directory.Source: §3.1 · p. 11
  • The LLM judge assessed each submission against sequential integrity, completeness, and scientific-quality gates; 21 submissions passed the integrity gate, and none passed all three.Each submission was evaluated four times, with repeats aggregated into a consensus verdict.Source: §3.1 · pp. 11–12

3.2. Released artifacts

  • The fixed release includes judge outputs, consensus verdicts, agent working directories, and some interaction trajectories; the analyses use public artifacts without rerunning agents.The snapshot is repository commit 612cdca.Source: §3.2 · p. 12
  • Complete working directories, reliably extractable LaTeX papers, and released node-level justifications enable claim-level provenance analysis.The artifacts include source files, code, configurations, logs, and result files.Source: §3.2 · p. 12

4. Methods

The methods combine analysis of official integrity judgments, a deterministic claim-to-evidence checker, human labeling, baseline comparisons, and re-grading by a second judge. The checker evaluates whether a number’s context identifies the result it claims to represent, rather than merely whether the value appears in an agent’s files.

4.1. Re-analysis of the official judgments

  • 2,376 consensus G1 node verdicts are analyzed by rubric family, repeat agreement, and a model of run duration with agent and task fixed effects.The family analysis separates checks of code–paper alignment and number traceability from checks for overt misconduct.Source: §4.1 · pp. 12–14

4.2. Claim-to-evidence checker

  • The deterministic checker extracts numerical claims from papers, matches them to machine-written result files, and labels them certified, unattributed, or untraced.Claims include numbers with decimal points, percent signs, or scientific exponents; evidence excludes the paper, code, and human-written notes.Source: §4.2 · pp. 14–15
  • Unlike value matching alone, the checker asks whether the evidence context identifies the claimed quantity; a value with 24 matches can remain unattributed when its best match is a configuration setting.In the worked example, human annotators also labeled that claim wrong-source.Source: §4.2 · pp. 14, 17
  • It scores non-bulk matches by claim–evidence context overlap and calibrates each claim against numerical decoys; strict thresholds were fixed before human labeling.The strict operating point uses τ = 0.12 and ρ = 0.1.Source: §4.2 · pp. 15–16

4.3. Human labeling

  • Annotators distinguish traceable, wrong-source, not-found, and not-a-claim labels; checker evaluation uses weighted precision and recall and Cohen’s κ for annotator agreement.“Not found” means absent from the available artifacts, not necessarily fabricated.Source: §4.3 · pp. 17, 19
  • Human labeling uses development and confirmation samples, with confirmation drawn after operating points were frozen and labeled against written rules by one or two annotators.The development sample retains 184 claims; the confirmation sample contains 72 claims from eight submissions.Source: §4.3 · p. 18

4.4. Baselines

  • The baselines include an embedding scorer and an LLM verifier given the paper context and the three best-matching evidence entries; methods are compared on 129 untuned real claims.The embedding baseline replaces token overlap with sentence-embedding similarity.Source: §4.4 · pp. 19–20

4.5. Second judge

  • A different-vendor judge re-grades a stratified sample of 30 submissions, with node-level and gate-level agreement and vendor effects assessed across submissions.The analysis uses signed-rank, Kruskal–Wallis, permutation, and generalized estimating equation approaches.Source: §4.5 · pp. 20–21

5. Results

The results locate most integrity failures in mismatches between papers and the work, with unsupported numbers usually traceable to the wrong source. A deterministic provenance checker certifies claims precisely but covers only a subset, while benchmark rankings persist across judges despite differences in leniency.

5.1. What the official judge fails (RQ1)

  • 52% of code–paper alignment nodes and 38% of number-traceability nodes failed, while hard-coded values and external-code criteria each failed 1%.Failures therefore concentrated in write-up/work alignment rather than criteria targeting overt misconduct.Source: §5.1 · p. 21, Figure 2 · p. 22

5.2. Misattribution versus absence in human labels (RQ2)

  • 47 of 53 unsupported claims came from the wrong source, and population reweighting puts the wrong-source share at 81%; only 6 values were not found.Wrong sources included other experiments, metrics, runs, or statistics.Source: §5.2 · pp. 22–23

5.3. Can provenance be checked automatically? (RQ3)

  • Value matching accepts 87–95% of wrong-source and not-found claims, whereas context scoring raises precision from 0.77 to 0.92 while retaining 60% of traceable claims.The decoy test separately reduces false certification from 18.9% to 1.3%.Source: §5.3 · pp. 24–26
  • 78 of 85 certified claims were traceable (0.92 precision; exact 95% CI 0.84–0.97) across 211 labeled claims.The held-out sample also had 27 of 29 certified claims labeled traceable.Source: §5.3 · pp. 24–25
  • The strict checker accepts 7 of 53 non-traceable claims, compared with 19 for the LLM verifier; when both accept, 51 of 52 held-out claims are traceable.The LLM recovers more traceable claims, but the methods are complementary rather than interchangeable.Source: §5.3 · p. 27
  • The strict checker certifies about a third of traceable claims, and 57% of uncertified held-out claims were nevertheless labeled traceable.Non-certification therefore cannot be treated as evidence that a claim is unsupported.Source: §5.3 · p. 28

5.4. Traceability across the benchmark

  • The checker certifies 76% of astra-6’s claims and 63% of sol-5.6’s, versus 8–17% for each of the other seven agents.For opus-5.5, the judge’s mean integrity score was 93 while checker certification was 14%; confirming the proposed explanation requires an agent-specific labeled sample.Source: §5.4 · pp. 28–30

5.5. Robustness to the judge (RQ4)

  • Both judges rank vendor groups identically, but the second judge fails 5.4 fewer G1 nodes per submission on average and passes 12 submissions versus the official judge’s 6.The absolute pass rate varies by judge even though the relative ranking is stable.Source: §5.5 · pp. 30–31, Figure 5 · p. 31

6. Discussion

In this benchmark, unsupported numbers usually reflect misattribution rather than invention, calling for provenance-focused reporting and verification. The discussion also sets practical limits on what the checker and benchmark results establish.

6.1. What the findings mean

  • Unsupported numbers usually exist in agents’ files but belong to another experiment, metric, or run, making misattribution the typical integrity failure.The authors distinguish this reporting problem from fabrication, which calls for detection and sanction.Source: §6.1 · p. 32
  • Misattribution calls for structured, explicitly keyed result files and reports generated from recorded values, rather than retyped numbers.The authors also call for checks linking each reported number to the run that produced it.Source: §6.1 · p. 32, §6.2 · p. 33

6.2. Implications for practice

  • 23% of held-out claims were certified at over 90% precision; uncertified claims should go to human review, not be treated as failures.The authors caution that certified claims are reliable only to the extent that the underlying experiment is sound.Source: §6.2 · p. 33
  • A second judge passed twice as many submissions while preserving the official judge’s agent ranking, so benchmarks should report which judge produced pass rates.Claim-level checking complements, rather than replaces, node-level judging.Source: §6.2 · p. 33

6.3. Limitations

  • Findings come from one benchmark of nine agents and 15 tasks; the agents are recent frontier systems expected to be superseded.The authors do not claim that agents generally misreport more often than they fabricate.Source: §6.3 · pp. 33–34
  • The 256-claim labels came from two research-team annotators, not an external panel, and several confidence intervals are wide.The held-out set contained 60 real claims.Source: §6.3 · p. 34
  • The checker is tailored to papers in LATEX with machine-readable result files, certifies provenance rather than correctness, and has about one-third recall.It would need adaptation for other paper formats or results stored in binary files or databases.Source: §6.3 · p. 34
  • The second-judge sample comprised 30 submissions, mostly graded once, and the 438 nodes where judges disagreed were not adjudicated.Different agent harnesses may also partly explain differences between agents.Source: §6.3 · p. 34

7. Conclusions

In this benchmark, unsupported numbers were mostly real values reported against the wrong experiment, metric, or run. The checker verifies provenance with high precision on held-out data, but only for a subset of claims.

7. Conclusions

  • Unsupported numbers were mostly real values reported against the wrong experiment, metric, or run, rather than values with no source.The checker’s ranking of agents agreed with the official judge, and a second judge preserved that ranking while showing that pass rates depend on judge choice.Source: §7 · p. 35
  • The checker establishes whether a reported number is tied to the result it claims to describe, rather than merely checking whether the value appears in agent files.Most claims it cannot certify are nevertheless traceable, so non-verification is not evidence of falsehood.Source: §7 · p. 35
  • The checker’s limited recall confines automation to a subset of claims, while still allowing high-precision verification of that subset.The authors conclude that AI-produced research should be audited by tracing each number to its claimed result.Source: §7 · p. 35

Related papers

Loading 2610.38691v1…