PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims

Delip Rao, Ahmed Muharram, Chris Callison-Burch

arXiv:2610.20457v1cs.CL

TL;DR

Scientific claims can admit several defensible interpretations, yet benchmarks compare verdicts with expert labels without recording those interpretations in a comparable form. The paper introduces Claim Cards to record interpretation fields before evidence gathering and tests them with an LLM agent. Deliberately changing a card’s interpretation changed verdicts on about a third of claims, while ordinary run-to-run interpretation variation did not measurably track verdicts.

  • Problem

    Benchmarks score verdicts against one gold label without recording expert and system interpretations in comparable form, leaving interpretation-related disagreements difficult to distinguish from evidence-related ones.

    Source: §1 · p. 2

  • Method

    Claim Cards record an agent’s interpretation in nine fields before evidence gathering, marking values as stated, inferred, or default.

    Source: §1 · p. 2

  • Results

    Universal-quantifier and stricter-threshold edits changed verdicts on 19 of 56 and 19 of 58 claims, respectively, while run-to-run interpretation variation did not measurably track verdicts.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors argue that each verdict should be reported with its interpretation, since both verdicts and calculations can depend on that interpretation.

    Source: §6 · p. 8

    The parser builds cards from the agent’s first reasoning step without seeing the claim, mixing agent decisions with parser-filled values and making their origins indistinguishable.

    Source: §7 · p. 9

Abstract

from arXiv · show

Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-language scientific claims, written for human experts, are often underspecified and so admit several defensible interpretations. Such verifiers usually pick one, often tacitly, and benchmarks score their verdicts against expert labels without recording the expert's interpretation or the verifier's in a comparable form. This leads to disagreements with expert labels that cannot be traced to the evidence or the interpretation, and to compute spent on unchecked interpretations. We introduce Claim Cards to make such interpretations explicit in the reasoning itself. A Claim Card records a verifier's interpretation as nine fields before evidence gathering, each marked stated, inferred, or default. To test whether an agent's interpretation can change its verdict, we ran a GPT-5.4 agent five times on each of 60 expert-labeled materials-science and biomedical claims with a card, without one, and with one card frozen across runs (900 runs). Over 40% of card fields were not marked stated, and a card added about a tenth to a run's GPT-5.4 tokens and time without measurably changing agreement with expert labels. The agent's own interpretations varied between runs, although within a claim this variation did not measurably track the verdict. A deliberate change to one field of the frozen card, however, making the quantifier universal or the threshold for support stricter, changed the verdict on about a third of the claims, against about one in seven expected by chance (p < 0.001 for each edit). On 24 materials-science claims run with computational tools, the stricter threshold also changed what the agent asked those tools to compute. We therefore argue that verdicts should be reported with their interpretations and that benchmarks could publish the interpretation behind each label.

1 Introduction

Scientific claims can leave key assessment choices open, while benchmarks record gold labels and system verdicts without comparable records of the interpretations behind them. The paper introduces Claim Cards to record those interpretations and tests whether they affect verdicts and tool use.

1 Introduction

  • Benchmarks compare verdicts with one gold label but do not record expert and system interpretations in comparable fields, obscuring whether disagreements reflect evidence or interpretation.The opening alloy claim illustrates this: the agent’s calculations agreed with the expert’s diagram, but its verdict differed because the expert relied on an unstated temperature range.Source: §1 · p. 2
  • A Claim Card records nine interpretation fields before evidence gathering, marking each value as stated, inferred, or default, and users can edit a disputed field before rerunning.The card is extracted from the agent’s first reasoning step and printed before its first tool call.Source: §1 · p. 2
  • The study tested ClaimSpy on 60 expert-labeled claims across materials science and biomedicine, using card, no-card, and frozen-card conditions, with five runs per claim and condition.It also tested edited cards and computational-tool use on 24 materials-science claims.Source: §1 · p. 3
  • Edits making the quantifier universal or the support threshold stricter changed verdicts on 19 of 56 and 19 of 58 claims, respectively, while run-to-run interpretation variation did not measurably track verdicts.Both edit counts exceeded twice the number expected if edits had no effect (p < 0.001 for each).Source: §1 · p. 3
  • More than 40% of card fields were not marked as stated, and adding a card did not measurably change accuracy, stability, or calculation frequency.A card added about a tenth to a run’s tokens and time; 50 of 60 audited cards had no field flagged as unsupported or contradictory.Source: §1 · p. 3

2 Interpretations and Claim Cards

The paper represents a claim interpretation through shared fields so that ambiguity can be counted, compared, and deliberately edited. Claim Cards make those values and their provenance visible before evidence gathering and can be frozen for a rerun.

2.1 The Fields of an Interpretation

  • An interpretation consists of values for nine fields describing the system, intervention, and evaluation criteria, with each field capturing a dimension on which defensible interpretations can differ.The quantifier sets the stopping rule: one threshold-clearing instance suffices for an existential claim, whereas a counterexample is decisive negative evidence against a typical or universal claim.Source: §2.1 · p. 3
  • A Claim Card records a value for each interpretation field and a provenance mark—stated, inferred, or default—for each value.These common fields let the authors count omissions, compare runs field by field, and change one field while holding others constant.Source: §2.1 · pp. 3–4

2.2 How a Card Is Produced and Used

  • The agent writes its interpretation in its first reasoning step; a second model, seeing only that step, converts it to a card and fills implicit fields and provenance marks.The parser does not see the claim, so its marks reflect its judgment rather than a direct comparison with the claim.Source: §2.2 · p. 4
  • In card mode the parsed card is printed before the selected tool call, but the agent is not shown it; in frozen mode, the agent and scorer receive the same stored card.A user can edit a field and rerun the agent with the edited card frozen.Source: §2.2 · p. 4

3 Experimental Setup

The experiments compare ClaimSpy’s card, no-card, and frozen-card conditions across materials-science and biomedical claims, with a separate computational-tool setting for some materials-science claims. The setup also defines the measures used to compare interpretations, verdicts, and tool use.

3 Experimental Setup

  • ClaimSpy uses GPT-5.4 as its backbone, first tries to falsify a claim, then seeks a condition-matched measurement, with quantifier-based stopping rules.Its tools include a search agent and a coding agent.Source: §3 · p. 4
  • The 60-claim set comprised 30 materials-science claims from SFBench and 30 biomedical claims from SciFact; a separate set of 24 SFBench claims was run with computational tools.The 60-claim runs lacked computational libraries, so the agent relied mainly on search; the two sets differ in tool setting, frozen-card source, and run timing.Source: §3 · p. 4
  • Each claim was run five times in card, no-card, and frozen-card conditions; one-field edits made frozen cards harder to satisfy, and each edited card was run once.Across the 60 and 24 claims, the study reports 1,582 GPT-5.4 runs, excluding a 20-claim pilot.Source: §3 · p. 5
  • Search outages affected 17 frozen and 10 edited-card runs without computational libraries, and frozen runs occurred at a different time from card and no-card runs.The authors therefore report outage-affected results both with and without those runs and treat frozen-condition comparisons as descriptive.Source: §3 · p. 5
  • A field audit classified card values against claims, while consistency measured agreement across run pairs and sign accuracy compared verdict signs with gold-label signs.Run-to-run variance was the standard deviation of each claim’s five scores; intervals were claim-paired 95% bootstrap intervals, and intervals including zero were treated as no measurable difference.Source: §3 · p. 5

4 Results

Across six results subsections, the study finds that scientific claims leave important interpretation fields unstated and that agent interpretations vary between runs. Deliberately editing an interpretation can change verdicts and tool questions, while cards did not measurably improve agreement with expert labels.

4.1 Claims Leave Much of Their Interpretation Unstated

  • 42% of materials-science and 44% of biomedical card fields were not marked stated; units, comparators, and thresholds were often left open.System and observable fields were usually marked stated in materials-science cards, unlike units, comparator, and threshold fields.Source: §4.1 · p. 6

4.2 Interpretations Vary Between Runs, Most in the Threshold, Comparator and Conditions

  • All nine fields expressed the same interpretation in only 2 of 51 claims, and interpretation consistency did not measurably differ between card and no-card runs.The five explicit cards agreed least on threshold (60%), comparator (62%), and conditions (69%).Source: §4.2 · p. 6

4.3 Changing the Interpretation Can Change the Verdict

  • 19 claims under each edit had a sign different from the most common unedited sign, versus 7.5 and 8.3 expected; both edits lowered scores on average.The quantifier edit shifted scores by −0.62 over 56 claims, and the threshold edit by −0.54 over 58.Source: §4.3 · p. 6, Table 2 · p. 6
  • The edits reached the agent’s own answers, but cannot establish which interpretation is right or separate the edited value from general caution under added constraints.In 22 of 24 edited tool-enabled runs, the edited threshold appeared in questions sent to tools or search, versus none of 120 unedited runs.Source: §4.3 · p. 7

4.4 How Well a Card Matches Its Claim, and What It Costs

  • Audit judges flagged 3% of materials-science and 4% of biomedical fields, while cards added 13% input tokens and 8% wall-clock time.Most biomedical disagreements involved fields marked stated that the judge considered reasonable defaults.Source: §4.4 · p. 7, §4.6 · p. 7

4.5 Observations on Accuracy and Stability

  • Card versus no-card sign accuracy was 0.68 versus 0.71 on 60 claims and 0.67 versus 0.64 on 24 tool-enabled claims, with no measurable accuracy or stability difference.The scorer repeated its own stated score in 472 of 478 runs; the authors think most variation arose from evidence found and how it was weighed.Source: §4.5 · p. 7

4.6 Effect of Claim Cards on Tool Use

  • Card wording changed tool questions, although tool-call rates did not measurably differ; frozen runs were descriptively less often correct, mostly because of extra materials-science abstentions.Cards appeared in 64 of 120 tool-enabled runs that called CALPHAD or CHGNet, versus 72 no-card runs; edited thresholds changed computation questions on three claims with composition ranges.Source: §4.6 · p. 7

5 Related Work

Prior work addresses claim verification, ambiguity, and agent assumptions through related but distinct approaches. The paper distinguishes Claim Cards by recording adopted values and their sources before evidence gathering.

Claim verification and feasibility

  • Binary feasibility labels in Matter-of-Fact cannot record that a claim is too underspecified to judge, while Claim Cards record open details and their sources before checking.The authors use SFBench, which includes expert rationales, and do not compare with Matter-of-Fact’s binary baselines.Source: §5 · p. 8

Ambiguity and underspecification

  • AmbiFC models veracity-label disagreement without comparable interpretations, while Dis2Dis rewrites claims after evidence gathering rather than recording an interpretation beforehand.The authors report that no dataset they know of lists interpretations of scientific claims.Source: §5 · p. 8

Agents and visible interpretations

  • Clarification methods need an answer during the run, whereas Claim Cards make an interpretation visible for batch assessment without that interaction.Prior interfaces let users edit interpretations, while the paper tests edits as an intervention.Source: §5 · p. 8

6 Conclusion

The conclusion argues that feasibility verdicts should be reported with the interpretation behind them, because different interpretations can change both verdicts and computations. Its experiments show that deliberate edits can change verdicts, while ordinary run-to-run interpretation variation did not measurably track verdicts.

6 Conclusion

  • A feasibility verdict concerns one interpretation of a claim, so systems should report verdicts with that interpretation and benchmarks could publish the interpretation behind each expert label.The authors’ Al–Zn example shows that calculations can agree with an expert’s diagram while the verdict differs because the claim leaves its temperature range unstated.Source: §6 · p. 8
  • Deliberate interpretation edits changed verdicts, but ordinary variation between frozen-card interpretations did not measurably track verdicts without computational libraries.The edits made claims harder to satisfy, so they cannot separate the edited value from general caution under added constraints.Source: §6 · p. 8
  • The authors identify testing expert usefulness of default-marked fields and whether expert corrections improve verdicts as an initial next step.They call for evaluation on more datasets and with more than one LLM backbone.Source: §6 · p. 8

7 Limitations

The study’s small, uneven claim sets and choices in the runs constrain how broadly its findings can be generalized. Several comparisons are exploratory or confounded by run timing, card construction, and tool settings.

7 Limitations

  • The claim sets are small, and the 24 tool-tagged claims have a positive-label imbalance that makes their accuracy descriptive.Fourteen of 24 labels are positive; accuracy should be compared with the 0.58 score from always predicting positive, and CALPHAD may share COST 507 data with the experts.Source: §7 · pp. 8–10
  • The study uses one backbone, GPT-5.4, and findings may differ with other agents, guidelines, or training-data exposure.Materials-science guidelines were refined on earlier SFBench claims; SciFact labels have been public since 2020, and the agent searched the open web while labels refer to cited abstracts.Source: §7 · p. 9
  • Card values and provenance marks mix agent decisions with parser-supplied fields, and parser variability may contribute to run-to-run differences.A judge flagged 3% of materials-science fields and 4% of biomedical fields as unsupported or contradictory, but could not identify which values came from the agent.Source: §7 · p. 9
  • Judge-model labels were not systematically checked by people, and several analyses rely on codes assigned by one coder.The consistency analysis covered 51 of 60 claims and 22 of 24 tool claims.Source: §7 · p. 9
  • Frozen-card comparisons span different run times and search outages, while the 60-claim edit shifts barely changed when outage-hit runs were excluded.The 24-claim threshold-edit comparisons were made a day apart; the edited-card effect is based on one run per setting and other contrasts are exploratory.Source: §7 · pp. 9–10
  • The card/no-card comparison tests the whole card condition, because no-card runs also omit guideline sections and withhold cards from scoring and reporting.No domain expert edited a card or reran the agent; whether card visibility helps users detect wrong defaults or increases over-trust remains untested.Source: §7 · p. 10

Ethics Statement

Claim Cards are intended to make automated verdicts easier to inspect and contest, but they do not ensure correctness. The appendix illustrates how claims leave experimental details open and explains why quantifier and provenance fields matter.

Ethics Statement

  • Cards may make verdicts easier to inspect, but a readable card can still rest on a wrong default and its provenance marks are a second model’s judgment.The paper also notes that explanations may increase reliance, while making answers cheaper to check may reduce over-reliance.Source: Ethics Statement · p. 10

A.1 Underspecification in a Single Claim

  • The GaN claim leaves at least thirteen details open, spanning geometry, bonding, strain state, measurement conditions, reproducibility, and material properties.These omissions can affect strain transfer, feasibility, measurement, and interpretation of the reported bandgap modulation.Source: §A.1 · p. 16
  • Other open GaN details include measurement modality, the meaning of the 200 meV range, spatial uniformity, residual stress, environmental conditions, and required strain magnitude.The claim also leaves process yield, cycle durability, doping, and surface passivation unspecified.Source: §A.1 · pp. 16–17
  • The Claim Card marks eight of nine GaN fields as stated, but a whole-field mark cannot reveal details left open within a stated value.The figure’s margin notes connect card fields to gaps in the example claim.Source: §A.1 · p. 17

A.2 The Quantifier and the Provenance Marks

  • Quantifier choices change the evidence rule: one matched instance can support an existential claim, whereas one counterexample can defeat a typical or universal claim.A few positive instances do not establish a typical or universal claim, and failing to find an existential instance is not negative evidence.Source: §A.2 · p. 17
  • Provenance marks distinguish values stated in the claim, inferred with domain knowledge, or assumed by default; non-stated fields estimate what the claim leaves open.Every schema field is required, so an open field receives an explicit default or is marked unspecified.Source: §A.2 · p. 17

A.3 How a Card Is Produced and Used

ClaimSpy makes a claim’s interpretation explicit before evidence gathering, then uses the card to guide scoring and reporting. Its schema structures the interpretation into required fields with provenance marks.

A.3 How a Card Is Produced and Used

  • In preflight analysis, ClaimSpy turns a claim into an explicit interpretation before evidence gathering, filling silent fields from domain knowledge and marking them inferred or default.The card records both what the claim says and what ClaimSpy decided it means.Source: §A.3 · p. 17
  • The agent first writes a free-text interpretation under a guideline that prefers the weakest faithful reading and avoids upgrading quantifiers or adding hidden constraints.It names the system and evidence sought; the median first-step paragraph is about 90 words.Source: §A.3 · pp. 17, 19
  • A second model converts that first reasoning step into the required card schema, supplying implicit values and assigning provenance marks without seeing the claim or guideline.The parser’s marks are its judgments of where values came from, not statements by the agent.Source: §A.3 · p. 19
  • In card mode, ClaimSpy prints and saves the parsed card before the selected tool call, then supplies it to the scorer and report step; frozen mode injects a stored card unchanged.The agent gathers evidence under its own written interpretation and is not shown the parsed card in card mode.Source: §A.3 · p. 19

A.4 Preflight Schema

  • The nine-field schema covers the system, state, conditions, manipulation, observable, units, quantifier, threshold, and comparator, with provenance recorded for each content field.Field descriptions vary by domain; the same schema is used across domains.Source: §A.4 · pp. 19–20
  • The quantifier specifies existential, typical, or universal scope, while threshold and comparator fields state support criteria and the baseline or control.Other fields encode conditions, units, and the intervention or measured quantity.Source: §A.4 · p. 20

B Experimental Setup in Detail

The setup compares ClaimSpy’s card, no-card, and frozen-card runs across selected materials-science and biomedical claims, with additional tool-setting and one-field-edit runs. It also specifies how interpretations, verdicts, overhead, and statistical contrasts are measured.

B.1 System and Tool-Use Policies

  • ClaimSpy decomposes claims, checks implied quantities against physical bounds, applies domain rubrics, and uses search plus policy-limited coding or computational tools.Materials-science runs could call CALPHAD and CHGNet; biomedical runs could call a general coding agent.Source: §B.1 · p. 20

B.4 Conditions and Edits

  • The three conditions were card, no-card, and frozen-card; each of 60 claims ran five times per condition, while edited cards changed one field and ran once.For 24 claims, frozen and edited cards also ran five times with computational tools and five times with search only; threshold edits were not checked against the prompt that requested them.Source: Table 3 · p. 21, §B.4 · p. 23

B.2 Computational Tools and Run Settings

  • Of 90 computational-tool calls in the 60 materials-science claims, only one returned a calculation; many card and no-card CALPHAD calls on Fe–Al claims used different databases from frozen runs.Of the 89 calls returning no calculation, 81 reported a missing library; 43 supplied literature, recalled, or hand-estimated values instead. On Fe–Al claims, 12 of 15 card and 7 of 14 no-card CALPHAD calls used another database, versus none of 15 frozen calls.Source: §B.2 · p. 22

B.4 Conditions and Edits

  • No-card runs retained plausibility checks and quantifier-based stopping rules but removed card-related guideline subsections; frozen runs received the same stored card across repeats.The 60-claim draws were stratified by gold label, and the computational-tool sample retained each claim’s original text and label.Source: §B.4 · pp. 22–23, C Selected Claims · pp. 24–25

B.5 Measures

  • Interpretation consistency groups equivalent field values across runs, while verdict variability uses sign disagreement; overhead compares tokens, time, steps, and searches.Edit outcomes include score shifts and whether edited scores fall outside unedited-run ranges; computational-tool use is measured by calls and returned calculations.Source: §B.5 · p. 23

B.6 Statistics

  • Uncertainty estimates use bootstrap intervals over claims, while exact sign-flip tests assess paired mean differences and a cluster bootstrap plus group sign-flip tests address shared chemical systems.The cluster analysis groups the 24 claims into 12 chemical-system or host-compound groups; its tests agreed with claim-level intervals on whether contrasts differed from zero.Source: §B.6 · pp. 23–24

C Selected Claims

  • The 60-claim sample contains 30 SFBench materials-science and 30 SciFact biomedical claims; 24 tool-run materials-science claims were randomly selected from 37 tagged claims and had more positive labels than the 13 excluded claims.The 24 included 14 positive labels, while the 13 excluded claims had eight −2 labels and only one positive label.Source: C Selected Claims · pp. 24–25

D Unstated Fields and Run-to-Run Consistency

  • In the 60-claim setting, 42% of materials-science and 44% of biomedical card-field values were not stated; in the 24 tool-run claims, 25% were inferred and 11% default.In the 60-claim cards, the state was unstated in 57% of materials-science and 85% of biomedical cards; units, comparator, and quantifier were often unstated in the tool-run cards.Source: D Unstated Fields and Run-to-Run… · p. 27

D.1 Consistency

Explicit cards and recovered no-card interpretations were similarly consistent across repeated runs. Within-claim interpretation differences generally did not track verdict differences, though tool-run quantifier comparisons showed mixed patterns.

D.1 Consistency

  • Without computational libraries, first-step keyword counts rarely identified assumptions or defaults, and the parser supplied provenance marks only in card mode.“Default” appeared in none of 300 no-card first steps and in six of 300 card steps; these counts are a lower bound on assumptions expressed without those terms.Source: §D.1 · p. 27
  • With no computational libraries, explicit-card interpretations agreed on 74% of run pairs and recovered no-card interpretations on 71%, a difference whose interval included zero.On claims grouped both ways, explicit cards and interpretations recovered from the same card runs each agreed on 75% of pairs; Figure 5 displays agreement by field.Source: §D.1 · p. 27, Figure 5 · p. 28
  • With computational tools, explicit cards agreed on 76% of run pairs versus 73% for recovered no-card interpretations, again without a measurable consistency difference.The paired differences were +0.03 and +0.02, and both intervals included zero.Source: §D.1 · p. 28
  • Without computational libraries, within-claim pairs with matching thresholds or quantifiers had nearly the same sign agreement as differing pairs, so interpretation variation did not consistently track verdict variation.The comparisons were exploratory and observational, and the scorer received each run’s card.Source: §D.1 · p. 28
  • With computational tools, stricter-versus-weaker quantifier scores differed by −0.25 across 13 claims, while matching-quantifier pairs agreed in sign less often than differing pairs.The quantifier score difference interval excluded zero, but the paired-sign pattern ran opposite to what verdict tracking would predict.Source: §D.1 · p. 29

E Card Edits in Detail

Making a frozen card’s quantifier universal or its support threshold stricter generally lowered feasibility scores, often below every unedited run. Threshold edits also increased abstention in search-only runs and changed what computational tools were asked to calculate.

E Card Edits in Detail

  • Stricter interpretations lowered scores mainly on claims with positive unedited means, while claims with means of zero or below stayed about where they were.Without the claim at the score floor, mean shifts were −0.63 for quantifier edits and −0.55 for threshold edits.Source: E Card Edits in Detail · p. 31

E.1 A Stricter Threshold That Search Cannot Meet Can Lead to Abstention

  • 15 of 24 threshold-edited materials-science runs abstained without computational libraries, versus 45 of 120 unedited runs; with tools, the counts were 5 of 24 versus 14 of 120.Search-only threshold edits also increased searches and empty search results; the authors suggest the agent could not find evidence meeting the stricter threshold.Source: §E.1 · pp. 32–33, §E.2 · p. 33

E.2 Edit Propagation

  • Edited values appeared more often in edited-run reasoning and tool inputs than in unedited runs, showing that edits reached beyond the scorer to the agent’s work.For threshold edits on the 60 claims, the edited threshold appeared in reasoning in 34 edited runs versus 5 of 280 unedited runs.Source: §E.2 · pp. 33–34
  • With computational tools, threshold-edited runs searched more, and on three claims calculated over the edited composition range rather than the unedited range.On the Al–Zn single-phase claim, the edited run computed x(Zn) at 0.45, 0.50 and 0.55, compared with 0.50 in all five unedited runs.Source: §E.2 · p. 34
  • Some computational-tool examples had irregularities, including a lost CALPHAD result and a τ-phase mole fraction read as a percentage.The corresponding edited scores were excluded from the relevant robustness rows.Source: Table 9 · p. 30, §E.2 · p. 35

F Same Calculation, Different Verdicts

The same calculation could accompany different verdicts, with differences linked to how agents interpreted or applied a claim. The agent also matched expert calculations on fewer than half of the 24 claims, and calculation matches did not measurably align with gold-label signs.

F Same Calculation, Different Verdicts

  • Among tool runs with the same calculation result, average same-score shares were 0.77 under frozen cards and 0.72 in card runs, yet four of five illustrated claims had opposite verdicts.These cases illustrate the pattern rather than estimate how often it occurs.Source: F Same Calculation, Different Verdicts · p. 35, Table 12 · p. 36
  • On the opening claim, all 15 tool runs found a single solid phase only above about 600 K, matching the expert’s diagram, yet all scored −1 against the expert’s label of 0.Every card required a single solid phase at every temperature below 700 K.Source: F Same Calculation, Different Verdicts · p. 35

F.1 Agent and Expert Calculations

  • On the 24 claims, the agent matched the expert’s calculation on 11, used a different setup on 9, differed in value on 1, and calculated nothing on 3.No evidence showed that runs matching the expert’s calculation agreed more often with the gold-label sign.Source: §F.1 · pp. 36–37
  • Some expert calculation records themselves had setup or composition inconsistencies, and the study did not establish whether these changed gold labels.Examples included swapped Mg and Zn in code relative to the claim and an eight-atom cell labeled as 4 at.% solute.Source: §F.1 · p. 37

F Same Calculation, Different Verdicts

  • Different verdicts also arose when runs applied the same result under different criteria or framed nearly equal calculated limits in opposite terms.On the Li–AlCu reaction claim, card and no-card runs scoring −1 and frozen-card runs scoring +2 obtained the same phase field.Source: F Same Calculation, Different Verdicts · p. 35, §F.1 · p. 37

G Card Audit and Overhead

The card audit found that provenance marks did not always match a judge’s assessment, and cards could differ from interpretations agents later applied. Adding a card raised measured run costs, while card and no-card runs showed no measurable difference in sign accuracy.

G Card Audit and Overhead

  • The judge found provenance-mark disagreements in both domains, with materials-science fields mostly marked inferred but judged faithful and biomedical fields mostly marked stated but judged reasonable defaults.For 11 materials-science cards with a judge ruling, each card was compatible with the expert rationale.Source: G Card Audit and Overhead · pp. 37–38
  • On the opening claim, five cards without computational libraries and four of five cards with tools marked as stated a threshold requiring one solid phase below 700 K, although the expert judged the range unspecified.The first-step card could also differ from the interpretation applied later in a run; this was not measured systematically.Source: G Card Audit and Overhead · pp. 37–38
  • On 60 claims, card runs used 40.4K input tokens versus 35.6K without a card, took 3.8 versus 3.5 minutes, and cost $0.27 versus $0.24 per run.The authors note that some model components’ tokens were not counted, so these are not full-run costs.Source: G Card Audit and Overhead · p. 38

H Accuracy, Abstention and Variance

  • On the 60 claims, sign accuracy did not differ measurably between frozen-card and no-card runs.Scores were identical on 47 of 60 pairs of backbone and claim, and the two conditions never gave opposite signs when both decided.Source: H Accuracy, Abstention and Variance · p. 38

H.1 Abstention and Variance Sources

Abstention differed across conditions, especially on materials-science claims, but failed tool calls did not explain the excess in frozen runs. On the 24 claims, zeros were concentrated on two claims, while run-to-run score variation often reflected different verdicts despite similar evidence or calculations.

H.1 Abstention and Variance Sources

  • On the 60 claims, abstention was 0.14 with a card, 0.10 without one, and 0.18 with a frozen card, with most zeros on materials-science claims.The corresponding card-minus-no-card difference was +0.07 for materials science and +0.01 for biomedical claims.Source: §H.1 · p. 38
  • Missing or partial evidence was the primary reason for 32/39 card zeros (82%), 22/29 no-card zeros (76%), and 41/54 frozen zeros (76%); no rationale cited conflicting evidence.Source: §H.1 · p. 39
  • Failed computational-tool calls did not explain frozen runs’ excess abstention on materials-science claims, which persisted after those runs were excluded.Runs with failed calls had lower sign accuracy than other runs in each condition, but comparisons among tool-calling runs are selected on a condition that may influence calls.Source: §H.1 · pp. 39–40
  • On the 24 claims, abstention did not differ measurably between conditions, and two claims accounted for nearly every zero across all settings.Missing or unmatched evidence was the primary reason for most zeros; rejected calculation proxies were more common among no-card zeros, while missing quantities accounted for 10 search-only zeros.Source: §H.1 · p. 41
  • On the 60 claims, score variability was not measurably different across conditions; on the 24, different verdicts on similar evidence accounted for most weighted variation.Different decisive search evidence was another major source of variation, whereas different calculation results contributed none in the coded runs with tools.Source: §H.1 · pp. 41–42

I Pilot on Three LLM Backbones

A 20-claim pilot compared three LLM backbones with and without a Claim Card. The supplied passages describe the pilot’s design but do not report its outcomes.

I Pilot on Three LLM Backbones

  • The pilot compared sign accuracy and abstention for three LLM backbones, each run once with and without a card on 20 claims, 10 per dataset.The supplied table caption identifies the pilot’s measures and setup but gives no outcome values.Source: Table 16 · p. 42, I Pilot on Three LLM Backbones · p. 42

J Tool Use in Detail

This section examines how frozen Claim Cards relate to computational-tool use and tool-mediated verdicts, while considering whether extracted cards faithfully represent an agent’s reasoning. It also situates Claim Cards as records of the assumptions behind a single verdict before evidence gathering.

J Tool Use in Detail

  • Cards were associated with slightly fewer tool calls, but the possible savings were not established because card runs took longer and searched more often.Card minus no-card was −0.05 calls per run; calls returning calculations took a median 2.6 minutes, while searches averaged 1.42 per run with a card versus 1.24 without.Source: J Tool Use in Detail · pp. 42–43
  • Frozen-card tool calls fell on different claims than card and no-card calls, but contrasts were descriptive and may reflect run-day, computer, or card-source differences.Across all 24 claims, frozen runs called tools in 64 of 120 runs, the same total as card runs; permutation tests found differences in which claims received calls, but the authors note potential confounds.Source: J Tool Use in Detail · p. 43
  • Frozen-card wording appeared more in calculation questions than wording in independently generated card or no-card questions, while card fields did not measurably change how often tools were called.On 15 claims, frozen-versus-card overlap was 0.15 versus 0.06, and frozen-versus-no-card overlap was 0.14 versus 0.06; the authors treat this as evidence the card reached the tools, not a tool-use finding.Source: J Tool Use in Detail · p. 44

J.1 Frozen Cards with the Computational Tools and with Search Only

  • Sign accuracy was 0.68 with computational tools versus 0.57 with search only, a paired difference of +0.12, entirely from fewer abstentions.Abstentions were 14 of 120 tool runs versus 28 of 120 search-only runs; wrong signs were 24 runs (0.20) in both settings. The comparison is exploratory, and the authors do not attribute the gain to calculations themselves.Source: §J.1 · p. 45, Figure 7 · p. 45, Table 18 · p. 45

K Extended Related Work

  • Claim Cards address scientific ambiguity by recording the interpretation an agent adopts, unlike related benchmarks that enumerate interpretations, provide evidence, or assess analytic choices after analysis.The paper argues that scientific claims omit domain details needed for assessment, so measures based on predefined interpretation lists do not fit its task.Source: K Extended Related Work · pp. 47–48
  • Because another model extracts a card from the agent’s first reasoning step, the card may be plausible without faithfully reflecting what drives the answer.The passage applies the same faithfulness caution raised for model reasoning and explanations (Chen et al., 2025).Source: K Extended Related Work · p. 48
  • Editing a frozen card field can change the score and often alter tool questions, search context, and the final answer.The edited value can reach the tools through the agent’s questions and the context passed to the search agent.Source: K Extended Related Work · p. 49
  • Interfaces that expose and let users edit a system’s interpretation have helped steer text-to-SQL, programming, data-analysis, code-generation, and planning systems.Claim Cards are shown before the agent’s first tool call; unlike these interfaces, automated fact-checking explanations typically justify verdicts after evidence gathering, while interviewed fact-checkers requested uncertainty and information gaps.Source: K Extended Related Work · p. 49
  • PICO and SciClaim omit support thresholds, value provenance, and feasibility labels, while a Claim Card records assumptions for a single verdict before evidence is gathered.FActScore decomposes text into atomic facts; model cards, datasheets, and data statements document models or datasets, while preregistration commits to hypotheses and analysis plans. Claim Cards are saved with the report.Source: K Extended Related Work · p. 49

L Further Discussion

The discussion shows that computational tools can leave interpretation unresolved even when runs obtain the same calculation, while some verdict-driving choices remain outside the Claim Card. The prompts specify the card fields, frozen-card procedure, scoring inputs, and consistency judgments.

L Further Discussion

  • On two claims, experts used proxies—total magnetic moment and bulk modulus—while the frozen cards named saturation magnetization and Young’s modulus.The runs and experts may therefore have assessed different observables, so a disagreement need not mean the run reached a wrong verdict.Source: L Further Discussion · p. 49
  • Six card/no-card runs and all five frozen runs found the same Li–AlCu three-phase field yet reached opposite verdicts over what “two-phase reaction” meant.The groups ran under different conditions, so the contrast may also reflect those differences.Source: L Further Discussion · p. 49
  • Five frozen runs computed the same Fe–Al B2 limit under a typical-case quantifier, yet their scores ranged from +2 to −1.The card did not specify which result counted as a counterexample; the authors propose decision-rule and tolerance fields for testing.Source: L Further Discussion · p. 49
  • Where a claim has an author to ask, inferred or default card fields could become clarification questions before the run.Source: L Further Discussion · p. 50

M Prompts

  • The materials-science card prompt covers the system and state, method and conditions, observable and units, quantifier, threshold, comparator logic, and sufficiency set.Source: M Prompts · p. 50
  • The prompt distinguishes stated, inferred, and default provenance, and asks for assumed defaults, near-match tolerances, and evidence that would change the score.Source: M Prompts · p. 50
  • Frozen mode instructs the agent not to alter the locked card, while the scorer takes the card and reasoning as inputs and returns a Likert score and confidence.No-card runs remove the card input and card items from the scoring guideline.Source: M Prompts · p. 50
  • The consistency judge groups runs by whether their interpretations would lead an expert to the same test, pass criterion, or scope.Source: M Prompts · p. 51

Related papers

Loading 2610.20457v1…