Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

• Long tool outputs can be delivered inline or saved for on-demand reading, but prior work leaves unclear whether resulting errors reflect computation, exposure, or answering despite seeing the correct value.The paper identifies this question using a deterministic simulator whose reports can be traced value by value.Source: §1 · p. 1
• The study evaluates 158 execution-verified multiple-choice items by re-running logged simulations and distinguishing whether the keyed value was computed, reached the agent, and yielded a correct answer.The tool is PHREEQC, whose reports reach 846.8k characters; the interface changes report delivery between inline and indexed retrieval.Source: §1 · p. 1
• Pooled over runs, TOC changes accuracy relative to inline output by −11.3 pp for GPT-5.1; Claude Opus 4.6, Claude Sonnet 4.6, and GPT-5.4 range from −0.6 to +0.2 pp, with intervals including zero.GPT-5.4 loses about ten points to evidence on numeric items but computes the keyed value more often under retrieval, so its net change is indistinguishable from zero.Source: §1 · pp. 1–2
• GPT-5.1 and GPT-5.2 compute the keyed value about as often under retrieval but often skip the report, use their own higher-precision output files, and truncate values to neighboring options.The paper calls this evidence substitution; holding the simulation fixed, TOC still costs GPT-5.2 9.4 pp.Source: §1 · p. 2
• Naming the report as authoritative raises accuracy by +6.3, +7.9, and +7.8 pp for GPT-5.1, GPT-5.2, and GPT-5.4; inlining reports that fit has the same effect, but rounding instructions do not.The report instruction cuts report skipping to under 2% and raises closure to 99% without changing build.Source: §1 · p. 2

2.1 Items and Verification

• The researchers retained 158 of 200 items from 19 setups after re-execution confirmed that each printed value matched its keyed option exactly and uniquely and remained stable under stricter numerical settings.The item author confirmed the 42 excluded items as defective; exclusion did not use model outputs.Source: §2.1 · pp. 2–3

2.2 Agents and Output Delivery

• Agents used file, line-range read, listing, and PHREEQC execution tools, with conditions returning reports inline, through a section index, through keyword search, or via a hybrid interface.The evaluation covered Claude, GPT, and Gemini models, with one headline run per model and condition and additional reruns or interventions on subsets.Source: §2.2 · p. 3

2.3 Locating Where an Agent Fails

• The failure analysis re-ran 11,007 simulator calls and labeled whether an agent computed the keyed value, saw it in a result, and answered correctly given it.Accuracy decomposes as p s c plus answers given without the keyed value in view; the Shapley decomposition allocates observed differences among stages but does not estimate independent causal effects.Source: §2.3 · p. 3

• Simulator access adds 15.8–45.6 pp over the better no-tool baseline for five Claude and GPT models, while Gemini 3 Flash has no parseable answer on 46% of items inline and 71% under TOC.Gemini is therefore left out of the analyses below.Source: §3 · p. 3
• −11.3 pp for GPT-5.1 and −7.9 pp for GPT-5.2 are the pooled TOC-minus-inline accuracy differences; Claude Opus 4.6, Claude Sonnet 4.6, and GPT-5.4 show no detectable net cost.The reported differences for the latter models are −0.6, −0.5, and +0.2 pp, respectively, with intervals including zero.Source: Table 1 · p. 4, §3 · p. 4
• Grep, which does not use PHREEQC section structure, does no better than TOC, and the retrieval cost is not specific to the TOC index.For both GPT-5.1 and GPT-5.2, the inline runs score higher than every TOC run and the Grep run.Source: §3 · p. 4

4 Agents Answer from Their Own Output

• GPT-5.1 and GPT-5.2 produce the keyed value at similar rates across interfaces, so their retrieval losses are not detectably due to computation.The build contribution to the accuracy change stays within 2 pp, with intervals that include zero.Source: §4 · p. 4
• −9.4 pp [−14.5, −4.6]: GPT-5.2’s TOC accuracy loss persists when its simulation is held fixed.The loss is also −10.1 pp on reports that fit inline and −9.9 pp when the frozen report contains the keyed value.Source: §4 · p. 4
• For GPT-5.2 under TOC, closure accounts for −7.7 pp and exposure for −2.5 pp of the −11.4 pp numeric-item change; GPT-5.1’s loss splits about evenly between them.These stage totals are relative to one logged inline run and cover numeric items, unlike the pooled contrast.Source: §4 · p. 4
• Under TOC, GPT-5.2 and GPT-5.1 receive no part of result.out in 60.3% and 53.0% of trajectories, often reading their own selected-output files instead.Those files print more digits than the report, and GPT models often truncate values to a neighboring option.Source: §4 · pp. 4–5
• When a higher-precision value rounds to the key but truncates to a distractor, GPT-5.2 chooses the neighbor in 26/35 TOC trajectories and GPT-5.1 in 19/27, versus 1/35 for Sonnet.These cases make up about two thirds of TOC trajectories that saw the keyed value but still answered wrongly; the comparison is observational because the trajectory groups can differ in their items.Source: §4 · p. 5
• GPT-5.4’s TOC evidence-use losses of −2.6 pp in exposure and −7.2 pp in closure are offset by a +4.1 [+0.0, +8.1] pp build gain, leaving a net change indistinguishable from zero.The net is −4.4 [−12.4, +3.4] pp on numeric items and +0.2 pp pooled over runs and items.Source: §4 · pp. 5–6

5 Intervening on What Agents Read

• +7.9 pp for GPT-5.2 and +6.3 pp for GPT-5.1: naming result.out as authoritative raises accuracy over all TOC runs.The instruction reduces report skipping to 0.0% and 1.6%, raises closure to 98.8% and 99.2%, and leaves build unchanged within run-to-run variation.Source: §5 · p. 6
• +0.6 pp for GPT-5.2 and +1.2 pp for GPT-5.1: asking agents to round correctly changes accuracy only slightly relative to all TOC runs.They still skip the report about as often, closure stays near 82%, and they choose the truncated neighbor in 17/25 and 14/19 trap cases.Source: §5 · p. 6
• +7.8 pp [+2.2, +13.6]: the instruction raises GPT-5.4 accuracy over its TOC runs to 81.6% in both runs.Report skipping falls to 0.0% and closure rises to 99.0%, while the retrieval build gain remains +3.4 pp relative to inline.Source: §5 · p. 6
• +4.6 pp for GPT-5.2 and +8.0 pp for GPT-5.1: Hybrid, which inlines reports that fit, raises accuracy over the TOC control.Report skipping nearly disappears and closure returns to its inline level; GPT-5.1’s remaining difference from inline is outside closure.Source: §5 · p. 6
• Both interventions restore closure without changing build; Hybrid needs no tool knowledge, while retrieval-preserving instructions must name the canonical output.The GPT-5.2 paired replay also shows the loss with computation fixed.Source: §5 · p. 6

6 Discussion

• Retrieval makes the report compete with files the agent wrote itself; inline delivery instead keeps one source in view.The paper relates this difference to which information a bounded agent can use, not whether the interfaces contain the same information.Source: §6 · p. 7
• A near-zero net accuracy change can hide an evidence-use loss, so harness delivery should be validated for each model and by failure stage.The paper identifies inlining outputs that fit as a safe default and recommends naming the authoritative output when agents can create their own files.Source: §6 · p. 7

7 Related Work

• Prior harnesses truncate, mask, compress, or file long outputs; Sen et al. (2026) report lower accuracy with file-based search results in nine of ten harness–model pairs.The cited work hypothesizes a read–integrate–retry cycle, while this paper locates the loss within trajectories and tests it by intervention.Source: §7 · p. 7
• Work on bounded receivers distinguishes information being present from information being usable; this paper maps exposure and closure to absorption and closure.The paper notes that deterministic simulation has no externalization loss.Source: §7 · p. 7
• Prior agent-failure studies use judges, counterfactual replay, or reference-claim diagnostics; re-executing deterministic simulator inputs supplies ground truth at each stage without a judge.The related-work discussion also notes that models prefer self-generated over retrieved context and round significant digits unreliably.Source: §7 · p. 8

8 Conclusion

• On 158 PHREEQC items, retrieval costs GPT-5.1 and GPT-5.2 accuracy, with no detectable net cost for Claude Opus 4.6, Claude Sonnet 4.6, or GPT-5.4.GPT-5.4’s more frequent correct simulations mask an evidence-use loss, leaving its net change indistinguishable from zero.Source: §8 · p. 8
• Under retrieval, GPT models often skip the report, answer from higher-precision output files, and misround the keyed value to a neighboring option.With simulation held fixed, retrieval still costs GPT-5.2 9.4 points.Source: §8 · p. 8
• Naming the authoritative output or inlining reports that fit removes the closure loss, while asking for correct rounding does not.Deterministic tools make evidence-use losses visible even when final accuracy hides them.Source: §8 · p. 8

9 Limitations

• Evidence substitution appears in all three GPT models and rarely in Claude Sonnet 4.6, but the study does not determine which model property drives that difference.Full trajectory logs were available for Claude Sonnet 4.6.Source: §9 · p. 8
• The Claude agents used temperature 0 without extended thinking, while GPT and Gemini used API defaults without explicit reasoning effort, leaving persistence under explicit reasoning unknown.The study tests one wording of each instruction.Source: §9 · p. 8
• API-budget constraints left some models unrun, unrepeated, or unlogged in some conditions, and possible served-model changes between run dates were not modeled.The four TOC runs shared one run on the 33 long-output items; the paired replay covers GPT-5.2 and starts from inline-run checkpoints.Source: §9 · p. 8
• Reproducing a retained item’s key with PHREEQC is not an independent scientific review of the question.The text under-specifies reference inputs for 24 retained items; excluding them leaves 134 items and the same conclusions.Source: §9 · p. 8

A Item Verification

• The authors re-executed the 200-item pool and retained 158 items from 19 setups only when the keyed value matched uniquely and remained stable under specified checks.The item author confirmed all 42 excluded items were defective as written.Source: A Item Verification · p. 10
• Accuracy rises after defective items are removed, but model ordering and the TOC-versus-Raw100k pattern remain the same across the 200-, 158-, and 134-item sets.The pattern is no detectable difference for Claude models and GPT-5.4, and a loss for GPT-5.1 and GPT-5.2.Source: A Item Verification · p. 10, Table 6 · p. 10

B Prompts, Tool Results, and Conditions

• The prompt changes across conditions are limited to the execute_phreeqc description and the named mode; Grep alone adds a fifth tool definition.No-tool baselines receive the item text alone with one of two system prompts.Source: B Prompts, Tool Results, and Conditions · p. 10
• Descriptions may partly induce different reading behavior because inline discourages additional reads while retrieval requires them, though paired replay still shows a loss with shared prompts.The inline condition is Raw100k; the retrieval condition uses TOC.Source: B Prompts, Tool Results, and Conditions · p. 10
• Raw100k returns the report’s first and last 50,000 characters, while TOC returns section headers and starting lines for ranged reads.TOC+Report and TOC+Round append their instructions to the tool-result hint as well as the system prompt.Source: B Prompts, Tool Results, and Conditions · pp. 10–11
• In Figure 4’s GPT-5.1 TOC trajectory on item 121, the agent read its selected-output file after the run, answered with the printed value’s truncation, and never opened result.out.The index pointed to the section that prints the keyed value.Source: B Prompts, Tool Results, and Conditions · p. 11
• TOC+Report reached 67.1% and 66.5% for GPT-5.2, 62.7% and 60.8% for GPT-5.1, and 81.6% in both GPT-5.4 runs.GPT-5.4’s logged inline run reached 76.6%.Source: B Prompts, Tool Results, and Conditions · p. 11
• TOC+Round reached 62.0% and 57.0% for GPT-5.2 and 55.7% and 57.6% for GPT-5.1.Unlike TOC+Report, this condition asks for correct rounding but does not name an authoritative file.Source: B Prompts, Tool Results, and Conditions · pp. 10–11

C Failure Localization

• The analysis rebuilds and re-runs 11,007 simulator calls, then classifies numeric trajectories by whether the keyed value was produced, seen, or followed by a correct answer.Matching is numeric: a printed value can count even if it appears on a line for a different quantity.Source: C Failure Localization · pp. 11–12
• Re-execution reproduces stored reports in 4,971 of 5,009 cases; the remaining differences are only in line order.The keyed value appears in reference output for all 137 numeric items, though numeric matching can mistake a value for another quantity.Source: C Failure Localization · p. 12
• In a 68-trajectory manual audit, seven of 24 TOC losses involved the retrieval channel, two were mistakes with the value in hand, and fifteen were build failures.Outcome-selected losses were dominated by build failures that churn between runs; the sample was not a population estimate.Source: C Failure Localization · p. 13
• The manual and automatic stages agreed in 52 of 58 numeric trajectories, with five disagreements caused by matching a number printed for another step or solution.Navigating to the wrong report section occurred once, and all eight Sonnet failures in the audit were build failures.Source: C Failure Localization · pp. 13–14
• On 93 numeric items, TOC closure was −9.8 [−15.4, −4.5] for GPT-5.2, −6.8 [−11.2, −2.9] for GPT-5.1, and −7.4 [−11.7, −3.8] for GPT-5.4.TOC exposure was −3.2 [−6.5, −0.0], −4.9 [−8.0, −2.5], and −3.3 [−6.3, −0.2] pp for GPT-5.2, GPT-5.1, and GPT-5.4, respectively; intervals are 95% item-bootstrap intervals.Source: C Failure Localization · p. 12, Table 9 · p. 14
• GPT-5.1 declined to choose an option in 6.1% of TOC trajectories versus 1.9% inline; 19 of 22 distinct numeric TOC non-commits had never produced the keyed value.About half of GPT-5.2’s distinct TOC non-commits were step-budget exhaustion.Source: C Failure Localization · p. 14

D Paired Replay with Computation Held Fixed

• Paired continuations changed only the pending result’s delivery: Raw100k returned the report inline, while TOC returned its section index; simulation files stayed fixed.Three continuations per item and condition used reconstructed, shared files verified by hashes.Source: D Paired Replay with Computation Held… · p. 14
• −9.4 pp mean accuracy difference (TOC minus inline; 95% interval [−14.5, −4.6]) across 131 frozen checkpoints.The sensitivity interval resampling 19 setups was [−13.9, −5.2].Source: Table 10 · p. 15, D Paired Replay with Computation Held… · p. 15
• For 115 reports that fit inline, accuracy was 75.4% inline versus 65.2% under TOC (−10.1 pp); for 16 longer reports, it was 41.7% versus 37.5%.On 84 items whose frozen report contained the keyed value, accuracy was 93.7% inline and 83.7% under TOC.Source: D Paired Replay with Computation Held… · p. 15

E Gemini 3 Flash Outcomes under TOC and Raw100k

• 27.8% under TOC versus 50.0% with Raw100k for Gemini 3 Flash; inline delivery increased answer commitments, while committed-answer accuracy was similar (44 of 46 versus 79 of 85).The table covers outcomes on 158 items in the headline runs.Source: E Gemini 3 Flash Outcomes under TOC… · p. 15, Table 11 · p. 15