PaperlayerView exact revision on arXiv

Source-linked AI summary

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents

Zifu Tao, Changqing Yin

arXiv:2609.37153v1cs.AIcs.CR

TL;DR

Data agents can receive plausible but incorrect evidence from successfully executed tools, making it important to understand both what evidence checking provides and which answer agents adopt. ToxicBench pairs clean and poisoned observations over fixed source data; poisoning lowers task success, and repeated poisoning reveals adoption of wrong answers even after checking.

  • Problem

    Tool execution can succeed while returning plausible evidence that supports an incorrect analytical conclusion, and evaluation must distinguish the evidence checked from the answer ultimately adopted.

    Source: §1 · p. 1

  • Method

    ToxicBench pairs clean and poisoned tool observations over fixed source data and tracks checking, answer adoption, and recovery across numerical and semantic interventions.

    Source: §1 · p. 2

  • Results

    Across the 118-task GPT evaluation, poisoning lowered task success by 26–39 percentage points; ordinary retries helped under one-shot poisoning, while repeated poisoning still produced wrong-answer adoption after checking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings motivate evaluating what evidence a check returns and which answer the agent selects, alongside whether it checks at all.

    Source: §5 · p. 9

    Core tasks use small synthetic tables, and generalization to production workflows and naturally occurring errors remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.

1 INTRODUCTION

Tool calls can succeed while returning plausible evidence that supports a wrong conclusion, so data agents must judge whether observations are trustworthy. ToxicBench pairs clean and poisoned observations over fixed source data to examine checking, adoption, and recovery.

1 INTRODUCTION

  • A wrong final answer alone cannot reveal whether an agent accepted its first observation, checked against further bad evidence, or obtained a correction but selected the wrong conclusion.Additional tool calls alone do not establish recovery; evaluation must track subsequent evidence and the adopted answer.Source: §1 · pp. 1–2
  • ToxicBench pairs clean and poisoned environments over fixed source data and tracks whether agents check evidence, adopt poisoned answers, or recover.Numerical and semantic interventions cover altered quantities and changed entity, column, metadata, or retrieved-evidence bindings.Source: §1 · p. 2
  • The benchmark distinguishes silent tool poisoning from altered source data: a proxy changes a successfully executed tool’s observation while the source remains fixed.Unlike indirect prompt injection, the intervention changes returned evidence without asking the agent to follow malicious instructions.Source: §1 · p. 2
  • The comparisons separate the benefit of another attempt from explicit verification and test how later evidence conditions affect recovery.They include controlled comparisons on three public tables and repeated poisoning, where checks may also be corrupted.Source: §1 · p. 2
  • 26–39 percentage points: poisoning lowers task success across three adapters in the 118-task GPT evaluation.Ordinary retries are a strong control under one-shot poisoning, while repeated poisoning reveals wrong-answer adoption after checking; human scoring agrees on task success for 96% of 200 trajectories.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses tool invocation, interactive task solving, silent errors, and trajectory-level evaluation. ToxicBench distinguishes answers supported by returned observations from answers correct against source data, while tracking checking and adoption.

2 RELATED WORK

  • TRACE assesses trajectory efficiency, hallucination, and adaptivity with an evidence bank and LLM judges; ToxicBench instead tracks checking and whether the adopted answer is source-correct.ToxicBench separates support by returned observations from correctness according to source data.Source: §2 · pp. 2–3
  • Prior benchmarks study silent tool errors, broader failures and recovery, stage-aligned diagnosis, and threats to metadata or retrieved content.ToxicBench studies paired data-analysis tasks with fixed source tables and separate checking, adoption, and recovery metrics under one-shot and repeated corruption.Source: §2 · p. 3
  • Data validation targets source quality, while feedback, self-monitoring, and calibration address revision or tool decisions; ToxicBench compares retries and verification under shared budget limits.In ToxicBench, source preservation leaves data available as a checking path.Source: §2 · p. 3

3 TOXICBENCH

ToxicBench pairs clean and poisoned tool environments over fixed source data, making it possible to trace altered observations through checks to the final answer. Its task design, corruption operators, metrics, and verification controls separate evidence availability from answer selection.

3 TOXICBENCH

  • ToxicBench changes a successfully executed tool’s returned observation while leaving source data unchanged, then relates the changed observation to later checks and the final answer.Tasks require inspectable answers, controlled corruptions, and identifiable recovery routes.Source: §3 · p. 3

3.1 TASK CONSTRUCTION AND QUALITY ASSURANCE

  • Synthetic tables, dictionaries, and evidence snippets cover aggregates, rates, rankings, and evidence-backed entity or field choices; construction prioritizes error-mechanism coverage over production workload sampling.Numerical tables have 4–12 rows (median 5.5), and semantic tables have 3–6 (median 3.5). Model-free checks found that all 119 tasks with an unambiguous, conclusion-changing return admit scripted raw-data recovery under one-shot poisoning; this establishes feasibility, not agent success.Source: §3.1 · p. 4

3.2 POISONING OPERATORS AND METRICS

  • Numerical and semantic/schema operators alter quantities, rankings, entity bindings, column meanings, metadata, or retrieved evidence, including errors that numerical plausibility checks cannot resolve.The operators include scaling, sign flips, rank or label swaps, treatment/control flips, stale metadata, and biased retrieval.Source: §3.2 · p. 5
  • Task Success Rate compares final answers with clean references, while Poisoned Degradation is clean TSR minus poisoned TSR; behavioral rates use exposed runs to distinguish delivery from response.Anomaly Detection and Validation measure discrepancy recognition and fresh task-relevant evidence; Recovery requires an accepted clean answer after either, while Poison Adoption, Blind Compliance, and Validated Poison Adoption distinguish poisoned answers by whether checking occurred.Source: §3.2 · p. 5
  • Scoring extracts task-relevant conclusions and corrections before matching clean and poisoned references, and unresolved or conflicting selections remain in the denominator.The same scorer is used across methods; scoring decisions and paired method comparisons are checked against human judgments.Source: §3.2 · p. 5

3.3 VERIFICATION PROTOCOLS

  • Double-pass controls for another opportunity to solve the task; Verification-only hands the first answer to route two as an untrusted claim, while Generic Guard adds expectation prompts and a checklist.The two-route methods share model, route count, step limit, and per-request token cap, but these are not matched total-token or computation budgets. One-shot poisoning leaves later calls clean; repeated poisoning corrupts each eligible observation with probabilities 0.25–1.00.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate poisoned versus clean tool observations across agents and models, then test whether additional attempts, supplied evidence, and human review change recovery. They find substantial poisoning-related declines, retry benefits under one-shot poisoning, and continued adoption after checking under repeated poisoning.

4.1 SETUP

  • The evaluation covers LangGraph ReAct, smolagents, and AutoGen across model families, with an expanded GPT defense comparison on 118 retained instances.The 118 instances comprise 58 numerical and 60 semantic/schema tasks; clean and poisoned runs are paired by task and use the same frozen scorer and audited references.Source: §4.1 · p. 6

4.2 BLIND COMPLIANCE ACROSS AGENTS AND MODELS

  • Task success falls by 0.26–0.39 in the 118-instance GPT evaluation, despite clean TSR of 0.92–0.99 across three adapters.Nonzero BCR in all three adapters identifies adoption of poisoned conclusions without detection or checking.Source: §4.2 · p. 6, Table 1 · p. 7
  • All nine fully crossed configurations show numerical TSR declines, and eight show semantic/schema declines, with vulnerability varying across configurations and task families.Operator profiles suggest magnitude checks can expose numerical errors but not label swaps that preserve amounts while changing entities; these profiles are descriptive, not causal rankings.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 WHAT DOES ADDITIONAL VERIFICATION BUY?

  • Double-pass reaches poisoned TSR 1.00 versus 0.89 for Base, 0.90 for Verification-only, and 0.97 for Generic Guard; explicit verification shows no clear advantage over retry.Later calls may receive unmodified observations after one-shot poisoning, so Double-pass's gain does not isolate verification reasoning.Source: §4.3 · p. 7, Figure 4 · p. 8
  • With fresh clean evidence, retries are correct on 48/48 repetitions and reviews on 47/48, while both methods are correct on 0/48 under full-target corruption.The 24-task control fixes the initial tool call and starts reviews from the same saved poisoned transcript, isolating the local effect of supplied evidence without establishing verification superiority.Source: §4.3 · pp. 7–8
  • Among 101 VPA cases in 1,572 retained trajectories, 78 had only corrupted qualifying follow-up checks and 23 included an unmodified check.The result establishes continued adoption after checking, not that all agents ignored sufficient correct evidence; routes may share sources and backends.Source: §4.3 · p. 8
  • Generic Guard averages 39.79 seconds on retained toxic runs, versus 16.91 for Base, 35.89 for Double-pass, and 33.62 for Verification-only.Protocols share budget caps, not realized compute; gateway token billing is unavailable.Source: §4.3 · pp. 8–9

4.4 HUMAN CHECKS OF SCORING AND METHOD COMPARISONS

  • Automatic task-success scoring agrees with human judgments on 192/200 trajectories (96.0%), and human judgments preserve Double-pass's gain over Base on 20 core tasks.Human poisoned TSR is 1.00 versus 0.80 for Double-pass and Base respectively; human review also identifies all four human VPA cases among 77 exposed trajectories.Source: §4.4 · p. 9

5 CONCLUSION

ToxicBench pairs clean and corrupted tool observations over fixed source data to make evidence use explicit in data-agent evaluation. Results show retry benefits under one-shot poisoning and continued poison adoption after checking under repeated poisoning, motivating evaluation of evidence and answer selection.

5 CONCLUSION

  • ToxicBench pairs clean and corrupted tool observations over fixed source data to make evidence use explicit in data-agent evaluation.The experiments report clean-to-poisoned degradation, retry benefits under one-shot poisoning, and continued poison adoption after checking under repeated poisoning.Source: §5 · p. 9
  • Human judgments corroborate retry benefits over Base on audited tasks and adoption after checking, while public-table controls identify a local effect of supplied evidence on recovery.These findings motivate evaluating what evidence checks return and which answer agents select, alongside whether they check.Source: §5 · p. 9
  • Future work will extend ToxicBench to longer, multi-source workflows and naturally occurring tool errors.Another proposed direction is budget-aware verification that seeks independent evidence and resolves conflicting observations.Source: §5 · p. 9

6 LIMITATIONS

The study's core tasks use small synthetic tables, and generalization to production workflows and naturally occurring errors remains untested.

6 LIMITATIONS

  • Core tasks use small synthetic tables; public-table controls broaden the setting but retain seeded calls and structured answers.Generalization to production workflows and naturally occurring errors remains untested.Source: §6 · p. 9
  • Scorer recall varies across metrics and answer styles, so fine-grained method rankings should be read alongside human comparisons.Historical delivery exceptions and sensitivity analyses are documented separately.Source: §6 · p. 9
  • Reported intervals condition on the selected tasks and sources.The study does not establish generalization to production workflows or naturally occurring errors.Source: §6 · p. 9

A BENCHMARK CONSTRUCTION AND THREAT MODEL

ToxicBench pairs clean and poisoned tool observations while keeping source tables fixed, making evidence checking and answer selection inspectable. Its scoring distinguishes successful execution, delivered corruption, checking, and adopted answers, while acceptance and audit procedures delimit what the benchmark establishes.

A BENCHMARK CONSTRUCTION AND THREAT MODEL

  • The benchmark preserves source tables while proxy interventions alter successful tool returns, targeting plausible false observations rather than malformed outputs or prompt control.The task construction is coverage-oriented, using synthetic tables and explicit specifications rather than queries sampled from user logs; poisoning targets selected matching returns rather than unrestricted source control.Source: A BENCHMARK CONSTRUCTION AND THREAT… · p. 14
  • Each task pairs clean and toxic runs with the same task, model, adapter, tools, and budget, while logging hidden clean observations without supplying them to the model.Tasks specify clean and poisoned references, tolerances or aliases, and an eligible event; delivery checks distinguish valid exposure from ineligible calls and textual-change flags that overcount corruption.Source: A BENCHMARK CONSTRUCTION AND THREAT… · p. 14
  • The inventory groups normalized dataset, query, and poison specifications, not independent semantic problems; its multi-table suite uses four pairs of CSVs.Cross-model suites are contained within expanded suites, while the 13 multi-table tasks use separate source files and the public-data controls are reported separately.Source: Table 3 · p. 14, A BENCHMARK CONSTRUCTION AND THREAT… · p. 14
  • Acceptance checks independently execute references and specifications across all 131 retained synthetic tasks, while the delivery audit flags 204 of 3,319 recorded corruptions in 157 trajectories.The reference audit covers arithmetic, dictionary and evidence selection, table discovery, and declared schema-role bindings; flagged delivery events are not a certification that unflagged events are valid.Source: A BENCHMARK CONSTRUCTION AND THREAT… · p. 15
  • Under fixed calls and rendering, 119 tasks receive an unambiguous conclusion-changing return and admit scripted raw-data recovery under one-shot poisoning; this measures feasibility, not autonomous recovery.Nine multi-table raw-data returns are themselves corrupted under repeated poisoning, and non-exposure is not counted as recovery.Source: A BENCHMARK CONSTRUCTION AND THREAT… · p. 15, Table 4 · p. 15

B SCORING AND HUMAN VALIDATION

  • The scorer separates TSR over all retained runs from behavior rates over exposed runs, and defines poisoned-answer adoption independently from whether checking or validation occurred.A fresh relevant post-poison computation can qualify as verification even when its evidence is corrupted; generic doubt or merely saying “verified” does not establish the corresponding behavior.Source: B SCORING AND HUMAN VALIDATION · pp. 15–16

B.1 HUMAN VALIDATION AND ARCHIVED AUDITS

The post-freeze human evaluation compares frozen scorer judgments with annotations on selected trajectories, while separate development and archived audits have different validation scopes. Human labels corroborate retry gains on audited tasks and show that checking can coexist with adoption of a poisoned answer.

B.1 HUMAN VALIDATION AND ARCHIVED AUDITS

  • The human evaluation contains 200 trajectories from 20 task IDs, including core clean/toxic runs, repeated-poison runs, and AutoGen numerical runs.The task IDs are disjoint from known development packets but share task families and synthetic provenance; annotators agreed on all 200 trajectories with no ambiguous cases.Source: §B.1 · p. 16
  • 96.0% TSR agreement on 200 post-freeze trajectories compares the frozen scorer with human reference labels.Table 7 reports 176 true positives, 16 true negatives, eight false negatives, and no false positives; behavioral metrics use 77 exposed trajectories.Source: §B.1 · p. 16, Table 7 · p. 17
  • On selected core tasks, human Double-pass exceeded Base poisoned TSR by +0.20 [0.05, 0.35]; human repeated-poison Guard–Double-pass contrast was −0.10 [−0.30, 0.00], with clean TSR 1.00 for all four methods.Source: §B.1 · p. 17
  • The development audit used 120 exposed trajectories and blind adjudication, but did not separately label TSR, PAR, or VPA.A targeted review found 18 VR-positive cases relevant, but that review is not an independent accuracy estimate.Source: §B.1 · p. 18
  • The archived scorer recheck reaches 238/240 agreement after repair, but those cases informed the repair and are not independent validation.The original scorer missed 64 of 193 human-correct answers and 36 of 72 recoveries; only three BCR and four VPA positives were available.Source: §B.1 · p. 18, Table 10 · p. 19
  • Four adjudicated VPA cases establish adoption after checking, but not necessarily rejection of sufficient correct evidence.The scorer also identifies all four human VPA cases in the 200-trajectory evaluation, with two each from Double-pass and Guard under repeated poisoning.Source: §B.1 · pp. 16, 19

B.2 FROZEN REVISION AND REFERENCE AUDIT

The revision audit preserves historical trajectories while correcting references and scoring decisions. Its numerical audit identifies concrete reference problems, and the affected-row review supports correctness only for eligible cases, not behavioral-rate comparisons.

B.2 FROZEN REVISION AND REFERENCE AUDIT

  • The versioned comparison covers 5,396 archived trajectories and stores baseline, frozen-parser, and revised decisions while preserving raw answers, events, task files, and human labels.Table 12 combines parser and reference changes on the same 944 retained defense trajectories, so it is not a parser-only effect.Source: §B.2 · p. 19, Table 12 · p. 20
  • The numerical audit found seven out-of-tolerance instances, two omitted ties, and two ambiguous channel-ranking queries among 60 expanded tasks.Both A100 and B200 are accepted as tied at revenue per unit 200; the two denominator-ambiguous queries are excluded consistently across revised methods.Source: §B.2 · p. 19
  • Of 40 affected historical audit rows, 28 eligible trajectories are human-correct and match the frozen scorer, while 12 ambiguous or ineligible references are excluded from accuracy comparisons.All seven exposed trajectories in this review belong to the ineligible group, leaving no eligible behavioral-rate comparison; these results are separate from the post-freeze evaluation.Source: §B.2 · p. 19

C AGENT PROTOCOLS AND FRAMEWORK BOUNDARIES

The agent protocols contrast ordinary retries with explicit verification and guarded verification while sharing route budgets, and the supplementary analyses qualify comparisons by exposure, dependence, uncertainty, and measured cost. Verification uses fresh evidence-producing actions but can still share upstream sources or override a correct answer.

C AGENT PROTOCOLS AND FRAMEWORK BOUNDARIES

  • Double-pass runs two ordinary routes and selects the second response, while Verification-only passes the first answer as untrusted and Generic Guard adds expectation prompts.All two-route variants use equal step and output-token caps, but equal caps do not imply equal realized tokens, calls, or latency.Source: C AGENT PROTOCOLS AND FRAMEWORK… · p. 20
  • The 13-task multi-table extension reconstructs joins from raw keys, providing a fresh execution route but not source diversity.It cannot detect shared CSV, parser, or join-key errors.Source: C AGENT PROTOCOLS AND FRAMEWORK… · p. 20
  • Abstention on route disagreement lowers detected BCR but also answer coverage; the deterministic and keyword gates are not learned or generally calibrated.These alternative policies are fixed-subset diagnostics.Source: C AGENT PROTOCOLS AND FRAMEWORK… · p. 20
  • Table 14 reports suite-level TSR, exposure-conditioned behavior, and delivery; its detected BCR events are 8/40 for numerical Base, 1/59 for semantic Base, and zero for other variants.Conditional behavior rates use exposed runs and are not direct rankings of trust calibration.Source: Table 14 · p. 21, D SUPPLEMENTARY RESULTS AND… · p. 21

D.1 PAIRED INFERENCE AND MEASURED COST

  • The combined paired interval includes zero, and behavior contrasts require exposure in both variants rather than comparing marginal rates.Zero detected BCR differences do not prove equal human-judged risk; the 118 instances form 87 template clusters across 30 CSVs, so cluster-resampling addresses dependence rather than scorer error or API variance.Source: §D.1 · p. 21
  • Zero-event BCR cells still have approximate one-sided 95% upper bounds of 0.07 for numerical two-route variants and 0.05 for semantic variants.A degenerate empirical bootstrap is not evidence of zero true risk.Source: §D.1 · pp. 21–22
  • For 13 join tasks at p = 1, mean latency is 41.59 seconds for Verification-only, 61.48 for Double-pass, and 54.34 for Guard.These small-sample timings are sensitive to backend queueing; tool-event counts are measured costs, not token billing.Source: §D.1 · p. 22

D.2 REPEATED POISONING AND EVIDENCE AVAILABILITY

Repeated poisoning tests whether agents can recover when checks may also be corrupted. At p = 1, no cell has detected BCR, yet wrong-answer adoption after checking appears in adjudicated cases.

D.2 REPEATED POISONING AND EVIDENCE AVAILABILITY

  • Repeated poisoning applies a deterministic hash gate at p ∈ {0.25, 0.50, 0.75, 1.00} to eligible operator-matching returns, while nonmatching calls can remain unmodified even at p = 1.The revised matrix retains 1,572 trajectories after reference exclusions and has no single-route Base control.Source: §D.2 · p. 22
  • At p = 1, all nine cells have zero detected BCR but nonzero PAR and VPA.The table reports suite-specific behavioral rates; they condition on exposed runs.Source: §D.2 · p. 22, Table 17 · p. 23
  • Of 101 VPA trajectories, 78 had only corrupted qualifying checks and 23 had at least one unmodified check; these trace categories do not prove sufficient correct evidence was rejected.Only one trajectory had an automatically identifiable clean-reference match, which still required human review.Source: §D.2 · p. 22
  • The 13-task join estimates are imprecise, and finite-sample curves need not be monotonic.Source: §D.2 · p. 22, Table 18 · p. 23
  • Four adjudicated VPA cases establish adoption after checking, although a corrupted check remains a possible explanation.Source: §D.2 · p. 23
  • Fluent arithmetic can repeat a wrong entity binding, while a fresh group-by or reconstructed join can restore the clean answer; extra checking can also exhaust budgets or return errors.Reduced BCR alone therefore does not establish recovery.Source: §D.2 · p. 23

E.1 HISTORICAL DELIVERY AND REPAIRED SCALAR CONTROLS

Historical delivery audits flag identifiable intervention errors and test sensitivity without changing final answers. Repaired-injector controls examine selected tasks and public-table execution, but do not establish broad generalization or recovery from fully removed correct evidence.

E.1 HISTORICAL DELIVERY AND REPAIRED SCALAR CONTROLS

  • 204 of 3,319 recorded corruptions across 157 trajectories were mechanically flagged, including wrong-target selections, partial numeric tokens, and value-preserving edits.The flags are not an exhaustive semantic-validity assessment, and unflagged events are not certified valid.Source: §E.1 · p. 23
  • Double-pass minus Base TSR was +0.118 (95% interval [0.064, 0.182]) on 110 defense tasks after removing eight sign-flip instances.Guard minus Double-pass was −0.027 [−0.064, 0.000], and Verification-only minus Double-pass was −0.109 [−0.173, −0.055].Source: §E.1 · p. 23, Table 21 · p. 25
  • The repaired injector targets configured clean values and rejects ambiguous, missing, or value-preserving scalar matches, reducing false delivery but potentially lowering delivery rates.A scalar match does not prove semantic field identity; observations remain available for inspection.Source: §E.1 · p. 24
  • All four six-task public-data conditions achieved TSR 6/6, but each exposed Base return retained an unchanged correctly rounded answer alongside the altered scalar.The six tasks use one 344-row Palmer Penguins source; this is a partial-corruption execution check, not recovery after correct evidence was removed.Source: §E.1 · pp. 24–25, Table 23 · p. 26
  • On the frozen 16-task control subset, Base already solved 16/16 toxic tasks, so per-route poisoning did not replicate the historical retry advantage.The subset deliberately covers a discovered defect and is not a representative prevalence estimate.Source: §E.1 · pp. 24–25, Table 22 · p. 26
  • The pre-execution protocol fixed tasks and analyses, but intervals cover selected tasks and one execution per condition; production generalization remains untested.Source: §E.1 · p. 25

E.2 FIELD-BOUND, MATCHED-PARENT EVIDENCE CONTROLS

Structured-evidence controls vary whether supplied target evidence is clean or fully corrupted while holding review histories matched. Clean evidence supports high second-stage correctness, whereas full corruption yields widespread adoption of incorrect answers.

E.2 FIELD-BOUND, MATCHED-PARENT EVIDENCE CONTROLS

  • Full-target corruption changes all three aliases of the target result, while raw rows, source files, and alternative aggregates remain unmodified.Thus, full-target corruption does not remove every possible correct evidence route.Source: §E.2 · p. 26, Table 24 · p. 27
  • Review branches copied the full-poison parent transcript, while retries restarted from the original question; both received a mandatory fresh target-aggregate observation.Each policy-condition had two completions per task, and later matching calls followed the assigned clean or full-corruption condition.Source: §E.2 · p. 26, Table 24 · p. 27
  • First-stage correctness was 24/24 clean, 20/24 partial, and 0/24 full; full-minus-partial TSR was −0.833 (95% interval [−0.958, −0.708]).All 24 full-poison primary answers adopted the incorrect scalar.Source: §E.2 · pp. 27–28
  • Second-stage correctness was 48/48 for clean retries and 47/48 for clean reviews, versus 0/48 for each policy under full corruption.The cells aggregate 24 tasks with two completions per policy-condition, not 48 independent tasks.Source: Table 24 · p. 27, Figure 8 · p. 27, §E.2 · p. 28
  • Under full corruption, all 48 retry answers and 46 of 48 review answers matched the poisoned scalar; two other reviews scaled that poisoned value again.Absence of a direct poisoned-value match therefore need not mean rejection of its evidence.Source: §E.2 · p. 28
  • Clean-review branches ran unmodified alternative queries in 10/48 trajectories, but full-poison reviews made no successful alternative query beyond the mandated observation.The experiment does not test self-initiated verification or establish how much later clean evidence explains historical retry gains.Source: §E.2 · p. 28

Related papers

Loading 2609.37153v1…