PaperlayerView exact revision on arXiv

Source-linked AI summary

Correct Answers, Unsupported Findings: Evidence Binding in Forensic Reconstruction of LLM Agent Logs

Taehyeon Yun, Dongho Kim, Geonwoo Kim, Juyoung Seo, Minseok Hur, Moohong Min

arXiv:2610.09581v1cs.CRcs.AI

TL;DR

Forensic reconstruction must establish which preserved record supports an agent-log finding, because agreement with a complete reference can coexist with an unsupported citation claim. The paper audits literal and citation relations under controlled changes to visible evidence and bindings, finding that explicit bindings improve grounded reconstruction while identifier renaming alone does not reliably do so.

  • Problem

    A citation identifier names a source only when its binding to a particular record is preserved, so complete-reference agreement alone may not show that the supplied evidence supports a finding.

    Source: Introduction · p. 1

  • Method

    The paper audits 64 saved Banking cases, separating reference agreement, evidence-grounded findings, abstention, and unsupported assertions while varying evidence bundles, bindings, and identifier spelling.

    Source: Introduction · p. 1, Study II: binding availability and… · p. 5

  • Results

    26 of 28 C00 mapping-required cases drew unsupported citation-source assertions from Sonnet, and 22 matched the complete reference; explicit bindings improved grounded reconstruction for both readers, while renaming alone had no consistent remedy.

    Source: Introduction · p. 2, Conclusion · p. 10

  • Takeaways & Limitations

    For the tested questions, preserve record content, execution order, channel identity, and execution-scoped bindings, and report support status separately from agreement with a complete reference.

    Source: Conclusion · p. 10

    The audit covers 64 cases from 13 tasks in one simulated Banking environment and does not estimate operational prevalence or quantify benefits to casework.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Forensic reconstruction of LLM-agent actions requires not only recovering the correct value, but establishing which preserved record supports that finding. Tool logs, generated explanations, and local citation identifiers capture different parts of this evidence, yet a citation identifier does not establish a source unless its binding to a record is preserved. We audit this distinction using 64 mechanically checkable cases from saved AgentDojo Banking executions. Two LLM readers reconstruct source relationships under controlled variations in visible evidence and identifier-to-record bindings. We separately evaluate complete-record agreement, evidence-grounded findings, justified abstention, and unsupported assertions. With original identifiers and no binding table, Sonnet recovered every literal source location but made unsupported citation-source assertions in 26 of 28 cases requiring the missing relation; 22 nevertheless matched the complete reference. Adding explicit bindings improved grounded reconstruction for both readers, whereas identifier renaming alone provided no consistent remedy. A deterministic same-packet comparator correctly resolved the bounded task or abstained throughout. These results show that factual agreement alone is insufficient for evaluating forensic reconstruction of agent logs and motivate preserving explicit record bindings to distinguish supported findings from correct guesses.

Introduction

Forensic reconstruction must establish which preserved record supports a finding, not merely recover a correct value. The paper audits literal occurrence and recorded citation relations separately, against a background where complete-reference agreement can conceal unsupported links.

Agent attacks and the evidence left behind

  • External documents can carry prompt-injection instructions and later be examined as evidence, so agreement alone cannot establish that a finding is supported by the supplied records.Prior work identifies limitations of LLM assistance and the need for task-specific validation [23] [24].Source: Introduction · p. 1

Introduction

  • The packet-level audit separates complete-reference agreement, supported findings, justified abstention, and unsupported assertions when checking literal occurrence and recorded citation relations.It is evaluated on saved agent executions; the studies use 64 cases.Source: Introduction · p. 1

Supported attribution and model-reader validation

  • Prior attribution and factuality evaluations distinguish answer correctness from citation or claim support,,,; this audit tests literal occurrence and recorded ID-to-record relations.Citation correctness is also distinguished from causal faithfulness.Source: Supported attribution and model-reader… · p. 2

Forensic provenance and reconstruction validation

The audit applies provenance and task-specific validation ideas to source-reference findings in saved agent records. Its reconstruction studies use controlled evidence packets and a simulated Banking environment.

Forensic provenance and reconstruction validation

  • Forensic studies of investigative assistance motivate checking generated findings against records available to support them [23] [24] [15].Function-oriented validation similarly specifies operations and their evidential prerequisites.Source: Forensic provenance and reconstruction… · p. 2
  • W3C PROV-DM and CASE represent provenance entities, traces, and relations; this audit applies that perspective to execution-local citation IDs through an ID-to-locator table.The table makes a recorded relation checkable without adding the underlying request or document text.Source: Forensic provenance and reconstruction… · p. 2
  • Matched records and explicit visibility rules separate accurate literal retrieval, supported citation relations, and justified abstention when a required binding is unavailable.The retrospective audit tests findings against the packet supplied for analysis.Source: Forensic provenance and reconstruction… · p. 2

Execution environment and public accounts

  • Both studies reuse the same 64 applicable cases, while the second varies citation bindings and identifier spelling; all readers and the comparator receive the same packet within each condition.Full-log references are used only for scoring, with agreement and visible-evidence support assessed separately; the Banking environment is simulated.Source: Execution environment and public… · pp. 2–3, Figure 1 · p. 3

Corpus-generation context

The paper retrospectively audits a fixed inventory drawn from an earlier agent campaign, whose outcomes and saved-view comparisons provide corpus-generation context. Reconstruction cases were selected independently of incident status and reader correctness.

Corpus-generation context

  • The earlier campaign completed 694 of 696 assignments, with 23 incidents, 394 nonincidents, and 277 unresolved episodes; only three cases met the source-eligibility rule.No A3 payload was delivered in its 214 completed assignments.Source: Corpus-generation context · p. 3
  • The pilot removal experiment had one eligible case and no incident in either condition (0/2 each), leaving the removal effect unidentified; the follow-up and public-payload outcomes were also limited in scope.The public-payload development had eligible incidents of 0/18, 1/18, and 3/18 for baseline, AutoDojo-derived, and TopicAttack-derived payloads; all four incidents were in Mini on one bill task.Source: Corpus-generation context · p. 3

Earlier saved views

  • L0/L1 answers were identical in 642/694 episodes, including 607 none/none pairs, despite a 13.9% increase in input tokens; these are stability and input-volume observations.The 694 episodes yielded 2,082 original Luna readings across the saved views.Source: Earlier saved views · p. 3

Fixed reconstruction inventory

  • The inventory contains 64 applicable episodes from 96 candidates—22 Mini, 17 Haiku, and 25 Flash—across 13 tasks; the other 32 are structurally inapplicable and were not replaced.A fixed rule selected the first qualifying action and argument; selection did not use incidents, task utility, attack delivery, or reader correctness.Source: Fixed reconstruction inventory · pp. 3–4

Reconstruction method

The reconstruction contract asks what preserved records support, rather than what probably happened inside a model. It separates literal occurrence in the request, earlier records containing the value, and records explicitly named by the saved citation through a binding.

Illustrative reconstruction case

  • In the example, both packets support locating the value in the earlier bill record, but only the packet with the ID-to-record table establishes that the saved citation names it.Neither packet supplies the authentic request, so Q1 requires abstention; the target action’s repeated value is not earlier evidence.Source: Illustrative reconstruction case · p. 4
  • Sonnet returns the same Q3 answer with and without the table, but without the binding it is a correct guess against the complete record rather than a finding supported by the packet.The example is selected as the first case by ID among 22 matching unsupported answers and is illustrative, not an estimate of error prevalence.Source: Illustrative reconstruction case · p. 4

Questions and observation references

  • Q2 collects addresses of all earlier external records containing the target literal; matching is literal and case-sensitive, and target-action or later results are excluded.If several earlier records contain the value, all are included; if none does, the set is empty.Source: Questions and observation references · p. 4
  • Q3 intersects those Q2 addresses with the record addresses obtained by translating the saved Basis IDs through their mapping.Complete-record references are scorer-only; a missing binding requires abstention when earlier matches and a nonempty Basis leave the relation unresolved.Source: Questions and observation references · p. 4

Scoring findings and justified abstention

The scoring separates agreement with the complete record from correctness justified by the visible packet. It also distinguishes justified abstention, supported definite findings, and unsupported assertions.

Scoring findings and justified abstention

  • Complete-reference agreement (W) can reward a matching answer even when the reader could not see the evidence needed to support it.In the example, Sonnet’s unchanged answer has W = 1 in both conditions, but G and D change when the binding table is supplied.Source: Scoring findings and justified… · pp. 4–5
  • Grounded correctness (G) scores an answer when it matches what the visible packet justifies, including a justified abstention.Supported definite correctness (D) counts only grounded answers that make a definite finding.Source: Scoring findings and justified… · p. 5
  • Justified abstention earns G = 1 and D = 0, while a supported definite answer earns G = 1 and D = 1.The justified-abstention count is A = G − D.Source: Scoring findings and justified… · p. 5
  • Unsupported assertions include definite answers to unanswerable questions and claims to evidence or locators outside the visible contract; they are not simply incorrect answers.A correct guess can have W = 1 and U = 1, while omitting part of a valid source set can be wrong without adding an unsupported locator.Source: Scoring findings and justified… · p. 5

Study I: actor accounts and native inputs

Study I compares bundled evidence conditions while keeping the tool log and target fixed. Study II varies access to ID-to-record bindings and identifier spelling, with packet-only comparator decisions providing a bounded reference for Q3.

Study I: actor accounts and native inputs

  • Study I compares four bundled evidence conditions, retaining the same tool log and target while varying actor-account and native-input evidence.Incident labels, attack assignments, clean authorization references, previous readings, and answer references are withheld.Source: Study I: actor accounts and native… · p. 5
  • Q2 is Study I’s primary endpoint because all four conditions provide the relevant literal evidence; Q1 and Q3 are secondary, with answerability reported.Sonnet read the same 256 packets after Luna, so the comparison is retrospective rather than an independently held-out replication.Source: Study I: actor accounts and native… · p. 5

Study II: binding availability and identifier spelling

  • Study II retains 64 cases and compares adding the complete ID-to-record table with consistently renaming actor-local IDs, separately and together.The table adds no user or document text; Q1 requires abstention in every condition, and Q2 is a negative control because its literal evidence is unchanged.Source: Study II: binding availability and… · p. 5
  • Table 2 scopes IDs and locators to one saved execution, and its mapping rows provide relations rather than record text.The case manifest identifies the execution.Source: Table 2 · p. 6
  • With no binding table, the comparator abstains on 28 Q3 cases; with the table, it abstains on 18, while all 64 decisions in either condition are grounded and none unsupported.The table reports G = 64 and U = 0 per condition; supported decisions include nonempty sets, empty sets, and observed Basis absence.Source: Study II: binding availability and… · p. 5, Table 3 · p. 6

Execution, pairing, and uncertainty

The studies use paired readings of the same recorded actions and calculate task-equal changes across 13 tasks. Their bootstrap intervals are exploratory, and retained artifacts support auditing the reported comparisons.

Execution, pairing, and uncertainty

  • All 1,024 assignments were repeated readings of 64 cases nested in 13 tasks, with each condition contrast paired on the same action and target literal.Task-level correctness changes are averaged within each task and then across tasks, giving each task equal weight despite differing case counts.Source: Execution, pairing, and uncertainty · p. 6
  • Equation 4’s contrast uses question-specific correctness: case changes contribute +1, −1, or zero, and multiplying the task-equal average by 100 gives percentage points.Study I uses Q2 and Study II uses Q3; the questions are not combined.Source: Execution, pairing, and uncertainty · p. 6
  • Intervals from 10,000 whole-task bootstrap draws are exploratory descriptions, not confirmatory tests, equivalence bounds, or acceptance gates.The analysis has 13 selected task clusters, and bootstrap validity depends on the cluster-dependence model.Source: Execution, pairing, and uncertainty · p. 6

Reproducibility and retained evidence

  • The retained artifacts trace reported comparisons through case manifests, visible packets, references, reader outputs, and scoring decisions.The package also retains native requests and responses, prompts, schemas, renaming tables, and run metadata.Source: Reproducibility and retained evidence · p. 6

No observed retrieval gain from the actor account in Study I

Study I shows no observed retrieval gain from adding the actor account, while Study II finds that explicit bindings improve evidence-grounded citation findings. Complete-reference agreement and grounded correctness capture distinct outcomes.

No observed retrieval gain from the actor account in Study I

  • Study I shows no observed Q2 retrieval gain from the actor account: Luna scores 60/64 and 55/64 with tools alone and actor account, versus 64/64 and 63/64 with native inputs; Sonnet scores 64/64, 63/64, 64/64, and 64/64.The counts do not establish harm or equivalence; interpretation is constrained by Sonnet’s ceiling and 13 task clusters.Source: No observed retrieval gain from the… · p. 7, Table 4 · p. 7
  • In R10, Luna and Sonnet make 27 and 28 unsupported Q3 assertions, of which 18 and 23 match the complete reference; the account bundle increases input tokens by 18.1% without native inputs and 11.7% with them.A post-hoc intersection finds 16 shared reference-matching unsupported answers across seven tasks.Source: No observed retrieval gain from the… · p. 7

Bindings and the support for recorded citations

  • In mapping-required cases, Sonnet makes 26 unsupported Q3 assertions without bindings, 22 matching the complete reference; adding the table supports definite answers in 24/28 cases.Luna makes 21 unsupported assertions without bindings, 12 matching the reference, and has supported definite answers in 21/28 cases with the table.Source: Bindings and the support for recorded… · p. 7, Table 5 · p. 7
  • With original IDs, adding bindings changes complete-reference agreement from 43 to 49 for Luna and 46 to 49 for Sonnet across 64 cases, distinct from the supported-finding gains.Some previously correct guesses become supportable when the missing relation is supplied.Source: Bindings and the support for recorded… · pp. 7–8
  • Across the other 36 cases, grounded Q3 counts do not improve uniformly, and both readers continue to err even when complete tables are supplied.C10 has 15 nongrounded answers for each reader; binding preservation alone does not ensure a correct reading.Source: Bindings and the support for recorded… · p. 8

Renaming, controls, and shared errors

Opaque identifier renaming does not provide a consistent remedy for unsupported or changing answers. The paper recommends preserving execution-scoped bindings alongside the records needed to check findings.

Renaming, controls, and shared errors

  • Opaque renaming without a table reduces Luna’s unsupported Q3 assertions from 21 to 14 in mapping-required cases, while Sonnet remains at 26; every task-equal Q3 renaming-accuracy interval includes zero.Luna’s pooled grounded count rises from 38 to 42, but its task-equal change is −1.15 percentage points.Source: Renaming, controls, and shared errors · p. 8
  • Renaming changes Q3 answers in 25/64 Luna and 9/64 Sonnet cases without a table, but single responses per condition and changing token sequences prevent isolating an identifier heuristic.Q2 also changes despite unchanged literal evidence, cautioning against attributing every Q3 change to binding use.Source: Renaming, controls, and shared errors · p. 8
  • With the authentic user text withheld, justified Q1 answers require abstention, yet Luna makes 9/10/4/10 unsupported assertions across C00/C10/C01/C11 and Sonnet makes 0/1/0/0.A mapping to a user-request locator is not the request itself.Source: Renaming, controls, and shared errors · p. 8
  • Cross-reader Q3 exact-answer agreement is 41/44/28/41 cases across C00/C10/C01/C11, but shared reference-matching unsupported answers occur in 11 C00 and eight C01 cases, and none in table conditions.Agreement therefore still requires checking against supplied evidence.Source: Renaming, controls, and shared errors · pp. 8–9

Preservation and evidence export

  • Evidence export should retain native tool calls and results, execution order, authentic request and channel identity, the public actor account, and execution-scoped input-ID bindings.The experiment removes bindings from derived packets rather than measuring their loss in deployed systems; source authenticity and operational chain of custody require separate validation.Source: Preservation and evidence export · p. 9

Reporting the supported relation

The illustrative case separates evidence that a value appears in an earlier record from evidence that a saved citation refers to it. Assisted findings should preserve their support status and the evidence version used for checking.

Reporting the supported relation

  • In rcase-001, both packets establish that the recipient appears in the earlier bill result, but only the packet with the binding table establishes that the saved Basis refers to it.That finding concerns the recorded citation relationship, not why the actor selected the recipient.Source: Reporting the supported relation · p. 9, Table 8 · p. 9
  • Reports should distinguish an empty relation, an observed absent Basis field, and insufficient supplied evidence; none warrants extending a finding to unobserved actor behavior.An empty set reports the defined record query, while a missing field reports the saved account’s state.Source: Reporting the supported relation · p. 9

Validation of assisted findings

  • Accurate retrieval, complete-reference agreement, and cross-reader agreement can coexist with unsupported citation findings, so evaluation should report supported findings and justified abstention separately.A deterministic same-packet comparator answered or abstained correctly throughout these bounded questions, while readers still produced 12–16 nongrounded Q3 answers with complete tables.Source: Validation of assisted findings · p. 9
  • Keeping the finding’s supplied content and bindings with its support status makes later reviewers able to inspect whether a definite claim was justified by that packet.In rcase-001, the table supports Sonnet’s unchanged answer when the earlier packet could not.Source: Validation of assisted findings · p. 9
  • If a required binding was never recorded, the audit leaves the relation unresolved; the proposed reduced-trace use depends on preserving execution-scoped bindings during collection.An exporter can include retained bindings with the records used to verify a finding.Source: Validation of assisted findings · pp. 9–10

Limitations

The conclusion separates answer agreement from packet-supported citation relations and specifies preservation requirements; the Limitations bound the findings to the studied corpus and reader configurations.

Limitations

  • A saved citation records the actor’s stated reference, not semantic authorization, harmful intent, or internal causal reliance.Source: Limitations · p. 10
  • The audit covers 64 cases from 13 tasks in one simulated Banking environment, so its derived conditions do not estimate operational missing-binding prevalence.No investigator evaluation was conducted, so the results do not quantify casework benefits.Source: Limitations · p. 10
  • Retrospective analyses fixed inputs before each pass, reused cases, and compared two specific reader configurations; hosted-model nondeterminism limits attribution of individual output changes.Task-bootstrap intervals are exploratory, and Sonnet was selected after Luna’s results.Source: Limitations · p. 10

Conclusion

  • In C00, Sonnet retrieved every literal-location set but made 26 unsupported citation-source assertions in 28 mapping-required cases, 22 of which matched the complete reference.Explicit ID-to-record bindings improved grounded reconstruction for the measured readers; identifier renaming alone provided no consistent remedy.Source: Conclusion · p. 10
  • Preservation should retain record content, execution order, channel identity, and execution-scoped bindings, while keeping generated accounts distinguishable from referenced records.Reporting supported definite findings, observed empty or missing states, and unresolved relations separately prevents complete-reference agreement from being treated as packet support.Source: Conclusion · p. 10
Loading 2610.09581v1…