Source-linked AI summary
LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents
Daehong Kim, Haichao Miao, Shusen Liu
TL;DR
Reviewers must audit which actions, artifacts, checks, and evidence support an agent’s conclusions, but observability alone leaves those connections to reconstruct. LEDGER builds layered claim-to-evidence trace graphs over agent sessions, and two case studies show explicit artifact-lineage and code-grounding review paths.
Problem
Observability exposes agent events, but reviewers still must reconstruct which actions, artifacts, checks, and evidence support a particular conclusion.
Method
LEDGER reorganizes observed session records into layered semantic trace graphs linking claims to actions, artifacts, and validation checks through typed edges.
Results
Across data-analysis and coding case studies, traces supported artifact lineage from source data to findings and code grounding from repository inspection through patches and tests.
Takeaways & Limitations
Layered trace graphs make actions, artifacts, checks, and claims inspectable as connected evidence while preserving access to underlying source records.
Takeaways & Limitations
The graph above the Trace Record layer is not fully deterministic because inferred groupings, workflow nodes, and semantic edges can be incomplete, unstable, or wrong.
Abstract
from arXiv · showhide
Large language model (LLM) agents can now carry out long-horizon technical workflows involving complex tool use, code execution, file edits, and generated artifacts. As agents do more work faster, the productivity bottleneck shifts from producing outputs to auditing whether those outputs are correct and trustworthy. Agent observability systems make fine-grained execution events visible, but visibility alone still leaves reviewers to reconstruct which actions, artifacts, and validation steps matter for a particular conclusion. We introduce LEDGER - Layered Evidence and Decision Graphs for Execution Review, a tracing and review system that builds layered trace graphs over observed agent sessions. LEDGER preserves Trace Records while grouping them into Evidence Nodes and Workflow Nodes, representing artifacts as evidence anchors, and adding typed semantic edges that connect claims to supporting actions, artifacts, and checks. Through data-analysis and coding examples, we show how the resulting traces expose workflow decisions, artifact lineage, repair steps, validation coverage, and claim-support paths for evidence-centered audit.
1 Introduction
As LLM agents undertake longer technical workflows, auditing plausible outputs requires tracing the decisions, actions, artifacts, and checks behind final claims. LEDGER addresses this evidence-centered review problem by reorganizing observed sessions into layered claim-to-evidence graphs that connect conclusions with supporting work and validation.
- Motivation: Longer agent sessions combine planning, inspection, tool calls, execution, file edits, and generated artifacts, making final results depend on many intermediate decisions and actions.These artifacts include scripts, plots, tables, patches, and reports.
- Motivation: Auditing a plausible result requires identifying inspected inputs, explored choices, changed files, produced artifacts, executed checks, and evidence supporting the final claim.Productive use therefore depends on more than generation speed.
- Problem: Fine-grained observability exposes prompts, tool calls, responses, errors, intermediate outputs, and other execution records, but visibility alone does not show which events support a particular conclusion.Reviewers must otherwise reconstruct task structure and evidence from a flat record.
- LEDGER: LEDGER constructs layered trace graphs over observed sessions, reorganizing records around audit needs and connecting conclusions to contributing actions, artifacts, and checks.It builds on observability rather than replacing it.
- LEDGER: The sidecar tracer parses session data into Trace Records, groups them into Evidence and Workflow Nodes, represents inspectable objects as artifact nodes, and links claims, work, and validation with typed semantic edges.The review interface lets users move between graph-level audit paths, source records, artifacts, graph updates, and tracer-agent call records.
2 Related Work
Prior work captures agent execution, evaluates runs, and visualizes reasoning, but LEDGER focuses on the review layer that organizes observed actions, artifacts, and checks around claims. Its post-capture evidence graph differs from control-flow orchestration graphs and extends inspection to saved artifacts.
- Capture and review layers: Observability and provenance systems capture execution records and production histories, whereas LEDGER arranges those records around claims, artifacts, actions, and checks for review.The passage identifies observability and provenance as the capture layer and LEDGER’s focus as the review layer above it.
- Agent evaluation: Agent-evaluation methods assess programs, trajectories, safety behavior, or monitorability, motivating the use of intermediate actions as evidence.Examples include ScienceAgentBench, Graphectory, AgentAuditor, and monitorability work.
- Evidence relations versus control flow: Explanation-faithfulness work supports inspecting observed execution evidence, while LEDGER’s post-capture graph represents evidence relations rather than orchestration control flow.The audit record reaches observed actions, artifacts, and checks, unlike graphs whose edges define control flow before or during execution.
- Inspection scope: Visualization and explanatory-debugging systems make reasoning or generated work inspectable, but execution review must also cover changed files, patches, command outputs, tables, plots, and other saved artifacts.The passage contrasts reasoning or generation traces with the broader artifact coverage required for execution review.
3 Building the Layered Trace Graph
LEDGER constructs a layered trace graph from captured session records and transcripts, preserving provenance while organizing activity into inspectable evidence units and workflow phases. Artifact anchors and typed semantic edges connect actions, artifacts, checks, and claims so reviewers can trace conclusions back to underlying records.
- Trace Records: Trace Records preserve the session’s stable evidence substrate, including messages, tool calls and results, lifecycle boundaries, artifacts, timing, and transcript links.They are observations rather than interpretations of the agent’s reasoning or task structure.
- Trace Records: LEDGER retains raw transcript evidence and normalizes it into trace-record families while making parse errors and unfamiliar transcript rows explicit.The hook path stores unchanged transcript JSONL alongside trace.json and coverage records, supporting provenance-preserving review.
- Layered Graph: The graph has three layers: Trace Records preserve captured evidence, Evidence Nodes group records into inspectable work units, and Workflow Nodes organize those units into phases.This layered structure keeps long sessions readable while allowing reviewers to return to underlying records.
- Artifacts: Artifact nodes represent inspectable outputs such as command results, changed files, plots, and tables, enabling claims to connect to concrete evidence outside the message stream.The graph records which evidence produced an artifact, which later evidence used it, and whether a final claim is supported.
- Semantic Edges: Typed directed edges connect actions with artifacts at the Evidence Layer, relate workflow phases and claims, and link Workflow Nodes back to supporting Evidence Nodes.Relations include using files, producing output, checking changed files, and framing, informing, checking, or supporting another phase or claim.
4 LEDGER Interface
LEDGER’s local dashboard supports graph-guided workflow review by combining a two-level graph with node details, artifact views, and event logs. Reviewers can inspect workflow structure, source evidence, and how tracing records were constructed.
- Graph-guided workflow review: The two-level graph summarizes workflow phases and evidence relations, letting reviewers inspect concrete messages, tool calls, outputs, files, patches, and artifacts.Workflow Nodes summarize phases, while Evidence Nodes show the recorded work under those phases.
- Evidence inspection: Node details and artifact views connect graph elements to evidence such as terminal output, patches, files, plots, and result tables.Reviewers can follow supporting edges from a final claim to the recorded work.
- Trace-construction audit: Event and log views make graph construction reviewable by exposing Trace Records, graph updates, tracer events, model-call records, and trace errors.These views help distinguish an agent error from a tracing error when the graph appears incomplete.
5 Case Studies
The case studies show how LEDGER supports evidence-centered review across data-analysis and coding workflows by exposing phases, decisions, artifacts, checks, repairs, and claim-support paths. In both examples, reviewers can trace final outputs back through the recorded actions and validation evidence.
- Overview: LEDGER’s two case studies cover artifact lineage in data analysis and code grounding in feature development.The data-analysis example traces findings through tables and a plot to source data, while the coding example traces a code change through supporting records.
- Data-analysis case study: The data-analysis task required cleaning hourly air-quality data and characterizing typical daily pollutant patterns.The workflow included sensor readings, reference measurements, timestamps, weather variables, code, cleaning, execution results, and cost.
- Data-analysis case study: The data-analysis trace linked the cleaning and summary plan to file inspection, CSV previews, row-count checks, ending-row inspection, and environment checks before script generation.This exposed how analysis choices were grounded in the observed file format and made the workflow decisions inspectable.
- Data-analysis case study: A failed script execution caused by an unavailable optional dependency was linked to patches, a successful rerun, and validation of the report, tables, and plot.The output-checking phase connected the final claim to those artifacts and back to the recorded process that produced or checked them.
- Coding case study: The coding trace separated the initial implementation from a later guard patch, regression-test patch, and focused test run after checking existing missing-node behavior.Although the final diff combined the stages into one function, the trace preserved them as separate phases with separate evidence and recorded which check covered which behavior.
- Coding case study: Repository searches and file reads grounded the utility’s placement in the generic shortest-path module and linked the implementation patch to the ordered-intermediate-node requirement.The inspection covered shortest-path exports, the generic module, nearby tests, and documentation lists.
6 Discussion
The discussion presents LEDGER as a graph-based audit aid that connects claims to supporting actions, artifacts, checks, and source Trace Records. It also identifies inferred graph structure as a limitation and calls for stronger provenance, clearer relation types, and uncertainty-aware visualization.
- Discussion: LEDGER lets reviewers start from a claim, follow evidence edges to supporting actions and artifacts, and open the underlying Trace Records.It reorganizes agent sessions from message streams into graphs of evidence relations.
- Discussion: The case studies demonstrate artifact lineage for data analysis and code grounding from repository inspection through patches and tests.Both review paths use Trace Records together with Evidence Nodes, Workflow Nodes, artifact anchors, and semantic edges.
- Limitations: LEDGER’s graph layer is not fully deterministic because the tracer infers record groupings, Workflow Nodes, and semantic edges that may be incomplete, unstable, or wrong.The system therefore treats the graph as an audit aid rather than a source of truth.
- Future Work: Future work should replace model-inferred structure with deterministic or independently verifiable tracking through stronger instrumentation, structured tool records, and provenance backends.Suggested targets include file operations, patches, command outputs, test runs, generated artifacts, and data-level dependencies.
- Future Work: Future systems should distinguish temporal, dataflow, validation, and evidential relations while exposing edge provenance and encoding uncertainty or missing support.Improved visual encodings could separate deterministic from inferred structure and emphasize unsupported conclusions.
A Case Study Prompts
The case study prompt asks an agent to add a NetworkX utility for paths from a source to a target through ordered intermediate nodes. It requires reuse of existing shortest-path functions, weighted and unweighted support, boundary deduplication, and established error conventions.
- A.2 Case Study 2: Feature Addition Prompt: The requested utility returns a path from a source to a target while visiting specified intermediate nodes in order.The prompt frames the result as a path or walk through the required nodes.
- A.2 Case Study 2: Feature Addition Prompt: The implementation should build on existing NetworkX shortest-path functions and support both weighted and unweighted graphs.
- A.2 Case Study 2: Feature Addition Prompt: The utility should concatenate segment paths without repeating boundary nodes and follow NetworkX conventions for missing nodes or paths.
A.1 Case Study 1: Tabular Data Analysis Prompt
The case study defines a reproducible tabular analysis of an hourly air-quality dataset. It requires dataset cleaning, reporting basic structure and missingness, and characterizing daily pollutant patterns by hour.
- Data preparation: The analysis begins by inspecting and cleaning data/airqualityuci.csv, including timestamp parsing, numeric conversion, and handling the documented missing-value convention.These steps establish a reproducible preprocessing procedure.
- Dataset reporting: It reports the number of rows, date range, and missingness for each column.The requested outputs summarize dataset size, temporal coverage, and data completeness.
- Daily pollutant patterns: The analysis characterizes typical daily patterns in the main pollutant measurements by computing mean values for each hour of the day and identifying the peak hour.The passage specifies hourly aggregation as the basis for the daily-pattern analysis.
B Case Study Full Trace Graphs · B.1 Case Study 1 Trace Graph
The section presents a full trace graph for Case Study 1, illustrating the paper’s trace-graph representation of an agent workflow.
- B.1 Case Study 1 Trace Graph: Figure 7 presents the Case Study 1 Trace Graph.
- B.1 Case Study 1 Trace Graph: The figure serves as the section’s visual record of the Case Study 1 trace graph.
- B Case Study Full Trace Graphs: The case study is represented through a trace graph rather than a standalone output.
- B Case Study Full Trace Graphs: The section groups Case Study 1 within the paper’s broader full-trace-graph case studies.
- B.1 Case Study 1 Trace Graph: Figure 7 identifies the displayed graph as belonging specifically to Case Study 1.
- B.1 Case Study 1 Trace Graph: The figure provides the named trace-graph artifact for this case-study section.
B.2 Case Study 2 Trace Graph
Figure 8 presents the trace graph for Case Study 2.
- Figure 8 shows the Case Study 2 Trace Graph.