Source-linked AI summary
AgentTrails: Towards Trust and Reuse for Agentic Tasks
Eden Wu, Sonia Castelo, Yurong Liu, Cláudio T. Silva, Juliana Freire
TL;DR
Chronological agent logs obscure dependencies between actions and artifacts, limiting understanding, comparison, debugging, and reuse. AgentTrails reconstructs trace-derived provenance graphs and joins aligned activity structures across executions. Usage scenarios show hidden dependencies and workflow patterns beyond raw logs, while provenance benchmarks and scaling remain open challenges.
Problem
Chronological agent logs obscure dependencies between actions and artifacts, limiting developers' ability to understand, compare, debug, and reuse agent computations.
Source: §1 · p. 1Method
AgentTrails reconstructs provenance graphs from trace evidence and aligns activity structures across traces in a joined graph for downstream analysis.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Trace-derived provenance provides a foundation for agent debugging, workflow reuse, and skill extraction grounded in auditable structures.
Source: §5 · p. 4Provenance quality lacks ground-truth benchmarks, capsule similarity and weights need principled evaluation, and scalability to very long traces or large collections is unassessed.
Source: §5 · p. 4
Abstract
from arXiv · showhide
LLM-powered agents increasingly tackle complex tasks by invoking tools, querying databases, executing code, and manipulating intermediate artifacts. These agents follow trajectories that are typically stored as chronological logs, obscuring the underlying dataflow -- the dependencies between their actions and the artifacts they create and manipulate. This limits developers' ability to understand the agents' trails, compare executions, debug failures, and re-use the computations. We present AgentTrails, a prototype system for agent provenance and sensemaking. AgentTrails converts raw trajectories into structured provenance graphs, where tool calls are modeled as computational actions and inputs and outputs as data artifacts. The system supports the comparison of executions by placing multiple provenance graphs on a shared canvas and constructing a joined quotient graph that aligns recurring tools, artifacts, and dependency structures across trajectories. On top of this representation, AgentTrails supports pattern extraction, downstream analysis, and skill abstraction. We demonstrate AgentTrails on real-world agent trajectories, showing that it reveals hidden dependencies, aligns divergent executions, and surfaces recurring tool-use patterns beyond chronological logs.
1 INTRODUCTION
Agent trajectories are rich but unstructured chronological logs, making them difficult to understand and compare because they hide workflow dependencies. AgentTrails reconstructs provenance and aligns multiple traces to expose shared workflows and divergent branches for downstream sensemaking.
1 INTRODUCTION
- Repeated structures across successful executions may reveal reusable skills or workflows, and AgentTrails frames its contributions around provenance reconstruction and joined-graph alignment for sensemaking.Source: §1 · pp. 1–2
- Chronological logs hide dependencies between tool calls and artifacts, making workflow structure difficult to understand and trace comparison difficult when executions diverge.Traces may reuse earlier artifacts, combine outputs, or branch; adjacent calls may be unrelated.Source: §1 · p. 1
- AgentTrails reconstructs provenance graphs from trace evidence, representing tool calls as activities and inputs, outputs, and intermediate artifacts as data artifacts.A joined quotient graph aligns recurring tools, artifacts, and dependency structures while preserving trace-specific branches.Source: §1 · pp. 1–2
- An LLM-based analysis agent inspects reconstructed and joined graphs to summarize observed patterns and explain selected branches, while the core workflow remains grounded in trace-derived structures.Source: §1 · p. 2
2 RELATED WORKS
Prior provenance systems commonly assume explicitly defined or recoverable workflows, while agent trajectories arrive as stochastic chronological logs with implicit dependencies. AgentTrails reconstructs provenance post hoc and aligns executions in a joined graph, complementing trajectory reuse approaches that treat traces as flat sequences.
2 RELATED WORKS
- Scientific workflow provenance supports reproducibility and reuse, but commonly assumes workflows are explicitly defined or recoverable from controlled environments.Source: §2 · p. 2
- Agent trajectory tools support debugging and visualization, but prior efforts largely focus on single-run inspection or domain-specific settings.AgentTrails instead reconstructs provenance post hoc from raw logs without instrumentation and aligns multiple executions through a joined graph.Source: §2 · p. 2
- ExpeL, Reflexion, and Trace2Skill use prior trajectories for workflow induction, reflection, or skill extraction, while AgentTrails exposes dependency structure to identify reusable motifs and prune redundant calls.Source: §2 · p. 2
3 SYSTEM OVERVIEW
AgentTrails converts heterogeneous agent traces into provenance graphs and aligns activity capsules across runs to represent shared structures and trace-specific branches. It provides tool-coverage and graph views, with a copilot for lightweight interpretation of trace-derived structures.
3 SYSTEM OVERVIEW
- AgentTrails normalizes heterogeneous traces into action sequences while retaining evidence such as arguments, responses, artifact identifiers, paths, URLs, returned objects, and reused semantic values.It does not assume a fixed agent architecture or tool schema.Source: §3 · p. 2
- Its UpSet-style coverage matrix counts each tool's invocations per trace and supports filtering and selection, but does not infer dependencies.Rows encode traces, columns encode tools, and cells encode repeated calls; a side rail can show score, cost, latency, or token count.Source: §3 · p. 2
- Provenance graphs make producer-consumer dependencies explicit by representing tool calls as activities and inputs, outputs, artifacts, and returned values as entities.Exact references to recovered artifact keys capture high-precision reuse while temporal validity keeps the graph acyclic.Source: §3 · p. 2
- Semantic matches become dependency candidates rather than asserted facts, and constrained LLM edits are accepted only when they preserve valid references, temporal order, and the activity/entity schema.Exact edges, semantic candidates, and LLM-refined edits remain distinguishable for auditability.Source: §3 · p. 2
- For multi-trace comparison, activity capsules summarize tool identity, input/output structure, local graph context, and upstream lineage before clustering and remapping edges into a joined graph.Different tool orders, repeated calls, missing steps, and alternative branches make raw sequence or exact tool-name comparison insufficient.Source: §3 · p. 3
- Joined nodes and edges retain trace support and multiplicity, exposing recurring motifs and low-support branches; the views encode support through node size and edge width while preserving trace membership by color.Source: §3 · p. 3
- Joined provenance construction remains one design choice among many, and alternative abstractions and scalable candidate alignment are identified as future work.Source: §3 · p. 3
- The provenance copilot can inspect graph structures, retrieve tool inputs and outputs, expand joined nodes to trace members, and summarize visible patterns without defining provenance.Source: §3 · p. 3
4 USAGE SCENARIOS
Usage scenarios show AgentTrails recovering scientific dependencies hidden in a physics trace and supporting workflow inspection and comparison across Discovera runs.
Inspecting a SciAgentGym Physics Trace
- The SciAgentGym graph connects independent physical quantities to the hyperfine constant, branches into two energy-shift calculations, and grounds the final transition frequency in prior results.The graph exposes a fan-in/fan-out structure that is obscured by the chronological trace.Source: §4 · p. 3
- The reconstructed SciAgentGym graph covers the benchmark’s required scientific steps and reveals how upstream quantities feed the hyperfine constant and downstream frequency calculation.This scenario demonstrates workflow inspection and validation, not benchmarking the underlying model.Source: §4 · pp. 3–4
Discovera Gene-Set Exploration
- In Discovera, the top-ranked trace uses fewer tool calls than several alternatives, prompting provenance-level inspection.Runs are grouped by task and ranked by score in the trace-level overview.Source: §4 · p. 4, Figure 4 · p. 4
- The inspected Discovera trace filters and joins two Wang 2025 Cancer Cell datasets before ora_pipe enrichment, forming a coherent path from ingestion to analysis.The workflow applies FDR < 0.05 filtering and removes global differentially expressed genes before enrichment.Source: §4 · p. 4
- Across the joined Discovera view, the high-scoring run follows a direct, well-supported dataflow, while lower-scoring runs add branches and detours that are candidates for pruning or prompt fixes.Node support, edge support, and trace membership distinguish shared structure from trace-specific deviations.Source: §4 · p. 4
5 CONCLUSION
The conclusion presents AgentTrails as preliminary infrastructure for recovering and comparing provenance in agent traces. Usage scenarios demonstrate structure beyond raw logs, while provenance evaluation, capsule similarity, and scalability remain open.
5 CONCLUSION
- AgentTrails combines a provenance framing, evidence-based dependency reconstruction from chronological logs, and a joined quotient graph for comparing executions.The joined representation aligns traces to expose shared workflows, divergent branches, and low-support behaviors.Source: §5 · p. 4
- Provenance quality lacks ground-truth benchmarks, capsule similarity and its weights need principled evaluation, and scalability to very long traces or large collections remains unassessed.These open questions bound the current evidence for the system.Source: §5 · p. 4
- The authors frame AgentTrails as a foundation for future work on debugging, workflow reuse, and skill extraction grounded in auditable, trace-derived provenance.Source: §5 · p. 4