Source-linked AI summary
Explaining AI Agents Through Execution Traces
Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli, Claudia Di Carlo, Matteo Silvestri, Gabriele Tolomei
TL;DR
Traditional XAI methods provide limited process-level explanations for autonomous agents operating through multi-step interactions. This paper turns execution traces into auditable structured reports and trace-grounded narratives; across benchmarks and architectures, the reports and narratives are judged more faithful than direct LLM-generated explanations.
Problem
Traditional XAI approaches designed for static, single-step models do not adequately explain autonomous agents’ dynamic interactions, trajectories, and tool use.
Source: §1 · p. 1Method
The framework transforms logged agent execution traces into a structured, auditable report and a concise natural-language explanation grounded in observed behavior.
Source: §1 · p. 2Results
Across benchmarks and architectures, structured reports were more faithful than direct LLM-generated reports, and report-grounded narratives received higher faithfulness, exhaustiveness, and overall ratings than direct trace-to-text explanations.
Source: §1 · p. 2, §6 · p. 11Takeaways & Limitations
The structured report provides an auditable intermediate layer for identifying mismatches between an agent’s available knowledge and observed behavior.
Source: §1 · p. 2Evaluation focuses on tool-mediated QA tasks and a relatively small human-evaluated subset, and reconstruction is limited to behavior and evidence recorded in the trace.
Source: §6 · p. 11
Abstract
from arXiv · showhide
AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human oversight. This creates a pressing need for reliable and auditable explanations of what an agent did and why. However, traditional Explainable AI (XAI) methods fall short of providing the process-level transparency required for such interactive, multi-step systems, motivating a paradigm shift toward approaches specifically designed for AI Agents. To address this gap, we present a post-hoc XAI framework that transforms a lengthy agent's execution trace into a structured report and a faithful natural-language explanation explicitly grounded in its observable behavior. Because it relies solely on execution traces, the framework applies across different agent architectures, environments, and tasks. Human and automated evaluations across multiple benchmarks and architectures show that our framework produces high-quality, trace-faithful explanations while reliably identifying unsupported claims, unjustified actions, and evidence gaps, outperforming naive LLM-generated explanations.
1 Introduction
The paper proposes a post-hoc, execution-trace-grounded framework for user-facing explanations of AI agents. It combines an auditable structured report with a natural-language narrative, and evaluates explanation quality across agent architectures and benchmarks.
1 Introduction
- Existing XAI approaches designed for static, single-step models inadequately explain agents’ dynamic, multi-step interactions and trajectories.The paper also identifies error cascades, responsibility gaps, and multi-step error propagation as transparency challenges.Source: §1 · p. 1
- The framework transforms execution traces into human-readable explanations for end users, addressing a gap in approaches primarily aimed at debugging, monitoring, and failure analysis.It operates on logged interaction traces and is intended to apply across agent architectures, tool ecosystems, and task domains.Source: §1 · p. 2
- It produces a step-by-step structured report and a concise natural-language narrative, with the report serving as an auditable bridge from trace to explanation.The report supports identifying mismatches between the agent’s available knowledge and observed behavior.Source: §1 · p. 2
- The paper’s taxonomy characterizes agent behavior by how available knowledge relates to information acquired through environmental interaction.It distinguishes aligned and mismatched knowledge states to help identify and interpret behavior in execution traces.Source: §1 · p. 2
- The framework’s reports are substantially more faithful to execution traces than directly prompted LLM reports, while report-grounded narratives score higher on faithfulness, exhaustiveness, and overall ratings than direct trace-to-text explanations.The evaluation spans two agentic benchmarks, three architectures, two baselines, human evaluation, and LLM-as-a-judge.Source: §1 · p. 2
2 Related Work
Prior trace-based research primarily targets developer-facing debugging, evaluation, and failure diagnosis, rather than explanations of agent decisions for end users. The paper positions its framework as a way to use traces for user-understandable explanations and behavioral auditing.
2 Related Work
- The related work describes prior trace-based methods as developer-facing tools for debugging, evaluation, and failure diagnosis, rather than user-facing explanations of how agents reached decisions.Examples include detecting trace-visible failures, diagnosing execution issues, checking procedural compliance, and localizing root causes.Source: §2 · p. 2
- The paper identifies a gap in frameworks that make Agentic AI decision processes transparent to end users.It contrasts its aim with prior work focused on why an agent failed rather than how it reached a decision.Source: §2 · p. 2
- The proposed framework uses execution traces to support user-understandable explanations and structured inspection of agent behavior.The related work connects this emphasis to arguments for explicit execution provenance in responsibility attribution.Source: §2 · p. 2
3 Trace-Grounded XAI Methodology
The methodology reconstructs an agent’s execution trace as an auditable account of its behavior and evidence, then renders that account as a concise natural-language explanation.
3.1 Main Components
- The framework assumes goals can be decomposed into independently verifiable requirements and that execution logs contain enough information to reconstruct the agent’s behavioral trajectory.The trace represents actions, observations, and other interactions that may alter the information available to the agent.Source: §3.1 · p. 3
- Behavioral commitment is inferred from the next relevant action, while internal and external evidence are tracked separately; positive statuses require trace support, and labels diagnose commitment–evidence discrepancies.The evidence channels distinguish agent-surfaced information from environment-returned observations; unsupported statuses remain open.Source: §3.1 · pp. 3–4
- The structured report records each step’s action, observations, knowledge–action state, transition, and grounding report; a constrained narrator turns this audited history into per-step explanations and an episode summary.The report is the inspectable input to the explanation.Source: §3.1 · pp. 3–4
3.2 Operational Pipeline
- The pipeline normalizes heterogeneous logs, decomposes goals into requirements, extracts pointer-backed updates, validates them deterministically, reconciles states, then constrains narration to audited report content.Requirements can be generated upfront, incrementally, or through a hybrid strategy.Source: §3.2 · pp. 4–5
4 Experiments
The experiments evaluate trace-grounded reports and explanations across agent traces, comparing the pipeline with direct LLM-generated artifacts. Evaluation combines human references and LLM-as-a-judge assessment of report fidelity and narrative quality.
4.1 Experimental Setup
- The corpus covers GAIA executions from TRAIL, ReAct, and plan-and-act agents, plus HotpotQA traces from ReAct and plan-and-act, totaling 548 executions.The independently implemented agents were run on common GAIA questions, while HotpotQA added a complementary multi-hop setting.Source: §4.1 · p. 6
- Matched comparisons vary artifact sources: Our–Our uses trace-grounded reports, LLM–Our uses LLM-generated reports with the same narrator, and LLM-only generates narratives directly from traces.The configurations use the same underlying LLM where comparisons are available, controlling for model capacity.Source: §4.1 · pp. 6–7
- The proposed pipeline had no generation failures; failures occurred primarily in single-shot LLM configurations and were more frequent on longer, more complex traces.Failed generations were excluded from artifact-quality evaluation but retained in the reliability analysis.Source: Table 2 · p. 6, §4.1 · p. 7
4.2 Evaluation
- Human references were independently reconstructed from 32 stratified traces across 12 queries and three agent systems; narrative evaluation used human assessment and an LLM judge.Structured-report fidelity was assessed against the human-annotated trace reference.Source: §4.2 · p. 7
- Narratives were assessed for faithfulness, exhaustiveness, linguistic quality, and overall quality, while report-to-trace fidelity and narrative-to-report faithfulness were evaluated separately.Human assessment directly compared narratives with the trace; narrative-to-report assessment applied where a structured report was available.Source: §4.2 · p. 8
- Comparisons were paired by query, but bootstrap intervals on the 12-query human subset are indicative; architecture-invariance analysis was limited to that subset.The study leaves LLM-as-a-judge assessment of the remaining architectures for future work.Source: §4.2 · pp. 9–10
5 Results
The full pipeline outperformed direct LLM explanation, while structured reports scored far higher than LLM-generated reports against human annotations. The results also support architecture-consistent effects and using the structured representation as a proxy for trace-level faithfulness, with limits to automated judge agreement.
Structured-representation fidelity (H0)
- Across 63 reports, structured reports achieved NFI = 0.816 (SD = 0.219), versus 0.061 (SD = 0.124) for the LLM baseline.Under strict matching, 74% of LLM reports scored zero, compared with none of the structured reports; results support using the structured report as the narrative-faithfulness reference.Source: §5 · p. 10
LLM-as-judge validation
- Judge-human agreement varied by configuration, and the judge’s positive calibration led it to miss most human-negative M2 cases.The judge recovered 9/25 human-negative M2 cases; pooled κw was 0.63 for cross-level faithfulness, 0.52 for linguistic quality, and 0.41 for overall quality.Source: §5 · p. 10
End-to-End Superiority (H1)
- On the LLM-as-judge corpus, the full pipeline improved all evaluated criteria over LLM-only, with the largest effect on M3 (∆= 1.18).M2 improved by ∆= 0.07 and M4 by ∆= 0.62; the M3 and M4 effects had p < 0.0001.Source: Table 6 · p. 11, §5 · p. 11
Value of the Structured Representation (H2)
- On the LLM-as-judge corpus, adding the structured representation improved M2 (∆= 0.49) and M4 (∆= 0.82), but not M3 (∆≈0, p = 0.6418).The M3 95% CI was [−0.03, 0.03].Source: Table 6 · p. 11, §5 · p. 11
Cross-level faithfulness comparison
- Without access to the ground-truth trace, the judge identified 14/16 narratives that humans judged unfaithful to the trace, supporting the structured report as a practical proxy for trace-level assessment.The proxy is useful when direct trace-based evaluation is costly.Source: §5 · pp. 10–11
6 Conclusion
The paper presents a post-hoc framework that turns agent execution traces into auditable structured reports and trace-grounded natural-language explanations. Evaluations show stronger faithfulness and quality than direct trace-to-text explanations, while the study’s task and logging scope constrain what it establishes.
Conclusion
- The framework transforms heterogeneous execution traces into auditable structured reports that separate behavioral commitments from available evidence, then produces trace-grounded natural-language explanations.Across three agent architectures and two QA benchmarks, reports were more faithful to human references than direct LLM generation, and report-based narratives scored higher in faithfulness, exhaustiveness, and overall quality than direct trace-to-text explanations.Source: §6 · p. 11
- The structured representation can serve as a practical proxy for trace-level faithfulness when direct trace inspection is costly.Source: §6 · p. 11
Limitations
- The evaluation focuses on tool-mediated QA tasks and a relatively small human-evaluated subset; reconstruction is limited to behavior and evidence recorded in the trace.Incomplete or insufficient logging limits what can be reconstructed.Source: §6 · p. 11
A Detailed Human-Evaluation Results
Human evaluations favor the full pipeline over direct LLM explanation and show gains from structured reports on content quality. Effects retain their direction across agent architectures, while ratings also reveal that faithfulness and content contribute strongly to overall quality.
End-to-end superiority (H1)
- Across four criteria and 26 paired traces, Our-Our beat LLM-only in 80 of 104 comparisons, tied in 23, and lost once.The full pipeline significantly outperformed direct prompting on all four criteria, with pBH = 0.0019 throughout; mean paired differences were ∆= 1.35 for overall quality, 1.11 for trace-level faithfulness, 1.04 for linguistic quality, and 0.85 for exhaustiveness.Source: A Detailed Human-Evaluation Results · p. 12, Table 8 · p. 14
Value of the Structured Representation (H2)
- Our-Our significantly outperformed LLM-Our on exhaustiveness (∆= 1.03), overall quality (∆= 1.00), source-report faithfulness (∆= 0.55), and trace-level faithfulness (∆= 0.52).Linguistic quality did not significantly differ (∆= 0.39); the configurations share the same narrator. The structuring-stage advantage is chiefly in content quality: exhaustiveness was positive in 48% versus 94%, and overall quality in 55% versus 97%.Source: A Detailed Human-Evaluation Results · p. 13, Table 8 · p. 14
Invariance Across Agent Architectures (H3)
- Effects were direction-consistent across ReAct, Plan-and-act, and TRAIL; under H2, M3 was approximately zero across architectures, as expected under a fixed narrator.Subgroups contained only 8–12 queries, insufficient for a reliable condition-by-architecture interaction test, so the analysis reports directions and magnitudes without p-values.Source: A Detailed Human-Evaluation Results · p. 13
End-to-end superiority (H1)
- LLM-only narratives were rated positive for exhaustiveness in 73% of cases, but only 8% reached the highest category; 46% received outright Disagree for trace-level faithfulness.Overall quality was positive for 31% of LLM-only narratives versus 97% for Our-Our.Source: A Detailed Human-Evaluation Results · pp. 12–13, Figure 2 · p. 14
Relations among criteria
- Overall quality correlated strongly with trace-level faithfulness (ρ = 0.85), exhaustiveness (ρ = 0.84), and delivery (ρ = 0.81); faithfulness to the structured report correlated weakly with narrative criteria (ρ = 0.23–0.40), while NFI showed moderate associations (ρ = 0.33–0.48).Source: A Detailed Human-Evaluation Results · p. 13