Source-linked AI summary
From Features to Actions: Explainability in Traditional and Agentic AI Systems
Sindhuja Chaduvula, Jessee Ho, Kina Kim, Aravind Narayanan, Ahmed Y. Radwan, Mahshid Alinoori, Muskan Garg, Dhanesh Ramachandram, Shaina Raza
TL;DR
The paper addresses how explainability methods for single predictions translate to agentic systems whose success depends on multi-step trajectories. It compares attribution-based explanations with trace-grounded rubric diagnostics and finds that trace-based methods better diagnose execution failures, motivating trajectory-level explainability. The paper also identifies scope and faithfulness limits of its post-hoc, rubric-based approach.
Problem
Agentic systems require explanations of trajectories involving states, actions, and observations, but it remains unclear how static prediction explanations translate to these settings.
Method
The paper formally distinguishes static and agentic explainability and compares attribution-based explanations with trace-grounded behavioral rubrics across static classification and agentic benchmarks.
Results
Trace-based behavioral rubrics provide more actionable localization of agentic failures, whereas attribution methods do not reliably identify violated constraints in specific failed executions.
Takeaways & Limitations
Evaluating autonomous agent behaviour requires trajectory-level explanation artifacts grounded in execution context rather than relying only on feature-level attribution.
Takeaways & Limitations
The post-hoc rubric abstraction is coarse, primarily correlational, dependent on subjective LLM judging, and assumes complete execution traces.
Abstract
from arXiv · showhide
Over the last decade, Explainable AI has primarily focused on interpreting individual model predictions, producing post-hoc explanations that relate inputs to outputs under a fixed decision structure. Recent advances in large language models (LLMs) have enabled agentic AI systems whose behaviour unfolds over multi-step trajectories. In these settings, success and failure are determined by sequences of decisions rather than a single output. It remains unclear how explanation approaches designed for static predictions translate to agentic settings where behaviour emerges over time. In this work, we bridge this gap by comparing attribution-based explanations with trace-based diagnostics across both settings. Our results show that while attribution methods achieve stable feature rankings in static settings (Spearman \r{ho} = 0.86), they cannot be applied reliably to diagnose execution-level failures in agentic trajectories. In contrast, trace-grounded rubric evaluation for agentic settings consistently localizes behaviour breakdowns and reveals that state tracking inconsistency is 2.7x more prevalent in failed runs and reduces success probability by 49%. These findings motivate a shift towards trajectory-level explainability for evaluating and diagnosing autonomous AI behaviour in agentic systems. Code: https://github.com/VectorInstitute/unified-xai-evaluation-framework Project page: https://vectorinstitute.github.io/unified-xai-evaluation-framework
1 Introduction
Traditional XAI explains fixed input–output predictions, whereas agentic systems require explanations of multi-step trajectories. This paper formalizes that distinction and finds that trace-based diagnostics better localize agentic failures than attribution methods.
- Traditional post-hoc explanations infer model behaviour for fixed input–output mappings without modifying the underlying system.
- Agentic LLM systems challenge static explainability because behaviour unfolds through observations, reasoning steps, tool invocations, and state updates.
- Trajectory-level questions concern tool choices, strategy abandonment, incorrect state propagation, and failure recovery rather than a single prediction.
- The paper distinguishes static and agentic explainability, proposes a cross-paradigm taxonomy, and compares attribution-based with trace-based explanations.
- Static attribution methods produce stable aggregate feature or behavioural rankings, but do not reliably identify the violated constraint in a specific failed agent execution.
- Trace-based rubric evaluations grounded in execution logs provide per-run explanations by identifying failures such as tool misuse, state inconsistency, and failed recovery.
2 Related Work
Related work spans static-model explanations, LLM reasoning traces, and agent evaluation frameworks, but these strands have not been empirically unified. The paper positions its contribution as a cross-paradigm test of explanation quality for diagnosing agent behaviour.
- Traditional XAI includes post-hoc, model-agnostic and model-specific methods, with attribution techniques explaining influential input features or regions.
- Concept-based methods shift explanations toward human-interpretable abstractions, while mechanistic interpretability seeks internal computations and causal circuits.
- Traditional XAI does not explain how decisions unfold over time, how states change, or how agents adapt.
- LLM-oriented approaches expose rationales, action logs, tool use, retrieved evidence, or provenance, but rationales and attention are not guaranteed to be faithful explanations.
- Agent evaluation frameworks assess multi-step planning, tool use, and execution rather than only final outputs.
- Prior work had not empirically evaluated explanation quality across static and agentic paradigms, motivating this paper’s unified comparison.
3 Explainability Across Static and Agentic AI Paradigms
Static explainability targets single predictions, whereas agentic explainability must account for trajectories of states, actions, observations, and tool interactions. The framework compares attribution and trace-based approaches using shared criteria and trace-grounded behavioural evaluation.
- Paradigm distinction: Static explanations target a fixed input–output mapping, while agentic explanations target full trajectories whose states, actions, and observations determine success or failure.
- Paradigm distinction: Agentic explanation scope expands from single decisions to tool calls, state updates, retrieved evidence, and environmental feedback.
- Mapping XAI methods: Attributions in agentic systems require action sets, conditioning state variables, and composition into trajectory-level accounts; global summaries can miss where failures arise.
- Mapping XAI methods: Reasoning traces are self-reported and become more reliable when aligned with interaction logs that link intent to actions, tool outputs, and state updates.
- Evaluation: The framework evaluates scope, grounding, reliability or faithfulness, and auditability across static and agentic explanation settings.
- Evaluation: Trace-based rubric evaluation uses complete execution traces to label behavioural violations without access to task outcomes or ground truth, avoiding outcome leakage.
- MEP: MEP packages explanation artifacts with linked evidence and verification signals, supporting auditing and replay rather than standalone narratives.
4 Experimental Setup
The experiments evaluate explainability in one static text-classification setting and two agentic tool-use benchmarks under a unified MEP-based framework. Static attribution stability is compared with trace-derived behavioural signals associated with agent success and failure.
- Evaluation design: The study evaluates a static prediction task and agentic tasks using shared MEP-based criteria, while reporting setting-specific metrics.
- Static setting: The static evaluation uses the Online Job Postings dataset for binary IT versus non-IT text classification.
- Agentic setting: Agentic evaluation uses TAU-bench Airline for API-mediated customer-service tasks and AssistantBench for web-based multi-step assistance.
- Static setting: The static baselines are TF–IDF with Logistic Regression and a Text CNN.
- Trace evaluation: Docent applies fixed GPT-5 rubric judgments to agent traces, while the study reports rubric signals and their relationship to task outcomes.
- Trace collection: HAL-Harness collects action, tool-call, and observation trajectories, which serve as the primary artifacts for trajectory-level analysis.
5 Results and Analysis
The evaluation contrasts stable attribution-based explanations for static predictions with trace-based diagnostics for agentic trajectories. Trace-level analysis identifies distinct failure modes and localizes how behavioural constraints break down across runs.
- Experimental setup: The study calibrates explanation reliability on static classification before comparing attribution-based and trace-based explanations on TAU-bench Airline and AssistantBench.The bridging experiment also projects trajectories into rubric features for a shared comparison.
- Static explainability: 0.8577 Spearman ρ is the explanation stability for TF–IDF + Logistic Regression, compared with 0.6127 for the Text CNN.The static task uses binary IT versus non-IT classification on the Online Job Postings dataset.
- Static explainability: Static post-hoc methods explain final predictions but provide limited visibility into intermediate reasoning, state changes, and multi-step decision dynamics.This motivates trace-based evaluation for failures involving state updates, tool choices, and long-horizon trajectories.
- Agentic explainability: 2.7× is the State Tracking Consistency failure ratio in TAU-bench Airline, while AssistantBench Tool Choice Accuracy appears only in unsuccessful runs with Ratio = ∞.The benchmarks achieve 56.0% and 17.39% accuracy respectively, and their failure modes differ qualitatively.
- Agentic explainability: RR = 0.51 makes State Tracking Consistency the clearest TAU-bench Airline failure predictor, with success dropping by 36 percentage points when violated.AssistantBench Tool Choice Accuracy and Plan Adherence violations correspond to zero success, while Tool Correctness correlates positively with success at RR = 1.24.
- Bridging paradigms: Trace-based rubric evaluation localizes violated constraints in specific reasoning steps, tool invocations, and intermediate states, whereas SHAP over rubric features provides aggregate importance rankings rather than run-specific diagnoses.Attribution becomes useful as a complementary summary only after trajectories are compressed into behaviourally grounded rubric features.
6 Discussion
The discussion finds a clear divide between static attribution and trajectory-level diagnosis: trace-grounded rubrics localize agentic failures more effectively, while important scope and methodological limitations remain.
- Main findings: Trace-grounded rubric analysis produces per-run explanations that identify violated constraints such as tool misuse, state inconsistency, and failed recovery.These accounts support auditing, debugging, and reliability assessment across agentic benchmarks.
- Main findings: State inconsistency is the strongest failure signal in TAU-bench Airline, reflecting divergence between the agent’s evolving plan or memory and environment state.The resulting failures arise from degraded trajectory integrity rather than a single wrong output.
- Main findings: Attribution methods capture aggregate behavioural correlates but cannot reliably localize violated constraints in specific failed executions.Tool correctness may coincide with successful runs, indicating that some violations are minor or recoverable rather than fundamental breakdowns.
- Limitations: The evaluation studies a small set of tool-using LLM agents on TAU-bench Airline and AssistantBench using HAL-Harness.These benchmarks do not cover embodied agents, multi-agent coordination, online learning, or persistent long-term memory, limiting generalization.
- Limitations: Post-hoc rubric explanations are necessarily coarse, primarily correlational, dependent on complete traces, and subject to LLM-judge subjectivity.Docent also has limited support for automated counterfactual interventions, causal validation, and integration with internal model representations.
7 Conclusion
The conclusion formalizes the distinction between traditional and agentic explainability and introduces a trajectory-aware framework for evaluating complex agent behaviour. Experiments find attribution insufficient for diagnosing agentic failures, while trace-based rubrics improve visibility into failures affecting task success.
- The work introduces the Minimal Explanation Packet as a structured framework for capturing reasoning behaviour, tool interactions, execution traces, and verification signals.
- Conventional attribution methods are insufficient for diagnosing failures in agentic workflows, whereas trace-based behavioural rubrics reveal planning failures, execution inconsistencies, and reasoning deviations.
- The findings support evolving agentic explainability from static feature attribution toward trajectory-aware behavioural verification.
- Open challenges include scalable human-grounded evaluation, automated-judge robustness, and explanation management for extremely long trajectories.
Appendix
The appendix illustrates local and global attribution for static classification and summarizes the shared evaluation representation, experimental settings, and metrics used across static and agentic explainability evaluations.
- Static attribution: LIME shows per-prediction feature importance for a single job posting, with orange tokens favoring IT and blue tokens favoring non-IT.Domain-specific terms carry the strongest local signal.
- Static attribution: SHAP summarizes corpus-level feature effects, with software pushing toward IT, accounting toward non-IT, and development near zero.
- Static attribution: Figure 4 contrasts local LIME attribution with global SHAP attribution for the static IT classifier.
- Evaluation design: Table 8 compares traditional attribution-based and trace-based agentic explainability under a shared rubric-derived representation.
- Evaluation design: Table 9 summarizes experimental settings, including hardware, frameworks, benchmarks, models, training configurations, and explanation parameters.The listed elements support reproducibility across static and agentic evaluations.
- Evaluation design: Table 10 lists evaluation metrics for static and agentic settings, including explanation stability and custom rubric signals operationalized with Docent.