PaperlayerView exact revision on arXiv

Source-linked AI summary

The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution

Chen Qian, Peng Wang, Dongrui Liu, Junyao Yang, Dadi Guo, Ling Tang, Jilin Mei, Qihan Ren, Shuai Shao, Yong Liu, Jie Fu, Jing Shao, Xia Hu

arXiv:2601.15075v2cs.AIcs.CL

TL;DR

Failure-focused attribution localizes explicit errors in unsuccessful trajectories, but it does not explain questionable decisions that lack an explicit error signal. This paper uses hierarchical attribution to identify influential interaction components and sentence evidence, and reports that it reliably pinpoints drivers of agent behavior across diverse scenarios. The authors describe this as a step toward safer, more accountable deployment.

  • Problem

    Failure attribution focuses on explicit errors in unsuccessful trajectories, leaving the rationale behind questionable decisions without explicit errors unexplained.

    Source: §1 · p. 1

  • Method

    The framework uses temporal likelihood dynamics to localize influential interaction components, then perturbation-based analysis to identify sentence-level evidence.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across diverse agentic scenarios, qualitative analysis reports accurate identification of pivotal historical events and sentence evidence, with comparisons demonstrating robustness and effectiveness.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors present granular attribution as a step toward more interpretable, verifiable, and trustworthy AI systems and safer, more accountable agent deployment.

    Source: §6 · p. 8, Impact Statements · p. 9

    Further research is necessary to fully automate interpretation of attribution signals in large-scale, real-world applications.

    Source: Impact Statements · p. 9

Abstract

from arXiv · show

Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering. As these systems become more autonomous and are deployed at scale, understanding why an agent takes a particular action becomes increasingly important for accountability and governance. However, existing research predominantly focuses on \textit{failure attribution} to localize explicit errors in unsuccessful trajectories, which is insufficient for explaining \textbf{the reason behind agent behaviors}. To bridge this gap, we propose a novel framework for \textbf{general agentic attribution}, designed to identify the internal factors driving agent actions regardless of the task outcome. Our framework operates hierarchically to manage the complexity of agent interactions. Specifically, at the \textit{component level}, we employ temporal likelihood dynamics to identify critical interaction steps; then at the \textit{sentence level}, we refine this localization using perturbation-based analysis to isolate the specific textual evidence. We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias. Experimental results demonstrate that the proposed framework reliably pinpoints pivotal historical events and sentences behind the agent behavior, offering a critical step toward safer and more accountable agentic systems. Codes are available at https://github.com/AI45Lab/AgentDoG.

1. Introduction

Failure attribution targets explicit errors in unsuccessful trajectories, but questionable decisions can occur without an explicit failure signal. The paper proposes hierarchical attribution to identify the historical components and sentences driving an agent’s action, and reports that it pinpoints such evidence across diverse scenarios.

1. Introduction

  • LLM-based agents are applied across dynamic tasks, including software engineering, web navigation, and scientific discovery.Their capabilities combine long-term memory, external tools, and iterative reasoning loops.Source: §1 · p. 1
  • Failure-focused methods address explicit errors in unsuccessful trajectories, leaving questionable decisions with acceptable outcomes outside their scope.The refund example illustrates a decision with no explicit error signal, driven by memory of past positive feedback.Source: §1 · p. 1, Figure 1 · p. 2
  • Understanding why agents act matters for transparency, accountability, and governance as agentic systems enter real-world settings.The paper frames attribution around whether decisions rely on appropriate semantic evidence and an expected reasoning pathway.Source: §1 · pp. 1–2
  • Temporal likelihood dynamics localize influential interaction steps, followed by perturbation-based sentence attribution to isolate textual evidence.The sentence-level module can also accommodate other attribution techniques, including gradient-based saliency or attention-based analysis.Source: §1 · p. 2
  • Qualitative analysis reports accurate identification of pivotal historical events and sentence evidence across standard tool use and reliability-risk scenarios.The scenarios include memory-induced bias and tool-conditioned hallucinations; comparisons of sentence-level methods also demonstrate robustness and effectiveness.Source: §1 · p. 2

2. Problem Formulation

The problem formulation represents an agent’s interaction history as ordered components and defines attribution at both component and sentence levels. The resulting objectives assign scores to historical components and, within influential components, to sentences contributing to the final action.

2. Problem Formulation

  • The formulation structures agent–environment interaction to support analysis at both component and sentence levels.It frames the task around a completed trajectory, an agentic action, and the agent policy.Source: §2 · p. 2, §2.2 · p. 3

2.1. Structured Agent Interaction

  • At each discrete time step, the agent receives an environment observation and produces an action according to its policy.The policy is conditioned on the interaction history available at that step.Source: §2.1 · pp. 2–3
  • The interaction history contains the initial input followed by observations and actions through the current time step.The initial input can be a system prompt or user request; observations can include tool results, retrieved documents, or memories.Source: §2.1 · p. 3
  • The full trajectory is transformed into an ordered component sequence, with each component corresponding to one observation or action.Components are contiguous token sequences and preserve temporal order.Source: §2.1 · p. 3
  • This componentized trajectory represents agent behavior as structured parts that jointly determine its actions and enable unified attribution.Each component is associated with a functional type.Source: §2.1 · p. 3

2.2. Attribution Objectives

  • The objectives quantify how individual trajectory components contribute to generating the final action under the agent policy.The formulation distinguishes component-level attribution from sentence-level attribution.Source: §2.2 · p. 3
  • Component-level attribution assigns each component a scalar score reflecting its influence on generation of the final action.The score is defined under the given agent policy.Source: §2.2 · p. 3
  • Sentence-level attribution refines analysis within high-scoring components by assigning scores to their individual sentences.The scores indicate each sentence’s relative contribution to the agent’s output.Source: §2.2 · p. 3

3. Agentic Attribution Framework

The framework attributes agent actions hierarchically: it first localizes influential interaction components, then identifies supporting sentences within them. Component scores track how action likelihood changes as history unfolds, while sentence scores use perturbations to refine the attribution.

3. Agentic Attribution Framework

  • The framework first localizes influential interaction components, then identifies precise textual evidence within them.This coarse-to-fine design targets both where in the trajectory influence arises and which sentences support the action.Source: §3 · p. 3

3.1. Component Attribution by Temporal Likelihood Dynamics

  • Incremental replay tracks realized-action likelihood as ordered components arrive; sharp positive gains flag possible drivers while preserving temporal order.The approach contrasts with flat-context attribution by respecting the order of tool use, memory updates, and reasoning steps.Source: §3.1 · pp. 3–4
  • For safety-critical use, reward-model scoring of action appropriateness is suggested as an alternative, but its exploration is left to future work.The described component-level instantiation uses temporal gain derived from generative log-likelihood.Source: §3.1 · p. 4

3.2. Sentence-Level Attribution inside High-Impact Components

  • Within a selected high-impact component, sentence ablation measures probability drop and probability hold, and the combined scores rank fine-grained evidence.Probability drop identifies sentences whose removal reduces action likelihood; probability hold measures whether a sentence alone provides substantial support.Source: §3.2 · pp. 4–5
  • Sentence scores are not tied to perturbation: gradient-, attention-, and linear-model-based attribution methods are also considered within the framework.The component-level score may likewise use metrics other than generative log-likelihood, including embedding similarity or reward-model scores.Source: §3.1 · p. 4, §3.2 · p. 5

4. Experiments

The experiments test agentic attribution on memory- and tool-driven scenarios through representative case studies and a quantitative comparison of sentence-level methods.

4.1. Scenarios and Cases Design

  • Eight Llama-3.1-70B-Instruct trajectories cover benign to implicitly undesirable memory- and tool-driven behavior, testing attribution of decision drivers rather than task correctness.The scenarios target internal factors behind decisions, unlike benchmarks focused on task correctness or explicit runtime-error localization.Source: §4.1 · p. 5

4.2. Analyzing Attribution in Representative Scenarios

  • Llama-3.1-70B-Instruct is used to analyze four representative cases in depth, with analyses of the remaining cases included in Appendix B.The cases serve as the testbed for qualitative analysis.Source: §4.2 · p. 5

4.2.1. MEMORY-DRIVEN INTERACTION.

  • Memory attribution links the delivery change to updated high-risk and heavy-rain fields, but links the chair recommendation to a prior laptop success generalized across contexts.The delivery fields align with the prescribed decision logic; the laptop outcome is reused as a strong signal despite the different purchase context.Source: §4.2.1 · p. 6

4.2.2. TOOL-DRIVEN INTERACTION.

  • Tool attribution isolates malicious email instructions as the driver of leaked private data, while document-question answers trace to the request and relevant retrieved lines.The retrieved document lines concern the updated safe combination and meeting location.Source: §4.2.2 · p. 7

4.3. Comparative Evaluation of Attribution Methods

  • Across test cases, Prob. Drop&Hold achieves the best overall performance among evaluated methods, while other sentence-level methods also perform strongly.Table 1 compares average Hit@1, Hit@3, and Hit@5 scores.Source: Table 1 · p. 8, §4.3 · p. 8

5. Related Work

Related work covers LLM-agent memory and tool use, LLM input attribution methods, and approaches to detecting failures in agent trajectories.

LLM-based agents

  • Agent memory stores, retrieves, and updates information across interactions, while external-tool paradigms integrate tool calls into reasoning.The passage distinguishes short-term, long-term, and structured memory.Source: §5 · p. 8

LLM input attribution

  • Gradient-based salience metrics are reported only for successfully completed cases because long cases caused out-of-memory issues.Source: §5 · p. 8
  • LLM input attribution methods include counterfactual ablation, internal-signal proxies such as gradients or embedding similarity, and Shapley-based approaches.The passage also describes surrogate models, hierarchical search, and approximations across text-generation and retrieval-augmented settings.Source: §5 · p. 8

Agent failure detection

  • Agent failure-detection work localizes explicit errors in failed trajectories, whereas this framework asks why an agent makes a particular decision.The cited approaches include LLM-as-judge strategies and a benchmark reporting difficulty with precise step-level fault localization.Source: §5 · p. 8

6. Conclusion

The paper proposes hierarchical attribution to identify internal drivers of agent actions, with empirical evaluation across benign and potentially risky behaviors.

6. Conclusion

  • The framework combines temporal-likelihood analysis of historical components with perturbation-based sentence scoring to localize rationale behind agent actions.The paper presents this as a step toward reliable and accountable autonomous-agent deployment.Source: §6 · p. 8

Impact Statements

The impact statement frames the framework as supporting transparency and accountability, while noting that interpretation automation remains future work.

Impact Statements

  • The framework is intended to enhance transparency and accountability in autonomous agent systems.Source: Impact Statements · p. 9
  • The authors emphasize understanding agent behavior before deployment and aim to contribute to interpretable, verifiable, and trustworthy AI systems.Source: Impact Statements · p. 9
  • Further research is needed to fully automate interpretation of attribution signals in large-scale, real-world applications.Source: Impact Statements · p. 9

A. More Details about the Agentic Case Design

The authors assemble eight agent-interaction trajectories for attribution, combining seven custom-designed cases with one complex GAIA retrieval task. The suite covers memory- and tool-driven scenarios from benign operations to implicitly undesirable actions.

A. More Details about the Agentic Case Design

B. Attribution Analysis for Remaining Cases

The remaining case analyses show attribution identifying influential events and text in both problematic and reliable behavior. They trace questionable decisions to early tool signals or user-provided details, and reliable advice or calculations to relevant memory and file content.

Case Study 5: Enforcing Safety Constraints via Long-term Memory in Fitness Scenario

  • The prior injury acknowledgment and retrieved memory drive Coach Max’s safety-first advice, including low-impact bodyweight exercises despite the user’s partial recovery.The component analysis highlights both the earlier response and the later memory retrieval.Source: B. Attribution Analysis for Remaining… · pp. 12–13

Case Study 6: Spurious Tool Signal Leading to a Wrong Answer

  • The early search result containing a “BELARUSIAN” label is the strongest driver of the wrong answer, despite later reasoning acknowledging uncertainty; the attribution localizes a potential vulnerability in weighting early tool signals.The runner-up attribution also highlights uncertainty and insufficient evidence in the agent’s reasoning.Source: B. Attribution Analysis for Remaining… · pp. 13–14

Case Study 7: Hallucinated Content Generation from User Prompt

  • For the hallucinated manual, attribution identifies the detailed user prompt—not tool outputs—as the dominant driver, exposing that the claimed “alternative sources” were fabricated.High sentence-level scores fall on the report title and descriptions of its findings.Source: B. Attribution Analysis for Remaining… · p. 14

Case Study 8: Cost Calculation with Tool-Assisted File Reading

  • For the correct procurement total, attribution concentrates on the file-reading tool output and its structured item rows, which provide the needed numerical data.The discussion describes the calculation as grounded in those rows.Source: B. Attribution Analysis for Remaining… · p. 14
Loading 2601.15075v2…