PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes

Mehil B Shah, Mohammad Mehdi Morovati, Mohammad Masudur Rahman, Foutse Khomh

arXiv:2603.06847v2cs.SE

TL;DR

Existing agentic-AI failure studies provide limited insight into how faults originate in system components and manifest as symptoms. This study derives component-grounded taxonomies, mines associations among faults, symptoms, and causes, and validates them with practitioners, identifying 34 fault types across four architectural dimensions.

  • Problem

    Prior studies largely analyze task outcomes or behavioral errors, leaving the component origins, observable symptoms, and propagation mechanisms of agentic-AI faults insufficiently characterized.

  • Method

    The study analyzes 385 faults sampled from 13,602 documented faults across 40 repositories, derives taxonomies using grounded theory, mines associations with Apriori, and surveys 145 practitioners.

  • Results

    The study identifies 34 fault types across four architectural dimensions, with high-confidence associations among fault types, symptoms, and root causes and strong practitioner alignment.

  • Takeaways & Limitations

    The taxonomy and association patterns provide an empirical basis for diagnosing faults and understanding how failures span components in agentic-AI systems.

  • Takeaways & Limitations

    Findings may not generalize fully beyond Python-based GitHub repositories to other programming ecosystems, industrial deployments, or proprietary systems.

Abstract

from arXiv · show

Agentic AI systems combine LLM-based reasoning, orchestration, tool invocation, and interaction with external environments. These systems introduce faults that are difficult to characterize using existing taxonomies. To address this gap, we present an empirical study of faults in agentic AI systems. We collected 13,602 issues and pull requests from 40 repositories and, using stratified sampling, selected 385 faults for analysis. Through grounded theory, we derived taxonomies of fault types, symptoms, and root causes. We then used Apriori-based association rule mining to identify relationships among faults, symptoms, and root causes, and validated the taxonomy through a developer study with 145 practitioners. Our analysis produced a taxonomy of 34 fault types, organized into four architectural dimensions. These faults manifested as failures in structured-output interpretation, tool calls, runtime execution, and exception handling, with root causes including data schema mismatches, dependency drift, state management complexity, and model interface instability. Furthermore, association rules showed recurring cross-component propagation, linking structured data, dependency, and state management faults to their symptoms and root causes. Practitioners considered the taxonomy representative of agentic AI failures and suggested refinements related to multi-agent coordination and observability. These findings provide an empirical basis for diagnosing faults and improving reliability in agentic AI systems.

1 Introduction

Agentic AI faults arise across orchestration, evolving state, tool and environment interactions, motivating a component-grounded taxonomy of fault types, symptoms, and root causes. The study derives this taxonomy from 385 sampled faults, analyzes associations among its dimensions, and validates practical relevance with developers.

  • Agentic systems can misinterpret tool outputs, enter infinite reasoning cycles, or retain incorrect assumptions about external state.
  • The study analyzes 13,602 documented faults from 40 systems, samples 385 faults, and uses grounded theory to derive taxonomies of fault types, symptoms, and root causes.
  • 34 fault types are organized into 14 categories and four architectural dimensions, alongside 12 symptom and 12 root-cause categories.
  • Data Schema Mismatches account for 28.0% and Dependency Drift for 21.9% of dominant root causes, reflecting contract violations between generated artifacts and external software constraints.
  • A developer study with 145 practitioners found strong practical alignment: mean relevance was 3.97/5, 74.9% of ratings were at least 4, and 83.8% reported coverage of encountered faults.
  • Apriori-based association mining identifies high-confidence relationships among fault types, symptoms, and root causes, while avoiding causal interpretation.

2 Methodology

The methodology combines repository screening, representative sampling, grounded-theory coding, and Apriori association analysis to characterize agentic AI faults and their relationships. Reliability checks and iterative annotation support the resulting taxonomy, while high-confidence rules guide symptom–cause investigation without establishing causality.

  • Repository Selection: Repositories were screened for active, widely used Python agent projects, reducing 82 candidates to 47 before manual annotation produced 40 repositories.
  • Sampling: The corpus contains 13,602 items, from which stratified random sampling selected 385 faults at a 95% confidence level and 5% margin of error.
  • Grounded-Theory Analysis: Grounded-theory analysis manually coded faults, symptoms, and root causes through iterative batch reviews, disagreement resolution, and evolving coding guidelines.
  • Grounded-Theory Analysis: The resulting taxonomy captures both low-level technical faults and broader architectural issues, including reasoning breakdowns, integration errors, and operating-environment vulnerabilities.
  • Apriori Analysis: Apriori analysis encodes each fault using Fault Type, Symptom, and Root Cause attributes to identify recurring associations among them.
  • Apriori Analysis: High-confidence association rules summarize symptom–cause relationships that can guide root-cause investigation but do not establish causality.

3 Study Results

The study results characterize faults in agentic AI systems through a taxonomy grounded in 385 analysed faults. The taxonomy organizes these faults by architectural dimensions and reports their frequencies.

  • Figure 2 reports the frequency of faults using numbers in parentheses.
  • 385 real-world faults were analysed to characterize failures in agentic AI systems.
  • The study presents a taxonomy of faults organized according to agentic AI system architecture.

I. Agent Reasoning & Control (86 faults).

Agent reasoning and control faults arise from interactions between LLM behavior, orchestration logic, iterative execution, and state updates. The identified fault categories include model-interface problems and lifecycle defects affecting scheduling, delegation, transitions, and termination.

  • Agent Reasoning & Control: LLM Interaction faults occur at the interface between agent frameworks and LLM providers.Examples include incorrect model or provider configuration, incompatible API usage, token-accounting errors, and context-window handling problems.
  • Agent Reasoning & Control: Model-interface faults can result from incorrect capability assumptions, outdated API schemas, or incorrect tokenizers.
  • Agent Reasoning & Control: Agent Lifecycle and State faults affect scheduling, task delegation, state transitions, and termination logic across iterative control loops.
  • Agent Reasoning & Control: Reported lifecycle examples include prevented coworker execution, redundant builds and tool calls, inconsistent assistant behaviour, and failure to stop execution.

II. Context & Memory (67 faults).

Context and memory faults arise when agents store, retrieve, parse, or interpret heterogeneous information incorrectly. These defects can corrupt contextual artifacts, create temporal inconsistencies, and disrupt downstream processing.

  • Context & Memory: Memory Management faults affect serialization, deserialization, timestamp handling, and persistent storage of contextual artifacts.They can prevent reliable preservation of context across executions or sessions.
  • Context & Memory: Timestamp mishandling introduced temporal inconsistencies in stored interaction history, while oversized screenshot paths prevented state artifacts from being written.
  • Context & Memory: Input Interpretation faults arise when heterogeneous agent inputs are incorrectly parsed, validated, transformed, or interpreted.The category includes incorrect assumptions about input structures or types and flawed parsing logic.
  • Context & Memory: Parsing, return-value, and encoding defects caused runtime crashes, failed tabular extraction, and failed downloads involving Cyrillic filenames.

III. Tooling, Integration & Actuation (62 faults).

Tooling, integration, and actuation faults occur at boundaries between agents and external tools, services, resources, and execution infrastructure. The categories cover invocation contracts, access, resources, coordination, and observability.

  • Tooling, Integration & Actuation: Tool Invocation faults involve defective tools or incorrect calls, including API-contract violations, unsupported operations, and parameter mismatches.
  • Tooling, Integration & Actuation: Incorrectly structured or unsupported tool arguments caused invocation failures and program crashes.
  • Tooling, Integration & Actuation: External Access faults involve invalid endpoints, misconfigured connections, and poorly managed credentials or permissions.These problems can prevent communication with external systems or execution of authorized operations.
  • Tooling, Integration & Actuation: Resource Interaction faults involve invalid paths, filesystem assumptions, or storage and database configuration errors.They can prevent tools from reading, writing, or managing required artifacts.
  • Tooling, Integration & Actuation: System Coordination faults affect concurrent, streaming, or asynchronous execution and can produce race conditions or incoherent output.
  • Tooling, Integration & Actuation: Execution Monitoring faults arise from missing or incorrect logging, tracing, or metric instrumentation that reduces failure visibility.

IV. System Infrastructure & Reliability (155 faults).

System infrastructure and reliability faults span dependencies, platforms, failure handling, interfaces, documentation, and execution symptoms across the agent pipeline.

  • Infrastructure faults: 72 dependency and environment-management faults involved incompatible versions, missing packages, or inconsistent installations.These faults produced setup failures, resolution conflicts, broken functionality after upgrades, and inconsistent behavior across installations.
  • Infrastructure faults: 15 platform and integration compatibility faults arose when runtime assumptions about operating systems, architectures, filesystems, or utilities were violated.
  • Reliability faults: 47 failure-handling and implementation-robustness faults involved suppressed exceptions, incorrect error propagation, or inadequate recovery mechanisms.These mechanisms are especially important because agents interact with multiple external services and execute long-running tasks.
  • Supporting defects: User-interface defects and documentation issues affected how execution state was represented or understood, with 13 and 8 faults respectively.The corresponding problems included incomplete visualizations, misleading interfaces, incorrect examples, and incomplete behavior descriptions.
  • Symptoms: Observable symptoms included 80 data and validation errors, 68 runtime execution errors, 45 unhandled exceptions, and 32 build errors.Data and validation failures involve malformed or incompatible intermediate artifacts, while runtime and exception failures interrupt workflows or terminate execution.
  • Symptoms: Other symptoms included 28 dependency-installation errors, 24 agent-behavior anomalies, 22 LLM interaction errors, and 17 network connectivity errors.These symptoms range from failed environment setup and rejected model requests to coordination breakdowns and unreachable external services.

3.2 RQ2: What statistically significant associations do exist among faults, symptoms, and root causes in agentic AI systems?

Apriori mining of 385 fault transactions identified structured associations linking fault types to symptoms and symptoms to root causes. The rules indicate that failures can propagate across components before becoming observable at runtime.

  • Mining approach: Apriori association-rule mining encoded each of 385 labelled reports as a transaction and retained rules with confidence ≥0.30.The transactions captured fault types, symptoms, and root-cause categories.
  • Fault type → symptom: UI Rendering Defect → User Interface Errors had confidence 0.7692, linking rendering defects to observed interface errors.The rule suggests that many UI errors are attributable to rendering defects rather than upstream components.
  • Fault type → symptom: Structured Data Error → Data and Validation Errors had confidence 0.6087, linking generated-output mismatches to validation-stage failures.The paper connects these mismatches to structured inputs passed from LLM outputs to downstream tools.
  • Propagation: Runtime execution acts as an aggregation point where faults from different components become observable after propagating through multiple steps.This reflects tightly coupled orchestration and motivates step-wise tracing across the execution pipeline.
  • Symptom → root cause: Dependency Installation Errors → Dependency Drift had confidence 0.9286, associating installation failures with inconsistent dependency versions or environment configuration.The paper highlights dependency specifications and version pinning as relevant controls for environment setup.
  • Propagation: The resulting rules show propagation shaped by orchestration, autonomy, and environmental dependence, with early faults surfacing later as shared runtime errors or exceptions.The summary of RQ2 links implicit assumptions about data formats, interface contracts, and state to recurring failures.

3.3 RQ3: To what extent do the derived taxonomies of bugs, symptoms, and root causes align with the practical experiences of agentic system developers?

A survey of 145 developers found broad practical relevance for the taxonomy across all four architectural dimensions and experience levels. Feedback supported its coverage while identifying refinements for semantic failures, coordination, human workflows, observability, and resource exhaustion.

  • Quantitative validation: 145 participants rated 34 fault types at a mean relevance of 3.97 out of 5, with 74.9% of responses rated 4 or higher.The median was 4.0, the standard deviation was 0.87, and 4.4% of responses were rated 2 or lower.
  • Quantitative validation: Mean relevance ratings across the four dimensions ranged from 3.93 to 4.04, with Tooling, Integration & Actuation highest at 4.04.Agent Reasoning & Control scored 4.00, Context & Memory 3.94, and System Infrastructure & Reliability 3.93.
  • Quantitative validation: 33 of 34 fault types, or 97.1%, received relevance ratings of at least 4 from a majority of participants.
  • Experience analysis: Relevance was consistent across experience levels, with mean ratings of 3.91, 4.00, and 4.13 for increasing experience groups.Figure 5 reports non-significant differences across groups at p>0.05.
  • Qualitative feedback: 122 of 145 participants, or 83.8%, said the taxonomy broadly reflected their experience with agentic systems.Participants particularly valued separating LLM reasoning, tool integration, and infrastructure failures.
  • Refinements: Participants suggested refinements for syntactic versus semantic failures, multi-agent coordination, human approval workflows, observability, and behavioral resource exhaustion.Examples included logically incorrect but structurally valid outputs, inter-agent coordination breakdowns, stalled approval workflows, missing cross-agent traces, and runaway API usage.

4 Discussion

The discussion treats agentic failures as pipeline-level phenomena arising across probabilistic reasoning and deterministic components. It emphasizes schema validation, propagation-aware debugging, stable integrations, observability, and early detection as reliability priorities.

  • Implications: Agentic failures arise from interactions among LLM reasoning, tools, external services, runtime environments, and structured data, and rarely remain localized.The discussion frames failures as propagating across reasoning pipelines, tool invocation, and runtime environments.
  • Design implications: Interfaces between LLM outputs and downstream components should use clearly defined schemas, validation layers, and fallback mechanisms.The recommendation follows the reported prevalence of data-validation errors and schema mismatches.
  • Debugging implications: Developers should validate outputs at multiple pipeline stages, especially after LLM generation and before tool invocation.The paper connects this practice to nondeterministic behavior and propagation patterns that can make failures difficult to reproduce.
  • Infrastructure implications: Tooling and infrastructure providers should prioritize API stability and backward compatibility because dependency drift and integration changes can disrupt downstream pipelines.Existing observability platforms record prompts, responses, tool calls, costs, latency, and multi-agent interactions.
  • Observability implications: Agentic debugging support should correlate traces across agents, connect tool failures to upstream reasoning or state transitions, and replay failure-inducing paths.These capabilities would help distinguish local failures from propagation through delegation and shared context.
  • Reliability implications: Reliability engineering should treat failures as multi-stage processes and detect common signatures before they propagate.Examples include schema-violating outputs and state updates that conflict with predefined invariants.
  • Research implications: Researchers can use the taxonomy to annotate failure datasets, compare agent frameworks, and report benchmark results with consistent labels.

5 Threats to Validity

The study addresses internal, construct, and external validity threats through coding safeguards, resolved-issue filtering, practitioner validation, and stratified sampling. Its findings remain bounded by potential annotation and filtering bias, GitHub-based data, and a focus on Python repositories.

  • Internal validity: Manual coding and GPT-4.1 filtering may introduce bias, mitigated through consensus discussions, ground-truth comparison, and representative inspection.GPT-4.1 achieved 83% accuracy and 97% recall against human-labelled data.
  • Internal validity: Association rules were retained at confidence ≥30% and interpreted as high-confidence co-occurrences rather than causal relationships.
  • Construct validity: GitHub issue reports may be incomplete or speculative, so the study retained only issues showing evidence of resolution.Accepted evidence included closure, a linked merged pull request, or discussion confirming the fault was fixed.
  • Construct validity: The taxonomy’s completeness was supported by iterative coding of 385 faults from 40 agent systems and a practitioner survey showing strong alignment with developer experience.The survey also identified areas for refinement.
  • External validity: Generalizability is limited by the focus on Python repositories and GitHub systems, despite stratified sampling across four repository types.Repositories were selected with more than 1,000 stars and at least 30 issues to favor mature, actively maintained projects.

6 Related Work

Related work categorizes or diagnoses agentic failures from platform, interaction, task, or behavioral perspectives. This study complements those approaches with component-grounded characterization and propagation analysis across agentic system architecture.

  • Existing taxonomies: Existing failure taxonomies commonly adopt platform-centric, interaction-centric, or task-centric perspectives.Platform-centric work links orchestration-infrastructure constraints to system-wide crashes.
  • Failure diagnosis: Voting-based consensus mechanisms address responsibility attribution among agents in non-deterministic multi-agent tasks.
  • Failure diagnosis: Self-reflective feedback loops use execution traces to refine prompts and reasoning strategies, targeting robustness and debugging during execution.These approaches largely treat agents as black-box components and emphasize behavioral attribution or prompt refinement.
  • Positioning of this work: This study instead maps 385 real-world faults from 40 frameworks, libraries, and other systems to agent functionality, memory, planning, tool interfaces, and runtime environments.

7 Conclusion and Future Work

The study develops an architecture-grounded account of agentic AI faults and shows that failures often cross component boundaries. It concludes with engineering implications for interfaces, state management, dependency resilience, and future expansion of the evidence base.

  • Conclusion: 13,602 closed issues and merged pull requests from 40 repositories yielded 385 manually analyzed representative faults.The analysis used grounded theory to derive taxonomies mapped to key agentic-system components.
  • Conclusion: Failures frequently cross component boundaries, propagate through stateful control loops, and are amplified by external-tool integration and evolving dependencies.
  • Implications: Reliable agentic systems need robust model-program interfaces, systematic internal-state validation, and safeguards against dependency and external-service failures.Suggested safeguards include pinned versions, runtime API-contract checks, adapters, and fallback behaviors.
  • Future work: Future work will broaden the dataset across programming languages, deployment environments, and industrial systems while strengthening model-code contracts and pipeline observability.
  • Conclusion: The study provides empirical foundations for improving reliability, diagnosability, and engineering practices in agentic AI systems.
Loading 2603.06847v2…