Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows

1 Introduction

• LLM fragility and complex orchestration make low-code agentic workflows prone to failure in practice.Failures are harder to repair because traces entangle natural-language prompts, agent communications, tool calls, and dynamic control; fixes may require prompt or workflow changes, not just code edits.Source: §1 · p. 2
• Existing taxonomies describe failure symptoms, while prior localization and repair approaches do not explain underlying mechanisms or offer actionable root-cause guidance.A surface symptom such as repeated steps does not identify why the failure occurred or how to repair it.Source: §1 · p. 2
• AgentFail contains 307 failure trajectories from two agentic workflow platforms, with annotations covering failure location, root cause, and repair strategy.Each instance also includes the user query, workflow configuration, and execution failure logs.Source: §1 · p. 2
• The study analyzes failure patterns and repair difficulty, evaluates automated repair, and derives guidance for failure attribution, repair, and workflow design.Its analyses examine failure causes and agent nodes, with repair evaluated by success rate and cost.Source: §1 · p. 3

2 Background

• Platform workflows connect functional nodes for reasoning, control, and external interactions; start and termination nodes mark entry and exit rather than execution failures.The described node types include LLM and agent, knowledge, logic and control, code and template, and tool and integration nodes.Source: §2 · p. 3
• Executions that interrupt are failures; concluded executions are judged against ground truth when available or by majority vote among independent LLM evaluators otherwise.Ground-truth checks include test-based verification or exact reference matching; LLM judging is used for tasks lacking deterministic ground truth.Source: §2 · p. 3
• A node action is decisive when replacing its wrong action with a correct one, while keeping prior steps unchanged, changes failure to success; the earliest decisive action is the root.The identified location is then used to analyze underlying root causes.Source: §2 · p. 3

3 The AgentFail Dataset

• AgentFail contains 307 failure logs from ten workflows across Dify and Coze, with queries, traces, workflow configurations, and annotations for failure location, cause, and repair strategy.The annotations also explain why each failure occurred.Source: §3 · p. 3

3.1 System selection

• The selected workflows cover software development, information insight, planning, and question answering, alongside serial, parallel, branching, looping, and hybrid structures.The selection includes platform templates and user-defined systems to cover typical task and orchestration patterns.Source: §3.1 · p. 4

3.2 Failure log collection

• Failure logs combine reproduced community reports with controlled runs over public or hand-crafted datasets, yielding 307 cases.Controlled runs used 1,387 test data in total; for large-scale test sources, the dataset information caption says a random subset of 100 samples forms the test set.Source: Table 1 · p. 3, §3.2 · p. 4

3.3 Failure Root Location & Cause Annotation

• Three expert annotators independently identified decisive errors and root causes, resolved uncertain cases through discussion, and cross-checked labels until discrepancies were settled.The study uses Grounded Theory to identify root-cause patterns from empirical data.Source: §3.3 · p. 4

4.1.1 Failure Manifestation.

• LLM and agent nodes account for over half of failure root causes; logic and control nodes rank second, knowledge nodes third, and code and template nodes comprise very few cases.The authors associate these patterns with agent-level reasoning and planning, orchestration, and knowledge retrieval failures, respectively.Source: §4.1.1 · p. 4
• 32% of failures have root nodes distinct from symptom nodes, and over 10% propagate more than 40% of workflow length; LLM and agent roots differ in about 40% of cases, versus 45% for logic and control.The remaining distance analyses, including root distance from start and termination, are reported on the website; the authors characterize the findings as evidence of non-local propagation.Source: §4.1.1 · pp. 4–5, Figure 2 · p. 4

4.1.2 Automatic Failure Attribution Performance.

• Attribution first narrows suspicious nodes by examining traces and workflow configuration, then re-rolls out candidate corrections to locate the earliest node whose correction resolves failure.The candidate output is selectively replaced with an oracle-generated correction while the rest of execution remains unchanged.Source: §4.1.2 · p. 5
• 65.8% attribution accuracy with re-rollout verification exceeds 46.3% without it, supporting counterfactual root-location detection over one-shot attribution.The authors report that cross-node dependencies and long-range propagation can mislead attribution based on local evidence, noisy logs, or surface symptoms.Source: §4.1.2 · p. 5

4.2.1 Taxonomy & Distribution.

• The taxonomy maps system abstraction levels to concrete root causes and corresponding repair strategies.It distinguishes agent-level, structure-level, and platform-level failures.Source: §4.2.1 · p. 5
• Agent-level failures dominate, especially knowledge and reasoning limitations (F1.4) and poor prompt design (F1.5).Missing input validation (F2.1) and unreasonable node dependency (F2.2) are the most common structure-level causes; platform-level causes are least common.Source: §4.2.1 · p. 5

4.2.2 Automatic Root Cause Identification Performance.

• Providing the taxonomy alone raises root-cause identification accuracy from 9.6% to 27.4%, while location alone yields smaller gains.The section presents identification as a semantic reasoning task: location indicates where a failure is observed, not why it occurs.Source: §4.2.2 · p. 5
• 45.6% accuracy is achieved when taxonomy and failure location are both provided, versus 9.6% when neither is available.The section characterizes taxonomy and location as complementary inputs to root-cause identification.Source: §4.2.2 · pp. 5–6, Table 2 · p. 6

4.3.1 Repair Strategy Taxonomy.

• The repair taxonomy links response-format errors to format verification and reasoning limitations to model upgrades or external augmentation.Figure 5 illustrates these two failure modes and corresponding repairs.Source: §4.3.1 · p. 6
• A four-expert repair team works in stages using failure trajectories, workflow configurations, and available failure-location and root-cause information.The experimental settings vary whether location and root-cause guidance are used.Source: §4.3.2 · pp. 6–7

4.3.2 Automatic Repair Performance.

• Structure-level causes involving control logic, node dependencies, or task decomposition have lower repair success and require more iterations than agent-level causes.Missing input verification (F2.1) is an exception, often repaired by adding input validation.Source: §4.3.2 · p. 7
• 66.8% repair success and a 2.8% new error rate are achieved when both failure location and root-cause information are provided.New errors are measured by rerunning repaired workflows on test inputs that previously passed.Source: Table 3 · p. 7, §4.3.2 · p. 7
• Code and template nodes are more consistently repairable than LLM & Agent and tool & integration nodes.Formatting (F1.2) and encoding (F1.6) are relatively insensitive to guidance, while poor prompt design (F1.5) is harder to repair without root-cause information and repair rates drop as the failure location is further removed.Source: §4.3.2 · p. 7

5 Discussion

• Unrepaired cases stem from repair-agent capability limits, narrow cognitive boundaries, or platform-mechanism restrictions.The authors recommend challenging assumptions with broader or opposing test scenarios and using memory or template-based strategies to reuse repair experience.Source: §5 · p. 7
• Clear role specification and modular prompts address planning errors and response misalignment, while explicit input and output validation helps prevent cascading errors.The latter applies to response-format errors (F1.2) and missing input validation (F2.1).Source: §5 · p. 7
• Secondary validation agents or alternative tool paths can address problems locally before propagation, but may add overhead; progressive workflow design is also recommended.The authors recommend starting with simple serial or parallel flows before introducing more complex patterns.Source: §5 · p. 8

6 Related Work

• The studied low-code workflows are a distinct subclass of agentic systems, with visual, template-driven configurations that bring workflow-design, debugging, and repair challenges.Prior examples include software-engineering agents such as SWE-agent [32] and systems for GAIA tasks such as Magentic-One [8].Source: §6 · p. 8
• Prior failure research addresses localization, diagnosis, repair, optimization, and taxonomies of failure patterns.Examples include Who&When [34], Aegis [24], Maestro [27], and the MAST taxonomy [1].Source: §6 · p. 8
• The paper identifies a gap in work focused mainly on where failures occur or how they manifest, and emphasizes root causes linked to systematic repair.It characterizes many existing failure types as surface phenomena that do not directly provide actionable repair guidance.Source: §6 · p. 8

• AgentFail contains 307 annotated failure traces, enabling fine-grained analysis of where failures originate, how they propagate, and how to repair them.Source: §7 · p. 8
• The study introduces a repair-oriented root cause taxonomy linking failure manifestations to actionable repair strategies.Source: §7 · p. 8
• The analysis examines failure occurrence patterns, underlying causes, and repair difficulty across different root causes and workflow nodes.Source: §7 · p. 8
• The paper offers practical workflow-design guidelines emphasizing modular prompt design, explicit validation, comprehensive checks, and progressive design.Source: §7 · p. 8