Source-linked AI summary
Flow-of-Action: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysis
Changhua Pei, Zexin Wang, Fengrui Liu, Zeyan Li, Yang Liu, Xiao He, Rong Kang, Tieying Zhang, Jianjun Chen, Jianhui Li, Gaogang Xie, Dan Pei
TL;DR
Frequent microservice incidents and complex RCA evidence motivate methods that can guide LLM agents beyond hallucination-prone, difficult action selection. Flow-of-Action combines SOP-guided orchestration, action sets, and auxiliary agents in a ReAct-based multi-agent system. The paper reports improved localization accuracy and identifies a 20-step process limit as a practical constraint.
Problem
Frequent incidents and variable observations make RCA difficult, while LLM agents can hallucinate and struggle to select actions among many possibilities.
Method
Flow-of-Action combines SOP flow, action sets, and multiple specialized agents within a ReAct-based multi-agent system.
Results
Flow-of-Action improves fault-localization accuracy over ReAct, and ablation results underscore the crucial role of SOP knowledge.
Takeaways & Limitations
SOP guidance, action-set design, and auxiliary-agent collaboration jointly refine the RCA process within the paper’s evaluated setting.
Takeaways & Limitations
The RCA process is limited to 20 steps to prevent unbounded loops that could make it unending.
Abstract
from arXiv · showhide
In the realm of microservices architecture, the occurrence of frequent incidents necessitates the employment of Root Cause Analysis (RCA) for swift issue resolution. It is common that a serious incident can take several domain experts hours to identify the root cause. Consequently, a contemporary trend involves harnessing Large Language Models (LLMs) as automated agents for RCA. Though the recent ReAct framework aligns well with the Site Reliability Engineers (SREs) for its thought-action-observation paradigm, its hallucinations often lead to irrelevant actions and directly affect subsequent results. Additionally, the complex and variable clues of the incident can overwhelm the model one step further. To confront these challenges, we propose Flow-of-Action, a pioneering Standard Operation Procedure (SOP) enhanced LLM-based multi-agent system. By explicitly summarizing the diagnosis steps of SREs, SOP imposes constraints on LLMs at crucial junctures, guiding the RCA process towards the correct trajectory. To facilitate the rational and effective utilization of SOPs, we design an SOP-centric framework called SOP flow. SOP flow contains a series of tools, including one for finding relevant SOPs for incidents, another for automatically generating SOPs for incidents without relevant ones, and a tool for converting SOPs into code. This significantly alleviates the hallucination issues of ReAct in RCA tasks. We also design multiple auxiliary agents to assist the main agent by removing useless noise, narrowing the search space, and informing the main agent whether the RCA procedure can stop. Compared to the ReAct method's 35.50% accuracy, our Flow-of-Action method achieves 64.01%, meeting the accuracy requirements for RCA in real-world systems.
1 Introduction
Frequent incidents and complex, variable evidence make RCA difficult, while LLM agents can introduce hallucinations and poor action choices. Flow-of-Action addresses these challenges with SOP-guided orchestration and collaborating agents.
- Microservices generate recurrent incidents that complicate reliability assurance and require RCA for swift resolution.
- Traditional deep-learning RCA methods adapt poorly to new scenarios and provide only the root cause rather than the complete diagnosis process.
- LLM-based RCA is vulnerable to hallucinations, inaccurate parameter extraction, and context-action discrepancies across multimodal data and API tools.
- Large tool sets and multiple plausible actions make action selection difficult even when observations are identical.
- Flow-of-Action integrates SOPs and a multi-agent design to constrain orchestration, reduce hallucinations, and support root-cause identification.Its auxiliary agents include JudgeAgent for assessing whether the root cause has been found and ObAgent for extracting fault types and key information.
- 35% to 64%: Flow-of-Action increased localization accuracy compared with ReAct in a fault-injection simulation of a real-world e-commerce system.
2 Flow-of-Action
Flow-of-Action is a multi-agent system built on ReAct with three design components: SOP flow, action set, and multi-agent system.
- Flow-of-Action is a multi-agent system built upon ReAct for root-cause analysis.
- Its three key design components are the SOP flow, the action set, and the multi-agent system.
2.1 Knowledge Base of Agents
The knowledge base combines SOP knowledge with historical incident knowledge to guide RCA and retrieve relevant diagnostic information.
- The knowledge base integrates SOP knowledge with historical incident knowledge for RCA retrieval.
- SOPs constrain LLMs at crucial junctures and are represented as self-contained units containing a name and steps.The SOP name is translated into a vector for subsequent retrieval.
- Historical incidents are characterized by manifestation and type, with similarity assessed between the current observation and historical manifestations.
2.2 Tools of Agents
Flow-of-Action uses tools to collect and analyze multimodal data, operate an SOP-centered workflow, and query system-specific operational state.
- Agent tools fall into multimodal data collection and analysis, SOP-flow tools, and other tools.
- The monitoring system aggregates metrics, traces, and logs, while preprocessing denoises structured data and converts it into text for LLM comprehension.
- The SOP flow follows common SRE workflows and includes tools for retrieving relevant metrics and handling unexpected incidents.
- Additional MSA analysis tools query operational state through service- and pod-level analysis.
2.3 SOP Flow
SOP flow structures RCA around retrieving or generating SOPs, converting them into executable code, and using observations to guide progressive localization. It constrains LLM orchestration while supporting execution of interdependent diagnostic steps.
- SOP flow: SOP flow instructs LLMs to use standardized RCA steps, including matching relevant SOPs and generating new ones when no suitable SOP exists.The full flow is provided to the MainAgent and ActionAgent through prompts.
- Fault Type/Information→SOP: Matching ranks SOP-name embeddings and selects the top k matches subject to a relevance threshold.This filtering reduces matches with highly irrelevant SOPs.
- Fault Type/Information→SOP: When existing SOPs do not match a fault, generate_sop uses an LLM and existing SOPs as few-shot prompts to create a new standardized procedure.The generated IO-error SOP may lack precision but provides a rigorous and useful diagnostic direction for SREs.
- SOP hierarchy: Hierarchical SOPs guide RCA from broad fault categories to specific problems, such as progressing from network issues to network partition problems.This macro-to-micro progression is intended to mirror real-world diagnosis.
- SOP→SOP Code: generate_sop_code converts an SOP into executable code so interdependent or multiple atomic actions can be executed more accurately and simultaneously.The resulting code is run through run_sop, with regeneration expected when syntax or variable errors occur.
- Observation-guided localization: After run_sop produces an observation, match_observation retrieves similar historical incidents and ObAgent proposes potential fault types for subsequent localization.This supports the layered progression of the RCA process.
2.4 Action Set
The action set addresses RCA planning difficulty by generating and evaluating multiple plausible actions before execution. It combines agent suggestions with flow-based rules to balance flexibility against uncontrolled randomness.
- Motivation: RCA planning is difficult because observations vary and multiple actions may be plausible for the same observation.The LLM must identify and execute a reasonable action from many viable choices.
- Action-set construction: The action set generates several candidate actions, each paired with a textual rationale for its selection.Candidates come from the ActionAgent and JudgeAgent.
- Action-set construction: Flow-based rules add necessary follow-up actions, such as generate_sop_code after generate_sop, while JudgeAgent adds Speak when the root cause is identified.These rules make the candidate set more comprehensive and logically connected to the SOP flow.
- Design effect: The action set mitigates planning problems from diverse observations and many feasible actions while balancing stochasticity with determinism.The design aims to avoid both diagnostic divergence from excessive randomness and rigid scripted behavior from excessive determinism.
2.5 Multi-Agent System
Flow-of-Action uses a multi-agent system in which a MainAgent orchestrates RCA and auxiliary agents provide specialized suggestions. The auxiliary roles support action selection, observation interpretation, and stopping decisions.
- Agent roles: The MainAgent is the principal agent with authority and orchestrates the entire localization process.Other agents provide suggestions rather than controlling the process.
- Agent roles: The ActionAgent supplies a feasible action set for the MainAgent to choose from.This role supports planning under multiple plausible actions.
- Agent roles: The ObAgent provides potential anomaly types or information after match_observation, while the JudgeAgent determines whether the root cause has been identified.Together these roles support interpretation and termination of localization.
3 Evaluation
The evaluation uses a Kubernetes microservices testbed, metrics for root-cause location, type, and path efficiency, and comparisons against open-source RCA baselines. Flow-of-Action outperforms the baselines, while ablations show that SOP flow, action sets, and auxiliary agents contribute differently to accuracy and efficiency.
- Dataset: The dataset contains 90 incidents from nine anomaly types injected into GoogleOnlineBoutique2, a Kubernetes deployment with over 10 services.Prometheus, Elastic, DeepFlow, and Jaeger collect metric, log, and trace data.
- Metrics: Evaluation measures Root Cause Location Accuracy, Root Cause Type Accuracy, and Average Path Length for LLM-agent methods.The location and type metrics penalize incorrect causes with a hyperparameter σ, while APL measures diagnosis-path efficiency.
- Overall Performance: Flow-of-Action surpasses the SOTA by 23% in location accuracy and 28% in type accuracy.Baselines include Kubernetes RCA tools and general-purpose CoT, ReAct, and Reflexion frameworks.
- Overall Performance: Flow-of-Action maintains an acceptable APL, whereas ReAct tends to identify causes incorrectly and Reflexion incurs many iterations.Excessive path lengths can increase resource consumption and knowledge clutter.
- Ablation Study: Removing SOP knowledge causes significantly low accuracy, while removing SOP-flow prompts mainly decreases location accuracy and leaves type accuracy relatively effective.Without SOP flow, the system cannot fully execute the SOP and loses location information.
- Ablation Study: Removing the action set harms judgments in complex and rare scenarios but lowers APL, while removing individual auxiliary agents decreases accuracy.The multi-agent design reduces cognitive load by allowing the main agent to use other agents’ opinions.
- Ablation Study: The APL effects of most ablations are similar because diagnosis is capped at 20 steps to prevent unbounded loops.The action-set ablation has the strongest APL effect.
4 Conclusion
Flow-of-Action addresses frequent-incident RCA by combining SOP guidance, action-set selection, and auxiliary agents in a multi-agent system. The authors report experimental evidence supporting its efficacy.
- Flow-of-Action is a novel SOP-enhanced multi-agent system for RCA.
- Its SOP flow alleviates hallucinations during orchestration, while the action-set mechanism addresses action selection under diverse observations.
- A main agent supported by multiple auxiliary agents refines responsibility allocation and enhances overall accuracy.
A Multimodal Data Collection
The data collection setup combines Prometheus and DeepFlow for metrics, Jaeger for traces, and Elastic for logs. Rule-based or keyword-based methods detect anomalies in the collected data.
- Prometheus collects architecture-level metrics, while DeepFlow gathers business-level metrics such as business traffic data.Prometheus covers pod-level and node-level indicators; DeepFlow captures business-level measurements.
- Jaeger collects trace data represented as call chains of spans, and trace anomalies are detected by extracting span error messages.Each span corresponds to a single call, and a span failure typically includes an associated error message.
- Elastic collects logs, with keyword-based extraction used to identify anomalies in abnormal log messages.
- Figures 8–11 depict the Prometheus, DeepFlow, Jaeger, and Elastic dashboards used by the data collection systems.
B Tools
The section presents Table 5 as a description of the tools.
- Table 5 provides a description of the tools.