Source-linked AI summary
AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds
Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, Saravan Rajmohan
TL;DR
AIOps research lacks unified, realistic frameworks for evaluating agents across complex and evolving cloud incidents. AIOPSLAB addresses this gap by orchestrating microservice environments, faults, workloads, telemetry, and agent interactions, then demonstrates the framework with a 48-problem benchmark and four agents. The evaluation exposes distinct agent capabilities and limitations across operational tasks.
Problem
Existing AIOps evaluation approaches lack integrated, realistic support for diverse agents operating across complex cloud incidents.
Method
AIOPSLAB combines service deployment, fault injection, workload generation, telemetry, orchestration, and agent-cloud interfaces into an end-to-end evaluation framework.
Results
The framework constructs a benchmark with 48 problems across AIOps tasks and evaluates four LLM-based agents, revealing distinct challenges across tasks.
Takeaways & Limitations
AIOPSLAB supports interactive evaluation of agents across different AIOps tasks and provides analysis of their performance and limitations.
Takeaways & Limitations
Agents often waste steps through repeated or invalid API actions and can become overloaded when consuming telemetry data.
Abstract
from arXiv · showhide
AI for IT Operations (AIOps) aims to automate complex operational tasks, such as fault localization and root cause analysis, to reduce human workload and minimize customer impact. While traditional DevOps tools and AIOps algorithms often focus on addressing isolated operational tasks, recent advances in Large Language Models (LLMs) and AI agents are revolutionizing AIOps by enabling end-to-end and multitask automation. This paper envisions a future where AI agents autonomously manage operational tasks throughout the entire incident lifecycle, leading to self-healing cloud systems, a paradigm we term AgentOps. Realizing this vision requires a comprehensive framework to guide the design, development, and evaluation of these agents. To this end, we present AIOPSLAB, a framework that not only deploys microservice cloud environments, injects faults, generates workloads, and exports telemetry data but also orchestrates these components and provides interfaces for interacting with and evaluating agents. We discuss the key requirements for such a holistic framework and demonstrate how AIOPSLAB can facilitate the evaluation of next-generation AIOps agents. Through evaluations of state-of-the-art LLM agents within the benchmark created by AIOPSLAB, we provide insights into their capabilities and limitations in handling complex operational tasks in cloud environments.
1 INTRODUCTION
AIOpsLab addresses the lack of realistic, unified benchmarks for evaluating agents across dynamic cloud incidents. It provides an end-to-end framework and benchmark suite for interactive evaluation of LLM-based AIOps agents.
- Cloud systems scale through distributed architectures but introduce operational complexity that can cascade into costly outages.An Amazon outage, for example, can cause $100 million in losses in one hour.
- AIOPSLAB automates service deployment, fault injection, workload generation, agent-cloud interaction, and result analysis through a unified framework.Its Agent-Cloud Interface lets agents communicate, act, and receive feedback while solving operational problems.
- Existing AIOps benchmarks often rely on static datasets, proprietary services, or isolated incident-lifecycle tasks, limiting comprehensive evaluation.These approaches do not capture fluctuating workloads and incidents in dynamic cloud environments.
- The framework models realistic microservice incidents with fine-grained root causes to assess agents’ diagnostic and mitigation abilities.Its scenarios are organized by a task-level taxonomy spanning the incident management lifecycle.
- AIOPSLAB created 48 evaluation problems and registered four LLM-based agents, revealing distinct challenges across tasks.Agent registration required less than a hundred lines of code.
2 AIOPSLAB
This section introduces AIOPSLAB’s design and implementation as a framework composed of coordinated components for evaluating AIOps agents.
- AIOPSLAB’s design and implementation are organized around the components illustrated in Figure 2.
2.1 Problem Definition
AIOPSLAB formalizes each evaluation scenario as a task, context, and expected solution, enabling realistic incident problems with task-specific evaluation criteria.
- Each AIOps problem is represented as P = ⟨T, C, S⟩, where T is the task, C the context, and S the expected solution.Tasks include detection, localization, root-cause analysis, and mitigation.
- AIOPSLAB evaluates problems using success criteria and metrics tailored to each task, such as Time-to-Detect for detection.
- The context combines the operational environment with problem information shared directly or queried by the agent at runtime.It includes services, fault and workload models, descriptions, API documentation, logs, metrics, and traces.
- Solutions are task-specific, while mitigation problems may accept multiple valid actions and evaluate the resulting state of the entire system.For example, evaluation can check whether all services are running rather than only the targeted resource.
- A Kubernetes target-port misconfiguration example defines localization of the fault and identifies user-service as the expected solution.The scenario uses a social-network application, a library fault, and a standard wrk workload.
2.2 Orchestrator
The Orchestrator separates agents from services through a unified interface that deploys and perturbs environments, exposes telemetry and actions, manages sessions, and evaluates outcomes.
- Agent-Cloud Interface: The Orchestrator provides a central separation between agents and cloud services while supporting integration and extension of system components.
- Agent-Cloud Interface: The Agent-Cloud Interface defines valid agent actions and conveys service-state observations through concise, documented APIs.Examples include retrieving logs, metrics, and traces, or executing filtered shell commands.
- Agent and service lifecycle: A session is created for each agent solving a problem, with the Orchestrator providing context and polling the agent for successive actions.
- Problem Initializers: The Orchestrator deploys services and creates realistic problems by combining workload generation with controlled, fine-grained fault injection.AIOPSLAB uses infrastructure-as-code deployment, wrk2 workloads, and an extensible fault library.
- Problem Evaluators: The Orchestrator compares agent solutions with task-specific success criteria and metrics, including detection time, action steps, and generated tokens.
2.3 Cloud Services
AIOPSLAB deploys live microservice applications as cloud environments, using DeathStarBench’s SocialNetwork and HotelReservation applications as current testbeds.
- Cloud service testbeds: AIOPSLAB currently integrates the SocialNetwork and HotelReservation applications from DeathStarBench as live cloud environments.SocialNetwork includes 28 microservices, while HotelReservation supports hotel recommendation and reservation services.
2.4 Task-oriented Fault Library
AIOPSLAB’s task-oriented fault library supports progressively complex incident-management tasks by injecting symptomatic and functional faults into realistic service scenarios.
- Task taxonomy: The task taxonomy organizes AIOps evaluation across incident-management stages with progressively increasing complexity and impact.Higher levels represent harder and more impactful tasks.
- Fault categories: Fault injection constructs problems across task levels and classifies them as symptomatic or functional faults.Symptomatic faults support detection and localization, while functional faults expose deeper operational causes.
- Functional faults: Functional-fault scenarios require agents to detect, localize, diagnose root causes, and apply mitigation strategies.An example revokes MongoDB authentication for the geographic microservice, producing abnormal behavior and error logs in the Geo service.
- Fault implementation: AIOPSLAB’s fault-injection library includes an application-level revoke-authentication injector with injection and recovery operations.The example code targets the HotelReservation application and revokes MongoDB administrative privileges.
2.5 Observability
AIOPSLAB provides an observability layer that collects distributed traces, application logs, and system metrics for monitoring its benchmark environments.
- Telemetry collection: AIOPSLAB’s telemetry collector gathers Jaeger traces, application logs, and Prometheus system metrics.The observability layer combines end-to-end request paths, logs retrieved or recorded through its logging components, and system-level measurements.
3 EVALUATION
The evaluation uses AIOPSLAB to compare diverse LLM and traditional AIOps agents on realistic, interactive incident tasks. Results show strong but uneven capabilities: agents outperform traditional methods on detection and localization, while complex tasks expose accuracy, efficiency, and telemetry-use limitations.
- Evaluation Setup: AIOPSLAB evaluates four LLM-based agents and three non-LLM AIOps algorithms using task-specific correctness, time, steps, and token-cost metrics.The LLM agents include GPT-W-SHELL, REACT, and FLASH; traditional methods target detection or localization.
- Problem Pool of AIOPSLAB Benchmark: The benchmark contains 48 problems spanning detection, localization, root-cause analysis, and mitigation across functional and symptomatic faults.Functional faults cover all four task levels, whereas symptomatic faults are limited to detection and localization.
- Performance Results: FLASH achieves the highest overall accuracy, while GPT-3.5-TURBO is fastest but has the lowest accuracy at 15.25%.This result illustrates a trade-off between speed and correctness across agents.
- Performance Results: RCA and mitigation are the most challenging tasks; only FLASH answers all detection problems correctly, and GPT-3.5-W-SHELL recovers no mitigation failures.REACT performs best for localization when evaluated using its top 3 candidate services, but its top-1 accuracy is lower.
- Performance Results: None of the agents consistently achieves high accuracy across all four task categories, with even FLASH limited on complex mitigation problems.The benchmark therefore reveals task-dependent weaknesses despite promise on realistic operational scenarios.
- Agent Behavior: The Good, the Bad and the Gaps: Agents often waste steps through repeated or invalid API calls, excessive multiagent communication, and indiscriminate telemetry consumption that can exhaust context and tokens.GPT-3.5-W-SHELL repeats incorrect API commands, while telemetry dumps can add noise; REACT sometimes self-corrects invalid commands.
4 DISCUSSION
AIOPSLAB supports customized incident scenarios and controlled agent evaluation through Agent-Cloud Interfaces. Its optional LLM-as-Judge can assess whether an agent’s reasoning matches the fault, workload, and environment.
- AIOPSLAB helps engineers create customized incident scenarios while ACIs constrain agents to controlled evaluation environments.Users define incidents and the specific problems agents should solve.
- AIOPSLAB can accommodate anomaly-detection scenarios in which agents label workload or telemetry data.The framework is adaptable to additional fault types and task formulations.
- Optional LLM-as-Judge evaluators compare agent reasoning with the problem description, including the fault, workload, and environment setup.This addresses cases where an agent selects the correct answer but gives an incorrect explanation.
5 RELATED WORK
Prior AgentOps systems demonstrate LLM use in cloud operations, but evaluation remains constrained by proprietary implementations and the absence of unified, realistic benchmarks. Existing AIOps benchmarks typically use static metrics or fixed question-answer datasets rather than dynamic cloud environments.
- AgentOps approaches apply LLMs to monitoring and analyzing complex system behaviors, but many lack publicly available implementations and associated private datasets.
- A unified benchmark for realistic evaluation across operational tasks remains absent.The related-work discussion identifies this as a gap for assessing agents’ performance across tasks.
- Existing AIOps benchmarks primarily rely on static system metrics or fixed question-answer formats.These formats are typically time series data or text-based datasets.
6 CONCLUSION
The paper presents AIOPSLAB as a comprehensive framework for simulating cloud incidents and evaluating autonomous AIOps agents. It combines infrastructure components with an agent-cloud interface and demonstrates the framework using 48 problems and four agents.
- AIOPSLAB combines a fault injector, workload generator, cloud-agent orchestrator, and telemetry observer to simulate cloud incidents.
- The framework provides an agent-cloud interface for orchestrating and evaluating AIOps agents.
- 48 problems and four agents demonstrate AIOPSLAB’s application across different types of AIOps tasks.