PaperlayerView exact revision on arXiv

Source-linked AI summary

ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutlin, Saki Takano, Divya Pathak, Felix George, Xinbo Wu, Bekir O. Turkkan, Gerard Vanloo, Michael Nidd, Ting Dai, Oishik Chatterjee, Pranjal Gupta, Suranjana Samanta, Pooja Aggarwal, Rong Lee, Pavankumar Murali, Jae-wook Ahn, Debanjana Kar, Ameet Rahane, Carlos Fonseca, Amit Paradkar, Yu Deng, Pratibha Moogi, Prateeti Mohapatra, Naoki Abe, Chandrasekhar Narayanaswami, Tianyin Xu, Lav R. Varshney, Ruchi Mahindru, Anca Sailer, Laura Shwartz, Daby Sow, Nicholas C. M. Fuller, Ruchir Puri

arXiv:2502.05352v1cs.AIcs.DCcs.MA

TL;DR

AI agents need reliable evaluation on complex real-world IT tasks before deployment. ITBench provides an extensible benchmarking framework with push-button workflows and interpretable evaluation across SRE, CISO, and FinOps; current agents resolve only a minority of scenarios, while real-world deployment also raises security risks.

  • Problem

    The complexity of IT tasks creates a need for systematic evaluation of AI agents on real tasks and environments before production deployment.

  • Method

    ITBench benchmarks real-world IT automation through scenario-based environments, specialized tools, automated partial scoring, and push-button deployment infrastructure.

  • Results

    Agents achieve low success rates across ITBench, with GPT-4o attaining 13.81% diagnosis and 11.43% mitigation pass@1 on SRE scenarios, while removing traces lowers these scores further.

  • Takeaways & Limitations

    ITBench provides a reproducible platform for studying complex IT-agent performance and open problems in observability, specialization, collaboration, safety, and human-in-the-loop operation.

  • Takeaways & Limitations

    Real-world use of ITBench agents could enable malicious code or offensive security activity, requiring constraints and careful control of deployment.

Abstract

from arXiv · show

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.

1. Introduction

Modern IT complexity and high-impact failures create an urgent need to evaluate AI agents on real IT tasks before deployment. ITBench addresses this need with broad, extensible benchmarking across SRE, CISO, and FinOps, supported by automated evaluation and deployment tooling.

  • Modern IT complexity makes it difficult for organizations to maintain resilient, reliable, secure, and cost-effective systems.
  • A $5.4 billion CrowdStrike outage underscored the need for intelligent incident resolution, compliance, and risk-management capabilities.
  • AI-agent research needs systematic evaluation methods grounded in real IT tasks and their corresponding environments before production deployment.
  • ITBench benchmarks complex real-life IT automation across SRE for availability and resiliency, CISO for compliance and security, and FinOps for cost efficiency and return on investment.
  • ITBench reflects production requirements, supports vertical and horizontal expansion, and provides automated evaluation with partial scoring.
  • ITBench provides push-button deployment and tooling for environments, runtime agents, guardrails, authorization, and authentication, and initially covers 94 scenarios.
  • The framework is intended to accelerate performance improvements in real-world IT agents through reproducible, community-oriented benchmarking.

2. Related Work

Existing IT benchmarks address specific tasks but provide limited coverage of real-world IT problems and automated evaluation. ITBench is positioned as a broader comparison framework spanning SRE, CISO, and FinOps use cases.

  • ITBench targets IT Operations personas and tasks, initially focusing on Site Reliability Engineering, Compliance and Security Operations, and Financial Operations.
  • Related benchmarks include incident scenarios, interactive SRE environments, ticket-analysis scenarios, and CIS security recommendations, but cover narrower use cases.
  • Table 1 compares ITBench with related benchmarks across their supported IT domains and evaluation capabilities.
  • FinOps Foundation benchmarks compare cloud financial performance using metrics such as resource utilization efficiency, contract coverage, and cost apportionment.
  • Existing benchmarks are limited in reflecting real-world IT problems, covering the broader IT landscape, and automating evaluation.

3. ITBench

ITBench is a systematic benchmarking framework and runtime environment for evaluating AI agents on complex IT operations through scenarios, interactive environments, task metrics, and comparative reporting.

  • ITBench combines an AI Agent, Scenario Specification and Environment, Evaluator, and Leaderboard for comprehensive performance assessment.
  • 3.1. Scenario Specification and Environment: Each scenario represents an IT problem formalized as metadata, an operational environment, triggering events, and a desired outcome.Scenario metadata includes names, descriptions, domains, classes, complexity, and ground truth.
  • 3.2. AI Agents: Agents interact with scenario environments through tools, receive observations, choose actions, and continue until they take the stop action or achieve the final goal.The environment exposes APIs for observing and controlling system components; agent decisions depend on prior observations and actions.
  • 3. ITBench: ITBench evaluates SRE incident resolution, CISO compliance posture assessment, and FinOps cost-management tasks using independently assessed task metrics.Examples include diagnosing incidents, collecting compliance evidence, scanning assessment posture, identifying inefficiency, and mitigating inefficiency.
  • 3.3. Baseline AI Agents: Baseline agents use ReAct planning, reflection, disaggregation, and specialized tools for traces, metrics, logs, Kubernetes operations, and summarization.The SRE and FinOps agents use specialized tools to reduce the burden of large operational data volumes on language-model context windows.
  • An open leaderboard supports reproducibility and comparative analysis, while ITBench is designed for extensible scenario and task coverage.The framework supports agent registration and scenario selection, and plans call for additional tasks and domains.

4. Results

ITBench evaluates agents across SRE, CISO, and FinOps scenarios using task-specific effectiveness metrics. Baseline agents often struggle, with performance varying by model, scenario complexity, observability, and repeated runs.

  • SRE: GPT-4o achieved the highest SRE diagnosis pass@1 at 13.81% and mitigation pass@1 at 11.43%.
  • SRE: Removing traces reduced GPT-4o’s SRE diagnosis pass@1 from 13.81% to 9.52% and mitigation pass@1 to 2.86%.The results identify observability as important for evaluating SRE-agent performance under varying conditions.
  • CISO: CISO-agent evaluation measures compliance detection or confirmation, with GPT-based models dominating pass@1 and Time to Process metrics.Their pass@1 was nearly twice that of the second-best models.
  • FinOps: GPT-4o achieved a 33% pass rate for diagnosing FinOps cost-increase alerts, while no model attained optimal CPU and memory cost or high CPU efficiency.
  • Scenario complexity: GPT-4o’s SRE diagnosis success declined from 36% on Easy scenarios to 7.73% on Medium and 5.0% on Hard scenarios.GPT-4o mitigation success similarly declined from 21% to 12.27% and 0.0%.
  • Run variability: Repeated runs revealed substantial GPT-4o variability, with minor telemetry changes altering token generation and exposing robustness differences.ITBench tracks this dynamic behavior across runs to provide insight into agent reliability.

5. Discussion and Conclusion

ITBench is presented as a reproducible framework and experimental platform for benchmarking AI agents on complex IT automation tasks. Its extensible design supports broader scenario coverage, community contributions, and further investigation of safety and multi-agent automation.

  • Conclusion: ITBench benchmarks AI agents for IT automation while capturing the complexity of modern systems and the diversity of IT tasks.The framework is intended to remain reproducible despite nondeterminism in large-scale IT systems.
  • Future scope: ITBench is designed to support diverse IT systems and extend to new scenarios, processes, and event types through community contributions.The authors plan to expand beyond the current event-triggered scenarios and enrich benchmark coverage.
  • Open problems: The paper identifies domain-specialized agents, multi-agent orchestration, and safety of agent-driven solutions as open problems for future research.

6. Statements

ITBench frames open, extensible benchmarking as a way to evaluate AI agents on complex IT automation tasks while addressing security and deployment risks.

  • Contributions: ITBench measures AI-agent performance across complex, real-life IT automation tasks and is intended to support correct, safe, and fast automation.The framework is open-source and can also support organizations developing and benchmarking proprietary solutions.
  • Reproducibility: Open-source resources, repositories, and support forums are provided to enable reproduction and further development of ITBench.The authors describe the framework as encouraging community knowledge sharing and lowering barriers to IT-domain innovation.
  • Motivation: Real-world IT automation remains difficult because incidents vary, systems are complex, and AI systems must incorporate contextual knowledge.Existing research has not yet made fully automated incident resolution or actionable insights routine.
  • FinOps gap: FinOps benchmarking lacks specific use cases that test automated optimization of IT investments and reduction of resource waste.The paper identifies a need to move beyond comparative scorecards and broad taxonomies.

B.3. Leaderboard

The Leaderboard standardizes agent evaluation by registering agents, provisioning scenarios, collecting statuses, evaluating outcomes, and publishing interpretable performance metrics.

  • Metadata: The initial Leaderboard covers SRE, CISO, and FinOps agent types, with beginner, intermediate, and expert levels mapped to scenario complexity.CISO scenario classes include RHEL9 with OPA and Kubernetes with Kyverno or OPA/Kyverno updates.
  • Leaderboard: The Leaderboard provides predefined, extensible performance metrics that clarify agent performance relative to evaluation criteria.It supports both API and UI interfaces for the benchmarking workflow.
  • Workflow: Agents register endpoints, retrieve scenarios, interact with provisioned environments, report status, and receive aggregated results through the Leaderboard.The workflow supports hosted automatic deployment or manual deployment in a user-managed environment.
  • Workflow: The Benchmark Runner provisions environments, injects faults, marks scenarios ready, monitors agent execution, evaluates results, and deletes scenarios.Completed results across scenarios are aggregated and published to the Leaderboard.
  • SRE tasks: SRE failure management comprises detection, diagnosis, and mitigation, with ITBench currently focusing on diagnosis and mitigation.The paper describes failures as observable effects arising when faults propagate through errors.
  • Complexity: Overall task complexity is defined as the geometric mean of fault-propagation length, resolution-plan size, and technology heterogeneity.Figure 9 is used to show the resulting distribution of complexity across incident scenarios.

C.4.1. EXPERIMENTAL SETUP

ITBench implements reproducible fault-injection experiments using configurable Kubernetes network policies and supports both larger AWS clusters and lightweight Kind clusters.

  • Fault injection: Fault injection and removal are implemented as configurable Ansible playbooks that apply or remove Kubernetes network policies blocking target-service ports.Variables select the target service, port, namespace, and whether the playbook performs injection or removal.
  • Fault injection: The network-policy implementation applies a deny policy for a target service and removes that policy during fault removal.The policy uses Kubernetes ingress rules and selected target-port and namespace variables.
  • Infrastructure: Experiments used an AWS m4 xlarge cluster with one control-plane node and three worker nodes.The worker nodes had 12 cores and 48 GiB of RAM, with 16 cores and 64 GiB used in total.
  • Infrastructure: ITBench also supports Kind clusters as lightweight, portable environments for local testing and reproducible fault scenarios.The validated Kind setup used one control-plane node, 12 CPU cores, and 16 GB RAM on Red Hat Enterprise Linux.
  • Agent execution: SRE agents interact with target environments, collect observability data, and execute actions to diagnose and mitigate incidents.The SRE goal is defined around incident diagnosis and mitigation within the benchmark environment.

C.5.1. OBSERVABILITY DATA

SRE-Agent processes multimodal observability data through sequential diagnosis and mitigation agents, using tools that gather evidence, generate plans, and execute corrective actions.

  • Observability data: SRE tasks analyze logs, traces, and metrics as multimodal observability data.Logs record events, traces represent distributed request execution, and metrics provide time-series indicators such as latency and error rates.
  • Agent architecture: SRE-Agent uses two sequential LM-based agents: a Diagnosis Agent followed by a Resolution or Mitigation Agent.Sequential execution is selected because it fits the incident-resolution workflow.
  • Definitions: In ITBench, an agent uses a language model to plan, decide, interact with the target environment, and execute actions toward a goal.Tasks define the goals and tools define functions or API calls for completing subtasks.
  • Tooling: Tools expose observability retrieval and environment actions through natural-language outputs or function calls.LM-based tools can use reflection with linters to correct invalid function calls, while summarizers condense long outputs.
  • Diagnosis: The Diagnosis Agent retrieves active alerts and iteratively gathers traces, logs, metrics, and Kubernetes information before producing a diagnosis.Its diagnosis task identifies the fault-propagation chain and localizes the exact cause.
  • Mitigation: The Mitigation Agent converts the diagnosis report into a natural-language plan, executes it with Kubernetes tools, and checks whether alerts clear.The plans explain why commands were executed and improve agent explainability.

C.6.1. EXPERIMENTAL DETAILS

The SRE-Agent is evaluated across diagnosis and mitigation tasks using exact-match, topology-aware, and time-based metrics. ITBench also introduces NTAM to measure partial correctness in fault localization and propagation chains.

  • Experimental setup: Four non-fine-tuned language models are evaluated across 42 SRE scenarios.The models are gpt-4o, granite-3.1-8b-instruct, llama-3.3-70b-instruct, and llama-3.1-8b-instruct.
  • Diagnosis: Diagnosis measures fault localization, propagation-chain quality, and mean time to diagnosis.Fault localization uses pass@1 and NTAM; propagation chains use NTAM; MTTD measures diagnostic speed.
  • Mitigation: Mitigation measures incident resolution with pass@1 and mean time to repair.A successful mitigation clears the relevant alerts, while MTTR measures how quickly the action succeeds.
  • Metrics: Pass@1 measures correctness in single trials: exact root-cause matching for localization and cleared alerts for mitigation.The metric is described as an unbiased estimator of correctness when k=1.
  • NTAM: NTAM evaluates predicted root causes and propagation chains using topology-aware distance rather than exact matching alone.Its components reward nearby and discriminative entities, account for root-cause position, and penalize length mismatches.

C.6.4. EVALUATION RESULTS

Across 42 SRE scenarios, gpt-4o performs best overall, but performance falls sharply with scenario complexity and depends strongly on trace availability. Diagnosis and mitigation can diverge, and repeated runs can be inconsistent.

  • Overall results: gpt-4o achieves the strongest overall results, with 13.81% diagnosis pass@1 and 11.43% mitigation pass@1.It also leads on NTAM metrics, while no agent repairs the hard scenarios.
  • Scenario complexity: Performance declines from Easy to Hard scenarios, and none of the five hard scenarios is resolved by any agent.Five of eight easy scenarios are repaired by at least one agent.
  • Diagnosis and mitigation: Diagnosis and mitigation can succeed independently: agents sometimes repair incidents after misidentifying the root cause, while accurate diagnosis may not yield repair.Scenario 15 illustrates symptom-level mitigation after an incorrect diagnosis; scenario 13 shows strong diagnosis without successful repair.
  • Run consistency: Repeated runs can produce inconsistent outcomes because real-time telemetry fluctuations alter model outputs.In one example, telemetry changes accompany outcomes ranging from occasional success to substantially varying execution times.
  • Tracing: Masking traces substantially reduces performance: gpt-4o diagnosis pass@1 falls from 18.10% to 9.52%, while mitigation pass@1 falls to 2.86%.For llama-3.3-70B, diagnosis declines from 5.24% to 0.95%; only three scenarios remain resolved by gpt-4o after masking.

D.6. CISO ITBench Evaluation

The CISO evaluation tests compliance assessment across 50 scenarios using multiple language models and code-focused models. Agents identify compliant or non-compliant artifact configurations, with success and processing speed measured separately.

  • Evaluation setup: The CISO compliance assessment agent is evaluated on 50 scenarios spanning four scenario classes.Classes impose distinct CIS-Benchmark requirements, complexity levels, and scenario-specific code artifacts.
  • Models: The evaluation uses GPT-4o, Llama, Granite, GPT-4o-mini, Llama-3.1-405b-instruct, and Mixtral-8x7b-instruct models.Natural-language tasks use reasoning models, while code-focused tasks additionally use specialized model choices.
  • Assessment task: CISO agents assess whether artifact configurations are compliant or contain injected misconfigurations.Examples include container namespace requirements and threshold violations.
  • Metrics: Success is measured with pass@1, while Time to Process measures how quickly an assessment is completed.TTP is calculated for scenarios whose posture is correctly identified.

D.6.2. METRIC DEFINITIONS

The CISO metric definitions specify correctness and processing-time measures for posture assessment. Results show GPT-based models leading overall, while increasing scenario difficulty reduces performance.

  • Metric definitions: Pass@1 measures whether the agent correctly classifies posture as pass or fail.It is described as an unbiased estimator of correctness in k=1 trials.
  • Metric definitions: Time to Process measures how quickly an agent correctly assesses a scenario, with unsuccessful cases assigned infinite time.A compliant configuration should receive pass, while a misconfiguration should receive fail.
  • Results: GPT-based models dominate both pass@1 and Time to Process, with pass@1 nearly twice that of the second-best models.The second-best models alternate between llama-3.1-405b-instruct and mistral-large-2.
  • Results: All models struggle more as CISO scenario difficulty rises from Easy Kyverno to Hard Kyverno-update classes.The mapping places k8s-opa and rhel-opa at Medium complexity.

D.6.4. CISO COMPLIANCE ASSESSMENT AGENT TRAJECTORIES

The trajectories contrast a successful compliance assessment workflow with an unsuccessful one in which repeated Rego syntax errors prevent evaluation from passing.

  • Successful scenario: The successful scenario collects host data with a generated playbook, creates an OPA policy, and detects compliance issues through bench evaluation.It assesses whether cron is restricted to authorized users, including requirements for /etc/cron.allow and /etc/cron.deny.
  • Unsuccessful scenario: The unsuccessful scenario addresses SSH LoginGraceTime but fails to create a syntactically valid Rego policy despite successfully generating and testing the playbook.The requirement is to set SSH LoginGraceTime to one minute or less.
  • Unsuccessful scenario: After three retries, the maximum retry limit is reached and the syntactically incorrect policy is evaluated, so the scenario does not pass.Only the currently saved playbook.yml and policy.rego are used at evaluation.

E.2. Motivating Example and FinOps Tasks

The FinOps tasks evaluate agents on diagnosing cost alerts and recommending mitigations, using pass@1 and proximity to optimal cost and efficiency. A motivating example shows that interpreting utilization and load is necessary to avoid harmful scaling recommendations.

  • Motivating example: The motivating incident involves a cost increase above 20% caused by increased replicas during a load spike, while unchanged budget thresholds generate a false alert.The application remains healthy, and the agent recommends updating budget alerts based on the new load level.
  • Evaluation setup: OpenCost monitors Kubernetes costs and raises alerts when predefined budget or efficiency thresholds are crossed, using a custom pricing model in the experiments.The setup is based on the evaluation framework used for SRE scenarios.
  • Evaluation tasks: FinOps tasks evaluate agents on two primary tasks: diagnosis of budget-alert root causes and mitigation through resolution and cost-efficiency recommendations.Diagnosis uses pass@1 accuracy, while mitigation uses pass@1 and proximity scores.
  • Metrics: Proximity scores measure proportional absolute difference from optimal values, with 1 indicating optimal performance across cost and workload-efficiency metrics.The measured metrics include hourly CPU cost, hourly memory cost, workload CPU efficiency, and workload memory efficiency.
  • Motivating example: An inaccurate diagnosis leads to a scaling-down recommendation despite high application load, whereas a correct trajectory analyzes utilization and considers threshold updates or safe deployment scaling.The recommended choice depends on whether total budget allocation permits threshold changes and whether deployments can be scaled without harming performance.
Loading 2502.05352v1…