ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

1. Introduction

• Cloud-driven complexity strains IT teams tasked with resilient, reliable, secure, cost-effective systems, while agent adoption makes real-task predeployment evaluation urgent.Source: §1 · p. 1
• ITBench targets SRE availability and resiliency, CISO compliance and security, and FinOps cost efficiency and return on investment.Source: §1 · p. 1
• As an open-source hub, ITBench supports vertical and horizontal expansion and partial-scoring feedback, aiming to spur improved real-world IT agents.Source: §1 · p. 2
• Successful scenario handling rates were 13.8% for SRE, 25.2% for CISO, and 0% for FinOps across ITBench’s 94 scenarios.The benchmark comprises 42 SRE, 50 CISO, and 2 FinOps scenarios.Source: §1 · p. 2
• Push-button deployment bundles environment and runtime-agent setup, guardrails, authorization, and authentication for reactive incident, compliance, and cost-management tasks.Source: §1 · p. 2

Related Work

• TrainTicket covers 22 incident scenarios for fault localization, while AIOpsLab offers 10 interactive, real-environment SRE problems.TrainTicket uses hardcoded faults; AIOpsLab provides access to text, time-series, and tabular data.Source: §2 · p. 2
• InsightBench (Sahu et al., 2024) analyzes 100 ticket-data scenarios using static tables and synthetic cases, while TSB-AD (Liu and Paparrizos, 2024a) focuses on 40 synthetic anomaly-detection scenarios.Source: §2 · p. 2
• CIS-Benchmark (CIS, 2024) offers security recommendations but no experimental platform; OSCAL-compass (OSCAL-compass, 2024) enables programmatic compliance, which ITBench CISO automation uses to assess policy requirements.Source: §2 · p. 2
• FinOps Foundation benchmarks compare cloud financial performance across organizations using KPIs, and the authors note no known FinOps benchmarks beyond scorecards.The cited KPIs include resource-utilization efficiency, contract coverage, and cost apportionment.Source: §2 · p. 3
• Existing benchmarks are limited in real-world relevance, breadth across IT, and evaluation automation; ITBench is designed to address these gaps, with Table 1 comparing it to related benchmarks.Source: §2 · p. 3, Table 1 · p. 3

3. ITBench

• ITBench combines an AI agent, scenario specification and environment, evaluator, and leaderboard to assess agents automating IT operations.Its leaderboard supports reproducibility and comparative analysis.Source: §3 · p. 3, §3.4 · p. 6

3.1. Scenario Specification and Environment

• Scenarios model IT problems through metadata, an operational testbed, triggering events, and a desired outcome.Metadata includes domain, class, complexity, and ground truth; SRE scenarios draw on observed incidents, CISO scenarios on CIS benchmarks, and FinOps scenarios on FinOps Foundation domains and capabilities. Testbeds expose APIs for observing and controlling the environment, and the runner instantiates them for selected scenarios.Source: §3.1 · pp. 3–4

3.2. AI Agents

• Agents address SRE incident resolution, CISO compliance posture assessment, and FinOps cost management by interacting with the environment through tools.Tasks are assessed independently using defined metrics; agents observe and act iteratively until stopping, targeting success across their scenario set.Source: §3.2 · pp. 4–5

3.3. Baseline AI Agents

• Baseline agents for SRE, CISO, and FinOps use techniques including ReAct planning, reflection, and task disaggregation.SRE and FinOps agents use specialized tools to structure traces, metrics, and logs and perform Kubernetes operations; CISO agents can generate policies and evidence-collection scripts, access code repositories, and deploy and execute policies.Source: §3.3 · pp. 5–6

3.4. Leaderboard

• The leaderboard provides predefined, extensible performance metrics and scores partially correct solutions by comparing final system states with scenario ground truth.Eleven of the 94 scenarios are open-sourced with baseline agents; the remaining scenarios are reserved for evaluating submitted solutions.Source: §3.4 · p. 6

4.1. Evaluation Setup

• Baseline evaluations span models for natural-language reasoning and code generation, and the findings show that open-source and proprietary models often struggle with real-world tasks.All models use 128K-token context windows; experiments primarily run on AWS EC2 m4.xlarge instances, though the benchmark can also run on a consumer-grade laptop pseudo-cluster.Source: §4.1 · p. 6

4.2. Overall Results

• 13.81% diagnosis pass@1 and 11.43% mitigation pass@1 made GPT-4o the top SRE model, with the highest NTAM FL and FPC scores.Without traces, GPT-4o’s diagnosis pass@1 fell to 9.52% and mitigation pass@1 to 2.86%.Source: Table 4 · p. 7, §4.2 · p. 7
• GPT-based models dominate CISO pass@1 and time-to-process metrics, with pass@1 nearly 2x that of the second-best models.The second-best model alternates between Llama-3.1-405b-instruct and Mistral-Large-2 across scenario classes.Source: §4.2 · p. 7, Table 5 · p. 8
• 33% pass rate made GPT-4o the top FinOps model for diagnosing cost increases, while no model achieved optimal CPU and memory cost or high CPU efficiency.Performance across the additional cost and workload-efficiency metrics was comparable across models.Source: §4.2 · p. 7, Table 6 · p. 8

4.3. Impact of Scenario Complexity

• SRE success declines with complexity: GPT-4o diagnosed 36%, 7.73%, and 5.0% of Easy, Medium, and Hard scenarios, and mitigated 21%, 12.27%, and 0.0%, respectively.No model mitigated any Hard SRE scenario; CISO results also declined from Easy to Hard, while neither model diagnosed (except GPT-4o) nor mitigated FinOps’s Hard scenario.Source: §4.3 · p. 8

4.4. Inherent Non-determinism in the Environment

• GPT-4o’s SRE outcomes varied across repeated runs despite consistency-oriented tuning, and similar non-determinism appeared in CISO and FinOps.Minor real-time telemetry changes can alter token generation; tracking behavior across runs reveals agent robustness and reliability.Source: §4.4 · p. 8

• ITBench provides a reproducible framework and experimental platform for benchmarking AI agents on complex, diverse IT automation tasks.Its reproducibility is intended to support a community-driven effort despite the nondeterminism of large-scale IT systems.Source: §5 · p. 9
• The authors plan to expand ITBench with additional processes, broader scenario coverage, and scenarios beyond event-triggered tasks.They invite the community to contribute real-world-inspired incidents reproduced in synthetic, sandboxed environments.Source: §5 · p. 9
• Open questions include specializing agents for IT tasks, coordinating agents with different expertise, ensuring safety, and using human-in-the-loop approaches.The authors invite the community to help address these questions and advance AI-agent automation of critical IT tasks.Source: §5 · p. 9

6.1. Ethics & Broader Impacts

• ITBench agents pose code-execution, malicious-evaluation, and real-world deployment risks; safeguards include containerization, official-channel guidance, and research into constraints on agent actions.The framework also provides a self-contained Kubernetes environment and invites community improvements to its codebase.Source: §6.1 · p. 9

6.2. Reproducibility

• The project open-sources its code and resources, documents how to run and modify them, and provides scripts and an online forum to support reproducing results.The available materials include the interactive pipeline, context management, command implementations, and interface design.Source: §6.2 · pp. 9–10

Appendix; A. Related Work

• The appendix adds analyses of ITBench, its SRE, ComplianceOps, and FinOps personas, and agent performance; related work surveys AI/ML for IT automation.Source: Appendix · p. 16, A. Related Work · p. 16

A.1. Site Reliability Engineering

• Prior SRE work spans incident detection, diagnosis, and mitigation, but reported agent success varies and existing benchmarks have limited scope, motivating standardized, open evaluation.Existing efforts include SRE-only incident scenarios, synthetic ticket analysis, and synthetic anomaly-detection datasets.Source: §A.1 · pp. 16–17

A.2. Compliance

• Compliance automation research maps requirements to control frameworks, while ITBench bridges compliance as code to policy as code; the passage reports no prior compliance-automation benchmark.Source: §A.2 · p. 17

A.3. FinOps

• Prior FinOps work applies machine learning, constrained reinforcement learning, forecasting, and optimization to cloud costs, but standardized benchmarks for forecasting, anomaly detection, and cost optimization are absent.The section situates FinOps alongside IT financial management, technology business management, and portfolio business management.Source: §A.3 · pp. 17–18

B. ITBench; B.1. Benchmark Registration; Scenario Development and Registration

• Benchmark Submitters register realistic scenarios with environment and metadata specifications, then define scenario-specific tasks and evaluation metrics; ITBench plans to add tasks and domains.Source: B. ITBench · p. 18, §B.1 · p. 18, Scenario Development and Registration · p. 18

B.2. Agent Registration; B.2.1. AGENT REGISTRATION

• Agent Submitters register agent metadata, after which ITBench retrieves matching benchmarks using agent type, level, and scenario class and supplies tasks with predefined evaluation metrics.The phase begins with the submitter registering as a platform user.Source: §B.2 · p. 18, §B.2.1 · p. 18

B.3. Leaderboard

• ITBench’s leaderboard offers an extensible set of performance metrics and API and UI workflows for registering agents, deploying scenarios, and publishing status updates.Scenarios can be deployed automatically in ITBench’s hosted environment or manually in a user-hosted environment while using the same Leaderboard API.Source: §B.3 · p. 19

C.1. Background

• ITBench currently focuses on diagnosis and mitigation, and some scenarios inject multiple faults so agents must prioritize alerts amid a flood.The framework plans to add tasks such as incident detection and preventing recurrence.Source: §C.1 · pp. 20–21

C.2. Real-world Incident Example

• A real-world incident example traces accidental resource deletion through shard unavailability and Elasticsearch failure to an error-rate SLO violation.The incident report highlights detection, diagnosis, and mitigation timing; customer impact; fault propagation; resolution plans; and longer-term fixes as useful information for incident automation.Source: §C.2 · pp. 21–22

C.3. ITBench Architecture

• ITBench uses open-source technologies to create repeatable Kubernetes incidents, orchestrating agent enrollment, scenario setup and fault injection, task execution, evaluation, and cleanup.Agents can use NL2Alerts, NL2Logs, NL2Metrics, and NL2Traces during active scenarios.Source: §C.3 · p. 22

C.3.1. PRINCIPLES

• ITBench’s principles include SRE best practices, real-incident patterns, observability, variable data modalities, scalable scenario parameters, and deterministic readiness checks.Detection is provided out of the box; the team examined 105 incidents and reports a mix of 24% Easy, 24% Medium, and 52% Hard incidents. Grafana is the default observability tool, with support for other platforms and configurable telemetry availability.Source: §C.3.1 · p. 22

C.3.2. RECREATING INCIDENTS IN ITBENCH USING REAL-WORLD SCENARIOS

• The recreated Elasticsearch scenarios reproduce production-like disruptions, failure propagation, metric impacts, and mitigation decisions based on real incident reports.Examples include network changes, node or volume deletion, and disabling shard writes in a multi-node cluster with EBS volumes.Source: §C.3.2 · pp. 22, 24, Table 11 · p. 25

C.4. Characterizing ITBench incidents

• ITBench defines incident complexity as the geometric mean of fault-propagation length, resolution steps, and technologies involved.The measure gives the three incident-inherent factors proportional influence rather than allowing one to dominate; skill requirements and diagnostic interactions are excluded because they vary with platforms and teams.Source: §C.4 · pp. 24–26, Figure 8 · p. 26
• Figure 9 shows a diverse distribution of incident complexity, reflecting variation in propagation depth, resolution steps, and multi-technology integration.The distribution uses a geometric-mean complexity formulation and provides a basis for future analyses of how tools, observability, or operator training might change scenario difficulty.Source: Figure 9 · p. 26, §C.4 · p. 26

C.4.1. EXPERIMENTAL SETUP

• Fault-injection tasks are implemented as configurable Ansible playbooks, with Kubernetes network policies used to block service ports and Grafana alerts validated for each scenario.Roles support fault injection and removal, and variables allow targeting different services.Source: §C.4.1 · p. 26
• The AWS experiments used a cluster with one control-plane node and three workers, and measured PVC usage, CPU consumption, and memory utilization during a one-hour test.The resource measurements are summarized in Table 13; the supplied passages do not give its cell values.Source: §C.4.1 · p. 27, Table 13 · p. 27
• Kind clusters provide a lightweight, portable setup for reproducible fault-scenario testing; Incident 22 averaged 361.71% CPU usage and 93.53% memory consumption.The validation machine had one control-plane node, 12 CPU cores, and 16 GB RAM.Source: §C.4.1 · p. 27

C.5.1. OBSERVABILITY DATA

• SRE tasks analyze logs, traces, and metrics to diagnose and mitigate incidents.Logs capture runtime events, traces represent service-invocation paths and exceptions, and metrics track performance indicators such as latency, errors, and resource use.Source: §C.5.1 · p. 27, Figure 10 · p. 27, §C.5 · p. 27

C.5.2. SRE-AGENT ARCHITECTURE AND IMPLEMENTATION

• SRE-Agent tools are LM-based and include reflection with linter or execution-error feedback, while summarizers condense long outputs from tools such as NL2Logs and NL2Traces.Tools can generate natural language or function calls.Source: §C.5.2 · p. 28, Table 14 · p. 28
• SRE-Agent uses separate diagnosis and mitigation agents in sequence, matching the requirement to diagnose an incident before resolving it.Diagnosis targets the fault-propagation chain and exact fault location; mitigation produces and executes a plan to clear the triggering alert.Source: §C.5.2 · pp. 28–29
• The Diagnosis Agent retrieves active alerts, iteratively gathers observability data and environment information, then emits a structured JSON diagnosis for the Mitigation Agent.The Mitigation Agent implements its plan with NL2Kubectl, checks alerts, and can wait to verify that they remain cleared before reporting its actions in JSON.Source: §C.5.2 · p. 29

C.6.1. EXPERIMENTAL DETAILS

• The SRE-Agent was evaluated on 42 SRE scenarios using gpt-4o, granite-3.1-8b-instruct, llama-3.3-70b-instruct, and llama-3.1-8b-instruct, none fine-tuned.These models served as the agent’s LM-based planning component.Source: §C.6.1 · p. 29
• Hyperparameters were chosen to make results as deterministic as possible, and decoding_method applied to all models except gpt-4o.Table 15 lists the model hyperparameters, but their values are not included in the supplied passages.Source: §C.6.1 · p. 29, Table 15 · p. 29

C.6.2. EVALUATION METRICS

• Diagnosis assesses fault localization and propagation chains: localization uses pass@1 and NTAM, propagation uses NTAM, and MTTD tracks diagnostic efficiency.Localization identifies the resource causing a problem; propagation chains show how faults cascade across the application stack and can support blast-radius analysis.Source: §C.6.2 · p. 29
• Mitigation evaluates whether agents clear incident alerts, measuring success with pass@1 and efficiency with MTTR.“%Agent submission” is the percentage of completed trials in which the agent returned results.Source: §C.6.2 · pp. 29–30
• ITBench cannot yet automatically evaluate unstructured fault-condition descriptions, but plans to extend this task using LM-as-a-judge (Zheng et al., 2023).The unstructured output describes what is wrong with the identified resource.Source: §C.6.2 · p. 30

C.6.3. METRIC DEFINITIONS

• pass@1 is an unbiased estimator of correctness in k=1 trials across scenarios; correctness means an exact root-cause match for localization or cleared alerts for mitigation.Both fault localization and mitigation use pass@1.Source: §C.6.3 · p. 30
• NTAM addresses exact-match metrics’ inability to distinguish near-miss predictions by measuring agent performance against ground truth with topology-aware distance.It evaluates both fault propagation chains and fault localization.Source: §C.6.3 · p. 30
• NTAM uses a system topology graph and scores entities using edge types and subtree size, rewarding closer predictions and more discriminative nodes.A node-importance factor rewards correctly predicting the root cause over another chain entity, and length-mismatch penalties lower scores for predictions with too many or too few entities.Source: §C.6.3 · p. 30
• NTAM components have tunable hyperparameters, and its score is normalized from 0 to 1; for localization, it considers only predicted and ground-truth root causes.A score of 1 indicates a perfect match.Source: §C.6.3 · p. 30
• MTTD and MTTR measure diagnosis and mitigation time in seconds only when the agent succeeds; otherwise, each metric is set to infinity.MTTD requires finding root-cause entities, while MTTR requires executing an automated action that clears alerts.Source: §C.6.3 · p. 30

C.6.4. EVALUATION RESULTS

• gpt-4o leads the four-agent SRE evaluation, reaching 13.81% diagnosis pass@1 and 11.43% mitigation pass@1; no agent repaired any hard scenario.llama-3.1-8B had the fastest detection (MTTD 57.50s) and repair (MTTR 245.13 seconds) among successful attempts, while llama-3.3-70B ranked second overall.Source: §C.6.4 · p. 31
• SRE performance declines from Easy to Hard: no agent resolved any of five hard scenarios, while at least one agent repaired five of eight easy scenarios.Six easy scenarios were diagnosed correctly; the authors report comparing task-level success rates using difference-of-proportions z-tests.Source: §C.6.4 · p. 31
• Diagnosis and mitigation can diverge: an agent resolved Scenario 15 despite a wrong root cause, while Scenario 13 resisted repair despite strong diagnosis and Scenario 19 defeated both.In Scenario 13, gpt-4o achieved roughly 80% diagnosis pass@1 and over 0.7 on both FPC and RC NTAM metrics.Source: §C.6.4 · p. 31
• Telemetry masking and repeated runs affect reliability: gpt-4o diagnosis pass@1 fell from 18.10% with traces to 9.52% without, and its mitigation pass@1 fell to 2.86%.Only three scenarios remained successfully resolved by gpt-4o with traces masked; minor telemetry differences also produced variable outcomes, including 100–800 second run times.Source: §C.6.4 · pp. 31–32
• ITBench supports new scenarios and tasks, integrates four applications, and lets collaborators add an application with an Ansible playbook in around four human hours.Its faults span system layers, including application and virtualization layers.Source: §C.6.4 · p. 32

C.6.5. DIAGNOSIS AND MITIGATION REPORT

• The SRE agent’s JSON report lists encountered faulty entities, its best-guess incident cause, and mitigation actions; evaluation covers 21 scenarios with 10 runs per scenario, under tracing-enabled and tracing-disabled conditions.Source: §C.6.5 · pp. 32, 38, Table 19 · p. 38, Table 20 · p. 38

D.2. Real-World Benchmarking

• The CISO benchmark evaluates an end-to-end agent workflow that discovers the assessment engine, generates policy-as-code, manages a Git PR, deploys and executes the code, and reports posture.Its scenario model defines agents, tasks, scenarios, classes, environments, states, and goal states as agents move environments toward desired states.Source: §D.2 · p. 39, §D.2.1 · pp. 39–40

D.2.2. CISO COMPLIANCE ASSESSMENT TASKS

• CISO tasks include collecting evidence and scanning assessment posture; ITBench’s first release supports these tasks, checked against expected pass/fail outcomes.The framework uses reproducible Kubernetes scenarios and deployable stacks for environment preparation, fault injection, evaluation, and cleanup; it draws on CIS benchmarks, OSCAL-based observability, and deterministic cleanup.Source: §D.2.2 · p. 40, §D.3.1 · p. 41, §D.3.2 · p. 41

D.3.2. ITBENCH ARCHITECTURE

• CISO runs present agents with compliance requirements, let them gather evidence and deploy policies in an accessible scenario environment, then aggregate measured metrics on a leaderboard.Mini-Bench automates setup and measurement, while a Bench Server API supports local or remote environments.Source: §D.3.2 · pp. 41–42

D.4. ITBench Real-World CISO Scenarios

• ITBench’s CISO scenarios use CIS Benchmarks for Kubernetes and RHEL9, with each scenario designed to mirror the complexity of its compliance recommendation.The framework can also use other technologies covered by CIS Benchmarks.Source: §D.4 · p. 42

D.5.1. NEW-K8S-CIS-B-KYVERNO

• New-K8s-CIS-b-Kyverno defines 10 easy Kubernetes Pod Security Policy scenarios in a preconfigured cluster, where agents generate and deploy Kyverno policies to identify unknown misconfigurations.Evaluation checks whether the policy reflects the required configuration, whether a policy and report are generated in the required locations, and whether the report matches the expected posture.Source: §D.5.1 · p. 42

D.5.2. NEW-K8S-CIS-B-OPAREGO

• New-K8s-CIS-b-Kubectl-OPARego comprises 10 medium-complexity Kubernetes Pod Security Policy scenarios requiring a kubectl evidence-fetcher script and an OPA Rego checker.The fetcher runs against a real cluster; evaluation checks generation, evidence collection, and whether the checker produces the expected compliance assessment.Source: §D.5.2 · pp. 42–43

D.5.3. NEW-RHEL9-CIS-B-ANSIBLE-OPA

• New-RHEL9-CIS-b-Ansible-OPA targets RHEL9 hosts with Ansible playbooks as evidence fetchers and OPA Rego policies as checkers.The playbook gathers evidence from a real RHEL9 host, which the checker evaluates against expected compliance outcomes.Source: §D.5.3 · pp. 43–44

D.5.4. UPDATE-K8S-CIS-B-KYVERNO

• Update-K8s-CIS-b-Kyverno covers 10 hard scenarios in which agents modify an existing Kyverno policy to meet revised requirements and deploy the updated policy.Its validation follows the methodology used for New-K8s-CIS-b-Kyverno.Source: §D.5.4 · p. 44

D.6. CISO ITBench Evaluation

• The evaluation measures CISO agents across 50 scenarios in four classes that vary in CIS-benchmark requirements, complexity, and scenario-specific code artifacts.The experiments primarily use AWS EC2 m4.xlarge instances, though ITBench can also run on a consumer-grade laptop using a pseudo-cluster.Source: §D.6 · p. 44
• The evaluation considers natural-language reasoning and code-focused LLMs, with all models using 128K-token context windows.The evaluated models include GPT-4o, Llama-3.3-70B-instruct, Llama-3.1-8B-instruct, Granite-3.1-8B-instruct, GPT-4o-mini, Llama-3.1-405b-instruct, and Mixtral-8x7b-instruct.Source: §D.6 · p. 44

D.6.1. EVALUATION METRICS

• The agent is evaluated on detecting artifact misconfigurations or confirming proper configurations across scenario-class environments.Examples of non-compliance include lacking a minimum count of containers sharing a namespace or exceeding the threshold.Source: §D.6.1 · p. 44

D.6.2. METRIC DEFINITIONS

• Success rate is measured with pass@1, which evaluates correct pass/fail posture assessments and is an unbiased correctness estimator across scenarios in k=1 trials.The metric is attributed to Chen et al. (2021).Source: §D.6.2 · p. 46, §D.6.1 · p. 46
• Efficiency is measured by Time to Process (TTP), the time taken for a successful posture assessment; unsuccessful scenarios receive infinite TTP.A successful assessment labels a misconfiguration “fail” and a compliant configuration “pass.”Source: §D.6.2 · p. 46, §D.6.1 · p. 46

D.6.3. EVALUATION RESULTS

• GPT-based models dominate on both pass@1 and Time to Process metrics across the scenario classes.The reported results are in Table 5 and Figure 25.Source: §D.6.3 · p. 46
• Pass@1 is nearly 2x better for GPT-based models than for the second-best models, which alternate between llama-3.1-405b-instruct and mistral-large-2.Source: §D.6.3 · p. 46
• GPT-based models handle scenarios in minimal Time to Process across the scenario classes.Source: §D.6.3 · p. 46

Impact of Scenario Complexity

• Kyverno scenarios are classified as Easy complexity.Source: Impact of Scenario Complexity · p. 46
• The k8s-opa and rhel-opa scenarios are classified as Medium complexity.Source: Impact of Scenario Complexity · p. 46
• Kyverno-update scenarios are classified as Hard, and models struggle more as difficulty rises from Easy Kyverno to Hard Kyverno-update.Source: Impact of Scenario Complexity · p. 46

D.6.4. CISO COMPLIANCE ASSESSMENT AGENT TRAJECTORIES

• The two LLaMa 405B examples cover RHEL9 CIS controls 5.1.8 and 5.2.19, addressing cron access and an SSH LoginGraceTime of one minute or less.Both tasks require collecting host configuration with an Ansible playbook and checking compliance with an OPA Rego policy.Source: §D.6.4 · p. 46
• The successful trajectory generated and independently tested a playbook, collected host data, created an OPA policy, and detected issues during benchmark evaluation.The task concerned restricting cron to authorized users, including requiring /etc/cron.allow and prohibiting /etc/cron.deny.Source: §D.6.4 · p. 46
• In the failed trajectory, playbook generation and data collection succeeded, but invalid Rego syntax triggered repeated OPA errors until retries were exhausted and evaluation failed.After three repeats, evaluation used the saved playbook.yml and policy.rego, and the policy did not pass.Source: §D.6.4 · pp. 46–47

E.1. Background

• FinOps aims to maximize cloud business value through shared financial accountability, using iterative, real-time insights from application performance, resource management, and finance.The background describes Inform, Optimize, and Operate as its three phases and notes priorities including reducing waste, managing commitments, and forecasting spend.Source: §E.1 · p. 47

E.1.1. KEY TERMS

• FinOps’ Inform phase develops data for budgeting and analysis, Optimize identifies efficiency opportunities, and Operate implements FinOps using capabilities from the earlier phases.The phases work iteratively on framework capabilities.Source: §E.1.1 · p. 48
• The KPI catalog spans utilization, spend variance, autoscaling, and usage or spend forecasting; current scenarios use spend-variance alerts, while other KPIs offer bases for additional scenarios.Other listed measures include commitment discounts and unused resources.Source: §E.1.1 · pp. 48–49

E.2. Motivating Example and FinOps Tasks

• One budget alert followed a demand-driven replica increase while the application remained healthy; another reflected low-utilization scale-up caused by faulty autoscaler thresholds.The agent recommends updating budget alerts in the first scenario and adjusting autoscaler configuration in the second.Source: §E.2 · p. 49, Table 25 · p. 51, Table 26 · p. 51

E.3. ITBench Architecture for Constructing FinOps Task Scenarios

• FinOps scenarios reuse the SRE setup and use OpenCost to monitor costs and raise alerts when budget or efficiency thresholds are crossed.The experiments use custom hourly CPU and memory pricing; networking and spot-instance costs are outside the main scope and are set to negligible values.Source: §E.3 · pp. 51–52

E.4. Evaluation

• The evaluation reuses the SRE-bench framework with OpenCost and cost alerts in the scenarios.Source: §E.4 · p. 52

E.4.1. EVALUATION METRICS

• Agents are evaluated on budget-alert diagnosis and mitigation: diagnosis uses pass@1 for root-cause accuracy, while mitigation uses pass@1 success rate and proximity scores for cost and efficiency.The described evaluations could be expanded to measure performance automatically with an LLM-as-a-judge.Source: §E.4.1 · p. 52

E.4.2. METRIC DEFINITIONS

• Proximity scores cover hourly CPU cost, hourly memory cost, workload CPU efficiency, and workload memory efficiency, with a score of 1 indicating optimal performance.Each score is 1 minus the proportional absolute difference between the observed and ground-truth optimal value.Source: §E.4.2 · p. 52
• The metric definitions use custom hourly CPU and memory costs and average container utilization divided by requests for workload CPU and memory efficiency.The paper takes the arithmetic mean across runs for each metric, calculates standard deviation, and presents results in Table 6.Source: §E.4.2 · p. 52

E.5. Example Trajectories

• FinOps diagnosis must pair infrastructure and deployment checks with load and utilization analysis; in Scenario 37, skipping utilization checks produced an erroneous scale-down recommendation despite high application load.A correct trajectory would assess whether budget thresholds can be updated within the total allocation or identify deployments that can be scaled down without harming performance.Source: §E.5 · pp. 52–53, Figure 26 · p. 53