PaperlayerView exact revision on arXiv

Source-linked AI summary

DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle

Yuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, Giovanni Vigna, Dawn Song, William Yang Wang, Lun Wang, Yangruibo Ding, Zhenkai Liang, Wenbo Guo

arXiv:2601.20882v1cs.SEcs.AIcs.CR

TL;DR

Existing benchmarks leave agent capabilities across realistic DevOps workflows unclear, while operational stages still demand substantial manual intervention. DEVOPS-GYM evaluates agents across four workflows using real-world repositories and DevOps tools; its results show low success rates across stages.

  • Problem

    Existing benchmarks focus on isolated tasks and lack dynamic execution environments and tool-calling interfaces for evaluating agents across realistic DevOps workflows.

    Source: §1 · p. 2

  • Method

    DEVOPS-GYM evaluates agents across four core DevOps workflows using real-world repositories, DevOps tools, and a dynamic tool-calling environment.

    Source: §1 · p. 2

  • Results

    Top-performing agent success rates were 51.85% on build and configuration, 20.56% on monitoring, 23.87% on issue resolving, and 13.87% on test generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results highlight the need for research on automating the full DevOps cycle with AI agents.

    Source: §1 · p. 2

Abstract

from arXiv · show

Even though demonstrating extraordinary capabilities in code generation and software issue resolving, AI agents' capabilities in the full software DevOps cycle are still unknown. Different from pure code generation, handling the DevOps cycle in real-world software, including developing, deploying, and managing, requires analyzing large-scale projects, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions. However, existing benchmarks focus on isolated problems and lack environments and tool interfaces for DevOps. We introduce DevOps-Gym, the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation. DevOps-Gym includes 700+ real-world tasks collected from 30+ projects in Java and Go. We develop a semi-automated data collection mechanism with rigorous and non-trivial expert efforts in ensuring the task coverage and quality. Our evaluation of state-of-the-art models and agents reveals fundamental limitations: they struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration. These results highlight the need for essential research in automating the full DevOps cycle with AI agents.

1 INTRODUCTION

Existing benchmarks leave a gap in evaluating agents across realistic DevOps workflows, which involve dynamic system behavior, domain-specific tools, and multi-step decisions. DevOps-Gym addresses this gap, while evaluation reveals weak performance across workflows and difficulty with planning and tool use.

1 INTRODUCTION

  • Existing benchmarks largely target isolated development tasks, while realistic end-to-end agent evaluation lacks dynamic environments and tool-calling interfaces.Operational stages such as building, deployment, and monitoring still demand substantial manual intervention.Source: §1 · pp. 1–2
  • DEVOPS-GYM evaluates four core DevOps workflows using real-world repositories, DevOps tools, multi-step tasks, and a dynamic tool-calling environment.The benchmark includes 704 tasks from 30+ projects, with expert effort devoted to reproducing tasks and filtering contamination.Source: §1 · p. 2
  • Top-performing agent success rates were 51.85% on build and configuration, 20.56% on monitoring, 23.87% on issue resolving, and 13.87% on test generation.The evaluation covered three widely used coding agents across five LLMs and four agentic frameworks.Source: §1 · p. 2
  • Agents frequently fail at high-level planning and use DevOps-specific tools incorrectly, especially in build and monitoring workflows.Agents also struggle to process dynamic information such as program states, runtime logs, and system status.Source: §1 · p. 2
  • State-of-the-art agents that perform strongly on Python benchmarks drop significantly on Java and Go issue-resolving and test-generation tasks.The authors suggest possible data contamination and note the compilation, dependency, and syntax complexity of non-script languages.Source: §1 · p. 2

2 RELATED WORK

Prior benchmarks evaluate code generation and individual DevOps-related tasks, but coverage of end-to-end, agentic workflows in realistic environments remains limited. DEVOPS-GYM addresses this gap with four stages and distinct command-line tools.

2 RELATED WORK

  • Existing code benchmarks range from function-level generation to repository-level tasks, but still focus mainly on development rather than a holistic DevOps cycle.SWE-bench targets Python issue resolving, while other benchmarks extend repository-level and task-specific evaluations.Source: §2 · pp. 2–3
  • DevOps-related benchmarks separately address test generation, build configuration, or monitoring, often in Python or narrow infrastructure settings.DEVOPS-GYM’s four core stages are build and configuration, monitoring, issue resolving, and test generation, each using distinct command-line tools.Source: §2 · p. 3, Figure 1 · p. 3
  • Existing benchmarks lack end-to-end DevOps coverage on real repositories, agentic multi-step workflows, and dynamic execution environments with tool-calling interfaces.DEVOPS-GYM is constructed to evaluate real-world tasks with metrics, tool-call support, and execution environments.Source: §2 · p. 3

3 CONSTRUCTION OF DEVOPS-GYM

DEVOPS-GYM combines realistic Java and Go tasks, tool-based agentic workflows, and coverage of four DevOps stages. Its construction includes expert-driven task reproduction and evaluation across isolated tasks and sequential end-to-end pipelines.

3.1 OVERVIEW

  • DEVOPS-GYM targets Java and Go and covers build/configuration (54 tasks), monitoring (34), issue resolving (310), test generation (310), and sequential end-to-end workflows (18), across 30+ repositories.The researchers manually analyzed over 1,000 GitHub issues to categorize tasks for each stage.Source: §3.1 · p. 4
  • Construction addresses contamination, task reproduction, and rigorous evaluation through repository filtering, environment reconstruction, and standardized tool interfaces.Reproducing a monitoring or building issue can take one expert more than 10 hours; evaluation and engineering also require extensive manual effort.Source: §3.1 · pp. 4–5

3.2 TASK CONSTRUCTION FOR BUILD AND CONFIGURATION

  • Build tasks include repairs across five failure types and implementation scenarios such as framework migration, target release, plugin integration, and dependency upgrades.Repair tasks mine CI failure-success pairs; approximately 40% of initially selected issues required multiple iterations for reproducibility.Source: §3.2 · p. 5

3.3 TASK CONSTRUCTION FOR MONITORING

  • Monitoring tasks target subtle runtime performance and resource anomalies, including healthy cases where agents must avoid false-positive diagnoses.Tasks require capturing runtime and system states with command-line tools; anomalies include gradual resource degradation and performance problems that do not cause immediate failure.Source: §3.3 · p. 6

3.4 TASK CONSTRUCTION FOR ISSUE RESOLVING AND TEST GENERATION

  • Issue-resolving patches must pass the provided tests, while generated tests must fail on buggy code and pass on the patched code.The collection adapts the Multi-SWE-bench pipeline for Java and Go issues with test-validated fixes.Source: §3.4 · p. 7

3.5 END-TO-END TASKS

  • End-to-end tasks chain build, monitoring, issue resolving, and validation, requiring agents to propagate diagnostic context and complete every stage sequentially.Failure at any stage terminates the pipeline, so partial solutions do not count as success.Source: §3.5 · p. 7

4 EVALUATION

The evaluation compares advanced agent frameworks and language models on DevOps-Gym, finding uneven results across tasks and no successful completion of the full end-to-end workflow.

4.1 EXPERIMENTAL SETUP

  • The evaluation pairs OpenHands with five LLMs and compares three frameworks using Claude-4-Sonnet, focusing initially on advanced systems as an upper bound.Source: §4.1 · p. 8

4.2 RESULTS

  • Claude-4-Sonnet outperforms other state-of-the-art models by more than 20%, while Claude Code beats mini-SWE-Agent with the same model.Build failures also reflect toolchain and environment limitations, sequential-planning failures, and domain-specific knowledge gaps; 17% of all failures are categorized as inherently difficult.Source: §4.2 · pp. 8–9
  • Monitoring remains exceptionally weak, with state-of-the-art models reporting 0%; agents struggle with continuous system state, sustained focus, and distinguishing anomalies from normal variation.Common monitoring errors include inadequate methodology (37%), premature conclusions (26%), insufficient temporal granularity (11%), and interpretation failures (26%).Source: §4.2 · p. 9
  • OpenHands + Claude-4-Sonnet resolves 23.87% of Java and Go issues versus 70.4% on SWE-Bench-Verified, and newer models still struggle with non-Python ecosystems.The authors identify compilation, dependency management, and environment configuration as challenges for Python-centric agents.Source: §4.2 · pp. 9–10
  • Generating high-quality tests is harder than issue resolving because tests must reproduce the reported failure and validate the patch through dynamic reasoning.DevOps-Gym uses a strict test metric requiring the generated test to reproduce the issue failure and then pass on the patched code.Source: §4.2 · p. 10
  • Claude Code achieves the best overall performance, while Claude-4-Sonnet leads across most tasks and Claude Code reaches 58.33% on build and configuration.Source: §4.2 · p. 10
  • Agents achieve 0% success on end-to-end tasks requiring all four stages, with failures linked to difficulty completing monitoring and chaining stages.The paper attributes the overall workflow gap to missing context propagation and multi-stage planning capabilities.Source: §4.2 · pp. 10–11

4.3 STABILITY OF AGENT PERFORMANCE ACROSS MULTIPLE INDEPENDENT RUNS

  • Across five runs on 50 sampled tasks, Claude Code’s mean accuracy is 18.8% and OpenHands’ is 16.8%, with relatively small run-to-run variance.Aggregating five trajectories raises success rates to 26% for Claude Code and 20% for OpenHands without materially changing their relative ranking or the study’s conclusions.Source: Table 2 · p. 11, §4.3 · p. 11

5 CONCLUSION AND FUTURE WORKS

DevOps-Gym evaluates four core DevOps stages and finds substantial limitations in current agentic systems, particularly on monitoring and build and configuration tasks. The authors identify scope expansion and broader task coverage as future directions.

5 CONCLUSION AND FUTURE WORKS

  • DevOps-Gym evaluates build and configuration, monitoring, issue resolving, and test generation, and current agents perform particularly poorly on monitoring and build and configuration.Source: §5 · p. 11
  • CI/CD automation and infrastructure management remain outside the benchmark because mutable external systems make task creation and evaluation difficult.Source: §5 · p. 11
  • Future work includes adding tasks, error types, and projects, extending to other programming languages and DevOps stages, and enriching the benchmark with scenarios and metrics.Source: §5 · p. 11

B DATASET CONTAMINATION PREVENTION

The benchmark screens repositories for possible training-data contamination using sampled code continuations, similarity metrics, and repository-history isolation.

B DATASET CONTAMINATION PREVENTION

  • For each repository, 20 code snippets from CI/CD configurations, build files, and tests are sampled for prefix-completion contamination checks.A model continues a 50-token prefix, and its prediction is compared with the original suffix.Source: B DATASET CONTAMINATION PREVENTION · p. 15
  • Five similarity metrics flag a snippet as high-risk when any exceeds its threshold; repository contamination rate C is the high-risk share of 20 samples.Thresholds are s > 0.7, t > 30, p > 30, l > 30, or b > 0.5.Source: B DATASET CONTAMINATION PREVENTION · p. 15
  • Repositories with C ≥ 0.2 are excluded, retaining LOW or LOW-MODERATE contamination repositories.The threshold aims to balance contamination control with dataset scale; even the cleanest repositories show 10% contamination, while the threshold leaves 80%+ of snippets without detected memorization.Source: B DATASET CONTAMINATION PREVENTION · pp. 15–16
  • Git metadata is removed so agents cannot query future commits or branches containing fixes.Agents receive the codebase state from issue creation.Source: B DATASET CONTAMINATION PREVENTION · p. 16

C.1 ISSUE RESOLVING

The benchmark evaluates issue fixing and regression-test generation as distinct DevOps tasks, with tests generated without access to the proposed fix.

C.1 ISSUE RESOLVING

  • Issue resolving asks agents to turn bug descriptions into minimal code fixes while preserving normal functionality, regardless of how the bug was discovered.Inputs include a buggy repository, natural-language description, development tools, and language-specific test runners; the output is a unified-diff patch.Source: §C.1 · p. 16
  • Issue-resolution tasks use merged pull-request fixes or expert patches, with success requiring a fail-to-pass test transition and no new failures.Collection selects maintained Java and Go projects, filters issue-resolving pull requests with test modifications, and validates test transitions before and after patching.Source: §C.1 · p. 17

C.2 TEST GENERATION

  • Test-generation agents receive only the buggy repository and bug description, not the ground-truth patch, so tests must independently capture the bug.This setting reflects independent validation of patch quality and is especially challenging for performance or resource issues requiring specialized assertions.Source: §C.2 · pp. 17–18
  • A generated test succeeds when at least one test fails on buggy code and passes after patching, with no failures on the patched code.The evaluation reports the share of tasks with valid fail-to-pass tests and no spurious failures.Source: §C.2 · p. 18
  • Each issue-resolution task also yields a corresponding test-generation task, using developer-provided tests when available or expert-validated regression tests.The generated test is delivered as a diff that captures the bug’s behavior.Source: §C.2 · p. 18

D.1 BUILD

The build example concerns migrating a Maven project to Gradle while maintaining functional equivalence.

D.1 BUILD

  • The agent fails to properly handle test-execution failures while still achieving the core migration objectives.Source: §D.1 · p. 18
  • The task asks for migration of the MobArena plugin build system from Maven to Gradle while maintaining functional equivalence.The environment includes build artifacts and logs from the Maven build.Source: §D.1 · p. 18

A Maven project at /app/mobarena/ - this is your starting point

The example shows a Maven-to-Gradle migration task with explicit build, test, and artifact-equivalence criteria, alongside failures in the agent’s verification.

A Maven project at /app/mobarena/ - this is your starting point

  • The migration task requires a complete Gradle configuration, Maven-equivalent functionality, a successful build, passing tests, and equivalent artifacts.The requested checks include ./gradlew clean build and ./gradlew test.Source: A Maven project at /app/mobarena/ -… · p. 18
  • The agent began by examining project structure and Maven configuration, then ran the Gradle tests for verification.The transcript includes a query about the current configuration and a test command using ./gradlew test.Source: A Maven project at /app/mobarena/ -… · p. 19
  • The Gradle test run completed 253 tests, with 88 failing.The agent attributed failures to missing runtime dependencies and mocking issues.Source: A Maven project at /app/mobarena/ -… · p. 19
  • The agent migrated the build system and produced correct artifacts, but did not resolve all test failures despite that being a success criterion.The account says the successful build and JARs met the primary migration goals, but not the full test criterion.Source: A Maven project at /app/mobarena/ -… · p. 19
  • In a separate Caddy GoReleaser case, the agent failed to include Caddy-specific build tags in the configuration.Source: A Maven project at /app/mobarena/ -… · p. 19

A Go project at /app/caddy-project - this is your starting point

The task asks an agent to configure GoReleaser for Caddy releases across platforms, but the attempted configuration is functionally incomplete because it omits Caddy-specific build tags.

A Go project at /app/caddy-project - this is your starting point

  • The task requests cross-platform builds for Linux, Windows, macOS, and FreeBSD, plus cross-compilation across six architectures with platform exclusions.It also specifies archive generation, build tags, and linker flags.Source: A Go project at /app/caddy-project -… · p. 20
  • Success criteria include a successful goreleaser check, a snapshot build, and binaries for Linux, Windows, macOS, and FreeBSD.Generated binaries must preserve original functionality and use GoReleaser archive naming conventions.Source: A Go project at /app/caddy-project -… · p. 20
  • After inspecting the project, the agent concludes that Go modules handle version information, so custom version-injection ldflags are unnecessary.The agent then starts creating a GoReleaser configuration.Source: A Go project at /app/caddy-project -… · pp. 20–21
  • The proposed configuration builds from ./cmd/caddy/main.go and disables CGO to produce static binaries.It also includes build and linker flag sections.Source: A Go project at /app/caddy-project -… · p. 21
  • The GoReleaser configuration validates syntactically but omits Caddy-specific build tags nobadger, nomysql, and nopgx, leaving it functionally incomplete.The comments identify these tags as essential for proper Caddy builds.Source: A Go project at /app/caddy-project -… · p. 22

D.2 MONITORING

The monitoring examples call for observing a running server over time, but the feedback identifies failures to monitor continuously and an incorrect conclusion attributed to limited context.

D.2 MONITORING

  • Case feedback says the agent fails to monitor the system continuously, despite the task requiring observation of system behavior over time.The feedback suggests repeated sampling, such as watching /proc/14/io every second.Source: §D.2 · pp. 22–23
  • The agent proposes periodic CPU, memory, and file-descriptor logging, while the feedback emphasizes continuous monitoring.Source: §D.2 · p. 23
  • The case attributes the agent’s wrong final conclusion to limited context length.Source: §D.2 · p. 23
Loading 2601.20882v1…