PaperlayerView exact revision on arXiv

Source-linked AI summary

SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

Avi Arora, Jinu Jang, Roshanak Zilouchian Moghaddam

arXiv:2507.09063v1cs.SEcs.AIcs.LG

TL;DR

Existing software-agent benchmarks commonly provide pre-installed environments, leaving bootstrap capability under-tested. SetupBench evaluates agents on 93 setup tasks from bare sandboxes; performance remains limited, with recurring tooling, constraint, and persistence failures.

  • Problem

    Many software-agent benchmarks use pre-baked environments, so they do not test the setup and dependency work developers encounter before code can run.

    Source: §1 · pp. 1–2

  • Method

    SetupBench evaluates agents on 93 bare-sandbox tasks involving package installation, dependency conflicts, databases, and services, with deterministic validation commands.

    Source: §1 · p. 2, §2 · p. 2, §2.1 · p. 3

  • Results

    Across five models and 93 tasks, performance remained challenging; Claude 4 achieved 62.4% success, with failures involving implicit tooling, hallucinated constraints, and non-persistent configuration.

    Source: §1 · p. 2, §4 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    SetupBench provides a reproducible yardstick for environment-bootstrap skill and releases prompts, scripts, and evaluation harnesses for community contributions.

    Source: §5 · p. 9

    Agents ran with root privileges and unrestricted outbound networking, unlike permission- or network-restricted CI and production environments.

    Source: §5 · p. 8

Abstract

from arXiv · show

Modern Large Language Model (LLM) agents promise end to end assistance with real-world software tasks, yet existing benchmarks evaluate LLM agents almost exclusively in pre-baked environments where every dependency is pre-installed. To fill this gap, we introduce SetupBench, a 93 instance benchmark that isolates the environment-bootstrap skill: starting from a bare Linux sandbox, an agent must install packages, resolve dependency conflicts, initialize databases, and configure background services. Our tasks span seven language ecosystems, five database engines, and multi-service orchestration scenarios, each accompanies by a natural language problem statement and a deterministic success command. Through evaluation of OpenHands, a state-of-the-art coding agent, we find low success rates across task categories, with particular challenges in repository setup (38.9-57.4%) and local database configuration (20.0-53.3%). Our analysis reveals systematic failure modes including incomplete development tooling installation, hallucinated task constraints, and non-persistent environment modifications that break agent-human collaboration workflows. We identify substantial inefficiencies in agent exploration strategies, with 38-89% of actions being unnecessary compared to optimal human behavior. These findings highlight gaps in current agents' practical environment-bootstrap capabilities. By targeting this critical yet under-evaluated capability, SetupBench provides a rigorous yard-stick for the next generation of software developer agents aiming to solve end to end real-wold tasks.

1 Introduction

Existing benchmarks largely assume dependencies are already installed, leaving environment setup untested; SetupBench evaluates this bootstrap work and OpenHands results reveal low success and recurring failures.

1 Introduction

  • As agents move from code assistance toward autonomous development, people increasingly define requirements, guide work, and validate outcomes.Examples include OpenAI Codex, GitHub Copilot Coding Agent, and Devin.Source: §1 · p. 1
  • Existing agent benchmarks distribute tasks in pre-baked environments with required libraries, services, and configuration files already installed.This leaves environment-bootstrap ability untested, although setup and dependency work is a major source of developer frustration.Source: §1 · p. 1
  • SetupBench introduces 93 tasks starting from bare sandboxes and requiring package installation, database initialization, service configuration, or dependency-conflict resolution.Each task includes a natural-language prompt, workspace snapshot, and deterministic success command.Source: §1 · p. 2
  • OpenHands achieved 34.4-62.4% success across models, with incomplete tooling, hallucinated constraints, non-persistent changes, and 38-69% wasted steps.The reported failures also break agent-human collaboration workflows.Source: §1 · p. 2

2 SetupBench

SetupBench evaluates practical environment-bootstrap tasks from minimal sandboxes using deterministic validation, spanning multiple task types, languages, databases, and service configurations.

2 SetupBench

  • SetupBench contains 93 instances across four practical environment-bootstrap categories, each with a prompt, workspace snapshot, and deterministic validation command.The command reports success or failure based on whether environment changes took effect.Source: §2 · p. 2

2.1 Task construction

  • Repository tasks cover seven languages; dependency-conflict tasks reproduce validated npm and Bundler lock-file conflicts mined from real GitHub issues.Repository setup steps were manually documented and validated in fresh sandboxes; the dependency set contains nine npm and seven Bundler cases.Source: §2.1 · pp. 2–3
  • Database tasks span five engines and three difficulty tiers, from installation and seeding to configuration, migrations, and troubleshooting deliberate obstacles.Orchestration tasks coordinate long-running services through supervisord and validate observable effects such as HTTP responses, Redis keys, or logs.Source: §2.1 · p. 3

2.2 Metrics & Evaluation

  • Evaluation tracks success rate, token usage, and action count, distinguishing setup correctness from the efficiency of reaching success.The authors describe token and step use as practical efficiency measures relevant to costs, response times, and context limits.Source: §2.2 · p. 3

2.3 Benchmark characteristics

  • Single-line literal success/failure commands make evaluation deterministic rather than relying on LLM judges or potentially flaky test suites.SetupBench spans seven languages, five database engines, multiple package managers, and tasks from simple installations to multi-service orchestration.Source: §2.3 · pp. 3–4
  • Fresh minimal Linux containers require agents to install packages, configure databases, and resolve conflicts from scratch, unlike pre-configured benchmark containers.The benchmark is presented as reflecting cloud-sandbox setup tasks and practical systems-administration capabilities.Source: §2.3 · p. 4

3 Analysis and Results

Evaluation of OpenHands on SetupBench reveals setup failures and substantial exploration overhead, alongside measurable differences in resolve rates across model variants. The analysis identifies failure patterns and proposes changes to repository exploration, environment persistence, constraint checking, and model selection.

3.2 Performance results

  • 62.4% resolve rate: Claude 4 Sonnet led the OpenHands variants, using about 30% more tokens and 32% more steps than Claude 3.7 Sonnet.Runs used standardized Docker images; the harness checked task-specific validation commands in a fresh terminal, with a two-hour timeout.Source: §3.1 · pp. 4–5, §3.2 · p. 5

3.3 Failure Mode Analysis

  • Failures included missed test tooling (17–26% of unsuccessful repo setups), invented task constraints, and tools unavailable in fresh shells after non-persistent installs.Manual inspection attributed the test-tooling failure mode to 5–7 instances per model, depending on the model.Source: §3.3 · p. 5

3.4 Efficiency Analysis

  • The efficiency baseline mapped human setup behavior from 10 instances to agent steps, counting each folder exploration as two steps and each file read as one.The comparison excluded agent actions without meaningful human equivalents, including internal reasoning, completion signals, initial context steps, and process monitoring.Source: §3.4 · p. 6
  • 38–68% of steps were wasted across the 10 analyzed instances, ranging from Claude 3.5 to Claude 4.The analysis compared agent trajectories with the estimated minimum necessary actions.Source: Table 4 · p. 6, §3.4 · p. 6
  • The identified inefficiencies were redundant partial file reads, poor instruction following, and off-target reads of files without actionable setup instructions.Examples include repeated reads of increasing file ranges and checks for installations despite explicit instructions about the environment.Source: §3.4 · p. 7

3.5 Design implications

  • The authors propose ranking setup-relevant files and exposing repository structure early, then persisting environment changes and summarizing them for human handoffs.Persistence recommendations include writing changes to configuration files and sourcing them in the current session.Source: §3.5 · p. 7
  • They also propose caching repository structure, batching exploration, validating configuration decisions against documentation, and selecting models according to task complexity and resource constraints.The cited model comparison reports 57% success and 99% more tokens for Claude 3.7 than GPT-4.1 at 50% success.Source: §3.5 · pp. 7–8

4 Discussion

The discussion reports that coding agents can resolve many common setup tasks, but still fail on implicit tooling requirements and environment state that does not persist across sessions. It also highlights resource trade-offs and proposes extending evaluation beyond setup alone.

4 Discussion

  • 62.4% resolve rate: Claude 4 achieved the highest rate, with 75.0% on background-service setup and 87.5% on dependency resolution.GPT-4.1 reached 50.5% and Claude 3.5 Sonnet 53.8%.Source: §4 · p. 8
  • Agents missed implicit development-tooling requirements, failed to preserve shell state across sessions, and sometimes hallucinated ports or made unnecessary configuration edits.The discussion links these failures to gaps in implicit reasoning, session management, and grounding in task specifications.Source: §4 · p. 8
  • 104.9 million tokens and 4,377 tool steps accompanied Claude 4’s strong performance, compared with 37.7 million tokens and 1,793 steps for Claude 3.5 Sonnet.Claude 3.5 achieved 3.5% better performance than GPT-4.1 while using 6% fewer tokens and 34% fewer steps.Source: §4 · p. 8
  • Proposed benchmark extensions include chained development tasks, cloud infrastructure management, and system migrations beyond basic setup correctness.These extensions would test long-term planning, adaptability, and task continuity.Source: §4 · p. 8

5 Limitations

SetupBench's manual curation constrains scale, and its privileged networked sandbox and omitted domains limit the settings represented. The authors provide the benchmark materials as a foundation for future contributions.

5 Limitations

  • All 93 tasks were manually reviewed; scaling to hundreds or thousands will require greater automation.Roughly half were authored by humans and the remainder adapted from repositories.Source: §5 · p. 8
  • Agents run as root with unrestricted outbound networking, unlike environments with limited permissions or restricted network access.This setting simplifies execution and focuses evaluation on functionality and correctness.Source: §5 · p. 8
  • SetupBench covers seven language ecosystems and five databases but omits GPU drivers, most message queues, container orchestration, and infrastructure-as-code tools.Source: §5 · p. 9
  • The authors release prompts, scripts, and evaluation harnesses to encourage community contributions addressing these gaps.Source: §5 · p. 9

6 Related Work

Related benchmarks cover code editing, environment setup, or generic tool use, but do not directly assess whether agents can configure an environment until code runs. SetupBench targets that remaining evaluation gap.

6 Related Work

  • SWE-BENCH and DEVBENCH [Li et al., 2024a] provide preconfigured containers, while AGENTBENCH [Liu et al., 2024] has few software tasks and no system configuration.SetupBench isolates the environment-bootstrap phase before code changes.Source: §6 · p. 9
  • ENVBENCH [Eliseeva et al., 2025] omits OS packages, databases, and daemon orchestration, while LADS [Khan et al., 2025] and OPSEVAL [Liu et al., 2025] lack runnable end-to-end setup tasks.SetupBench spans languages, databases, and process managers.Source: §6 · p. 9
  • TOOLBENCH [Qin et al., 2024] and STABLETOOLBENCH [Guo et al., 2024] use mocked APIs, TOOLRET [Shi et al., 2025] measures API retrieval, and NL2BASH [Lin et al., 2018] focuses on single-line commands.SetupBench instead involves multi-command planning, package installation, and service supervision.Source: §6 · p. 9
  • Prior benchmarks address useful software-engineering facets, but none directly evaluates whether an agent can get code to run.Source: §6 · p. 9

7 Conclusion and Future Work

Across 93 tasks, the best-performing model, Claude 4, achieved 62.4% success, while the evaluation identified recurring setup failures and wasted steps. The authors propose persistent file-system representations, semantic search, and benchmark extensions as future directions.

7 Conclusion and Future Work

  • Claude 4 achieved 62.4% success across the 93-task evaluation, while agents showed three recurring failure modes and 38–69% wasted steps.Failures involved implicit development tooling, hallucinated constraints, and non-persistent configurations; inefficiency reflected redundant reads, poor instruction following, and off-target exploration.Source: §7 · p. 9
  • The authors suggest persistent file-system representations, semantic search, and hybrid approaches balancing efficiency with accuracy for future agent development.Source: §7 · p. 9
  • Future benchmark extensions include multi-repository setups and interactive configuration scenarios.Source: §7 · p. 9

A Example Dataset Entry

The appendix illustrates dataset entries, prompt construction, and scripts for mining and validating dependency conflicts. Together, these materials show how setup tasks and deterministic success checks are prepared.

A Example Dataset Entry

  • A sample Ruby dependency-conflict entry specifies an ecosystem, base commit, success command, instance identifier, task description, and base image.The field conventions also list task type, with ecosystem and language as category-specific keys.Source: A Example Dataset Entry · p. 11

B.1 Setup-Instruction Derivation Prompt

  • The setup-instruction prompt requests repository analysis, step-by-step setup instructions, and a single-line success criterion for an Ubuntu 22.04 sandbox with nothing pre-installed.Source: §B.1 · p. 12

B.2 Success-Command Synthesis Prompt

  • The success-command prompt asks for a command that tests key functionality, requires full configuration, and reports success or failure; repository Markdown is supplied as context.Source: §B.2 · p. 12

C.1 Mining script

  • The mining script searches closed GitHub issues for ecosystem-specific resolver errors and retains cases with a lockfile, recording the base commit and matched error snippet.Source: §C.1 · pp. 13–14

C.2 Validation script

  • The validation script uses ecosystem-specific install commands in Docker containers and records whether validation succeeded along with installation output.Source: §C.2 · pp. 15–16

D Database-setup examples: MySQL tier ladder

The MySQL examples include a Tier 1 instance labeled dbsetup-mysql-1.

D.1 Tier 1 — basic install + single dump

  • The Tier 1 basic-install and single-dump example is identified as dbsetup-mysql-1.Source: §D.1 · p. 17

Task requirements

The basic MySQL task covers installation and importing a compressed dump; the next instance is labeled for ordered migrations and a charset tweak.

Task requirements

  • The basic task requires non-interactive MySQL installation with root login and importing dump.sql.gz.The dump must first be decompressed.Source: Task requirements · p. 17

D.2 Tier 2 — ordered migrations + charset tweak

  • The next instance is dbsetup-mysql-2, labeled for ordered migrations and a charset tweak.Source: §D.2 · p. 17

Task requirements

The requirements progress from MySQL migrations and authentication settings to a stricter database configuration and a supervised Gunicorn Unix-socket service.

Task requirements

  • The migration-and-charset tier requires numbered compressed migrations with foreign keys, utf8mb4, and root password authentication.Source: Task requirements · p. 17

D.3 Tier 3 — port change, strict mode, user permissions

  • The Tier 3 MySQL task uses port 3307, STRICT_TRANS_TABLES, corrected migration order and DEFINER references, and a privileged benchmark_user account.Port 3306 is blocked; the account credentials are benchmark_user/benchmark_pass.Source: Task requirements · p. 18, §D.3 · p. 18

E Background-service example: Gunicorn + Unix socket

Related papers

Loading 2507.09063v1…