PaperlayerView exact revision on arXiv

Source-linked AI summary

EnvBench: A Benchmark for Automated Environment Setup

Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, Yaroslav Zharov

arXiv:2503.14443v1cs.LGcs.SE

TL;DR

Prior environment-setup studies use limited datasets, leaving broad benchmarking of repository configuration underrepresented. This paper introduces ENVBENCH, a Python and JVM benchmark with automatic checks; Bash Agent with GPT-4o achieves the highest success rates, but setup remains challenging, especially for Python.

  • Problem

    Prior environment-setup studies evaluate approaches on limited datasets, while setup varies across repositories and is often incorporated into broader tasks.

    Source: §2 · p. 3

  • Method

    The authors introduce ENVBENCH, a benchmark of Python and JVM repositories evaluated with static analysis for Python and compilation checks for JVM languages.

    Source: §1 · p. 2, §7 · p. 8

  • Results

    Bash Agent with GPT-4o leads both tasks at pass@1: 29.47% of JVM repositories and 6.69% of Python repositories.

    Source: §5 · p. 7

  • Takeaways & Limitations

    ENVBENCH provides systematic assessment of setup strategies across a larger repository set using language-specific checks.

    Source: §7 · p. 8

    Static analysis and compilation checks may miss runtime issues that appear only during execution.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have enabled researchers to focus on practical repository-level tasks in software engineering domain. In this work, we consider a cornerstone task for automating work with software repositories-environment setup, i.e., a task of configuring a repository-specific development environment on a system. Existing studies on environment setup introduce innovative agentic strategies, but their evaluation is often based on small datasets that may not capture the full range of configuration challenges encountered in practice. To address this gap, we introduce a comprehensive environment setup benchmark EnvBench. It encompasses 329 Python and 665 JVM-based (Java, Kotlin) repositories, with a focus on repositories that present genuine configuration challenges, excluding projects that can be fully configured by simple deterministic scripts. To enable further benchmark extension and usage for model tuning, we implement two automatic metrics: a static analysis check for missing imports in Python and a compilation check for JVM languages. We demonstrate the applicability of our benchmark by evaluating three environment setup approaches, including a simple zero-shot baseline and two agentic workflows, that we test with two powerful LLM backbones, GPT-4o and GPT-4o-mini. The best approach manages to successfully configure 6.69% repositories for Python and 29.47% repositories for JVM, suggesting that EnvBench remains challenging for current approaches. Our benchmark suite is publicly available at https://github.com/JetBrains-Research/EnvBench. The dataset and experiment trajectories are available at https://jb.gg/envbench.

1 INTRODUCTION

Environment setup is a practical repository-level task, but prior dedicated studies emphasize agent strategies more than broad benchmarking. ENVBENCH evaluates setup approaches across Python and JVM repositories and finds that even its best-performing approach succeeds on a minority of repositories.

1 INTRODUCTION

  • Prior dedicated environment-setup studies emphasize agent strategies but cover limited datasets, including 40 Python repositories in one study and 10 repositories per language in another.Environment setup can take several hours for reproducing NLP results, and developers report incomplete or inappropriate installation instructions as concerns.Source: §1 · pp. 1–2
  • ENVBENCH covers 329 Python and 665 JVM repositories, uses language-specific automatic checks, and excludes projects configured by deterministic scripts.The evaluated approaches include zero-shot LLM, Bash Agent, and Installamatic Agent workflows with GPT-4o and GPT-4o-mini.Source: §1 · p. 2
  • 29.47% of JVM repositories and 6.69% of Python repositories are successfully configured by Bash Agent with GPT-4o, the highest-success approach.LLM-based approaches also reduce missing imports relative to the deterministic script for many Python repositories, while scripts without error feedback commonly contain errors.Source: §1 · p. 2
  • The benchmark suite, dataset, and experiment trajectories are publicly available.The suite is hosted on GitHub, while the dataset and trajectories are available separately.Source: §1 · p. 2

2 RELATED WORKS

Environment setup varies across repositories, and existing dedicated benchmarks cover limited projects or evaluate it as part of broader tasks. ENVBENCH broadens repository coverage and uses lightweight checks to assess setup.

2 RELATED WORKS

  • Setup steps vary across repositories, complicating automation; selecting executable repositories for execution-based benchmarks also commonly requires substantial manual work [Tang et al., 2024; Zhao et al., 2024].Source: §2 · pp. 2–3
  • Prior environment-setup benchmarks cover 40 Python repositories or 10 repositories per language across five languages, and rely on test-based evaluation that can require manual inspection.Broader scientific-reproduction and machine-learning benchmarks include setup but provide limited insight into setup-specific challenges.Source: §2 · p. 3
  • ENVBENCH covers 994 repositories across Python, Java, and Kotlin, excludes projects handled by deterministic scripts, and uses static analysis or compilation checks instead of test execution.It also provides predefined Dockerfiles to keep base system configuration consistent across approaches.Source: §2 · p. 3

3 ENVBENCH BENCHMARK

ENVBENCH evaluates repository-specific environment setup by asking approaches to produce shell scripts and checking their outcomes. Its dataset filters active Python and JVM projects to retain repositories not handled by simple deterministic setup scripts.

3 ENVBENCH BENCHMARK

  • ENVBENCH targets Python and JVM repositories, specifically Java and Kotlin, as distinct technology stacks.The benchmark and its evaluation code are publicly available.Source: §3 · p. 3

3.1 TASK DEFINITION

  • Approaches receive full repository contents and must produce a shell script; Python setup is checked for missing imports, while JVM setup is checked by building with Gradle or Maven.These checks provide binary success and error-count measures, but unlike test-suite metrics they miss problems appearing only at runtime; Docker containers isolate script execution.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 DATA COLLECTION AND FILTERING

  • The collection process applies activity and quality filters, retains repositories with a single supported dependency-manager configuration, and removes Docker-configured projects and those handled by deterministic scripts.The final dataset contains 329 Python and 665 JVM repositories after the deterministic scripts configure 62 Python and 309 JVM projects successfully.Source: §3.3 · pp. 4–5

4 EXPERIMENTAL SETUP

The experiments assess setup using pass@1 and average reported errors on ENVBENCH repositories. They compare zero-shot and agentic workflows using GPT-4o and GPT-4o-mini.

4.1 DATASET & METRICS

  • ENVBENCH contains 329 Python and 665 JVM repositories; pass@1 requires zero exit code and zero reported errors, while avgErrs measures reported errors when scripts finish successfully.Source: §4.1 · p. 5

4.2 BASELINES

  • The baselines use GPT-4o and GPT-4o-mini and include a single-request zero-shot script generator, an adapted INSTALLAMATIC agent, and a ReAct Bash Agent.INSTALLAMATIC gathers context before generating a script in one request, with regeneration disabled; Bash Agent iteratively gathers context and generates scripts through bash interaction.Source: §4.2 · pp. 5–6

5 RESULTS & DISCUSSION

Bash Agent with GPT-4o has the highest pass@1 for both JVM and Python, though full Python setup remains challenging. Baselines can outperform a deterministic script on Python, while error-count comparisons are restricted to scripts that execute successfully.

5 RESULTS & DISCUSSION

  • 29.47% for JVM and 6.69% for Python repositories: Bash Agent with GPT-4o has the highest pass@1 in both tasks.pass@1 counts repositories set up with zero exit code and zero reported issues.Source: Table 1 · p. 6, §5 · p. 7
  • 52.00 missing imports per repository for Bash Agent with GPT-4o on Python, and 21.43 errors per repository for Installamatic Agent with GPT-4o on JVM Maven repositories.avgErrs is computed only where setup scripts finish with zero exit code, limiting the comparison; this issue is more prominent for JVM.Source: §5 · p. 7
  • On a 30-repository Python sample, expert-produced scripts achieved 66.7% pass@1, compared with 10.0% for the best baseline.The expert scripts serve as an upper bound in this comparison.Source: §5 · p. 7
  • 31% for Bash Agent with GPT-4o to 53% for Zero-shot LLM with GPT-4o: baselines reduced missing imports versus the deterministic script on these shares of repositories.For the baselines' underperforming cases, the average increase in missing imports was at least 200%; repositories with more missing imports may inform future setup strategies.Source: §5 · pp. 7–8

6 LIMITATIONS

The benchmark excludes Docker-dependent projects, and its results may also reflect data contamination risks and variability in open-source repository quality. Its setup-success measures use static analysis for Python and compilation checks for JVM rather than runtime tests.

6 LIMITATIONS

  • Docker-dependent projects are excluded, so the benchmark does not cover repositories requiring Docker for setup.The authors cite evaluation complexity and metrics as reasons for the exclusion.Source: §6 · p. 8
  • Public-code pretraining may expose models to setup instructions for some benchmark repositories, although the authors believe this does not significantly affect findings.The benchmark can be updated with newer repositories to reduce contamination risk as training datasets evolve.Source: §6 · p. 8
  • Variable documentation quality and invalid repositories may influence results, reflecting real-world repository conditions.Future work could manually verify the benchmark and remove invalid samples.Source: §6 · p. 8
  • Static analysis and compilation checks may miss runtime issues that appear only during execution.These proxies enable efficient evaluation across more repositories without the complexity of running test suites.Source: §6 · p. 8

7 CONCLUSION

EnvBench covers Python and JVM repositories with static-analysis and compilation-based checks, and the evaluated methods configure only a minority of repositories successfully. Erroneous script generation remains a challenge, while the benchmark and code are publicly available for further research.

7 CONCLUSION

  • The benchmark covers 329 Python and 665 JVM repositories, using static analysis for Python and compilation checks for JVM setup.Appendix A also describes the benchmark, evaluation suite, and deterministic scripts used in benchmark construction.Source: §7 · p. 8, A BENCHMARK · p. 11
  • 29.47% of JVM and 6.69% of Python repositories were successfully configured by the best-performing evaluated method.The evaluation compared three methods, including two AI agents, using two LLM backbones.Source: §7 · p. 8
  • Erroneous script generation is a key challenge, with further investigation and mitigation left to future work.The benchmark and associated code are publicly available; future extensions could add manual verification, repositories, languages, and runtime-based metrics.Source: §7 · p. 9

A.1 BENCHMARK STATISTICS

The appendix points readers to repository statistics and documents the shared Docker environments and deterministic scripts used for benchmark construction.

A.1 BENCHMARK STATISTICS

  • Repository statistics for the benchmark are presented in Table 3, but the supplied text gives no individual table values.The table is identified as repository statistics for the benchmark.Source: §A.1 · p. 11, Table 3 · p. 11

A.2 DOCKER ENVIRONMENT

  • The evaluation uses separate Python and JVM Docker environments based on Ubuntu 22.04, with commonly needed tools preinstalled for reproducible, isolated runs.The appendix lists Python tools including pyenv, Poetry, uv, and pyright, and SDKMAN with Java 11.0.20-tem for JVM.Source: §A.2 · p. 11

A.3 DETERMINISTIC SCRIPTS

  • Deterministic scripts use repository files to select environment managers, versions, dependencies, and build commands, handling simple setup cases.For Python they recognize configuration and dependency files; for JVM they detect Maven or Gradle, infer Java versions, and run builds with common flags.Source: §A.3 · p. 12

B LLM BASELINES IMPLEMENTATION DETAILS

The appendix describes how the zero-shot LLM, Bash Agent, and Installamatic Agent are implemented for environment setup. It also introduces additional experimental results.

B LLM BASELINES IMPLEMENTATION DETAILS

  • Zero-shot LLM receives repository-specific context and generates an environment setup shell script.Context includes repository structure, documentation, environment information, and language-specific configuration or dependency files.Source: B LLM BASELINES IMPLEMENTATION DETAILS · p. 13
  • Bash Agent uses a ReAct workflow with Docker-terminal commands and constructs its resulting script from commands that completed successfully.It is allowed up to 30 iterations, with a 360-second timeout per command.Source: B LLM BASELINES IMPLEMENTATION DETAILS · p. 13
  • Installamatic Agent runs documentation-gathering and Dockerfile-building stages, omitting the original repair stage and producing a shell script instead of a Dockerfile.Each of the two stages is allowed one full iteration.Source: B LLM BASELINES IMPLEMENTATION DETAILS · p. 13

C EVALUATION RESULTS

C.1 CASE STUDY

The Tablib case study compares baseline setup scripts for a Python project managed through pyproject.toml. It illustrates how version, dependency, and installation choices can affect setup outcomes.

C.1 CASE STUDY

  • The Bash Agent successfully sets up Tablib by reading its Python requirement and installing all optional dependencies with .[all].It uses the minimum necessary steps, while the project has no README installation instructions or separate setup.py.Source: §C.1 · p. 14
  • The zero-shot approach finishes without errors but misses optional dependencies, while using a compatible Python version and adding unnecessary setup steps.It creates a virtual environment, upgrades pip, and installs test requirements that are not needed in this case.Source: §C.1 · p. 14
  • The Installamatic Agent fails with exit code 100 on the Tablib setup.Source: §C.1 · p. 14
  • The compared scripts illustrate failure points involving Python-version compatibility and installation method.One example uses Python 3.8.18 despite a requirement of at least 3.9; another installs dependencies through apt instead of pip and exits non-zero.Source: §C.1 · p. 14

C.2 COSTS

The experiments compare token use and script exit codes across approaches, then examine common failure causes and generated shell scripts. Agent-based approaches use more tokens, while Bash Agent with GPT-4o has the lowest reported Python script-failure rate among the compared approaches.

C.2 COSTS

  • Agent-based approaches consume 5-10x more tokens than zero-shot LLM approaches, and JVM projects require more tokens across approaches, primarily because their build and dependency logs are more verbose.The most token-intensive agent-based approach averages $0.25 per repository.Source: §C.2 · p. 15

C.3 EXIT CODES

  • Missing system dependencies and Python or Java version mismatches are among the causes of scripts terminating with non-zero exit codes.Zero-shot LLM and Installamatic Agent commonly use tools unavailable on the system; Bash Agent can react to error feedback but sometimes repeats a failing action.Source: §C.3 · pp. 15–16
  • 17.74% (Bash Agent with GPT-4o) to 81.20% (Zero-shot LLM with GPT-4o) of JVM scripts failed; Python failures fell from 28.57% (deterministic script) to 5.78% (Bash Agent with GPT-4o).The Python reduction is reported as 22.79%; Table 5 reports exit-code distributions, which do not alone establish successful setup.Source: §C.3 · p. 15, Table 5 · p. 16

C.4 SHELL SCRIPTS ANALYSIS

  • Bash Agent tends to produce the shortest scripts; its commands often explore filesystems, with Python-specific tools and package managers differing from JVM commands.The reported frequent commands include cat and ls, pyenv, pip, python, poetry, sdk, ./gradlew, and maven.Source: §C.4 · p. 16, Figure 3 · p. 17, Figure 4 · p. 17

D EXPERT-PRODUCED SCRIPTS

Expert-produced scripts were tested on 30 randomly selected Python repositories and outperformed the considered environment-setup baselines on that sample. Their results also show limitations of the Python static-analysis metric, including cases involving obsolete dependencies and dynamically resolved imports.

D EXPERT-PRODUCED SCRIPTS

  • 66.7% pass@1 and 9.8 avgErrs were achieved by expert-produced scripts on 30 randomly sampled Python repositories, outperforming all considered environment-setup baselines.All expert scripts finished with zero exit code; bootstrap resampling further showed a wide gap between expert scripts and Bash Agent with GPT-4o-mini.Source: D EXPERT-PRODUCED SCRIPTS · p. 17, Table 7 · p. 18, Figure 5 · p. 18
  • Obsolete dependencies and dynamically resolved imports that static type checking cannot process correctly are among the issues preventing successful setup or affecting evaluation.The examples include a legacy Python 2.x module in a Python 3.x codebase.Source: D EXPERT-PRODUCED SCRIPTS · p. 17

Related papers

Loading 2503.14443v1…