PaperlayerView exact revision on arXiv

Source-linked AI summary

EnvPilot: Systematic Design and Evaluation of an Experience-Augmented Agent for Software Environment Setup

Hanwu Chen, Hanyu Lin, Zhanjiang Yang, Linhao Zhang, Aoyan Li, Jinxi Li, Meng Li, Yin Chen, Daoguang Zan

arXiv:2609.07357v1cs.SE

TL;DR

Automated environment setup must prepare reproducible runtimes so projects can be built and tested. EnvPilot reuses structured experience from execution trajectories through memory and context-aware retrieval, achieving a 75.00% success rate on AES-Bench.

  • Problem

    Environment setup must prepare a reproducible runtime so a project can be built and its tests reliably executed.

    Source: §2 · p. 4

  • Method

    EnvPilot converts execution trajectories into reusable structured experience and retrieves relevant guidance using project context during setup.

    Source: §1 · p. 3

  • Results

    75.00% success rate on AES-Bench, outperforming existing methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Ablations show that both TDM and Context-aware Retrieval contribute to performance gains, with their combination yielding the most significant improvement.

    Source: §1 · p. 3

    The evaluation focuses on open-source repositories with public dependencies and standard registries, so it does not establish performance outside that setting.

    Source: §8 · p. 27

Abstract

from arXiv · show

Environment Setup is a critical yet complex task in software engineering that relies heavily on expert knowledge. Existing automated environment setup methods lack the ability to accumulate experience from past execution trajectories and to evolve over time. As a result, their performance is limited because they often perform redundant exploration, ignore useful past solutions, and fail to generalize across diverse software ecosystems. We present the systematic design and empirical validation of EnvPilot, an experience-augmented agent that operationalizes trajectory-derived experience reuse for software environment setup. EnvPilot maintains an expandable Trajectory-Derived Memory (TDM), initialized with 667 high-quality experiences. It systematically transforms implicit knowledge from historical execution trajectories into structured experience and retrieves the most relevant guidance during task execution through the Context-aware Retrieval mechanism. This enables EnvPilot to combine multiple validated setup strategies, providing more precise and detailed guidance than methods that rely solely on static project files or web retrieval. To evaluate EnvPilot, we construct AES-Bench, a multilingual benchmark of 112 real-world GitHub instances across 9 programming languages. Experiments show that EnvPilot achieves a new state-of-the-art (SOTA) with a 75.00% Pass@1 success rate while reducing reasoning costs. Our empirical study shows that both the structured experience representation and the Context-aware Retrieval mechanism are essential.

1 Introduction

Existing setup methods rely on static information or web retrieval and treat tasks in isolation, while trajectory-derived experience offers a way to reuse validated solutions. EnvPilot structures past setup trajectories into reusable memory and retrieves relevant guidance; on AES-Bench, it achieves a 75.00% success rate, the highest among compared methods.

1 Introduction

  • Existing methods rely mainly on static project information or web retrieval and treat setup tasks in isolation, limiting experience reuse across tasks.The paper identifies noisy, lengthy trajectories, cross-toolchain fix steps, and retrieval only after failures as challenges for applying existing memory approaches directly.Source: §1 · p. 2
  • EnvPilot converts historical execution trajectories into an expandable TDM of reusable problem–solution–action experiences, initialized with 667 high-quality entries.Each entry captures errors, corrective strategies, and executable commands; new executed tasks add validated knowledge.Source: §1 · p. 3
  • Context-aware Retrieval profiles a project, generates multiple queries for potential setup problems, and combines retrieved cases with Reciprocal Rank Fusion.The selected experiences guide the agent through its reasoning loop as part of a Learn–Retrieve–Act–Evaluate cycle.Source: §1 · p. 3
  • 75.00% success rate on AES-Bench makes EnvPilot the highest-performing method among those compared.AES-Bench covers 112 real-world GitHub instances across 9 programming languages; ablations indicate that TDM and Context-aware Retrieval both contribute to performance gains.Source: §1 · pp. 3–4
  • AES-Bench comprises 112 version-locked GitHub instances across 9 languages, each with executable tests and authentic build and dependency practices.The benchmark is designed to evaluate automated environment setup methods.Source: §1 · p. 3

2 Problem Statement

The task is to prepare a correct runtime environment for a specified software project so it can be built and its tests reliably executed. The paper frames an agent as a TDM-equipped system that observes, retrieves, acts, and evaluates in a closed loop.

Problem definition

  • The task is to set up the correct runtime environment for a project identified by its repository and version, using structured experience from past attempts.The intended environment supports smooth builds and reliable test execution.Source: §2 · p. 4

Expected output

  • The agent configures the environment in a provided container, runs the project test suite, and returns commands, execution trajectory, and test logs.The command sequence includes both successful and failed commands.Source: §2 · p. 4

Key challenges

  • The problem has three challenges: structuring raw trajectories, retrieving relevant experiences for new projects, and reliably evaluating setup outcomes without manual intervention.Evaluation must meet preset precision requirements.Source: §2 · p. 4

3 Method

EnvPilot distills prior setup trajectories into reusable TDM experiences, retrieves guidance using project context, and updates its memory through a Learn–Retrieve–Act–Evaluate cycle. Its method combines structured experience construction, context-aware retrieval, isolated execution, and LLM-based evaluation.

3.1 Overview

  • EnvPilot’s Learn–Retrieve–Act–Evaluate cycle retrieves contextually relevant experiences, executes setup in an isolated container, evaluates the outcome, and adds new insights to TDM.This feedback loop is designed to refine the memory across tasks.Source: §3.1 · pp. 4–5

3.2 Trajectory-Derived Memory

  • Each TDM entry records the encountered problem, its corrective strategy, and executable commands, making historical setup knowledge concise and reusable.For example, the entry for a missing Rust runtime library identifies the issue and solution and gives the command `apk add libgcc`.Source: §3.2.1 · p. 5, §3.2.2 · p. 6
  • Processing 781 raw execution trajectories produced 667 distilled experience entries, an 85.4% retention rate; root-cause analysis extracts validated interventions and converts implicit insights into standardized JSON objects.DeepSeek-R1 performs reasoning-based distillation. A manual audit of a random 5% sample (approx. 34 entries) confirmed the entries accurately reflected original environment challenges and provided technically sound interventions.Source: §3.2.2 · p. 6

3.3 Experience-augmented Automated Environment Setup

  • EnvPilot profiles project versions, dependency managers, and commands, generates hypothetical obstacle queries, and combines query-specific similarity rankings with Reciprocal Rank Fusion.The retrieved memory and project profile are supplied to the agent as setup guidance.Source: §3.3.1 · pp. 7–8
  • A moderate retrieved set balances added guidance against longer context and potentially distracting irrelevant commands; containerized runs also supply trajectories for incremental TDM updates.Successful runs add validated strategies, while failures contribute negative experiences.Source: §3.3.1 · p. 8, §3.3.2 · pp. 8–9

3.4 Automated Evaluation Framework

  • The LLM judge evaluates environment integrity rather than repository code correctness, counting assertion failures as passes but runtime, dependency, or compilation errors as failures.A successful setup also requires a complete test lifecycle and an explicitly triggered test command.Source: §3.4 · p. 9

4 Benchmark

AES-Bench targets reproducible repository-level environment setup evaluation through version-locked commits, native test execution, and multilingual coverage. Its construction and varied language and tool practices provide a realistic evaluation basis.

4 Benchmark

  • AES-Bench evaluates repository-level setup with fixed commits, native test execution, and 112 instances across 9 programming languages.The benchmark addresses earlier limitations involving restricted language coverage, omitted test execution, and unreproducible code versions.Source: §4 · pp. 9–10

4.1 Dataset Construction

  • The benchmark selects 112 instances from 99 qualified repositories, records each instance’s identifier, commit hash, and test command, and excludes repositories used to build TDM.The repository separation tests transfer to unseen projects rather than memorization of prior complete solutions.Source: §4.1 · p. 10

4.2 Dataset Statistics

  • AES-Bench spans ecosystems with contrasting tool practices: Go, Rust, and PHP are standardized, while JavaScript and C/C++ are more diverse and complex.The benchmark includes 9 languages; Java projects have the largest average file counts, and C/C++ projects average nearly 6,000 files.Source: §4.2 · p. 11

5 Experimental Setup

The experimental setup compares EnvPilot with existing setup agents, defines success and resource metrics, and specifies the memory, models, and run protocols used for evaluation.

5 Experimental Setup

  • The seven research questions cover comparative success and efficiency, module contributions, judge reliability, retrieval quantity, run stability, behavioral effects, and remaining failures.The setup includes comparison with existing methods, ablations, human validation, repeated runs, and a repository case study.Source: §5 · p. 12
  • The study also examines how experience changes behavior in a repository case study and analyzes failure patterns to identify remaining setup difficulties.The case study compares EnvPilot with and without experience under controlled conditions.Source: §5 · p. 12

5.1 Baselines

  • Baselines include adapted SWE-Agent, RepoLaunch, ExecutionAgent, Repo2Run, Installamatic, and a generalized Bash-based Agent.Repo2Run is evaluated only on the Python subset because of its language constraints.Source: §5.1 · pp. 12–13

5.2 Evaluation Metrics

  • Environment Configuration Success Rate requires the project’s tests to execute to completion, while API calls, tokens, and dollar cost measure resource use and interaction efficiency.Test cases need not pass or fail in a particular way; success indicates a full test lifecycle and environment integrity.Source: §5.2 · p. 13

5.3 Implementation Details

  • The TDM starts with 667 trajectory-derived experiences, and evaluation freezes this memory while using up to 10 retrieved items by default.EnvPilot uses DeepSeek-V3-0324 with deterministic settings; EnvPilot and adapted SWE-Agent receive three runs per instance, with median Pass@1 reported, while other baselines receive one run.Source: §5.3 · pp. 13–15

6 Experiment Results

EnvPilot outperformed baseline agents on AES-Bench, and the experiments examined its components, retrieval settings, stability, and remaining failure modes. Results also assessed the reliability of the automated evaluator and illustrated how stored experience changes task execution.

6.1 RQ1: Comparison with Baseline Methods for Environment Setup Performance

  • On the Full dataset, DeepSeek-V3-0324 achieved 75.00% success with EnvPilot, exceeding SWE-Agent (62.50%), RepoLaunch (55.36%), and ExecutionAgent (43.75%), supporting EnvPilot’s effectiveness on AES-Bench.EnvPilot used 220K tokens on the Full set, compared with 200K for SWE-Agent and 420K for ExecutionAgent.Source: §6.1 · p. 15

6.2 RQ2: Ablation Study on the Contribution of the TDM and the Context-aware Retrieval mechanism

  • 75.00% success for full EnvPilot exceeded every ablated variant (62.50%–66.96%) by combining the complete problem–solution–action format with Context-aware Retrieval.The full system used the fewest API calls (28), but its 220K token usage was not the smallest.Source: §6.2 · pp. 17–18, Table 5 · p. 18

6.3 RQ3: Reliability of Automated Evaluation

  • 98.04% accuracy and 98.34% F1 were achieved by the LLM judge against human labels across 460 trajectories.The judge recorded 99.26% precision and 97.45% recall.Source: §6.3 · p. 19, Table 6 · p. 19

6.4 RQ4: Impact of Experiences Number

  • 75.00% success at k=10 exceeded 68.75% at k=5, while success fell to 74.11% at k=15 and 71.43% at k=20.Token use rose from 200K at k=5 to 230K at k=20, while API calls remained between 27 and 30; the study selected k=10 as the default.Source: §6.4 · p. 20, Table 7 · p. 20

6.5 RQ5: Impact of LLM Nondeterminism and Result Stability

  • 74.11% ± 4.09 mean success for EnvPilot exceeded 62.20% ± 2.25 without TDM across three trials.EnvPilot averaged 28 ± 0.58 API calls versus 30 ± 0.58, but used 218K ± 4.93 tokens versus 199K ± 4.16.Source: §6.5 · p. 21, Table 8 · p. 21

6.6 RQ6: Case Study and Behavioral Interpretation

  • In the Nuxt.js case, experience guided the agent to use nvm, Node.js 20, pnpm, build before testing, and install missing Playwright binaries.The agent without experience instead repeated ineffective dependency and test-repair attempts and failed to complete the task.Source: §6.6 · pp. 21–22, Figure 7 · p. 22

6.7 RQ7: Failure Mode Analysis

  • 60% of failures in the best-performing trial arose from toolchain and runtime issues; platform constraints and native dependency failures each accounted for 16%, and repo-config mismatch 8%.The 25 failed instances were manually inspected and categorized into four root-cause groups.Source: §6.7 · pp. 23–24, Figure 8 · p. 23

7 Related Work

Prior work addresses configuration and builds, environment-setup agents, agent memory, and evaluation benchmarks, but leaves gaps in cross-task setup experience and broad benchmark coverage. EnvPilot uses trajectory-derived experience across repositories, while AES-Bench broadens language and ecosystem coverage.

Software Engineering Configuration, Dependencies, and Reproducibility

  • Configuration, dependency, and reproducible-build tools automate declared setups, yet incomplete specifications and changing dependencies still require iterative troubleshooting.This motivates automation that interprets failure signals and infers implicit dependencies or tooling requirements.Source: §7 · p. 24

LLMs for Software Engineering

  • LLMs have been studied across code generation, completion, bug fixing, review, and testing, alongside work on environment setup agents.The related work positions setup within broader software-engineering automation.Source: §7 · pp. 24–25

LLM-based Agents in Environment Setup

  • Earlier setup agents relied on documentation, trial-and-error, or Python-specific tooling; EnvPilot instead retrieves validated solutions across repositories.EvoConfig adapts within a task but does not use external memory modules.Source: §7 · p. 25

Memory Systems in LLM-based Agents

  • Memory research spans latent, graph, hierarchical, and multimodal representations, but much of it stores abstractions rather than directly executable software-engineering actions.Experience-integration approaches include cross-task knowledge bases and trajectory-derived memories, while prior work does not explicitly target setup and dependency resolution.Source: §7 · pp. 25–26
  • Replay methods reuse execution traces, but long noisy logs, repository-specific workflows, and reactive retrieval make them a poor direct fit for heterogeneous setup tasks.EnvPilot distills traces into structured problem–solution–action experiences and retrieves them proactively using codebase profiling and hypothetical queries.Source: §7 · p. 26

Benchmarks in Environment Setup

  • AES-Bench responds to EnvBench’s limited coverage of Python and Java by incorporating a broader range of environment-setup tasks and an automated evaluation framework for more efficient and comprehensive assessment.Installamatic covered around 40 projects; ExecutionAgent remained constrained by a small number of projects and the human effort required for evaluation.Source: §7 · pp. 26–27

8 Threats to Validity

The authors discuss external, internal, and construct validity. Their findings are bounded by benchmark and execution settings, one attempt per task, possible pretraining overlap, and coarse binary success scoring.

Threats to external validity

  • Results may change in industrial settings with private dependencies, internal workflows, stricter security policies, or different operating systems and resource limits.The benchmark uses public dependencies and standard registries on a specific Linux distribution and hardware profile.Source: §8 · p. 27
  • AES-Bench omits very large monorepos, data-intensive pipelines, hardware-dependent projects, and mobile or embedded platforms, so findings best reflect common-stack server-side and library projects.The authors do not claim full generality.Source: §8 · p. 27

Threats to internal validity

  • Experiments use one setup attempt per task, and possible pretraining overlap remains despite excluding target repositories from the TDM.The authors focus on relative improvements under the same model family and toolchain.Source: §8 · p. 27

Threats to construct validity

  • Binary success scoring conflates setups requiring all tests to pass with those needing only key commands or core functions to run.The authors describe the current evaluation as coarse-grained and suggest reporting finer levels of success.Source: §8 · p. 28

9 Conclusions

The conclusion summarizes EnvPilot’s trajectory-derived experience and Context-aware Retrieval approach for software environment setup, its evaluation and success criterion, and future directions.

9 Conclusions

  • 75.00% success rate, with high efficiency, is reported for EnvPilot in the conclusion.The passage gives no comparison values or further efficiency metrics.Source: §9 · p. 28
  • EnvPilot turns execution trajectories into structured experience and uses Context-aware Retrieval to guide setup, helping it learn from past tasks, avoid repeated errors, and adapt across projects and languages.Source: §9 · p. 28
  • AES-Bench provides a multilingual evaluation benchmark of 112 real-world instances across 9 programming languages.The conclusion describes it as a rigorous way to evaluate automated environment setup systems.Source: §9 · p. 28
  • Ablation studies identify structured experience and Context-aware Retrieval as key to performance; planned future work includes larger repositories, richer execution feedback, and developer collaboration in real workflows.Source: §9 · p. 28

A Collections of Used Prompts

  • A successful setup requires a completed test lifecycle, a generated report, and zero errors; assertion failures are allowed, while missing tests, crashes, runtime or compilation errors, and incomplete reports fail.The prompt directs evaluators to judge the final test or compilation command in the trajectory rather than earlier trial-and-error output.Source: A Collections of Used Prompts · p. 38

B Baseline Adaptation Details: SWE-Agent for Environment Setup

The appendix documents how SWE-Agent was adapted for environment setup and provides reproducible details for a fair comparison.

B Baseline Adaptation Details: SWE-Agent for Environment Setup

  • The adaptation supplies a command-line interaction setting and requires one tool call per response, followed by waiting for the shell before continuing.Interactive session commands such as Python or Vim are unsupported.Source: B Baseline Adaptation Details:… · pp. 40–41
  • The task is to configure an Ubuntu environment, install the project from source when needed, and run its tests without directly installing the repository as a package.The test framework should follow repository contents and CI/CD configuration; tests must not be modified.Source: B Baseline Adaptation Details:… · pp. 40–41
  • The adaptation directs agents to inspect repository structure and prioritize CI/CD configuration before dependency files, README instructions, and other configuration files.CI/CD matrix configurations can guide environment setup across language versions.Source: B Baseline Adaptation Details:… · pp. 40–41
  • A success-reflection prompt asks agents to extract the problem, critical fix steps, reusable heuristics, and environment or tooling insights from session history.The reflection is intended to summarize takeaways from successful repair trajectories.Source: B Baseline Adaptation Details:… · p. 42

C Complete List of AES-Bench Instances

AES-Bench’s appendix provides a reproducible inventory of its instances.

C Complete List of AES-Bench Instances

C.1 C Language Instances (12 instances)

The C-language instance listing identifies repositories and pull requests alongside base commits.

C.1 C Language Instances (12 instances)

  • The listing includes two fluent/fluent-bit instances, identified by pull requests 3663 and 10007 and their base commits.Source: §C.1 · p. 42
  • The listing includes two mruby/mruby instances, identified by pull requests 3649 and 6442 and their base commits.Source: §C.1 · p. 42
  • Other listed repositories include OpenMathLib/OpenBLAS, php/php-src, redis/redis, valkey-io/valkey, mpv-player/mpv, taosdata/TDengine, and obsproject/obs-studio.Source: §C.1 · p. 43

C.2 C++ Language Instances (13 instances)

The C++-language instance listing identifies repositories and pull requests alongside base commits.

C.2 C++ Language Instances (13 instances)

  • The listing includes CGAL/cgal and root-project/root, with pull requests 8736 and 17731 and their respective base commits.Source: §C.2 · p. 43
  • Four listed instances come from halide/Halide, associated with pull requests 7506, 6533, 5545, and 8490 and their base commits.Source: §C.2 · p. 43
  • Other listed repositories include godotengine/godot, ggml-org/llama.cpp, dragonflydb/dragonfly, taichi-dev/taichi, rethinkdb/rethinkdb, google/flatbuffers, and google/libphonenumber.Source: §C.2 · p. 43

C.3 Go Language Instances (13 instances)

The benchmark instance listing spans repositories from seven programming-language ecosystems.

C.3 Go Language Instances (13 instances)

  • The Go subset lists 13 instances, including beego/beego, caddyserver/caddy, and istio/istio.Source: §C.3 · p. 43

C.4 Java Language Instances (13 instances)

  • The Java subset lists 13 instances, including checkstyle/checkstyle and junit-team/junit5.Source: §C.4 · p. 44

C.5 JavaScript Language Instances (12 instances)

  • The JavaScript subset lists 12 instances, including Automattic/mongoose, caolan/async, and facebook/react.Source: §C.5 · p. 44

C.6 PHP Language Instances (12 instances)

  • The PHP subset lists 12 instances, including briannesbitt/Carbon, composer/composer, and laravel/framework.Source: §C.6 · p. 44

C.7 Python Language Instances (12 instances)

  • The Python subset lists 12 instances, including astropy/astropy, django/django, and scikit-learn/scikit-learn.Source: §C.7 · p. 45

C.8 Rust Language Instances (12 instances)

  • The Rust subset lists 12 instances, including alacritty/alacritty, fish-shell/fish-shell, and rust-lang/mdBook.Source: §C.8 · p. 45

C.9 TypeScript Language Instances (13 instances)

  • The TypeScript subset lists 13 instances, including colinhacks/zod, trpc/trpc, and nuxt/nuxt.Source: §C.9 · p. 45

D Cross Language Analysis

EnvPilot leads overall across the reported benchmarks and backbone models, while results vary in some C/C++ and JS/TS settings.

  • Toolchain heterogeneity and negative transfer are identified as the main challenges, especially in C/C++ and some JS/TS settings.On AES-Bench, EnvPilot's C/C++ rate is 56.00%, below SWE-Agent's 64.00%; its JS/TS rate of 80.00% matches RepoLaunch.Source: D Cross Language Analysis · pp. 46–47
  • 75.00% (84/112) ECSR on AES-Bench; across two benchmarks and two backbone models, EnvPilot achieves the best overall results and remains robust in PHP and Rust.Reported AES-Bench rates include 84.62% in Java and Go, 80.00% in JS/TS, 83.33% in Rust, and 91.67% in PHP.Source: D Cross Language Analysis · pp. 46–47, Table 19 · p. 48
  • 46.43% (52/112) overall on AES-Bench with GPT-4o-mini; EnvPilot's PHP rate is 50.00%, versus RepoLaunch's 0.00%.The authors attribute small-model gains in PHP to retrieving expert knowledge for missing domain-specific setup knowledge.Source: D Cross Language Analysis · p. 46, Table 19 · p. 48
  • 72.00% (36/50) overall on ExecutionAgent Bench with DeepSeek-V3, at 26 API calls, 198K tokens, and $0.057 cost.Source: D Cross Language Analysis · p. 47, Table 20 · p. 49
  • 46.00% (23/50) overall on ExecutionAgent Bench with GPT-4o-mini; EnvPilot leads in C/C++ (55.56%) and Python (36.36%) but trails JS/TS baselines (36.36% versus 45.45%).Source: D Cross Language Analysis · p. 47, Table 20 · p. 49
Loading 2609.07357v1…