PaperlayerView exact revision on arXiv

Source-linked AI summary

Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

Bhanu Prakash Vangala, Tanu Malik

arXiv:2610.00425v1cs.SEcs.AI

TL;DR

Functional-code benchmarks largely omit whether generated software specifies the environment needed to execute. The paper evaluates environment generation through iterative repair and comparisons of declared, installed, and runtime-traced dependencies. It finds systematic dependency misspecification that varies across agents and ecosystems.

  • Problem

    Current coding-agent benchmarks focus almost exclusively on functional correctness and do not measure dependencies, package versions, or system configurations required for execution.

    Source: §1 · p. 1

  • Method

    The protocol prompts agents to generate code and an environment manifest, then iteratively repair failures and compare declared, installed, and independently traced runtime dependencies.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Dependency declarations are inconsistent across agents on identical tasks, rarely reproduce within an agent across runs, and have zero unanimous consensus across languages.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Successful code execution alone does not establish environment correctness; evaluation and training should target accurate, minimal dependency specifications alongside task completion.

    Source: §8 · p. 14

    The controlled benchmark covers 50 tasks rather than fully open-ended real-world projects, so its rates are evidence about structured code-generation settings, not exhaustive estimates for all software development.

    Source: §6.6 · p. 12

Abstract

from arXiv · show

Code generation has emerged as a central capability of large language models, with coding agents now able to produce functionally correct software projects from natural language prompts. However, functional correctness alone does not capture a critical dimension of generation quality: environment specification, defined as the accurate identification of the dependencies required to execute generated code, is equally critical. We develop an agent protocol for environment specification and introduce a three-layer framework comprising declared, runtime-installed, and necessary-and-sufficient dependencies to systematically assess coding agents for environment specification. Using this protocol, we evaluate the extent to which coding agents systematically misspecify software environment dependencies and how this misspecification varies across three agents, four languages, and fifty programming tasks. Our results show that current coding agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across agents, dependency set agreement is as low as 7% for identical tasks, and newer agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency. The largest divergence occurs between the declared and runtime dependency layers, implicating environment priors learned from the models' training distributions as the primary driver. Our findings establish environment specification as a distinct, measurable axis of code generation quality that current benchmarks do not capture, and motivate training objectives and evaluation protocols that jointly optimize for functional correctness and environmental portability.

1 Introduction

Functional-code benchmarks largely omit whether generated software has the dependencies and environment configuration needed to execute. The paper introduces an iterative evaluation framework and reports systematic, variable dependency misspecification across agents and ecosystems.

1 Introduction

  • Existing agent benchmarks focus almost exclusively on functional correctness, leaving dependencies, package versions, and system configuration unmeasured despite their necessity for execution.The paper argues that environments are distinct from source code because they manage external, stateful ecosystems prone to silent runtime failures and version conflicts.Source: §1 · p. 1
  • The framework evaluates initial environment generation, self-repair after failures, and over-specification against independently traced runtime dependencies.Its protocol prompts agents to generate code and a manifest, then iteratively diagnose and resolve environment failures when needed.Source: §1 · p. 2
  • Dependency declarations are inconsistent across agents on identical tasks, vary across independent runs, and show zero unanimous consensus across languages.The declared-to-runtime gap is wide, directionally biased, and ecosystem-dependent.Source: §1 · p. 2
  • The authors propose treating environment configuration as a core LLM training objective and measuring dependency precision, recall, and F1 against traced runtime ground truth.The supplied passage ends before completing the sentence about these metrics.Source: §1 · p. 2
  • The study spans three production coding agents, four language ecosystems, and 1,000 verified project instances with execution traces in isolated environments.The dataset is described as open-source and supports analysis across agents, languages, and task domains.Source: §1 · p. 2

2 Agent Evaluation Protocol

The protocol models environment generation as an initial code-and-manifest attempt followed, when execution fails, by iterative repair. It compares declared, installed, and runtime-traced dependencies to expose distinct gaps.

2 Agent Evaluation Protocol

  • Agents generate initial code and a dependency manifest, then use runtime errors to iteratively revise both until execution succeeds or the repair budget is exhausted.The loop stops at the first successful execution or ends after the allotted iterations.Source: §2 · pp. 2–3
  • The framework compares initial declared dependencies, packages installed after repair, and dependencies traced during successful runtime execution.The runtime-traced set is agent-independent and serves as a proxy for ground truth for the observed execution.Source: §2 · p. 3
  • Declared-versus-traced gaps identify phantom dependencies and hidden dependencies, while installed-versus-traced gaps reveal dependency bloat.Phantom dependencies are declared but unnecessary; hidden dependencies are needed but undeclared; bloat is installed but unused.Source: §2 · p. 3
  • Comparing initial declarations with the final installed set isolates packages added or removed through iterative repair.Figure 1 summarizes the pairwise comparisons and the gaps they expose.Source: §2 · p. 3

3 Protocol Primitives: Dependency Generation and Extraction

The protocol generates and repairs projects, then compares declared, installed, and runtime-traced dependencies. Its extraction primitives define each layer and the comparisons used to identify dependency gaps.

3.1 Dependency Generation Primitives

  • The generation prompt controls task and language framing, with implicit and explicit completeness instructions as two prompt variations.Explicit prompts request source code, a pinned manifest covering direct and transitive dependencies, and a README; passing the task’s tests validates prompt adequacy.Source: §3.1 · p. 3

3.2.1 Parse Primitive

  • Manifest parsing deterministically extracts declared packages from language-specific manifest fields, excluding JavaScript devDependencies.The rules cover Python requirements files, Java pom.xml, JavaScript package.json, and CMake declarations for C++.Source: §3.2.1 · p. 4

3.2.2 Resolve primitive

  • The package manager resolves the final manifest and reports installed packages, including transitive dependencies, while failures return to Repair.The framework measures transitive overhead as the additional installed set and an inflation ratio; C++ has no canonical package manager, so its comparison is uninformative by construction.Source: §3.2.2 · p. 5

3.2 Dependency Extraction Primitives

  • The protocol compares declared dependencies from the initial manifest, installed dependencies after resolving the final manifest, and dependencies loaded during successful execution.Runtime tracing defines the necessary-and-sufficient set for the observed execution.Source: §3 · p. 3, §3.2.1 · p. 4, §3.2.3 · p. 5

3.2.3 Trace primitive

  • Runtime tracing records dependency files accessed by the successfully executing program and maps those paths to package names for comparison with declarations and installations.The trace is language- and agent-independent, but requires successful resolution.Source: §3.2.3 · p. 5
  • A single execution can omit conditionally imported packages, and file-access tracing cannot observe header-only libraries compiled into the program.The authors assume conditional dependencies are rare and do not materially affect aggregate behavior across tasks.Source: §3.2.3 · p. 5

4 Experiment Setup

The study applies the protocol to 50 tasks in four language ecosystems using three coding agents, with 1,000 total runs. Runs use isolated Docker environments and manually conducted generation and repair interactions.

4.1 Protocol Instantiation and Execution

  • The study covers 50 platform-independent, self-contained tasks across Python, Java, JavaScript, and C++, spanning eight application domains.Tasks require no external services, APIs, or credentials, helping attribute dependency failures to agent generation and repair.Source: §4.1 · p. 6, Table L · p. 6
  • Three production coding agents receive every task-language prompt, yielding 600 primary runs; two additional independent trials per pair for one agent bring the total to 1,000.All primary runs use explicit prompts, so the reported misspecification rates are described as lower bounds; comparison with implicit prompts is left for future work.Source: §4.1 · p. 6, Table L · p. 6
  • Each run uses a per-language Docker container with only the language toolchain baseline, and each task starts in a fresh container.The 1,000 generation-and-repair interactions were conducted manually, including transferring artifacts, transcribing runtime errors, and recording outcomes.Source: §4.1 · pp. 6–7

4.2 Primitive Configuration

  • Tracing uses Sciunit system-call interception; Java fat JARs are unpacked for classpath tracing, while C++ traces shared libraries but misses header-only dependencies.Classpath-based Java execution produced functionally identical output to fat JAR execution in the traced cases.Source: §4.2 · p. 7

5 Evaluation Dimensions

The evaluation dimensions cover whether agents converge through repair and how their dependency specifications compare with runtime ground truth. They also assess cross-agent, cross-language, and task-complexity variation.

5 Evaluation Dimensions

  • Specification quality compares declared, installed, and runtime-traced sets against runtime ground truth, measuring phantom, hidden, bloated, and transitive dependencies.The framework reports precision, recall, F1, and the direction and magnitude of over- or under-specification.Source: §5 · p. 7
  • Protocol convergence measures whether agents reach successful execution within the repair budget and how many iterations they consume.Repair depth is the first iteration at which the runtime error is empty.Source: §5 · p. 7
  • The evaluation tests whether agents differ on identical task-language pairs and whether specification quality varies across the four language ecosystems.It also assesses whether quality degrades with task complexity or differs by programming domain.Source: §5 · pp. 7–8

6 Evaluation Results

The evaluation measures environment generation, repair, and dependency quality across agents, languages, and tasks. Results show that performance and dependency specification vary sharply by ecosystem, while declared dependencies are unstable across agents and runs.

6.1 Protocol Convergence

  • After repair, Claude converges in all four languages, Codex reaches 94% in C++, and Gemini reaches 96% in Java and 98% in C++; first-attempt success varies widely.Claude’s first-attempt success ranges from 18% in Java to 96% in JavaScript; most repairs finish within 1–3 cycles, and deep repair chains are rare outside C++.Source: §6.1 · p. 8, §6.2 · p. 9

6.2 Environment Specification Quality

  • JavaScript has about 49% phantom declarations, 62% hidden runtime packages, and 22% installed-but-unused packages, while Python and Java have about 5% phantom rates.JavaScript’s inflation ratio averages 11.40; Table 4 reports initial-manifest macro F1 of 0.765 for Python, 0.708 for Java, 0.219 for JavaScript, and 0.018 for C++.Source: §6.2 · pp. 9–10, Table L · p. 10

6.3 Cross- and Intra-Agent Comparison

  • Mean cross-agent Jaccard similarity is below 0.5 even in Python, and Claude’s unanimous three-trial consensus rate is 0% in every language.Complete disjointness averages 53.3% across ecosystems; JavaScript reaches 63.3%, and Codex and Gemini share a mean Jaccard similarity as low as 0.073 there.Source: §6.3 · p. 11, Table L · p. 11

6.4 Cross-Language Variation

  • Python combines the strongest initial-manifest accuracy with mild inflation, while JavaScript has the weakest accuracy and greatest inflation; C++ failures often reflect unavailable system libraries.Claude’s C++ system-library assumption rate is 87.5%, and all such failures are repaired after feedback; Codex’s rate is 17.9%, also with perfect recovery.Source: §6.4 · pp. 11–12, Table L · p. 11

6.5 Task Complexity Sensitivity

  • Final success is at least 95% for every agent–domain pair, but first-attempt success ranges from 40% to 88%, with the weakest domain differing by agent.On tasks solvable with standard libraries alone, Claude still declares external packages in 91.8% on macro average, including 100% in C++.Source: §6.5 · p. 12, §6.4 · p. 12

6.6 Limitations

  • C++ runtime tracing can miss header-only and compile-time dependencies, and the 50-task controlled benchmark, fixed prompt, manual judgments, and evaluated agent versions limit generalization.The authors characterize findings as evidence about structured code-generation settings, not exhaustive estimates for all software development contexts.Source: §6.6 · p. 12

7 Related Work

Prior work largely evaluates isolated-snippet correctness, while dependency specification across the declaration-to-runtime lifecycle remains underexamined. This paper positions its cross-ecosystem measurements as relevant to manifest accuracy, runtime provenance, and multidimensional LLM evaluation.

LLM Evaluation Methodology

  • Most benchmarks assess functional correctness of isolated snippets, leaving dependency stability and management across declaration, resolution, and runtime underexamined.The paper distinguishes its focus on existing-package misspecification from package hallucinations and earlier work on output non-determinism.Source: §7 · p. 13

JavaScript/npm

  • JavaScript’s mean inflation ratio is 11.40, with some three-package declarations expanding beyond 140 installed packages.Prior work describes npm’s deep dependency graphs; the paper measures their expansion in agent-generated projects.Source: §7 · p. 13

Java/Maven

  • Agent-generated Java projects show pervasive dependency bloat, extending prior studies of bloated dependencies in human-authored Maven artifacts.The paper notes that Maven’s transitive resolution can obscure the footprint induced by a small set of declared dependencies.Source: §7 · p. 13

C++/CMake

  • The paper measures C++ system-library assumption gaps across agents and reports that they account for most C++ build failures.It links these assumptions to agents’ exposure to code from richer development environments and reports differences across agents.Source: §7 · p. 13

Python/pip

  • Even in comparatively tractable Python, agent manifests reach macro F1 of 0.765 against runtime ground truth.The passage also notes native-extension system-library requirements that can still create environment issues.Source: §7 · p. 13

Cross-ecosystem analysis

  • The paper presents a unified comparison of dependency specification across Python, Java, JavaScript, and C++, rather than treating each ecosystem separately.It frames ecosystem-specific resolution semantics and failure modes as important to evaluating generated software.Source: §7 · p. 13

Software Bills of Materials and Manifest Accuracy

  • The declared dependency set can be evaluated as an agent-generated SBOM by measuring its precision, recall, and F1 against runtime ground truth.The paper reports macro F1 ranging from 0.765 for Python to 0.018 for C++ and discusses implications for SBOM trustworthiness.Source: §7 · p. 14

LLM Evaluation Methodology

  • The framework treats environment specification as a distinct evaluation dimension, separating declaration accuracy, resolution behavior, and runtime footprint.The paper uses Sciunit runtime tracing as a dependency oracle and dynamic tracing to capture what programs load on exercised execution paths.Source: §7 · pp. 13–14

8 Conclusion

The paper concludes that coding agents systematically produce inconsistent, incomplete, or excessive dependency specifications, even when projects can be repaired to functional success. It argues that evaluation and training should target dependency accuracy alongside task completion.

8 Conclusion

  • Current coding agents systematically misspecify dependencies through both under-specification and over-specification, with reliability varying across agents and ecosystems.Projects may reach functional success after repair even though their dependency specifications remain incorrect or nonminimal.Source: §8 · p. 14
  • Successful code generation alone is not evidence of environment correctness, so dependency specification needs explicit validation alongside functional testing.The paper identifies environment-grounded fine-tuning, dependency-aware rewards, and manifest verification as possible directions.Source: §8 · p. 14
  • The paper calls for evaluation frameworks and training objectives that jointly address functional task completion and accurate, minimal dependency specification.Its stated goal is more reliable and portable code generation.Source: §8 · p. 14

A Supplementary Structural Metrics

The appendix examines whether declared dependency-set size is associated with first-attempt success, using Spearman correlations by ecosystem.

Supplementary Structural Metrics

  • The appendix presents these analyses as supplementary results supporting interpretation of the primary findings.It also cross-references the eight primary metrics with the subsections where their results appear.Source: A Supplementary Structural Metrics · p. 16
  • The appendix tests the association between declared dependency-set size and round-zero success using Spearman rank correlation for each ecosystem.A negative correlation is described as indicating that manifest bloat may be a source of build fragility.Source: A Supplementary Structural Metrics · p. 16
  • Larger declared dependency sets are associated with lower first-attempt success in Python, Java, and C++, while JavaScript’s observed association is not statistically meaningful in this benchmark.The strongest negative association is reported for C++.Source: A Supplementary Structural Metrics · p. 16

B Complete Problem List

The appendix includes ranked diagnostics of recurring phantom and hidden packages, alongside tables listing benchmark problems and package frequencies.

Complete Problem List

  • Ranked phantom and hidden package frequencies are used as qualitative diagnostics of recurring over-declaration and omission patterns.Examples include phantom pycparser, yargs, and nlohmann_json, and hidden six and Java Jackson packages.Source: B Complete Problem List · p. 16
  • The appendix includes a table covering all 50 programming problems.Source: Table L · p. 17
  • Separate tables list the most frequent phantom and hidden packages by ecosystem.Source: Table L · p. 17

Related papers

Loading 2610.00425v1…