Source-linked AI summary
MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering
Chuanzhe Guo, Jingjing Wu, Sijun He, Yang Chen, Zhaoqi Kuang, Shilong Fan, Bingjin Chen, Siqi Bao, Jing Liu, Hua Wu, Qingfu Zhu, Wanxiang Che, Haifeng Wang
TL;DR
Scalable SWE evaluation and training require executable environments, but constructing them across languages is complex and costly. MEnvAgent combines iterative multi-agent repair with reuse of historical environments, and on MEnvBench it improves F2P rates by 8.6% while reducing time costs by 43%.
Problem
Scalable SWE evaluation and training are constrained by the complexity and cost of constructing executable environments across diverse languages.
Method
MEnvAgent uses a multi-agent Planning-Execution-Verification architecture and incrementally patches retrieved historical environments.
Results
MEnvAgent improves Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43% on MEnvBench.
Takeaways & Limitations
MEnvAgent supports scalable construction of verifiable polyglot SWE data, including MEnvData-SWE and solution trajectories associated with downstream SWE gains.
Takeaways & Limitations
Automated environment setup and code execution may enable harmful software or unauthorized scripts, and erroneous commands may have severe consequences.
Abstract
from arXiv · showhide
The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages. To address this, we introduce MEnvAgent, a Multi-language framework for automated Environment construction that facilitates scalable generation of verifiable task instances. MEnvAgent employs a multi-agent Planning-Execution-Verification architecture to autonomously resolve construction failures and integrates a novel Environment Reuse Mechanism that reduces computational overhead by incrementally patching historical environments. Evaluations on MEnvBench, a new benchmark comprising 1,000 tasks across 10 languages, demonstrate that MEnvAgent outperforms baselines, improving Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43%. Additionally, we demonstrate the utility of MEnvAgent by constructing MEnvData-SWE, the largest open-source polyglot dataset of realistic verifiable Docker environments to date, alongside solution trajectories that enable consistent performance gains on SWE tasks across a wide range of models. Our code, benchmark, and dataset are available at https://github.com/ernie-research/MEnvAgent.
1. Introduction
MEnvAgent addresses the difficulty and cost of constructing executable environments for multilingual SWE tasks through automated multi-agent construction and environment reuse. Evaluations and dataset construction show broad benchmark coverage and downstream utility.
- Motivation: Executable environment construction limits scalable evaluation and training of software-engineering agents.Existing workflows require repository exploration, patch generation, and test execution, while scalable environment construction remains constrained.
- Challenges: Frequent dependency conflicts, compilation errors, inconsistent tests, and clean-slate rebuilds make construction complex and time-consuming.These failures reduce success rates and create prohibitive overhead for large-scale data expansion.
- Approach: MEnvAgent combines a multi-agent Planning-Execution-Verification loop with an Environment Reuse Mechanism.The loop resolves construction failures, while reuse adapts historical environments through incremental patches instead of rebuilding from scratch.
- Evaluation: MEnvBench contains 1,000 tasks across 10 languages with execution-based evaluation and quality assurance.The benchmark is designed to evaluate multilingual executable environment construction rigorously.
- Results: MEnvAgent improves Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43% against state-of-the-art baselines.The framework is also used to construct MEnvData-SWE, a realistic verifiable SWE training dataset with solution trajectories.
2. Problem Formulation
The paper formalizes environment construction as producing a configuration that makes a fixed issue-fixing task executable and verifiable. Validity requires both failure on the original repository state and success after applying the fix.
- Task Definition: A verifiable task instance contains a repository snapshot, an issue with its related pull request, a fix patch, and a test patch.Rfix denotes the repository state after applying the fix patch.
- Task Definition: Environment construction determines a triplet (B, P, T) comprising a base image, build process, and test configuration.The environment state is defined as S = δ(B, P).
- Executability: The executability condition requires the fixed repository state Rfix to pass the configured tests.The paper denotes successful test execution with ε(·)=0.
- Validity: The Fail-to-Pass criterion requires the original state to fail while Rfix passes: ε(R, S, T) = 1 ∧ ε(Rfix, S, T) = 0.This differential outcome checks both reproduction of the issue and verification of its resolution.
3. MEnvAgent Design
MEnvAgent constructs environments through iterative planning, execution, and verification, then accelerates construction by retrieving and incrementally adapting previously verified environments.
- Multi-Agent Architecture: MEnvAgent has three iterative stages: Planning, Execution, and Verification.The architecture is designed both to construct executable environments and to resolve construction failures.
- Planning: Planning agents analyze the repository, select a base image, and generate the installation script.Repository analysis summarizes project type, dependencies, and entry points for downstream setup decisions.
- Execution: During execution, the system runs installation commands, repairs immediate errors, and returns to planning after repeated unresolved failures.Examples include missing packages and version conflicts.
- Verification: Verification runs the configured tests and attributes failures to missing dependencies or incorrect test commands.Diagnostic feedback is propagated back to planning for iterative refinement.
- Environment Reuse: The Environment Reuse Mechanism retrieves a similar verified environment and adapts it with an incremental patch ΔP.Retrieval prioritizes version consistency and same-repository history, while verification-driven adaptation either reuses the environment directly or synthesizes further commands.
4. MEnvBench Construction
MEnvBench is built through a quality-controlled pipeline targeting broad multilingual and repository coverage, then evaluates executability, validity, and construction efficiency. Its performance analysis compares pass outcomes against time cost.
- Pipeline: MEnvBench is constructed through a strict pipeline intended to ensure execution validity, quality, and broad representativeness.The benchmark is explicitly designed to address limitations of existing benchmarks.
- Pipeline: The acquisition pipeline transforms raw GitHub data into a high-quality candidate pool in two phases.Repository acquisition is followed by instance extraction and quality assurance.
- Repository Acquisition: 8,000 repositories were selected across 10 mainstream languages using thresholds for stars, forks, issues, pull requests, and language concentration.The criteria were intended to reduce failures caused by inherent code defects.
- Instance Quality: 213,766 issue instances remained after retaining linked closed issues with test patches and filtering low-quality issues.The extraction covered issue-pull-request pairs from 2018–2025.
- Benchmark Composition: MEnvBench comprises 1,000 tasks sampled across languages, repositories, historical versions, domains, and repository-size bands.The allocation balances inter-project breadth with intraproject variability.
- Evaluation Metrics: The benchmark reports Pass Rate, Fail-to-Pass Rate, and Time Cost as measures of executability, validity, and efficiency.Figure 3 places average time cost on the x-axis and pass rate or F2P rate on the y-axis.
5. Experiment
MEnvBench evaluates MEnvAgent against baselines across diverse languages, measuring environment-construction effectiveness and efficiency. MEnvAgent occupies the strongest efficiency-quality position and improves key metrics over SWE-Factory.
- Experimental Design: MEnvBench evaluates environment-construction success and computational efficiency across 10 programming languages.The benchmark comparison targets state-of-the-art baselines using aggregated metrics.
- Experimental Design: The experiments include Repo2Run, SWE-Bench-Live, and SWE-Factory as comparison baselines with different language coverage.Repo2Run is Python-specialized, SWE-Bench-Live supports six languages, and SWE-Factory is evaluated across all ten.
- Overall Results: MEnvAgent consistently achieves high validity with low computational overhead across backbone models and programming languages.In the efficiency-quality plots, it occupies the upper-left quadrant, while baselines show either higher latency or lower validity.
- Data-Scale Analysis: Increasing instances per repository from 1 to 10 improves reuse probability and overall efficiency.Figure 4 tracks Reuse Success Rate, Time Cost, and Pass Rate as data scale increases.
- Overall Results: 8.6% higher strict F2P Rate, 11.0% higher Pass Rate, and 43.0% lower time cost are achieved against SWE-Factory when averaged across models.Table 2 provides the direct comparison with the strongest baseline.
6. Analysis
The analyses examine component ablations, data and repository scale, language-specific failures, and downstream SWE training utility. Environment reuse and patching improve efficiency and pass rates, while dataset fine-tuning boosts performance across evaluated models.
- Ablation Study: 39.0% to 25.0% is the Reuse Success Rate drop without EnvPatchAgent, accompanied by a 20% increase in time cost.The direct-reuse variant frequently falls back to scratch builds, showing the contribution of incremental patching.
- Ablation Study: 46.0% lower average computational time and 18.5% higher Pass Rate distinguish the full framework from the no-reuse baseline.The reported gains are attributed to avoiding error-prone dependency resolution from scratch.
- Data-Scale Analysis: 39% Reuse Success Rate is reached at 10 instances per repository, with concurrent Pass Rate improvements and Time Cost reductions.At one historical instance, reuse is negligible and performance resembles the scratch baseline.
- Repository-Scale Analysis: F2P rates decrease significantly as repository size increases, consistent with larger projects having more intricate dependency graphs and build overhead.These characteristics increase the complexity of automated environment configuration.
- Language-Specific Analysis: Go and Python show high F2P resolution rates, whereas Java exhibits greater ecosystem difficulty and C/C++ frequently encounter compilation and timeout failures.Gemini-3-Flash has lower setup failure rates than Kimi-K2 across most languages, with the largest difference in Java.
- Downstream Utility: Fine-tuning on MEnvData-SWE consistently boosts performance across evaluated models on SWE-bench Verified and SWE-bench Multilingual.Qwen2.5-Coder-32B matches GPT-4.1 on SWE-bench Verified and outperforms it on the Multilingual benchmark.
7. Related Work
Prior environment-construction approaches differ in automation strategy, language coverage, and verification scope. Existing benchmarks increasingly use execution-based evaluation but remain limited in coverage or completeness.
- Automated Environment Construction: Repo2Run uses a dual-agent, Python-specific framework focused on environment installation with fixed test commands and no verification-test execution.Its scope is narrower than multilingual environment construction.
- Automated Environment Construction: SWE-Bench-Live supports environment setup and test configuration through a single-agent interactive-bash method.The framework extends beyond installation but retains a single-agent construction approach.
- Automated Environment Construction: SWE-Factory broadens automated environment construction to four programming languages with a collaborative multi-agent architecture.It represents a wider-scope prior framework than Python-specialized approaches.
- Environment Construction Benchmarks: INSTALLAMATIC and EXECUTIONAGENT establish execution-based standards for evaluating environment construction, while dedicated benchmarks still have coverage limitations.Earlier comprehensive-task evaluations provided little insight into isolated construction challenges.
8. Conclusion
MEnvAgent automates polyglot environment construction with Planning-Execution-Verification and environment reuse. On MEnvBench, it achieves higher F2P rates and lower time costs, and supports construction of MEnvData-SWE.
- MEnvAgent automates complex environment construction through a multi-agent Planning-Execution-Verification architecture.
- Its environment reuse mechanism reduces computational overhead by adapting historical environments instead of rebuilding from scratch.
- MEnvAgent achieves significantly higher F2P rates and lower time costs than state-of-the-art baselines on MEnvBench’s 1,000 tasks across 10 languages.
- MEnvAgent constructs MEnvData-SWE, whose experiments demonstrate consistent performance gains for models on downstream SWE tasks.
Impact Statement
The paper positions MEnvAgent as infrastructure for scalable, execution-verified polyglot software-engineering research. It describes both its broader utility and safety boundary: automated execution should remain isolated in Docker sandboxes.
- Impact Statement: MEnvAgent is intended to lower barriers to large-scale, polyglot software-engineering studies by automating environment construction.
- Impact Statement: Its 10-language execution-verified data construction can support more robust and versatile coding agents.
- Impact Statement: The framework carries risks because malicious users could create harmful software environments or unauthorized scripts, while erroneous commands may cause severe consequences.
- Impact Statement: All tasks run in an isolated Docker sandbox by default, and users are strongly advised to retain this configuration for safety and isolation.
- Impact Statement: MEnvAgent can accelerate benchmark maintenance by enabling continuous updates with fresh real-world repositories.
- Impact Statement: Unlike synthetic mutations or limited base environments, MEnvAgent scales training data from diverse real-world scenarios and can support other generation pipelines.
- Task and Environment Definition: The construction objective is to find (B, P, T), where B is the base image, P the installation sequence, and T the test configuration.
F. Detailed Results on MEnvBench
MEnvAgent delivers strong performance and cost efficiency across MEnvBench’s 10-language evaluation. Its gains are measured through F2P, PASS, TIME, and token-cost analyses against multiple baselines and backbones.
- Performance: MEnvAgent consistently achieves superior stability and resolution rates compared with baselines across all 10 evaluated programming languages.
- Performance: MEnvAgent’s F2P, PASS, and TIME results are evaluated using both Kimi-K2 and Gemini-3-Flash backbones.
- Cost: MEnvAgent maintains minimal token consumption per task, with only marginally higher cost than SWE-Factory.
- Cost: The reported cost increment over SWE-Factory accompanies gains in F2P rate and time efficiency.
G. Cross-Language Ablation Study on Environment Reuse
The cross-language ablation evaluates environment reuse on 200 tasks spanning 10 programming languages. Reuse improves pass rates consistently across languages while reducing execution time.
- Study Setup: The ablation study covers 200 tasks drawn from 10 programming languages, with four repositories and five instances per repository.
- Results: 21.5% average Reuse Success Rate accompanies an +8.5% absolute improvement in instance-level PASS across all 10 languages.
- Results: The PASS improvement holds consistently on a per-language basis across the polyglot benchmark.
- Results: Approximately 19% lower average overall execution time demonstrates the reuse mechanism’s computational efficiency.
H. Detailed Failure Analysis on C++ Tasks
C++ environment construction remains a major bottleneck, driven by compilation, build-system, dependency, library, and network failures. The pipeline addresses scalability through automated construction and strict execution-based verification, producing a multilingual dataset with broad ecosystem coverage.
- C++ Failure Analysis: 26% PASS with Gemini-3-Flash and 23% with Kimi-K2 in C++ demonstrate a substantial bottleneck relative to Python's 66%.The analysis identifies compiled languages, especially C++ and C, as the most difficult environments to construct.
- C++ Failure Analysis: 22% of failures arise from compilation OOM, while 20% involve incompatibility with non-standard build systems.Heavy template expansion and parallel compilation can exhaust sandbox memory, while tools such as Bazel, Nix, SCons, and Colcon require contextual setup logic.
- C++ Failure Analysis: 18% of failures involve heavy dependency installation, with additional failures from missing system libraries and network download timeouts.Large upstream dependencies can trigger package-manager hangs, network drops, version conflicts, or broken mirrors; library naming also varies across distributions.
- C++ Failure Analysis: The authors characterize these bottlenecks as structural and infrastructural rather than purely algorithmic, motivating more flexible sandboxes and specialized toolsets.The proposed future direction includes improved resource allocation and tool support for systemic failures.
- Dataset Construction: MEnvData-SWE retains only instances that reproduce failure before the fix and pass after the fix, yielding reproducible executable bug-fix scenarios.The dataset pipeline uses test patches for the failing state and both test and fix patches for the passing state.
- Dataset Construction: MEnvData-SWE covers multiple ecosystems and maintains representation for Rust, JavaScript, C, Java, and PHP, supporting multilingual generalization.The dataset is described as the largest open-source polyglot collection of realistic verifiable Docker environments, with verified solution trajectories.