Source-linked AI summary
R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, Ion Stoica
TL;DR
Open-weight SWE agents lag behind proprietary models, while scalable executable training environments and effective test-time compute scaling remain challenging. This paper introduces R2E-GYM, SWEGEN synthetic data curation, and hybrid execution-based and execution-free scaling, achieving 51% on SWEBENCH-VERIFIED with an open-weights 32B model.
Problem
Open-weight SWE agents lag behind proprietary models, while scalable curation of executable training environments and effective test-time compute aggregation remain challenging.
Method
The paper introduces R2E-GYM, uses SWEGEN to curate executable environments from commits through backtranslation and test collection or generation, and combines execution-based testing with execution-free verifiers for test-time scaling.
Results
51% on SWEBENCH-VERIFIED was achieved by the open-weights 32B model, establishing a new state-of-the-art for open-weight SWE agents.
Takeaways & Limitations
Synthetic data curation enables more scalable SWE-agent training, while complementary verifier strengths can produce higher test-time gains.
Takeaways & Limitations
Execution-based verifiers have limited discriminative capacity because most problems have fewer than 20% of tests providing discriminative signal.
Abstract
from arXiv · showhide
Improving open-source models on real-world SWE tasks (solving GITHUB issues) faces two key challenges: 1) scalable curation of execution environments to train these models, and, 2) optimal scaling of test-time compute. We introduce AgentGym, the largest procedurally-curated executable gym environment for training real-world SWE-agents, consisting of more than 8.7K tasks. AgentGym is powered by two main contributions: 1) SYNGEN: a synthetic data curation recipe that enables scalable curation of executable environments using test-generation and back-translation directly from commits, thereby reducing reliance on human-written issues or unit tests. We show that this enables more scalable training leading to pass@1 performance of 34.4% on SWE-Bench Verified benchmark with our 32B model. 2) Hybrid Test-time Scaling: we provide an in-depth analysis of two test-time scaling axes; execution-based and execution-free verifiers, demonstrating that they exhibit complementary strengths and limitations. Test-based verifiers suffer from low distinguishability, while execution-free verifiers are biased and often rely on stylistic features. Surprisingly, we find that while each approach individually saturates around 42-43%, significantly higher gains can be obtained by leveraging their complementary strengths. Overall, our approach achieves 51% on the SWE-Bench Verified benchmark, reflecting a new state-of-the-art for open-weight SWE-agents and for the first time showing competitive performance with proprietary models such as o1, o1-preview and sonnet-3.5-v2 (with tools). We will open-source our environments, models, and agent trajectories.
1 Introduction
R2E-Gym addresses open-weight SWE agents’ lag behind proprietary models by tackling scalable environment curation and test-time compute scaling. Its synthetic curation recipe and hybrid verifier strategy support stronger training and inference-time performance.
- Open-weight SWE agents lag behind proprietary models despite progress on real-world software-engineering tasks.
- Scalable training requires executable environments, which existing resources often lack or curate only through human-written issues and tests.
- R2E-Gym provides more than 8.1K procedurally curated problems with executable environments, unit tests, and natural-language task descriptions.
- SWEGEN curates executable environments from commits through backtranslation and test collection or generation, reducing reliance on human-written PRs and unit tests.
- 34.4% PASS@1 is achieved on SWEBENCH-VERIFIED by the resulting training approach.
- Execution-based and execution-free verifiers have complementary strengths: direct correctness signals versus stronger discrimination but heuristic bias.
- Hybrid inference-time scaling combines both verifier types, improving performance beyond either approach’s roughly 42–43% plateau.
- The open-weights 32B model achieves 51% on SWEBENCH-VERIFIED, reported as state-of-the-art for open-weight SWE agents and competitive with commercial models.
2 R2E-GYM: Procedural Synthetic Data Generation
SWEGEN procedurally builds executable SWE-agent environments from repository commits, tests, and generated problem statements rather than depending on human-written issues. The resulting collection exceeds 8.1K problems and provides over 2.5 times more problems than issue-based collection.
- SWEGEN uses backtranslation and test generation to procedurally create environments from commits, reducing reliance on human-written issues and test cases.
- Repository and commit curation identifies Python repositories with many commits, extracts code changes, filters relevant commits, and collects build scripts.
- Existing Fail→Pass tests are collected from curated commits, while automatically generated Fail→Pass tests supplement commits without associated tests.
- F2P test cases are included in backtranslation prompts to produce higher-quality problem statements that capture the code changes.
- Over 8.1K problem statements are collected; decontamination yields 4578 problems in R2E-Gym-Subset across non-overlapping repositories.
- Over 2.5 times more problems are collected with SWEGEN than through data collection relying on GITHUB issues.
3 Training SWE-Agents using R2E-GYM Environments
R2E-GYM training uses collected agent trajectories to improve SWE-agent performance, with gains from scaling data, synthetic problem statements, and explicit thought traces.
- Training Setup: The training pipeline collects 3,321 successful trajectories from 2,048 task environments and uses them for supervised fine-tuning of 7B, 14B, and 32B Qwen-2.5-Coder models.The trajectories come from R2E-Gym-Subset, which contains 4,578 executable environments across 10 repositories.
- Results: 34.4% PASS@1: the R2E-GYM-trained 32B model improves over SWE-Gym’s 20.6% baseline on SWEBENCH-VERIFIED.The comparison uses the same base-model type and scale, representing a 14% improvement.
- Results: Performance improves as trajectory counts increase, but returns diminish; the 14B model begins saturating near 800 samples while the 32B model continues improving.The study evaluates trajectory counts from 100 to 3,200 and reports stronger continued gains for the larger model.
- Results: 27.8% PASS@1 versus 28.0%: synthetic problem statements perform nearly identically to real GitHub issues when each supplies 400 trajectories.Synthetic data also provides greater scalability by avoiding reliance on human-written descriptions and test cases.
- Results: 34.2% versus 30.4%: training with explicit thought demonstrations outperforms training without them.The training targets include both agent thought processes and actions.
4 Efficient Inference Time Scaling With Hybrid Verifiers
The paper compares execution-based and execution-free verifiers for inference-time scaling, then combines their complementary strengths in a hybrid verifier. Hybrid verification improves scaling beyond the individual methods, while targeted test-agent rollouts can provide more compute-efficient gains.
- Verifier Designs: Execution-based verifiers generate reproduction tests and apply regression-test filtering, while execution-free verifiers score trajectories using learned models.Execution-based scores combine generated-test results with regression-test scores; execution-free models assess task descriptions, trajectories, and patches.
- Comparative Scaling: 43.7% and 42.8% are the respective BEST@K plateaus for execution-based and execution-free verifiers as editing-agent rollouts increase.Both methods improve with additional rollouts, but their performance quickly saturates at similar levels.
- Verifier Limitations: Less than 20% of tests provide discriminative signal for most problems, limiting execution-based re-ranking.Execution-based verifiers also face rare toxic tests that pass incorrect patches while failing correct ones.
- Verifier Limitations: Removing agent thoughts reduces execution-free BEST@26 from 42.8% to 37.6%, indicating reliance on trajectory information beyond final patches.The ablation shows that execution-free verification can depend on heuristics encoded in agent thoughts.
- Hybrid Scaling: The hybrid verifier adds 7-8% performance over the individual methods by combining execution-based and execution-free verification.The hybrid approach exhibits substantially better scaling as agent rollouts increase.
5 Related Work
Prior SWE-agent research spans programming agents, executable training environments, and verifiers, but open-weight systems remain constrained by limited data and analysis of verification strategies. This work extends these directions with synthetic executable-environment collection and hybrid verification.
- Programming Agents: Existing GITHUB issue-resolution agents largely rely on proprietary models because open-weight models and datasets are scarce.
- Agent Training Environments: SWE-Bench lacks executable training environments, R2E contains 246 function-completion instances, and SWE-Gym relies on human-written issues and tests.
- SWE-Agent Training: Prior work studies synthetic code-editing tasks, agent trajectories, inference scaling, and reinforcement learning on real-world GITHUB issues.
- Verifiers for SWE-Coding Tasks: SWE-task verifiers include majority voting, reproduction and regression tests, LLM committee review, and trajectory-based patch reranking.
- Verifiers for General Coding Tasks: Verifier research also covers isolated programming puzzles, interviews, competitions, and test generation, while LLM judges can perform poorly on code correctness.
- Extension: This work analyzes execution-based and execution-free SWE verifiers and combines their strengths, achieving 51.0% on SWE-BENCH-VERIFIED.
6 Conclusion
R2E-Gym is presented as a training framework for scaling open-weight SWE agents through synthetic data curation and hybrid test-time scaling. The final approach achieves 51% on SWE-Bench Verified and competitive performance with some proprietary models.
- R2E-Gym is a gym environment and training framework for scaling open-weight SWE agents.
- Synthetic data curation enables more scalable training on software-engineering tasks.
- Hybrid test-time scaling combines execution-based testing agents and execution-free verifiers because their strengths are complementary.
- 51% on SWE-Bench Verified is reported as a new state-of-the-art for open-weight SWE agents and competitive with some proprietary models.
A Dataset Details
The dataset pipeline procedurally filters commits, reconstructs historical environments, generates synthetic issues and tests, minimizes patches, and applies quality filtering. It produces executable environments and synthetic issues intended to support scalable SWE-agent training.
- Dataset Overview: 8,135 instances comprise the complete R2E-Gym dataset across repositories.
- Commit Filtering: Commit filtering prioritizes small, non-documentation changes with correlated code and test matches, using line- and AST-level heuristics.
- Commit Filtering: The pipeline limits commits to 5 non-test files, 100 edited lines, 2,000 characters, 1 deletion, 3 additions, 3 edits, and 10 statement-level changes.
- Repository Installation: Historical commits are installed in Docker using search-based dependency resolution that tests multiple dependency configurations until one works.
- Repository Installation: The installation process is semi-manual and challenging to scale, motivating greater future reliance on LLMs.
- Issue and Test Generation: SWEGEN backtranslates commits into synthetic issues using code changes and test results, while test generation uses the ground-truth patch as context.
- Issue Generation: Synthetic issue prompts include failing tests, error messages, expected behavior, commit context, patches, old and new executions, and relevant test functions.
- Issue Generation: The resulting issues are designed to be concise, natural, informative, and solution-free, and an LLM judge further filters the dataset.
B SFT Training
The training setup uses executable R2E-Gym tasks to train code-editing and testing agents, alongside execution-free outcome-supervised verifiers. Agents operate end-to-end with lightweight tools, while verifiers score candidate patches from trajectories without execution.
- Code-Editing Agent: The code-editing agent receives an executable environment and problem description, then solves the issue end-to-end by reproducing, localizing, and fixing the bug.
- Agent and Tools: The agent uses a REACT scaffold with file editing, search, bash execution, and submission tools, without internet or browser access.
- Testing Agent: The testing agent generates M = 10 diverse reproduction tests to assess whether a candidate patch resolves the issue.
- Testing Agent: Testing-agent training uses 2,203 Sonnet trajectories with at most 40 steps, 20K tokens, five minutes per trajectory, and 60 seconds per action.
- Execution-Free Verifier: The execution-free verifier scores a problem, trajectory, and output patch with sEF ∈[0, 1], predicting the probability that the patch is correct.
- Execution-Free Verifier: Verifier training uses balanced positive and negative samples from 5,700 trajectories, including on-policy trajectories from the trained 32B model.
C.3 Execution-Based Analysis
Execution-based verification is assessed through distinguishability and toxicity of generated tests. Most tests provide weak discrimination, while a smaller toxic subset can undermine reliability and motivate combining execution-based and execution-free verification.
- Metrics: Distinguishability measures whether a test behaves differently on correct and incorrect patches.A test that passes correct patches and fails incorrect patches has perfect distinguishability; identical behavior provides no useful signal.
- Findings: Less than 20% of generated tests typically distinguish effectively between correct and incorrect patches.This limits execution-based verification’s ability to identify the best patch, especially as candidate-patch counts increase.
- Metrics: Toxicity measures the proportion of tests that incorrectly favor incorrect patches over correct ones.Toxic tests pass incorrect patches but fail correct patches.
- Findings: Toxicity reaches up to 10% for some problems, despite toxic tests generally being rare.Such tests can significantly affect verification reliability and support combining execution-based verification with other approaches.
C.4 Execution-Free Analysis
The execution-free verifier has a documented limitation: its assessment can be biased by an agent’s thoughts and actions rather than relying on the final output patch.
- Limitation: Execution-free verification can be biased by the agent’s thoughts and actions.The limitation concerns what the verifier attends to when assessing trajectories.
- Limitation: The stated limitation is a mismatch between trajectory-derived signals and final-patch assessment.The passage specifically contrasts agent thoughts and actions with the final output patch.
D Example Testing Agent Outputs
This section presents examples of generated testing-agent outputs and visual analyses of agent evaluation and behavior. The examples include verifier scaling and a short successful trajectory using the 32B model.
- Overview: The section provides examples of test cases generated by the approach.
- Verifier analysis: Figure 13 analyzes the four sliding windows over an agent trajectory with the highest mean attention score.The visualization concerns execution-free verifier limitations on a SymPy SWE-Bench task.
- Evaluation examples: Figure 14 plots Pass@K for the authors’ agent and Agentless tests, while Figure 15 shows the problem statement presented to the agent.
D.1 Example 1: SymPy Relational Parsing Tests
The examples illustrate generated tests for SymPy relational parsing and Django ModelChoiceField validation, alongside a visualized agent trajectory from issue description to verified solution. They show both successful detection and failure modes involving unhandled exceptions and nondistinguishing tests.
- D.1 Example 1: SymPy Relational Parsing Tests: Generated SymPy tests target relational parsing in response to SymPy PR #24661.The example demonstrates multiple test cases addressing a library issue.
- D.1 Example 1: SymPy Relational Parsing Tests: The truncated SymPy test suite successfully detects incorrect code from correct code.
- D.1 Example 1: SymPy Relational Parsing Tests: Generated Django tests target ModelChoiceField validation and include setup and teardown code for Django PR #13933.
- D.1 Example 1: SymPy Relational Parsing Tests: Most Django test cases error because of unhandled exceptions and do not distinguish between patches.
- Trajectory example: The visualized trajectory presents the agent’s step-by-step process from problem statement through solution while solving a software-engineering task.The trajectory is described as a visual example of the agent’s approach to real-world programming issues.
- Trajectory example: The trajectory emphasizes systematic codebase exploration, root-cause analysis using issue.py, and planning before implementation.These capabilities are described as enabling the agent to tackle complex software-engineering tasks involving code structure, language semantics, and design principles.