Source-linked AI summary
Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards
Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, Sean Hendryx
TL;DR
RLVR has succeeded in verifiable reasoning tasks, but agentic settings require multi-step interaction and make effective reward-driven training difficult. Agent-RLVR guides agents through failed attempts and uses environment rewards for training. On SWE-Bench Verified, Qwen-2.5-72B-Instruct pass@1 rises from 9.4% to 22.4%.
Problem
In agentic settings, multi-step interaction and complex problem solving make successful trajectories less likely and RLVR training difficult.
Source: §1 · p. 2Method
Agent-RLVR uses guidance about failed patches and environment interactions to steer reattempts, then trains from environment-verified trajectories.
Source: §1 · p. 2, §2.2.2 · pp. 4–5Results
9.4% to 22.4%: Qwen-2.5-72B-Instruct pass@1 on SWE-Bench Verified improves with Agent-RLVR; reward-model ranking further raises pass@1 from 22.4% to 27.8%.
Source: §3.2 · pp. 6, 8Takeaways & Limitations
The paper reports that guidance reduces reward sparsity and improves agent PASS@k, and that guidance-based RL data can train a reward model for test-time patch selection.
Source: §2.2.2 · p. 5, §6 · p. 9The training data covers a relatively small set of repositories, and possible pre-training contamination remains despite disjoint training and test repositories.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
Reinforcement Learning from Verifiable Rewards (RLVR) has been widely adopted as the de facto method for enhancing the reasoning capabilities of large language models and has demonstrated notable success in verifiable domains like math and competitive programming tasks. However, the efficacy of RLVR diminishes significantly when applied to agentic environments. These settings, characterized by multi-step, complex problem solving, lead to high failure rates even for frontier LLMs, as the reward landscape is too sparse for effective model training via conventional RLVR. In this work, we introduce Agent-RLVR, a framework that makes RLVR effective in challenging agentic settings, with an initial focus on software engineering tasks. Inspired by human pedagogy, Agent-RLVR introduces agent guidance, a mechanism that actively steers the agent towards successful trajectories by leveraging diverse informational cues. These cues, ranging from high-level strategic plans to dynamic feedback on the agent's errors and environmental interactions, emulate a teacher's guidance, enabling the agent to navigate difficult solution spaces and promotes active self-improvement via additional environment exploration. In the Agent-RLVR training loop, agents first attempt to solve tasks to produce initial trajectories, which are then validated by unit tests and supplemented with agent guidance. Agents then reattempt with guidance, and the agent policy is updated with RLVR based on the rewards of these guided trajectories. Agent-RLVR elevates the pass@1 performance of Qwen-2.5-72B-Instruct from 9.4% to 22.4% on SWE-Bench Verified. We find that our guidance-augmented RLVR data is additionally useful for test-time reward model training, shown by further boosting pass@1 to 27.8%. Agent-RLVR lays the groundwork for training agents with RLVR in complex, real-world environments where conventional RL methods struggle.
1 Introduction
RLVR performs well on verifiable reasoning tasks, but agentic settings make successful trajectories harder to generate and reward signals harder to obtain. Agent-RLVR adds guidance to agent training and improves SWE-Bench Verified performance.
1 Introduction
- Figure 1 depicts an unguided attempt evaluated by unit tests, guidance for failed patches, and guided reattempts whose positive trajectories support training.The training loop updates the policy via offline DPO in an iterative manner.Source: Figure 1 · p. 2
- Agentic tasks require multi-turn reasoning, sequential decisions, and environment interaction, while infrastructure setup and execution time add training complexity.These factors make correct trajectories less likely and complicate effective RLVR training.Source: §1 · p. 2
- Agent-RLVR uses teacher guidance about failing patches or test expectations while relying on rewards from agent-environment interaction.The paper focuses on software engineering tasks with objectively verifiable solutions such as unit tests.Source: §1 · p. 2
- On SWE-bench Verified, Agent-RLVR raises PASS@1 from 9.4% to 22.4% using just 817 training environments, demonstrating substantial improvement with a relatively small training dataset.Source: §1 · p. 2
- The framework addresses sparse rewards in multi-step environments while preserving the verifiability of the ultimate reward signal.The paper also curates SWE tasks with environments and expert guidance for training.Source: §1 · p. 2
2 Agent-RLVR: An RL Framework for SWE Agents via Guidance and Environment Rewards
Agent-RLVR applies RLVR to software engineering agents by pairing environment-verified trajectories with guidance for failed attempts. It uses unit-test outcomes to form preference data and updates the policy through SFT and DPO.
2 Agent-RLVR: An RL Framework for SWE Agents via Guidance and Environment Rewards
- Agent-RLVR uses agent guidance to encourage more effective environment exploration and steer agents toward correct trajectories.The experiments describe one iteration cycle.Source: §2 · p. 3
2.1 Background: Reinforcement Learning from Verifiable Rewards
- RLVR uses a grader or unit tests as verifiable feedback; DPO trains on response pairs with higher and lower rewards.In Agent-RLVR, preferred code solutions pass all unit tests and rejected solutions fail at least one.Source: §2.1 · p. 3
2.2 Extending RLVR to Agentic Tasks with Agent-RLVR
- Guidance helps agents tackle difficult multi-turn tasks, while asynchronous environment runs can speed reward generation in this offline-RL framework.The challenges include costly environment infrastructure, sequential interaction, and tasks where the agent may never succeed unaided.Source: §2.2 · p. 3
2.2.1 Problem Formulation
- Tasks provide a GitHub issue, a Python repository snapshot, a Docker development environment, and unit tests as the solution’s ultimate validation.Training repositories do not overlap with repositories in the SWE-Bench test set.Source: §2.2.1 · pp. 3–4
2.2.2 Guidance
- Guidance is provided during trajectory rollouts, can help generate correct trajectories, and is used during training rather than test-time inference.The three forms are plans, environment feedback correcting earlier mistakes, and environment interaction pointing to relevant files or patch locations.Source: §2.2.2 · pp. 4–5
2.2.3 Training Dataset
- The training dataset contains 817 SWE environments across 27 repositories and 8,186 trajectories, including 4,093 positive trajectories.The data includes 593 SWE-Gym problems and 219 self-collected problems; 20% of positive trajectories are used for instruct-tuning.Source: §2.2.3 · p. 5
2.2.4 Agent-RLVR Training Loop
- The training loop evaluates initial trajectories, retries failures with guidance, then pairs correct guided trajectories with incorrect unguided ones for DPO.Before DPO, an untuned model receives SFT on a sampled 20% of non-guidance-augmented trajectories.Source: §2.2.4 · p. 5
3 Experiments
Experiments evaluate Agent-RLVR on SWE-Bench Verified and compare it with SFT, guidance-free training, and test-time reward-model filtering. The reported results show improved benchmark performance with Agent-RLVR and additional gains from guidance and reward-model filtering.
3.1 Setup
- The training set contains 817 environments, with both on-policy Qwen2.5-72B-Instruct trajectories and off-policy guidance-model trajectories.The trajectory collection contains 8,186 trajectories sampled from 16 trajectories per problem across guidance and non-guidance sampling.Source: §3.1 · pp. 5–6
- Evaluation uses the verified SWE-Bench portion; six instances could not be compiled, but were retained in the metrics for consistency with other methods.Source: §3.1 · p. 6
- The experiments use a simplified Agentless-based scaffold with localization and repair steps, and report PASS@1 and BEST@1 using the specified decoding procedures.The scaffold omits patch selection, while guidance is supplied through an additional prompt during self-improvement.Source: §3.1 · p. 6
3.2 Evaluation Results
- Qwen-2.5-72B-Instruct improves from 9.4% to 22.4% on SWE-Bench Verified with Agent-RLVR.Table 1 also reports 27.8% for Agent-RLVR-RM-Qwen-2.5-72B-Instruct.Source: Table 1 · p. 7, §3.2 · p. 7
- With guidance, an average of 27 more trajectories succeed and the empty-patch percentage decreases by 1.51%.Source: Table 2 · p. 7
- Agent-RLVR scores 22.4% on SWE-Bench Verified, compared with 20.8% for the rejection-sampling SFT baseline.The authors note that SFT and RLVR are not mutually exclusive and suggest combining their data in future work.Source: §3.2 · p. 7
- Guidance-augmented Agent-RLVR performs better than its guidance-free version at PASS@1, with the gap increasing at higher PASS@k.The authors hypothesize that sparse reward landscapes help explain the difference.Source: §3.2 · p. 7, Figure 2 · p. 7
- Reward-model filtering improves PASS@1 from 22.4% to 27.8%.The reward model is trained on RLVR task data and ranks generated patches at test time.Source: §3.2 · pp. 7–8
4 Broader Impacts and Limitations
The paper presents software engineering agents as potential collaborators that could reduce tedious work and accelerate production-ready code. Its scope is limited by Python-only experiments, small training data, limited validated evaluation data, and an offline-RL focus.
4.1 Broader Impacts
- SWE agents could help experienced developers by eliminating tedious work and accelerating teams’ progress from ideas to production-ready code.The authors frame this as a hoped-for long-term impact.Source: §4.1 · p. 8
4.2 Limitations
- Experiments are limited to Python, with other programming languages left for future work.The paper selects Python partly because SWE-Bench and Agentless infrastructure support it.Source: §4.2 · p. 8
- Training data covers few repositories, only one substantial human-validated SWE-agent test set is noted, and pretraining contamination remains possible despite disjoint repositories.The small repository set reflects cost considerations.Source: §4.2 · p. 8
- The study explores Agent-RLVR as an offline algorithm, leaving online RL for future work because it requires additional environment infrastructure.The paper notes that online RL could use multiple GRPO or PPO iterations.Source: §4.2 · p. 8
5 Related Work
Related work spans software-engineering agent scaffolds and model training, RLVR with verifiable feedback, and training for other environment-based language-model agents.
5.1 Training software engineering agents
- SWE agents are evaluated by repository patches passing tests; prior work improves scaffolds or trains models via rejection sampling, RL, or specialized modules.Scaffold examples include tool-use and multi-step localization, repair, and validation; training approaches include SWE-Gym, SWE-RL, and SWE-Fixer.Source: §5.1 · pp. 8–9
5.2 Reinforcement Learning from Verifiable Rewards
- RLVR applies RLHF methods such as PPO, DPO, and GRPO with grader or unit-test feedback instead of a reward model; PlanSearch reports that plans improve coding-search diversity.The passage also places RLVR in the Tulu3 and DeepSeek V3 post-training pipelines.Source: §5.2 · p. 9
5.3 Progress in large language model agents
- Computer-use, web, and tool-use benchmarks provide environment-based agent evaluations, while UI-Tars, OpenAI Operator, and WebRL exemplify trained GUI or computer- and web-use agents.The passage characterizes training work for agentic tasks as sparser than benchmark availability.Source: §5.3 · p. 9
6 Conclusion
The conclusion presents Agent-RLVR as a pipeline for training agents with RLVR through guidance and environment exploration. It reports improved SWE-agent performance over SFT baselines and identifies guidance-based RL data as useful for reward-model patch selection.
6 Conclusion
- Agent-RLVR uses guidance for environment exploration, improves SWE-agent performance over SFT baselines, and yields RL data useful for reward-model patch selection at test time.The guidance gap widens at higher PASS@k.Source: §6 · p. 9
A Technical Appendices and Supplementary Material
The appendix visuals show a guidance-generation prompt, its use during trajectory generation, and a comparison of SFT with and without guidance.
A Technical Appendices and Supplementary Material
- Table 4 compares SFT with and without guidance.The supplied caption states the comparison scope but does not report its outcome.Source: Table 4 · p. 13
- Figure 3’s guidance-generation prompt uses fields for the repository, task statement, reference patch, and prior-attempt stacktrace.The caption identifies what each field refers to.Source: Figure 3 · p. 13
- Figure 4 illustrates guidance being appended to the trajectory during trajectory generation.Source: Figure 4 · p. 13