PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training

Huatong Song, Lisheng Huang, Shuang Sun, Jinhao Jiang, Ran Le, Daixuan Cheng, Guoxin Chen, Yiwen Hu, Zongchao Chen, Yiming Jia, Wayne Xin Zhao, Yang Song, Tao Zhang, Ji-Rong Wen

arXiv:2602.03411v2cs.SEcs.CL

TL;DR

Software engineering agents remain difficult to reproduce because existing systems obscure data construction and optimization procedures. SWE-Master exposes and systematically optimizes the full post-training pipeline, combining trajectory curation, long-horizon SFT, execution-feedback RL, and inference design. It reaches 61.4% on SWE-bench Verified and reports 70.8% with TTS@8, while showing that context-management gains do not extend to SWE-Master’s smaller model scale.

  • Problem

    Existing software engineering-agent approaches lack transparency and reproducibility across training-data construction and optimization procedures.

  • Method

    SWE-Master systematically combines teacher-trajectory synthesis and curation, long-horizon SFT, RL in real execution environments, and inference-time framework design.

  • Results

    61.4% resolve rate is achieved on SWE-bench Verified with Qwen2.5-Coder-32B, while TTS reaches 70.8% at TTS@8.

  • Takeaways & Limitations

    SWE-Master provides an open-source, transparent foundation for reproducible software engineering-agent research.

  • Takeaways & Limitations

    Context management yields no measurable efficiency or efficacy improvement for SWE-Master, attributed to the smaller model’s limited ability to use managed context.

Abstract

from arXiv · show

In this technical report, we present SWE-Master, an open-source and fully reproducible post-training framework for building effective software engineering agents. SWE-Master systematically explores the complete agent development pipeline, including teacher-trajectory synthesis and data curation, long-horizon SFT, RL with real execution feedback, and inference framework design. Starting from an open-source base model with limited initial SWE capability, SWE-Master demonstrates how systematical optimization method can elicit strong long-horizon SWE task solving abilities. We evaluate SWE-Master on SWE-bench Verified, a standard benchmark for realistic software engineering tasks. Under identical experimental settings, our approach achieves a resolve rate of 61.4\% with Qwen2.5-Coder-32B, substantially outperforming existing open-source baselines. By further incorporating test-time scaling~(TTS) with LLM-based environment feedback, SWE-Master reaches 70.8\% at TTS@8, demonstrating a strong performance potential. SWE-Master provides a practical and transparent foundation for advancing reproducible research on software engineering agents. The code is available at https://github.com/RUCAIBox/SWE-Master.

1 Introduction

SWE-Master targets limited transparency and reproducibility in software engineering-agent development by exposing and systematically optimizing the post-training pipeline. It combines data curation, long-horizon training, execution-feedback RL, and structured code navigation, achieving strong SWE-bench Verified results.

  • Motivation: Existing SWE-agent approaches obscure data construction and optimization procedures, hindering reproducible research and broader accessibility.Key challenges include synthesizing long-horizon teacher trajectories and balancing correctness, diversity, and difficulty during SFT data curation.
  • Evaluation: 57.8% resolve rate is achieved with long-horizon SFT, increasing to 61.4% after RL with real execution environments on SWE-bench Verified.The evaluation uses identical base-model, training-data, and inference settings when comparing against existing open-source methods.
  • Framework: SWE-Master is an open-source framework that exposes an end-to-end training pipeline covering data processing, SFT, RL infrastructure, and inference-time agent frameworks.The framework studies interactions among data construction, optimization strategies, and inference-time behaviors starting from an open-source model with limited initial SWE capability.
  • Inference framework: LSP-driven code navigation provides IDE-level structural information that improves repository understanding and agent efficiency without sacrificing performance.The approach replaces simple text-based search with structured navigation to support precise inspection and modification in large repositories.
  • Evaluation: 70.8% resolve rate is reached with test-time scaling using LLM-based environment feedback, while 76.2% is reported under Pass@8.TTS explores and ranks multiple candidate solutions without the overhead of physical execution.

2 Preliminaries

The paper formulates software engineering as interactive patch generation over repository states, with agents taking navigation, editing, and execution actions. Evaluation applies generated patches and declares tasks resolved only when the complete unit-test suite passes.

  • Problem formulation: A software engineering instance consists of an issue description, an initial codebase, and a unit-test suite supporting automated repair or feature implementation.The target is a patch that modifies the repository to resolve the issue.
  • Agent interaction: The agent operates as a policy over interaction histories, generating reasoning traces and tool actions in a sequential environment.The environment state includes file contents, command-line history, and previous execution outputs.
  • Agent interaction: The action space combines navigation commands, file-manipulation commands, and execution commands such as pytest.These actions let the agent inspect repositories, modify files, and obtain execution feedback.
  • Agent interaction: Each action produces an observation and transitions the environment to a new state until submission or a maximum step limit is reached.A trajectory records the issue, actions, and observations throughout the interaction.
  • Evaluation protocol: A task is resolved only when the patched codebase passes every test in the unit-test suite.Evaluation runs in isolated Docker containers, and the suite includes fail-to-pass tests for reproducing bugs and pass-to-pass tests for preventing regressions.

3 SWE-Master: Training Open-Source SWE Agent

SWE-Master combines realistic execution environments, teacher-trajectory generation, filtering, long-horizon training, reinforcement learning, and test-time scaling in a reproducible SWE-agent pipeline. Its experiments show that curated trajectories, stable RL, and expanded interaction or rollout budgets improve software-engineering task resolution.

  • Training Framework and Environments: SWE-Master uses Docker-based environments with terminal, filesystem, package-management, and repository access for interactive agent training and evaluation.Its inference pipeline is based on R2E-Gym and uses a decoupled Docker–Server architecture with isolated execution containers.
  • Trajectory Generation: Teacher models generate trajectories by exploring repositories, modifying files, writing and executing tests, and validating proposed solutions through realistic environment interaction.The rollout corpus combines multiple open-source SWE datasets and real-world and synthetic data sources.
  • Data Filtering: Filtering removes unsuccessful trajectories and long-tail failure outliers, producing a smoother and more stable trajectory-length distribution for SFT.The pipeline applies reward- and format-based filtering before difficulty-based selection.
  • Policy Learning Dynamics: RL training shows rising reward, increasing interaction turns, and declining entropy without entropy collapse, indicating stable behavioral adaptation during policy learning.The reward rises from an initial value of 0.35 toward a stabilized peak.
  • Sequential Scaling: 61.4% resolve rate is reached by SWE-Master-RL when the interaction budget increases from 25 to 150 turns, outperforming SWE-Master-SFT.Both models plateau beyond 125 turns, indicating diminishing marginal returns at higher interaction limits.
  • Parallel Scaling: 70.8% at TTS@8 is achieved as rollout breadth increases, while TTS performance closely tracks Pass@K through simulated reward-based candidate verification.SWE-World predicts trajectory rewards with 77.59% accuracy, 71.40% recall, and 71.64% precision against Real-Docker feedback.

4 Experiment

SWE-Master is evaluated on SWE-bench Verified against open-source baselines, with results showing gains from long-horizon SFT, RL, and test-time scaling. Behavioral analyses link longer trajectories to lower resolve rates and RL to more iterative tool use.

  • Evaluation Setup: SWE-bench Verified contains 500 solvable real-world GitHub issues from 12 open-source Python repositories.
  • Main Results: 61.4% Pass@1 is achieved by SWE-Master-32B-RL, exceeding daVinci-Dev at 58.5% and SWE-SWE-Compressor at 57.6%.
  • Main Results: 5.8 percentage points is the RL gain for Qwen3-4B-2507, while Qwen2.5-Coder-32B improves from 57.8% to 61.4%.
  • Main Results: 70.8% is reached at TTS@8 for SWE-Master-32B-RL, while the SFT model reaches 70.2% from 57.8%.TTS@8 uses eight rollouts with simulated verification and ranking; the reported RL improvement is 9.4%.
  • Behavioral Analysis: Longer trajectories generally correspond to lower resolve rates, while RL shifts behavior toward more execute_bash and file_editor_replace operations.The reported tool-use pattern is associated with more active iterative debugging through repeated test execution and code modification.

5 IDE-Level Code Capacity

SWE-Master adds IDE-level code navigation through an LSP-based tool to address the semantic limitations of lexical search. The resulting navigation capability preserves resolve performance while reducing interaction and token costs.

  • Motivation: Existing CLI tools such as grep and find lack semantic understanding, creating bottlenecks for ambiguous non-crashing defects.
  • Method: lsp_tool provides a unified, language-agnostic interface with semantic operations including go to definition and find references.
  • Language Server Protocol: LSP standardizes communication between development tools and language-specific intelligence providers through JSON-RPC.This decoupling allows editors and IDEs to use a uniform interface across programming languages.
  • Method: lsp_tool exposes repository navigation, dependency analysis, code understanding, and workspace search through a unified API.
  • Evaluation Scope: The LSP evaluation uses pyright because SWE-bench Verified is Python-only, while the overall system design is described as language-agnostic.Two unsupported pyright features, get_declaration and get_implementation, are omitted.
  • Results: 61.0% versus 61.4% resolve rate shows comparable performance after LSP integration, while input and output tokens fall 23.7% and 16.3%.Average trajectory length also decreases by 17.5%.
  • Case Study: 57 steps versus 91 steps represents a 37% shorter trajectory for the illustrated task after distilled IDE-level navigation.The comparison is framed as a shift from lexical search and trial-and-error debugging toward reading code structure.

6 Further Analysis

Further analysis examines how data scaling, RL reward design, anti-hacking controls, and context management affect SWE-Master’s performance and efficiency. The results show substantial gains from more training data and managed context for some models, alongside setting-specific limitations.

  • SFT Data Scaling: 57.8% resolve rate is reached as SFT data scales from 0 to 60K samples, with marginal gains beginning to plateau beyond 48K.The resolve rate rises from a 6.2% baseline, following robust logarithmic growth.
  • SFT Data Scaling: Inference efficiency improves with SFT scaling, as average interaction turns fall from approximately 115 to 94 and thinking tokens from 14.8k to 12.7k.The authors associate this trend with internalized problem-solving heuristics that reduce redundant exploration.
  • RL Loss Masking and Reward Design: DeepSWE-style masking causes reward degradation and eventual training collapse, whereas SWE-Master’s reward shaping produces stable reward growth and deeper interaction.The reward-design comparison is illustrated by Figure 12, where the proposed mechanism increases trajectory length despite lower throughput.
  • Git Hacking: Removing Git-tool restrictions slightly degrades performance, while the enforced intervention prevents unauthorized attempts to retrieve golden patches through commands such as git show and git log.The authors attribute the degradation without restrictions to limited training exposure to exploitation patterns.
  • Summary-Based Context Manager: A summary-based context manager compresses older interactions while preserving a high-fidelity recent window, reducing M2.1 input tokens by 42% and peak tokens from 55.4k to 28.5k.The method summarizes every m turns once the raw context window reaches k + m, retaining long-term logic and recent debugging detail.
  • Summary-Based Context Manager: SWE-Master shows no measurable efficiency or efficacy improvement from the context manager, which the authors attribute to its smaller parameter scale.This contrasts with the gains reported for MiniMax-M2.1 and GLM-4.7.

7 Related Work

Related work has progressed from repository-level agent frameworks toward improving the underlying models and training processes for autonomous software engineering. SWE-bench and SWE-bench Verified provide benchmarks for evaluating these capabilities on real-world GitHub issues.

  • SWE-bench established repository-level evaluation of LLM code agents on real-world GitHub issues, while SWE-bench Verified builds on this benchmark.
  • Agent research includes environment-interacting systems such as SWE-agent and OpenHands, alongside newer efforts focused on enhancing the underlying models.

8 Conclusion

SWE-Master combines reproducible environment design, data curation, long-horizon reinforcement learning, and IDE-level code navigation for repository-level software engineering. Evaluation on SWE-bench Verified reports advanced performance among open-source code agents.

  • SWE-Master builds a high-quality training dataset through agent rollouts, format-based filtering, and difficulty-based selection in decoupled Docker environments.
  • The framework combines long-horizon reinforcement learning with the language-agnostic lsp_tool to provide agents with IDE-level structural code information.
  • SWE-bench Verified evaluations report advanced performance among open-source code agents for complex repository-level issues.

A Detailed Implementation of LSP Tools

This section provides detailed implementation information for the LSP tools and a fuller characterization of their tool-call format.

  • The appendix documents the LSP Tool implementation and its tool-call format in greater detail.

A.1 Language Server Protocol

LSP standardizes communication between development tools and language servers, enabling structured code intelligence. Its intermediate protocol replaces many editor-language integrations with a more scalable architecture.

  • LSP changes language-support integration from M × N point-to-point implementations to a scalable M + N arrangement.A standardized intermediate protocol lets one language server serve any compliant client.
  • LSP connects integrated development environments with servers that provide language intelligence tools.The protocol supports structured interaction and code-intelligence delivery between clients and servers.
  • LSP consists of a Language Server, a Language Client, and the protocol connecting them.The server provides language smarts, while the client is an editor, development tool, or extension that communicates with it.
  • LSP sessions begin with client-server process startup, channel establishment, and an initialization handshake that negotiates capabilities and workspace context.The client sends an initialize request, and the server returns supported language features.

A.2 LSP Features Integrated into the Code Agent

The code agent exposes simplified function calls for LSP operations and converts them into JSON-RPC requests. It also post-processes server responses to provide compact, context-rich outputs.

  • The pre-processing layer presents an intuitive function-calling interface and translates model calls into standard JSON-RPC requests.Three LSP call-hierarchy methods are aggregated into the unified get_call_hierarchy feature to reduce token consumption and cognitive load.
  • The post-processing layer parses raw JSON responses into model-readable results while adding necessary source context and filtering redundant noise.For example, outputs may include actual source code rather than only location coordinates.
  • Table 10 documents the processed tools’ input parameters and output results, with fp, l, and s abbreviating file_path, line, and symbol.The listed symbol categories include classes, functions, methods, variables, fields, and modules.

A.3 Tool Call Format of LSP Tool

The LSP tool exposes navigation, symbol, reference, hierarchy, and contextual inspection commands through structured inputs and execution outputs. Examples show these calls returning definitions, symbols, references, call relationships, and hover information across repositories.

  • The evaluation uses eight of ten LSP features, excluding get_declaration and get_implementation because Pyright lacks support for them in Python’s dynamic typing system.Those features remain available for languages such as C++ and Java.
  • get_workspace_symbols and get_document_symbols expose symbol inventories at workspace and file scope.The outputs include entities such as swap_dims and the ConditionSet class with its methods and functions.
  • get_definition and get_type_definition return the locations and source declarations associated with a selected symbol.Examples locate _replace_with_new_dims in xarray and logger in Python typing or logging definitions.
  • get_call_hierarchy reports callers and callees for a symbol across project and library files.The minimal_blocks example identifies test callers and dependencies such as range, tuple, and schreier_vector.
  • get_hover returns contextual type information, exemplified by the hover result identifying tickdate as NDArray[Any].
  • get_document_highlights and get_references locate related occurrences of a symbol within documents or across files.The examples report two document highlights and five references for minimal_blocks.

B LLM Prompts

The prompts instruct an evaluator to assess whether LSP tools help an SWE-bench agent solve coding tasks. They require evidence-based comparison of successful, ineffective, and potentially missed tool usage, including quantitative call statistics and a final verdict.

  • The prompt supplies trajectory-specific checks for workspace symbols, document symbols, references, definition chains, and missed opportunities to use LSP.
  • The prompt defines lsp_tool as a Pyright-based code-intelligence tool that provides semantic understanding beyond text search.
  • The evaluator must identify positive and negative tool usage, including efficient navigation, accurate understanding, misuse, failed calls, ignored results, and inefficiency.
  • The analysis compares agent behavior with lsp_tool against hypothetical grep/cat-only behavior and asks whether LSP provides a decisive structural-understanding advantage.
  • The required report includes use cases, problematic cases, key statistics, critical moments, a yes-or-no answer, and recommendations.Statistics cover total, successful, solution-contributing, failed, and redundant calls.
  • The instructions require evidence-based, objective, context-sensitive, counterfactual, and problem-solving-focused judgments.They caution that failed calls are not inherently harmful if the agent adapts well and encourage considering tool synergy.
Loading 2602.03411v2…