Source-linked AI summary
SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving
Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, Zhiming Mao, Xinyu Wang, Lifeng Shang, Haoli Bai
TL;DR
SWE-Lego addresses limitations in SWE-agent training data and lightweight SFT by combining executable hybrid data with refined supervised fine-tuning and verifier-guided test-time scaling. Using this approach, its models achieve strong hack-free SWE-bench Verified performance, which increases further with TTS@16.
Problem
Prior SWE-agent training efforts lack sufficient executable, realistic instances for high-quality trajectories, while alternative mid-training approaches require substantially more computation and data.
Method
SWE-Lego combines real and synthetic executable data with validated expert trajectories, step-level error masking, difficulty-based curriculum learning, and verifier-guided test-time scaling.
Results
SWE-Lego-Qwen3-8B and SWE-Lego-Qwen3-32B reach 42.2% and 52.6% on SWE-bench Verified using SFT, rising to 49.6% and 58.8% under TTS@16.
Takeaways & Limitations
A reproducible, lightweight SFT pipeline can provide strong hack-free SWE issue-resolving performance and further benefit from verifier-guided inference scaling.
Abstract
from arXiv · showhide
We present SWE-Lego, a supervised fine-tuning (SFT) recipe designed to achieve state-ofthe-art performance in software engineering (SWE) issue resolving. In contrast to prevalent methods that rely on complex training paradigms (e.g., mid-training, SFT, reinforcement learning, and their combinations), we explore how to push the limits of a lightweight SFT-only approach for SWE tasks. SWE-Lego comprises three core building blocks, with key findings summarized as follows: 1) the SWE-Lego dataset, a collection of 32k highquality task instances and 18k validated trajectories, combining real and synthetic data to complement each other in both quality and quantity; 2) a refined SFT procedure with error masking and a difficulty-based curriculum, which demonstrably improves action quality and overall performance. Empirical results show that with these two building bricks alone,the SFT can push SWE-Lego models to state-of-the-art performance among open-source models of comparable size on SWE-bench Verified: SWE-Lego-Qwen3-8B reaches 42.2%, and SWE-Lego-Qwen3-32B attains 52.6%. 3) We further evaluate and improve test-time scaling (TTS) built upon the SFT foundation. Based on a well-trained verifier, SWE-Lego models can be significantly boosted--for example, 42.2% to 49.6% and 52.6% to 58.8% under TTS@16 for the 8B and 32B models, respectively.
1 Introduction
SWE-Lego investigates whether a carefully designed SFT pipeline can resolve software issues effectively without relying on more complex training paradigms. It combines executable, realistic data with refined training procedures and reports strong SWE-bench Verified performance.
- Prior SFT efforts lack executable environments and realistic bug instances, while mid-training requires substantially more computation and data.
- SWE-Lego combines real pull requests with synthetic bugs and expert trajectories to build a scalable training pipeline.Its pipeline includes environment construction, hybrid task creation, and expert-trajectory generation and curation.
- 32k high-quality task instances and 18k validated expert trajectories form the SWE-Lego dataset.The dataset uses complementary real-world and synthetic sources and filters trajectories to promote genuine problem-solving skills.
- 42.2% and 52.6% are achieved by SWE-Lego-Qwen3-8B and SWE-Lego-Qwen3-32B, respectively, using only SFT on SWE-bench Verified.These results establish strong performance among open-source models of comparable size.
- 49.6% and 58.8% are reached under TTS@16 for the 8B and 32B models, respectively.The reported gains come from adding test-time scaling to the SFT foundation.
2 SWE-LEGO Dataset: Combining Real-world and Synthetic Data
SWE-Lego builds an executable SWE dataset by combining authentic but limited real-world tasks with scalable synthetic instances. This hybrid strategy expands validated supervision while preserving complementary coverage of complex and focused bugs.
- 32k executable task instances and 18k validated expert trajectories form the SWE-Lego training dataset.
- Hybrid data construction uses a unified schema while varying data origin and sandbox granularity between real and synthetic tasks.
- Real-world tasks provide production-level authenticity and complexity, whereas synthetic tasks provide scalable quantity and coverage.
- Real-world instances touch more files and change more lines, while synthetic instances are more focused and yield more valid trajectories.
- Scaling synthetic instances from real-only data to real plus synthetic instances consistently improves valid trajectories and SWE-bench Verified resolve rate.
- Trajectory quality is further supported by Git-history sanitization, malformed-tool correction, ineffective-tool pruning, and post-hoc validation and filtering.
3 Refined Supervised Fine-tuning
SWE-Lego refines supervised fine-tuning by masking erroneous trajectory steps and progressively increasing task difficulty. These changes target the transition from basic process failures to strategic and implementation bottlenecks during training.
- Step-level error masking preserves trajectory context while excluding incorrect intermediate actions from the loss.
- Curriculum learning progressively exposes the model to increasingly complex tasks to improve training efficiency and generalization.
- The trajectory-length heuristic outperformed model-based difficulty scoring by 0.5 points and was more computationally efficient and less prone to mislabeling.
- A three-stage curriculum trains on successive difficulty tiers while retaining preceding-stage data to mitigate catastrophic forgetting.
- 42.2% resolve rate was reached by epoch 4, while failure modes shifted from reproduction and turn-budget failures toward implementation and localization errors.
- 3.8% improvement over conventional SFT accompanied SWE-Lego-Qwen3-8B's 42.2% and SWE-Lego-Qwen3-32B's 52.6% on SWE-Bench Verified.
- Test-time scaling varies resolve rate with maximum interaction turns and parallel rollouts, shifting from sequential gains toward parallel gains after saturation.
4 Test-time Scaling
SWE-Lego studies how to allocate inference compute between sequential interaction turns and parallel candidate rollouts, using verifier-based selection for parallel scaling. Sequential scaling is favored initially, while parallel scaling becomes more effective after saturation; generative verifiers consistently outperform regressive ones.
- Balancing Sequential and Parallel Scaling: Latency grows super-linearly with interaction turns but sub-linearly with batched candidate rollouts, motivating different scaling choices across compute budgets.Sequential trajectories incur increasing per-turn context computation, whereas parallel candidates execute in batch.
- Balancing Sequential and Parallel Scaling: Sequential scaling is preferred under tight latency budgets, but performance saturates around 100–140 turns, after which parallel scaling becomes more effective.Under equivalent latency, the optimal allocation shifts from sequential-dominated to parallel-dominated as the total latency budget increases.
- Improved Parallel Scaling: 49.6% TTS@16 makes SWE-Lego-Verifier-8B stronger than OpenHands-Critic-32B at 44.0% and R2E-Gym-Verifier-14B at 47.0%.Generative verifiers maintain monotonic improvement as rollout count increases, while the regressive OpenHands verifier degrades at higher K.
- Balancing Sequential and Parallel Scaling: Sequential-then-parallel scaling prioritizes additional interaction turns before the saturation point, then assigns remaining compute to parallel rollouts.This allocation follows the observed shift in effectiveness under increasing latency budgets.
- Improved Parallel Scaling: Generative verifiers consistently outperform regressive verifiers across 8B and 32B models and different rollout budgets.The study compares verifier paradigms using the same training data and backbone settings where applicable.
5 Comparisons with Existing Methods
SWE-Lego reports hack-free comparisons on SWE-bench Verified against proprietary and open-source baselines. Its models achieve strong resolve rates with SFT alone and improve further with TTS@16.
- SWE-bench Verified: 42.2% resolve rate for SWE-Lego-Qwen3-8B rises to 49.6% with TTS@16, while SWE-Lego-Qwen3-32B rises from 52.6% to 58.8%.These results use the latest SWE-bench Verified Docker images without Git hacking.
- SWE-bench Verified: The hack-free results surpass most open-source models and several larger proprietary models, whose reported results are generally inflated by Git hacking.The comparison reports results with and without Git hacking for fairness with prior methods.
6 Related Work
Related work spans code-data construction, foundation models and repository-context methods, and increasingly realistic repository-level evaluation for autonomous software engineering.
- Code Data and Foundation Models: Code intelligence research has progressed from filtered pretraining corpora toward instruction tuning, Git-commit data, and synthetic generation.Examples include The Stack, self-instruct methods, Git commit structures, Evol-Instruct, and OSS-Instruct.
- Code Data and Foundation Models: Foundation models such as Code Llama and DeepSeek Coder support software engineering agents through large-scale code pretraining and extended context windows.Repository-context approaches also integrate retrieval augmentation or AST-based retrieval.
- Repository-Level Evaluation: Evaluation has expanded from function-level synthesis to repository-level tasks and realistic issue resolving with benchmarks such as SWE-bench.Related benchmarks include ClassEval, CrossCodeEval, RepoBench, Defects4J, and InterCode.
7 Conclusions
SWE-Lego combines hybrid executable data construction, refined SFT, and verifier-guided TTS to achieve strong hack-free SWE-bench Verified performance. Its data, training, and inference strategies form a reproducible approach for software engineering agents.
- Conclusion: SWE-Lego combines hybrid executable data, refined SFT, and verifier-guided TTS for hack-free SWE-bench Verified performance.The framework uses scalable verifiable data, error masking, curriculum learning, and a generative verifier.
- Conclusion: Scalable, verifiable hybrid data instances and expert trajectories are identified as the core drivers of performance gains.Error masking and curriculum learning stabilize training, while generative verification allocates inference compute efficiently.
- Conclusion: The combined strategies provide a reproducible paradigm for building robust agentic systems for software engineering issue resolving.
A Additional Information on Data Construction
SWE-Lego constructs task instances from real and synthetic repositories, with golden patches and test outcomes defined differently for each source. These fields support executable issue-solving data and validated trajectories.
- Real-world golden patches are merged pull-request diffs representing human-authored fixes.
- Synthetic golden patches reverse injected bugs by recording the difference between buggy and clean repository versions.The buggy branch must be checked out before solving a synthetic task.
- FAIL-to-PASS tests fail before a real-world fix and pass afterward, while PASS TO PASS tests pass in both states.Synthetic instances derive analogous labels by running the repository test suite before and after injection reversal.
A.2 Analysis of Data Categories
The category analysis contrasts real and synthetic SWE-Lego data and examines how scaling and trajectory behavior affect issue resolution. The sources provide complementary coverage of external interactions and internal implementation defects.
- Data categories: Real-world data emphasizes external interaction and boundary defense, including API changes, input validation, and protocol conformance.
- Sequential scaling: Beyond roughly 140 turns, truncation and average-turn changes diminish, indicating saturation in sequential scaling.Most trajectories terminate or become unproductive before exhausting larger budgets.
- Sequential scaling: Very short and very long trajectories tend to have lower resolve rates than trajectories with intermediate turn counts.
- Data categories: Synthetic data emphasizes internal implementation and structural integrity, including logic flow and dependency management.Its inclusion supplements rare structural defects and rebalances interface-focused training.
B.1 Sequential Scaling Analysis
Sequential scaling shows diminishing returns after a turn-limit threshold, while resolve rates peak at intermediate trajectory lengths. These findings motivate shifting to parallel scaling with verifier selection after sequential scaling saturates.
- Additional turns beyond roughly 140 provide little marginal utility as truncation declines marginally and average trajectory length plateaus.Most trajectories terminate or enter unproductive states before using large turn budgets.
- Resolve rates are low at very small and very large turn counts, with the highest rates at intermediate ranges.Small counts limit exploration, whereas large counts often reflect ineffective retries.
- Sequential scaling saturation motivates parallel scaling with verifier selection once extra sequential compute yields limited benefit.
B.2 Impact of Training Data Scale and Model Scale
Verifier performance improves with more training trajectories and, for larger agent rollouts, with a larger verifier. The broader task scope also exposes unresolved limitations in data, evaluation, and cross-task generalization.
- Training data scale: 49.6% versus 47.6% at TTS@16 shows that an 18K-trajectory verifier outperforms a 6K verifier on 8B rollouts.The advantage widens as the candidate pool K increases.
- Scope limitations: Extending beyond issue resolving remains constrained by ambiguous requirements, nuanced acceptance criteria, non-regression constraints, and complex evaluation needs.Higher-order behaviors such as design decisions, integration, and refactoring lack well-defined learning signals.
- Scope limitations: Multilingual SWE is limited by language-specific toolchains, sandboxes, datasets, dependency practices, testing paradigms, and code organization.These differences complicate transfer of navigation, localization, editing, and validation skills across languages.
- Scope limitations: Heterogeneous code tasks create uneven capability demands, while differing difficulty and interaction requirements complicate balanced performance assessment.The lack of standardized cross-task metrics further limits evaluation of this imbalance.
D Example of Git Hacking
Git hacking can let an SWE agent retrieve the pre-existing solution from commit history, bypassing independent problem-solving and inflating evaluation performance.
- An SWE agent may inspect Git commit history and retrieve the pre-existing solution.
- This behavior can bypass independent problem-solving and falsely inflate SWE evaluation results.