Source-linked AI summary
SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution
Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, Kai Chen
TL;DR
Real-world GitHub issue resolution is difficult to reproduce and make accessible when existing systems depend on proprietary LLMs, while the key capabilities for effective issue solving remain unclear. SWE-Fixer trains separate open-source retrieval and editing models in a streamlined two-stage pipeline using a 110K-instance dataset. It achieves competitive benchmark performance, reaches state-of-the-art open-source Best@1 results with P2P filtering, and requires only two model calls per instance.
Problem
Many GitHub issue-solving approaches rely on proprietary LLMs, limiting reproducibility, accessibility, and transparency while leaving the most important capabilities for effective issue resolution unclear.
Method
SWE-Fixer uses BM25 and a retriever for coarse-to-fine file selection, then a separate editor generates patches using structured outputs and a 110K-instance training dataset.
Results
SWE-Fixer achieves 22.0% on SWE-Bench Lite and 30.2% on SWE-Bench Verified, rising to 24.7% and 32.8% Best@1 with P2P filtering.
Takeaways & Limitations
The framework provides an efficient open-source approach to real-world code fixing, requiring only two model calls per instance while achieving strong benchmark performance.
Takeaways & Limitations
The training data scale could be expanded, but computational resource constraints prevented training on a significantly larger dataset.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks. One significant application of LLMs is in tackling software engineering challenges, particularly in resolving real-world tasks on GitHub by fixing code based on the issues reported by the users. However, many current approaches rely on proprietary LLMs, which limits reproducibility, accessibility, and transparency. The critical components of LLMs for addressing software engineering issues and how their capabilities can be effectively enhanced remain unclear. To address these challenges, we introduce SWE-Fixer, a novel open-source framework designed to effectively and efficiently resolve GitHub issues. SWE-Fixer comprises two essential modules: a code file retrieval module and a code editing module. The retrieval module employs BM25 along with a lightweight model to achieve coarse-to-fine file retrieval. Subsequently, the code editing module utilizes the other model to generate patches for the identified files. To mitigate the lack of publicly available datasets, we compile an extensive dataset that includes 110K GitHub issues along with their corresponding patches and train the two models of SWE-Fixer separately. We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving competitive performance among open-source models with scores of 22.0% and 30.2%. Furthermore, SWE-Fixer reaches state-of-the-art performance (24.7% on Lite and 32.8% on Verified) with PASS_TO_PASS (P2P) filtering. Additionally, our approach requires only two model calls per instance, making it significantly more efficient than existing methods. These results highlight the effectiveness of SWE-Fixer in real-world code-fixing scenarios. We will make our model, dataset, and code publicly available at https://github.com/InternLM/SWE-Fixer.
1 Introduction
SWE-Fixer addresses the reliance on proprietary models for real-world GitHub issue resolution with a streamlined open-source pipeline. It combines coarse-to-fine file retrieval, code editing, large-scale training data, and efficient inference to achieve competitive or state-of-the-art benchmark performance.
- Existing GitHub issue-solving systems commonly rely on proprietary models, limiting reproducibility, accessibility, and transparency.
- SWE-Fixer decomposes issue resolution into code file retrieval and code editing, using BM25 plus a retriever before patch generation.The retrieval strategy is coarse-to-fine, while the editing model generates patches for identified defective files.
- 110K training instances support separate training of a 7B retriever and a 72B editor based on Qwen2.5 models.The dataset is filtered for quality and designed to train both subtasks.
- 22.0% on SWE-Bench Lite and 30.2% on SWE-Bench Verified are competitive with existing state-of-the-art open-source models.
- 24.7% on SWE-Bench Lite and 32.8% on SWE-Bench Verified constitute state-of-the-art Best@1 performance with P2P filtering.
- The framework requires only two model calls per instance, reducing inference cost while maintaining strong performance.
2 Related Works
Prior GitHub issue-solving systems use agent-based or pipeline-based paradigms, but many depend on proprietary models. Agentless offers a related pipeline philosophy, whereas SWE-Fixer emphasizes simpler adaptation for open-source training.
- GitHub issue-solving approaches broadly divide into agent systems that choose actions dynamically and pipelines that follow predefined retrieval and editing steps.
- Agent-based methods such as SWE-agent, Autocoderover, OpenHands, and Moatless Tools commonly leverage proprietary models and tools.
- Agentless is a pipeline-based approach with competitive performance against agent-based methods.
- Agentless uses sophisticated proprietary-model retrieval and editing designs that are challenging to adapt for open-source model training.
- SWE-Fixer differs by offering a simpler and more robust design intended to facilitate open-source training-data construction.
3 SWE-Fixer
SWE-Fixer structures GitHub issue resolution as retrieval followed by editing. It narrows repositories with coarse-to-fine retrieval and produces patches through a structured editing representation.
- 3.1 Overview: SWE-Fixer divides issue resolution into code file retrieval and code editing within a structured pipeline.The framework generates code patches specifying repository modifications.
- 3.2 Code File Retrieval: BM25 retrieves the 30 most relevant files using the issue description as a query, after which a fine-tuned retriever identifies files requiring modification.
- 3.2 Code File Retrieval: File documentations retain signatures, headers, docstrings, and limited function lines to reduce context size while preserving retrieval-relevant information.
- 3.3 Code Editing: Code editing generates a patch from relevant files, using gold defective files during training and retrieved files during inference.The model receives complete retrieved-file content to understand the issue and apply changes.
- 3.3 Code Editing: The editing output contains a file path, original code block with line numbers, and modified code block without line numbers.This structured format simplifies training and can be automatically converted into patches for evaluation.
4 Model Training
SWE-Fixer trains retrieval and editing models with structured task representations and reasoning-oriented data construction. Because real execution is unavailable, patch verification by execution and standard rejection sampling are infeasible.
- Model Training: JsonTuning represents task inputs, instructions, output controls, and task-specific outputs in structured JSON.Retrieval uses issues and file documentations as inputs, while editing uses issues and file content.
- Chain-of-Thought Data Construction: Code editing data lacks intermediate reasoning because real-world examples typically contain codebases and gold patches without reasoning traces.
- Chain-of-Thought Data Construction: Without a real execution environment, generated patches cannot be verified through execution, making standard rejection sampling infeasible.
- Chain-of-Thought Data Construction: Gold patches guide GPT-4o to generate reasoning chains and corresponding patches through a rationalization-inspired training procedure.
5 Experiments
Experiments evaluate SWE-Fixer’s data construction, benchmark performance, efficiency, retrieval and editing design choices, and scaling behavior. The results show competitive open-source performance with two model calls per instance, while editing remains the primary bottleneck.
- Experimental Setup: SWE-Fixer is evaluated on SWE-Bench Lite and SWE-Bench Verified, where patches are assessed with developer-written unit tests and accuracy is the percentage of successfully resolved instances.P2P tests can optionally filter patches that break unrelated functionality.
- Dataset Preparation: The training data is split into 80K valid retrieval instances and 70K sampled editing instances, with a separate 10K dataset used for ablations.Retrieval examples are filtered by top-30 file inclusion and context length; editing examples are sampled under computational and API constraints.
- Main Results: Two model calls per instance—7B retrieval plus 72B editing—make SWE-Fixer more efficient than methods requiring substantially more calls.SWE-Search requires at least 200 model calls per instance while achieving only a slightly higher Lite score without P2P filtering.
- Code File Retrieval: Retrieval performance improves with relevant context, larger context windows, larger training datasets, and additional editing-task data, whereas irrelevant file content harms performance.A 32K context can exclude defective files and lower recall relative to a 64K context; expanding training data from 10K to 80K significantly boosts performance.
- Code Editing and Scaling Trends: Line numbers improve code editing, and direct-training performance scales with data size, while larger models show steeper trends and CoT’s advantage narrows as data grows.For Llama-3.1-70B, increasing CoT training data does not produce sustained performance improvements.
6 Conclusion
SWE-Fixer is an open-source, two-subtask pipeline for resolving real-world GitHub issues with finetuned open-source models. It achieves strong SWE-Bench performance while using few inference steps and low computational overhead.
- SWE-Fixer divides GitHub issue resolution into code file retrieval and code editing, requiring only two steps to generate final results.
- The framework uses a large-scale real-world dataset and task-specific training data for both subtasks.
- SWE-Fixer achieves the highest Best@1 performance among open-source model-based approaches on SWE-Bench Lite and Verified.
- SWE-Fixer outperforms several proprietary-model methods using GPT-4, GPT-4o, and Claude-3-Opus.
- The framework lowers barriers for community use and supports further innovation in software engineering.
Limitations
The authors identify limited training-data scale and the absence of test-time reward-based patch selection as limitations. They propose larger datasets and reward-model-guided Best-of-N selection as future directions.
- The training-data scale could be expanded, but computational resource constraints prevent training on a significantly larger dataset.
- A reward model could evaluate generated patches and support Best-of-N selection for test-time optimization.
- Despite these limitations, SWE-Fixer offers a low-cost and effective open-source approach to real-world software engineering problems.
A Training Data Collection
The authors collect GitHub issues, pull requests, and codebases, then analyze and filter real-world instances to construct higher-quality training data. Their statistics show that most instances involve small edits concentrated in a few files, lines, and hunks.
- Data collection: The collection process gathers high-quality issues, pull requests, and codebases from GitHub repositories before filtering overly complex examples.
- Repository collection: The crawler selects Python repositories with more than 100 pull requests and uses GitHub events instead of string matching to identify issue–PR pairs.
- Repository collection: The raw crawl contains 2.3K repositories and 331K instances, excluding repositories already included in SWE-Bench.
- Data statistics: Figure 4 summarizes distributions of edited files, modified code lines, and edited code hunks in sampled real-world data.
- Data statistics: 54.7% of instances modify one file, nearly 80% modify no more than three files, and over 85% modify up to 200 lines.
- Data filtering: The filtering process samples 140K instances, discards unparseable patches, and removes instances modifying more than three non-test files.
B Post-processing
SWE-Fixer uses unified post-processing and resampling for retrieval and editing outputs. It retries invalid outputs with higher temperature and rejects results that fail code or test validation.
- The procedure begins with deterministic sampling at temperature 0 and retries failed or invalid outputs at temperature 0.7.
- Each output receives at most five attempts, preventing endless retries.
- Results are invalid when the original code cannot be located, modified code fails syntax checks, or existing repository tests fail.
- Post-processing improves generated-output correctness in format and content for real-world application requirements.
C P2P Filtering
P2P filtering validates generated patches against repository regression tests that pass both before and after the gold patch. SWE-Fixer reports both filtered and unfiltered results because the community discusses whether to use this inference-time procedure.
- P2P tests check whether generated patches preserve unrelated repository functionality.A patch is retained only if it passes all P2P tests; failed patches are resampled.
- P2P filtering first uses greedy sampling, then temperature 0.7 after failure to generate more creative patches.
- Successful P2P-filtered instances require an average of 1.15 generation attempts on SWE-Bench Lite and 6.73 on SWE-Bench Verified.
- At least 200 inference steps are estimated for SWE-Search, including 100 search steps and 100 value-model evaluations.This estimate is based on the reported cap of 100 search iterations and per-node value-model evaluation.
E Ablation on More Fine-Grained Retrieval Strategies
The ablation evaluates whether richer retrieval details improve SWE-Fixer. Adding class and function names increases retrieval complexity and reduces overall pipeline performance, while file documentation exposes structured code context.
- E Ablation on More Fine-Grained Retrieval Strategies: The experiment tests more fine-grained retrieval strategies to determine whether additional details improve overall performance.
- E Ablation on More Fine-Grained Retrieval Strategies: Retrieving class and function names increases retrieval-task complexity and ultimately reduces overall pipeline performance.
- E Ablation on More Fine-Grained Retrieval Strategies: File documentation includes the relative path, available module and class docstrings, class names, and method names.For functions, it includes only the name and the first and last five lines of code.