PaperlayerView exact revision on arXiv

Source-linked AI summary

Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement

Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, Yongbin Li

arXiv:2411.00622v1cs.SEcs.AI

TL;DR

Automated software engineering needs capable open models and training that captures the dynamic, iterative nature of software development rather than static code alone. Lingma SWE-GPT addresses this with process-centric learning from simulated issue-resolution workflows, and its 72B model resolves 30.20% of SWE-bench Verified issues while its 7B model resolves 18.20%.

  • Problem

    ASE research depends heavily on closed-source models and static-code training that misses developers’ reasoning, tool use, and iterative software-development processes.

    Source: §1 · p. 2

  • Method

    Lingma SWE-GPT trains 7B and 72B open-source models to simulate repository understanding, fault localization, and patch generation through iterative reasoning and tool interactions.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    30.20% of SWE-bench Verified issues are resolved by Lingma SWE-GPT 72B, approaching GPT-4o’s 31.80%, while the 7B model resolves 18.20%.

    Source: §1 · p. 3, §5.1 · p. 13

  • Takeaways & Limitations

    Process-centric training enables an open-source model series to achieve competitive software-improvement performance, including strong results from a smaller 7B model.

    Source: §1 · p. 3, §7 · p. 18

    The study evaluates primarily on SWE-bench benchmarks, which may not fully capture complexity and applicability relative to human developers.

    Source: §6 · p. 17

Abstract

from arXiv · show

Recent advancements in LLM-based agents have led to significant progress in automatic software engineering, particularly in software maintenance and evolution. Despite these encouraging advances, current research faces two major challenges. First, SOTA performance primarily depends on closed-source models, which significantly limits the technology's accessibility, and potential for customization in diverse SE tasks. Second, these models are predominantly trained on static code data, lacking a deep understanding of the dynamic interactions, iterative problem-solving processes, and evolutionary characteristics inherent in software development. To address these challenges, our study adopts a software engineering perspective. We recognize that real-world software maintenance and evolution processes encompass not only static code data but also developers' thought processes, utilization of external tools, and the interaction between different functional personnel. Consequently, we introduce the Lingma SWE-GPT series, comprising Lingma SWE-GPT 7B and 72B. By learning from and simulating real-world code submission activities, Lingma SWE-GPT systematically incorporates the dynamic interactions and iterative problem-solving inherent in software development process, thereby achieving a more comprehensive understanding of software improvement processes. We conducted experimental evaluations using SWE-bench Verified benchmark. The results demonstrate that Lingma SWE-GPT 72B successfully resolves 30.20% of the GitHub issues, marking a significant improvement in automatic issue resolution (22.76% relative improvement compared to Llama 3.1 405B), approaching the performance of closed-source models (31.80\% issues of GPT-4o resolved). Notably, Lingma SWE-GPT 7B resolves 18.20% of the issues, highlighting the potential for applying smaller models to ASE tasks.

1 Introduction

ASE research is advancing, but remains constrained by dependence on closed-source models and training that misses dynamic software-development processes. Lingma SWE-GPT addresses these gaps with process-centric training and achieves competitive SWE-bench performance.

Motivation

  • Current ASE performance relies heavily on closed-source models, limiting accessibility, customization, and applicability to private software repositories.Closed-source model use may raise security concerns for user code repositories.Source: §1 · p. 2
  • Static-code training overlooks developers’ reasoning, tool use, iterative problem-solving, and cross-file project context.Existing pull-request data often contains commit messages and final patches without detailed reasoning traces.Source: §1 · p. 2

Approach

  • The approach simulates repository understanding, fault localization, and patch generation through iterative Chain-of-Thought interactions and tool use.The process progressively analyzes repository structure, retrieves relevant code, and debugs generated patches.Source: §1 · pp. 2–3

Results

  • 30.20% of SWE-bench Verified issues are resolved by Lingma SWE-GPT 72B, a 22.76% relative improvement over Llama 3.1 405B and near GPT-4o’s 31.80%.The 7B version resolves 18.20% of issues, exceeding Llama 3.1 70B’s 17.20%.Source: §1 · p. 3

Contribution

  • Lingma SWE-GPT is an open-source large language model series specifically optimized for automated software improvement.Source: §1 · p. 3

2 Related Work

Prior work spans code-focused LLMs, tool-using software-engineering agents, and benchmarks for end-to-end GitHub issue resolution. Together, these lines of research motivate evaluating models on realistic, repository-level software tasks.

2.1 Large Language Models for Code

  • Code LLMs support generation, testing, editing, refactoring, translation, debugging, and automated program repair, but mostly learn from static code data.The paper positions dynamic tool-use and reasoning as missing capabilities for real-world software engineering.Source: §2.1 · p. 4

2.2 LLM-based Software Engineering Agents

  • LLM-based software-engineering agents combine environment awareness, planning, reasoning, and tools to perform project-level tasks.Examples include Devin’s end-to-end workflow and SWE-agent’s repository navigation and program execution.Source: §2.2 · p. 4
  • Existing agents such as AutoCodeRover and RepoUnderstander iteratively retrieve context or localize faults, but most rely heavily on closed-source models.The paper frames Lingma SWE-GPT as an open-source attempt to narrow this performance gap.Source: §2.2 · p. 5

2.3 Evaluation of Real-world Software Engineering Tasks

  • SWE-bench evaluates agents on real GitHub issues requiring repository navigation, fault localization, debugging, code generation, and repair.SWE-bench Lite and Verified are smaller, human-filtered subsets recommended for evaluation.Source: §2.3 · p. 5

3 Lingma SWE-GPT

Lingma SWE-GPT trains open-source models from issue–pull-request data to simulate iterative software improvement. Its workflow combines repository understanding, fault localization, patch generation, filtering, and curriculum-based training.

3 Lingma SWE-GPT

  • Lingma SWE-GPT uses an iterative loop in which Chain-of-Thought reasoning and actions produce observations that guide subsequent repository work.The three stages are repository understanding, fault localization, and patch generation.Source: §3 · p. 5

3.1 Issue Data Collection

  • Approximately 90,000 merged pull requests from 4,000 repositories form the collected training dataset.Repositories had at least 50 stars, excluded SWE-bench overlap, and were filtered for substantive issues and code changes.Source: §3.1 · p. 6

3.2 Development Process Data Synthesis

  • SWESynInfer synthesizes development-process data by extending AutoCodeRover with enhancements that simulate expert developers’ cognitive processes.The workflow captures intermediate reasoning, tool interactions, and iterative refinement.Source: §3.2 · pp. 6–8
  • Repository understanding compresses and navigates project structure to identify relevant files, classes, functions, and a repair plan.RepoUer uses repository-structure, navigation, and code-compression tools.Source: §3.2 · pp. 7–8
  • Fault localization iteratively searches and reflects over code context, while patch generation applies, validates, and debugs patches with tools.The process continues until a fault is localized or an iteration limit is reached, and patch retries are bounded.Source: §3.2 · p. 8
  • Synthetic examples are retained using fault-localization and patch-similarity thresholds, while valid localization reasoning can be preserved even when patches differ.The filtering uses Jaccard similarity for locations and n-gram or CodeBLEU scores for patches.Source: §3.2 · pp. 8–9

3.3 Model training

  • Training maximizes the likelihood of target reasoning and actions, uses curriculum learning, and restarts each iteration from the original checkpoint.This strategy is intended to reduce the effect of low-quality early synthetic examples and improve stability.Source: §3.3 · p. 10

4 Experiment Setup

The experiments evaluate Lingma SWE-GPT against closed- and open-source baselines on SWE-bench Verified and Lite. They use developer-written tests, resolved-task rates, API cost, and repeated inference to assess effectiveness and consistency.

4.1 Benchmark and Evaluation Metric

  • SWE-bench Verified contains 500 issues and SWE-bench Lite contains 300, with developer-written unit tests validating generated patches.Each task provides an issue description and repository code, and expects a corrective patch.Source: §4.1 · p. 10
  • The evaluation reports resolved-task percentage and average closed-source API inference cost, while repeating experiments three times and reporting pass@3.Open-source API calls are assigned no direct API cost, excluding deployment expenses.Source: §4.1 · p. 10

4.2 Baselines

  • The baselines include GPT, Claude, and Gemini closed-source models alongside Llama and Qwen open-source models.Results combine reported studies with deployed evaluations, depending on the baseline.Source: §4.2 · p. 11

4.3 Implementation Details

  • Lingma SWE-GPT uses Qwen2.5 72B Instruct and Qwen2.5 Coder 7B foundations, with temperature 0.3 and fixed stage-specific inference limits.Training uses 64 NVIDIA A100 GPUs, 90 iterations, and the official SWE-bench Docker environment.Source: §4.3 · p. 11

5 Evaluation

Lingma SWE-GPT is evaluated on realistic software-issue resolution, open-source comparisons, fault localization, and execution consistency. Its 72B model approaches leading closed-source performance, outperforms open-source baselines, and remains stable across runs, while localization remains challenging.

5.1 RQ1: Overall Effectiveness in Resolving Real-life Software Issues

  • 30.20% of SWE-bench Verified issues were resolved by Lingma SWE-GPT 72B, approaching GPT-4o’s 31.80% and exceeding several open-source comparisons.On SWE-bench Lite, it resolved 22.00% versus GPT-4o’s 20.67%; Claude 3.5 Sonnet remained highest on both benchmarks.Source: §5.1 · pp. 12–13
  • Using GPT-4o to solve 500 SWE-bench Verified problems cost approximately $390, or $0.78 per issue.The paper contrasts these direct API costs with open-source models such as Lingma SWE-GPT, which do not incur direct API costs due to public accessibility.Source: §5.1 · p. 13

5.2 RQ2: Performance Against State-of-the-Art Open-Source Models

  • 30.20% success on SWE-bench Verified represented an 18.90% relative improvement over Qwen2.5-instruct 72B and 22.76% over Llama-3.1-instruct 405B.The comparison used the same inference process across open-source models, with prompt and tool adjustments for models with limited instruction-following.Source: §5.2 · pp. 13–14
  • Lingma SWE-GPT 7B resolved 18.20% of Verified and 12.00% of Lite issues, outperforming Llama-3.1-instruct 70B on both benchmarks.The reported comparisons were 18.20% versus 17.20% on Verified and 12.00% versus 7.00% on Lite.Source: §5.1 · p. 13, §5.2 · p. 14
  • Lingma SWE-GPT 72B outperformed Llama 3.1 405B instruct in 9 of 12 repositories and approached closed-source models in numerous instances.The repository comparison covered diverse software repositories in SWE-bench Verified.Source: §5.2 · p. 14, Figure 4 · p. 14
  • Resolved-issue overlaps were complementary: Lingma SWE-GPT 72B uniquely solved 34 issues, while its 7B model uniquely solved 7.The reported uniqueness suggests model ensembles as a possible direction for improving overall automated program improvement.Source: §5.2 · p. 15, Figure 5 · p. 15

5.3 RQ3: Fault Localization Effectiveness

  • Lingma SWE-GPT showed better fault localization than other open-source models at chunk, function, and file levels, but localization remained difficult overall.Even Claude 3.5 Sonnet reached only 54.90% chunk-level and 58.08% function-level accuracy; localization success generally correlated positively with issue resolution.Source: §5.3 · pp. 15–16

5.4 RQ4: Model Consistency and Randomness Impact

  • Across three runs, Lingma SWE-GPT 72B resolved 30.20%, 29.00%, and 30.20% of SWE-bench Verified issues, indicating stable performance.Pass@3 increased issue resolution to 39.80% and fault localization to 61.63% at chunk, 66.28% at function, and 80.85% at file level.Source: §5.4 · pp. 16–17

6 Limitation and Threats to Validity

The paper identifies missing solution verification and limited benchmark-based evaluation as key limitations affecting the practical and real-world validity of Lingma SWE-GPT.

Automatic Solution Verification

  • Automated patch verification is absent, so generated solutions are not yet checked through unit testing for correctness.The paper attributes this limitation partly to scarce end-to-end testing and debugging data and difficult environment construction.Source: §6 · p. 17

Model Evaluation

  • SWE-bench Verified and SWE-bench Lite may not fully capture the complexity or real-world applicability of software improvement compared with human developers.The authors propose direct evaluation on open GitHub issues and more sophisticated assessments of complex software engineering tasks.Source: §6 · p. 17

Future Directions

  • The authors frame these limitations as directions for future research toward more robust, adaptable, and effective developer assistance.Future work includes solution verification mechanisms and broader evaluation approaches.Source: §6 · pp. 17–18

7 Conclusion

The paper concludes that Lingma SWE-GPT is an open-source model series for complex software improvement that simulates dynamic development processes. Its 72B model reaches 30.20% on SWE-bench Verified, while the 7B model resolves 18.20% of issues, with strong fault localization and consistent multi-run performance.

Conclusion

  • 30.20% of SWE-bench Verified issues are resolved by Lingma SWE-GPT 72B, while the 7B variant resolves 18.20%.The 72B model approximates closed-source performance and the 7B result indicates potential for resource-constrained environments.Source: §7 · p. 18
  • Lingma SWE-GPT simulates dynamic software development through tool-usage reasoning and interactive problem solving rather than relying solely on static code data.The series includes 7B and 72B variants for different computational resource requirements.Source: §7 · p. 18
  • Lingma SWE-GPT achieves exceptional fault localization across chunk, function, and file levels.Source: §7 · p. 18
  • Consistent performance across multiple runs and gains from pass@3 highlight the model’s reliability and potential for ensemble methods.Source: §7 · p. 18

Related papers

Loading 2411.00622v1…