PaperlayerView exact revision on arXiv

Source-linked AI summary

SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuning

Zexiong Ma, Chao Peng, Pengfei Gao, Xiangxin Meng, Yanzhen Zou, Bing Xie

arXiv:2502.20127v1cs.SEcs.AIcs.CL

TL;DR

Commercial-model issue resolution raises cost and privacy concerns, while existing open-source fine-tuning can generalize poorly. SoRFT trains on ground-truth-derived issue-resolving subtasks using filtered supervised data and rule-based reinforcement learning. It achieves state-of-the-art performance among open-source models on SWE-Bench Verified and Lite.

  • Problem

    Commercial models dominate automated software-development systems despite cost and privacy concerns, while SFT-only issue-resolving approaches can generalize poorly.

    Source: §1 · p. 1

  • Method

    SoRFT derives ground-truth answers from issue-associated pull requests for four subtasks, then applies rejection-sampled SFT and rule-based PPO.

    Source: §1 · p. 2

  • Results

    SoRFT achieves state-of-the-art performance among open-source models on SWE-Bench Verified and Lite, with SoRFT-Qwen-7B scoring 21.4 versus SWE-Gym-Qwen-32B’s 20.6 on Verified.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The authors conclude that SoRFT improves performance and generalization while offering a cost-efficient alternative to commercial models.

    Source: §7 · p. 9

    Comparing edits with a single ground truth can misclassify valid solutions as failures; the authors suggest incorporating unit-test results.

    Source: §8 · p. 9

Abstract

from arXiv · show

Mainstream issue-resolving frameworks predominantly rely on commercial models, leading to high costs and privacy concerns. Existing training approaches for issue resolving struggle with poor generalization and fail to fully leverage open-source development resources. We propose Subtask-oriented Reinforced Fine-Tuning (SoRFT), a novel training approach to enhance the issue resolving capability of LLMs. We decomposes issue resolving into structured subtasks: file localization, function localization, line localization, and code edit generation. SoRFT consists of two training stages: (1) rejection-sampled supervised fine-tuning, Chain of Thought (CoT) data is filtered using ground-truth before fine-tuning the LLM, and (2) rule-based reinforcement learning, which leverages PPO with ground-truth based rewards. We evaluate the SoRFT-trained model on SWE-Bench Verified and SWE-Bench Lite, achieving state-of-the-art (SOTA) performance among open-source models (e.g., resolve 21.4% issues on SWE-Bench Verified with SoRFT-Qwen-7B). The experimental results demonstrate that SoRFT significantly enhances issue-resolving performance, improves model generalization, and provides a cost-efficient alternative to commercial models.

1 Introduction

Commercial-model issue resolution raises cost and privacy concerns, while supervised fine-tuning of open-source models can generalize poorly. SoRFT uses ground-truth-derived subtasks and two-stage training to improve open-source LLMs’ issue-resolving capabilities.

1 Introduction

  • Mainstream automated software-development systems mainly use commercial models, whose API calls are costly and pose privacy-leakage concerns.The paper frames these concerns as limiting commercial models’ application in industrial development processes.Source: §1 · p. 1
  • SFT-only issue-resolving approaches can generalize poorly and become more susceptible to hallucinations and factual errors.Prior approaches sample CoT data and filter negative samples using ground truth before fine-tuning.Source: §1 · p. 1
  • Resolved open-source issues provide (issue, patch) pairs that can supply ground truth for rule-based reinforcement learning.The paper raises using these pairs to improve language models’ issue-resolving capabilities.Source: §1 · p. 2
  • Issue resolving is divided into file, function, and line localization plus code edit generation, with subtask answers derived from associated pull requests.The authors say end-to-end training data is challenging to construct for this task.Source: §1 · p. 2
  • SoRFT combines rejection-sampled supervised fine-tuning with rule-based reinforcement learning using PPO and ground-truth-based rewards.A teacher LLM generates subtask Chain-of-Thought data; negative samples are filtered before supervised fine-tuning.Source: §1 · p. 2
  • The authors apply SoRFT to open-source models and validate it within the Agentless framework.They also investigate how different reward rules affect PPO training to inform more robust reward design.Source: §1 · p. 2

2 Background

The section introduces SWE-Bench and its curated subsets, reviews issue-resolving framework types, and motivates pipeline-based evaluation. It also explains the paper’s use of Agentless and PPO with rule-based scoring.

2.1 SWE-Bench

  • SWE-Bench evaluates issue resolution from an issue description and full repository; Lite focuses on functional bug fixes, while Verified is human-verified against benchmark quality issues.The paper conducts experiments on SWE-Bench Lite and SWE-Bench Verified.Source: §2.1 · p. 2

2.2 Issue Resolving Framework

  • Issue-resolving systems include agent-based and pipeline-based frameworks; the paper uses Agentless 1.0 for inference and designs training subtasks around it.Pipeline frameworks offer a clearer way to evaluate reasoning accuracy at each stage than agent frameworks, according to the section.Source: §2.2 · pp. 2–3, §2.3 · p. 3

2.3 Reinforcement Learning

  • PPO calculates reward using an output score and then updates the policy model; the paper selects PPO for stable and efficient policy updates.The reward uses a reward model and a KL penalty relative to a reference policy; the paper instead describes ground-truth-based scoring rules for issue resolving.Source: §2.3 · p. 3

3 Approach

SoRFT organizes issue resolution into four subtasks, then trains with rejection-sampled supervised fine-tuning and rule-based reinforcement learning. Ground-truth answers derived from resolved issues guide both data filtering and subtask rewards.

3 Approach

  • SoRFT combines issue-resolving subtask construction, rejection-sampled supervised fine-tuning, and rule-based reinforcement learning.The framework is designed to address the challenge of constructing end-to-end training data for issue resolving.Source: §3 · p. 3, §3.1 · p. 3

3.1 Issue Resolving Subtasks

  • File localization maps an issue and repository skeleton to modified file names, excluding non-Python files and test scripts from inputs and outputs.Ground-truth answers for the subtasks are extracted from pull requests associated with resolved issues.Source: §3.1 · p. 4
  • Function localization uses an issue and file function-name skeleton to predict the modified function names from the pull request.Source: §3.1 · p. 4
  • Line localization takes an issue and function content as input and predicts the modified lines from the pull request.Source: §3.1 · p. 4
  • Code edit generation takes an issue and localized code snippet as input and produces the corresponding pull-request edits in Search/Replace format.The format has a search target and a replacement code snippet.Source: §3.1 · p. 4

3.2 Rejection-sampled Supervised Fine-Tuning

  • Rejection-sampled SFT filters generated CoT samples against ground-truth overlap, then fine-tunes on data integrated across all subtasks.Localization samples without overlap with the ground-truth file, function, or line are filtered; code-edit samples without overlap with ground-truth modified lines are also filtered.Source: §3.2 · p. 4

3.3 Ruled-based Reinforcement Learning

  • Rule-based PPO rewards localization and code-edit outputs with Fβ scores against ground truth, assigning zero when outputs are empty or targets are absent from the problem description.The paper sets β = 3 to prioritize recall while maintaining a reasonable trade-off with precision.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate SoRFT-trained models on SWE-Bench Verified and SWE-Bench Lite using Agentless and Qwen2.5-Coder base models, alongside framework-matched baselines. The section also specifies benchmark metrics and training-data and implementation details.

4.1 Benchmark

  • SWE-Bench Verified has 500 manually verified issues with executable tests, while SWE-Bench Lite has 300 instances focused on functional bug fixes.Frameworks must generate patches that pass all test cases on Verified.Source: §4.1 · p. 5

4.2 Metrics

  • Evaluation reports %Resolved and %Applied, measuring test-passing edits and valid edits applicable to repositories, respectively.%Resolved requires that the generated edit pass all test cases.Source: §4.2 · p. 5

4.3 Framework and Model

  • Experiments use Agentless with Qwen2.5-Coder-7B-Instruct and Qwen2.5-Coder-32B-Instruct as the base models.Baselines include OpenHands with SWE-Gym-Qwen, SWE-Fixer with SWE-Fixer-Qwen, and SWE-SynInfer with Lingma-SWE-GPT.Source: §4.3 · p. 5, §4.4 · p. 5

4.4 Baselines

  • Table 1 lists SoRFT-Qwen-32B at 30.8 and 24.0, versus Lingma-SWE-GPT-72B at 30.2 and 22.0.The comparison is among %Resolved results on SWE-Bench Verified and SWE-Bench Lite, within corresponding frameworks.Source: Table 1 · p. 6

4.5 Implementation Details

  • Training data comes from 30,000 issue–PR pairs across 100 repositories, yielding 60,000 filtered SFT samples; 30,000 samples are selected for RL.The selected repositories exclude SWE-Bench test repositories to prevent data contamination.Source: §4.5 · pp. 6–7
  • The 7B and 32B models are fine-tuned for two epochs with a global batch size of 128; PPO samples completions at temperature 1.0.Fine-tuning uses 4×8 96G H20 GPUs; the 7B and 32B setups use FSDP and DeepSpeed, respectively.Source: §4.5 · p. 7

5 Results and Analysis

SoRFT outperforms open-source models of comparable size on SWE-Bench Verified, and full SoRFT training beats SFT alone. Reward-rule comparisons identify differences in generated reasoning and answer behavior, while evaluations also consider general code tasks.

SoRFT achieves SOTA performance among open-source LLMs

  • 21.4% for SoRFT-Qwen-7B exceeds SWE-Gym-Qwen-32B’s 20.6% on SWE-Bench Verified, while SoRFT-Qwen-32B’s 30.8% exceeds Lingma-SWE-GPT-72B’s 30.2%.The authors report state-of-the-art performance among open-source models of the same parameter size and say SoRFT-Qwen-32B slightly outperforms the larger model.Source: §5 · p. 7
  • Pipeline-based frameworks can provide intermediate-step supervision, enabling finer-grained CoT filtering and more precise reward calculation than agent-based supervision.The authors suggest this may explain why SWE-Gym-Qwen underperforms despite OpenHands performing optimally with proprietary models.Source: §5 · p. 7

SoRFT achieves higher accuracy than solely supervised fine-tuning

  • Full SoRFT training consistently outperforms SFT alone for both 7B and 32B models on SWE-Bench Verified, across %Resolved and %Applied.The passage attributes improved issue-resolving ability to the results on both metrics.Source: §5 · p. 7, Table 2 · p. 7

The robustness of reward rules is crucial for reinforcement learning

  • Hit-score rewards prompt reward hacking with fewer thoughts and more answers, whereas Fβ rewards reduce redundant answers and stabilize answer length.The paper reports a performance gap between reward strategies and notes thought length first decreases, then increases during PPO training.Source: §5 · p. 7, Figure 3 · p. 8

SoRFT also enhances performance on general code tasks

  • SoRFT is reported to have potential to enhance code-related tasks beyond issue resolving, evaluated with LiveCodeBench and RepoQA.LiveCodeBench covers self-contained code generation; RepoQA evaluates information extraction from long-context code.Source: §5 · pp. 7–8, Table 3 · p. 8

6 Related Work

Prior issue-resolving fine-tuning work used supervised learning, while SoRFT applies reinforced fine-tuning to the task. Related rule-based reinforcement-learning research had focused mainly on mathematics.

LLM Training for Issue Resolving

  • Earlier issue-resolving approaches built training data from open-source development resources and fine-tuned models with SFT; the authors claim SoRFT is the first to use reinforced fine-tuning for this task.Prior work filtered agent trajectories or CoT data generated using closed-source models.Source: §6 · p. 8

Reinforcement Learning with Rule-based Reward

  • Recent rule-based reinforcement-learning work includes models and studies focused on mathematical tasks, while this paper applies subtask-oriented rule-based RL to issue resolving.The related-work passage describes ground-truth rewards as straightforward to compute for mathematics.Source: §6 · p. 8
  • SoRFT’s issue-resolving approach draws on rule-based reinforcement learning, an area also explored in work on long-form reasoning.The cited examples include DeepSeek R1, Kimi-k1.5, and PPO training on math tasks.Source: §6 · p. 8

7 Conclusion

The conclusion presents SoRFT as a subtask-oriented reinforced fine-tuning approach for improving LLM issue resolution. It reports better performance and generalization, and describes the approach as a cost-efficient alternative to commercial models.

7 Conclusion

  • SoRFT is presented as a subtask-oriented reinforced fine-tuning approach intended to improve LLM issue-resolving capabilities.Source: §7 · p. 8
  • The authors report that SoRFT improves performance while ensuring better generalization.Source: §7 · p. 9
  • The conclusion describes SoRFT as a cost-efficient alternative to commercial models.Source: §7 · p. 9

8 Limitations

The paper identifies two limitations: rule-based rewards can reject valid alternative solutions, and the experiments cover only Python repositories.

8 Limitations

  • A single ground-truth patch can cause rule-based rewards to classify valid alternative issue resolutions as failures.The authors suggest unit-test results as a more objective and fair measure of code-edit quality.Source: §8 · p. 9
  • Because no multilingual SWE-Bench test set was available, experiments were limited to Python repositories; performance in other languages remains untested.The authors state that SoRFT is language-agnostic and may have potential in other languages.Source: §8 · p. 9

A Evaluation Details

The evaluation used SWE-Bench’s official scripts for Verified and Lite, alongside specified LiveCodeBench and RepoQA evaluations. Figure 4 plots reward over PPO training steps.

A Evaluation Details

B PPO Training Details

SoRFT’s subtask results show improved localization hit rates, while the supplementary material describes its PPO reward trend, reasoning example, and subtask prompts.

B PPO Training Details

  • With the designed reward rules, reward steadily increased as PPO training steps progressed.The PPO setup used a train batch size of 64, actor learning rate 5e-7, and critic learning rate 9e-6.Source: B PPO Training Details · p. 12

C Additional Reasoning Example

  • After SoRFT training, the 7B model continued reasoning and identified the ground-truth function sql_flush in the example.The model explored whether potential answers had been overlooked before concluding.Source: C Additional Reasoning Example · p. 12

D Subtask Prompt

  • Each subtask prompt combines a subtask description, issue, context, and output-format instructions.The prompts ask the model to analyze the issue, select relevant content from context, and return a formatted answer.Source: D Subtask Prompt · p. 12

E Effectiveness of SoRFT on issue resolving subtasks

F Training Data Ablation

The training-data ablation reports that SoRFT outperformed SFT even at the same 90k-sample amount, while increasing SFT data had mixed outcomes.

F Training Data Ablation

  • 90k-sample SFT still fell short of SoRFT when both used the same amount of training data.The comparison was conducted to assess whether SoRFT’s advantage reflected its larger training-data amount.Source: F Training Data Ablation · pp. 12–13
  • Increasing SFT from 60k to 90k samples slightly improved %resolved but decreased %Applied; the authors suggest excessive SFT might impair generalization.They characterize SoRFT as more robustly enhancing issue-resolving capability.Source: F Training Data Ablation · p. 13

Related papers

Loading 2502.20127v1…