Source-linked AI summary
SWE-Prime: Fewer Trajectories, Better Performance
Dewu Zheng, Ruizhe Ye, Yanlin Wang, Yang Ye, Hongyu Zhang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jianxing Yu, Zibin Zheng
TL;DR
Successful coding-agent trajectories can still provide noisy supervision when they contain ineffective, redundant, or risky behaviors, so task success alone is insufficient. SWE-Prime uses two-stage selection over trajectories and semantic segments while preserving full-sequence context, and its selected 10% subset outperforms training on the full resolved pool. The paper reports maximum relative gains of 12.2% on SWE-Bench Pro and 24.2% on SWE-Bench Verified.
Problem
Task success does not guarantee suitable SFT supervision because successful trajectories may contain ineffective, redundant, risky, or non-contributory behaviors.
Method
SWE-Prime first selects high-quality and representative successful trajectories, then selects valuable semantic segments while retaining all segments as context and applying loss only to selected segments.
Results
12.2% and 24.2% are the maximum relative gains on SWE-Bench Pro and SWE-Bench Verified, respectively, for the selected 10% trajectory subset versus the full resolved-trajectory pool.
Takeaways & Limitations
Effective coding-agent SFT depends on supervision quality, representativeness, and learnability in addition to successful task outcomes.
Takeaways & Limitations
Sliding-window processing can split ongoing behaviors at window boundaries, causing incorrect segmentation and unreliable segment scoring.
Abstract
from arXiv · showhide
To improve large language models' ability to resolve real-world software issues, prior work has focused on constructing large-scale agent trajectory datasets and performing supervised fine-tuning (SFT) on successful trajectories. However, task success does not guarantee high-quality supervision: successful trajectories may still contain ineffective, redundant, or risky steps. Directly using such trajectories for SFT can introduce noisy supervision and encourage models to imitate undesirable problem-solving behaviors. Therefore, we propose SWE-Prime, a multi-granularity, two-stage SFT data selection method that progressively filters training data at the trajectory and segment levels. Specifically, the first stage performs trajectory-level screening based on process quality, result quality, and data representativeness, selecting a high-quality and representative subset of successful trajectories. The second stage performs segment-level selection by grouping consecutive steps into semantic segments and assessing each segment based on its contribution to the final solution, learnability, and potential risks. During SFT, all segments remain in the sequence to preserve context, while only selected segments contribute to the loss computation. Experiments on SWE-Bench Pro and SWE-Bench Verified show that training on the 10% trajectory subset selected by SWE-Prime outperforms training on the full resolved dataset, yielding relative performance gains of up to 12.2% and 24.2%, respectively.
Introduction
Successful software-issue trajectories can contain ineffective, redundant, risky, or otherwise unsuitable supervision despite achieving task success. SWE-Prime addresses this by progressively selecting trajectories and semantic segments, and its 10% subset improves performance over full successful-trajectory training.
- Motivation: Task success alone does not guarantee high-quality supervision because successful trajectories may contain ineffective, redundant, risky, or irrelevant behaviors.These behaviors include unstable tool use, repeated actions, leakage-based shortcuts, unnecessary file modifications, and segments that do not contribute to the final fix.
- Motivation: Trajectory-level assessment distinguishes demonstrations by problem-solving pattern, tool-call reliability, redundancy, shortcut use, and result quality.For example, one trajectory follows an observe-edit-verify workflow, while another modifies code without exploration and repeats localization actions.
- Motivation: Even high-quality trajectories can contain low-value semantic segments whose successful steps do not contribute to the final fix.Semantic segments group multiple actions and observations so their intent, outcome, and contribution can be assessed together.
- SWE-Prime: SWE-Prime progressively filters supervision at the trajectory and semantic-segment levels, retaining complete sequences as context while computing loss only on selected segments.The method selects a high-quality, representative subset of successful trajectories before identifying high-value behaviors within them.
- Results: 12.2% and 24.2% are the maximum relative gains on SWE-Bench Pro and SWE-Bench Verified, respectively, when training on the selected 10% subset versus all successful trajectories.The experiments evaluate SWE-Prime on both benchmarks.
Related Work
Coding-agent SFT commonly relies on large collections of successful, test-verified trajectories, but success and explicit errors do not fully identify valuable supervision. SWE-Prime extends filtering to quality-aware trajectory and semantic-segment selection.
- Coding-agent SFT: Repository-level issue resolution requires agents to understand context, locate code, implement changes, and validate patches without regressions.These tasks are described in natural language and require iterative interaction with repositories and execution signals.
- Existing datasets: Prior work scales executable-task and trajectory construction using real-world issues, synthetic bugs, or code commits, then retains successful trajectories for SFT.SWE-Gym is an example that constructs executable environments from GitHub issues and collects test-verified trajectories.
- Remaining gap: Filtering based only on task success and explicit errors can miss unreliable processes, unfocused patches, redundant behaviors, and risky segments.This motivates evaluating supervision value at both trajectory and semantic-segment levels.
SWE-Prime
SWE-Prime progressively selects supervision at trajectory and semantic-segment levels, retaining complete trajectories as context while applying loss only to selected segments.
- Overview: SWE-Prime uses two stages: trajectory-level selection followed by semantic-segment selection within retained trajectories.Stage 1 selects high-quality, representative trajectories; Stage 2 selects valuable behaviors within them.
- Stage 1: Trajectory-Level Selection: Trajectory scoring combines process quality, result quality, and data representativeness to select focused, reliable demonstrations.Process signals include workflow grounding, tool-call success, redundancy, and Git-hacking detection; result quality measures modification scope.
- Stage 1: Trajectory-Level Selection: HDBSCAN groups semantically similar issues so high-scoring trajectories can be selected within issue groups, reducing redundancy and improving coverage.Trajectories are ranked by Straj within each group under the target budget.
- Stage 2: Segment-Level Selection: Consecutive steps sharing an intent form semantic segments because individual actions or observations often lack enough context to assess behavior.Sliding-window processing can truncate boundaries, so the final segment is carried into the next window and reevaluated with later steps.
- Stage 2: Segment-Level Selection: Each segment is scored for contribution to the final fix, local learnability, and behavioral risk, then selected when qi,j ≥ δ.Selected segments receive mask mi,j = 1; others receive mi,j = 0.
- Stage 2: Segment-Level Selection: All segments remain in the sequence as context, but only assistant-response tokens in selected segments contribute to the selective SFT loss.The mask excludes unselected segments from loss computation while their tokens remain available for subsequent predictions.
Experimental Setup
The experiments assess SWE-Prime across three base models and two software issue-resolution benchmarks, using resolved trajectories as the initial candidate pool and comparing targeted selection with established baselines.
- Research Questions: SWE-Prime addresses three research questions covering effectiveness, hyperparameter sensitivity, and behavioral changes.The questions examine performance improvement, configuration generalization, and how gains appear in model behavior.
- Training Data: The initial candidate pool contains 32,161 trajectories that successfully resolve issues, drawn from 67,074 OpenHands trajectories.The trajectories come from the SWE-rebench OpenHands Trajectories dataset.
- Models and Benchmarks: Evaluation uses GLM-4.7-Flash2, Qwen3-30B-A3B-Instruct-25073, and Qwen3-Coder-30B-A3B-Instruct4 on SWE-Bench Verified and SWE-Bench Pro.SWE-Bench Verified contains 500 human-validated tasks, while the study uses the 731-task public split of SWE-Bench Pro.
- Baselines: Baselines include the raw model, SFT on all 32,161 resolved trajectories, and a random 10% subset under the same trajectory budget.These comparisons separate pre-SFT performance, unfiltered successful-trajectory training, and targeted selection.
- Training Configuration: Training uses a maximum sequence length of 131,072, batch size 32, AdamW, cosine learning-rate scheduling, and a peak learning rate of 4 × 10^-6.SWE-Prime differs through its training data and loss masks, while applying loss to assistant tokens in selected segments.
- Evaluation Setting: Evaluation allows up to 100 interaction turns per task, uses model-default sampling parameters, and validates final patches in each benchmark’s execution environment.The maximum evaluation sequence length is 131,072.
Results
SWE-Prime improves issue-resolution performance across models and benchmarks while using a 10% trajectory subset, and segment-level selection adds value beyond trajectory filtering alone. The selected configuration generalizes across settings and is accompanied by more structured, efficient behavior.
- Overall Effectiveness: 18.8%–57.1% relative improvement on SWE-Bench Verified and 17.6%–140.2% on SWE-Bench Pro over raw models.SWE-Prime achieves the highest Resolved Rate across all evaluated models and both benchmarks.
- Segment-Level Selection: SWE-Prime consistently outperforms its Stage 1-only variant across all evaluated settings.The comparison indicates that trajectory-level filtering alone does not fully address low-value behaviors within otherwise high-quality trajectories.
- Hyperparameter Analysis: A 10% trajectory retention ratio and segment score threshold of 7 are selected and applied unchanged throughout the main evaluation.The configuration remains effective beyond the model and benchmark used for hyperparameter analysis.
- Behavioral Analysis: SWE-Prime improves all three process-quality metrics and reduces average interaction turns by 4.1 to 12.7 turns versus Resolved-Trajectory SFT.The reported changes include lower redundancy, higher observe-before-edit rates, and higher tool success rates across evaluated settings.
Conclusion
SWE-Prime selects high-quality supervision through trajectory- and segment-level filtering rather than relying on successful outcomes alone. Using only 10% of trajectories, it outperforms training on the entire resolved-trajectory pool, with gains of up to 12.2% and 24.2%.
- Conclusion: SWE-Prime selects representative trajectories and valuable semantic segments using process quality, result quality, contribution, learnability, and risk criteria.The method targets supervision quality beyond task success.