Source-linked AI summary
Open-SWE-Traces: Advancing Dual-Mode Multilingual Distillation for Software Engineering Agents
Wasi Uddin Ahmad, Nikolai Ludwig, Somshubra Majumdar, Boris Ginsburg
TL;DR
The software-engineering agent community lacks large-scale interaction traces and pre-built environments for robust model development. OPEN-SWE-TRACES uses MiniMax-M2.5 for reasoning-heavy traces and Qwen3.5-122B for diverse tool-use, with thinking/non-thinking trajectories and multi-stage quality filtering. 61.7% on SWE-bench Verified, 57.1% on SWE-bench Multilingual, and 36.8% on SWE-bench Pro were achieved by the best fine-tuned model.
Problem
The software-engineering agent community lacks large-scale interaction traces and pre-built environments for robust model development.
Method
OPEN-SWE-TRACES uses MiniMax-M2.5 for reasoning-heavy traces and Qwen3.5-122B for diverse tool-use, with thinking/non-thinking trajectories and multi-stage quality filtering.
Results
61.7% on SWE-bench Verified, 57.1% on SWE-bench Multilingual, and 36.8% on SWE-bench Pro were achieved by the best fine-tuned model.
Takeaways & Limitations
The results support OPEN-SWE-TRACES as a resource for distilling software-engineering capabilities into efficient, open-source dual-mode agents.
Takeaways & Limitations
Student behavior is tethered to teacher-model biases and errors, while stochastic models and volatile execution environments introduce reproducibility variance.
Abstract
from arXiv · showhide
The path toward autonomous software engineering is currently bottlenecked by a severe deficit of diverse, large-scale trajectory data. We address this by introducing \ourdataset, an expansive dataset of 207,489 agentic trajectories spanning nine programming languages (Python, Go, TS, JS, Rust, Java, PHP, C, C++). Sourced from 20,000 real-world PRs via OpenHands and SWE-agent harnesses, the dataset utilizes a hybrid-reasoning synthesis: Minimax-M2.5 generates trajectories with explicit "thinking" processes, while Qwen3.5-122B provides high-quality "non-thinking" traces. Filtered for permissive licenses (MIT, Apache, BSD) from SWE-rebench-V2, this data facilitates the training of models capable of long-horizon reasoning. We validate the dataset by fine-tuning the Qwen3-30B-A3B series (Thinking, Instruct, and Coder). The best performing model achieves resolve rates of 61.7% on SWE-bench Verified, 57.1% on SWE-bench Multilingual, and 36.8% on SWE-bench Pro. These results establish Open-SWE-Traces as a premier resource for distilling human-level software engineering capabilities into efficient, open-source agentic LLMs.
1 Introduction
OPEN-SWE-TRACES addresses the shortage of large-scale multilingual software-engineering interaction data with dual-mode trajectories and validates them through strong benchmark results. Its thinking traces also improve execution efficiency by reducing unproductive interaction turns.
- 207,489 multilingual trajectories address the shortage of large-scale interaction traces and executable environments for agent development.
- Dual-mode distillation combines reasoning-heavy MiniMax-M2.5 traces with direct tool-use traces from Qwen3.5-122B.
- Thinking-enabled traces reduce average OpenHands assistant turns from 94.08 to 58.22, a 38% increase in execution efficiency.
- 61.7% on SWE-bench Verified, 57.1% on SWE-bench Multilingual, and 36.8% on SWE-bench Pro were achieved by the best fine-tuned model.
- The corpus supports analysis of base models, filtering strategies, multilingual scaling, thinking versus non-thinking modalities, and generalization across execution harnesses.
2 OPEN-SWE-TRACES: From Trajectory Synthesis to Quality Filtering
OPEN-SWE-TRACES is built from multilingual, permissively licensed software tasks and synthesized through heterogeneous agent frameworks. Multi-stage validation removes corrupted, incomplete, structurally invalid, anomalous, and potentially deceptive trajectories before standardization.
- 2.1 Repository Selection and Criteria: The dataset selects SWE-rebench v2 instances in Python, Java, C, C++, JavaScript, TypeScript, Rust, Go, and PHP under MIT, Apache-2.0, or BSD licenses.
- 2.2 Agentic Trajectory Synthesis: MiniMax-M2.5 and Qwen3.5-122B generate heterogeneous traces within OpenHands and SWE-agent while agents explore repositories, run commands, and iteratively refine patches.
- 2.3 Multi-Stage Quality Filtering: The filtering pipeline aggregates logs, discards runtime failures, removes incomplete or invalid patches and malformed tool use, then standardizes role, content, tool-call, and reasoning fields.
- 2.3 Multi-Stage Quality Filtering: AST-based TrajectoryScanner removes git-hacking attempts that inspect repository metadata or internal histories for solution leaks.
- 2.4 Dataset Statistics: 207,489 trajectories span nine languages, with 51.7% containing MiniMax-M2.5 reasoning traces and 50.8% synthesized through OpenHands.
- 2.4 Dataset Statistics: 65,244 trajectories resolved their issues in unit testing, yielding an overall pass rate of approximately 40.6%.
3 Experiments
The experiments evaluate multilingual agent scaffolding, distillation variants, and benchmark performance across SWE-bench Verified, Multilingual, and Pro. OPEN-SWE-AGENT achieves strong resolve rates, while results vary by harness and reasoning mode.
- 3.1 Experiment Setup: The evaluation uses MOpenHands and MSWE-agent scaffolds, Qwen3-30B-A3B Thinking, Instruct, and Code variants, and three SWE-bench benchmarks.The dual-mode model uses the full corpus, while Thinking and Instruct variants use their respective trace subsets.
- 3.2 Experiment Results: OPEN-SWE-AGENT-INSTRUCT and OPEN-SWE-AGENT-THINKING improve by over 30.0% versus their base models on SWE-bench Verified.The comparison is reported for the monolingual evaluation in table 3.
- 3.2 Experiment Results: 61.7% resolve rate is achieved on SWE-bench Verified by OPEN-SWE-AGENT in the no-think setting, compared with 59.3% in think mode.These results represent performance under the primary evaluation settings described for the model.
- 3.2 Experiment Results: OPEN-SWE-AGENT improves over its base model by 7.7–10.1 absolute points, reaching 59.3% in think mode and 61.7% in no-think mode.The base model performance reported for comparison is 51.6%.
- 3.2 Experiment Results: 57.1% resolved rate is achieved on SWE-bench Multilingual, while SWE-bench Pro reaches 36.8% under the MSWE-agent framework.The multilingual score rises from 33.5% to 57.1%, and Pro performance rises from 28.4% to 36.8%.
- 3.2 Experiment Results: On SWE-bench Multilingual, distilled models underperform in think mode relative to no-think, while MOpenHands produces framework-specific performance drops and loops in 5–10% of cases.The paper hypothesizes that internal reasoning requires more extensive training and reports exhaustion of the maximum turn limit in these loops.
4 Ablation and Analyses
The ablations identify multilingual data and inclusion of unresolved trajectories as major performance drivers, while cross-harness transfer remains constrained by framework-specific interaction patterns.
- 4 Ablation and Analyses: The ablation study evaluates how training configurations affect agentic capabilities across environments and reasoning modalities.The analyzed factors include harness transfer, multilingual versus Python-only data, and resolved-only versus full trajectory inclusion.
- 4.1 Cross-Harness Generalization: Cross-harness transfer is achievable but incurs a consistent performance penalty across Python-centric and multilingual benchmarks.The degradation is attributed to overfitting to each framework’s interaction patterns, action spaces, or observation formats.
- 4.1 Cross-Harness Generalization: Models trained on MSWE-agent transfer more stably to MOpenHands, whereas models trained on MOpenHands suffer steeper drops when evaluated with MSWE-agent.The analysis identifies MSWE-agent as the more robust baseline for cross-harness generalization.
- 4.2 Cross-Lingual Transfer: Multilingual data raises SWE-bench Multilingual no-think performance from 43.1% to 57.1%, a +14% absolute gain over Python-only training.On SWE-bench Verified, multilingual data also slightly improves think-mode performance from 54.9% to 58.1%.
- 4.3 Data Filtering Impact: Including unresolved trajectories improves SWE-bench Multilingual from 40.5% to 47.6% in think mode and from 49.6% to 57.1% in no-think mode.The full corpus also improves SWE-bench Verified think performance from 55.3% to 58.1%.
- 4 Ablation and Analyses: The paper’s overall hierarchy of performance drivers places multilingual data and full trajectory inclusion above harness consistency, which remains a technical challenge.Peak performance requires careful alignment between training and evaluation environments.
5 Related Works
Related work spans increasingly specialized SWE benchmarks, repository-level dataset construction, and model-centric approaches that distill reasoning into software engineering agents.
- SWE Benchmarks: SWE benchmarks evolved from SWE-bench and SWE-bench-Verified toward evaluations of multimodal integration, crosslinguistic proficiency, and long-horizon reasoning.The cited benchmark ecosystem probes multiple specialized dimensions of autonomous software engineering.
- SWE Datasets: SWE programming datasets use synthetic generation, real-world mining, learned surrogate feedback, and hybrid combinations of authentic pull requests with synthetic instances.Examples include R2E-Gym, SWE-smith, SWE-Mirror, SWE-Gym, SWE-rebench, SWE-World, and SWE-Lego.
- SWE Models and Agents: Autonomous coding research is shifting from environment-based prompting toward model-centric optimization through mid-training, expert-trajectory supervised fine-tuning, and reinforcement learning.Early systems such as SWE-agent and OpenHands relied on environment-based prompting, while newer work embeds reasoning into model weights.
6 Conclusion
OPEN-SWE-TRACES provides over 200,000 software-engineering trajectories across nine languages and supports strong resolve rates across three SWE-bench evaluations.
- 61.7% on SWE-bench Verified, 57.1% on Multilingual, and 36.8% on SWE-bench Pro demonstrate strong evaluation results.
- Over 200,000 trajectories across nine languages address the scarcity of large-scale agentic software-engineering data.
- Combining explicit thinking modalities with high-quality behavioral traces is presented as an effective path for developing specialized autonomous agents.
Limitations
The distillation framework remains constrained by dependence on its teacher models and by variability from stochastic language models and software execution environments.
- Teacher-model biases and systemic errors may be inherited by students trained from Minimax-M2.5 and Qwen3.5-122B.
- Infrastructure dependencies and environmental factors can cause minor score fluctuations that affect precise reproducibility.Triplerun aggregation mitigates, but does not eliminate, variance from LLM stochasticity and volatile execution environments.
Ethics Statement
The project commits to releasing OPEN-SWE-TRACES with safeguards for permissive licensing, privacy, and credential protection.
- Agent trajectories come from open-source repositories using permissive licenses including MIT, Apache 2.0, and BSD.
- Automated filtering pipelines redact Personally Identifiable Information and sensitive credentials from the released data.
- Gemini assisted with manuscript language refinement and generation of the Figure 1 visual data-flow illustration.
A Implementation Details
The implementation appendix documents the training and inference hyperparameters used for OPEN-SWE-AGENT in Table 6.
- The SFT and inference hyperparameters are detailed in Tables 6a and 6b.
- Table 6 provides a comprehensive breakdown of OPEN-SWE-AGENT hyperparameters.