Source-linked AI summary
SEAlign: Alignment Training for Software Engineering Agent
Kechi Zhang, Huangzhao Zhang, Ge Li, Jinliang You, Jia Li, Yunfei Zhao, Zhi Jin
TL;DR
Code models trained largely on competitive-programming tasks remain misaligned with practical agentic software engineering, which requires repository retrieval and context-sensitive use of algorithms. SEAlign aligns software-agent trajectories through MCTS and preference optimization, and reports state-of-the-art results on real-world engineering benchmarks.
Problem
Existing code-model post-training focuses mainly on competitive programming, while practical software engineering requires repository retrieval and applying algorithms in code context.
Source: §1 · p. 1Method
SEAlign uses high-quality software-agent trajectories for fine-grained MCTS alignment across multi-step decisions, followed by preference optimization on critical actions.
Source: §1 · pp. 1–2Results
SEAlign achieves state-of-the-art performance across real-world software-engineering benchmarks; its 14B model reaches a 17.7% resolved rate on SWE-Bench-Lite.
Source: §1 · p. 2Takeaways & Limitations
Applying SEAlign in software-engineering scenarios successfully automated the creation of several small applications.
Source: §8 · p. 11Scaling is constrained by extended execution time and the resource costs of constructing larger datasets.
Source: §6.4 · p. 9
Abstract
from arXiv · showhide
Recent advances in code generation models have demonstrated impressive capabilities in automating software development tasks, yet these models still struggle in real-world software engineering scenarios. Although current training methods, particularly post-training, excel at solving competitive programming problems, they fail to adequately prepare models for the complexities of practical software development. This misalignment raises the critical question: Are existing alignment training methods well suited for real-world software engineering tasks? In this study, we identify this issue and propose SEAlign, a novel alignment framework designed to bridge the gap between code generation models and real-world software development tasks. SEAlign leverages the unique characteristics of software engineering processes, including high-quality workflow steps, to enhance model capabilities. Our framework further employs Monte Carlo Tree Search for fine-grained alignment in multi-step decision processes, followed by preference optimization on critical actions to ensure models meet real-world requirements. We evaluate SEAlign on three standard agentic benchmarks for real-world software engineering, including HumanEvalFix, SWE-Bench-Lite, and SWE-Bench-Verified. Experimental results demonstrate state-of-the-art performance with minimal training overhead. In addition, we develop an agent-based software development platform using SEAlign, which successfully automates the creation of several small applications. Human evaluations of these applications highlight significant improvements in both task performance and user experience. Our findings underscore the potential of SEAlign to accelerate the adoption of large code models in real-world software development. We believe that this research makes a meaningful step towards fully automated software engineering.
1 Introduction
The introduction argues that contest-oriented code-model post-training misses real-world software-engineering demands and presents SEAlign as a workflow-alignment approach. It reports benchmark performance and an application-building demonstration.
1 Introduction
- Existing code-model post-training focuses mainly on competitive programming, while practical software engineering requires repository information retrieval and context-sensitive use of algorithms.The authors argue that this gap matters for software-development automation.Source: §1 · p. 1
- Qwen2.5-Coder-Instruct-14B with OpenHands resolved only 3.7% of SWE-Bench-Lite problems; failures involved instruction following, tool use, and agentic loops.The preliminary evaluation is presented as evidence that existing post-training does not adequately prepare models for agentic software-engineering tasks.Source: §1 · p. 1
- SEAlign collects software-development trajectories, identifies critical workflow actions, scores multi-step decisions with MCTS, and then applies preference optimization.The authors describe this as aligning models with real-world requirements and agentic workflows.Source: §1 · pp. 1–2
- 17.7% on SWE-Bench-Lite and 21.8% on SWE-Bench-Verified were resolved by SEAlign’s 14B model; the paper reports state-of-the-art benchmark performance.The authors also report performance comparable to several commercial products with hundreds of optimization samples.Source: §1 · p. 2
- An SEAlign-based development platform automated several small applications, and human evaluations found significant improvements in effectiveness and user experience.The introduction frames this as a practical demonstration of the approach.Source: §1 · p. 2
2 Related Work
Related work traces code models from pretraining and supervised fine-tuning to post-training, then identifies a mismatch between contest-oriented alignment and practical software-engineering workflows.
2.1 Large Language Models for Code
- Code-generation research uses pretraining and SFT, while self-instruct methods synthesize training data; Evol-Instruct [21] uses complex prompts and OSSinstruct [35] uses real-world code snippets.The cited approaches seek to improve synthesized-data quality or relevance.Source: §2.1 · p. 2
- Pretraining supplies language and programming abilities, while SFT supports instruction following; code models have reached human-level performance on competitive-programming benchmarks such as HumanEval [3] and LiveCodeBench [11].The related-work discussion presents post-training as a response to more complicated real-world tasks.Source: §2.1 · p. 2
2.2 Alignment Post-Training for Code Models
- Code-Optimise [5], CodeDPO [43], and PLUM [41] align models using trivial programming problems, but practical engineering also requires codebase navigation, contextual edits, and architecture restructuring.The authors identify this difference as a need for alignment training tailored to practical agentic workflows.Source: §2.2 · p. 2
3 A Motivating Example
The motivating example contrasts standalone code-generation strengths with the demands of environment-based software engineering. Its preliminary agent evaluation exposes low task resolution alongside instruction, tool-use, and looping failures.
3.1 Code Agents & Agentic Tasks
- Real-world developers must navigate codebases and understand architecture; agents interact with their environment through repeated tool actions and observations.The trajectory begins with a task prompt and continues through agent actions and environment responses.Source: §3.1 · p. 3
- Long, diverse, context-dependent trajectories make it difficult to construct datasets and train models that generalize across complex scenarios.The passage identifies trajectory length and project diversity as dataset and generalization challenges.Source: §3.1 · p. 3
3.2 Failure Caused by Misalignment
- Despite code-generation strengths on algorithmic tasks, Qwen2.5-Coder-Instruct-14B with OpenHands resolved only 3.7% of SWE-Bench-Lite problems.The benchmark simulates real-world software-engineering issues.Source: §3.2 · p. 3, §3 · p. 3
- Observed failure modes include misreading complex multi-step prompts, incorrect tool selection or use, and repetitive loops that can cause task failure or excess computation.The prompt failures can involve output constraints and workflow stages; loops may repeat actions without progress.Source: §3.2 · p. 3
4 SEAlign
SEAlign aligns code models to software-engineering agent workflows through trajectory collection, tree-based scoring of critical actions, and preference optimization. Its training uses correct trajectories for SFT and critical action pairs for DPO.
4 SEAlign
- SEAlign collects agent trajectories, builds trajectory trees, scores critical actions, and preference-optimizes the model to favor actions associated with higher success rates.The framework is designed for agentic software-development systems and uses partial trajectory pairs for post-training.Source: §4 · pp. 3–4
4.1 Trajectory Dataset Collection
- Trajectories from agents built with strong models and frameworks are paired with task outcomes, while evaluation repositories and code snippets are excluded to avoid data leakage.Each trajectory records actions and observations, and its final result indicates task success or failure.Source: §4.1 · p. 4
4.2 Trajectory Tree Construction
- Shared action prefixes across trajectories are merged into trees, where non-leaf nodes represent decisions and leaves represent task success or failure.The framework also removes trajectories that repeat actions without progress or fail to overlap meaningfully with others.Source: §4.2 · pp. 4–5
4.3 Node Scoring & Critical Action Extraction
- A score difference greater than 0.5 between sibling non-leaf nodes marks their actions as a critical pair for training toward the higher-scoring action.The algorithm identifies actions whose alternatives are associated with substantially different success rates.Source: §4.3 · p. 5, §4.2 · p. 5
4.4 Alignment with Preference Optimization
- Training first uses SFT on correct trajectories, masking environment observations, then applies DPO to critical preferred and dispreferred partial trajectories sharing the same action prefix.The preferred action is associated with a higher success rate, and the score-difference threshold is 0.5.Source: §4.4 · pp. 5–6
5 Experimental Setups
The experimental setup evaluates SEAlign on software-engineering and debugging benchmarks, with ablations, data scaling, and human evaluations among its research questions. It specifies benchmark metrics, coding-model baselines, and training and inference settings.
5 Experimental Setups
- The study asks whether SEAlign improves real-world code correctness and general agent capability, and examines preference-optimization design, data scaling, and user experience.The user-experience evaluation includes applications constructed using SEAlign.Source: §5 · p. 6
5.1 Benchmarks
- Evaluation uses SWE-Bench-Lite, SWE-Bench-Verified, and HumanEvalFix, and excludes their repositories and code snippets from training data to prevent contamination.The SWE-Bench variants test real-world software tasks; HumanEvalFix tests debugging of functions with subtle bugs.Source: §5.1 · p. 6
5.2 Metrics
- SWE-Bench tasks are primarily measured by resolved rate, while HumanEvalFix uses unit-test pass rate; the setup also records specified patch and loop failure rates.SWE-Bench evaluations stop at 30 interaction turns or the model’s 32k context limit, whichever comes first.Source: §5.2 · pp. 6–7
5.3 Baselines
- The evaluated coding baselines are Qwen-2.5-Coder-Instruct-7B and Qwen-2.5-Coder-Instruct-14B, and training uses the OpenHands agentic framework.OpenHands uses the CodeActAgent strategy for tool-use and reasoning decisions.Source: §5.3 · p. 7
5.4 Training & Inference Setups
- Training trajectories are constructed using Python-focused data, and inference uses greedy search for deterministic evaluation.Training is capped at 10 epochs, with the checkpoint selected by lowest validation loss.Source: §5.4 · p. 7
6 Experimental Results
SEAlign improves SWE-Bench performance against baselines and many state-of-the-art methods, while showing robustness on HumanEvalFix. Ablations highlight the role of fine-grained preference optimization, and experiments report gains from more training data and improved application evaluations.
6.2 HumanEvalFix (RQ2)
- HumanEvalFix broadens evaluation to complex instruction following and tool use, with no HumanEvalFix training data and substantially different data styles and formats.Table 4 reports HumanEvalFix performance and defines Invalid as the invalid-patch rate within agentic mode.Source: §6.2 · p. 7, Table 4 · p. 8
6.1 SWE-Bench Family (RQ1)
- SEAlign improves SWE-Bench resolved rates over baselines, matches GPT-4o with OpenHands and a 14B LLM, and leads comparable-size open-source methods, behind only a few 72B models.The comparisons cover SWE-Bench-Lite and SWE-Bench-Verified; the reported baseline improvement uses less than a thousand training samples.Source: Table 2 · p. 7, §6.1 · p. 7, Table 3 · p. 8
6.3 Ablation Studies (RQ3)
- SFT provides a foundation that DPO further improves, while omitting critical-action identification and optimizing whole trajectories significantly degrades results.The fine-grained preference optimization aligns critical decisions at key workflow steps and contributes substantially to task success.Source: §6.3 · p. 8, Table 5 · p. 8
6.4 Data Scaling Law for SEAlign (RQ4)
- More training examples consistently improve target-LLM performance, but scaling is constrained by execution time and the resources required to construct larger datasets.The experiments balance performance gains against training costs.Source: §6.4 · pp. 8–9, Table 6 · p. 8
6.5 User Experience (RQ5)
- In five manually designed development tasks, human evaluation found that SEAlign improves user experience; the to-do-list case contrasts complete feature implementation with a baseline that struggled.Evaluators assessed functionality completeness, code quality, and aesthetic appeal.Source: §6.5 · p. 9
7 Discussions & Future Work
The discussion finds that SEAlign performs well within its target agentic framework but transfers less effectively across frameworks, while execution validation constrains scaling. It also identifies limits in the RL comparison and in the study’s validity and evaluation scope.
7.1 Generalization across Agentic Frameworks
- Framework-specific alignment supports proficiency within its target workflow, but OpenHands-trained SEAlign-14B transfers only reasonably above baseline and falls behind GPT-4o on another framework.SEAlign-14B is reported as comparable to GPT-4o on OpenHands, while the performance gap widens on AutoCoderRover.Source: §7.1 · pp. 9–10, Table 8 · p. 10
7.2 Overhead for Execution Validation
- Execution validation constrains scaling: it requires about 1TB of free disk space and nearly 5 minutes per sample on a 16-CPU server.The resulting time and resource costs affect dataset construction and experimental evaluation; larger datasets or more environments require more efficient validation.Source: §7.2 · p. 10
7.3 Comparison with Advanced RL Techniques
- The study has not extensively compared SEAlign with advanced RL alignment; a simplified online iterative DPO experiment showed limited improvement, leaving effectiveness and efficiency comparisons open.The experiment used an additional 50% on-policy dataset based on SEAlign.Source: §7.3 · p. 10
7.4 Threats to Validity
- Validity is bounded by trajectory and hyperparameter choices, benchmark coverage, and resolve rate’s omission of efficiency, code quality, and usability.The selected benchmarks may not represent all software-engineering tasks or repositories.Source: §7.4 · p. 10
8 Conclusion
SEAlign targets the gap between code-model training and real-world software-engineering agents through workflow-informed alignment. The paper reports state-of-the-art results across multiple benchmarks and automated creation of several small applications.
8 Conclusion
- SEAlign identifies critical decision points and combines fine-grained alignment with preference optimization to help models handle complex software-engineering tasks in agentic frameworks.The framework is designed to address the gap between existing code-model training methods and real-world agent requirements.Source: §8 · pp. 10–11
- SEAlign achieves state-of-the-art performance across multiple benchmarks designed to simulate real-world software-engineering challenges.The conclusion does not specify benchmark names or metric values in these passages.Source: §8 · p. 11
- Using SEAlign, the authors automate the creation of several small applications in real-world software-engineering scenarios.Source: §8 · p. 11