Source-linked AI summary
daVinci-Dev: Agent-native Mid-training for Software Engineering
Ji Zeng, Dayuan Fu, Tiantian Mi, Yumin Zhuang, Yaxing Huang, Xuefeng Li, Lyumanshan Ye, Muhang Xie, Qishuo Hua, Zhen Huang, Mohan Jiang, Hanning Wang, Jifan Lin, Yang Xiao, Jie Sun, Yunze Wu, Pengfei Liu
TL;DR
Agentic mid-training is underexplored despite the need to teach models software-engineering behaviors that mirror iterative repository work. This paper studies agent-native data combining context-rich pull-request supervision with executable, test-verified trajectories, achieving 58.5% on SWE-Bench Verified and surpassing recent baselines such as Kimi-Dev.
Problem
Agentic mid-training remains underexplored, although mid-training could expose models to large-scale data mirroring file navigation, contextual edits, tool invocations, and test-driven iterations.
Method
The study combines contextually-native trajectories that preserve complete task-level action sequences with environmentally-native trajectories collected from executable, test-verified environments.
Results
58.5% on SWE-Bench Verified was achieved by daVinci-Dev-72B, surpassing recent baselines such as Kimi-Dev.
Takeaways & Limitations
Agent-native PR supervision and executable verified trajectories provide a token-efficient path to strong agentic coding performance.
Takeaways & Limitations
Developer identifiers were not explicitly removed from PR text in the general subset, raising privacy concerns and potentially enabling memorization of contributor names.
Abstract
from arXiv · showhide
Recently, the frontier of Large Language Model (LLM) capabilities has shifted from single-turn code generation to agentic software engineering-a paradigm where models autonomously navigate, edit, and test complex repositories. While post-training methods have become the de facto approach for code agents, **agentic mid-training**-mid-training (MT) on large-scale data that mirrors authentic agentic workflows-remains critically underexplored due to substantial resource requirements, despite offering a more scalable path to instilling foundational agentic behaviors than relying solely on expensive reinforcement learning. A central challenge in realizing effective agentic mid-training is the distribution mismatch between static training data and the dynamic, feedback-rich environment of real development. To address this, we present a systematic study of agentic mid-training, establishing both the data synthesis principles and training methodology for effective agent development at scale. Central to our approach is **agent-native data**-supervision comprising two complementary types of trajectories: **contextually-native trajectories** that preserve the complete information flow an agent experiences, offering broad coverage and diversity; and **environmentally-native trajectories** collected from executable repositories where observations stem from actual tool invocations and test executions, providing depth and interaction authenticity. We verify the model's agentic capabilities on `SWE-Bench Verified`. We demonstrate our superiority over the previous open software engineering mid-training recipe `Kimi-Dev` under two post-training settings with an aligned base model and agentic scaffold, while using less than half mid-training tokens (73.1B). Besides relative advantage, our best performing 32B and 72B models achieve **56.1%** and **58.5%** resolution rates, respectively, which are ...
1. Introduction
Agentic mid-training addresses the mismatch between static code data and dynamic software-engineering workflows. The paper proposes agent-native supervision through complementary contextually-native and environmentally-native trajectories.
- Agentic software engineering requires models to navigate repositories, edit code, execute tests, and iteratively respond to feedback.
- Agentic mid-training remains underexplored despite its potential to instill foundational behaviors before post-training.
- The core mismatch is that conventional data omits the complete action–observation flow governing real agentic deployment.
- Contextually-native trajectories preserve complete task-level information flow, broad repository coverage, and diverse engineering patterns.
- Environmentally-native trajectories preserve authentic tool interactions, test executions, runtime errors, and system feedback.
- The synthesis effort combines 68.6B-token contextually-native trajectories with 3.1B-token environmentally-native trajectories.
2. Background and Problem Setup
The paper formalizes agentic software engineering as sequential interaction between an agent and an executable repository environment. It motivates agent-native data as a remedy for training data that exposes final artifacts without the workflows and feedback needed at deployment.
- Agent-native data preserves complete development workflows and environmental feedback through contextually-native and environmentally-native trajectories.
- An agentic task is represented as (R, q, E), comprising repository state, problem description, and evaluation oracle.
- Actions include searching, reading, editing, and testing, while observations provide file contents, compiler errors, and test results.
- The typical workflow is localize → read → edit → test → revise, with steps potentially repeated or interleaved.
- Traditional training data creates distribution mismatch by showing completed artifacts rather than sequential action–observation pairs.
3. Agent-Native Data: Design and Synthesis
The dataset combines contextually-native PR trajectories with environmentally-native executable rollouts. Together, these data preserve broad development context and authentic execution feedback while producing a 71.7B-token corpus.
- Figure 2 contrasts static files and factorized subtasks with trajectories that bundle retrieval context, sequential edits, and execution feedback.
- Contextually-native trajectories reconstruct complete workflows from GitHub PRs by bundling issues, repository files, modifications, and validation signals.
- The contextually-native corpus combines general cross-language coverage with Python-focused alignment to SWE-Bench Verified.
- PR collection retrieves metadata, linked issues, base-commit files, commit sequences, and diffs through GitHub APIs.
- Filtering retains merged, non-bot PRs and constrains Python tasks to one-to-five changed Python files.
- Reconstruction enhances PR context and uses different structural formats for general and Python subsets.
- The workflow representation maps file paths, contents, edits, and generated text to localization, reading, editing, and reasoning stages.
- The Python subset contains 41.9B tokens from 6 million PRs across 7.4 × 10^5 repositories, while the general subset contains 26.7B tokens from 4 million PRs.
4 Experiments
The experiments evaluate agentic mid-training across datasets, post-training regimes, model scales, open baselines, and transfer tasks. Results show consistent SWE-Bench Verified gains, including strong performance against Kimi-Dev and generalization beyond software engineering.
- Experimental Setup: Evaluation uses SWE-Bench Verified with SWE-AGENT at temperature 0, a 128k context, 100 steps, and Pass@1 averaged across four runs.A small number of test cases were manually fixed because the provided ground-truth patches could not pass.
- Experimental Setup: The experiments combine mid-training on PR data and trajectories with supervised fine-tuning on agentic trajectories, using Dctx, Denv, and comparison datasets.Dctx includes Python-focused and general subsets totaling 68.6B tokens; Denv contains approximately 4.5B effective rollout tokens and its passing subset is upsampled threefold.
- Robustness Across SFT Regimes: 58.5% is the strongest 72B result with Dctx + Denv mid-training, while Dctx alone reaches 46.4% with weak SFT and 58.2% with strong SFT.The 72B results show gains across both post-training regimes, with Dctx alone matching Kimi-Dev under weak SFT and surpassing the RL-tuned Kimi-Dev checkpoint under strong SFT.
- Robustness Across Scales: 56.1% is the best 32B result with Dctx + Denv mid-training, 3.1% above the strong SFT baseline.Dctx mid-training improves the weak SFT baseline by 4.7% and the strong SFT baseline by 1.1% at 32B.
- Comparison with Open Recipes: 58.5% surpasses Kimi-Dev’s 48.6% at 72B, while 56.1% is state-of-the-art among open 32B training recipes using agentic scaffolds.The comparison uses the Qwen2.5 model family and agentic scaffolds; daVinci-Dev-32B starts from a non-coder Qwen2.5-32B-Base model.
- Generalization Beyond SWE Tasks: Agentic mid-training improves code-generation and scientific-reasoning benchmarks across both 32B and 72B scales.The reported transfer includes gains on HumanEval, EvalPlus, GPQA, and SciBench relative to the base models.
5. Analysis
The analysis finds that contextually-native data is more token-efficient and complementary to environmentally-native trajectories, while performance continues scaling with training data and steps. The strongest models reach 56.1% and 58.5% resolution rates at 32B and 72B, respectively, with substantial additional corpus headroom.
- Token efficiency: 68.6B contextually-native tokens consistently outperform Kimi-Dev’s approximately 150B-token recipe, with an additional 4.5B effective environmentally-native tokens improving performance further.The authors attribute this efficiency to contextually-native representations being closer to the software-engineering process.
- Synergy: 47.1% (72B) rises to 54.8% when environmentally-native trajectories are mixed with the Python contextually-native subset, a +7.7% gain.The ablation indicates that executable trajectories teach environmental interaction, while contextually-native data supplies knowledge and code-modification diversity.
- Synergy: For the 72B model, adding trajectories during mid-training raises the final SFT score from 56.5% to 57.8%.This supports using trajectory exposure during mid-training as an initialization benefit for later alignment.
- Scaling: 58.5% (72B) and 56.1% (32B) are obtained by scaling the contextually-native foundation from 41.9B to 68.6B tokens.The analysis identifies contextually-native scale and diversity as dominant performance factors, while trajectory mixing remains beneficial.
- Scaling: R2 ≈0.90 characterizes robust log-linear scaling of Pass@1 with training steps for both model sizes, reaching 54.9% for 72B and 49.9% for 32B.The monotonic trends suggest that performance had not saturated in this experiment.
- Scaling: The corpus has substantial scaling headroom: surveys indicate approximately 3 × 10^8 pull requests across approximately 10^9 public repositories, while executable processing adds depth and verifiability.Raw PR mining provides breadth, whereas Docker environments and unit tests provide authentic feedback and validated tasks.
6 Limitations
The paper identifies privacy and attribution risks in its general contextually-native corpus because developer identifiers were not explicitly removed. This may enable memorization of contributor names.
- Data privacy and attribution: Developer identifiers were not explicitly removed from PR text in the general subset Dctxgen, raising privacy concerns and potential contributor-name memorization.The limitation concerns the treatment of developer identifiers in the general contextually-native data.
7. Related Work
Related work positions mid-training as a bridge between pre-training and post-training, while agentic training has developed through SFT and RL over environment-derived trajectories. Data synthesis has progressed from recombination and filtering toward persona-driven and agent-oriented generation.
- Scope and evaluation: The paper’s evaluation remains limited to one base-model family and one benchmark, with broader model families and real-world agentic tasks left for future work.Some results also depend on a patched evaluation harness that fixes benchmark issues and introduces an additional source of variance.
- Mid-training: Mid-training introduces higher-quality, task-structured, or instruction-oriented data between web-scale pre-training and later SFT or RL.Prior work associates this stage with improved sample efficiency and a higher achievable performance ceiling.
- Agentic training: Early agentic systems commonly sampled trajectories in specific environments using closed models and distilled them into smaller task-specialized models through SFT.The related-work discussion places this trajectory-distillation pattern before the paper’s agentic mid-training focus.
- Data synthesis: Synthetic-data methods evolved from corpus recombination, rewriting, and reject sampling toward persona-driven synthesis that expands task coverage.The cited progression frames later agent-oriented synthesis as a newer direction in data construction.
8 Conclusion
The conclusion presents agent-native data as a data-centric strategy combining context-rich pull-request supervision with executable, verified rollouts. It reports strong SWE-Bench Verified performance and identifies structural PR representation, verified trajectories, and their synergy as central findings.
- Conclusion: 68.6B context-rich PR tokens combined with verified executable rollouts produce daVinci-Dev-72B with 58.5% on SWE-Bench Verified, surpassing Kimi-Dev.The conclusion frames this as a unified training recipe for enhancing agentic coding capabilities.
- Conclusion: Agent-native data consists of context-complete PR supervision plus executable, test-verified rollouts.The two sources respectively provide software-engineering priors and specialized agentic behavior.
- Conclusion: Keeping relevant file contents and commit edits together better mirrors the localize-read-edit loop than decomposing pull requests into isolated subtasks.The conclusion identifies structural PR representation as a key driver of effectiveness.
- Conclusion: Executable, test-verified passing trajectories yield higher gains than static or simulated traces, while PR data supplies broader software-engineering knowledge and modification diversity.The conclusion describes their combination as a token-efficient route to strong performance.
- Conclusion: Public repositories and increasingly automated environment verification provide headroom to expand the approach across languages and more complex software-maintenance tasks.The authors emphasize bridging static historical data and dynamic execution environments as the field moves toward autonomous engineering.
A.1 Dataset Components and Staging
The recipe stages contextually-native data across general and Python-focused subsets, while combining complementary datasets for agentic software-engineering training. It specifies separate mid-training and supervised fine-tuning configurations and uses structured PR formats with enriched interaction context.
- Dataset Staging: 68.6B tokens are staged as 26.7B general data followed by 41.9B Python-focused data in Dctx training.The general subset establishes broad software-engineering coverage, while the Python subset specializes agent-native patterns.
- Dataset Staging: 73.1B-token Dctx + Denv training uses the general Dctx subset first, followed by the other two datasets.The first stage contains 26.7B general tokens; the second stage contains the remaining datasets.
- Training Configuration: Mid-training uses a 1024-sample batch, an 8 × 10−5 peak learning rate, 5% warmup, cosine decay, and one epoch without loss masking.The schedule consumes all samples once, with no loss mask applied during mid-training.
- Training Configuration: Supervised fine-tuning uses a 128-sample batch, a 1 × 10−5 peak learning rate, 10% warmup, cosine decay, and standard masking of user and tool tokens.The schedule consumes all samples once per epoch.
- PR Data Formats: The General PR format concatenates chronological events, relevant file content, and grouped review-comment threads from highly starred repositories.Unlike The Stack v2, it includes relevant file content and grouped review threads without a 1–5 Python-file constraint.
- PR Data Formats: The Python PR format uses Markdown and search-and-replace actions, with generated PR summaries and refined commit messages simulating planning and textual reasoning.Git diffs are rewritten into the search-replace action space used by many agentic scaffolds.
D Benchmark decontamination
The benchmark decontamination procedure removes SWE-Bench Verified pull requests and measures n-gram overlap for HumanEval and EvalPlus before evaluation. A threshold-based contamination rule removes identified HumanEval instances.
- N-gram Matching: For HumanEval and EvalPlus, each benchmark reference concatenates the prompt and canonical solution before extracting unique 13-grams.The tokenized training corpus is scanned for overlapping n-grams in each training sample.
- Context Enrichment: The PR-summary prompt requests a concise 1–4-sentence account of the problem or feature, key changes, and important implementation details.Its template supplies repository, issue, changed-file, commit, and diff context.
- Context Enrichment: Commit-message refinement is prompted to improve clarity and educational value while remaining concise.The procedure optionally uses LLM-generated PR summaries and normalized commit messages during PR rendering.
- N-gram Matching: The contamination score for each benchmark instance is the maximum leakage ratio across all training samples.The leakage ratio compares unique 13-grams in the benchmark instance with those in a training sample.
- Thresholding: τ = 0.10 identifies high-overlap matches, leading to removal of 24 contaminated HumanEval instances from evaluation.The threshold was manually selected from case studies of high-overlap matches.