Source-linked AI summary
Towards an AI Software Factory for Data Systems
Anna Pavlenko, Bogdan Crivat, Brandon Haynes, Carlo Curino, Fotis Psallidas, Jaro Slawinski, Johannes Freischuetz, Laura Pereira Sanchez, Markus Weimer, Mathieu Demarne, Matthias Jasny, Mauktik Gandhi, Max Bovykin, Mirco Milletari, Purbasha Ghosh, Qiushi Bai, Raghu Ramakrishnan, Rahul Pandita, Sergiy Matusevich, Shivaram Venkataraman, Subru Krishnan, Md. Tareq Mahmood, Tiemo Bang, Venkatesh Emani, Xuan Zhao, Yiwen Zhu
TL;DR
Coding gains alone have limited impact on end-to-end SDLC performance, while cross-stage data and verification gaps constrain automation. The paper presents an AI Software Factory grounded by a World Model and agent-guided search; its Data Systems ECT deployments report gains over agentic coding, while general-purpose World Modeling remains open.
Problem
Coding productivity gains yield only modest end-to-end SDLC latency or throughput gains, while cross-stage decision data and formal verification remain limited.
Source: §1 · p. 1Method
The approach combines a World Model grounding SDLC work with an agent ideator pipeline that steers evolutionary software search.
Source: §2.1 · p. 2, §2.3 · p. 4Results
Takeaways & Limitations
These results convince the authors they are on track to accelerate the SDLC end-to-end with an AI Software Factory.
Source: §5 · p. 6General-purpose World Model construction and updating remains a hard, open problem despite success in specific Data Systems/ECT scenarios.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
AI-assisted coding tools deliver significant acceleration of coding, but only limited impact across the end-to-end software development lifecycle (SDLC)--an Amdahl's law effect! In this paper, we discuss our progress towards building an AI SW Factory that accelerates all the stages of SDLC-Targeting, Coding, Reviewing, and Ops. The AI SW Factory produces a metadata exhaust that enables self-improvement by fine-tuning model weights and updating our World Model (a rich data substrate). We focus on Data Systems and the important class of Evolutionary Coding Tasks (i.e., those with a measurable objective to hill-climb) and report on 1) scaled deployments at Microsoft (tens of repositories) leading to 3x engineering efficiency above agentic coding and up to 22x token efficiency, and 2) several open challenges.
1 INTRODUCTION
Coding has become a promising target for AI because software engineering provides rich decision traces and automated feedback. But coding gains alone have limited effect on end-to-end SDLC performance, and data and verification gaps make other stages harder to automate.
1 INTRODUCTION
- Decision tracing and automated feedback have helped make coding suitable for AI training and agentic loops.Decision tracing records how people resolve tasks; compilers, linters, and tests provide early feedback.Source: §1 · p. 1
- 10× higher coding productivity still yields only modest latency or throughput gains across the end-to-end SDLC.The paper frames this as an Amdahl’s Law effect.Source: §1 · p. 1
- Decision data is often missing or hard to correlate across SDLC stages.Source: §1 · p. 1
- LLMs lack formal grounding to verify generated code against ambiguous or incomplete intent, limiting automated feedback and leaving human supervision critical.Source: §1 · p. 1
- Data Systems add workload dependence, privacy constraints, and mission-critical workloads that make experimental disruption difficult or unacceptable.Source: §1 · p. 1
AI Software Factory
The paper presents an AI Software Factory aimed at automating more of the SDLC, focusing on Data Systems and Evolutionary Coding Tasks where objectives can be measured. It combines a World Model, targeting and coding agents, and a learning loop, while identifying review and generalization challenges.
AI Software Factory
- The factory focuses on Data Systems and Evolutionary Coding Tasks, where objectives such as query latency can be measured automatically.Data Systems offer stable public APIs and clear system semantics, while workload dependence, mission criticality, and privacy remain concerns.Source: AI Software Factory · pp. 1–2
- Targeting combines a World Model of system structure, business context, and telemetry with an agent that ranks optimization targets and builds production-derived benchmarks.The agent aims to maximize expected return on investment as lift in target metrics per token spend.Source: AI Software Factory · p. 2
- Darwin uses evolutionary strategies and specialized agents to guide software search toward measurable objectives, reducing reliance on human validation.Its ideator pipeline includes seeder, refiner, and matcher agents that shape which hypotheses are explored.Source: AI Software Factory · p. 2
- Darwin’s agent-first search delivers up to 22× better token efficiency than.The framework broadens ECT-style optimization to systematic, large-scale production-system optimization.Source: AI Software Factory · p. 2
- Reviewing remains a costly bottleneck; systematic flighting compares new code with production traffic, while privacy-preserving isolated flighting remains an open challenge.The faster SDLC also needs faster automated incident management, root-cause analysis, and automitigation.Source: AI Software Factory · p. 2
- Automating more SDLC work increases decision-trace quality and completeness, which can update the World Model or fine-tune model weights; this learning loop has fewer concrete results so far.Source: AI Software Factory · p. 2
2 AI SW FACTORY: BUILDING BLOCKS
The factory’s building blocks pair a World Model and Hotspan targeting with Darwin optimization, automated reviewing, and incident-response tooling.
2.1 World Model
- The World Model grounds decisions in raw artifacts, rebuildable materialized views, and task-scoped viewpoints, with retrieval savings of 8× on average and up to >100×.General-purpose World Model construction and updating remain open problems.Source: §2.1 · p. 2
2.2 Hotspan: Targeting Agent
- Hotspan turns ambiguous goals into steerable metrics, then combines telemetry, semantic code analysis, and review evidence to enumerate and corroborate optimization targets.Targets are ranked by evidence-weighted expected returns against cost, stability risk, and reviewer effort, then packaged with correctness-and-performance harnesses.Source: §2.2 · pp. 2–3
- For production-derived workloads, Hotspan uses scoped discrete-event simulators that mock interacting components; general-case benchmark generation, equivalence proofs, and tighter performance bounds remain open.The mock implementation is guided by production telemetry.Source: §2.2 · p. 3
2.3 Darwin: Evolutionary Agentic Swarms
- Darwin nests locally debugged code-test-debug-reflect agent sessions inside an evolutionary loop, while specialized agents generate, refine, and match hypotheses with candidate mutations.The loop synthesizes local self-correction with global objective search.Source: §2.3 · pp. 3–4
- 22× token efficiency versus OpenEvolve on circle packing; Darwin’s autonomous search also supports continuous optimization toward measurable objectives.A fuller ablation study is omitted for space.Source: §2.3 · p. 3
- Persistent hypotheses help prevent redundant concurrent exploration, while contextual-bandit weighting balances proven and untried strategies; noisy evaluations, slow inner loops, and formal search trade-offs remain challenges.Hypotheses can also tune mutation magnitude and model cost.Source: §2.3 · p. 4
2.4 Automated Reviewing
- Review agents use World-Model prompts learned from historical reviews and have anecdotally shortened cycles, while systematic evidence and flighting remain open challenges.Flighting remains largely manual because it affects security and privacy-sensitive core infrastructure.Source: §2.4 · p. 4
2.5 AutoSRE: Automated Ops
- AutoSRE distills incident records into a searchable investigation graph, letting agents retrieve evidence paths from similar past incidents to guide triage and resolution.The graph is a rebuildable materialized view of incident tickets, alerts, guides, mitigations, and investigations.Source: §2.5 · p. 4
3 AI SW FACTORY: SELF-EVOLUTION
The AI SW Factory turns SDLC execution traces into reusable learning signals, evolving both a shared World Model and model weights. The section describes World Model updates and Darwin-lineage fine-tuning, reports an agent-efficiency result, and identifies open challenges in trust calibration and learning from long traces.
3.1 Evolving the World Model
- The World Model combines user and AI-reflection knowledge with a clearing house that governs, redacts, and tiers interaction data before sharing it.Updates can be consent-promoted and reviewed, with rollback through replaying the source-of-truth log.Source: §3.1 · pp. 4–5
- Execution traces make the World Model self-improving: successful trajectories reinforce workflows, failures expose stale steps, and reviewed corrections enter shared context.Source: §3.1 · p. 5
- AutoSRE feeds incident-investigation traces back into its graph, reinforcing successful paths, decaying unused ones, and adding nodes and edges for novel resolutions.The process uses an ant-colony-optimization mechanism [19].Source: §3.1 · p. 5
- World Model updates are reviewed for safety, but that review is costly; calibrating trust as confidence grows remains open.Source: §3.1 · p. 5
3.2 Evolving the LLM: Fine-tuning Loop
- Darwin lineages provide preference-training examples without human labels: scored parent-child edits form comparisons, and run-level splits prevent lineage leakage.The paper uses offline DPO [15] with a Qwen3.6 27B model and a LoRA adapter [7].Source: §3.2 · p. 5
- On circle_packing, the tuned policy matched the base policy’s best solution with 72.5% fewer mutator agents and raised the valid sum-of-radii score from 2.618 to 2.631.The paper describes this as a +0.48% gain that closes 75% of the remaining distance to optimal, and attributes the matched-quality speedup to a trajectory prior.Source: §3.2 · p. 5
- Scaling trace-based learning to the diversity of real-software runs and to a fully on-policy loop remains ongoing work.The paper identifies effective use of the full signal in long SDLC traces as an open research challenge.Source: §3.2 · p. 5
4 REAL-WORLD EVIDENCE
The real-world evidence spans Darwin applications, OSS systems, and Microsoft production deployments. Autoperf’s application work outperformed manual development and current agentic coding tools by the reported engineering-efficiency measures, while Darwin showed gains across the targeted OSS systems.
Anecdotal evidence
- In a Fabric DW hash-join GPU kernel, Darwin combined five optimizations for a 2.6× speedup over the production kernel.The paper also reports 22× lower cost than MAP-elites on circle_packing and notes that its anecdotal ROI observations may reflect selecting problems suited to Darwin.Source: §4 · p. 5
OSS evidence
- 45 OSS systems all showed positive, sometimes modest, gains; the best-performing 20 averaged 13.8% performance gain.Darwin used a $2,500 LLM-inference budget per system; Figure 4 presents the improvements.Source: Figure 4 · p. 6, §4 · p. 6
- Estimated annual resource savings from deploying gains on 10% of top-20 OSS project production instances ranged from $230M to $616M.The authors note that impact is difficult to estimate and that these estimates could be off by 10×–100×.Source: §4 · p. 6
Production evidence
- 0.35 PRs per engineer-hour was estimated at 4.7× manual development and 3× today’s best agentic coding tools for Autoperf application work.Figure 5 reports PR gains, with 1.72× median and 10× at the 90th percentile.Source: Figure 5 · p. 6, §4 · p. 6
5 CONCLUSIONS
The paper presents an AI Software Factory vision for automating more of the SDLC and reports early implementation impacts in OSS and Microsoft production deployments. For Data Systems ECTs, it reports gains exceeding agentic coding alone while acknowledging open challenges.
5 CONCLUSIONS
- ECTs for Data Systems delivered gains far exceeding agentic coding alone and justified the token cost.The authors say these results increase their confidence in the AI Software Factory direction.Source: §5 · p. 6
- The paper reports an early AI Software Factory implementation and its impact in OSS and Microsoft production-level, large-scale deployments.Source: §5 · p. 6
- The authors present the AI Software Factory as a way to automate more of the SDLC, while noting that open challenges remain.Source: §5 · p. 6