Source-linked AI summary
SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, Xingyao Wang
TL;DR
Existing benchmarks largely miss the temporal dependencies and technical debt of continuous software evolution. The paper introduces DeepCommit to reconstruct executable Milestone DAGs and SWE-Milestone to evaluate agents on dependent milestone streams; scores fall from over 80% on isolated tasks to 38.03% in continuous settings, exposing a major long-term maintenance challenge.
Problem
Existing benchmarks evaluate isolated coding tasks and therefore do not capture temporal dependencies, technical debt, or sustained system integrity during software evolution.
Method
DeepCommit reconstructs verifiable Milestone DAGs from commit histories, while SWE-Milestone evaluates agents on continuous, dependency-driven milestone sequences in executable testbeds.
Results
38.03% is the continuous-setting score versus over 80% on isolated milestones, and even the strongest agents resolve only approximately 13% of full evolutionary sequences.
Takeaways & Limitations
Sustained, maintainable repository evolution remains a central open challenge because agents implement features but fail to prevent regressions and accumulating technical debt.
Takeaways & Limitations
Filtering drops documentation-only commits and commits without resolvable structural ties, biasing the benchmark toward dependency-rich evolution and underrepresenting independent maintenance work.
Abstract
from arXiv · showhide
Real-world software must continuously evolve to meet ever-changing and open-ended requirements. AI agents, increasingly deployed as long-running systems, are now entrusted to drive this evolution. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable SWE-Milestone, a benchmark that evaluates agents on streams of milestone-level tasks, requiring them to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to 38.03% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.
1. Introduction
Real-world software evolution is continuous and dependency-laden, but existing evaluations largely isolate coding tasks. DeepCommit and SWE-Milestone address this gap by reconstructing executable milestone sequences and testing agents’ ability to sustain codebase integrity over time.
- Real-world software evolves through ongoing, incremental changes driven by dynamic requirements, including dependent and parallel work.
- Existing benchmark categories often evaluate agents on isolated snapshots, whereas repository evolution requires building on accumulated development history and exposes error propagation.
- DeepCommit reconstructs verifiable software evolution itineraries as executable Milestone DAGs using static analysis, LLM-driven construction, and runtime validation.
- SWE-Milestone evaluates agents on dependency-constrained streams of milestone-level tasks across persistent codebases rather than independent issues.
- 38.03% is the highest reported continuous score, compared with over 80% on isolated tasks, while Gemini 3 Pro achieves a 13.37% Resolve Rate.
- Continuous evaluation reveals failures in regression prevention, accumulated-error control, and long-term system maintenance despite retained feature-implementation capability.
2. Related Work
Prior work spans coding agents, environment synthesis, issue resolution, codebase generation, continuous tasks, and optimization. SWE-Milestone differs by evaluating open-ended functional development through dependency-driven evolution of the same repository.
- Coding-agent research has progressed from basic shell-tool use and predefined scaffolds toward autonomous systems integrated into development workflows.
- Automated environment-synthesis methods construct Dockerized repository environments, whereas DeepCommit preserves temporal development structure by reorganizing commit histories.
- Issue-resolution benchmarks modify real repositories but generally present isolated issues rather than streams of dependent milestones.
- Codebase-generation benchmarks extend task scope from features to releases or repositories, but they do not necessarily model continuous dependent development on one evolving codebase.
- Continuous-SWE efforts examine long-horizon behavior, including structural erosion, verbosity drift, or CI loops, while dedicated benchmarks remain nascent.
- Optimization benchmarks evaluate closed-ended objectives such as runtime against expert references, unlike SWE-Milestone’s open-ended functional software development.
3. DeepCommit: An Automated Pipeline for Reconstructing Software Evolution
DeepCommit converts noisy repository histories into coherent, dependency-structured Milestone DAGs and resolves their runtime environments into reproducible testbeds. Its validation checks structural validity, executability, and test reliability.
- Raw commit histories are noisy and fragmented, obscuring developmental relationships through varied granularity, parallel work, merges, and non-functional changes.
- DeepCommit integrates commit-history preprocessing, Milestone DAG construction, and executable-environment resolution in an end-to-end pipeline.
- The pipeline models release-tag ranges as main-branch commit sequences and collects associated pull-request, issue, and release metadata.
- An iterative LLM-agent process discovers seeds, consolidates commits, infers dependencies, and decomposes or merges milestones to produce a valid DAG.
- Runtime resolution produces reproducible Docker testbeds through multi-agent analysis, localization, and specialized repair modules, with expert guidance at key decisions.
- Quality assurance verifies complete eligible-commit coverage, dependency consistency, acyclicity, compilability, test collection, and stable test signals.
- 87.1% is the achieved test collection rate, alongside Pass-to-Fail rates of ≤0.026% and environment-induced errors of ≤0.10%.
4. SWE-Milestone: Benchmarking Continuous Software Evolution
SWE-Milestone evaluates agents in a persistent, dependency-driven development loop using verified milestone suites derived from diverse repositories. The benchmark combines dynamic task unlocking, snapshot-based scoring, and human-verified execution contexts.
- SWE-Milestone evaluates agents’ ability to evolve and maintain a codebase over an extended lifecycle rather than solve independent issues.
- Tasks arrive as a stream and follow DAG dependencies, allowing independent milestones to proceed in parallel.
- An external planner unlocks milestones after prerequisites are completed, while the agent works in a persistent codebase and submitted states are evaluated asynchronously.
- The dataset is constructed by selecting dependency-rich release ranges, extracting itineraries with DeepCommit, refining acceptance criteria against tests, and adding environment-specific instructions.
- 98 verified milestones span 7 open-source repositories, five programming languages, and 109 inter-milestone dependencies.
- Milestone complexity varies substantially, with an average SRS length of 1,348 words and gold patches ranging from under 100 to over 1,500 lines of code.
- Verification combines Fail-to-Pass tests for feature correctness with Pass-to-Pass tests for regression prevention.
5. Results and Analysis
SWE-Milestone exposes severe weaknesses in continuous software evolution: agents make partial progress, but regressions and upstream errors accumulate across long horizons. Performance is shaped by milestone position, dependency depth, exploration, verification, and cost-effort trade-offs.
- Overall performance: 38.03% was the highest continuous Score, while Resolve Rate remained around 13%, showing partial progress without frequent full milestone completion.Resolved milestones were predominantly early milestones with few upstream dependencies.
- Cost and effort: Gemini 3 Flash at $12 and 24.2% Score, and GPT 5.3-Codex at $25 and 28.9% Score, lie on the reported cost-effectiveness frontier.Higher cost does not uniformly produce higher performance, and additional OpenHands compute does not consistently improve scores.
- Independent versus continuous evaluation: 93.2% independently versus 21.1% continuously for Claude Sonnet 4.6 on scikit-learn demonstrates that degradation is driven by long-horizon evaluation rather than inherent milestone difficulty.High independent-task performance across repositories indicates that milestones are individually solvable.
- Evolution depth and error accumulation: Later milestones and deeper DAG layers consistently receive lower Scores and Resolve Rates, as accumulated upstream errors make downstream milestones harder to complete correctly.Resolve Rate drops sharply beyond the earliest milestones and shallowest dependency layers, while Score can still reflect partial progress.
- Evolution dynamics: Recall grows near-linearly, but Precision saturates rapidly, indicating that agents retain feature-implementation ability while regressions accumulate in the evolving codebase.Stronger models reach higher Precision plateaus, yet none avoid saturation entirely.
- Failure analysis: Logic Error accounts for approximately 57% of error-chain events, with inherited propagation at 12% and missing-test propagation at 17%.The distribution links dominant logic errors with propagation patterns that can prevent downstream tests from running.
6. Conclusion
DeepCommit converts noisy Git histories into verifiable Milestone DAGs, while SWE-Milestone evaluates agents on continuous, dependency-driven development. The results expose a substantial gap between isolated task solving and sustained repository evolution.
- DeepCommit distills noisy, fine-grained Git histories into coherent, verifiable Milestone DAGs.
- SWE-Milestone evaluates LLM agents while they continuously evolve codebases through dependency-driven milestone streams.
- 38.03% continuous performance contrasts with over 80% on isolated milestones, revealing a major degradation during sustained evolution.
- Agents implement new features but fail to prevent regressions, producing accumulating technical debt and a snowball effect.
- Even the strongest agents resolve only approximately 13% of milestones in full evolutionary sequences.
Limitations
The benchmark and pipeline have limitations involving test-suite requirements, filtering bias, contamination risk, human oversight, and scale. These constraints bound which repositories and evolution ranges the current system can support.
- The benchmark requires well-maintained, executable test suites with reliable F2P and P2P signals.
- Repositories lacking rich test coverage or relying on inaccessible external services cannot currently be included.
- Filtering source-code commits with non-trivial dependencies may overrepresent dependency-rich evolution and underrepresent independent maintenance.
- Commit histories from high-impact open-source repositories may appear in model pretraining, so memorization on individual milestones cannot be fully ruled out.
- Two DeepCommit stages still require human-expert oversight, including runtime-resolution scheduling and SRS verification.
- The current pipeline targets release ranges under roughly 30k LoC because larger ranges exceed resolution budgets and often fail testbed-construction gates.
Impact Statement
The paper presents continuous software evolution as a prerequisite for deploying autonomous agents in production environments. Its construction process combines history preprocessing, milestone inference, balancing, reconstruction, and reliability validation.
- Impact Statement: Continuous software evolution is presented as essential for long-running agents that iteratively customize software for diverse user needs.
- Milestone Construction: Seed milestones are identified using commit-DAG topology, then expanded through file overlap, temporal proximity, and semantic association.
- Dependency Inference: Dependency candidates are ranked using file overlap, symbol references, temporal ordering, and author overlap.
- Milestone Balancing: Milestone sizes are balanced using coefficient of variation, achieving CV = 0.96 on SWE-Milestone against a target below 1.0.
- Milestone Refinement: Oversized squashed commits may be re-segmented along feature boundaries, with dependency edges recomputed through line-level git blame.
- Evaluation Reliability: Reliability validation checks graph validity, runtime executability, and evaluation reliability, including negligible environment-induced errors of at most 0.10%.
A.4.3. Evaluation Reliability
Evaluation reliability is assessed through test collection and size stability checks. The appendix records the benchmark’s construction artifacts alongside these validation results.
- Test names extracted from commit diffs are matched against runtime-collected node IDs to assess collection stability.
- 87.1% of tests are collected overall, corresponding to 3,563 of 4,090 tests.
- The appendix documents software requirements, unified evaluation configuration, repository statistics, and complete Milestone DAG visualizations.
B.1. Software Requirements Specifications
SWE-Milestone uses behavior- and contract-level Software Requirements Specifications (SRSs) to express milestone goals while preserving implementation freedom. Human-led verification refines these specifications, and agents execute them in standardized continuous-evaluation environments.
- Design principles: SRS design deliberately avoids patch-like instructions, line-level edits, and nonessential identifiers.The specification aims to describe required behavior and contracts rather than reconstruct a particular patch.
- SRS structure: Each milestone SRS contains Feature Requirements with a Problem statement, Requirements block, and Acceptance clause.This structure bounds the implementation space while leaving agents free to choose compliant solutions.
- Verification and refinement: 71% of milestones underwent at least one human-led revision, most commonly to remove implementation leakage or test leakage.Other revisions tightened inaccurate requirements or added missing functional requirements.
- Verification and refinement: The first refinement round shifted per-milestone scores by approximately 5 percentage points, while the second shifted them by at most 2 percentage points.The reported convergence indicates that residual SRS noise had limited effect on relative agent rankings.
- Evaluation configuration: All agents run in identical sandboxed Docker environments under a shared continuous-task system prompt.The prompt specifies continuous context, edit boundaries, one-shot tagged submission, a streaming queue, and a monitor-implement-submit loop.
C.1. Agent Code Quality Analysis
Qualitative analysis identifies recurring structural weaknesses in agent-generated patches despite apparent functional success. These weaknesses involve misplaced responsibilities, overly broad suppression, degraded APIs, and accumulating technical debt.
- Responsibility boundary misplacement: Agents can place correct logic at the wrong abstraction level, violating framework contracts and causing regressions.In apache/dubbo, checking stream parameters in accept() rather than resolveValue() triggered 3 P2P regressions.
- Shotgun fixes: Shotgun fixes replace targeted validation with broad error suppression, allowing tests to pass while weakening required behavior.In nushell, the agent globally suppressed NotInALoop errors instead of preserving a whitelist-based guard and narrow exception.
- API signature degradation: Local implementation choices can degrade public API contracts by replacing structured types with opaque strings.The nushell agent changed OutputMismatch.expected from Type to String, losing pattern-match capability and omitting cleanup and deduplication.
- Cross-case pattern: Technical debt from weaker signatures, missed deduplication, and retained workarounds can compound across milestones.The cited API-degradation case explicitly links these omissions to debt that accumulates during long-horizon evolution.
- Cross-case pattern: These cases show superficial test-level correctness coexisting with structural and maintainability problems that automated evaluation may miss.The recurring issues include misplaced abstractions, coarse-grained suppression, and degraded API contracts.
C.2. Cumulative Error Analysis on element-web
Continuous evaluation exposes how upstream mistakes propagate through later development, while DeepCommit’s topology-driven milestone construction differs materially from human semantic curation. The comparison supports both the benchmark’s maintenance challenge and its construction boundaries.
- Cumulative error: Continuous evaluation produces a snowball effect in which early regressions and unresolved bugs destabilize downstream milestones.Independent-task performance therefore acts as an optimistic upper bound for sustained software evolution.
- DAG comparison: DeepCommit’s DAG contains 12 milestones and 14 dependencies, compared with 14 milestones and 19 dependencies in the Human DAG.Both DAGs organize the scikit-learn release interval into six groups, but their construction principles differ.
- Construction principles: The Human DAG uses semantic release-planning themes, whereas DeepCommit partitions commits using dependency topology and phase structure.Human dependencies distinguish functional and process-level relations, while DeepCommit derives dependencies from structural signals.
- Coverage boundary: DeepCommit retains a tighter interconnected commit subgraph but excludes process-level patterns such as code-first, documentation-later workflows.The filtering strategy preferentially omits low-interdependency commits and some isolated development patterns.
- Clustering divergence: DeepCommit and Human clustering achieve an Adjusted Rand Index of 0.538 over 201 shared commits, indicating moderate agreement despite identical group counts.Their differing abstractions reflect topology-driven aggregation versus intent-based semantic grouping.
C.3.3. Agreement Analysis and Summary
DeepCommit aligns with human milestone boundaries when technical boundaries are clear, but differs for cross-module and intent-defined work. Its topology-driven abstractions capture executable dependencies while incompletely recovering developer reasoning.
- Agreement patterns: Documentation shows the strongest agreement: 89% of Human M6.1 commits, or 71 of 80, map to DeepCommit M6.1.Array API and performance-optimization milestones also show comparatively strong alignment in the reported overlap analysis.
- Summary: DeepCommit recovers human-consistent boundaries for technically clear work but falls back to phase-based partitioning for cross-module, intent-defined work.The Human DAG better captures strategic coherence, functional versus non-functional edges, and cross-cutting workflow structure.