Source-linked AI summary
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, Bing Zhao
TL;DR
Existing benchmarks mainly test one-shot functional correctness, whereas mature software requires long-term maintenance through evolving requirements. SWE-CI introduces a repository-level continuous-integration benchmark with real development histories and iterative agent workflows. Experiments show that current LLMs still struggle to sustain code quality over extended evolution, especially in controlling regressions.
Problem
Existing snapshot-style benchmarks measure one-shot functional correctness but do not reveal maintainability as codebases evolve through successive changes.
Method
SWE-CI evaluates agents on 100 repository tasks using an iterative Architect–Programmer CI loop and EvoScore, which tracks functional correctness on future modifications.
Results
Current state-of-the-art models still struggle to sustain code quality over extended evolution, particularly in controlling regressions.
Takeaways & Limitations
SWE-CI makes the cumulative consequences of agents’ design decisions visible as codebases evolve, beyond what snapshot-based benchmarks capture.
Abstract
from arXiv · showhide
Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations -- a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose SWE-CI, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability. The key insight is simple: Maintainability can be revealed by tracking how functional correctness changes over time. The benchmark comprises 100 tasks, each deriving from a real-world code repository with a development history spanning an average of 233 days and 71 consecutive commits. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.
1 Introduction
Existing coding benchmarks largely measure one-shot functional correctness, leaving long-term maintainability unobserved. SWE-CI addresses this gap by evaluating agents as codebases evolve through successive requirements and modifications.
- Motivation: Existing benchmarks predominantly evaluate whether agents produce functionally correct code from a single, complete requirement.This snapshot-style setup spans single-file, repository-level, and interactive evaluations but remains one-shot.
- Motivation: 60% to 80% of software lifecycle costs are attributed to maintenance activities, underscoring the need to evaluate code-maintenance ability.
- Motivation: Snapshot evaluations cannot distinguish brittle fixes from clean, extensible implementations because their maintainability differences emerge only as requirements and interfaces change.
- SWE-CI: SWE-CI evaluates agents across long-term code evolution using 100 real-world repository tasks averaging 233 days and 71 consecutive commits between base and target commits.
- SWE-CI: SWE-CI uses an Architect–Programmer CI-loop protocol and EvoScore to measure functional correctness on future modifications, rewarding earlier decisions that facilitate later evolution.
- Findings: More than 10 billion tokens of experiments show that state-of-the-art models still struggle to sustain code quality over extended evolution.
2 Measuring the Agent’s Ability to Maintain Codebase
SWE-CI formalizes coding as an iterative evolution process and measures maintainability through normalized changes in functional correctness over time. Its EvoScore gives greater weight to later iterations, emphasizing long-term stability.
- 2.1 Task formalization: SWE-CI models coding with requireT, which derives requirements from codebase gaps, and codeT, which updates a codebase according to those requirements.
- 2.1 Task formalization: Unlike snapshot evaluation, SWE-CI dynamically computes ri = requireT(ci, c∗) and updates ci+1 = codeT(ci, ri) across iterations.
- 2.2 Normalized Change: Normalized change replaces binary pass/fail evaluation with a finer-grained measure of the current codebase while accounting for regressions during evolution.
- 2.2 Normalized Change: The normalized change score lies in [−1, 1], with 1 indicating that the agent closed the entire test gap and −1 indicating that every initially passing test was broken.
- 2.3 EvoScore: EvoScore aggregates normalized changes across N iterations using a future-weighted mean, with γ ≥1 assigning greater weight to later iterations.
- Data curation: The data-curation process constructs repository histories and benchmark samples for evolution-based evaluation.
3 SWE-CI
SWE-CI is built from substantial real-world repository histories and evaluates agents through an Architect–Programmer workflow that iteratively reproduces continuous integration. The resulting benchmark contains 100 reproducible samples from 68 repositories.
- Data curation: SWE-CI constructs base/oracle pairs from chronologically ordered commits in actively maintained Python repositories.
- Data curation: The pipeline builds Docker environments, verifies oracle tests, applies automated case filters, and selects the top 100 candidates by time span and intervening commits.
- Benchmark statistics: 100 samples from 68 repositories span averages of 233 days and 71 consecutive commits, with at least 500 modified source lines per pair.
- Agent protocol: The Architect identifies functional gaps and issues requirements, while the Programmer implements those requirements to reproduce a continuous-integration loop.
- Agent protocol: The Programmer follows the requirements document rather than directly inspecting the full test gap, allowing targeted development during rapid iteration.
4 Experiments
Experiments show that current models differ in long-term maintenance ability, still struggle to control regressions, and often produce superficially clean but less maintainable code than human solutions.
- 4.2 Maintainability: Newer models consistently achieve higher EvoScore within providers, with Claude Opus leading and GLM-5 also performing strongly.The evaluation covers 20 models from 8 providers.
- 4.2 Maintainability: Provider rankings vary with γ: MiniMax, GPT, and DeepSeek favor short-term gains, whereas Kimi, GLM, and Qwen favor long-term maintainability.When γ > 1, later iterations receive greater weight and higher rankings indicate better codebase maintenance.
- 4.3 Regression: Most models have zero-regression rates below 0.25, while only two Claude-opus models exceed 0.5.Zero-regression rate measures samples with no regression throughout maintenance.
- 4.3 Regression: 12 of 20 models show regression rates positively correlated with iteration count, while 11 of 20 show regression magnitudes negatively correlated with iteration count.Regressions become more frequent but affect a shrinking portion of the test suite over time.
- 4.4 Coding style: 15 of 20 LLMs outperform human oracle code on Pylint score, but all 20 underperform on MI score.The comparison uses successfully solved problems; Pylint reflects surface conventions, while MI captures deeper maintainability.
- 4.4 Coding style: LLMs consistently produce fewer changed lines than human solutions, yet achieve lower MI scores.The results associate concise patches with lower maintainability relative to more verbose human solutions.
5 Conclusion
SWE-CI evaluates maintainability through functional correctness on future modifications, exposing cumulative design consequences during codebase evolution. Experiments across 20 models from 8 providers show continued difficulty sustaining code quality, especially controlling regressions.
- 5 Conclusion: SWE-CI operationalizes maintainability as functional correctness on future modifications, revealing cumulative consequences of design decisions as codebases evolve.This makes visible consequences that snapshot-based benchmarks cannot capture.
- 5 Conclusion: Across 20 models from 8 providers, current LLMs struggle to sustain code quality over extended evolution, particularly in controlling regressions.
B.1 Solved rate
Most models resolve fewer than half of SWE-CI’s requirements, and solved-rate rankings largely align with EvoScore rankings.
- B.1 Solved rate: Most models achieve a solved rate below 50%, confirming the difficulty of long-term code maintenance in SWE-CI.Solved rate is the proportion of samples where all requirements are fulfilled by the end of maintenance.
- B.1 Solved rate: Solved-rate rankings largely align with EvoScore rankings, suggesting that EvoScore captures long-term task completion ability.Solved rate can be interpreted as a limiting case of EvoScore when γ →∞.
B.2 Average turns
Across evaluated models, weaker models consume more turns without resolving all requirements, while stronger models finish earlier. The figure summarizes average turns per task under a 20-turn maximum.
- Weaker models tend to exhaust the 20-turn budget without resolving remaining requirements.Stronger models terminate early after completing tasks efficiently.
- Additional attempts do not compensate for limited model capability in long-term code maintenance.The reported bottleneck is model capability rather than the number of attempts.
- Average turns per task are reported across all 20 evaluated models and sorted in ascending order.
B.3 Maximum relative changes
Performance gains accumulate unevenly across maintenance iterations: all providers improve most steeply during the first 1–4 turns, followed by considerably slower progress.
- The steepest performance gains occur within the first 1–4 turns across all providers.The figure plots cumulative maximum relative change up to each binned turn interval.
- Improvement slows considerably after the initial 1–4 turns, producing diminishing returns in later iterations.The cumulative curves remain monotonically non-decreasing by definition.
- Later iterations tend to contain harder requirements after easier requirements are resolved first.
C Prompts
The prompts define a coordinated Architect–Programmer workflow for incremental, test-driven maintenance. The architect analyzes failures and writes prioritized XML requirements, while the programmer implements them through constrained small-step code changes.
- Architect Agent: The Architect agent identifies functional gaps from test feedback and writes high-quality software development requirement documents.Its workflow consults non-passed test summaries, traces corresponding tests, and analyzes dependencies, assertions, inputs, outputs, exceptions, and boundaries.
- Architect Agent: The architect prioritizes changes that enable the most non-passed tests, favoring common low-level modules and downstream base capabilities when benefits are similar.
- Agent Coordination: The Architect–Programmer setup produces XML behavioral requirements while preserving test integrity and restricting file modifications to the designated output scope.The architect generates only the requirement document, and the programmer modifies code under /app/code/ without changing tests or the requirement file.
- Architect Agent: Architect requirements are limited to 1–5 code-change items, each specifying location, current problem, behavioral contract, and acceptance criteria.
- Programmer Agent: The Programmer agent reads the XML requirements, inspects relevant source code and tests, and implements requirements in small-step iterations under a TDD workflow.The programmer is prohibited from actively executing tests and may modify only the application code, excluding the tests folder.