Source-linked AI summary
Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review
Joel Abenhaim
TL;DR
Large architectural refactors can exceed conventional review capacity when they span interdependent files and implicit invariants. This case study evaluates specification-first control for dismantling such an invariant, reporting convergence toward conformity through repeated specification and code audits in one codebase.
Problem
Large refactors spanning hundreds of interdependent files can make conventional review an unrealistic quality gate, especially when central structural guarantees must also be removed.
Method
The agent repeatedly checked a written specification against source code, generated against a frozen reference, and audited the implementation for architectural conformity.
Results
189 files were touched in three days, while repeated audits converged toward conformity without requiring single-pass model reliability.
Takeaways & Limitations
The case provides concrete evidence that control over a large AI-generated refactor can be relocated from post hoc code review to specification and conformity audits.
Takeaways & Limitations
This single task in one codebase and under one operator establishes neither outcome distributions nor reproducibility across other systems or problem classes.
Abstract
from arXiv · showhide
This paper reports a single, fully instrumented case study of a large-scale architectural refactoring by an AI coding agent under a specification-first protocol, with no human review of the generated code and no pre-existing oracle to validate the target behaviour. The task, dismantling a central invariant across a large interdependent codebase, was assessed by the author as effectively infeasible through incremental refactoring, the kind of change that conventionally calls for a rewrite instead. Under the protocol described here, the agent completed it successfully. The system is a 717,725-line production TypeScript application across 3,648 files. The task required dismantling a core lifetime invariant: the guarantee that a UI panel remains open for the duration of an AI request. The target behaviour was that a streaming generation survives the closing of its panel and can be reattached, on reopening, to the same live stream with no loss or duplication. The protocol: formal specification by the agent, 14 refinement cycles auditing that specification against the source code, atomic implementation, a compile/test feedback loop, then 17 verification cycles auditing the code against the frozen specification. Across 31 audit passes, 201 defects were corrected before any human executed the program. The convergence criterion was empirical: two consecutive verification passes returning zero findings. The change touched 189 files (31 new); with the extraction phase, the two commits total 288 files, 34,770 insertions, 16,422 deletions. Across the first and roughly thirty later sessions, the software behaved as specified, no bug observed. Elapsed: three days; cost: USD 2,430. The full specification and raw session logs, 1,500+ pages in French, are published as evidence, allowing inspection of the process and submission to a language model for consistency checking.
1. Introduction
The paper argues that for large, interdependent changes, control can shift from post-generation code review to specification-first auditing before and after code generation. It demonstrates this protocol in a deliberately difficult streaming-subsystem refactor, completed in three days.
- Motivation: Review becomes an unrealistic quality gate when a change spans hundreds of interdependent files and requires preserving many implicit invariants simultaneously.The passage identifies the reviewer’s inability to hold the full dependency graph in working memory as the limiting condition.
- Specification-first protocol: Specification-first control audits intent against the source before generation, freezes the resulting reference, and audits the generated code against it afterward.The specification is repeatedly re-checked until it produces no further findings.
- Case study: The case study removes a lifetime invariant in a live streaming subsystem, targeting a change ordinarily considered closer to a rewrite than a routine refactor.The task was selected as a stress test of the method rather than as representative routine maintenance.
- Case study: The reported execution completed the difficult refactor in three days.This timing is reported for the single documented stress-test execution.
2. Related work
Prior coding-agent evaluations and industrial deployments largely rely on pre-existing tests or other verification oracles, whereas this paper studies specification construction for one indivisible change with no such oracle. Its protocol combines fresh-session checking and extensive harness governance, positioning the contribution as narrower and complementary to existing loop-engineering and agentic-coding work.
- Oracle-based coding-agent evaluation: SWE-bench [2] evaluates coding agents by whether patches resolve real-world issues against held-out human-written tests, with SWE-bench Pro [3] extending this to longer-horizon industrial tasks.This paradigm assumes the correct behaviour is already encoded in an oracle.
- Oracle-free specification: This study instead targets behaviour absent before the change, so no pre-existing test suite can serve as an oracle; correctness must be specified and then checked against that specification.The target behaviour is a generation surviving closure of its panel.
- Loop engineering: Loop engineering automates work discovery, agent execution, independent checking, and external progress recording, but assumes units can be checked independently through tests or linters.The paper addresses a large indivisible change where those checks cannot distinguish correct from incorrect implementations.
- Checking and execution model: The protocol adopts fresh-context review differently from Anthropic’s guidance: a new session of the same agent compares the implementation with a specification written earlier.SWE-agent [4] also shows that the codebase interface affects agent reliability, while OpenAI’s harness report addresses review practice without this specification-refinement stage.
3. System under study
The study examines a proprietary VS Code extension implementing an AI coding agent, comprising 717,725 TypeScript lines across 3,648 files. Its architecture combines an extension-host orchestration backend, a React webview frontend, and a custom RPC bus.
- System under study: 717,725 lines of TypeScript span 3,648 files at the operation’s start.The codebase is comparable in size and functional category to commercially available agents.
- System architecture: The extension-host orchestration backend issues model requests and streams their output to a React frontend running in a webview.The frontend renders response text, reasoning traces, tool calls, code blocks, and other interleaved interface elements.
- System architecture: A custom RPC data bus connects the backend and frontend processes.The system is custom end to end, and its unpublished source excludes reproducing a solution from training data.
4. Task definition
The task required making standard AI-code sessions survive panel closure, remain visible in the activity sidebar, and reattach to the same live stream without token loss or duplication. This dismantled a structural lifetime invariant and required reconstructing partially consumed asynchronous state amid races and competing initialization, replay, and streaming operations.
- Functional requirement: Closing a running session panel must preserve generation, show the session in the activity sidebar, and restore indistinguishable live streaming when reopened.A detached stop control appears only while the panel is hidden and generation remains active.
- Architectural impact: The change violated a guarantee that panels remain open throughout requests, affecting request lifetime, memory ownership, cache restoration, asynchronous ordering, and stream reattachment.The task therefore crossed several interdependent architectural boundaries rather than changing panel visibility alone.
- Technical challenges: Reconstruction had to combine interleaved display content, response text, reasoning traces, tool calls, and code blocks from a partially consumed stream while handling closure races.On reopening, panel initialization, replay of generated content, and live token streaming competed concurrently.
- Feasibility assessment: The author judged dismantling this central invariant while reconstructing the system end to end as conventionally more suited to rewriting affected components than incremental refactoring.The published frozen specification, produced through fourteen refinement cycles, provides a concrete view of the task’s complexity; no formal complexity metric was computed.
- Task scope: The work began as a natural-language request describing the user-visible problem, intended outcome, and interface adaptation, without specifying types, modules, protocols, or asynchronous operations.The two-slice scope separated library extraction and generalization from applying that library to standard sessions and implementing open/close behavior.
5. Method
The study used a five-phase, specification-first workflow in which refinement audited the specification against source code and verification audited generated code against a frozen specification. Fourteen refinement cycles froze a specification after approximately 85 corrections, while seventeen verification cycles produced 116 code corrections and stopped after two zero-finding cycles.
- Protocol: The five-phase protocol comprised ideate, specify, refine, code, and verify, with separate sessions and outputs checked against defined references.The workflow prioritized process trust over line-by-line code review because the changes were too complex and interdependent for practical inspection.
- Refinement: Fourteen refinement cycles produced approximately 85 specification corrections and expanded the scope from 110 to 160 affected files before the specification was frozen.Cycle 14 returned no findings, so the cycle-13 specification became the reference for subsequent phases.
- Implementation: Implementation proceeded atomically in three consecutive segments after the agent rejected a partial migration as inconsistent with the specification.The implementation took 2 hours 21 minutes 44 seconds of agent time, while residual compilation, typing, and unit-test errors were corrected separately in under one hour.
- Verification: Seventeen verification cycles produced 116 code corrections by auditing architectural conformity against the frozen specification rather than merely compilation validity.Verification targeted residual architectural defects and subtle deviations, distinguishing it from the feedback loop for syntax, typing, and unit-test errors.
- Verification: The stopping rule required two consecutive verification cycles with zero findings, reached at cycles 16 and 17.The program was not run during end-to-end execution of phase 2; the first manual execution occurred after the seventeenth verification cycle.
6. Results
The refactor corrected 201 specification and code defects across 189 files in three days, then passed manual and regression testing without observed defects. The publicly released build preserved streaming across panel closure and reopening, at an inference cost of USD 2,430.
- Results: 201 defects, ambiguities, and architectural deviations were corrected across 31 audit passes before any human ran the code.Approximately 85 arose in the specification and 116 in the generated code.
- Results: 189 files were changed in three days, while the two commits together modified 288 files with 34,770 insertions and 16,422 deletions.The final codebase comprised approximately 736,000 lines; the 189-file count exceeded the 160 files tracked at specification freeze.
- Results: Manual execution and the pre-existing unit-test suite found no defect or regression, including panel closure and reopening during streaming, display restoration, and the detached stop control.The specified behavior was present on first manual execution, and unaffected as well as affected modules were exercised by the existing tests.
- Results: The publicly released version 2.3.0 allows independent verification that closing and reopening a panel neither interrupts nor duplicates the underlying generation.The build was released the same day and is publicly available for functional inspection.
- Results: A post-release interface revision required only one source file and its tests, suggesting that responsibility for that behavior was not scattered across the codebase.The paper presents this confinement as a possible indication of architectural cleanliness.
- Results: USD 2,430 was consumed in model inference during the three-day operation.The reported total covers the operation's daily inference spend.
7. Discussion
The discussion argues that repeated specification refinement and verification can converge toward conformity without requiring reliable individual passes. It treats the frozen, fully published specification as the foundation of that argument.
- 7. Discussion: Fourteen specification challenges followed by seventeen self-audits converged toward conformity through repetition, without requiring the model to be reliable on any single pass.Verification checked the generated code session by session against the frozen specification until no further deviations were found.
- 7. Discussion: Defects are cheaper to remove during specification refinement than after generation, because later fixes require interdependent code changes.The refinement cycles are presented as the mechanism for catching defects early.
- 7. Discussion: The frozen specification, produced through fourteen refinement cycles, is the artifact on which the argument rests and is published in full.The attached English specification is 55 pages long.
8. Scope of the claim
The case study concerns maintenance of an original problem, not a from-scratch project, transpilation, documented migration, or reimplementation of a likely training-data project. Its target behaviour had to be derived without a pre-existing test-suite oracle, and no directly comparable open-source case was known to the author.
- The case was not language-to-language transpilation, a documented framework or version migration, or reimplementation of an open-source project likely present in the model’s training data.
- The work was a maintenance operation on an original problem rather than a project written from scratch.
- The target behaviour had to be derived because it had not existed before, and no pre-existing test suite could serve as an oracle.
9. Limitations
The report is limited by its single, self-reported, closed-source case, lack of controls, model-specific scope, and incomplete instrumentation. Its clearest next step is independent replication on a public codebase.
- Scope and comparison: The study covers one task, codebase, and operator, so it establishes neither outcome distributions, success rates, nor reproducibility across systems or problem classes.The same task was also not attempted with other agents under comparable conditions.
- Independence: The result is self-reported: the author designed the tool, performed the operation, and reports it, although raw logs permit inspection.The reporting itself is therefore not independent.
- Evidence limits: “No bug observed” means no defect appeared during first execution, roughly thirty later sessions, or automated tests; it does not prove latent defects are absent.No test campaign or usage window of any length establishes such absence.
- Reproducibility and instrumentation: The closed-source codebase prevents third-party replay, while the published logs cover only the reported operation rather than the entire development history.The claim that the whole system used the same methodology is the author’s declaration because the 4,000-commit history could not be instrumented retroactively.
- Model dependence and next steps: The result was obtained with one frontier model in extended reasoning mode, so the protocol’s behaviour with weaker models remains uncharacterized.A clearer next step is independent execution of the same protocol by an operator on a public codebase, addressing the single-case, independence, and closed-source limitations.
10. Data availability
The paper publishes its frozen specification and complete raw session logs, totaling more than 1,500 pages in French and enabling direct or language-model-assisted consistency checking. The software is also publicly downloadable in versions that permit direct verification of the described behavior change.
- Published evidence: The complete raw session logs and frozen specification are published, totaling 1,500+ pages in French for direct reading or language-model consistency checking.The published material covers every cycle described in the paper.
- Public software: Version 2.3.0 and above preserves a query after its panel closes and reopens the live panel on the same ongoing stream, unlike versions below 2.3.0.The earlier behavior terminates the query when its panel is closed; the later behavior preserves it and reattaches the reopened panel to the same stream.
Competing interests
The author’s company, AI Sovereign Labs, designed and distributes the agent used in the study, while published logs enable direct inspection of the process.
- AI Sovereign Labs, the author’s company, designed the agent used in the study.
- The author’s company also distributes the agent used in the study.
- Published logs provide direct access for inspecting the study’s process.