A Case Study in Assuring AI-Written Software

1 Introduction

• Coding agents can outpace review and weaken code comprehension, motivating controls beyond exhaustive source inspection without making source review obsolete.A practitioner study reported weekly commits growing from a few hundred lines to 10–20K; another found an initial task-completion advantage largely disappeared on a later extension without the agent.Source: §1 · p. 1
• When generated code becomes impractical to inspect, earlier assurance approaches shifted toward proofs, conformance tests, translation validation, verified compilers, and differential testing.The analogy is limited: coding agents infer underspecified intent, act in stateful environments, and may modify their own tests.Source: §1 · pp. 1–2
• The study examines an operator’s evolving controls through two questions: where human control resided and what checks and reviewing agents contributed or failed to catch.The workflow emerged through trial and error as failures prompted new checks and review practices.Source: §1 · p. 2

2 Case and workflow

• The retrospective case study reconstructed six months of development from two operator interviews, project records, and cross-checks of interview claims where possible.Records included decision logs, audit reports, incident records, check inventories, and working notes.Source: §2 · p. 2
• The platform supported practice operations and background functions, but the operator could directly inspect staged behavior more easily than backup usability or audit completeness.The study does not assess healthcare compliance; production data remained outside agent reach through architecture and permissions.Source: §2 · p. 2
• At each session’s start, the coding agent reread a versioned rulebook, tasks, issues, and reusable skills; the operator approved or revised the proposed approach before implementation.Changes then passed project tests, fresh-context agent review, and private staging inspection; broader meta-audits could inspect wider system areas and records.Source: §2 · p. 3
• For difficult decisions, multiple models independently returned recommendations, reasons, and risks; production changes required staging and explicit operator approval.Confirmed failures prompted searches for similar issues and, where appropriate, regression tests; approved lessons carried forward to later sessions.Source: §2 · p. 3

3 RQ1

• Implementation and much technical review were delegated, but the operator retained authority over durable project lessons and production changes, while production data stayed outside agent reach.The arrangement bounded agent work and concentrated control at consequential decision points.Source: §3 · p. 3
• Adding reviewers shifted part of oversight to judging what each mechanism observed, what its output could support, and what action could follow.The operator did not need to reconstruct every implementation decision, but did need to understand each reviewer’s evidential scope and authority.Source: §3 · p. 3

4 RQ2

• The operator tested whether checks detected their target failures by deliberately recreating those failures and confirming that the checks failed.Confirmed problems also prompted fixes, searches for similar issues, and reusable lessons for later work.Source: §4 · p. 3
• 69 of 84 mapped items (82.1%) were covered by pre-existing checks, suggesting coverage of many concerns in established security guidance but not full platform security.Because the standards are public, the mapping does not demonstrate model generalization.Source: §4 · p. 4
• Monitors sometimes measured intermediate events rather than usable outcomes, audits stopped silently, missing checks vanished from pass-rate totals, and an incorrect repair disrupted operations.In each incident, a reassuring signal supported a stronger claim than the evidence justified.Source: §4 · p. 4
• The case identifies four assurance questions: whether checks observe intended outcomes, actually run, expose missing results, and have authority bounded against harmful action.The presence or apparent success of checks was not itself sufficient evidence of reliability.Source: §4 · p. 4

• Staging worked well for observable behavior, while the clearest assurance failures involved less visible backups, audit execution, monitoring completeness, and automated recovery.Source: §5 · p. 4
• Additional reviewers and monitors were useful, but their outputs could not be treated as ground truth; oversight also had to assess what supervisors observed and what their outputs justified.Source: §5 · p. 4
• This is one observational case involving one operator, platform, and evolving workflow; it does not establish fewer defects, security, compliance, or sufficiency for other systems.Source: §5 · p. 4

• Human control in this workflow was concentrated around intended outcomes, evidence, access boundaries, and consequential decisions.Source: §6 · p. 4
• Tests and reviewing agents introduced to support oversight could provide incomplete or misleading evidence.Source: §6 · p. 4
• Adding supervisory mechanisms shifted part of oversight to judging what they observed, what their outputs supported, and what authority followed.Source: §6 · p. 4

• Building, pre-merge review, and meta-audits used Anthropic Claude models invoked through the Claude Code command-line agent.Source: A Implementation Details · p. 6
• The multi-model council additionally queried Google Gemini and OpenAI GPT models through their APIs.Source: A Implementation Details · p. 6
• Reviewers and meta-auditors ran as separate instances without development-conversation access, but shared the builder’s model family, leaving shared blind spots possible.Source: A Implementation Details · p. 6