Building AI-Intensive Software with AI: Early Results and a Cautionary Tale on Measuring Development Cost

1 Introduction

• Evidence on complete systems built with pervasive AI assistance remains scarce, and circulating cost figures are rarely audited for measurement error.Prior evaluations often focus on isolated tasks, despite calls for more evidence on real-world adoption, outcomes, and costs.Source: §1 · p. 1
• The study examines a doubly recursive setting: engineers use AI to build a system whose core behavior also depends on AI.The assistant helps developers navigate legacy codebases through retrieval-augmented generation (RAG) [10], addressing an onboarding bottleneck [2] [9] and broader developer experience concerns [3] [7].Source: §1 · p. 1
• A six-person student team built the assistant over one academic term with near-total AI assistance for design and implementation, using a three-layer cost model.The model tracks real AI spend, self-reported human effort, and an estimated human counterfactual as a first step toward a general costing methodology.Source: §1 · p. 1
• 19.4× was corrected to 9.9× after two independent, easy-to-miss errors revealed how fragile the initial cost ratio was.The authors present the correction, methodology, and early lessons as preliminary evidence that cost claims about AI-assisted development merit greater scrutiny.Source: §1 · p. 1

AI-assisted software development

• AI coding agents can plan, edit multiple files, run commands, and iterate, while questions remain about correctness, review burden, and adoption economics.Most related work evaluates model capability on benchmark tasks; this study complements it by analyzing the process and cost of building a complete system with pervasive AI assistance.Source: §2 · p. 1

Program comprehension and developer onboarding

• LLMs can explain code and answer repository questions with adequate RAG context [10], while codebase health affects developer experience [7].This work targets the technical axis of onboarding, addressing a longstanding codebase-understanding bottleneck that existing tools often approach through pointwise questions or manually authored walkthroughs.Source: §2 · p. 1

Cost Measurement in Practice

• Cost-focused studies often omit pricing and counterfactual assumptions, so this case study treats measurement error as a finding.Transparent, audited cost measurements remain uncommon; prior productivity studies emphasize time or throughput rather than dollar cost.Source: §2 · p. 2

The system

• The team built a repository-grounded onboarding assistant with RAG chat, guided tours, dependency graphs, and technical-debt analysis, delivering 25+ features and 201 automated tests.The system used a Python/FastAPI backend and React frontend, with about 21,000 production lines and 2,000 test lines.Source: §3 · p. 2

Three-layer cost model

• The three-layer cost model recorded provider-billed AI spend, team members’ logged hours, and a bottom-up professional human counterfactual priced at regional hourly rates.Counterfactual rates were US$3.86/6.67/10.88 for junior/mid/senior professionals; seniority was assigned by task complexity, and effort estimates were self-reported and subject to hindsight bias.Source: §3 · p. 2

Development process

• Development proceeded through problem framing, solution design, build-and-test, and measurement, using a frontier-model assistant for early research and a mid-tier coding agent for implementation, testing, and documentation.Source: §3 · p. 2

The system

• The tool pairs repository-grounded RAG chat with automated guided tours, a module dependency graph, and technical-debt analysis, addressing ad hoc peer-support demands that draw experienced developers away from their work [3] [7] [9].Source: §3 · p. 2

Cost across phases

• US$304 actual project cost versus US$3,005 counterfactual yields a corrected ∼9.9× ratio.The team logged 35.2 hours; the counterfactual estimates 329 professional hours without AI. Table 2 consolidates the three cost layers, including US$69 in tooling cost.Source: §4 · pp. 2–3

Adjustments to the Cost Model

• The initial 19.4× ratio was inflated by mispricing flat-rate AI use and using counterfactual labor rates about 85% above regional rates.Token-based estimates attributed US$22.74 to problem framing and suggested it consumed 68% of the AI budget; billing records instead put total tooling cost at ∼US$69. Regional rates lowered the counterfactual from US$5,678 to US$3,005.Source: §4 · p. 3

Distribution of AI-assisted work

• AI assistance was highest for code writing and test generation (about 95%) and debugging (around 80%), but lowest for architectural decisions (about 20%) and prompt design (about 40%).Documentation was near 85% AI-assisted and requirements analysis near 70%; the reported pattern leaves decisions shaping the system with the team.Source: §4 · p. 3

Benefits and Limitations of AI Assistance

• AI helped the team explore unfamiliar domains and turn established design decisions into consistent, tested code, but optimistic estimates and silent context loss created friction.Long sessions sometimes forced reexploration of previously read files, and features prompted without written designs required significantly more rework than those built from explicit specifications.Source: §4 · p. 3

5.1 Lessons Learned

• Verify whether each tool is flat-rate, metered, or hybrid before optimizing: token-efficiency work is wasted under flat-rate billing, while metered services require it.The team mistakenly inferred per-token billing for a premium model that was covered by a flat monthly subscription; the authors recommend using billing dashboards and recording metered charges as they occur.Source: §5.1 · p. 3
• Running state notes and repository decision records mitigated silent context loss, while explicit specifications reduced iterations compared with conversational feature requests.Without written designs, the agent filled gaps with plausible but often incorrect assumptions; AI assistance appeared to amplify existing clarity rather than substitute for it.Source: §5.1 · p. 3
• Versioned, templated core prompts aided reuse and review; upfront credential, constant-time webhook-comparison, and centralized audit-logging rules were adopted as precautions against security debt.The authors note that the pace of AI-assisted development can make security debt easy to accumulate.Source: §5.1 · p. 3
• 9.9× is an order-of-magnitude signal, not a controlled productivity measure; its retrospective 329-hour counterfactual assumes a senior developer, while a mid-level profile could imply 13–16×.The numerator is grounded in billing records and time logs, but the denominator estimates work never performed.Source: §5.1 · p. 3

5.2 Practical Implications

• AI reallocated rather than reduced headcount: implementation was nearly fully AI-assisted, while architecture and prompt design stayed human-driven, and verbose work accelerated more than conceptual complexity.Lightweight Architectural Description Records appeared to reduce context loss, though the authors do not claim causation.Source: §5.2 · pp. 3–4
• In AI-assisted SE education, specifications, decision records, and critical AI-output review appeared to reduce rework; recording costs, effort, and counterfactuals exposed measurement biases and suggested pedagogical value.The project suggests, but does not validate, curricular directions.Source: §5.2 · p. 4

5.3 Limitations

• Findings remain preliminary: 9.9× relies on retrospective counterfactual judgment and self-reported student effort in one ongoing case, so it is not a benchmark and may change.The team consisted of students, unlike the professionals used to price the counterfactual; billing records and time logs ground verifiable terms where possible.Source: §5.3 · p. 4

6 Conclusion

• 35.2 hours of human effort and US$69 of tooling produced a system estimated to otherwise require 329 professional hours.The authors frame these early results as evidence that pervasive AI assistance can shift the feasibility frontier for small development teams building nontrivial software.Source: §6 · p. 4
• The corrected methodology is presented as as valuable as the 9.9x ratio itself.Source: §6 · p. 4
• The authors suspect that inferring flat-rate costs from tokens and misapplying distant labor rates are common measurement errors.Source: §6 · p. 4

Artifact Availability

• The replication package includes development logs, counterfactual references, redacted billing records, prompts, and documented limitations.The limitations cover self-reported effort, estimated counterfactual hours, and apportioned subscription costs.Source: Artifact Availability · p. 4

• Generative AI assisted with drafting text, structuring tables, and reformulating language, and the authors reviewed, edited, and verified the generated text and tables.Source: Generative AI Use Disclosure · p. 4
• The software system described in the paper was developed using pervasive AI assistance, as documented throughout the study.Source: Generative AI Use Disclosure · p. 4