Source-linked AI summary
Building AI-Intensive Software with AI: Early Results and a Cautionary Tale on Measuring Development Cost
Victor Barros de Miranda Neves, Kiev Santos da Gama, Vinicius Cardoso Garcia
TL;DR
Empirical, audited evidence on the cost of building complete software systems with pervasive AI assistance remains limited. This case study applies a three-layer costing model and finds that the initially reported 19.4× ratio was corrected to ~9.9× after identifying two measurement errors. The findings underscore the need for greater scrutiny of AI-assisted development cost claims.
Problem
Transparent, audited cost measurements for complete AI-intensive software products remain uncommon, especially in resource-constrained academic settings.
Method
A six-person student team was studied using a three-layer model covering AI spend, self-reported human effort, and an estimated human counterfactual.
Results
~9.9× was the corrected cost ratio, replacing the initial 19.4× estimate after two independent measurement errors were identified.
Takeaways & Limitations
The correction indicates that AI-assisted development cost claims warrant greater scrutiny and that the costing methodology is valuable alongside the ratio itself.
Takeaways & Limitations
As a single ongoing case with student participants, self-reported effort, and an unmeasured counterfactual, the ratio is preliminary rather than a benchmark.
Abstract
from arXiv · showhide
Empirical reports on the true cost of AI-intensive software development remain scarce, and the few that exist are easy to get wrong in ways that never surface in the final number. We report early results from an ongoing case study: a six-person student team built a full conversational onboarding assistant -- RAG-based code chat, guided tours, dependency graphs, technical-debt analysis -- over one academic term using pervasive AI assistance. We instrumented development with a three-layer cost model (real AI spend, self-reported human effort, human counterfactual) and initially reported a 19.4x cost ratio. A follow-up pass revealed two independent errors -- inferring per-token cost under a flat-rate subscription, and pricing the counterfactual with the wrong regional labor rates -- that together had inflated the ratio by roughly 2x; the corrected figure is ~9.9x. We present this correction as an early, generalizable finding in its own right: both errors are easy to make, invisible in the final number, and plausibly common in similar reports. We outline next steps toward a more robust, replicable costing methodology for AI-intensive development.
1 Introduction
This ongoing case study examines a six-person student team building an AI-intensive conversational onboarding assistant with pervasive AI assistance and a three-layer cost model. Its initial 19.4× cost ratio was corrected to 9.9× after two independent, easy-to-miss errors, underscoring the fragility of cost claims.
- Motivation: Existing evidence largely evaluates isolated software-engineering tasks, leaving the economics, correctness, and review burden of pervasive AI-assisted system development open.AI coding assistants now support planning, multi-file editing, and iteration, but broader evidence on complete systems remains limited.
- Study setting: The study targets a doubly recursive setting in which engineers use AI to build a system whose core behavior depends heavily on LLMs and retrieval pipelines.The concrete system is a RAG-based conversational onboarding assistant for navigating legacy codebases.
- Method: A six-person student team built the assistant over one academic term, relying almost entirely on AI assistance for design and implementation.The project was instrumented with real AI spend, self-reported human effort, and an estimated human counterfactual.
- Contribution: 9.9× was the corrected cost ratio, down from the initially reported 19.4× after two independent, easy-to-miss errors were identified.The authors present the correction, methodology, and early lessons as preliminary evidence that AI-assisted development cost claims require greater scrutiny.
2 Background and Related Work
AI-assisted software engineering has expanded from task-level coding support to agentic systems, but questions about correctness, review burden, and adoption economics remain open. This work complements benchmark-focused capability studies by examining the process and cost of building a complete AI-intensive system, addressing the scarcity of transparent, audited real-world cost measurements.
- AI-assisted software development: AI-assisted software engineering now spans applications from requirements to maintenance, including agents that plan, edit multiple files, run commands, and iterate.Recent surveys describe a rapidly expanding landscape, while agent modes extend beyond single-line code completion.
- AI-assisted software development: Reported benefits emphasize implementation throughput, while correctness, review burden, and the economics of adoption remain open questions.These unresolved issues motivate analysis beyond model capability or task-level performance.
- Program comprehension and developer onboarding: Program comprehension and developer onboarding remain long-standing bottlenecks in understanding unfamiliar codebases.The paper situates its onboarding assistant within this established software-engineering challenge.
- Cost Measurement in Practice: Transparent, audited cost measurements of AI-assisted development remain uncommon, especially for AI-intensive products built in resource-constrained academic settings.Existing evidence often reports time or throughput gains rather than dollar cost and rarely audits its own measurement pipeline.
3 Case and Method
The team built a conversational onboarding assistant that combines repository-grounded code chat with guided tours, dependency visualization, and technical-debt analysis. Its pipeline clones repositories, parses and embeds source code and commit history, and retrieves repository fragments for grounded responses.
- The system: The assistant provides RAG-based chat, automated guided tours, a module dependency graph, and technical-debt analysis grounded in real repository fragments.Its purpose addresses onboarding gaps where codebase orientation often depends on mentoring and ad hoc peer support.
- The system: The pipeline clones a Git repository and chunks source code with tree-sitter, a multi-language parser supporting 15 languages.
- The system: Source code and commit history are embedded into a vector store to support repository-grounded retrieval rather than generic model knowledge.
4 Results and Findings
The project’s actual AI-assisted cost was US$304, compared with a US$3,005 no-AI counterfactual, yielding a ∼9.9× ratio. This gap reflects distinct tooling, human-effort, and counterfactual layers that are easy to conflate.
- Cost across phases: US$69 in tooling costs covered a flat GitHub Copilot Business subscription, repository analysis, and embedding generation.The subscription cost US$19/month over three months; repository analysis cost US$12.13, and embeddings cost US$0.13.
- Cost across phases: 35.2 hours of logged human effort valued at US$235 produced a total project cost of US$304.The human-effort valuation used the mid-level rate.
- Cost across phases: US$3,005 was the counterfactual cost for 329 professional hours, producing a ∼9.9× ratio against the actual project cost.The counterfactual used the seniority profile each activity demands.
5 Discussion
The discussion emphasizes verifying how AI tools are billed before optimizing costs, and treating AI as reallocating tasks rather than reducing headcount. It also presents preliminary educational and methodological implications, while cautioning that the corrected 9.9× ratio may not be final.
- For SE Practice: Flat-rate billing makes token-usage optimization irrelevant, whereas metered services require token discipline.The authors recommend determining for each tool whether billing is flat-rate or metered before optimizing costs.
- For SE Practice: AI reallocated tasks rather than headcount, with implementation heavily AI-assisted while architecture and prompt design remained human-driven.The discussion suggests planning which activities to delegate, while keeping judgment-heavy roles staffed.
- For SE Education: Classical engineering practices appeared to reduce rework, while Architectural Description Records appeared to reduce context loss during long agent sessions.The authors frame these observations as suggestions for AI-assisted software engineering education and do not claim causation for the architectural records.
- Limitations: 9.9× may not be final because the counterfactual relied on retrospective student judgment rather than a measured baseline.The case study is ongoing and single-site; the team consisted of students, the counterfactual used professional labor rates, and effort was self-reported.
6 Conclusion
Pervasive AI assistance enabled a small team to build nontrivial software with 35.2 hours of human effort and US$69 of tooling, versus an estimated 329 professional hours without it. The corrected methodology and 9.9x ratio are both presented as valuable because flat-rate token costing and distant labor rates can produce common measurement errors.
- Conclusion: 35.2 hours of human effort and US$69 of tooling produced a nontrivial software system.The conclusion describes this as evidence that pervasive AI assistance can shift the feasibility frontier for small development teams.
- Conclusion: 329 professional hours is the estimated counterfactual effort without pervasive AI assistance.The estimate refers to how long the system would otherwise have taken.
- Conclusion: 9.9x is valuable alongside the corrected methodology because two measurement errors may be common.The errors were inferring flat-rate costs from tokens and misapplying distant labor rates.
Artifact Availability
A replication package provides the materials and records needed to examine the study’s development costs, along with documented limitations in the estimates.
- Artifact Availability: The replication package includes the development log, counterfactual references, redacted billing records, prompts, and documented limitations.Limitations cover self-reported effort, estimated counterfactual hours, and apportioned subscription costs.
Generative AI Use Disclosure
The authors disclose generative AI assistance in preparing the paper and developing its software system. They reviewed, edited, and verified all generated text and tables.
- Generative AI assisted in drafting text, structuring tables, and reformulating language.
- The authors reviewed, edited, and verified all generated text and tables.
- The software system was developed using pervasive AI assistance documented throughout the study.