Source-linked AI summary
Beyond Code Generation: Reliability, Verification, and Cost Economics in the Agentic Software Development Lifecycle
Happy Bhati
TL;DR
Agentic coding makes candidate changes cheaper and can increase coding activity, but production delivery remains constrained by verification, release, operational risk, and growing distributed costs. The paper synthesizes heterogeneous evidence into concepts for production-qualified output and risk- and budget-bounded autonomy. Its central conclusion is that engineering progress should be measured by production-qualified value rather than generated code alone.
Problem
The paper addresses how organizations should control agentic software work when coding acceleration outpaces verification, release capacity, and operational assurance.
Method
The paper presents a structured systems synthesis of research, benchmark audits, industrial reports, developer telemetry, and cost-management evidence.
Results
The synthesis finds that coding gains attenuate before release while agents generate substantial verification, correction, coordination, and cost burdens.
Takeaways & Limitations
Progress should be measured as production-qualified value delivered under reliability, cost, reviewer-attention, and operational-risk constraints.
Takeaways & Limitations
The review is not exhaustive, includes preprints and organization reports, and notes that company outcomes and market forecasts may not transfer or prove future costs.
Abstract
from arXiv · showhide
AI coding systems are moving from autocomplete and chat toward agents that can inspect repositories, edit multiple files, run tools, write tests, open pull requests, and work for long periods with limited supervision. This capability changes the bottleneck in software delivery. Recent field studies show meaningful gains in coding activity, but newer evidence also shows that those gains attenuate sharply between writing code and shipping reliable software. Review, integration, testing, security, deployment, and production operations remain constraining stages, while the economics are shifting from predictable per-seat licensing toward variable token, tool, sandbox, CI, and rework costs. This paper synthesizes peer-reviewed software-engineering research, university studies, benchmark audits, production reports from major technology companies, developer telemetry, and cost-management evidence released primarily from 2024 through September 2026. No new model experiment is claimed; numerical findings remain attributed to their original studies. The synthesis proposes four engineering concepts: the Agentic SDLC Throughput Paradox, Production-Qualified Change (PQC), the Verification Tax, and an Agentic SDLC Control Plane that allocates autonomy subject to cost, reliability, and human-attention budgets. An evidence-based horizon then maps today's supervised agents to future policy-bounded software factories. The central research question shifts from how much code an agent can generate to how much production-qualified value an engineering system can deliver per dollar, per reviewer-hour, and per unit of operational risk.
A Systems Synthesis of Industrial Evidence and a Research Agenda for Agentic Engineering
Agentic software engineering increases local coding output, but production delivery remains constrained by verification, coordination, operational risk, and distributed costs. This synthesis frames those constraints through production-qualified changes, verification workload, and policy-based autonomy.
- Industrial evidence: 26.08% pooled task-output growth shows AI assistance can improve coding productivity, while autonomous-agent gains attenuate from 180% at commits to 30% at releases.These findings describe different studies and should not be treated as a single experiment.
- Industrial evidence: Only 44% of agent-produced code survived into user commits, and users corrected, interrupted, or rejected outputs in 44% of turns.The same dataset also found more security vulnerabilities in agent-written code.
- Industrial evidence: Agentic delivery is constrained by verification, coordination, human judgment, and operational risk even when local coding productivity increases.These constraints become part of the runtime rather than remaining peripheral workflow concerns.
- Research agenda: The paper asks how organizations should allocate autonomy, models, verification, human intervention, and agent retries under cost and reliability constraints.Its source set mixes research studies, industrial reports, benchmark audits, and market evidence because production engineering extends beyond benchmark performance.
- Research agenda: The synthesis proposes Production-Qualified Change, the Verification Tax, and an Agentic SDLC Control Plane as frameworks for risk-adaptive output and autonomy.The control plane allocates models, context, parallelism, retries, tests, and human review according to risk, evidence, budgets, and capacity.
- Claim boundary: The synthesis reports no original benchmark, productivity, security, market, or deployment measurement; its numerical findings remain attributed to cited sources.It is a structured systems synthesis rather than a PRISMA-style exhaustive systematic review.
3 From Coding Capability to Production Capability
Benchmarks and field studies show that coding capability does not directly equal production capability: upstream generation accelerates more than downstream validation, coordination, and release. The paper therefore treats agent capability as an end-to-end human-agent system question rather than a patch-generation score.
- Benchmarks established the capability leap: SWE-bench shifted coding evaluation toward real GitHub issues, while audits found automated tests and benchmark scores imperfect proxies for software correctness.One audit found that 59.4% of 138 difficult tasks had material test-design or problem-description issues.
- From coding capability to production capability: Organizations should decompose agent capability by asking whether the human-agent system can move intent to production without creating more validation work than it removes.This reframes capability from patch generation toward production movement.
- From coding capability to production capability: A 180% autonomous-agent increase at the commit level attenuated to 50% at the project level and 30% at releases.The estimates use data from more than 100,000 GitHub developers and AI usage telemetry.
- Evidence across delivery systems: DORA associated higher AI adoption with higher delivery throughput but a negative relationship with delivery stability, without establishing that AI causes instability.The study is observational and organization-level.
- Evidence across delivery systems: A randomized trial of 16 experienced open-source developers measured a 19% slowdown on 246 tasks, showing that perceived acceleration can differ from measured end-to-end task time in mature repositories.The paper cautions against generalizing this result to every developer or later tools.
- The Agentic SDLC Throughput Paradox: The paper’s throughput paradox is that generation throughput rises faster than delivery throughput, so local coding gains do not translate linearly into trustworthy change reaching users.Delivery remains dependent on review, integration, testing, release, and downstream demand.
5 Code Review Becomes a Capacity Problem
As agents increase the volume of proposed changes, review becomes a capacity and assurance problem rather than a uniformly manual step. The paper argues for separating evidence types and adapting review and testing intensity to change risk.
- Review capacity: Google reports millions of reviewer comments annually and about 60 minutes of active author shepherding per change, while ML systems automate portions of review iteration.AutoCommenter was deployed to tens of thousands of developers.
- Independent assurance: If one model generates a change and another approves it, organizations still need independent evidence beyond model agreement.The paper rejects both unexamined model-to-model approval and requiring humans to read every line as complete solutions.
- Risk-adaptive review: High-risk changes may require static analysis, generated and human-authored tests, dependency and security checks, ownership review, and a canary, while low-risk documentation needs less.The proposed principle is risk-adaptive review rather than uniformly manual or automated review.
- Human filtering remains part of review: Only 44% of agent-produced code survived into user commits, and users pushed back in 44% of turns in SWE-chat sessions.The bimodal pattern included 41% of sessions with nearly all committed code authored by agents and 23% with all code authored by humans.
- Testing as assurance: Meta’s TestGen-LLM evaluation found that 25% of generated test cases increased coverage, and engineers accepted 73% of recommendations for production deployment.The system filtered generated tests through objective checks before recommendation.
- Testing as assurance: Test volume is a poor assurance metric because useful tests must add independent discriminatory power against plausible faults.The paper treats the verifier itself as a first-class artifact and notes that reward-hacking can exploit weak verification.
7 Reliability Is a Chain of Gates, Not a Model Score
Production reliability requires accumulating independent evidence through risk-appropriate gates, not merely producing a plausible change or passing a repository benchmark. The framework treats qualification as a system property shaped by harnesses, coordination, security validation, and explicit output criteria.
- A reliability-gate ladder distinguishes plausible change generation from accumulating enough independent evidence to operate a change safely.The ladder is a synthesis framework rather than an industry standard, and shows why repository benchmarks cover only part of production assurance.
- Coding agents are deployed as systems whose reliability depends on execution state, retrieval, memory, permissions, review interfaces, harnesses, and resource allocation.
- Security validation can pair agent autonomy with mandatory machine-verifiable gates before pull requests are finalized.GitHub describes checks including CodeQL, dependency advisories, and secret scanning for agent-generated changes.
- Agents were about 30% less successful when cooperating than when performing both tasks alone in CooperBench, with failures linked to communication and partner-modeling problems.The study included more than 600 collaborative coding tasks.
- Specialized agents therefore require explicit ownership, messaging, shared-state, conflict-resolution, and coordination contracts; agent count is not equivalent to engineering capacity.
- Production-Qualified Change credit is awarded only when a candidate satisfies the relevant qualification gates for its change class.The framework permits smaller gate sets for low-risk edits and larger ones for migrations, authentication changes, or payment paths.
9 The Economics of Agentic Software Delivery
Agentic software delivery shifts economics from visible model or token costs toward a broader cost surface spanning context, tools, CI, review, security, rework, and incidents. The paper proposes diagnosing verification cost against reliability rather than minimizing assurance indiscriminately.
- Agentic tasks incur costs from model inference, repository context, retrieval, tools, sandbox compute, CI, review attention, security scans, rework, and production failures.
- Total Cost to Deliver is presented as a taxonomy covering model, context, tools, sandbox, CI, review, security, rework, and incident costs.The model is conceptual and does not claim that every organization can perfectly allocate each term to an individual pull request.
- A cheaper model can increase total cost through retries, weaker patches, additional CI, or reviewer rework, while an expensive model can waste resources on low-risk tasks.
- Long-horizon SWE-Marathon rollouts averaged 27.2 million tokens, making retry policy, termination, context compression, verifier design, and model choice first-order economic decisions.
- The Verification Tax measures assurance cost relative to generation cost, but its variability requires diagnosis rather than gamification.A high ratio may reflect task risk, weak candidates, reliability-envelope violations, inefficient infrastructure, or redundant policy.
- The useful objective is to reduce verification cost for a fixed reliability target or improve reliability for a fixed assurance budget, not simply minimize verification.
- Different productivity studies can coexist because they measure different ratios between coding-generation gains and verification burden across system layers.
11 The Agentic Autonomy Budget
The paper treats autonomy as a budgeted privilege rather than a capability measured only by runtime or model size. Execution plans must fit money, reliability, and human-attention constraints simultaneously.
- Agent autonomy is better treated as a budgeted privilege than as a capability level defined by how long a system can run.
- Each task consumes money and compute, reliability and risk, and human-attention budgets.These include tokens and tools; expected failure impact and blast radius; and review, escalation, debugging, approval, and incident response.
- The feasible autonomy region is the intersection of money, reliability, and human-review capacity constraints.
- The control policy evaluates expected production-qualified value against monetary cost, expected failure loss, and human attention under mandatory security, policy, and SLO constraints.The weights are organizational policy rather than universal constants.
- Budgeted autonomy helps avoid both under-delegation of repetitive verified work and over-delegation on poorly verifiable or high-blast-radius tasks.
12 An Agentic SDLC Control Plane
The paper proposes an Agentic SDLC Control Plane as a centralized policy and telemetry layer around existing development infrastructure. It links task risk, execution choices, evidence gates, cost attribution, and outcomes to govern future autonomy.
- An Agentic SDLC Control Plane observes work, assigns execution policy, and records the evidence required to qualify results.
- The control plane can surround IDE agents, CI, Git hosting, security scanners, and deployment platforms rather than replacing them.
- Its responsibilities include classifying task risk, selecting models and harnesses, setting permissions and budgets, and binding risk classes to required evidence.
- The control plane should stop waste when marginal progress no longer justifies cost or repeated attempts cycle.
- It should attribute model, tool, CI, reviewer, and rework costs across task, pull-request, team, and repository levels.
- It should update policies using PQC rate, escaped defects, rollbacks, review corrections, token use, and human escalation.
- AI observability becomes an engineering control when cost and code-quality signals are connected to policy enforcement rather than reporting alone.
13 What Companies Are Actually Struggling With
Organizations are struggling to convert faster agentic generation into reliable, cost-aware delivery. The central challenge is coordinating verification, reproducible tooling, human review, and maturity-aware operating controls.
- Verification capacity: Verification capacity can become the bottleneck when companies accelerate generation without expanding inspection, integration, or shipping capacity.The resulting queue may move downstream rather than disappear.
- Reproducibility: Agent systems require versioned harnesses and reproducible runs because permissions, retrieval, context compression, test selection, sandbox state, and retry logic affect success and cost.
- Metrics: Production-qualified value should connect AI cost with qualification and reliability rather than treating pull-request output as equivalent to safely shipped change.
- Maturity and reliability: The emerging operating model remains human-accountable: agents propose and iterate while production policy is externally enforced.
- Maturity and reliability: Long-horizon research systems still show coordination penalties, low success, weak self-verification, reward hacking, and high token use.
16 Research Agenda
The research agenda treats agentic development as a constrained production technology requiring experiments on assurance, cost, coordination, reproducibility, and autonomy. It prioritizes system-level outcomes over generated activity.
- Empirical evaluation: Future studies should instrument real agentic pull requests across repositories and model versions to identify predictors of production qualification and post-release stability.
- Verification and control: Experiments should compare risk-adaptive gates while holding escaped-defect risk constant and measuring cost, latency, reviewer time, and qualification rate.
- Cost and autonomy: Long-horizon agents need stop policies that use verifier improvement, edit churn, repeated failures, context growth, and estimated remaining work.
- Cost and autonomy: Cost research should compare cost per PR, merged PR, Production-Qualified Change, escaped-defect-free service change, and total delivery cost.
- Verification and control: Evaluation should test mutation score, fault localization, regression detection, specification coverage, and verifier independence against post-release defects.
- Coordination: Agent teams require experiments on explicit commitments, shared plans, role ownership, transactional state, conflict detection, and coordinator architectures.
- Reproducibility: Model, harness, environment, retrieval, and policy effects must be separated through versioned evidence and replayable trajectories.
- Cost and autonomy: The practical economic decision is marginal value: additional inference is worthwhile only when it avoids expert work without raising risk.
18 Implications for Researchers and Universities
Research and university curricula are shifting toward socio-technical evaluation of agentic software systems. The paper argues that stronger engineering foundations, assurance practices, and cost governance are increasingly important as implementation becomes cheaper.
- Research direction: Software-engineering research is moving from isolated model evaluation toward broader system evaluation in realistic environments and production traces.
- Curriculum: A durable curriculum should cover architecture, change impact, testing theory, mutation, property-based testing, and test-oracle design.
- Curriculum: Students should learn secure coding, dependency risk, least-privilege tooling, CI/CD, observability, SLOs, incident response, and rollback.
- Curriculum: Training should include model and harness evaluation, benchmark design, cost attribution, token economics, resource budgets, and human-agent accountability.
- Implication: If implementation becomes cheaper, engineering judgment becomes more leveraged, supporting deeper instruction in foundational engineering concepts.
19 Limitations and Responsible Claims
The paper limits its claims as an independent synthesis rather than a new empirical evaluation. Its proposed constructs remain unvalidated, evidence comes from heterogeneous settings, and autonomy must vary with domain risk.
- Scope: This work is a structured systems synthesis, not a PRISMA-style exhaustive review, and includes preprints, organization reports, and company-reported outcomes.
- Evidence boundaries: Results from randomized experiments, observational GitHub data, and maintainer studies describe different populations and should not be treated as one experiment.
- Construct status: PQC, Verification Tax, Agentic Autonomy Budget, and the Control Plane are proposed hypotheses rather than validated standards.
- Responsible deployment: Autonomy policies must vary with reversibility, blast radius, regulation, and human consequence across domains.
- Responsible claims: The evidence does not establish that rising agent capability inevitably reduces employment; labor outcomes depend on demand, organizational design, education, and distribution.
- Conclusion boundary: Reliable software still requires downstream requirements, testing, review, security, deployment controls, and production telemetry beyond generated code.
- Conclusion boundary: The paper frames progress as production-qualified value delivered, measured through trusted change, end-to-end cost, expert attention, and introduced risk.
- Conclusion boundary: The likely future is a control system coordinating models, tools, tests, security, human expertise, and budgets rather than a single coding model replacing the SDLC.