Source-linked AI summary
Vibe Coding: Practice, Performance, Productivity, and Risk -A State-of-the-Art Review
Dominik L. Michels, Mutaz Abu Ghazaleh, Francois Lazzari, Nabil Kassem, Jonathan Klein
TL;DR
Vibe coding has generated a rapidly expanding but uneven evidence base spanning software engineering, HCI, economics, security, governance, and education. This state-of-the-art review synthesizes that corpus, finding that benchmark capability has risen sharply while task-level performance and productivity remain context-dependent. Its central conjecture is that gains are real on new code but shrink or reverse on mature codebases.
Problem
The review addresses limited, fragmented evidence about AI-assisted software development’s capabilities, benefits, dangers, uses, and future direction.
Method
The paper conducts a cross-disciplinary state-of-the-art review combining scientific literature with journalism, vendor and practitioner publications, social-media discussion, and source tracing.
Results
Capability has risen from 1.7% to 95% on SWE-Bench Verified, while measured productivity for experienced developers on mature codebases moved from +55% to –19%.
Takeaways & Limitations
The review concludes that vibe coding is durable, but its benefits are narrower, more population-specific, and shorter-horizon than headline figures claim.
Takeaways & Limitations
No study in the corpus stratifies AI-assisted productivity by codebase age, so the new-code versus mature-code conjecture remains untested.
Abstract
from arXiv · showhide
Vibe coding - AI-assisted software development in which the developer describes intent in natural language and validates results by running rather than reading the generated code - was named by Andrej Karpathy in February 2025 and produced its first body of empirical evidence within seventeen months. This state-of-the-art review assembles that evidence across a cross-disciplinary corpus spanning software engineering, human-computer interaction, labour economics, security research, governance, and education. We survey the model landscape, the tool ecosystem, and the performance record by task type, finding the early benchmarks saturated but task-level capability uneven: reliable code generation alongside weak fault detection and hard-to-audit documentation. The productivity record is at first contradictory: peer-reviewed field experiments report +26% more tasks per week, independent randomised trials measure a 19% slowdown, and team-level telemetry shows code-review time up +441%. We argue these readings are consistent once measurement method, scope, and time horizon are held constant, and identify six patterns behind the dispersion, among them effect-shrinkage under broader measurement, self-report diverging from independent measurement, output volume conflated with productivity, and bold claims walked back once tested over longer horizons. We further document security failures in deployed applications, code-quality degradation visible in large-scale code and developer telemetry, unsettled copyright exposure, and evidence of skill atrophy. The review closes with the open research questions and one falsifiable conjecture: that the gains are real on new code and shrink or reverse on mature codebases, which would account for most of the disagreement in the record.
1 Introduction
This review traces AI-assisted software development from earlier AI and code-generation lineages, then examines its capabilities, benefits, risks, adoption, and future direction. It synthesizes a cross-disciplinary, state-of-the-art corpus while treating productivity claims as measurements that require methodological comparison.
- AI-assisted software development evolved from symbolic AI, machine learning, deep learning, transformers, and generative models that progressively lowered the barrier from intent to code.
- The review asks what models and tools can do, what benefits and dangers are documented, how practitioners use them, and where the technology is headed.
- The paper evaluates model capability, tool landscapes, standardised benchmarks, task-level performance, productivity, production adoption, security, code quality, and skill atrophy.
- The review argues that apparently contradictory productivity findings reflect six stable patterns involving measurement, scope, time horizon, and the interpretation of headline claims.
- The corpus is a state-of-the-art rather than systematic review, combining scientific literature with journalism, vendor and practitioner sources, social-media discussion, and AI-assisted source tracing.
- Quantitative evidence receives exhaustive coverage, whereas illustrative registers establish that deployment failures, institutional reversals, and copyright exposure occur rather than measuring their frequency.
2 History
Modern code generation extends earlier rule-based and statistical completion through Codex, Copilot, vibe coding, conversational IDEs, and agentic repository-wide tools. The historical shift increasingly moves implementation work from human typing toward natural-language direction and autonomous task execution.
- Rule-based IDE completion and statistical models preceded transformer-based code generation, but static analysis and lookup architectures constrained what earlier systems could produce.
- Codex trained on filtered GitHub Python data and introduced HumanEval, achieving a 28.8% pass@1 rate for natural-language-to-code function synthesis.
- GitHub Copilot, built on Codex, reached general availability in June 2022 and became associated with the canonical 55% task-speedup figure.
- “Vibe coding” was coined by Andrej Karpathy on 2 February 2025 for coding that embraces generated output while disregarding the code itself.
- Vibe coding differs from reviewed AI assistance because the developer disengages from understanding generated code, whereas conversational IDEs keep review and testing in the loop.
- Agentic CLIs shifted the frontier toward autonomous task completion by reading repositories, modifying files, running tests, and iterating until completion or obstruction.
- Open-source maintainers increasingly distinguish AI as author from AI as reviewer, with generated bug-report volume contributing to curl’s shutdown of its HackerOne programme.
3 Models and toolkits
The ecosystem combines rapidly changing foundation models with multiple tool form factors, while broad multi-backend support reduces model-layer lock-in. Its market is stratified by capability, compliance, vendor dependence, and cost, from premium APIs to self-hosted open weights.
- The ecosystem has two coupled layers: foundation models that generate and reason about code, and toolkits exposing them through specialised interfaces.
- By mid-2026, at least ten vendors shipped frontier-grade coding models across US closed-weight, open-weight, European, and Chinese clusters.
- Tools span inline completion, conversational IDEs, browser-based full-stack builders, and agentic CLIs, serving progressively broader natural-language-to-application workflows.
- Virtually every serious tool supports multiple model backends, making vendor lock-in at the model layer structurally rare.
- Self-hosted open weights shift spending from metered API calls to GPU capital and offer a private, auditable, near-zero-marginal-cost deployment path.
- Subscription, token-metered, and enterprise pricing form three bands, with individual subscriptions stretching from free tiers to $200 monthly plans.
- GPT-5 high scored 88.0% at $29.08 per run, while DeepSeek-V3.2-Exp scored 74.2% at approximately $1.30, producing a fourteen-percentage-point accuracy gap.
4 Performance evaluation
Performance evidence shows strong and rapidly improving results on well-specified, mechanically verifiable coding tasks, but capability remains uneven where correctness is difficult to audit. Benchmark scores can overstate production reliability because contamination, weak tests, and evaluation artefacts separate measured performance from dependable software work.
- Benchmark trajectory: HumanEval and MBPP at or above 95% indicate that generation is no longer the binding constraint for complete, self-contained specifications.These benchmarks are now saturated, while incomplete specifications remain outside what they measure.
- Benchmark trajectory: 95.0% on SWE-Bench Verified marks rapid benchmark growth, but contamination-resistant and independently run evaluations reveal substantially lower performance.Claude Fable 5 falls from 95.0% on Verified to 80.0% on self-reported Pro, while Scale’s independent set reaches roughly 59–61%.
- Benchmark limitations: Benchmark scores are evidence of capability under favourable conditions, not evidence of production reliability.Evaluation concerns include inconsistent task solving, defects in 59.4% of tested problem descriptions or designs, benchmark exposure in training, reward hacking, and semantically incorrect patches passing weak suites.
- Task-type capability: AI systems show strong deployed bug-discovery results, but automated sweeps still carry false-positive risk and require careful interpretation of severity counts.AISLE and Big Sleep findings were externally corroborated by maintainers, whereas Codex Security’s reported counts are vendor self-reported.
- Task-type capability: The greenfield-versus-legacy divide is a broad project-scale boundary because accumulated codebase context supplies specifications that are difficult to check mechanically.The review links greenfield claims to near-entirely generated applications and contrasts them with maintenance concerns in established codebases.
- Task-type capability: Across task types, verifiability is decisive: mechanical feedback loops support stronger performance, whereas silent failures make tests, documentation, refactoring, and interfaces expensive to audit.Generated tests can compile and pass while weakly detecting faults; documentation has high surface quality, while frontend generation has outpaced peer-reviewed evaluation.
5 Productivity impact
The productivity record is contradictory at first reading because evidence ranges from vendor and self-reported gains to independent trials and telemetry showing slowdown and rising review costs. Broader, more rigorous, and longer-horizon measurement narrows the apparent gains and exposes distinctions between output volume, productivity, and downstream cost.
- Productivity evidence spans peer-reviewed trials, vendor claims, surveys, and telemetry, whose differing methods, populations, models, and measures complicate direct comparison.The review assembles the record chronologically and analyzes how the claims relate rather than treating the pre- and post-2025 periods as a break in the evidence.
- 55% faster task completion came from GitHub’s researchers testing 95 developers in a single two-hour JavaScript HTTP-server exercise.McKinsey’s crossover experiment independently reported task-level savings of the same order, but the review emphasizes the narrow setting behind the widely repeated figure.
- Independent RCTs and telemetry introduced negative findings: METR measured slowdown, while Faros AI found sharply increased pull-request review time despite perceived gains.These evidence types were largely absent from the earlier vendor-centered record.
- Effect sizes shrink as measurement broadens: Cui’s field experiment roughly halved the 55% anchor, METR reversed its sign, and DORA found individual gains alongside declining organisational throughput.The review attributes the dispersion to both measurement differences and a capability–context interaction involving developer experience and codebase maturity.
- Self-reported and vendor-reported gains diverge from independent measurement, with METR developers expecting and perceiving speed while measured outcomes were slower.The corpus places self-reported gains near its top and independently measured effects near its bottom, with roughly seventy percentage points between the extremes.
- Large adoption and output figures do not necessarily measure productivity: code volume, merged pull requests, and task counts can rise alongside review time and production issues.The review distinguishes value delivered per unit of developer time from displacement or output volume, and notes that the quantities diverge under load.
6 Dangers
Vibe coding’s documented risks span security and operational failures, degraded code quality, unsettled copyright, and erosion of developer skills and entry-level pathways. These risks can compound because maintaining lower-quality code requires skilled review while AI-assisted workflows may weaken the pipeline producing skilled reviewers.
- Four risk registers recur: deployed security failures, degraded code quality and maintainability, unsettled copyright exposure, and developer-skill erosion with entry-level labour-market effects.The review treats these as distinct but potentially compounding consequences of vibe coding.
- Security and operational failures: 1.5 million API authentication tokens and approximately 4.75 million database records were exposed after a zero-handwritten-code SaaS application lacked row-level database security.The incident followed public promotion of the application and occurred within forty-eight hours.
- Security and operational failures: 2,038 high-impact vulnerabilities and more than 400 exposed secrets were found across roughly 1,400 vibe-coded applications in an independent scan.A separate assessment identified approximately 380,000 publicly accessible assets, about 5,000 containing sensitive corporate data.
- Code quality and maintainability: 1.7× more issues per pull request appeared in AI co-authored code, including 1.75× more logic and correctness errors and 1.57× more security findings.The analysis found 10.83 issues per pull request versus 6.45, while authorship was inferred from co-author annotations.
- Code quality and maintainability: Refactoring fell from 25% to below 10% of changes, code duplication rose approximately fourfold, and churn nearly doubled from 2021 to 2024.The review links these longitudinal code-change patterns to cheaper new-code generation without a corresponding reduction in the cost of understanding existing code.
- Copyright and governance: AI-mediated reimplementation raises unresolved questions about clean-room independence, meaningful human authorship, and copyright enforcement for substantially AI-authored codebases.The review presents these issues as unsettled after a source-code leak enabled rapid AI-rewritten implementations.
- Skill atrophy and labour markets: Seventeen percentage points separated AI-assisted and hand-coding groups on an immediate comprehension quiz, with AI-assisted participants scoring lower despite finishing approximately two minutes faster.The trial involved fifty-two mostly-junior software engineers; the speed difference was not statistically significant.
- Skill atrophy and labour markets: Approximately 20% employment decline affected 22–25-year-old software developers from their late-2022 peak, while some coding-education providers reported major enrolment or viability declines.The employment estimate comes from ADP payroll data; not every education-sector closure was attributed specifically to AI.
7 Discussion and Conclusion
The review concludes that capability, measured productivity, and population-level skill costs move on different time scales and can point in opposite directions. Its central conjecture is that gains on new code shrink or reverse on mature codebases, but the evidence cannot yet test this directly.
- Discussion: SWE-Bench Verified reaches 95% while contamination-resistant and independently run evaluations remain lower, making benchmark saturation an unsettled interpretation.The review reads the residual gap on Verified as more consistent with dataset noise than a firm capability boundary.
- Discussion: A fourteen-percentage-point accuracy gap at a cost ratio above twenty to one already favours open-weight self-hosting for much work that does not require frontier capability.The review notes that the accuracy gap has been closing faster than the price gap.
- Discussion: Open-source maintainers increasingly distinguish accepted AI review and analysis from rejected autonomous authorship, with enterprise governance moving toward the same boundary.The review identifies this convergence among several projects within a twelve-month window.
- Open research questions: Long-term comprehension, three-to-five-year maintenance economics, and AI-mediated clean-room copyright remain unresolved because the available evidence does not span those horizons.The review states that no study can determine whether comprehension recovers, compounds, or settles lower under sustained assistance.
- Open research questions: No study stratifies AI-assisted productivity by codebase age, leaving the new-code versus mature-code conjecture untested.A proposed randomized comparison would use identical tasks across greenfield and five-to-ten-year-old codebases.
- Conclusion: Capability rose from 1.7% to 95% on SWE-Bench Verified in roughly thirty months, while mature-code productivity shifted from vendor gains to METR’s –19% result.The review presents capability, productivity, and skill costs as real curves operating on different time scales.
- Conclusion: The codebase-age conjecture proposes that gains are real on new code but shrink or reverse on mature code, potentially explaining much of the productivity dispersion without measurement error.The review concludes that benefits appear narrower and shorter-horizon than headline figures, while costs are slower and harder to attribute.