Source-linked AI summary
Characterizing the Quality Profile of AI-Generated C++ in Production
Michael Tran, Fred Lewis, Kun Yang, Saksham Thakur, Aditya Kini, Aditya Patil, Milad Hashemi, Parthasarathy Ranganathan
TL;DR
Production-scale evidence on AI-generated C++ quality remains limited by observability challenges. This paper analyzes 3.52 million enterprise code changes and finds a distinct quality profile with compute overhead that targeted feedback can mitigate.
Problem
Production-scale evaluation of AI-generated code remains difficult because quality must be tracked across revisions, tooling, submission, and deployment outcomes.
Method
The study uses authoring-time provenance to compare AI-generated and human-written C++ across static issues, source-level efficiency, downstream outcomes, and taxonomy-informed feedback.
Results
AI-generated C++ shows concentrated Interface and Coupling Burden and Copy and Allocation Overhead, with targeted feedback reducing static findings by 11.1% and improving R_eff by 31%.
Takeaways & Limitations
Taxonomy-driven feedback can guide category-specific mitigation of AI-generated C++ quality and efficiency issues in production workflows.
Takeaways & Limitations
The main analyses focus on C++ in one large industrial monorepo, limiting direct transfer to other languages and organizations.
Abstract
from arXiv · showhide
The widespread integration of AI coding assistants offers undeniable boosts to engineering velocity. Yet, recent studies point to a growing trade-off, revealing persistent challenges with code quality and maintainability. Industry leaders, including frontier AI labs, echo these concerns. As large language models are increasingly relied upon to author production code, understanding their impact on shipped software quality has become a critical priority. However, assessing these effects in industrial workflows remains difficult due to observability barriers. We study the impact of AI-generated code on production quality within a large enterprise operating global products relied upon by billions of users daily. Driven by this scale and user trust, the organization values code quality and has built thorough observability for every line of code deployed into production, enabling us to overcome measurement barriers to assess these effects. This study presents a large-scale empirical analysis of AI-generated C++ code from April 2025 to April 2026, tracking 3.52 million code changes across this enterprise's brownfield codebase. The core purpose is to understand the quality, performance, and maintenance characteristics of AI-generated code compared to human-written code in a production environment at scale. We find that AI-generated C++ code has a distinct quality profile, showing higher rates of interface and coupling burdens, copy and allocation overheads, and a reliance on explicit loops over optimized standard APIs. These issues translate into tangible downstream costs, including increased review effort and a 5-8% increase in compute resource consumption. However, we demonstrate that providing models with targeted, taxonomy-informed feedback can mitigate these effects, leading to an 11.1% reduction in targeted static analysis warnings and improved computational efficiency.
1. Introduction
The study addresses the gap between AI-coding productivity measures and production-lifecycle effects by analyzing AI-generated C++ at enterprise scale. It compares AI-generated and human-written code across upstream properties and downstream review, reliability, and compute outcomes, while testing taxonomy-informed feedback.
- Motivation and methodology: Production-lifecycle analysis is needed because completion rates and task speed omit whether generated code survives review, creates maintenance burden, or contributes to deployment-scale compute cost.Prior work commonly stops at controlled tasks, repository snapshots, or completions before review, limiting visibility into these effects.
- Motivation and methodology: Production evaluation must track generated edits across reviews, tests, static analyzers, deployments, and successive revisions before final submission.This captures whether generated lines are accepted, revised, moved, or removed across their production lifecycle.
- Motivation and methodology: 3.52 million submitted changes form the study’s production-scale analysis, with a focused C++ slice of 10.46 million lines used for static-analysis research.The data come from a monorepo environment with centralized review, authoring-time provenance, and post-submit operational monitoring.
- Study scope and contributions: The analysis compares AI-generated and human-written C++ on change structure, taxonomy-grounded static issues, and source-level efficiency measures.It also estimates associations between these upstream properties and later review, reliability, and compute outcomes.
- Study scope and contributions: The study frames four research questions covering AI-generation distribution, upstream code properties, downstream outcomes, and the effects of taxonomy-informed feedback.These questions organize the empirical analysis from workflow adoption through potential mitigation.
2. Related Work
Related work finds uneven quality in LLM-generated code and limited realism in benchmark-based evaluation, while workflow studies emphasize productivity and oversight rather than production source-level issues. Attribution remains difficult because developers edit, interleave, or rewrite model output, and detectors generalize poorly.
- Code quality and evaluation: Early Copilot and ChatGPT studies report uneven correctness and code quality across tasks, models, and prompts.Studies of real completion behavior show that benchmark completions do not fully capture what developers accept in practice, while taxonomies identify syntactic, semantic, and repair-related failures.
- Productivity and workflow: Controlled and enterprise studies report faster task completion, while interaction studies document steering, inspection, and trust costs.Industrial studies of LLM-based review and human-in-the-loop agents also examine review time, failure trajectories, and developer oversight.
- Production-code gap: Existing process studies focus on task time, interaction behavior, review-tool use, repository activity, or benchmark-solving traces rather than submitted production code’s source-level issue profile.This limits characterization of the concrete issues present in production code accepted through development workflows.
- Attribution: Attributing submitted code is difficult because developers may edit, interleave, or partly rewrite model output before finalization.AI-source detectors generalize poorly across models and settings, while memorization studies show that origin can be hard to infer from final code alone.
3. Study Design
The study tracks submitted changes and functions across a centralized C++ production workflow, linking byte-level AI provenance to code properties, static findings, review, reliability, and compute outcomes. A taxonomy-informed intervention then tests targeted feedback on generated C++ implementations.
- Study design: The analysis follows submitted code across 5 observation levels, from organization-wide provenance and C++ change structure to static findings, review, reliability, and function-level compute.The setting is a large technology company with a multi-language monorepo, centralized review, and standard build and static-analysis pipelines.
- Cohorts and analysis units: Submitted changes are the unit for distribution, code-property, review, and reliability analyses, whereas functions are the unit for compute analyses.Compute comparisons use approximately 25k AI-heavy and 12k human-typed functions, with approximately 6k matched functions in each cohort after edit-volume stratification.
- Intervention: The intervention uses a 50-function C++ benchmark, 3 prompt stages, and 3 independent runs per stage to evaluate targeted static findings and benchmark-based computational efficiency.The study produces 450 generated implementations before validation exclusions; computational efficiency is summarized by 𝑅eff from CPU instruction count and memory usage relative to the original implementation.
- Provenance and cohorts: Byte-level authorship annotations are projected onto final submitted lines, functions, and static findings, while AI-generation pools inline completion, conversational generation, agentic editing, and transformation-based editing.For multi-line findings, AI share uses a length-weighted mean across the matched line span.
- Static taxonomy: The 3-level taxonomy classifies line-level static findings into quality attributes, issue categories, and issue types, while downstream review, reliability, and compute outcomes remain separate.Findings are propagated from anchored lines or line ranges to enclosing functions, files, and code changes.
4. Results
AI-generated C++ became a substantial but uneven source of submitted code and showed a distinct quality profile, with larger changes, concentrated interface and coupling burdens, and more explicit local work. These upstream patterns coincided with higher build-failure rates, lower revert rates, increased review friction, and approximately 5% higher compute growth in production.
- AI adoption: 62.80% was the overall C++ monthly AI-generated share at the window’s end, rising from 28.56% while adoption varied across organizational slices and product domains.The study window covers submitted changes from April 1, 2025 to April 1, 2026; developers substantially filtered generated text before submission.
- Code quality: AI-generated C++ changes were larger and newer, with excess static burden concentrated in Interface and Coupling Burden and Copy and Allocation Overhead.Those two categories accounted for 82.21% of the total positive absolute rate gap, while AI-generated code relied more on explicit local work and less on standard-library/API delegation.
- Reliability: 1.3× was the median AI-generated-to-human build-failure ratio, while the median revert-rate ratio was ∼0.9×.This indicates higher pre-deployment instability but lower immediate rollback likelihood after review and automated checks.
- Production outcomes: Approximately 5% relative compute growth separated AI-generated from human code by early 2026, reaching 1.31× versus 1.25× baseline, respectively.AI-heavy functions also shifted 3.46% toward direct on-CPU execution, with increases in constructors (+1.36%), vectors (+0.10%), and map lookups (+0.07%).
5. Discussion
The discussion interprets the observed quality profile as a production-scale baseline and identifies concrete mechanisms behind static and efficiency differences. It also shows that taxonomy-informed feedback can mitigate targeted issues and motivates broader evaluation standards for AI-generated code.
- 5. Discussion: Interface and Coupling Burden and Copy and Allocation Overhead account for much of the positive static gap, offering practical categories for reviewers and tool builders.The taxonomy connects these categories to specific checks, including misc -include-cleaner, misc-definitions-in-headers, and runtime-missing-move.
- 5. Discussion: Higher loop/API contrast, map-access warnings, container-insertion warnings, and missing-move findings indicate more local work and less delegation to optimized C++ implementations.IPC and MIPS show no notable degradation in the supplied discussion passage.
- 5. Discussion: The taxonomy supports intervention by selecting recurring efficiency categories for targeted feedback, steering generated code away from the same local patterns.The discussion describes a tooling path in which static analysis identifies category-level risks and feedback redirects generation.
- 5. Discussion: 11.1% reduction in targeted static analysis findings and 31% boost in 𝑅𝑒𝑓𝑓 show that the identified weaknesses are not hard limits of model capability.The discussion attributes the weaknesses largely to historical default system configurations and insufficient context-aware prompting.
- 5. Discussion: Production evaluation should extend beyond pass rate, completion acceptance, and benchmark runtime to include edits before submission, reviewer-visible categories, deployed source patterns, and feedback responsiveness.This evidence standard links model assessment to observable production behavior and recurring static-analysis categories.
6. Threats to Validity
The study’s validity is limited by measurement ambiguity, observational analyses, evolving model and prompting conditions, human-review confounders, and a setting constrained to one industrial C++ monorepo. Enterprise-internal data and noisy or sparse outcomes further limit reproducibility and statistical certainty.
- Construct validity: Authoring-time provenance improves measurement over post hoc attribution but remains ambiguous because AI and human features can overlap and revisions may not map cleanly to final snapshots.Tool-specific static checks also introduce measurement choices.
- Internal validity: Review and compute analyses are observational, so correlations with task difficulty, repository context, author experience, and review norms do not establish causal effects.Sequentially adjusted models reduce confounding risk without converting the study into a causal estimate.
- Model Mix and Prompt Evolution: Masked model identities prevent separating model-specific effects, while improving prompting proficiency means older commits may reflect weaker human steering.The one-year timeline therefore combines changing model and developer conditions.
- Human-in-the-loop confounders and review dynamics: Reviewer fatigue or over-trust may not reliably intercept localized inefficiencies, making upstream automated interventions important for mitigating post-deployment compute costs.The passage links this concern to taxonomy-informed prompt feedback evaluated in RQ4.
- External validity: The analyses center on C++ in one large industrial monorepo, limiting direct transfer to other languages and organizations with different tools, rollout timing, and interaction mixes.The observed organization-wide distributions and heterogeneity patterns may therefore be organization-specific.
- Reproducibility and reporting validity; Statistical conclusion validity: Enterprise-internal data constrain full release, while sparse issue categories, noisy outcomes, skew, and collinearity can weaken statistical conclusions despite reporting and robustness safeguards.Safeguards include support thresholds, exposure-normalized rates, confidence intervals, staged models, and alternative denominator and overlap checks.
Data Availability
The study’s underlying enterprise data cannot be released publicly, but reviewers receive a replication package containing methodological materials and selected aggregated statistics supporting the main claims.
- Data Availability: Enterprise-internal provenance, review, repository, and production-monitoring data cannot be released publicly.The data are restricted because they include internal identifiers and records.
- Data Availability: The reviewer replication package includes the taxonomy, codebook, analysis specifications, model formulas, and selected aggregated statistics.It excludes raw data, person-level records, repository identifiers, service names, tool names, and other internal identifiers.
7. Conclusion and future works
The study finds that AI-generated C++ in production has a measurable, uneven quality profile involving explicit local work and reduced standard-library/API use. Taxonomy-driven feedback can mitigate targeted issues, while future work must test whether these patterns generalize beyond C++.
- Conclusion: 3.52 million submitted changes reveal a measurable AI-generated C++ quality profile spanning authoring, submission, review, deployment, and feedback.The code is widespread but unevenly distributed and substantially filtered before submission.
- Future works: Mitigating AI-generated code’s compute overhead is also framed as a sustainability imperative as AI-assisted authoring scales across enterprise software ecosystems.The authors connect these feedback loops to reducing the broader carbon footprint of enterprise software.
- Conclusion: AI-generated C++ uses local loop constructs roughly twice as much and standard-library or API calls 30% to 40% less often than human-written code.The observed compute overhead is linked to explicit imperative logic replacing optimized, shared libraries.
- Conclusion: Taxonomy-driven feedback reduces targeted static findings while improving benchmark efficiency, making category-specific feedback a practical mitigation approach.The study identifies move/copy and container patterns as concrete targets for review and tooling.
- Future works: Future research should determine whether these imperative biases and library-avoidance patterns generalize from C++ to Rust, Go, Java, or other performance-critical languages.The study focuses on C++ because of its rigorous performance constraints and mature static analysis tooling.