PaperlayerView exact revision on arXiv

Source-linked AI summary

A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption

Yegor Denisov-Blanch, Shyam Agarwal, Pavel Azaletskiy, Hao He, Rylan Schaeffer, Brando Miranda, Bogdan Vasilescu, Sanmi Koyejo

arXiv:2608.25241v1cs.SEcs.AI

TL;DR

Coding-agent studies report average effects despite wide differences in teams’ returns and AI configuration. This paper develops and validates RAMP from committed repository artifacts, then stratifies agent-adoption effects by maturity: development accelerates across strata, while quality costs are concentrated where no configuration is committed.

  • Problem

    Existing studies report average effects across AI-adopting repositories and do not examine whether committed AI configuration is associated with different outcomes.

  • Method

    The paper develops RAMP, a four-level cumulative maturity model grounded in version-controlled AI-configuration artifacts, and applies its validated classifier to stratify an existing agent-adoption panel.

  • Results

    Agents accelerate development across maturity strata, while agent-first repositories without committed AI configuration show roughly twice the complexity increase of Level 2-and-above repositories.

  • Takeaways & Limitations

    Committed AI configuration distinguishes repositories with similar agent adoption but different observed quality costs, making RAMP a reusable stratification instrument.

  • Takeaways & Limitations

    Because maturity is observational, unobserved AI usage intensity and model capability remain alternative explanations for the observed quality gradient.

Abstract

from arXiv · show

Coding agents increase development velocity but also technical debt. Prior work reports only average effects across adopters, hiding wide differences between teams. We introduce RAMP (Repository AI Maturity Profile), a four-level cumulative maturity model grounded in version-controlled artifacts that teams commit to configure AI tools. RAMP runs from behavioral rules and coding standards through named agent definitions to multi-agent orchestration, with observed practice concentrated in the first three levels. Across 441 repositories the levels behave as a cumulative scale, and independent human annotation reproduces RAMP's repository-level labels on 97% of a held-out sample. Adoption is cumulative, forward-only, and set-and-forget: 73.8% of artifacts are committed once and never modified. Re-estimating an existing agent-adoption panel within each stratum, agents accelerate development regardless of maturity (28-38% more commits), but quality diverges: among agent-first repositories, where the contrast is identified, those without committed AI configuration show roughly twice the increase in cognitive complexity (+53% versus +27%) and 1.7x the increase in static-analysis warnings. Because maturity is observational, correlated engineering discipline or model capability may explain part of the gap; we present these findings as hypothesis-generating and release RAMP as a reusable instrument.

1 Introduction

The paper introduces RAMP, a repository-level maturity model built from committed AI-configuration artifacts, and uses it to examine heterogeneous outcomes after coding-agent adoption. It finds that agents accelerate development across maturity strata, while quality degradation is concentrated in repositories without committed configuration.

  • Motivation: Existing studies report average effects across adopters and do not test whether committed AI configuration is associated with different outcomes.The paper motivates repository-level measurement because code-quality outcomes and observable configuration artifacts reside at the same unit.
  • RAMP instrument: RAMP defines four cumulative levels ranging from no committed configuration to behavioral rules, reusable agent capabilities, and multi-agent coordination.The progression is from context to capabilities to coordination.
  • RAMP instrument: The model is operationalized across 12 AI coding tools using pattern matching and embedding-based semantic classification validated against human annotation.The pipeline combines tool-pattern, path-semantic, and content-semantic signals.
  • Study findings: Agents accelerate development across maturity strata, while in the agent-first stratum complexity rises roughly 2× more at Level 1 than at Level 2 and above.The authors describe these maturity-linked differences as associations and hypothesis-generating because maturity is observational.
  • Study design: 441 repositories support the paper’s development and validation of RAMP as a repository-level measure of AI practice maturity.The broader study design develops and validates RAMP across corporate repositories before applying it to agent-adoption outcomes.

2 Background and Related Work

Prior work finds velocity gains from AI assistance but unsettled quality effects, while existing studies generally report averages rather than maturity-linked differences. The paper therefore frames RAMP-based stratification as a way to study adoption dynamics and heterogeneous post-adoption outcomes.

  • Prior evidence: IDE-based AI evaluations generally report short-term velocity gains, with magnitudes varying by developer experience and task complexity.Enterprise trials are described as confirming the direction of these effects.
  • Prior evidence: Longitudinal evidence links AI-tool adoption to persistent increases in static-analysis warnings and cognitive complexity, although quality findings remain unsettled.One cited study reports 30% higher warnings and 41% higher cognitive complexity for Cursor adoption.
  • Agent literature: Evidence on autonomous coding agents remains thinner, with prior work emphasizing pull-request acceptance or average adoption effects rather than downstream quality heterogeneity.The predecessor study reports large velocity gains when agents are the first AI tool and minimal gains when IDE tools came first.
  • Measurement gap: Version-controlled artifacts provide objective, contemporaneous, and scalable evidence for distinguishing ad hoc prompting from structured AI configuration.The paper treats committed rules and workflows as visible choices about how AI tools should operate in a codebase.
  • Research questions: Study 1 examines transitions between RAMP maturity levels, while Study 2 tests whether maturity is associated with differential quality and velocity outcomes after agent adoption.The second study applies RAMP to an existing panel dataset, and the research questions explicitly distinguish quality from velocity metrics.

3 RAMP: Repository AI Maturity Profile

RAMP classifies repositories by the highest cumulative level of committed AI-configuration artifacts, using hierarchical discovery and semantic classification. Across 441 repositories, the observed categories form a validated, layered scale, though tool-specific measurement bias and repository activity constrain comparisons.

  • Classification pipeline: The classifier discovers artifacts hierarchically, then combines tool-pattern matching, path semantics, and content semantics to classify files into nine categories.Content classification addresses semantically equivalent files with different names, while validated artifacts determine level assignment.
  • RAMP model: RAMP assigns Level 1 through Level 4 according to the highest level containing at least one classified artifact, with higher levels expected to include lower-level configuration.The levels progress from shared context and rules to reusable agents and multi-step coordination.
  • Adoption landscape: 66.7% of the 441 repositories are Level 1, 24.7% Level 2, 8.6% Level 3, and none Level 4, while validated-artifact averages rise from 0.5 to 2.7 to 15.8.Observed practice is concentrated in the first three levels, with materially richer configuration at higher maturity.
  • Scale structure: Category breadth correlates strongly with maturity (Spearman ρ=0.843), and category combinations form tight, layered groups rather than specialty-oriented configurations.The 210 scored repositories contain 25 distinct combinations, with silhouette 0.822 against a permutation null of 0.731 (p<0.001).
  • Validation: 97.1% of repository-level RAMP labels agree with human judgment on a held-out sample, with disagreements concentrated in rare procedural categories.The agreement covers 34 of 35 repositories and has quadratic-weighted κ=0.829; reported rates are stratum-weighted.
  • Robustness and caveats: Cross-tool maturity comparisons are constrained by measurement asymmetry, while repository activity retains an independent association with maturity after controlling for artifact count.Tool distributions differ substantially, and commit velocity retains partial ρ=0.269 (p≈8 × 10^-5).

4 Study 1: Adoption Dynamics (RQ1)

Study 1 finds that AI-configuration adoption is rare at first, usually begins with Level 2 artifacts, and then proceeds quickly or stops rather than reversing. Most committed artifacts are never modified, making repository maturity relatively stable.

  • Adoption begins after a long delay: the median time to the first validated artifact is at least 441 observed days, or 633 days among repositories with measurable pre-adoption history.
  • 95.9% of adopters enter at Level 2, while 83.1% introduce artifacts at exactly one maturity level.
  • Reversals are absent and abandonment is negligible: 0% reverse direction and only 0.5% abandon adoption.
  • The Level 2-to-Level 3 transition, when it occurs, takes a median of 154 days, far shorter than the 633-day latency to the first artifact.
  • 73.8% of AI configurations follow a set-and-forget lifecycle, being committed once and never modified.
  • Higher maturity involves broader initial investment, but maintenance effort does not increase after artifact count is controlled; Level 3 instead has greater author diversity.Level 3 repositories involve a mean of 3.4 unique contributors to AI artifacts, compared with 1.0–1.5 at Levels 1–2.

5 Study 2: Agent Adoption Effects by Maturity

Study 2 re-estimates agent-adoption effects within maturity strata, finding that agents increase development activity across groups but impose larger quality costs on repositories without committed AI configuration. The gradient is strongest for complexity and warnings, with some exceptions at higher maturity levels.

  • Interpretation: The comparison identifies heterogeneous treatment effects, not a causal effect of maturity itself, and may partly reflect more capable models in Level 2+ repositories.
  • Level 1 repositories show roughly twice the cognitive-complexity increase of Level 2+ repositories: +52.7% versus +26.7%.
  • Static-analysis warnings increase 1.7× more at Level 1 than Level 2+: +24.1% versus +14.0%.
  • Agents increase commit frequency in both strata: +37.6% at Level 1 and +27.5% at Level 2+.
  • Level 2+ repositories add more code after adoption, increasing lines of code by +68.7% versus +48.1% at Level 1.
  • Robustness: The four-level gradient confirms declining degradation: cognitive complexity rises +54.4% at Level 1 and +39.8% at Level 2, but is indistinguishable from zero at Levels 3–4.
  • Robustness: Level 4 is a partial exception: duplicated-line density rises 31.4%, the only significant Level 4 quality effect, while complexity remains statistically flat.
  • Robustness: The ordering Level 1 > Level 1–2 > Level 2+ holds for cognitive complexity and static-analysis warnings in the boundary-sensitivity test.

6 Discussion

The discussion interprets RAMP as a stable, repository-level way to connect committed AI practices with coding-agent outcomes. Agents increase development across maturity levels, while quality degradation is substantially lower beyond Level 1, though the observational design leaves mechanisms and causality unresolved.

  • Interpretation: 73.8% of AI artifacts are committed once and never revised, making repository maturity a largely stable stratification variable.Adoption is cumulative and forward-only, with most repositories plateauing at their entry level.
  • Interpretation: Agents accelerate development at every maturity level, but the agent-first quality cost at Level 1 is about twice that at Level 2+.The clearest divergence concerns cognitive complexity and static-analysis warnings.
  • Interpretation: +52.7% versus +26.7% separates cognitive-complexity increases for Level 1 and Level 2+ repositories within the agent-first stratum.The pooled average of 39% therefore describes neither group.
  • Interpretation: Committed rules may constrain agent output, while shared structured practices may support organizational learning; the design cannot distinguish these channels.The proposed mechanisms include direct guardrails and a shared, updatable reference for agent-generated code.
  • Practical Implications: Level 2 is the practical intervention point: a few pages of rules and standards are associated with roughly half the quality degradation of no committed configuration.The paper recommends committing at least a rules file and coding-standards document before or during early agent deployment.
  • Implications for SE Research: RAMP supports repository-scale sampling and stratification, while artifact specificity, upkeep, and within-repository variation remain open research questions.The classifier requires no survey or team cooperation, but it measures artifact presence rather than content quality or enforcement.
  • Practical Implications: Structured practices reduce but do not eliminate quality degradation, with Level 2+ repositories still experiencing a significant 27% cognitive-complexity increase.Whether artifact quality or specificity matters is left for future research.

7 Threats to Validity

The threats concern construct validity, identification, timing, outcomes, and scope. RAMP observes committed repository artifacts rather than all governance or randomly assigned maturity, and the reported associations remain hypothesis-generating.

  • Construct Validity: RAMP’s cumulative structure is partly imposed by category definitions, while rare categories and moderate human-annotation agreement limit validation strength.The development sample had no Level 4 evidence, and annotation agreement was α=0.572.
  • Treatment and Outcomes: The outcomes are SonarQube proxies rather than direct measures of defects or maintainability, and higher maturity may also reflect greater agent-treatment dose.The study does not condition on agent pull-request volume.
  • Internal Validity: Maturity is not randomly assigned, and unobserved AI usage intensity or model capability may explain part of the roughly 2× complexity and 1.7× warning gradients.The authors therefore frame Study 2 associations as hypothesis-generating rather than causal.
  • Timing and Reverse Causality: Only 11.0% of Level 2+ treated repositories committed a validated artifact before adoption, leaving reverse causality possible.Within the agent-first stratum, only 3.8% did so before adoption.
  • Identification and Stratification: The maturity contrast is identified mainly within the agent-first population because 91 of 113 IDE-first repositories are Level 2+ and their Level 1 subgroup is too small.The shared matched control pool also means within-stratum covariate balance is not guaranteed.
  • Scope: Repository-level labels miss governance in wikis, onboarding, review checklists, and pull-request gates, so Level 1 can include well-governed organizations.The corporate and open-source study populations also differ in setting.
  • Timing and Scope: The six-month post-adoption window cannot rule out later self-correction, and Level 1 durations are lower bounds.Little reversion appears within the observed horizon.
  • Identification and Stratification: The Level 1-versus-Level 2+ split follows consistent empirical patterns across alternative groupings but remains descriptive, not a qualitative threshold.A post-outcome boundary choice could otherwise overstate the gradient.

8 Conclusion

RAMP converts committed AI configuration into a validated maturity scale that exposes different quality costs among agent adopters. The decisive contrast is between no committed configuration and a few pages of rules and standards, although the associations are observational.

  • Conclusion: RAMP turns committed repository configuration into a validated four-level maturity scale for comparing AI practice.The scale enables stratification beyond a binary adopted-versus-not comparison.
  • Conclusion: Agents accelerate development at every maturity level, while quality costs fall almost entirely on repositories that commit no configuration.The largest practical gap is between Level 1 and Level 2+.
  • Conclusion: The reported associations may partly reflect correlated engineering discipline or model capability rather than committed configuration itself.RAMP nevertheless provides a reusable basis for future stratified, sampled, and filtered studies.

Data Availability

The replication package provides the paper’s analysis materials and uses publicly released outcome data from the predecessor study.

  • Data Availability: The replication package includes the full pipeline, tool patterns, and analysis notebooks.Study 2 outcome data derive from Agarwal et al.’s publicly released dataset.

A Human Validation Details

Validation against 175 human-labeled items shows that both the RAMP classifier and independent LLM labeler handle common artifact categories similarly, while rare procedural categories remain unreliable. Repository-cluster bootstrap intervals indicate that the sample supports comparing labelers, not precise item-level accuracy estimates.

  • Both labelers succeed on commands, rules, configuration, and agents, with the latter achieving perfect precision.They fail on the same rare procedural categories, skills and flows, where low support makes metric magnitudes anecdotal.
  • 63.3–95.6% and 67.5–90.3% are the repository-cluster bootstrap file-level accuracy intervals for the RAMP classifier and LLM labeler, respectively.The bootstrap uses 10,000 draws and accounts for labeled items clustering within 35 repositories.
  • Both labelers concentrate errors in the same confusion-matrix cells, especially rare procedural categories absorbed into rules or not-artifact.The full confusion matrices underlie the per-category metrics in Table A.1.

B Additional Classification-Validation Material

Additional validation tests support the proposed category-to-level mapping and show that the classification is insensitive to the tested threshold range. The figures also provide overall file-level agreement against the human reference.

  • All 21 tested classification thresholds yield the identical classification, so the threshold sweep would be a flat line.The sweep is described in the text rather than shown as a figure.
  • 81.7% and 81.1% are the file-level accuracies for the RAMP classifier and LLM labeler against the 175-item human reference.Figure A.1 displays rows as human reference labels and columns as machine labels, with ten classes collapsed into not-artifact for general-documentation and none.
  • 0.997 and 0.983 are the observed Coefficient of Reproducibility and Coefficient of Scalability in the mapping-permutation test.Across 10,000 random category-to-level assignments, these values fall at the 99.3rd and 99.5th percentiles, respectively.

C Maturity and Prior AI Exposure

The maturity contrast is concentrated in repositories with prior agent-first exposure rather than IDE-first exposure. In the agent-first population, Level 1 and Level 2+ repositories form substantial groups, while the IDE-first Level 1 group is small.

  • Table C.1 cross-tabulates RAMP maturity level against Agarwal et al.’s agent-first versus IDE-first prior AI-exposure partition.The table covers 509 classified Study 2 treated repositories.
  • 214 versus 182 agent-first repositories fall into Level 1 and Level 2+, respectively.This is the primary population in which the maturity contrast is identified.
  • 91 of 113 IDE-first repositories are Level 2+, leaving only 22 repositories in the IDE-first Level 1 stratum.The small IDE-first Level 1 group limits the corresponding maturity contrast.
Loading 2608.25241v1…