Source-linked AI summary
AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development
Shyam Agarwal, Hao He, Bogdan Vasilescu
TL;DR
The paper addresses limited real-world evidence on how autonomous coding agents affect software projects, especially relative to prior IDE-based AI assistance. Using matched staggered difference-in-differences on AIDev repositories, it finds that velocity gains are concentrated in agent-first projects, while quality risks persist across prior-exposure settings. The results support selective deployment with attention to maintainability and the study’s quasi-experimental limitations.
Problem
The real-world effects of autonomous repository-level coding agents, especially after prior IDE-based AI use, remain insufficiently understood.
Method
The study uses AIDev, defines adoption by the earliest agent-attributed pull request, and estimates monthly repository-level effects with matched staggered difference-in-differences.
Results
Velocity gains are large and sustained mainly when agents are the first observable AI tool, while warnings rise about 18% and cognitive complexity roughly 39% across exposure groups.
Takeaways & Limitations
Autonomous agents provide meaningful acceleration primarily in new-to-AI settings but consistently increase complexity and warning levels, highlighting a speed–maintainability trade-off.
Takeaways & Limitations
Isolated significant pre-treatment coefficients in warnings and complexity indicate a limitation of the quasi-experimental design, although they do not establish a sustained pre-trend or clear parallel-trends violation.
Abstract
from arXiv · showhide
Large language model (LLM) based coding agents increasingly act as autonomous contributors that generate and merge pull requests, yet their real-world effects on software projects are unclear-especially compared with widely adopted IDE-based AI assistants. We present a longitudinal causal study of agent adoption in open-source repositories using staggered difference-in-differences with matched controls. Using the AIDev dataset, we define adoption as the first agent-generated pull request and analyze monthly repository-level outcomes spanning development velocity (commits, lines added) and software quality (static-analysis warnings, cognitive complexity, duplication, and comment density). Results show large, front-loaded velocity gains only when agents are the first observable AI tool in a project; repositories with prior AI IDE usage experience minimal or short-lived throughput increases. In contrast, quality risks are persistent across settings, with static-analysis warnings and cognitive complexity rising by roughly 18% and 39%, indicating sustained agent-induced technical debt even when velocity advantages fade. These heterogeneous effects suggest diminishing returns to AI assistance and highlight the need for quality safeguards, provenance tracking, and selective deployment of autonomous agents. Our findings establish an empirical basis for understanding how agentic and IDE-based tools interact, and motivate research on balancing acceleration with maintainability in AI-integrated development workflows. The replication package for this study is publicly available at https://github.com/shyamagarwal13/agentic-coding-impact.
1 Introduction
AI coding has split into IDE-based assistants and autonomous repository-level agents, but evidence about agents’ real-world effects remains limited. This study examines repository-level velocity and quality outcomes, including how prior IDE-based AI exposure changes those effects.
- AI coding paradigms: IDE assistants provide synchronous inline suggestions, whereas coding agents asynchronously generate pull requests, implement features, and make substantial repository-level contributions.The distinction concerns both autonomy and contribution scale.
- Research gap: Research has extensively studied IDE assistants, but the effects of autonomous repository-level agents remain largely unexplored.Existing agent evaluations often rely on benchmarks without humans-in-the-loop.
- Why agents require separate study: Agentic tools raise distinct questions about code quality, maintainability, technical debt, and evaluation because they generate larger-scale contributions than individual coding assistants.Evidence-based understanding is relevant to organizations considering adoption.
- Study focus: The study uses the AIDev dataset to estimate repository-level effects of agent adoption and compares repositories with prior IDE-based AI use against repositories that adopted agents directly.The analysis focuses on autonomous contributors rather than inline assistants.
- Research questions: The paper asks how coding agents affect development velocity, software quality, and outcomes after prior IDE-based AI exposure.These questions define the study’s three research questions.
- Contributions: The authors present longitudinal repository-level evidence and causal evidence on transitions from IDE assistants to autonomous coding agents.The contribution extends prior results to newer data and a broader agent ecosystem.
2 Related Work
Prior research has concentrated on IDE-based assistants and commonly reports modest velocity improvements, while evidence on autonomous agents is newer and mixed. The treated repositories also differ by prior AI exposure, with IDE-first repositories generally more active and popular.
- Prior evidence: Real-world AI coding research predominantly studies IDE-based assistants, typically reporting modest velocity improvements across productivity and quality outcomes.Measures include task time, code volume, throughput, vulnerabilities, churn, warnings, and complexity.
- Agent evidence: Early agent studies are mixed: 83.8% of 567 Claude Code pull requests were merged, while controlled Cursor experiments found limited productivity benefits for experienced open-source developers.These findings motivate repository-level study of autonomous agents.
- Repository differences: IDE-first repositories are more starred, forked, and active, whereas agent-first repositories are older but smaller and less popular.Both groups nevertheless exhibit substantial agentic contributions.
3 Methods
The study applies staggered difference-in-differences with propensity-score matching to AIDev-linked repositories, defining adoption by the earliest observed agent-generated pull request. Outcomes are measured monthly across velocity and software quality, with prior AI exposure used to form agent-first and IDE-first groups.
- Estimation design: The causal design combines staggered difference-in-differences with propensity-score matching to estimate project-level effects of agent adoption.Later adopters and never-adopters provide counterfactual comparisons.
- Treatment definition: Adoption is defined as the earliest month containing an agent-attributed pull request in the AIDev dataset.The authors retrospectively parse January 2024 through November 2025 to reduce left-censoring.
- Agent identification: Agent attribution uses mutually exclusive labels and cascading signals including branch prefixes, PR authors, commit authors, bot status, and Claude-specific text patterns.The procedure prioritizes agent-specific signals before generic labels.
- Identification caveat: The analysis acknowledges that remaining attribution errors primarily add noise to treatment timing and may attenuate estimated effects toward zero.Recovering the earliest agentic pull request is therefore important for treatment assignment.
- Sample and exposure groups: Repositories are restricted to projects with at least 10 stars and at least 10 agentic pull requests, producing an unbalanced monthly panel around staggered adoption.Treated repositories are partitioned into agent-first and IDE-first groups according to prior IDE activity.
- Outcomes and covariates: Velocity is measured with monthly commits and lines added, while quality is measured with static-analysis warnings, duplicated-line density, and cognitive complexity using SonarQube.Repository–month outcomes are adjusted for time-varying project and activity covariates.
4 Results and Discussion
Agent adoption produces large and persistent velocity gains mainly in agent-first repositories, while IDE-first repositories show minimal or temporary throughput effects. Across both groups, quality risks persist through increases in warnings and cognitive complexity, creating a speed–maintainability trade-off.
- Development velocity: Agent-first repositories gain +36.3% commits and +76.6% lines added on average, while IDE-first repositories change by +3.1% and −6.3%, respectively.At adoption, agent-first repositories spike to about +111% commits and +216% lines added.
- Development velocity: IDE-first repositories show only a short-lived bump before estimates return near zero and turn negative by six months after adoption.At t=6, lines are approximately −61% and commits approximately −35%.
- Development velocity: Agent-first repositories sustain elevated activity, with lines added remaining roughly +49–+109% through t=6.This contrasts with the fading and eventually negative IDE-first dynamics.
- Software quality: Across agent-first and IDE-first repositories, static-analysis warnings rise about 18% and cognitive complexity roughly 39%.Complexity increases persist dynamically, reaching approximately +49% in agent-first repositories by t=5.
- Software quality: Complexity and warning increases persist even when IDE-first velocity gains are weak or negative, indicating agent-induced complexity debt.Duplication effects are small and inconsistent, suggesting structural complexity is the more consistent quality risk.
- Design limitation: The authors identify isolated significant pre-treatment coefficients in warnings and complexity as a limitation, although they do not view them as evidence of a sustained pre-trend or clear parallel-trends violation.These differences indicate that untreated potential outcomes may not be fully captured by additive fixed effects.
5 Conclusion
Autonomous coding agents provide meaningful velocity gains primarily in repositories new to AI assistance, while consistently increasing complexity and warning levels. Prior AI IDE exposure moderates the benefits but not the risks, highlighting a speed–maintainability trade-off and the need for oversight.
- Agents provide meaningful development-velocity gains mainly in repositories with no prior observable AI assistance.
- Quality risks persist across settings, with coding agents consistently raising software complexity and static-analysis warning levels.
- Prior AI IDE exposure moderates agents’ velocity benefits but does not eliminate their quality risks.
- The findings reinforce a speed–maintainability trade-off and support selective deployment with active oversight.