PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Refactoring: An Empirical Study of AI Coding Agents

Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan

arXiv:2511.04824v1cs.SE

TL;DR

Refactoring is important for sustainable software development, but it requires specialized skills and can introduce bugs or destabilize systems when performed improperly. This study analyzes agent-generated refactorings in real-world open-source Java projects and finds that they are common, favor localized consistency edits, and yield small but significant structural-quality improvements.

  • Problem

    Despite refactoring’s importance, empirical understanding of how AI agents use refactoring and affect code quality remains limited, while improper refactoring can introduce bugs or destabilize systems.

  • Method

    The study examines agentic refactorings across prevalence, operation types, purposes, and impacts using real-world open-source Java projects from the AIDev dataset.

  • Results

    Refactoring appears in 26.1% of agentic Java commits, is dominated by low-level consistency edits, and produces small but significant structural improvements without reducing the overall count of known design and implementation smells.

  • Takeaways & Limitations

    Agentic refactoring currently contributes localized structural and maintainability improvements while remaining less oriented toward high-level architectural restructuring than human refactoring.

  • Takeaways & Limitations

    Agentic refactoring is heavily dominated by low-level consistency-oriented edits, with fewer high-level design changes than human refactoring.

Abstract

from arXiv · show

Agentic coding tools, such as OpenAI Codex, Claude Code, and Cursor, are transforming the software engineering landscape. These AI-powered systems function as autonomous teammates capable of planning and executing complex development tasks. Agents have become active participants in refactoring, a cornerstone of sustainable software development aimed at improving internal code quality without altering observable behavior. Despite their increasing adoption, there is a critical lack of empirical understanding regarding how agentic refactoring is utilized in practice, how it compares to human-driven refactoring, and what impact it has on code quality. To address this empirical gap, we present a large-scale study of AI agent-generated refactorings in real-world open-source Java projects, analyzing 15,451 refactoring instances across 12,256 pull requests and 14,988 commits derived from the AIDev dataset. Our empirical analysis shows that refactoring is a common and intentional activity in this development paradigm, with agents explicitly targeting refactoring in 26.1% of commits. Analysis of refactoring types reveals that agentic efforts are dominated by low-level, consistency-oriented edits, such as Change Variable Type (11.8%), Rename Parameter (10.4%), and Rename Variable (8.5%), reflecting a preference for localized improvements over the high-level design changes common in human refactoring. Additionally, the motivations behind agentic refactoring focus overwhelmingly on internal quality concerns, with maintainability (52.5%) and readability (28.1%). Furthermore, quantitative evaluation of code quality metrics shows that agentic refactoring yields small but statistically significant improvements in structural metrics, particularly for medium-level changes, reducing class size and complexity (e.g., Class LOC median $Δ$ = -15.25).

1 Introduction

The paper studies how AI agents participate in refactoring, addressing limited empirical evidence about its prevalence, forms, purposes, and code-quality effects. It analyzes agent-generated Java refactorings across these four dimensions and reports frequent, mostly localized activity with small structural-quality improvements.

  • Motivation: The paper frames refactoring as improving internal software quality without changing observable behavior, while noting that improper refactoring can introduce bugs or destabilize systems.These risks motivate empirical evaluation of agentic refactoring.
  • Study scope: The study analyzes agentic refactoring across prevalence, types, purposes, and impacts using 15,451 instances from 12,256 pull requests and 14,998 Java commits.The data are derived from the AIDev dataset and target real-world open-source projects.
  • Prevalence: 26.1% of agentic Java commits explicitly target refactoring, containing 7,127 refactoring instances.This establishes refactoring as a common activity in the analyzed agentic commits.
  • Refactoring types: Agentic refactoring is dominated by low-level, consistency-oriented edits, favoring localized improvements over high-level design changes common in human refactoring.The reported pattern includes renaming and type changes.
  • Purposes: Maintainability and readability are the dominant stated purposes of agentic refactoring, accounting for 52.5% and 28.1%, respectively.These motivations concern internal code quality.
  • Code-quality impact: Agentic refactoring produces small but statistically significant structural-metric improvements, while not reducing the overall count of known design and implementation smells.The reported improvements concern code size and complexity.

2 Motivating Example

The motivating examples contrast agentic method extraction with systematic variable renaming. Together, they illustrate the range of agentic refactoring and motivate measuring whether these operations improve software quality.

  • Method extraction: Agents can refactor a long method into smaller helper methods, reducing method length and clarifying control flow.The example introduces helpers such as printUsageAndExit() and execute().
  • Variable renaming: Systematic variable renaming improves naming clarity and stylistic uniformity but has little direct effect on structural quality metrics.This example represents a lower-level syntactic refactoring.
  • Scope of agentic refactoring: The examples span low-level syntactic refactorings such as renaming and high-level structural changes such as method extraction.This contrast motivates evaluating measurable code-quality effects rather than inferring them from the edits alone.
  • Empirical gap: The study addresses limited empirical understanding of how often agents refactor, which operations they perform, why they are initiated, and how they affect internal quality metrics.These questions organize the study across four dimensions.

I. Data Mining

The section points to an overview of the study design and identifies it as the organizing visual for the data-mining process.

  • Study design: Figure 2 provides an overview of the study design.The figure is referenced as the overview for the study’s design.
  • Data mining: The data-mining section describes data mining and data filtering as outlined in Figure 2.The figure is used to organize these methodological stages.
  • Figure reference: The paper explicitly directs readers to Figure 2 for the study-design overview.No further stage-specific comparison is supplied in the provided passage.

3 Data Collection

The study constructs a filtered Java corpus from AIDev and GitHub data, detects refactoring operations, and labels explicit refactoring intent for analysis.

  • Data mining: The researchers expand AIDev metadata through the GitHub REST API, collecting 1,311,057 agentic commits before filtering.AIDev initially provides pull requests from five coding agents and repositories with more than 100 GitHub stars.
  • Filtering criteria: The study focuses on commits modifying at least one .java file and excludes merge commits to avoid duplicated changes and ambiguous parent relationships.These filters support automated analysis with RefactoringMiner and DesigniteJava.
  • Project classification: GPT-4.1-mini classifies repositories from README content, followed by manual review of repositories initially labeled toy_or_example.Seven projects were reclassified after manual verification.
  • Corpus construction: The final corpus contains 1,613 unique software projects and 14,998 non-merge Java commits suitable for automated refactoring detection.Projects are selected after classification, manual verification, exclusion of toy or uncertain repositories, and fork removal.
  • Refactoring detection: RefactoringMiner 3.08 detects 103 refactoring types and their locations by analyzing abstract syntax trees of modified Java files.The tool is reported to achieve an overall F-score of 99.5%.
  • Detection output: From 14,998 commits, RefactoringMiner identifies 5,789 commits containing at least one refactoring operation.The detected operations are linked back to their source commits.
  • Intent labeling: Agentic refactoring commits require both detected refactoring and a commit message signaling refactoring intent through patterns such as refactor*, cleanup, or restructure*.This procedure labels 3,907 commits as agentic refactoring commits and 11,091 as other commits.
  • Dataset overview: The analyzed commits come from 12,256 pull requests across 1,613 repositories, with 93.9% of pull requests closed and 86.9% merged.OpenAI Codex accounts for 89.3% of analyzed commits and 94.3% of analyzed pull requests.

4 Results

The study establishes how frequently refactoring appears in agentic commits and finds that explicit refactoring intent corresponds to more concentrated restructuring than incidental edits.

  • 4.1 Results: The analysis records refactoring-instance distributions across 14,998 commits and evaluates group differences with the Mann–Whitney U test and Cliff’s delta.Effect-size interpretation uses negligible, small, medium, and large thresholds based on |d|.
  • 4.1 Results: 26.1% of commits explicitly target refactoring, while additional refactoring instances appear in commits without stated refactoring intent.Agentic refactoring commits account for 3,907 of 14,998 commits; other commits contain 8,324 detected refactoring instances.
  • 4.1 Results: p ≤ 0.001 and Cliff’s delta d = 0.838 show that agentic refactoring and other commits differ significantly in refactoring instances per commit, with a large effect size.The comparison uses the distribution of refactoring instances per commit.
  • 4.1 Results: Explicit refactoring intent produces more concentrated and substantial restructuring, whereas unacknowledged refactoring occurs sporadically during feature implementation or bug fixing.This distinction characterizes how agents refactor across commit types.

4.2 𝑅𝑄2: What are the common types of agentic refactoring?

Agentic refactoring favors localized, low-level consistency edits over high-level structural changes, with maintainability and readability dominating its stated purposes.

  • Common refactoring operations: Change Variable Type, Rename Parameter, and Rename Variable are agents’ three most common operations, together comprising 30.7% of instances.Agents prioritize localized naming and type consistency edits, while humans more often perform method-level API changes.
  • Refactoring abstraction levels: 35.8% of agentic refactorings are low-level, compared with 24.4% for humans, while high-level refactorings account for 43.0% versus 54.9%.Medium-level refactorings occur at similar rates for agents and humans: 21.2% versus 20.7%.
  • Refactoring purposes: Maintainability motivates 52.5% of agentic refactoring commits, compared with 11.7% in human refactoring, while readability accounts for another 28.1%.Together, maintainability and readability represent more than 80% of agentic refactoring purposes.
  • Refactoring purposes: Design-level motivations are uncommon for agents: duplication accounts for 1.1% and repurpose/reuse for 4.6%, versus 13.7% and 12.9% for humans.This contrasts with the agents’ emphasis on localized quality cleanup and consistency edits.
  • Refactoring purposes: Corrective purposes remain secondary, including hard-to-debug code at 1.9% and legacy code at 2.0%.Examples include adding missing logging and upgrading Java versions rather than deep bug fixing.

4.4 𝑅𝑄4: To what extent does agentic refactoring affect code quality?

Agentic refactoring produces statistically significant but generally small structural changes, with the clearest improvements occurring in medium-level refactorings.

  • Smell-count changes: Design and implementation smell counts show statistically significant differences but negligible practical effects, with median Δ = 0.00 for both categories.Before-and-after distributions nearly overlap; FDR-adjusted p < 0.001, while Cohen’s d is -0.027 and -0.026.
  • Effects by abstraction level: Medium-level refactorings reduce Class LOC and WMC, with median changes of -15.25 and -2.07, respectively.Method-level LOC also decreases, with a median Δ of -1.79.
  • Effects by abstraction level: Low-level edits reduce method LOC by a median Δ of -0.42 but can slightly increase method cyclomatic complexity.High-level signature-only changes produce smaller metric shifts than medium-level structural changes.
  • Effects by abstraction level: Structural decomposition refactorings improve modularity and reduce class-level complexity by distributing responsibilities across more cohesive components.Refactoring type strongly influences measured code-quality impact.

Finding #12: Not all refactoring types improve measured metrics; some primarily support

Measured structural metrics do not improve uniformly across refactoring types: some changes primarily support readability, naming consistency, or API clarity.

  • Metric interpretation: Identifier renames, access changes, and annotation adjustments often show negligible structural-metric changes after FDR correction.Their benefits may instead concern readability, naming consistency, or API clarity, which the selected indicators do not capture.
  • Metric interpretation: Maintainability-oriented refactorings produce clearer metric gains, whereas readability-oriented refactorings such as renames rarely alter measured structural metrics.Move And Inline Method can reduce method LOC while leaving cyclomatic complexity unchanged or slightly increasing it.

5 Implications

The implications center on strategic delegation: agents handle routine, low-level cleanup, while human reviewers and developers retain responsibility for architectural changes, tangled commits, and long-term quality assessment.

  • Implications for Researchers: 53.9% of refactoring instances occur without explicit refactoring intent, increasing review burden because developers must validate both primary tasks and incidental changes.The study recommends separating refactorings from major tasks and making change intent explicit.
  • Implications for Researchers: 26.1% of agentic commits explicitly state refactoring intent, yet traditional quality metrics show minimal improvement, motivating research on whether localized edits deliver tangible benefits.The proposed research direction distinguishes improved comprehension from low-value code churn that leaves deeper structural issues unresolved.
  • Implications for Researchers: Medium-level structural gains remain small and require longitudinal validation against defect density, post-release defects, and future-modification effort.Class LOC median Δ = -15.25 and WMC median Δ = -2.07, while design and implementation smells are not consistently reduced.
  • Developers and Coding Agent Builders: Agents excel at low-level refactorings versus humans, 35.8% vs. 24.4%, but lag in high-level design changes, 43.0% vs. 54.9%.Developers can delegate routine cleanup while focusing human effort on design-level restructuring requiring domain knowledge and architectural intent.
  • Coding Agent Builders: Agent builders should train agents on curated high-level refactoring datasets and equip them with specialized analysis tools to detect and fix design flaws.The recommendations target the gap between tactical cleanup and autonomous architectural planning.

6 Related Work

Prior work frames refactoring as a maintenance practice with mixed quality effects and limited adoption of automated tools, while this study addresses the unexplored empirical role of autonomous coding agents.

  • Refactoring and Its Empirical Foundations: Refactoring improves internal quality without changing observable behavior and supports readability, maintainability, comprehension, and preparation for future changes.Empirical work also links dedicated refactoring efforts with reduced inter-module dependencies and post-release defects.
  • Large-Scale Empirical Analyses: Large-scale mining has established that refactoring is pervasive in industrial and open-source development, enabled by detectors such as RefactoringMiner and RefDiff.
  • Large-Scale Empirical Analyses: Empirical evidence on refactoring impact is mixed: some studies report quality and productivity benefits, while others find inconsistent smell removal and new smell introduction.One cited study found fewer than 10% of refactorings removed code smells and over 30% introduced new ones.
  • Automated Refactoring Techniques: Automated and semi-automated refactoring tools reduced manual effort but suffered from low adoption because of limited trust and semantic reasoning.Recent work also began evaluating refactoring capabilities of large language models.
  • Agentic Software Development and Research Gap: This study provides the first large-scale empirical view of agentic refactoring across prevalence, AI–human refactoring styles, intent, and quality impact.Agentic refactoring differs from prompt-based generation because agents plan, execute, and validate changes through iterative reasoning with minimal human intervention.

7 Threats to Validity

The study’s validity is constrained by imperfect analysis tools, uncertain human intervention in agentic commits, and limited generalizability beyond open-source Java projects.

  • Internal Validity: The study relies on automated detectors and classifiers that may introduce measurement errors, although manual validation of purpose classification achieved substantial agreement with Cohen’s kappa of 0.77.
  • Construct Validity: Agentic commits are identified from keywords and author information, making the extent of human intervention difficult to determine.The study therefore characterizes its setting as human–AI collaborative refactoring rather than purely autonomous AI contribution.
  • External Validity: Findings may not generalize to industrial closed-source projects because development practices, coding standards, and refactoring types may differ.
  • External Validity: The analysis is limited to commits involving Java files, so prevalence and impact may vary across programming languages.

8 Conclusion

The study finds that agentic refactoring is common and quality-oriented but primarily incremental: it produces small structural gains while remaining weak at architectural restructuring and smell reduction.

  • Conclusion: 15,451 refactoring instances across real-world open-source Java projects provide the study’s large-scale empirical basis.
  • Conclusion: 26.1% of agentic commits explicitly target refactoring, with maintainability and readability accounting for over 80% of motivations.Maintainability accounts for 52.5% and readability for 28.1%.
  • Conclusion: Agentic refactoring favors low-level consistency edits and performs fewer high-level design changes than human refactoring.The observed pattern indicates incremental improvements rather than architectural restructuring.
  • Conclusion: Class LOC median Δ = -15.25 and WMC median Δ = -2.07 show small, statistically significant structural improvements for medium-level changes.Agents nevertheless fail to consistently reduce the overall count of known design and implementation smells.
  • Conclusion: Agentic coding tools currently serve effectively as incremental cleanup partners, but substantial advances are needed for architecturally aware restructuring.
Loading 2511.04824v1…