PaperlayerView exact revision on arXiv

Source-linked AI summary

From Cosmetic to Structural: A Paired Experiment on Metric-Guided Agentic Refactoring

Nadim Hagatulah, Markus Borg, Emma Söderberg

arXiv:2610.33343v1cs.SE

TL;DR

Agent refactorings often favor superficial edits over structural changes needed to address technical debt, and evidence on metric-guided agentic refactoring is limited. This study compares paired Basic and Guided refactorings using iterative CodeScene feedback across three languages. Guidance raises effective-refactoring rates, with outcomes depending on initial CodeHealth and added interaction and token costs.

  • Problem

    Evidence is limited on whether agents can act structurally on smell reports, how guidance interacts with initial code quality, and what correctness and compute overhead it brings.

    Source: §2 · p. 5

  • Method

    The study compares paired Basic and Guided runs across C++, Java, and Python, with Guided runs using CodeScene MCP for iterative smell reports and CodeHealth feedback.

    Source: §1 · p. 3, §3.3 · p. 8

  • Results

    21–26 percentage points: metric guidance raises the effective-refactoring rate, with break-even points at CH 4.9–7.1 below which guidance lowers that rate.

    Source: §1 · p. 4, §7 · p. 29

  • Takeaways & Limitations

    The findings indicate that static-analysis feedback can redirect agents toward structural transformations, while the benefit depends on initial code quality and carries higher token and interaction costs.

    Source: §1 · p. 4, §7 · p. 29

    The Guided treatment bundles metric-referencing instructions, tool access, and external feedback, so the experiment does not isolate the contribution of metric content from iterative feedback.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

The role of coding agents is expanding beyond writing new code toward maintaining existing systems as AI teammates. Yet recent studies show that agent refactorings favor superficial, consistency-oriented edits over structural transformations needed to resolve Technical Debt (TD). We investigate if explicit static-analysis feedback closes this gap. Through the CodeScene MCP server, we give an open-source agent (gptme) access to code smell reports and CodeHealth (CH) scores (1--10, higher is better), and compare guided against unguided refactorings in a paired experiment covering $24{,}620$ single-file competitive programming solutions in C++, Java, and Python. We find that guidance shifts the agent from performing cosmetic edits toward structural operations, raising the rate of effective refactorings by $21$--$26$ percentage points. The benefit depends on initial code quality, with language-dependent break-even points at CH $4.9$--$7.1$, below where the structural transformations pushed by guidance increasingly risk regression, indicating where the agent needs more support to refactor. Guided runs also shift the failure profile toward logical test failures, introduce new coupling smells, and consume $\sim$$3$--$5\times$ more tokens over $\sim$$2\times$ as many interaction turns. Our results indicate that AI teammates can benefit from similar static-analysis feedback that human developers already use, enabling structural transformations toward more effective TD remediation, at a higher token cost and more interaction turns.

1 Introduction

This study tests whether iterative static-analysis feedback can redirect agentic refactoring from superficial edits toward structural transformations. Across a paired, three-language evaluation, guidance improves effective-refactoring rates, but its benefit varies with initial CodeHealth and comes with trade-offs.

1 Introduction

  • Prior studies find that agents favor consistency-oriented edits over structural changes, while refactoring remains risky and is often deprioritized by developers.The study asks whether agents can use the static-analysis feedback already surfaced to developers to address this gap.Source: §1 · p. 2
  • The paired evaluation covers 24,620 files in C++, Java, and Python, comparing Basic and Guided runs with the same agent, model, and environment.Guided runs add CodeScene MCP access to code-smell reports and file-level CodeHealth scores.Source: §1 · pp. 2–3
  • The authors caution that their single-file competitive-programming corpus isolates structural reasoning and does not establish production-code maintenance performance.Production systems may involve cross-file dependencies and build systems absent from this experimental setting.Source: §1 · p. 3
  • Guidance raises effective refactoring by 21–26 percentage points while redirecting agents toward structural transformations, with language-specific break-even points from CH 4.9 to 7.1.Below those break-even points, guidance lowers the effective-refactoring rate; above them, it helps by up to 34 percentage points.Source: §1 · pp. 3–4

2 Related Work

Prior work reports that autonomous agents often make localized or consistency-oriented edits rather than resolving underlying design issues. This study addresses the limited evidence on whether iterative static-analysis feedback can guide agents toward structural refactoring and how that guidance interacts with initial code quality and overhead.

2 Related Work

  • Large-scale evaluations report that autonomous agents favor low-level consistency changes and annotations over structural modifications that address underlying code smells and design issues.These studies describe a drift in refactoring intent relative to human-driven maintenance.Source: §2 · p. 4
  • The study tests whether iterative metric feedback can operationalize proposals to give agents explicit refactoring goals, rather than only enriching a static prompt.Cordeiro et al. argue for issue-specific guidance because LLMs without explicit goals can misalign with maintainability intent.Source: §2 · p. 4
  • Borg et al. found that human-comprehensible code is more tractable for AI refactoring, but evaluated single-shot Python refactoring rather than multi-turn agents across languages.This study extends that line of work to two additional languages and interactive metric feedback.Source: §2 · p. 5
  • Evidence remains limited on agents’ ability to act structurally on smell reports, how guidance interacts with initial code quality, and its correctness and compute overhead.The paper addresses these questions with a paired, multi-language evaluation of multi-turn refactoring.Source: §2 · p. 5

3 Methodology

The methodology uses a paired experiment to compare Basic refactoring with a Guided configuration that supplies iterative CodeScene feedback. It combines controlled single-file tasks, code-quality and correctness measures, and complementary statistical and visual analyses.

3.1 Dataset Construction

  • The dataset contains 9,825 C++, 9,795 Java, and 5,000 Python solutions with STDIN/STDOUT tests serving as behavioral oracles.C++ and Java sampling includes low-CodeHealth files; the reused Python subset is concentrated in the healthy range, with only 10 files below CH 6.Source: §3.1 · pp. 5–6

3.2 Agent and Model Selection

  • The study uses the open-source gptme harness and open-weight Qwen3-Coder-30B-A3B-Instruct model to support a reproducible experimental stack.The model runs on a single NVIDIA A100 GPU, and gptme supports the file-editing tools and MCP extension required by the setup.Source: §3.2 · p. 6

3.3 Experimental Setup

  • The paired experiment tests each solution under Basic and Guided conditions, with the same agent, model, and environment but different instructions and tool availability.Guided runs can query smell category, severity, location, and CodeHealth through the MCP server.Source: Figure 1 · p. 7, §3.3 · pp. 7–8

3.4 Evaluation Metrics

  • Effective refactoring requires a run to compile, pass the test suite, and strictly improve CodeHealth; the study also tracks pass rate, CH improvement, turns, and tokens.CH improvement alone does not require the solution to compile or pass tests, and CH is used as a proxy under CodeScene’s rules.Source: §3.4 · p. 8

3.5 Refactoring Operation Extraction

  • RefactoringMiner detects AST-difference operations for Java and Python, which are classified as low-, medium-, or high-level by change scope.C++ is excluded from this operation analysis; the framework does not rank levels by maintainability impact, and structural changes mainly refer to medium-level operations such as Extract Method.Source: §3.5 · p. 9

3.6 Failure Taxonomy

  • Failures are classified by their first failing stage as compile errors, runtime errors, test failures, or exceptions, with Python error classes mapped for cross-language comparison.Rule-based regular expressions map compile and runtime messages to causes.Source: §3.6 · p. 9

3.7 Statistical Analysis

  • Paired nominal outcomes use exact McNemar tests, while continuous outcomes use Wilcoxon signed-rank tests; operation shares and failure-category shifts receive specialized paired analyses.The analyses include Holm correction for per-smell tests, task-level aggregation for operations, GEE interaction models for moderation, and within-task permutations for failure modes.Source: §3.7 · pp. 9–10

3.8 Visualization Design; 3.9 Conversation Logs

  • Figure 2 visualizes per-file CH movement ordered by initial CH, complementing aggregate results with distributions of movement across Basic and Guided runs.The visual design is argued from information-visualization principles rather than validated in a user study; seven conversation logs provide illustrative observations without establishing representativeness.Source: §3.8 · pp. 10–11, §3.9 · p. 11

4 Results

The results examine how metric guidance affects refactoring effectiveness, structural smell remediation, behavioral correctness, interaction and token costs, and effectiveness across initial code health. Log observations also describe agents’ feedback-driven iterations, intermediate regressions, judgments about remaining smells, and limited or absent verification.

4.1 RQ1: Impact of Metric Guidance on Refactoring Outcomes

  • Metric guidance raises effective refactoring rates, measured under the guiding metric, by +25.2 pp in C++, +21.1 pp in Java, and +25.9 pp in Python (p < 0.01), while lowering pass rates by 2.3–8.6 pp.It raises strictly improved-CH runs from 30.6% to 69.6% in C++, 46.9% to 81.8% in Java, and 43.0% to 77.2% in Python.Source: §4.1 · pp. 11, 12, 14

4.2 RQ2: Changes in Refactoring Operations and Smell Remediation

  • Guided refactoring reduces complexity smells, including Large Method by 72.7–96.1%, but introduces coupling smells, especially Excess Arguments in Java.Excess Arguments instances in Java grow from 1,080 to 4,825; Primitive Obsession is the only tested smell where Basic outperforms Guided in both C++ and Java.Source: §4.2 · pp. 16–17, Figure 3 · p. 16
  • Metric guidance shifts operations from low-level edits, falling from 57% to 32% in Java and 70% to 49% in Python, toward structural, Extract Method-dominated transformations, rising from 12% to 49% and 11% to 32%, respectively.Source: §4.2 · p. 17

4.3 RQ3: Moderating Effect of Initial CH

  • Guidance’s effective-refactoring advantage varies with initial CH, with break-even points at CH 4.9 in C++, 5.9 in Java, and 7.1 in Python.Below these thresholds, Guided runs are less likely to be effective because they more often break behavior, although successful Guided transformations still deliver larger CH gains; estimates below CH 5 rely on few solutions.Source: Figure 4 · p. 18, §4.3 · pp. 19–20

4.4 RQ4: Shifts in Failure Modes

  • Among failed runs, Test-failure shares rise by 9.5 pp in C++, 5.6 pp in Java, and 3.9 pp in Python, while compile-error shares fall in C++ and Java.Guided failures also include more infrastructure and session-length exceptions; C++ Missing Symbol/Identifier errors make up 62.7% of Guided compile errors, and segmentation faults fall as timeouts grow.Source: §4.4 · pp. 20–22, Table 7 · p. 21

4.5 RQ5: Computational Overhead

  • Guidance costs 3.0–5.4× the total tokens and 2.1–2.2× the interaction turns per task.Input tokens account for about 98% of Guided tokens as prior context accumulates in the iterative feedback loop.Source: §4.5 · pp. 22–23, Table 9 · p. 23

4.6 Observations from Logs

  • Sampled logs show Basic runs typically apply one patch, while Guided runs iteratively propose structural changes, re-query code health, and make further edits.Intermediate edits can lower CH before later recovery; one Python session dropped from 8.95 to 7.83 before rising to 9.24, and a high-CH Java file was rewritten repeatedly without score improvement.Source: §4.6 · pp. 23–24
  • After three failed attempts, the agent judged the remaining DFS smell inherent to the algorithm and considered the refactoring complete.It said separating the two logical sections would require changing the algorithm, contrary to the task’s requirement to preserve semantic behavior while following CodeScene’s metric.Source: §4.6 · p. 24
  • In one Python session, mocked-input verification crashed, prompting a syntax-check fallback; other observed sessions ended without compilation or execution.The fallback was `python3 -m py_compile`.Source: §4.6 · p. 24

5 Discussion

The discussion finds that metric feedback redirects the agent toward structural remediation, but exposes trade-offs in coupling, behavioral correctness, and token cost. The benefit depends on initial code quality, and the authors interpret CH gains as metric-specific rather than tool-agnostic maintainability gains.

5.1 Towards Structural Remediation

  • Basic runs favor cosmetic edits, while metric feedback makes Extract Method a leading operation and supplies concrete targets for structural changes.Under Basic, variable renaming accounted for 41–57% of operations, and 31–47% of runs improved CH; Extract Method became most frequent in Java and second most frequent in Python.Source: §5.1 · pp. 24–25

5.2 Refactoring Side Effects

  • Structural decomposition trades complexity smells for coupling smells, including excess and string-heavy arguments; the agent lacks architectural foresight to recompose clean abstractions.Reviewers may need to restructure dataflow after decomposition.Source: §5.2 · p. 25
  • Compile and test failures rise partly because this setup does not require verification, leaving checks dependent on the model and harness.More frequent compilation and testing could avoid most of those fault classes.Source: §5.2 · p. 25

5.3 AI-Friendliness Threshold

  • At CH 5, Guided effective-refactoring advantage is +1 pp in C++, −8 pp in Java, and −25 pp in Python, versus +26–34 pp at CH 9; below break-even, agents need support.The estimated break-even points, CH*≈4.9–7.1, are rough and specific to the study’s languages, model, harness, and metric.Source: §5.3 · pp. 25–26

5.4 Optimizing the Metric

  • CH gains are improvements under the guiding metric, not necessarily tool-agnostic maintainability; CH serves as both guidance and outcome, and may reward proxy optimization.Logs also show intermediate CH dips and runs that stop before a perfect score when remaining smells seem inherent to the algorithm.Source: §5.4 · p. 26

5.5 Overhead of Guided Refactoring

  • Guidance uses ∼3–5× more tokens per run, while a retrospective CH-threshold routing simulation matched all-Guided effectiveness and improved pass rate and token efficiency.In Java, the routed policy achieved 54.9% effective rate and 69.0% pass rate using 194k mean tokens, versus 54.8%, 68.6%, and 198k for all-Guided; the estimated threshold requires local calibration.Source: §5.5 · pp. 26–27

5.6 Implications

  • Practitioners should not equate refactoring activity with debt remediation; tool builders should warn about smells edits may introduce and support integration-level verification.The authors also recommend evaluating how guidance interacts with initial code quality and separating guiding signals from assessment measures.Source: §5.6 · pp. 27–28

6 Threats to Validity

The authors qualify their findings by the measurement choices, experimental setup, and study corpus. In particular, the results are tied to one agent, model, metric, and isolated single-file solutions.

6.1 Construct Validity

  • Because CH is both the guidance signal and outcome, effective-refactoring gains indicate improvement under CH rather than a general notion of maintainability; token counts are approximate but comparable across conditions.Counts use tiktoken’s cl100k base rather than the model’s tokenizer, while applying the same measurement tools in both conditions.Source: §6.1 · p. 28

6.2 Internal Validity

  • Each solution was run once per condition, and the bundled Guided treatment cannot isolate the effects of instructions, tool access, and tool feedback.674 Guided runs with pre-refactoring MCP failures count as failures and contribute to the lower pass rate on degraded files; timeouts and context pressure may also affect outcomes.Source: §6.2 · p. 28

6.3 External Validity

  • The isolated single-file competitive-programming corpus omits enterprise dependencies and build systems, so success rates and break-even thresholds may differ in multi-module codebases.The study also uses one harness, one open-weight model, and one maintainability metric; broader replication is needed before generalizing trends.Source: §6.3 · pp. 28–29

7 Conclusion

In a paired study of 49,240 refactoring runs across C++, Java, and Python, static-analysis guidance redirected gptme toward structural changes, with effectiveness depending on initial code quality. The gains came with more behavioral failures and computational cost, motivating further work on guidance and evaluation.

Structural refactorings

  • Guidance raised the share of runs improving CH by 34–39 percentage points and the effective refactoring rate by 21–26 points under the guiding metric.Without explicit targets, agents defaulted to cosmetic, consistency-oriented edits; actionable feedback redirected them toward structural transformations such as Extract Method.Source: §7 · p. 29

The AI-Friendliness threshold

  • Below language-specific break-even points of CH ≈4.9–7.1, guided decomposition increasingly risks behavioral regression.The agent’s ability to restructure code safely depends on its initial quality.Source: §7 · p. 29

Shifting the failure risk

  • Guided runs shifted failures away from primitive crashes, especially C++ segmentation faults, toward logic errors and timeouts, increasing the importance of comprehensive test suites.The faults introduced by guided agents were more often semantic than syntactic.Source: §7 · p. 29

Computational overhead

  • The feedback loop doubled interaction turns and consumed approximately 3–5× more tokens than unguided refactoring.The authors propose ablations to separate the value of metric semantics from the general benefit of external feedback.Source: §7 · p. 29

Future research

  • Future work should monitor new trade-off smells, separate guidance from assessment metrics, and test the design in multi-file enterprise repositories.These directions address architectural issues introduced during refactoring, proxy optimization, and the limits of single-file experiments.Source: §7 · pp. 29–30

Declarations

The declarations disclose an author’s employment by the studied tool vendor and make the dataset, analysis scripts, and prompt templates available in the replication package.

Conflict of interest.

  • Markus Borg is employed by CodeScene, the vendor of the CodeScene CLI tool whose CH metric and MCP server are studied.Source: Conflict of interest · p. 30

Data availability.

  • The replication package contains the full agent-run dataset, including per-task outcomes, CH scores, RefactoringMiner output, and sampled conversation logs.Source: Data availability · p. 30

Code availability.

Appendix A Prompt Templates

The basic prompt asks the agent to refactor a single file for maintainability while preserving behavior and using specified read, save, and validation steps. The guided prompt adds CodeHealth feedback and a review loop for identified issues.

A.1 Basic Prompt

  • The basic prompt assigns the agent the role of an expert software engineer specialized in the target language.Source: §A.1 · p. 30
  • The basic task is to improve maintainability and code quality without changing semantic behavior.Source: §A.1 · p. 30
  • The basic prompt requires overwriting the same file, preserving its name and behavior, avoiding new dependencies, and limiting changes to maintainability remediation.Source: §A.1 · p. 31
  • The basic workflow reads and analyzes the file, plans and applies refactoring, then rereads and validates the result, repeating if issues remain.Source: §A.1 · p. 31

A.2 Guided Prompt

  • The guided prompt retains the expert role and directs refactoring by CodeHealth; it provides the CodeScene MCP server’s code_health_review tool.Source: §A.2 · p. 31
  • The guided prompt restricts edits to identified maintainability issues, preserves behavior and the filename, and disallows new dependencies or tools.Source: §A.2 · p. 31
  • The guided workflow runs code_health_review before and after refactoring, validates that issues are resolved without introducing new ones, and repeats if needed.Source: §A.2 · p. 32

Appendix B GEE Model Specification

The appendix specifies separate GEE logistic regressions for three binary outcomes, modeling guidance and initial CodeHealth together to estimate how initial quality moderates guidance’s association with outcomes.

Model specification

  • The RQ3 moderation analysis fits a separate GEE logistic regression for each binary outcome: effective refactoring, strictly increased CH, or passing tests.Source: Appendix B · p. 32
  • Each model predicts the probability of an outcome from an intercept, guided-prompt indicator, initial CH, and their interaction.Source: Appendix B · p. 32

Model interpretation

  • The interaction sign indicates whether guidance helps more or less as initial CH rises, and the break-even score is CH*=−βP/βP×CH.Models use exchangeable working correlation and cluster observations by task.Source: Appendix B · p. 32

Appendix C Refactoring Operation Levels

Table C2 catalogs the RefactoringMiner operation types detected in the Java and Python runs, grouped by refactoring level.

Appendix C Refactoring Operation Levels

  • Table C2 lists 84 RefactoringMiner operation types detected in Java and Python, grouped by the refactoring level of Horikawa et al..Source: Appendix C · p. 32

Appendix D SLoC across CH

Table D3 reports source lines of code across initial CodeHealth bins.

Appendix D SLoC across CH

  • Table D3 reports original-file SLoC by initial CH bin, with CH floored to an integer.Source: Appendix D · p. 32

Appendix E Code Smell Glossary

The appendix documents CodeScene smells and detection thresholds, alongside tables of refactoring operations, CH-bin SLoC, and regression results.

Appendix E Code Smell Glossary

  • Table E4 summarizes the study’s CodeScene smells and their default detection thresholds, identical across the C++, Java, and Python rule sets.A dash denotes composite-rule smells or rules without a tunable numeric threshold.Source: Appendix E · p. 32, Table E4 · p. 34
  • Table B1 presents GEE regressions for effective refactoring, strict CH improvement, and passing tests, matching Figure 4’s three rows.Coefficients are log-odds with task-clustered standard errors; CH* marks the initial-CH break-even point, with 95% confidence intervals for the primary outcome.Source: Table B1 · p. 33
  • Table C2 lists RefactoringMiner operation types detected in Java and Python, grouped by refactoring level.The table lists 84 operation types.Source: Table C2 · p. 33
  • Table D3 reports original-file SLoC by initial CH bin, with CH floored to an integer.Source: Table D3 · p. 34
  • Table E5 reports smell instances before and after refactoring, with net reduction calculated as 100 × (Before − After)/Before.The table covers tasks with CodeScene reviews of both refactored files; it omits language-specific smells with fewer than 20 initial instances.Source: Table E5 · p. 35

Related papers

Loading 2610.33343v1…