PaperlayerView exact revision on arXiv

Source-linked AI summary

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

Muhammad Waseem, Aakash Ahmad, Pekka Abrahamsson

arXiv:2608.23611v1cs.SEcs.AI

TL;DR

LLM refactoring can reduce visible code smells while risking behavioral or integration changes. REFINE uses an evidence-aware multi-agent workflow for Java file-level refactoring candidates and substantially reduces detected smells, though broader quality gains remain inconsistent and candidates require review.

  • Problem

    LLM-assisted refactoring may reduce visible code smells while altering assertions, public methods, exception handling, or framework-sensitive control flow.

  • Method

    REFINE combines static-analysis-guided smell identification, agent planning, LLM transformation, automated re-analysis, preservation checks, and structured reporting for Java file-level candidates.

  • Results

    68.26%–72.79% code-smell reduction occurred across all three LLM configurations, with the largest reductions for major smells and inconsistent broader quality improvements.

  • Takeaways & Limitations

    REFINE offers traceable, evidence-aware mitigation of targeted Java file-level smells, but candidates require compilation, testing, dependency analysis, and human review before adoption.

  • Takeaways & Limitations

    Evaluation is limited to Java file-level tasks and should not be interpreted as evidence of repository-wide or system-level refactoring effectiveness.

Abstract

from arXiv · show

Large Language Models (LLMs) offer new opportunities for automated code refactoring. However, generated changes must reduce targeted quality problems without introducing new issues or altering behaviour-relevant code structures. We introduce REFINE (Refactoring with Evidence-aware Flow for Integrated ageNtic Execution), a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level refactoring candidates. REFINE combines static-analysis-guided smell identification, smell-informed planning, LLM-based transformation, automated re-analysis, preservation checks, and structured reporting. We evaluate REFINE on 450 Java files from 15 open-source systems, producing 1,350 model-pass outputs using OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8. REFINE reduces detected code smells by 68.26%, 72.79%, and 68.49% across the three configurations, respectively, with the strongest reductions observed for major smells. A matched 150-file direct-prompt baseline shows that REFINE achieves a higher median code-smell reduction with smaller edits and fewer public-method removals. However, broader quality improvements are inconsistent, and preservation checks reveal residual risks, including assert/fail-call changes and public-method removal. Therefore, REFINE outputs should be treated as refactoring candidates requiring compilation, testing, dependency analysis, and human review before adoption in repository- or system-level settings.

I. INTRODUCTION

REFINE is an evidence-aware multi-agent workflow that generates traceable Java file-level refactoring candidates through analysis, transformation, verification, and reporting. Its evaluation examines smell reduction and preservation risks while emphasizing that outputs require additional validation before broader adoption.

  • Motivation: LLM-assisted refactoring can generate useful transformations but may also remove behaviour-relevant code structures or introduce practical risks.The introduction notes that simplifying methods or splitting classes can have unintended effects.
  • Approach: REFINE combines static-analysis-guided smell identification, smell-informed planning, LLM transformation, automated re-analysis, preservation checks, and structured reporting.Verification gates record whether candidates satisfy configured evidence checks.
  • Approach: REFINE makes each candidate traceable to motivating smells, verification outcomes, and quality or preservation risks observed after transformation.This traceability supports analysis of generated candidates rather than treating transformations as automatically safe.
  • Evaluation: 450 Java files from 15 open-source systems produced 1,350 model-pass outputs across three LLM configurations, alongside analysis of smells, quality, preservation, failures, and model behaviour.The study also compares REFINE with a matched direct-prompt baseline on a 150-file subset.
  • Implications: REFINE outputs require additional validation before dependency-rich or system-level adoption.The paper presents the workflow as a traceable way to study candidates while retaining the need for further validation.

II. APPROACH … C. Task Characterization

REFINE is an evidence-aware, multi-agent workflow for generating and evaluating Java file-level refactoring candidates. It combines static-analysis evidence, agent orchestration, verification, and trace collection while treating generated code as candidates rather than automatically accepted results.

  • II. APPROACH: REFINE provides an evidence-aware multi-agent workflow for generating and evaluating Java file-level refactoring candidates.The workflow includes implementation, task characterization, orchestration, and verification trace collection.
  • A. Overview: Static analysis identifies file-level code smells and rule-level evidence that guide agent planning, LLM transformation, and candidate verification.The workflow begins with selected Java files from open-source projects and uses detected smells as refactoring-stage input evidence.
  • A. Overview: REFINE links each target file with detected smells, refactoring instructions, public API information, and workspace context across characterization, orchestration, and trace collection.These responsibilities organize task preparation, transformation generation, verification, and persistence of workflow outputs.
  • A. Overview: REFINE is not a direct-prompt rewrite: verification gates record evidence-check outcomes, and generated code remains a refactoring candidate.Detected smells guide planning before the LLM generates a transformation, which is then verified against configured checks.
  • B. Tool Implementation: The prototype combines a Next.js interface, Java Spring Boot backend, and Python agent service coordinated with LangGraph v1.1.The Java backend uses PMD 7.x for before–after smell detection and recounting, while the Python service handles orchestration, routing, verification, and metric computation.
  • B. Tool Implementation: The empirical study saves candidates, verification outcomes, metadata, metrics, diagnostics, and reports without writing candidate files back to original repositories.This artifacts-only configuration separates controlled candidate evaluation from project integration and supports reproducibility.
  • C. Task Characterization: Each bounded task targets a single Java file from ZIP uploads, Git repository clones, or inline source content and preserves workspace context.Task records include a workspace identifier, file path, source content, selected static-analysis evidence, and protocol metadata.
  • C. Task Characterization: REFINE prioritizes detected smells before prompt construction and supplies source code, rule-based instructions, public API constraints, and retry notes to the refactoring prompt.PMD is the configured empirical backend, but the workflow is not tied to a specific static-analysis tool.

D. Refactoring Orchestration · E. Verification Trace · III. EXPERIMENTAL DESIGN

REFINE orchestrates evidence-guided Java refactoring through coordinated planning, transformation, verification, and reporting roles. It re-analyzes candidates, applies preservation-oriented gates, and records detailed traces for later assessment of improvements and risks.

  • D. Refactoring Orchestration: Eleven workflow roles coordinate file loading, smell detection, planning, size assessment, LLM transformation, verification, compilation checking, optional application, and reporting.The workflow processes static-analysis evidence and checks size feasibility before invoking the refactoring agent.
  • D. Refactoring Orchestration: Rule-based planning maps prioritized smells to refactoring instructions, while optional LLM refinement and a shared interface support different model configurations.Unchanged candidates may be retried with additional feedback, while verification failures are recorded as evidence rather than triggering full replanning.
  • E. Verification Trace: REFINE re-analyzes transformed files and defines absolute smell reduction using the before-and-after detected-smell counts B and A.B and A denote the numbers of detected code smells before and after refactoring, respectively.
  • E. Verification Trace: Positive smell-change values indicate reduced detected smells, whereas negative values indicate that additional smells were introduced.REFINE also computes relative code-smell improvement.
  • E. Verification Trace: Static verification gates check source difference, smell non-worsening, public API preservation, exception-handling risks, size bounds, and baseline method preservation.Candidates either pass configured checks in interactive configurations or remain as evidence in the artifacts-only study configuration.
  • E. Verification Trace: For each model pass, REFINE stores source versions, agent traces, before–after metrics, model metadata, diagnostics, rejection reasons, and reports.These artifacts link refactoring decisions to evidence from characterization, orchestration, and verification.
  • E. Verification Trace: The persisted trace supports later analysis of smell movement, quality indicators, change footprint, and preservation risks.This extends verification beyond immediate candidate acceptance to structured downstream analysis.

A. Research Questions · B. Study Scope and Comparative Context · C. Subject Systems and File Selection

The study examines REFINE as a controlled Java file-level workflow across 15 open-source systems, focusing on smell and quality changes, preservation evidence, and refactoring behaviour. A matched direct-prompt baseline provides limited comparative context, while stratified sampling defines the evaluated production files.

  • A. Research Questions: The matched direct-prompt baseline supplies comparative context rather than constituting a separate research question.The comparison concerns observed file-level outcomes, preservation evidence, and refactoring behaviour.
  • A. Research Questions: REFINE is evaluated through three research questions covering before–after smell and quality changes, preservation risks, and refactoring behaviours across LLM configurations.The behaviours include code addition or removal, method extraction or renaming, class splitting, duplicate-code removal, and public-method changes.
  • B. Study Scope and Comparative Context: The main study evaluates REFINE on 450 Java files from 15 systems using identical orchestration, prompting, and verification rules across three LLM configurations.The controlled file-level setting examines smell reduction, quality movement, preservation evidence, and refactoring behaviour.
  • B. Study Scope and Comparative Context: The direct-prompt baseline uses the same LLM configurations on a matched 150-file subset and supports comparison only within that subset.It is not a comprehensive benchmark of all refactoring tools, prompt designs, or agentic architectures.
  • B. Study Scope and Comparative Context: Each selected Java file is treated as an independent refactoring unit without cross-file dependencies, tests, build configurations, or repository-wide context in the LLM prompt.The prompt includes the target file, prioritized smells, refactoring instructions, public API constraints, and applicable retry notes.
  • C. Subject Systems and File Selection: The subject set contains 15 publicly available, nontrivial Java systems varied by project size, application domain, and codebase structure.Systems were required to be primarily Java, suitable for static analysis, and large enough to support file-level sampling.
  • C. Subject Systems and File Selection: Only production Java source files are eligible, excluding test paths, generated build outputs, version-control folders, and REFINE-internal folders.KLOC measures rounded thousands of physical production-source lines, while class and method counts provide lightweight static scale indicators.
  • C. Subject Systems and File Selection: Eligible files must contain at least one baseline smell and are sampled randomly across smell-count × LOC strata using seed 42, with batches capped at 15 files.The strata distinguish lightly, moderately, and heavily affected files, while the LOC dimension limits domination by one size range and previously selected paths are avoided.

D. Contextual Direct-Prompt Baseline … G. Data Analysis

The study compares REFINE with a matched direct-prompt baseline and evaluates model outputs using automated smell, quality, preservation, and change-footprint measures. Analyses distinguish the baseline from the main cohort and interpret results as file-level evidence rather than project-level causal claims.

  • D. Contextual Direct-Prompt Baseline: REFINE is compared with direct prompting on 150 matched Java files, with 10 files from each of 15 subject systems and 450 file-model pairs.Both approaches use the same original file and LLM configuration for each pair.
  • E. Experimental Setting: Detected smell reductions are measured with PMD 7.10.0 on Java 17 using the refactai-pmd-ruleset.xml ruleset.These reductions represent detected smells under the specified configuration, not manually validated Fowler-style instances.
  • E. Experimental Setting: The three configurations use GPT-5.5, Gemini 3.1 Pro Preview, and Claude Opus 4.8 with identical file-level tasks and REFINE constraints.Each task includes the complete Java file, prioritized smells, rule-based instructions, public API constraints, and retry notes when applicable.
  • E. Experimental Setting: All runs execute independently from a frozen baseline with shared prompts, workflow settings, verification, token-budget preflight, and line-count feasibility checks.The study follows an artifacts-only protocol recording candidates, metadata, before–after measurements, and verification traces.
  • F. Measures: RQ1 measures smell counts, severity changes, rule-level resolution, and static quality proxies including LOC, complexity, maintainability, testability, Halstead metrics, and cohesion.RQ2 uses verification, rejection, and preservation-risk indicators, while RQ3 measures edits, refactoring behaviour, API changes, and footprint patterns.
  • G. Data Analysis: The main study reports pooled and model-wise descriptive statistics at the model-pass level.The analysis covers the full cohort and distinguishes model-wise patterns from pooled summaries.
  • G. Data Analysis: RQ1 uses Holm-adjusted Wilcoxon signed-rank tests, RQ2 uses Cochran’s Q and McNemar tests with Wilson score 95% intervals, and RQ3 uses Kruskal–Wallis tests.These procedures address before–after, matched binary preservation, and code-smell-reducing-subset behaviour comparisons, respectively.
  • G. Data Analysis: The baseline uses Wilcoxon signed-rank tests for numeric paired outcomes and McNemar tests for binary paired outcomes, separately from the full 450-file analysis.Failed or incomplete passes are separated from verifier rejections, and inferential tests are interpreted as file-level evidence without project-level causal claims.

IV. RESULTS · A. Observed File-Level Outcomes (RQ1)

REFINE substantially reduces detected code smells across all three LLM configurations, with the strongest reductions for major smells and coverage across all 15 systems. Broader quality and structural improvements are mixed, varying by model and system, with Gemini 3.1 showing the clearest secondary gains.

  • IV. RESULTS: The section reports empirical results for three research questions and a contextual direct-prompt baseline using the dataset and supporting artifacts available online.This establishes the scope of the reported results rather than adding a separate outcome.
  • A. Observed File-Level Outcomes (RQ1): The results address RQ1 by comparing before–after measurements across three LLM configurations for smells, severity levels, quality indicators, and structural metrics.The analysis also examines whether observed changes are distributed across systems or concentrated in a few cases.
  • A. Observed File-Level Outcomes (RQ1): 68.26% to 72.79%: REFINE’s total code-smell reduction across the three LLM configurations.Reductions are statistically significant for all severity categories.
  • A. Observed File-Level Outcomes (RQ1): 86.51% to 91.60%: REFINE’s reduction in major smells, the largest reductions among severity categories.The workflow is most effective when the target is explicit and measurable through static smell evidence.
  • A. Observed File-Level Outcomes (RQ1): Cyclomatic complexity and LCOM improve mainly for Gemini 3.1, while broader quality and structural indicators show mixed movement.Maintainability, testability, Halstead effort, LOC, and mean method length are not consistently improved across models.
  • A. Observed File-Level Outcomes (RQ1): Code-smell reduction appears across all 15 systems, whereas non-smell indicators vary more by system and model.Gemini 3.1 shows the broadest beneficial pattern, while GPT-5.5 and Opus 4.8 show less stable structural effects.
  • A. Observed File-Level Outcomes (RQ1): The aggregate smell-reduction result is not driven by a single system, but broader quality movement is less uniform.This system-level profile distinguishes consistent targeted-smell reduction from variable secondary effects.

B. Preservation and Verification (RQ2) · C. Refactoring Behaviour (RQ3)

Most preservation indicators remain high, but critical assert/fail-call changes and public-method removal remain important risks, while smell reduction reflects distinct model edit profiles and greater absolute—but not relative—benefits from larger edits.

  • B. Preservation and Verification (RQ2): Most behavioural-risk and structural-preservation indicators are retained at high rates, including signatures, exception handling, framework contracts, and conditional constructs.The RQ2 analysis covers preservation indicators, verification outcomes, and failure diagnostics.
  • B. Preservation and Verification (RQ2): 81.8%–90.0% is the range of the complete static preservation proxy, but these checks do not prove behavioural equivalence.The proxy covers visible API- and structure-related preservation indicators.
  • B. Preservation and Verification (RQ2): 57.1% of outputs pass critical assert/fail-call preservation for each model, making it a common cross-configuration risk.Critical assert/fail-call failures account for 579 model-pass outputs.
  • B. Preservation and Verification (RQ2): Public-method removal is the clearest concrete failure mechanism, with Gemini 3.1 showing the most cases and highest mean removals per affected file.Critical assert/fail-call loss is larger overall but less explained by available diagnostics.
  • C. Refactoring Behaviour (RQ3): 990 model-pass outputs that reduced detected smells underpin the RQ3 analysis of refactoring behaviour and smell-reduction associations.The analysis reports model-level behaviour and correlations with absolute and relative smell reduction.
  • C. Refactoring Behaviour (RQ3): GPT-5.5 produces the most compact edits, Gemini 3.1 deletion-heavy edits, and Opus 4.8 extraction-heavy edits.Opus 4.8 has the largest number of method extractions and highest churn rate, while Gemini 3.1 has the largest line and method removal.
  • C. Refactoring Behaviour (RQ3): Larger edits remove more smells in absolute counts but not a larger proportion of the original smell count.Total churn, lines added, and lines removed correlate positively with absolute reduction but negatively with relative reduction.
  • C. Refactoring Behaviour (RQ3): Edit volume, removal, and extraction are the main behaviours associated with smell reduction, whereas duplicate-code removal, method addition, and class splitting show limited association.Method renaming differs in total counts, but its per-pass model difference is not statistically significant.

D. Contextual Direct-Prompt Baseline · V. DISCUSSION · A. Targeted Code-Smell Reduction Is Not General Quality Improvement

REFINE delivers substantially stronger smell reduction than direct prompting while producing smaller edits and fewer public-method removals, but it does not dominate all preservation indicators. Across models, targeted smell reduction is measurable yet does not reliably imply broader software-quality improvement.

  • D. Contextual Direct-Prompt Baseline: 100.0% vs. 20.8% (p < .001) median total smell reduction favors REFINE over direct prompting on the matched 150-file subset.The comparison yielded 450 file-model pairs.
  • D. Contextual Direct-Prompt Baseline: REFINE preserves public signatures more often, removes fewer public methods, and produces smaller edits than direct prompting.These outcomes indicate a more controlled candidate-generation footprint.
  • D. Contextual Direct-Prompt Baseline: Direct prompting preserves critical assert/fail constructs more often than REFINE, showing that REFINE does not dominate every preservation indicator.Stronger smell reduction can therefore coexist with preservation risks.
  • D. Contextual Direct-Prompt Baseline: The baseline comparison uses 450 matched file-model observations, with major and critical smell reductions evaluated on smaller samples.Major smell reduction uses n = 396, and critical smell reduction uses n = 69; binary outcomes use McNemar tests and numeric outcomes use Wilcoxon signed-rank tests.
  • A. Targeted Code-Smell Reduction Is Not General Quality Improvement: Across all three LLM configurations, REFINE reduces total detected code-smell counts, with the largest reductions observed for major code smells.The result is strongest when the target is explicit, operationalised through static analysis, and re-checked after generation.
  • A. Targeted Code-Smell Reduction Is Not General Quality Improvement: Code-smell reduction is not a proxy for general software-quality improvement because maintainability, testability, Halstead effort, Lines of Code, LCOM, and Mean Method Length change inconsistently.These indicators show mixed, adverse, or non-significant changes depending on the model and system.
  • A. Targeted Code-Smell Reduction Is Not General Quality Improvement: LLM refactoring should be evaluated multi-objectively across smell reduction, edit footprint, and preservation risk.The matched baseline shows stronger REFINE smell reduction with smaller edits and fewer public-method removals, while direct prompting better preserves critical assert/fail constructs.

B. Preservation Evidence Exposes Residual Refactoring Risks · C. Model Choice Shapes Refactoring Behaviour and Risk · D. Evidence-Aware Workflows Provide Control but Not Autonomy

REFINE provides evidence-guided control over file-level refactoring candidates, but preservation checks expose residual behavioural risks and do not establish equivalence. Model choice changes edit profiles and risk tradeoffs, while repository-level adoption still requires stronger validation and human review.

  • B. Preservation Evidence Exposes Residual Refactoring Risks: Public methods, exception handling, framework contracts, and conditionals are often preserved, but assert/fail-call preservation is weaker and public-method removal is the clearest diagnostic risk.Preservation remains distinct from code-smell mitigation.
  • B. Preservation Evidence Exposes Residual Refactoring Risks: Static preservation checks provide risk evidence rather than proof of behavioural equivalence, because candidates may alter assertions, control flow, public methods, or framework interactions.These checks expose residual risks for review.
  • C. Model Choice Shapes Refactoring Behaviour and Risk: GPT-5.5 produces comparatively compact edits, Gemini 3.1 has a deletion-heavy profile with stronger method removal and net LOC reduction, and Opus 4.8 performs substantially more method extractions.All three evaluated configurations reduce detected code smells, but their edit profiles differ.
  • C. Model Choice Shapes Refactoring Behaviour and Risk: Deletion-heavy edits can increase API-surface risk, extraction-heavy edits require review of greater structural change, and compact edits may yield fewer secondary structural gains.The model-specific profiles therefore create different practical tradeoffs.
  • D. Evidence-Aware Workflows Provide Control but Not Autonomy: REFINE adds value over direct prompting for the matched subset in code-smell reduction, public-signature preservation, public-method removal, and edit footprint, but the comparison is contextual rather than universal.The baseline compares REFINE with one direct-prompt strategy on the same files and LLM configurations.
  • D. Evidence-Aware Workflows Provide Control but Not Autonomy: Neither direct prompting nor REFINE should be treated as producing completed behaviour-preserving refactorings, and direct prompting preserves some critical assert/fail constructs more often in these results.LLM-based refactoring can still produce syntax errors or functionality-changing edits.
  • D. Evidence-Aware Workflows Provide Control but Not Autonomy: REFINE creates a controlled evaluation object by separating code-smell detection, planning, transformation, verification, and reporting.Broader validation evidence includes compilation, test execution, and automated refactoring detection.
  • D. Evidence-Aware Workflows Provide Control but Not Autonomy: Because the evidence is file-level, repository adoption requires project context, dependency analysis, compilation, regression testing, and human-in-the-loop acceptance.REFINE does not measure ripple effects across dependent files, callers, tests, build configurations, or framework execution paths.

E. Implications for Research and Practice … C. Conclusive Summary

REFINE frames LLM refactoring as evidence-guided generation of traceable Java file-level candidates rather than autonomous, behaviour-preserving refactoring. Its implications emphasize richer evaluation, project-specific validation, and clear limits relative to repository- and architecture-level work.

  • E. Implications for Research and Practice: Future evaluations should report edit behaviour, preservation risk, and structural side effects alongside aggregate code-smell reduction.Suggested metrics include churn, method extraction or removal, API-surface change, static preservation proxies, and direct-prompt baselines.
  • E. Implications for Research and Practice: REFINE provides a traceable way to generate and inspect Java file-level refactoring candidates, but it is not an autonomous refactoring system.Generated candidates require project-specific validation before integration.
  • E. Implications for Research and Practice: Model choice may depend on maintenance goals, with GPT-5.5 associated with compact edits, Gemini 3.1 Pro Preview with aggressive size reduction, and Opus 4.8 with decomposition-oriented refactoring.Gemini 3.1 Pro Preview also requires stronger API review.
  • VI. RELATED WORK — A. Automated and LLM-Based Refactoring: Prior refactoring research spans tool-supported transformations, opportunity identification, process automation, and maintainability-oriented quality metrics.Code smells are widely used as maintainability indicators, while prior reviews caution against treating smell removal as complete measurement.
  • VI. RELATED WORK — A. Automated and LLM-Based Refactoring: Recent studies examine LLMs for refactoring recommendations, code-quality improvement, and code transformation, including evaluations using smells, static quality metrics, and unit-test success.The supplied passage also reports work on identifying refactoring opportunities and recommending solutions, while noting that LLM suggestions can have limitations.
  • VI. RELATED WORK — B. Agentic and Repository-Level Refactoring: Agentic refactoring approaches structure LLM work through planning, generation, validation, and repair, with RefAgent using specialized agents for planning, execution, testing, and iterative refinement.RefAgent compares multi-agent, single-agent, and non-agentic alternatives and reports improvements in unit-test pass rate, smell reduction, and quality attributes.
  • VI. RELATED WORK — B. Agentic and Repository-Level Refactoring: Repository-level and architectural studies show that realistic evaluation requires repository context, while compound refactorings and cross-module design understanding remain difficult for current LLM agents.SWE-Refactor validates developer-written Java refactorings through compilation, testing, and automated detection; SmellBench examines architectural smell repair and net codebase impact.
  • C. Conclusive Summary: REFINE studies evidence-aware file-level code-smell mitigation and connects smell reduction with quality indicators, preservation proxies, diagnostics, and model-specific behaviour across 450 Java files, 15 systems, and 1,350 model-pass outputs.Its contribution is a controlled characterization of workflow outputs and remaining preservation and integration risks, not a general superiority claim over other approaches.

VII. THREATS TO VALIDITY · VIII. CONCLUSION

REFINE’s findings are bounded by measurement, implementation, sampling, statistical, baseline, and behavioural-validity limitations, supporting conclusions about controlled Java file-level candidate evaluation rather than complete or deployment-ready refactoring. The workflow shows promise over the matched direct-prompt baseline, but candidates still require repository-level validation and human review.

  • VII. THREATS TO VALIDITY: PMD-detected smell reductions do not establish complete removal of all design smells or manually validated Fowler-style instances.The study operationalises code smells through PMD under the selected ruleset.
  • VII. THREATS TO VALIDITY: REFINE’s results may vary with its ruleset, prompts, retry policy, verification thresholds, evidence gates, model behaviour, service updates, and routing infrastructure.The configurations shared orchestration and verification rules, but the study used an artifacts-only protocol.
  • VII. THREATS TO VALIDITY: Controlled candidate evaluation is not full project integration, so reported outcomes should not be treated as evidence from complete repository-level deployment.The artifacts-only protocol means the evaluation reflects controlled candidate evaluation rather than full project integration.
  • VII. THREATS TO VALIDITY: The evaluation covers Java file-level refactoring in 15 subject systems and may not generalise to other languages, settings, file types, or repository-level context.The sample excludes clean files, test code, generated code, and other maintenance tasks, while models receive file-level rather than full-repository context.
  • VII. THREATS TO VALIDITY: Model-pass and file-level outcomes may be affected by nested files, non-independence, and larger or more smell-dense files, despite statistical mitigations.The study reports model-wise and system-level profiles, non-parametric tests, multiple-comparison adjustments, and effect sizes; static improvement is not proof of practical main…
  • VII. THREATS TO VALIDITY: The matched direct-prompt baseline supports comparisons only for its 150-file subset and one strategy, not claims against all alternative workflows or tools.It does not cover all prompt designs, single-agent workflows, non-agentic pipelines, traditional refactoring tools, or component ablations.
  • VII. THREATS TO VALIDITY: Static preservation checks do not replace compilation, regression testing, dependency analysis, call-graph analysis, or human review before adoption.Candidates can reduce detected smells and pass configured checks while still introducing behavioural, integration, or design-level risks.
  • VIII. CONCLUSION: REFINE evaluated 450 Java files from 15 open-source systems, produced 1,350 model-pass outputs, and showed stronger overall smell reduction, smaller edits, and fewer public-method removals than the matched baseline.Direct prompting preserved critical assert/fail constructs more often, while REFINE remains a promising approach requiring validation before repository-level or production adoption.
Loading 2608.23611v1…