Source-linked AI summary
SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring
Yisen Xu, Jinqiu Yang, Tse-Hsun, Chen
TL;DR
Existing refactoring benchmarks provide limited scenario coverage, may mix refactoring with unrelated changes, and often lack repository-level context. SWE-Refactor addresses these gaps with a validated Java benchmark and evaluates nine LLMs, finding that complex and compound refactorings remain especially challenging, with GPT-5.1-Codex achieving 39.4% success on compound instances.
Problem
Existing refactoring benchmarks have limited scenario coverage, may include unrelated changes, and provide insufficient repository-level context for realistic evaluation.
Method
SWE-Refactor constructs a benchmark of pure developer-written Java refactorings using automated extraction, filtering, repository-level information enrichment, compilation, and test verification.
Results
Models generally perform better on simpler atomic refactorings than on complex and compound refactorings; GPT-5.1-Codex achieves 39.4% success on compound instances.
Takeaways & Limitations
SWE-Refactor provides released data, evaluation code, and results for realistic research on LLM-based code refactoring.
Takeaways & Limitations
The benchmark currently covers only Java projects and method-level refactorings, while its scale remains limited for comprehensive evaluation or fine-tuning.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have recently attracted wide interest for tackling software engineering tasks. In contrast to code generation, refactoring demands precise, semantics-preserving edits that improve program structure, which also makes automated evaluation challenging. However, existing refactoring benchmarks commonly suffer from three shortcomings: limited coverage of refactoring scenarios, the inclusion of instances that mix refactoring with unrelated changes, and insufficient repository-level context for realistic assessment. To mitigate these issues, we introduce SWE-Refactor, a new benchmark for LLM-based code refactoring. SWE-Refactor comprises 1,099 developer-written, behavior-preserving refactorings mined from 18 Java projects, including 922 atomic and 177 compound instances. Each instance is validated via compilation, test execution, and automated refactoring detection tools to ensure correctness. We evaluate nine widely used LLMs on SWE-Refactor, covering models such as GPT-4o-mini, DeepSeek-V3, and CodeLLaMa, to provide representative reference results. Our results show that complex and compound refactorings remain the primary source of failures; notably, an OpenAI Codex agent achieves only 39.4% success on compound instances. We release SWE-Refactor and all evaluation results to facilitate future research on LLM-based code refactoring.
1. Introduction
Refactoring requires repository-level reasoning and precise behavior-preserving edits, while existing benchmarks provide limited, noisy, and insufficiently realistic evaluation settings. SWE-Refactor addresses these gaps with a verified Java benchmark covering atomic and compound refactorings and evaluates nine LLMs.
- Motivation: Refactoring requires analyzing existing code and dependencies before applying precise transformations that preserve behavior.It also challenges agentic workflows through iterative planning, coordinated edits, and repeated verification.
- Benchmark gaps: Existing benchmarks often underrepresent compound refactorings, which require multiple coordinated transformations common in real development.They also involve manual construction, limited repository-level information, missing tests, or unrelated functional changes.
- Benchmark gaps: 95.6% of recent software engineering benchmarks are built exclusively on Python, limiting language diversity and representativeness.SWE-Refactor complements these benchmarks by evaluating refactoring on Java projects.
- SWE-Refactor: 1,099 pure refactorings from 18 widely used Java projects form SWE-Refactor, covering three atomic and three compound refactoring types.The benchmark uses developer-written changes and excludes unrelated modifications.
- SWE-Refactor: Each retained instance passes AST-based analysis, compilation, full test execution, and manual inspection of sampled cases.These checks target refactoring purity and behavioral equivalence.
- Evaluation: Nine LLMs are evaluated for functional correctness and human-likeness using compilation and tests, AST verification, and CodeBLEU.CodeBLEU is used only for human-likeness because it primarily measures surface-level similarity.
- Results: 39.4% is the reported success rate of OpenAI Codex on compound refactorings, which remain especially challenging alongside other complex transformations.Larger general-purpose models tend to outperform smaller open-source models overall.
2. Related Work
Prior refactoring benchmarks vary in construction and coverage, but commonly rely on synthetic or manually prepared data and omit aspects of realistic repository complexity. SWE-Refactor is positioned as an automatically built benchmark from developer-written Java commits covering atomic and compound refactorings.
- Refactoring benchmarks: RefactorBench uses LLMs to identify refactoring opportunities, whereas SWE-Refactor mines developer-written refactorings from real commits.SWE-Refactor therefore captures project features such as overridden methods, generics, exception handling, and inheritance.
- Benchmark construction: RefactorBench manually writes ground-truth solutions, and ref-Dataset manually reconstructs pre-refactoring code, creating time-consuming construction steps.These examples illustrate the varied manual processes used in prior benchmark creation.
- Refactoring benchmarks: Existing datasets differ substantially in scale and scope, including 100 pure atomic Java refactorings, 122 Extract Method instances, and 1,752 extended Extract Method instances.Each benchmark has limitations, while SWE-Refactor covers both atomic and compound refactorings from 18 modern Java projects.
- LLM-based refactoring: Recent refactoring studies explore prompt clarity, structured prompting, few-shot learning, hybrid rule-based systems, and direct prompting of models such as GPT-4.These works focus on techniques for improving LLM performance on refactoring tasks.
3. SWE-Refactor
SWE-Refactor represents each refactoring with the target operation, repository and code structure, developer-written reference code, build configuration, and test coverage. These components support evaluation of correctness and human-likeness in realistic repository contexts.
- Dataset components: Each SWE-Refactor sample contains six components describing the target method, refactoring, repository context, reference code, build configuration, and test coverage.Together, these fields provide inputs and verification information for repository-level refactoring evaluation.
- Dataset components: The refactoring-type field identifies operations such as Extract and Move Method, which combines extraction into a separate method with movement to another class.This explicitly represents compound transformations rather than only single operations.
- Dataset components: Developer-written refactored code serves as the reference for evaluating generated code quality.Build configuration records the commit, compatible JDK, and build commands, while test coverage shows how the target method is exercised.
- Evaluation design: Given a target method, refactoring type, and repository and source-code information, SWE-Refactor evaluates whether generated code is correct and human-like.The described evaluation uses compilation and tests, AST-based refactoring verification, and a human-likeness measure.
1 Compilation and Test success (Functional Verifica-
SWE-Refactor integrates generated code into the project, then compiles the project and runs its test suites to assess functional correctness.
- Functional Verification: Compilation and full test execution verify that generated refactored code preserves project functionality without breaking the build or introducing unexpected issues.The generated code is first integrated into the project before these checks are performed.
2 AST-Based Refactoring Verification (Refactoring Ver-
SWE-Refactor combines AST-based detection with repository-level structural analysis and compilation/test verification to construct and assess refactoring instances. Its evaluation also uses CodeBLEU to measure similarity to developer-written refactorings.
- Evaluation metrics: CodeBLEU measures human-likeness by comparing generated code with developer-written refactorings across textual, structural, and semantic dimensions.The metric is used alongside functional correctness and AST-based verification rather than as a correctness guarantee.
- Automated construction: RefactoringMiner detects refactoring types and extracts affected files, line numbers, and commit-level locations for each instance.The extracted locations support subsequent AST construction and structural information collection.
- Automated construction: PurityChecker filters extracted commits to retain only pure instances among the six targeted refactoring types.It uses specialized AST analysis to identify changes that follow predefined purity rules.
- Repository-level enrichment: Eclipse JDT enriches each instance with repository, class, and method-level information, including class hierarchies and caller-callee relationships.The analysis resolves types and method references across source files within the same package.
- Functional verification: Compilation and test execution verify refactoring correctness, while JaCoCo excludes instances whose refactored code is not exercised by tests.The pipeline attempts multiple JDK versions before running the project test suite and collecting coverage information.
4. Experiment
SWE-Refactor evaluates nine LLMs across six Java refactoring types using repository-level prompts and multiple correctness metrics. Results show stronger performance for general-purpose models and multi-agent workflows, while cross-file and compound transformations remain difficult.
- Evaluation Setup: The evaluation covers nine LLMs across three atomic and three compound refactoring types.The benchmark includes Extract Method, Move Method, Inline Method, Extract+Move, Move+Inline, and Move+Rename.
- LLMs’ Performance on SWE-Refactor: DeepSeek-V3 achieves 457 successful refactorings (41.58%), followed by GPT-4o-mini with 438 (39.85%).Success requires both Compilation and Test Success and AST-Based Refactoring Verification.
- Performance across Refactoring Types: DeepSeek-V3 leads Extract Method with 301 successes, while GPT-4o-mini generalizes more broadly to Move Method and Extract+Move.GPT-4o-mini records 92 Move Method and 33 Extract+Move successes.
- Performance across Refactoring Types: The comparison identifies local atomic edits as easier than cross-file and compound transformations.These harder transformations are presented as important tests of repository-level reasoning over structured software artifacts.
- Context Augmentation and Multi-Agent Workflows: Multi-Agent achieves 579 successful refactorings, outperforming RAG (451) and Simple Prompting (438).The advantage is especially pronounced for Move Method and Extract+Move cases.
- Scalability to Agentic Scaffolding: GPT-5.1-Codex solves 151/200 instances (75.5%) overall, but only 13/33 (39.4%) compound instances.Most failures involve partial compound edits or alternative transformations that do not satisfy the requested operation.
5. Discussion
The discussion identifies distinct failure modes across model settings and defines the benchmark’s main scope boundaries. SWE-Refactor is Java- and method-level focused, and its dataset remains limited in scale despite broad coverage.
- Error Taxonomy: Small code LLMs mainly fail on prompt-format requirements, while general LLMs still struggle with dependencies and repository-level information.GPT-4o-mini failures include undefined variables, parameter mismatches, and moving methods into nonexistent files.
- Error Taxonomy: Multi-agent failures often overfit to tests, producing outputs that compile and pass tests but fail AST-Based Refactoring Verification.One example is generating empty methods that satisfy compilation and testing without performing the required refactoring.
- Limitations: SWE-Refactor focuses only on Java projects, limiting language diversity while enabling reliable extraction with mature Java analysis tools.The authors plan to extend the benchmark to other languages.
- Limitations: The benchmark targets method-level refactorings because higher-level class refactorings are less frequent and often entangled with bug fixes.The authors plan to broaden the included refactoring types.
- Limitations: Although SWE-Refactor contains 1,099 refactorings from 18 projects, its scale remains limited for comprehensive evaluation or LLM fine-tuning.The dataset is intended to expand in coverage and diversity.
6. Conclusion
SWE-Refactor is a repository-level benchmark built from real-world Java refactorings and designed for realistic evaluation of LLM code-refactoring capabilities. Evaluation of nine LLMs shows that compound refactorings remain especially challenging.
- SWE-Refactor contains 1,099 pure refactorings extracted from 18 diverse Java projects, covering both atomic and compound types.The benchmark uses automated filtering, compilation, and test verification, and includes repository-level information.
- Evaluation of nine widely used LLMs shows that compound refactorings remain a major source of difficulty.
- 39.4% success is achieved by GPT-5.1-Codex on compound instances.
- The authors publicly release the benchmark data and evaluation results for future research on LLM-based code refactoring.
A. Dataset Hosting
The supplementary materials define the benchmark’s refactoring types, project selection, prompts, and retrieval-augmented generation pipeline. They also document repository-level inputs used to support realistic refactoring evaluation.
- Refactoring Type Definitions: The benchmark defines atomic and compound method refactorings using established refactoring descriptions.Defined types include Extract Method, Move Method, Inline Method, and compound combinations such as Extract and Move Method.
- Project Selection: The project corpus comprises 18 Java projects selected for domain diversity, rich development histories, and meaningful refactoring coverage.The projects were previously used in change-history studies and each has more than 2,000 commits.
- Prompt Templates: The prompts provide target code, class content, refactoring operations, call relationships, and project structure as task context.Prompt fields vary by refactoring type and may additionally include file paths and other structural information.
- RAG Construction: The RAG pipeline prepares inputs, generates descriptions, retrieves similar examples using text and embedding similarity, and merges the results.The pipeline uses a retrieval database of pure refactoring examples and supports repository-level context such as callers and callees.
Step 3: Constructing a Searchable Database of Refactoring Examples
The searchable refactoring-example database combines generated natural-language descriptions with code representations. It supports complementary lexical and semantic retrieval through separate indexes.
- Each database entry contains refactoring code and a generated description, indexed for lexical and semantic similarity.
- BM25 ranks examples by token overlap and structural similarity across the combined code and description.
- all-MiniLM-L6-v2 generates vector embeddings that support similarity based on meaning rather than syntax alone.
Step 4: Merging and Reranking the Results
The retrieval stage merges lexical and semantic rankings, reranks the candidates, and selects examples for few-shot prompting. A two-agent workflow then supports iterative code generation and review.
- Merging and Reranking: Reciprocal Rank Fusion combines independent text-based and embedding-based rankings, favoring examples ranked highly by either method.
- Merging and Reranking: A reranking step refines similarity assessment to prioritize examples aligned both lexically and semantically with the query.
- Merging and Reranking: The system selects the top 3 ranked examples as few-shot prompts for generating refactored code.
- Multi-Agent Workflow: The multi-agent workflow uses Developer and Reviewer Agents that collaborate through iterative reasoning and feedback.
- Multi-Agent Workflow: The Developer Agent analyzes source code, generates refactored code, and can improve outputs using repository utilities and reviewer feedback.
Reviewer Agent: Evaluation and Feedback
The Reviewer Agent evaluates generated refactorings with static analysis tools and provides feedback on their validity and needed improvements.
- The Reviewer Agent assesses the quality of generated refactorings.
- It uses refactoring detectors such as RefactoringMiner to analyze whether the code changes constitute valid refactorings.
- It applies style checkers such as Checkstyle to identify code smells or coding-convention violations before generating feedback.