Source-linked AI summary
RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents
Aziz Ben Amor, Drish Mali, Mann Acharya, Vijayasri Iyer, Sébastien Bratières
TL;DR
Repository-scale refactoring lacks controlled infrastructure for testing how agent design choices affect behavior-preserving multi-file changes. RefactorPlatform provides an auditable harness that fixes the environment while varying models, retrieval, prompts, and orchestration, and its 100-task demonstration shows strong structural retrieval gains alongside weaker delegation performance. The platform supports reproducible evaluation, although conclusions remain bounded by the mid-tier models and limited language-paradigm analysis evaluated here.
Problem
Repository-scale refactoring requires behavior-preserving changes across many interdependent files, but existing infrastructure does not isolate the design choices governing agent success.
Method
RefactorPlatform holds the environment fixed while varying model, retrieval, prompting, and orchestration configurations with isolated workspaces, verification gates, and auditable telemetry.
Results
AST-aware retrieval outperforms naive token-window chunking by at least 25pp across prompt modes, while retrieval reaches 86% versus 66% for the evaluated sub-agent configuration on matched tasks.
Takeaways & Limitations
The open-source harness enables controlled, reproducible, and auditable comparison of refactoring-agent configurations across pass rate and cost.
Takeaways & Limitations
The study focuses on mid-tier models and cannot determine whether frontier-scale models resolve the observed structural blindness or multi-agent penalties.
Abstract
from arXiv · showhide
Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, yet to our knowledge no existing harness isolates the design choices that determine agent success on this task. We present RefactorPlatform, an open-source evaluation harness that holds the environment fixed and varies each design axis explicitly: model backbone (via OpenRouter and GitHub Copilot CLI), execution regime (baseline, retrieval-augmented, and multi-agent), and prompt specificity. Each run executes in an isolated workspace with live terminal streaming, per-task logging of tokens, diffs, and transcripts, AST-based verification, and exportable telemetry for audit and reproduction. Demonstrating the platform on 100 multi-file RefactorBench tasks across four model families, we illustrate the analyses it supports: AST-aware chunking outperforms naive token-window chunking by 25-30% across prompt modes, whereas naive retrieval falls below the retrieval-free baseline; a lean retrieval-augmented single agent (86%) beats the sub-agent configuration we evaluated (66%) on matched tasks with no task passing under delegation that fails under retrieval; and retrieval's accuracy gains absorb its token overhead, leaving cost per successful refactoring unchanged. RefactorPlatform is open-sourced to make refactoring-agent evaluation reproducible and auditable.
1 Introduction
Repository-scale refactoring demands behavior-preserving changes across interdependent files, but existing benchmarks and agents provide limited evidence for this setting. RefactorPlatform addresses this gap with controlled, auditable evaluation across retrieval, prompting, orchestration, and model design choices.
- Motivation: Repository-scale refactoring propagates changes across interdependent files while preserving external behavior, making manual propagation error-prone.Function-level benchmarks reveal little about this regime, while repository-scale refactoring requires behavior-preserving multi-file edits.
- Motivation: Current agents exhibit structural blindness because they inspect limited repository context and often retrieve historical snippets instead of discovering live code sites.The paper identifies missed application sites as a central failure mode and notes mixed evidence for adding specialized agents.
- Contribution: RefactorPlatform isolates retrieval, prompting, orchestration, and model routing for repository-scale refactoring under controlled, auditable logging.The harness is designed to address the lack of comparable infrastructure for isolating these design choices.
- Contribution: The harness fixes the environment while varying baseline, retrieval-augmented, and orchestrated multi-agent regimes to study prompting and structural retrieval.Its questions include AST-aware chunking, delegation costs, and retrieval under under-specified prompts.
- Contribution: A 100-task campaign across four model families combines isolated workspaces, AST verification, live streaming, per-task telemetry, and exportable logs for reproducible analysis.Reported analyses include prompt sensitivity, context-layer comparisons, and a best-case pass rate of 86% at $0.13 per success.
2 RefactorPlatform Architecture and Workflow
RefactorPlatform evaluates repository-scale agents through isolated, observable runs whose model, execution, prompt, and dataset choices are explicit. Its verification and telemetry pipeline measures complete task success and cost rather than plausible-looking partial edits.
- Platform workflow: Each task runs in an isolated workspace while the harness records diffs, terminal logs, token usage, and complete interaction transcripts.Persistent Tmux sessions stream execution activity during agent runs.
- Configuration axes: Operators configure evaluations across model selection, execution regime, prompt mode, and target dataset tasks.Models are available through OpenRouter or the GitHub Copilot CLI, while execution regimes and prompt specificity are independently configurable.
- Execution regimes: S1 provides localized terminal tools without external context, whereas S2 adds task-scoped structural retrieval through MCP and hybrid vector-plus-keyword search.S2 uses indexed code chunks, embeddings, cosine similarity, BM25, and Reciprocal Rank Fusion for codebase discovery.
- Prompt modes: Descriptive prompts specify what, where, and how; Base prompts specify what and where; Lazy prompts specify only what.These modes systematically vary instruction clarity for refactoring tasks.
- Monitoring: The dashboard presents task status, execution traces, token and diff metrics, terminal streams, repository views, prompts, tests, and resource controls.Asynchronous workers run configured agents in isolated checkouts and stream telemetry to the interface.
- Execution regimes: S3 enables native sub-agent delegation, which the agent chooses to exercise, while fixed-role external orchestration is a separate configuration.The harness verifies delegation from the agent event stream and distinguishes unexercised setups.
- Evaluation tasks: RefactorBench supplies 100 multi-file Python tasks spanning nine repositories, with edits affecting 2–31 files and success checked by AST-based unit tests.The platform also integrates the Java-centric SWE-Refactor suite.
- Verification: A patch counts as correct only after the relevant verification gate passes: AST tests for Python, and extraction, compilation, tests, and RefactoringMiner checks for Java.CodeBLEU is logged but is not the correctness gate.
3 Results and Empirical Analysis
Across 100 RefactorBench tasks, the evaluation shows that prompt specificity and structural retrieval strongly shape refactoring success, while sub-agent delegation underperforms lean retrieval. Retrieval also absorbs its token overhead through fewer failed attempts, leaving cost per successful refactoring nearly unchanged.
- Prompt specificity: -9pp and an additional -16pp: baseline accuracy fell when prompts shifted from Descriptive to Base and then Lazy modes.The results identify prompt quality as the largest indicator of standalone-agent capability.
- Tooling feedback: +3.7pp: LSP diagnostics produced only a marginal average gain across configurations.The gain was just +2pp in Descriptive mode, indicating that linting feedback did not substitute for detailed task descriptions.
- Structural retrieval: At least 25pp: AST-driven indexing outperformed naive token-window chunking across every prompt mode.AST chunking preserves syntactic boundaries and self-contained functional units, unlike fragmenting token windows.
- Structural retrieval: 9.5pp: S2 improved average performance over S1 across four model families.S1 supports local file navigation, whereas S2 supplies structural context for cross-file reasoning.
- Execution regimes: 86% versus 66%: S2 retrieval outperformed S3 delegation on the same 100 Descriptive-prompt tasks.Against S1, the corresponding comparison was 73% versus 66%.
- Cost efficiency: No more than $0.01: retrieval changed cost per successful refactoring by at most this amount across four evaluated models.Although attempted-task cost increased from $1.07 to $1.23 for deepseek-v4-pro, fewer wasted attempts offset the token overhead.
4 Conclusion
RefactorPlatform is an open-source harness for evaluating repository-scale refactoring agents under controlled, auditable conditions. Its extensible design supports comparisons across models, retrieval, prompting, and orchestration while tracking pass rate and cost.
- The harness provides isolated workspaces, MCP-based retrieval, and auditable per-task logging across model, retrieval, prompting, and orchestration axes.
- Operators can plug in new harnesses and observe their effects on pass rate and cost under controlled conditions.
- RefactorPlatform is designed to extend to additional benchmarks, verification gates, and orchestration regimes.
Limitations
The evaluation is bounded by mid-tier model coverage, primarily Python-focused benchmarks, and single-run campaigns without uncertainty estimates. These boundaries leave frontier-scale models, broader language ecosystems, and repeated-trial evidence for future work.
- Model Scale and Cost: The study focuses on mid-tier models because full-scale codebase refactoring rapidly consumes tokens, leaving frontier-scale effects unverified.
- Testing frontier architectures and mapping the architecture across distinct language paradigms remain important next steps.
- Language and Benchmark Scope: The experiments focus primarily on Python through RefactorBench, while definitive conclusions about Java require a deeper dedicated study.
- Campaign scale: Single-run campaigns under fixed per-task budgets provide pass rates without uncertainty estimates, so reported differences are observed rather than statistically tested.
A SWE-Refactor: Triple-Verification Pipeline
The SWE-Refactor evaluation applies a triple-verification pipeline requiring clean extraction, compilation, and structural alignment. Execution feedback substantially improves compilation and final passing performance in the reported setting.
- Triple-Verification Pipeline: A patch passes the triple-verification pipeline only when it extracts cleanly, compiles, and satisfies RefactoringMiner.
- Triple-Verification Pipeline: 97% extraction for minimax-m3 falls to 34% compilation success, exposing a gap between plausible code and executable code.
- Execution Feedback: Compiler diagnostics raise gpt-5-mini’s compilation rate from 12% to 88% and final pass rate from 20% to 85% (28/33 tasks).
- Execution Feedback: In this setting, compiler feedback outperformed richer prompting and single-pass generation.
B Universal Failure Analysis
A subset of tasks fails across every evaluated configuration, indicating persistent difficulties beyond model size, prompting, retrieval, or context alignment. These failures concentrate in demanding multi-file and diff-generation cases.
- Universal Failure Analysis: 5% of tasks fail in all evaluated configurations regardless of model size, prompt engineering, or retrieval architecture.
- Universal Failure Analysis: Persistent failures point to limitations in multi-file and diff-generation heuristics rather than poor context retrieval or weak prompt alignment.
- Universal Failure Analysis: Tasks requiring downstream test-module updates or deep cross-module parameterization failed uniformly.
C Detailed Failure Taxonomy
The failure taxonomy identifies incomplete reference propagation as the primary repository-scale bottleneck, with failures concentrated in dependent-call-site updates and unresolved Java symbols.
- 50.9% of RefactorBench failures involve localized edits that do not update dependent call sites.The combined partial-completion and missing-import metrics identify incomplete reference propagation as the dominant failure pattern.
- 73.5% of SWE-Refactor failures arise from unresolved package symbols and global import mismatches in compiled Java.The compiled setting makes reference-propagation failures more pronounced than in RefactorBench.
D Granular Deep-Dives & Trace Case Studies
The deep-dive section uses execution histories and benchmark tables to connect granular failure categories with concrete architectural breakdowns across Python and Java evaluations.
- The case studies examine automated execution histories to illustrate recurring breakdown modes and architectural blind spots.They connect localized technical successes with broader systemic failures observed in the evaluations.
- Table 5 reports multi-model outcomes on the SWE-Refactor Java repository matrix, with smaller gpt-5-mini subsets because of API rate limits.The gpt-5-mini subsets contain n = 75 single-pass baseline tasks and n = 33 feedback-loop tasks, versus n = 177 for the full compound subset.
- Table 6 separates granular failure categories for RefactorBench from compilation-pipeline execution failures on SWE-Refactor.This panel structure supports comparing failure mechanisms across the Python and Java-oriented evaluations.