Source-linked AI summary
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na
TL;DR
Whole-repository migrations are expensive and existing behaviour-only benchmarks cannot establish that a migration occurred rather than an untouched implementation passing tests. The paper introduces SWE Refactor Bench and a three-stage protocol combining migration audits, fixed behavioural checks, and agentic verification. Across 520 evaluations, only 28 runs pass all three stages, while performance varies substantially across migration categories and the benchmark panel limits how broadly difficulty can be generalized.
Problem
Existing benchmarks measure behavioural correctness but cannot verify replacement of the original stack, allowing untouched repositories to earn perfect scores.
Method
SWE Refactor Bench evaluates 20 whole-repository migrations with Migration Audit, 130,118 fixed behavioural checks, and six independent agents generating differential tests.
Results
Only 28 of 520 evaluations (5.4%) pass all three stages, and 13 of 20 tasks receive no accepted solution.
Takeaways & Limitations
Migration completeness and behavioural correctness are distinct abilities, making SWE Refactor Bench a testbed for reliable whole-repository migration agents.
Takeaways & Limitations
Accepted submissions reflect survival against this fixed panel of adversaries and 20 tasks, rather than proof of intrinsic migration correctness or universal task difficulty.
Abstract
from arXiv · showhide
Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ($5.4\%$) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$. Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.
1 Introduction
SWE Refactor Bench addresses the inability of behaviour-only benchmarks to verify whole-repository migrations by combining migration auditing with behavioural and adversarial testing. Across 20 migrations, it exposes substantial capability gaps and distinct failure modes in current coding agents.
- Problem: Existing behavioural benchmarks cannot distinguish a completed migration from an untouched repository because both can receive perfect test scores.This shortcut is called Blindness: fixed tests observe preserved behaviour, not replacement of the original stack.
- Evaluation protocol: SWE Refactor Bench evaluates migrations in three stages: Migration Audit, Behavioural Tests, and Agentic Verification.The protocol checks that replacement occurred, requires 130,118 fixed checks to pass, and uses six independent agents to generate differential tests for hidden differences.
- Benchmark: The benchmark contains 20 real open-source whole-repository migrations spanning language, framework, platform, and build toolchain technical debt.Agents receive 6 to 30 hours per task, and the study evaluates 8 frontier models across 26 model–effort configurations and 520 runs.
- Results: Only 28 of 520 runs (5.4%) pass all three stages, while 13 of 20 tasks receive no accepted solution.The best configuration, claude-opus-5 at xhigh effort, scores 47.0/100.
- Results: Migration completeness and behavioural correctness are distinct abilities: 30 runs skip migration, 252 complete it but break behaviour, and agentic verification rejects 60 of 88 fixed-suite survivors.Only 28 runs survive all three stages, showing that passing one evaluation condition cannot substitute for the others.
2 SWE Refactor Bench: Benchmark Design
SWE Refactor Bench defines migration as replacing a repository’s stack while preserving observable behaviour, then evaluates both properties separately. Its tasks use real infrastructure and a hidden three-stage protocol designed to prevent untouched implementations from earning behavioural credit.
- Task formulation: A migration task transforms a working repository from source stack ΣA to target stack ΣB while preserving its observable interface.The repository must build on ΣB, with ΣA absent from the repository and build closure, while observations remain unchanged.
- Why behavioural tests fail: Behaviour-only tests are blind because the untouched original already passes every test derived from its own observations.Returning the original repository can therefore earn a perfect behavioural score while satisfying none of the migration condition.
- Task formulation: The benchmark selects load-bearing migrations with externally used interfaces and runnable reference repositories, rather than superficial exercises.Examples include language, framework, platform, and build-toolchain changes in infrastructure such as SQLite, zlib, libsodium, and GraphHopper.
- Task formulation: The agent receives the original repository, target-stack declaration, instruction, offline image, and artifact contract, then submits its final working tree for evaluation.The agent must arrange the repository rewrite and build process itself within the supplied environment.
- Hidden evaluation: The hidden evaluation applies Migration Audit, all-or-nothing Behavioural Tests, and Agentic Verification in sequence.Migration Audit checks that the old stack disappeared; Behavioural Tests compare 130,118 fixed checks against the original; six independent agents search for remaining differences.
3 Experiments
The experiments ask how far current coding agents progress on whole-repository, behaviour-preserving migrations and whether the evaluation protocol remains reliable. The section therefore examines both agent performance and the protocol’s ability to distinguish migration outcomes.
- Experimental questions: The experiments measure how far today’s agents get on whole-repository behaviour-preserving migration and whether the evaluation itself holds up.These are the two questions organizing the experimental analysis.
3.1 Setup and Metrics
The setup evaluates eight frontier models across 20 tasks and 26 configurations, while reporting separate metrics for migration, fixed-test correctness, acceptance, breakage, and blindness. Table 2 organizes the 520 runs by the stage at which they stop.
- Setup: 8 frontier models run all 20 tasks across 26 configurations, producing 520 scored runs.Each run is an independent attempt in a fresh task container under the task’s time budget and without network access beyond the model endpoint.
- Metrics: The benchmark reports counts alongside its composite score because counts state what happened to each submission before weighting.The metrics distinguish migration, perfect fixed-test performance, acceptance, verifier-detected breakage, and blindness.
- Funnel outcomes: Table 2 partitions each row’s 20 runs into five mutually exclusive outcomes and marks the stage where each run stops.The table distinguishes Migration Audit failures, Behavioural Tests failures, Agentic Verification failures, accepted runs, and blindness.
- Metrics: Blindness denotes runs that pass every fixed check but fail Migration Audit because the repository was not actually migrated.This outcome would appear perfect under fixed checks alone but receives no solution credit under the three-stage protocol.
3.2 Overall Performance
Across 520 runs, the three-stage funnel accepted few migrations, with substantial differences across models and migration categories.
- Model performance: 5 acceptances in 20 runs and a score of 47.0 made claude-opus-5 the strongest configuration, while 160 strongest-configuration runs produced only 14 acceptances.gpt-5.6-sol scored 28.5, kimi-k3 19.5, and claude-sonnet-5 15.0.
- Evaluation effects: Three models had no accepted runs even though each produced submissions that passed every fixed check.Under fixed checks alone, these models would have appeared to achieve perfect scores, but later stages accepted none.
- Overall funnel: 340 runs passed Stage I, 118 passed every fixed check, 88 passed both stages, and 28 survived all six verifiers.These correspond to 65.4%, 22.7%, and 5.4% for the first, second, and final counts respectively.
- Overall funnel: 13 of the 20 tasks were never solved by any model, despite the mean score reaching 79.43 among runs that reached Stage III.The mean score across all 520 runs was 13.44/100, showing that reaching the final stage was itself difficult.
- Migration categories: Build toolchain rewrites had the highest Stage I pass rate and mean score, while framework rewrites contributed 14 of the 28 accepted runs.Build toolchain rewrites recorded an 80.8% Stage I pass rate and a 31.4 mean score.
3.3 Analysis of Agent Behaviour
Agent runs reveal separate failure modes: some preserve behaviour without migrating, while many genuine migrations introduce behavioural errors or hidden differences.
- Getting the code right and the migration done: 30 runs passed every fixed check without migrating, whereas 252 completed the migration but broke behaviour.Stage II alone would reward doing nothing, while Stage I alone would reward damaging the system.
- Getting the code right and the migration done: On lang04, 20 of 26 runs passed Stage I but none passed every fixed check; on lang01, 5 passed every fixed check but all five were blindness.The lang01 submissions passed seven of eight audit criteria but transliterated the original implementation rather than redesigning ownership and memory management.
- Getting the code right and the migration done: The four submission types include unchanged copies, forwarding wrappers, partial rewrites, and genuine rewrites that change behaviour.The first two can pass behavioural tests, so mechanism-sensitive auditing is required to reject them.
- The last 1%: Among 340 migration-complete runs, 58% reached 99% of fixed checks but only 26% reached 100%.Only 26% made no fixed-suite error, and 35 of 123 runs at 99.9% were eliminated by the final step.
- The last 1%: Among 88 submissions with no fixed-check errors, 60 were rejected by verifiers, with counterexamples appearing after a median of 17.0 minutes.Only 28 survived all six verifiers; the average submission resisted 3.94 of 6 verifiers.
- Agent capability by migration category: Build toolchain rewrites scored 31.4 overall, while language rewrites scored 5.6, but category bottlenecks occurred at different evaluation stages.Stage III was the main bottleneck for build toolchain and platform rewrites, whereas Stage II was the main bottleneck for framework and language rewrites.
3.4 Analysis of Benchmark Validity
The benchmark’s three stages test distinct failure modes: fixed tests find anticipated behavioural differences, while agentic verification searches for hidden ones. Validation evidence indicates that majority-voted audit decisions are stable, but verifier strength and panel composition materially affect acceptance.
- Stage I validity: 96.3% of 3,536 criterion verdicts had unanimous agreement across three judge samples.Only 131 verdicts split 2:1, despite Stage I’s hard-gate design.
- Stage I validity: Majority voting raised Stage I passes from a potential 305 to 340 by preventing single dissenting samples from deciding disputed criteria.Among the 340 Stage I passes, 35 had at least one criterion decided 2:1.
- Stage I validity: Human researchers agreed with the judge on 89.7% of 156 runs, and downstream stages rejected nearly all disagreement cases.The remaining potentially undercounted cases were at most two runs that passed behavioural checks but were zeroed by Stage I alone.
- Stages II and III: Stage II checks author-anticipated observations, whereas Stage III exposes implementation-specific differences that fixed tests did not anticipate.Examples include divergent Content-Type parsing, timestamp parsing, and WAL cleanup behaviour.
- Stages II and III: An executable counterexample makes an unforeseen behavioural difference re-checkable rather than leaving it as an unverifiable verifier verdict.Both stages are decisive because one failed check can establish that the migration is incomplete.
- Verifier strength and diversity: The two claude-opus-5 verifiers broke 55.7% and 53.4% of submissions, compared with 21.6%–26.1% for the other four.Changing model configuration altered break rates by about two points, while changing model altered them by about thirty.
- Verifier strength and diversity: Retiring the two strongest verifiers would increase accepted submissions from 28 to 46, showing that results depend partly on the verifier panel.The authors therefore characterize acceptance as survival against the strongest available adversaries, not proof of correctness.
- Verifier strength and diversity: The verifiers are complementary: even the lowest-break-rate verifier found a counterexample missed by the other five.Exclusive breaks were observed for both opus verifiers and collectively among the weaker four.
4 Related Work
SWE Refactor Bench addresses a gap in repository-level benchmarks: fixed behavioural tests cannot establish that a migration occurred. It combines a non-behavioural migration veto with model-based differential testing to close that evaluation gap.
- Repository-level coding benchmarks: Repository-level benchmarks traditionally use a red-to-green signal in which a failing test passes after the patch.This paradigm has expanded across languages, refreshed task streams, library construction, and test-writing tasks.
- Code migration and reward hacking: Whole-repository migrations invalidate that signal because the starting repository already passes its fixed tests.An unchanged repository can therefore receive the same perfect behavioural score as a correct migration.
- Code migration and reward hacking: Behaviour-only scoring can say at most “nothing was broken,” never “something was changed.”The benchmark frames this as Blindness rather than loophole exploitation: the rules cannot observe whether migration occurred.
- Code migration and reward hacking: The remedy is a second check outside behaviour that can veto submissions, because additional behavioural tests cannot close the observability gap.Stage I supplies the non-behavioural check, while Stage III searches beyond author-anticipated tests.
- Model-based judging and differential testing: Model judging and differential testing provide established tools for the benchmark’s migration audit and search for hidden behavioural differences.The protocol narrows model-judge questions, uses repeated samples, and measures consistency to address known judging failure modes.
5 Conclusion
SWE Refactor Bench evaluates 20 long-horizon repository migrations with three complementary stages that separately test migration completion, fixed behavioural preservation, and hidden differences. Across 520 evaluations, accepted solutions were rare and failures exposed distinct migration and behaviour abilities.
- Conclusion: The benchmark contains 20 whole-repository migrations from real open-source infrastructure and uses Migration Audit, Behavioural Tests, and Agentic Verification.The stages require migration evidence, 130,118 fixed checks, and executable counterexamples from six coding agents.
- Conclusion: Only 28 of 520 evaluations (5.4%) passed all three stages, and 13 of 20 tasks were solved by nobody.The accepted set therefore remained small across the evaluated models and configurations.
- Conclusion: Migration completion and behavioural preservation were distinct abilities: some runs skipped migration, while most attempted migration and broke behaviour.These failure modes were stopped at Migration Audit and Behavioural Tests, respectively.
- Conclusion: Among runs that genuinely completed the migration, only 26% passed every fixed behavioural check, and many survivors still had agent-found counterexamples.The protocol therefore tests both anticipated and hidden behavioural differences after migration.
A How a Task Is Built
Tasks are built around load-bearing, overdue migrations with explicit stack requirements and hidden behavioural checks. The benchmark verifies task validity through identity runs, source-based audits, bounded verifier scopes, and a reference example exposing disagreement between migration and behaviour stages.
- Task construction: Each task starts from a buildable repository and specifies a target stack, instruction, offline image, artifact contract, and fixed behavioural expectations.The fixed tests are kept outside the agent’s container so they are unavailable during task execution.
- Task construction: Task selection begins with technical debt a maintainer would consider overdue, then chooses a repository where the old stack is load-bearing.This makes the migration reach the repository’s design rather than merely changing imports.
- Task construction: Instructions state requirements and build constraints but omit concrete behavioural checks, preventing agents from optimizing directly for the fixed suite.The agent must inspect, plan, rewrite, build, and validate the target-stack repository itself.
- Scoring and verification: Source-based scoring excludes generated and workspace artifacts, making removal of the old stack a property of source rather than leftover directories.Excluded paths include build outputs, target directories, object files, install prefixes, and .git.
- Scoring and verification: Stage III allow and deny lists must match the task instruction so verifier scope is neither unwinnable nor unloseable.For lang01, the verifier has fifteen allowed and eleven forbidden directions.
- Scoring and verification: Identity runs score State A against itself at image-build time, and the build is refused unless every behavioural module reaches 1.0.This makes the benchmark’s reference behavioural expectations self-validating.
- Scoring and verification: A lang01 submission passed all 4,184 checks across 15 modules yet failed one of eight migration criteria because it reproduced the C implementation layout in Rust.The example demonstrates why behavioural success cannot substitute for migration audit evidence.
B Per-Task Breakdown
The per-task breakdown shows that most tasks remain unsolved, either because agents fail behavioural checks or because behaviour-preserving submissions are rejected as Blindness.
- Evaluation funnel: Table 9 decomposes each of the 26 runs into migration completion, fixed-check success, Blindness, and final acceptance.This task-by-task funnel corresponds to the evaluation stages described in Section 3.2.
- Task outcomes: 13 of 20 tasks were never solved by any model.Seven tasks had no submission reach the verifiers.
- Failure modes: Three tasks had no run passing every fixed check, while four had behaviourally passing runs rejected for Blindness.The latter tasks were lang01, fw01, fw02, and fw07; the former were lang03, lang04, and pf02.
C Detailed Task Catalogue
The detailed task catalogue covers all 20 tasks and reports repository-scale metadata, including source size, time budgets, and migration criteria.
- Catalogue structure: Table 10 presents the task set task by task, while Table 1 summarizes it by migration class.The catalogue is the detailed counterpart to the class-level summary.
- Repository scale: The repositories collectively contain 10,594 version-controlled files, measured using consistent inclusion and exclusion rules.The LoC definition excludes tests, vendored trees, and documentation.
- Task requirements: Each task specifies an agent time budget and migration criteria, with any single failed criterion zeroing the submission.The catalogue records B as the time budget in hours and Criteria as the number of required prompt-form criteria.