PaperlayerView exact revision on arXiv

Source-linked AI summary

MigrationBench: Repository-Level Code Migration Benchmark from Java 8

Linbo Liu, Xinle Liu, Qiang Zhou, Lin Chen, Yihan Liu, Hoan Nguyen, Behrooz Omidvar-Tehrani, Xi Shen, Jun Huan, Omer Tripp, Anoop Deoras

arXiv:2505.09569v3cs.SE

TL;DR

Repository-level code migration involves interconnected changes across files, but coding benchmarks have mainly covered code generation and issue resolution. MigrationBench introduces a Java 8 migration dataset, evaluation framework, and agentic workflow. Its hybrid approach performs comparably to a purely agentic approach while reducing LLM usage by 11%.

  • Problem

    Existing coding benchmarks primarily focus on code generation and issue resolution, while repository-level migration requires interconnected changes across files.

    Source: §1 · p. 2

  • Method

    MigrationBench provides full and selected Java 8 repository datasets, an evaluation framework, and an agentic migration workflow.

    Source: §1 · p. 2

  • Results

    The hybrid approach achieves performance comparable to the purely agentic approach while reducing LLM usage by 11%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Released agent trajectories document intermediate decisions, tool invocations, and failure modes, and may support reproducibility and training learning-based migration methods.

    Source: §1 · p. 2

    The evaluation relies on existing test cases and does not enforce test coverage, so untested lines may receive only compile-level verification.

    Source: §9 · p. 9

Abstract

from arXiv · show

With the rapid advancement of powerful large language models (LLMs) in recent years, a wide range of software engineering tasks can now be addressed using LLMs, significantly enhancing productivity and scalability. Numerous benchmark datasets have been developed to evaluate the coding capabilities of these models, while they primarily focus on code generation and issue-resolution tasks. In contrast, we introduce a new coding benchmark MigrationBench with a distinct focus: code migration. MigrationBench aims to serve as a comprehensive benchmark for migration from Java 8 to the latest long-term support (LTS) versions (Java 17, 21), including a full dataset and its subset selected with 5,102 and 300 repositories respectively. selected is a representative subset curated for complexity and difficulty, offering a versatile resource to support research in the field of code migration. Additionally, we provide a comprehensive evaluation framework to facilitate rigorous and standardized assessment of LLMs on this challenging task. We further propose an agentic framework and demonstrate that LLMs can effectively tackle repository-level code migration to Java 17. For the selected subset with Claude-4.5-Sonnet, our agentic framework achieves 71.67% and 53.33% success rate (pass@1) for minimal and maximal migration respectively. The dataset and evaluation source code are available at: https://huggingface.co/collections/AmazonScience/migrationbench and https://github.com/amazon-science/MigrationBench respectively.

1 Introduction

MigrationBench addresses the lack of systematic evaluation for repository-level code migration, a task involving interconnected, multi-step changes across files. It introduces a Java 8 migration benchmark, an evaluation framework, and an agentic workflow.

1 Introduction

  • Repository-level migration requires multi-step changes across files to address interconnected issues, motivating evaluation beyond isolated code-generation and issue-resolution tasks.Source: §1 · p. 2
  • MigrationBench provides 5,102 Java 8 repositories in full and a 300-repository representative, challenging subset called selected.Source: §1 · p. 2
  • Its evaluation framework checks minimal migration requirements, approximately validates functional equivalence, and verifies dependency upgrades to latest major versions.Source: §1 · p. 2
  • The Strands-based agent is provided shell access and a migration knowledge base to address code migration tasks.Source: §1 · p. 2
  • The hybrid approach achieves performance comparable to the purely agentic approach while reducing LLM usage by 11%.Source: §1 · p. 2
  • Released agent trajectories record intermediate decisions, tool invocations, and failure modes, and may support reproducibility and training learning-based migration methods.Source: §1 · p. 2

2 Related Work

Prior LLM software-engineering research spans code generation, testing, repair, and issue resolution, while repository-level migration has lacked systematic evaluation. Existing repository translation studies are limited in scale or scope, and some lack ground-truth target repositories.

LLM Agents for Software Engineering

  • There has been no systematic evaluation of LLMs on code migration, motivating an agentic framework for MigrationBench.Source: §2 · p. 3
  • LLM-agent research uses multi-step planning, tool invocation, and environmental feedback for software-engineering tasks, including bug fixing and issue resolution [33] [39] [41] [42] [47].Source: §2 · p. 3

Benchmark Datasets: Code Generation, Maintenance and Translation

  • Code-generation benchmarks assess function-level or multilingual coding, while SWE-Bench and its extensions focus on issue resolution across repositories and languages.Source: §2 · p. 3
  • Prior repository translation work includes Java-to-Python on 10 repositories and Python-to-Java on 100, with neither providing a ground-truth target repository.Source: §2 · p. 3
  • CodePlan studies package migration and code edits using three repositories per use case [4], while FreshBrew includes 228 repositories with a less comprehensive curation and evaluation pipeline [27].Source: §2 · p. 3
  • MigrationBench is described as a benchmark of more than 5,000 repositories accompanied by a robust automated evaluation framework for systematic, reproducible migration evaluation.Source: §2 · p. 3

3 MigrationBench

MigrationBench defines repository-level migration from Java 8 Maven projects to LTS targets and builds curated datasets for evaluating that task. Its 300-repository selected subset emphasizes larger, multi-module projects and has typically larger repository statistics than full.

3 MigrationBench

  • MigrationBench covers Maven-based GitHub repositories migrating from Java 8 to long-term support targets such as Java 17 or 21.The paper describes large-scale curation using AWS EMR Serverless, Amazon Linux 2, and Maven 3.9.6.Source: §3 · p. 3

3.1 Problem Formulation

  • Migration requires a repository that passes verification under Java 8 to be transformed into one that passes under the target language version.The target passing state need not be unique, and repositories with hard-coded compiler versions may require seed changes.Source: §3.1 · p. 3

3.2 Data Collection

  • The study limits Java repositories to Maven, requires latest snapshots to pass `mvn clean verify` unmodified on Java 8, 11, or 17, and narrows the dataset to 16,154 repositories by March 2024.Concurrent Maven throttling may leave the 16k list incomplete.Source: §3.2 · p. 4
  • The full dataset contains 5,102 repositories with unit tests, alongside a 4,814-repository subset without unit tests.The paper primarily discusses full; it retains the subset without unit tests as a benchmark for unit test generation.Source: §3.2 · p. 4

3.3 migration-bench-selected (selected): A Representative and Challenging Subset

  • The selected subset contains 300 repositories curated from full using repository statistics and expert selection, emphasizing larger, multi-module projects.The subset is intended to represent diverse and complex real-world migration challenges while reducing the cost of evaluating all 5,000 repositories.Source: §3.3 · p. 4

3.4 Dataset Statistics

  • Full repositories average 2.4 modules, 7,698.3 cumulative LOC, 65.1 Java files, 8.7 test files, and 34.8 test cases; selected averages, medians, and standard deviations are typically larger.The passage reports these statistics for the full and selected subsets.Source: §3.4 · p. 4

4 Evaluation

Repository-level migration lacks a straightforward correctness check, and exact functional-equivalence testing is challenging. The section operationalizes evaluation through build and test criteria, migration invariants, and dependency-version requirements.

4 Evaluation

  • Repository-level migration lacks straightforward ground-truth evaluation, while comprehensive functional-equivalence checks are challenging across applications of different sizes.The evaluation therefore uses automated approximations rather than attempting exhaustive functional-equivalence verification.Source: §4 · pp. 4–5

4.1 Minimal Migration

  • The base commit is the last candidate snapshot passing Java-version, Maven-command, and compiled-class-version checks, under an assumption of monotonic version histories.The flowchart starts from the repository’s latest snapshot as of March 2024.Source: Figure 1 · p. 6
  • Minimal migration requires a successful Java 17 `mvn clean verify`, compiled classes at major version 61, unchanged test methods, and a non-decreasing test-case count.The checks are intended to ensure the build and tests pass without silently changing or disabling tests.Source: §4.1 · pp. 5–6

4.2 Maximal Migration

  • Maximal migration adds a requirement to use stable latest major dependency versions available on Maven Central as of November 2024.The paper lists version requirements for the 240 most frequent dependencies in the selected subset and checks effective versions with `mvn dependency:tree`.Source: §4.2 · p. 6
  • The migration specifications remain incomplete, with fuller functional-equivalence checks and requirements for modern syntax or security features identified as possible additions.The paper notes that test-coverage enhancements and modernization features could extend the specifications.Source: §4.2 · p. 6

5 Agentic Framework

The framework starts with a shell-and-edit Strands agent for Java 17 migration, then adds prompt guidance and dependency-version retrieval for maximal migration. Its hybrid variant upgrades dependencies deterministically before using the agent to resolve resulting problems.

5.1 Strands Agent: Baseline

  • The baseline Strands agent uses shell and edit tools to inspect and modify repositories, targeting a successful Java 17 `mvn clean verify`.It is intended to handle basic version migration and straightforward compilation or test failures.Source: §5.1 · p. 6

5.2 PE Agent (Baseline + PE)

  • Prompt engineering adds instructions to upgrade dependencies to their latest major releases, strengthening the baseline for maximal migration.The standalone agent is effective for minimal migration but does not effectively perform maximal migration.Source: §5.2 · pp. 6–7

5.3 RAG Agent (Baseline + PE + RAG)

  • The RAG-agent retrieves authoritative latest-major dependency-version information from a knowledge base before performing updates.Retrieval addresses cases where the agent’s internal knowledge may not identify correct releases, especially for fast-evolving or less common libraries.Source: §5.3 · p. 7

5.4 Hybrid Approach

  • The hybrid approach upgrades dependencies through static analysis first, then invokes the agent to fix build failures, incompatibilities, and remaining migration steps.The paper presents this combination as effective for maximal migration with lower overall cost.Source: §5.4 · p. 7

6 Agent Prompt Templates

The appendix specifies shared agent tools and prompts for baseline, prompt-engineered, retrieval-augmented, and hybrid configurations. All variants receive the same initial request to migrate an entire Java 8 repository to Java 17.

6 Agent Prompt Templates

  • The baseline agent receives shell and edit tools, sandboxed to the project directory, with an 80-turn maximum to balance effectiveness and cost.These tools support repository inspection, modification, and command execution.Source: §6 · p. 7

6.1.1 Strands Agent: Baseline.

  • The baseline prompt asks the agent to migrate Java 8 projects to Java 17 and stop once `mvn clean verify` succeeds.The prompt says no summary is needed after successful verification.Source: §6.1.1 · p. 7

6.1.2 PE Agent.

  • The PE prompt adds an instruction to update all `pom.xml` dependencies to their latest versions that support Java 17.This guides the agent toward dependency modernization beyond build success.Source: §6.1.2 · p. 7

6.1.3 RAG Agent.

  • The RAG prompt directs the agent to look up recommended Java 17-compatible dependency versions, using its own knowledge when a dependency is absent from the database.It also instructs the agent to update all dependencies to compatible versions.Source: §6.1.3 · p. 7

6.1.4 Hybrid Approach.

  • The hybrid prompt assumes dependencies are already upgraded and directs the agent to fix compatibility issues without downgrading them.A static parser performs the initial upgrades using version information from the knowledge base; the agent addresses resulting failures and remaining migration steps.Source: §6.1.4 · p. 7

6.2 Initial User Prompt

  • All four agent variants receive the same initial prompt to migrate the entire Java 8 codebase to Java 17, controlling the comparison input.The prompt identifies the repository by its path.Source: §6.2 · p. 8

7 Experiments

Experiments compare migration efficacy and LLM usage across agentic and static approaches, then examine how results change with repository difficulty, verification criteria, and model choice. The hybrid approach matches RAG efficacy with fewer LLM calls, while disabling unit tests substantially raises minimal-migration efficacy.

7.2.2 Efficiency Study of 𝜂maximal

  • Figure 2 shows efficacy increasing with additional LLM calls; the Strands baseline plateaus, while RAG and hybrid gains are more gradual and sustained.Prompt engineering improves early-stage efficacy; static analysis offloads dependency upgrades before agent reasoning in the hybrid approach.Source: Figure 2 · p. 8, §7.2.2 · pp. 8–9

7 Experiments

  • Experiments use Claude-4.5-Sonnet in thinking mode via AWS Bedrock for agentic methods and release selected-subset trajectories and evaluations.Efficacy is reported at a cutoff of 80 LLM API calls, and Figure 2 emphasizes dependence on inference-time compute.Source: §7 · p. 8, §7.1.1 · p. 8

7.1.2 Open-source models

  • The open-source-model experiment applies the hybrid approach to migration-bench-selected using DeepSeek-V3.1, Qwen3-Coder-480B, and GLM-5.The passage identifies Table 3 as the efficacy report but supplies no model-specific values.Source: §7.1.2 · p. 8

7.2.1 Minimal and Maximal Migration Efficacy

  • For Claude-4.5-Sonnet, minimal migration efficacy is 4.7× maximal migration efficacy, and maximal migration is more challenging for larger repositories.The comparison is based on repository module and Java-file counts; maximal migration may require major dependency upgrades that introduce deprecated or breaking APIs.Source: §7.2.1 · p. 8

7.2.2 Efficiency Study of 𝜂maximal

  • 53.33% maximal migration efficacy matches the RAG agent with fewer LLM calls; the hybrid approach outperforms the static baseline’s 2.00% and Strands baseline’s 15.33%.Prompt engineering reaches 45.67% maximal efficacy, at increased LLM usage; the Strands baseline averages 33.68 calls per repository.Source: §7.2.2 · p. 8, Table 2 · p. 9

7.2.3 Compilation Efficacy

  • 97.67% minimal migration efficacy under mvn clean compile compares with 71.67% under the default test-verifying criterion.The ablation disables unit tests, checking compilation but not test correctness.Source: §7.2.3 · p. 9

8 Conclusion

The paper presents MigrationBench for Java 8 migration to later LTS versions and an evaluation framework, alongside an agentic workflow with promising results on the selected subset.

8 Conclusion

  • MigrationBench contains 5,102 full-dataset and 300 selected-subset repositories; with Claude-4.5-Sonnet, the workflow reports 71.67% minimal and 53.33% maximal migration efficacy on the selected subset.The evaluation framework approximates functional equivalence and checks Maven validations, compiled class versions, test-method consistency, test-case counts, and optional dependency versions.Source: §8 · p. 9

9 Limitations

The paper identifies limits in test coverage and Maven reliability that constrain evaluation and dataset collection.

9 Limitations

  • The evaluation does not enforce test coverage and relies on existing tests; for uncovered lines, Maven verify is equivalent to compile only.The authors identify the UTG subset as a possible resource for unit-test-generation research.Source: §9 · p. 9
  • Maven Central throttling can cause false negatives and reproducibility issues, especially when searching for base commits across the full dataset.The authors say this is not a major issue for the selected subset, but it can lead to selecting an incorrect base commit or missing a repository.Source: §9 · p. 9

A Data Collection

The data-collection process searches repository history for Java 8 base commits, applying consistency checks and runtime cutoffs to balance reproducibility and dataset retention.

A.1 F4: Search for Base Commit ID 𝐻𝑏

  • The search assumes Java versions and compiled-class major versions are generally chronological, scans pom.xml history for initially hard-coded versions, and excludes repositories that violate version checks.The history search proceeds in reverse order up to March 2024.Source: §A.1 · p. 11
  • Repositories are excluded when base-commit verification takes over 15 minutes or their broken state prevents finding a valid commit within that limit.The cutoff is intended to improve reproducibility, reduce compute time, and preserve as many repositories as possible.Source: §A.1 · p. 11
  • The process runs twice and retains repositories only when both runs report the same base commit, trading possible omissions for stronger correctness and reproducibility.The authors note that inconsistency, including Maven throttling or runtime cutoffs, can cause repositories to be missed.Source: §A.1 · p. 11

B Benchmark Dataset Statistics: full and selected Comparison

The full and selected subsets are compared across repository-size and testing measures, with the evaluation section extending earlier evaluation guidance. Unit testing is presented as a functional-equivalence check, and additional tests are recommended when existing coverage is low.

B Benchmark Dataset Statistics: full and selected Comparison

  • The full and selected subsets are compared using discrete probability and cumulative distributions for the listed repository measures.The comparison is introduced as extending the dataset statistics reported in Table 1.Source: B Benchmark Dataset Statistics: full… · p. 11
  • The comparison covers module counts, Java lines of code, all or test-only Java files, and test-case counts.Module count uses pom.xml files; test cases are parsed from root-level mvn test output, with negative values indicating test-running or parsing issues, likely due to Maven throttling.Source: B Benchmark Dataset Statistics: full… · p. 11

C Evaluation

  • Unit testing checks whether migrated code retains intended functionality by detecting discrepancies against predefined inputs and expected outputs.The evaluation section provides more details and extends Section 4.Source: §C.1 · p. 11, C Evaluation · p. 11

C.1 Unit Test Coverage

  • Existing unit tests are required through mvn clean verify, while additional tests are recommended when existing coverage is very low.Source: §C.1 · p. 11

D Additional Experiments

Additional experiments compare maximal migration on a sampled subset of the full dataset with the selected subset. The selected subset has lower success and requires more LLM calls, while Figures 3–4 compare repository and testing distributions across the subsets.

D Additional Experiments

  • The additional experiment samples every 18th repository from migration-bench-full, yielding 283 repositories evaluated with the hybrid Claude-4-5-Sonnet approach.Source: D Additional Experiments · p. 11
  • The selected subset has lower success and requires more LLM calls than the sampled full-dataset comparison, indicating greater challenge.Table 4 reports maximal migration efficacy for the hybrid approach with Claude-4-5-Sonnet.Source: Table 4 · p. 11, D Additional Experiments · p. 11
  • Figures 3–4 compare full and selected distributions for modules, LOC, Java-file counts, and test cases using discrete and cumulative distributions.Figure 4 drops invalid negative values.Source: Figure 3 · p. 12, Figure 4 · p. 12

Related papers

Loading 2505.09569v3…