PaperlayerView exact revision on arXiv

Source-linked AI summary

An Extensive Empirical Study on Code Translation Technique

Ruihang Fan, Jiajun Jiang, Xinpeng Wang, Jiateng Fu, Fengjie Li, Jiasi Shen

arXiv:2608.20776v1cs.SEcs.PL

TL;DR

The relative strengths of code-translation approaches across paradigms and translation granularities remain insufficiently understood. This study compares representative techniques and finds that LLMs and LLM-based methods generally outperform learning-based methods in correctness, while class-level translation is harder than method-level translation.

  • Problem

    Despite prior code-translation studies, systematic large-scale evidence comparing paradigms across tasks remains limited, leaving their relative strengths and weaknesses insufficiently understood.

    Source: §I · p. 1

  • Method

    The study evaluates 11 representative learning-based, LLM-based, and general-purpose LLM techniques under unified settings on two multilingual method-level and one class-level benchmark using correctness and similarity metrics.

    Source: §VIII · p. 13

  • Results

    LLMs and LLM-based methods generally outperform learning-based methods in correctness, while class-level translation is harder than method-level translation and performance varies with translation direction.

    Source: §I · pp. 1–2, §VIII · p. 13

  • Takeaways & Limitations

    The findings point to improving type-aware, structure-aware, and context-aware translation and combining correctness- and similarity-oriented evaluation.

    Source: §VIII · p. 13

    Findings may not generalize beyond the selected benchmarks, programming languages, and methods, which may not cover all real-world scenarios, language pairs, or future models.

    Source: §VII · p. 13

Abstract

from arXiv · show

Automated code translation is increasingly important for software evolution, yet the relative strengths and limitations of learning-based and large language model (LLM)-based techniques remain insufficiently understood. To address this gap, we conduct a large-scale empirical study comparing representative code translation techniques across methodological paradigms and translation granularities. We evaluate learning-based methods, LLM-based methods, and general-purpose LLMs on multilingual method-level and class-level benchmarks involving multiple programming languages. Our analysis considers executable correctness, code similarity, translation direction, translation granularity, and failure patterns. The results show that LLMs and LLM-based methods generally outperform learning-based methods in method-level correctness, although similarity metrics alone do not reliably reflect functional correctness. Translation direction substantially affects performance, particularly when translating between languages with different type-system characteristics. Class-level translation remains considerably more difficult than method-level translation because it requires preserving global semantics, interfaces, member relationships, and cross-method dependencies. Our error analysis further shows that static semantic errors and logical errors are the primary challenges in existing code translation systems. These findings provide empirical evidence and practical guidance for developing more robust, type-aware, structure-aware, and context-aware code translation techniques.

I. INTRODUCTION

To address the lack of systematic cross-paradigm, cross-task-level comparisons, the study evaluates code translation techniques across methods and granularities. It finds correctness differences by approach, direction, and translation level, with static semantic and logical errors remaining major challenges.

  • LLMs and LLM-based methods generally outperform learning-based methods in method-level correctness, though gains over base LLMs vary and are less pronounced on CodeBLEU.Source: §I · p. 1
  • Translations from statically typed to dynamically typed languages generally achieve better correctness than translations in the reverse direction, with benchmark characteristics also affecting results.Source: §I · pp. 1–2
  • On ClassEval-T, methods achieve relatively high CSR but much lower CA, showing that compiling class-level code is easier than preserving class semantics and behavioral equivalence.Source: §I · p. 2
  • For class-level translation, Python targets generally achieve higher CA than Java or C++ targets, while translation into C++ is particularly difficult in several directions.Larger contexts and cross-member dependencies amplify challenges from language-specific type and structural constraints.Source: §I · p. 2
  • 49.37% of sampled method-level failures were static semantic errors and 27.46% were logical errors (76.83% combined), while static semantic errors were 70.67% of class-level failures.Function signature violations were the most common static semantic error; the manual analysis covered 570 method-level and 150 class-level failed outputs.Source: §I · p. 2
  • The study presents a first large-scale unified comparison of 11 representative techniques across method- and class-level translation, using multilingual benchmarks and five programming languages.It analyzes correctness and similarity, translation directions, and benchmark characteristics; it recommends type-aware translation and stronger semantic verification for larger units, and releases results and implementations.Source: §I · pp. 1–2

II. RELATED WORK · A. Code Translation Methods

Code translation methods are broadly divided into rule-based, learning-based, and LLM-based approaches. The literature progresses from manually specified transformations and data-driven translation to methods leveraging LLM reasoning, feedback, and intermediate representations.

II. RELATED WORK · A. Code Translation Methods

  • Existing code translation methods fall into rule-based, learning-based, and LLM-based categories.The paper’s Table I summarizes representative methods by publication year, supported languages, evaluation benchmarks, and open-source status.Source: A. Code Translation Methods · p. 2
  • Rule-based systems parse code into structured forms and apply handcrafted mappings, but are language-specific, costly to maintain, and may need manual refinement,,,,,.Their rules can be difficult to adapt to new language features and evolving standards.Source: A. Code Translation Methods · p. 2
  • Early learning-based work adapted statistical machine translation to code, including phrase-based methods and mppSMT’s divide-and-conquer approach,,.mppSMT was reported to achieve strong performance at the time.Source: A. Code Translation Methods · p. 2
  • Neural translation included a tree-based JavaScript–CoffeeScript model, while supervised methods faced parallel-corpus constraints; TransCoder instead used unsupervised learning, later enhanced with test filtering and intermediate representations,,,.Source: A. Code Translation Methods · p. 2
  • LLM-based approaches leverage reasoning and iterative refinement, with CoTran using compiler and symbolic-execution feedback for finetuning and CodeGeeX supporting generation and translation across 23 languages,.CoTran directly optimizes for compilation correctness and functional equivalence; CodeGeeX was evaluated with HumanEval-X, which extends beyond Python.Source: A. Code Translation Methods · pp. 2–3
  • Other LLM-based systems use test-driven feedback, multi-agent error correction, or intermediate languages to address translation challenges,,.UniTrans uses automated test-driven feedback, TranAgent multiple LLMs to correct syntactic and semantic errors, and InterTrans intermediate languages to bridge syntactic gaps.Source: A. Code Translation Methods · p. 3
  • Code-generation models including CodeBERT, CodeT5, CodeT5+, and MftCoder can also serve as translation systems, with CodeT5+ noted as a competitive baseline,, [18],.Source: A. Code Translation Methods · p. 3

B. Code Translation Benchmarks · C. Empirical Studies on Code Translation · III. METHODOLOGY

Code-translation benchmarks vary in language and task coverage, and many lack executable tests; prior empirical studies have not compared learning-based and LLM-based methods under a unified, broad evaluation.

B. Code Translation Benchmarks

  • CodeNet covers 55 programming languages, Avatar targets challenging Python–Java translation, and CodeXGLUE benchmarks parallel Java–C# programs.Source: B. Code Translation Benchmarks · p. 3
  • Many earlier benchmarks lack executable test cases, limiting reliable evaluation of translated code’s functional correctness.Source: B. Code Translation Benchmarks · p. 3
  • TransCoder-Uni uses GeeksforGeeks programs in Python, C++, and Java, with test suites for 568 of its 948 cases.Source: B. Code Translation Benchmarks · p. 3
  • G-TransEval contains 400 parallel translation instances across five languages, with difficulty labels and complete test suites for every language pair.Source: B. Code Translation Benchmarks · p. 3
  • ClassEval-T benchmarks class-level Python, C++, and Java translation with longer snippets, complex dependencies, and comprehensive test suites; listed benchmarks were each used by at least two domain works.Source: B. Code Translation Benchmarks · p. 3

C. Empirical Studies on Code Translation

  • Prior studies examined neural translation types, LLM efficiency [53] and bugs, iterative repair, and PolyHumanEval, but focused on particular paradigms or limited languages and perspectives.This study aims to compare recent learning-based and LLM-based methods in a unified setting using quantitative metrics and error-type analysis.Source: C. Empirical Studies on Code Translation · p. 3

A. Code Translation Method Selection

The study compares recent open-source learning-based and LLM-based translation methods, supplemented by three general-purpose LLMs. Selection prioritizes performance, recency, and language diversity.

A. Code Translation Method Selection

  • Methods published between 2020 and 2025 were selected from open-source approaches using performance, recency, and language diversity criteria.The study favors methods with top performance on at least one benchmark, those released or updated after 2024, and those supporting more programming languages.Source: A. Code Translation Method Selection · p. 4
  • The evaluation includes eight selected code-translation methods—four learning-based and four LLM-based—plus three general-purpose LLMs, for 11 representative approaches.The eight comprise VIM-PT, StructCoder, TransCoder-ST, CodeT5+, two InterTrans variants, UniTrans, and ExeCoder; the additional LLMs are GPT-3.5-turbo, GPT-4o-mini, and DeepSeek-V4-Flash.Source: A. Code Translation Method Selection · p. 4
  • The learning-based group combines three translation-specific systems with CodeT5+, while retaining TransCoder-ST and CodeT5+ as classic competitive baselines,,,.VIM-PT jointly learns shared semantics and language-specific features; StructCoder uses ASTs and data-flow graphs; TransCoder-ST self-trains with test-filtered translations; CodeT5+ supports multiple model modes and multi-stage training objectives.Source: A. Code Translation Method Selection · p. 4
  • The LLM-based selections include InterTrans, UniTrans, and ExeCoder, which use search and test verification, iterative test-feedback repair, and executability representations, respectively.The two InterTrans implementations use Magicoder-SDS-7B and StarCoder2-15B, labeled IT(Magi) and IT(Star).Source: A. Code Translation Method Selection · p. 4

B. Benchmark Selection … C. Implementation and Configuration

The study selects multilingual, testable benchmarks compatible with the evaluated methods, then defines correctness and similarity measures and describes model preparation and evaluation settings.

B. Benchmark Selection

  • The study selected G-TransEval, TransCoder-Uni, and ClassEval-T, covering 1,062 tasks with multilingual implementations and test suites; XLCoST was excluded from test-time evaluation because it trained learning-based methods.Method-level benchmarks suit nearly all studied methods, while ClassEval-T suits most LLMs and LLM-based methods; learning-based methods are difficult to adapt to class-level context limits.Source: B. Benchmark Selection · p. 5

A. Research Questions

  • The four research questions examine overall method-level performance, task and language-direction variation, class-level correctness, and common translation-error types and distributions.The study compares correctness with compilation and functional-correctness metrics, analyzes CodeBLEU, and manually categorizes failures.Source: A. Research Questions · p. 5

B. Evaluation Metrics

  • Computational Accuracy (CA) measures programs passing all official tests, Compilation Success Rate (CSR) measures successful compilation, and CodeBLEU (CB) combines four equally weighted similarity components.CA and CSR represent functional and syntactic correctness, while CodeBLEU measures textual and structural similarity using n-gram, keyword, abstract syntax tree, and data-flow matching.Source: B. Evaluation Metrics · p. 5

C. Implementation and Configuration

  • TransCoder-ST used the authors’ best checkpoint, while VIM-PT, StructCoder, and CodeT5+ were trained or fine-tuned on XLCoST with overlapping benchmark examples removed.Training hyperparameters followed the corresponding open-source repositories’ defaults.Source: C. Implementation and Configuration · pp. 5–6
  • Learning-based methods used their best validation-set checkpoint; repeated training was avoided partly because VIM-PT training took approximately 1,266 GPU hours.The authors cite lower randomness in learning-based methods than in LLMs as the other reason for this checkpoint choice.Source: C. Implementation and Configuration · p. 6
  • Because released UniTrans could not run directly on G-TransEval, the study adapted its general translation-enhancement and translation-repair modules to process and rectify LLM outputs.Source: C. Implementation and Configuration · p. 6
  • LLM-based methods used temperature 0.8, top p 0.9, and one generated output per run, with averages reported across three independent runs.Experiments ran on a local machine with dual Intel Xeon 6388 CPUs, 512GB RAM, two NVIDIA A800-SXM4-80GB GPUs, and Ubuntu 20.04.6LTS.Source: C. Implementation and Configuration · p. 6

V. RESULTS ANALYSIS · A. RQ1: Method-Level Code Translation Performance

For method-level translation, LLMs and LLM-based methods generally outperform learning-based methods in correctness, with UniTrans performing best overall. Similarity alone is less discriminative: learning-based outputs can resemble correct translations while failing to compile or pass tests.

A. RQ1: Method-Level Code Translation Performance

  • The evaluation compares first outputs using CA@1 and CSR@1, alongside CodeBLEU, on G-TransEval and TransCoder-Uni across language pairs.Tables IV and V report correctness results across language pairs on the two benchmarks.Source: A. RQ1: Method-Level Code Translation… · p. 6
  • LLMs and LLM-based methods generally outperform learning-based methods in correctness; TransCoder-ST, the strongest learning-based method, scores only 56.96% CA and 70.50% CSR.Most LLM-based methods, except IT(Star), maintain high average CA and CSR on both datasets.Source: A. RQ1: Method-Level Code Translation… · p. 6
  • UniTrans leads all methods, scoring average CSR (CA) 96.01% (94.36%) on G-TransEval and 96.59% (91.66%) on TransCoder-Uni.These are averages across translation tasks.Source: A. RQ1: Method-Level Code Translation… · p. 6, Table IV · p. 7, Table V · p. 8
  • UniTrans improves GPT-3.5-turbo by 1.18 percentage points CSR (1.13 percentage points CA) and 1.44 percentage points CSR (1.57 percentage points CA) on the two datasets.Improvement over a base LLM depends on method design, base-model capability, and task; ExeCoder and IT (Magi) perform close to GPT-4o-mini.Source: A. RQ1: Method-Level Code Translation… · p. 6
  • Code-translation-enhanced methods achieve strong correctness, but improvement over a base LLM depends on method design, base-model capability, and target task.ExeCoder and IT (Magi) perform close to GPT-4o-mini.Source: A. RQ1: Method-Level Code Translation… · p. 6
  • Almost all methods exceed 70.00% average CodeBLEU on both datasets, but learning-based methods can have similarity close to LLM-based methods despite lower correctness.StructCoder and VIM-PT have similarity scores close to UniTrans despite poor correctness, showing that surface resemblance may not indicate compilable or test-passing output.Source: A. RQ1: Method-Level Code Translation… · p. 6
  • Passed-versus-failed CodeBLEU gaps are 15.96% to 28.32% for learning-based methods and 9.51% to 13.25% for LLM-based methods, suggesting greater focus on semantic function than line-by-line translation.InterTrans was excluded from this analysis to avoid empty parser outputs affecting the calculation.Source: A. RQ1: Method-Level Code Translation… · pp. 6, 8, Figure 3 · p. 8

B. RQ2: Performance on Different Translation Tasks

Translation correctness depends on direction and language type-system characteristics: statically typed sources and dynamically typed targets are generally more favorable, though dataset complexity can weaken this effect.

B. RQ2: Performance on Different Translation Tasks

  • On G-TransEval, DeepSeek-V4-Flash’s CA was 99.25% for C++ to Python but 76.33% for Python to C++.The same method can achieve substantially different results in opposite directions between the same language pair; similar patterns occur for other LLMs and LLM-based methods.Source: B. RQ2: Performance on Different… · p. 8
  • On G-TransEval, Python or JavaScript targets have significantly higher CA than Java or C++ targets, while Java or C++ sources score higher than Python or JavaScript sources.The paper attributes this pattern to explicit type information in statically typed languages: translating from Python to Java or C++ requires inferring variable, parameter, and return types, which can introduce errors.Source: B. RQ2: Performance on Different… · p. 8
  • On TransCoder-Uni, Python source- or target-language effects are much less pronounced, which the authors speculate reflects fewer type annotations and more complex logic than in G-TransEval.The paper concludes that language effects on correctness depend on the dataset’s type and logic complexity.Source: B. RQ2: Performance on Different… · p. 8
  • Translation generally performs better from statically typed to dynamically typed languages, while the reverse direction yields notably lower correctness.The paper recommends statically typed source and dynamically typed target languages as the more favorable practical setting.Source: B. RQ2: Performance on Different… · p. 9

C. RQ3: Class-Level Code Translation Performance

Class-level code translation is substantially harder than method-level translation: systems often produce compilable classes without preserving semantics, and performance varies with translation direction and target-language constraints.

C. RQ3: Class-Level Code Translation Performance

  • Longer contexts, richer inter-method dependencies, and more complex type structures make class-level translation qualitatively harder, requiring preservation of global semantics, member relationships, and cross-language type consistency.As translation units grow and type interactions become more complex, understanding and translating cross-language type systems becomes a major bottleneck.Source: C. RQ3: Class-Level Code Translation… · pp. 9–10
  • UniTrans averages 24.82% CA and 89.39% CSR, versus general-purpose LLM averages of 18.82–22.87% CA and 72.15–83.51% CSR, leaving class-level correctness limited.Even the best-performing method correctly translates only a limited portion of class-level code.Source: C. RQ3: Class-Level Code Translation… · p. 9, Table VIII · p. 10
  • Class-level results decline for all methods, with CA falling more than CSR, showing that compilation is easier than generating semantically correct translations.The larger CA–CSR gap indicates that semantic correctness, rather than compilation, is the main challenge in complex class contexts.Source: C. RQ3: Class-Level Code Translation… · p. 9
  • In Java→C++ and Python→C++, CA is extremely low for all methods, while C++→Java and Python→Java generally retain CSR above 73.40% but CA at or below 25.53%.The gap suggests that many classes compile but fail to preserve semantics, structure, or behavioral equivalence as dependencies and types grow more complex.Source: C. RQ3: Class-Level Code Translation… · p. 9
  • Translations targeting Python achieve relatively higher CA, whereas C++ is consistently the hardest target; Java→C++ CA is at most 12.77%.Stricter type and structural constraints make type-aware translation especially important for statically typed targets.Source: C. RQ3: Class-Level Code Translation… · pp. 9–10

D. RQ4: Case Study of Translation Errors · 1) RQ4.1 Method-Level Error Analysis:

Method-level failures are dominated by static semantic and logical errors, with type inference and code complexity helping explain how error patterns differ across benchmarks. The analysis classifies failures into four categories and examines their causes and dataset-specific distributions.

1) RQ4.1 Method-Level Error Analysis:

  • Static semantic errors (49.37%) and logical errors (27.46%) dominate 570 method-level failures, while syntax errors (18.25%) and runtime errors (4.92%) are less common.These results indicate that semantic and functional correctness remains the main challenge for both learning-based and LLM-based translation methods.Source: Figure 5 · p. 10, 1) RQ4.1 Method-Level Error Analysis · p. 10
  • Function signature violations constitute 70.18% of static semantic errors, reflecting the challenge of inferring type mappings, especially from information-sparse to information-rich languages.Even state-of-the-art LLMs must understand translated code semantics to determine appropriate target-language types.Source: 1) RQ4.1 Method-Level Error Analysis · p. 10
  • A JavaScript-to-C# example shows an interface-breaking signature mismatch: the translation uses int[] where the reference requires List<int> for the return and first two parameters.It also uses ToArray() instead of ToList(); collection-type inference and target-language abstractions remain a bottleneck when translating from information-sparse JavaScript to statically typed C#.Source: 1) RQ4.1 Method-Level Error Analysis · pp. 10–11, Figure 7 · p. 11
  • A Python-to-C++ logical error places an assignment outside its loop, producing wrong outputs despite compiling and making complex algorithms difficult to validate without understanding their logic.The example is attributed to IT(Magi).Source: 1) RQ4.1 Method-Level Error Analysis · p. 11, Figure 8 · p. 11
  • G-TransEval averages 35.45% function signature violations and 21.52% logical errors, versus 19.67% and 34.00%, respectively, on TransCoder-Uni.The benchmarks therefore differ substantially in the balance of their two dominant error types.Source: 1) RQ4.1 Method-Level Error Analysis · p. 11
  • G-TransEval signatures use more complex types and compound STL types, whereas primitive types alone account for 75.15% of TransCoder-Uni signatures.The added need to infer value types and understand mapping rules contributes to more signature violations in G-TransEval.Source: 1) RQ4.1 Method-Level Error Analysis · p. 11
  • TransCoder-Uni has higher average cyclomatic complexity, with medium-or-higher-difficulty tasks in nearly 30% of cases versus 20.20% in G-TransEval, contributing to more logical errors.Its more complex logic but simpler data types also reduces the impact of inaccurate type translation, helping explain the smaller Python-versus-C++ target-language performance gap there.Source: 1) RQ4.1 Method-Level Error Analysis · p. 11
  • Type diversity and complexity make datasets more susceptible to language effects, consistent with the observed relationship between type inference, type mapping, and translation direction.The passage connects this pattern to the difference between translating from statically to dynamically typed languages and the reverse direction.Source: 1) RQ4.1 Method-Level Error Analysis · p. 11

2) RQ4.2 Class-Level Error Analysis:

Class-level translation has a similar broad error profile to method-level translation, but substantially amplifies static semantic consistency problems. Preserving function signatures, symbol bindings, and cross-member dependencies is a primary bottleneck as translation granularity increases.

2) RQ4.2 Class-Level Error Analysis

  • Static semantic errors account for 70.67 of class-level failures, logical errors 19.33%, syntax errors 8.00%, and runtime errors 2.00%; semantic consistency matters more than syntax.Compared with method-level translation, class-level translation is more constrained by preserving interfaces, member bindings, and cross-method dependencies.Source: 2) RQ4.2 Class-Level Error Analysis · pp. 11–12
  • Function signature violations make up 75.47% of static semantic errors and 53.33% of all class-level failures, underscoring interface consistency as a primary bottleneck.Class-level translation must maintain consistency across fields, methods, and class-level scopes, not merely map individual types.Source: 2) RQ4.2 Class-Level Error Analysis · pp. 11–12
  • UniTrans and commercial models’ failures are dominated by static semantic errors, especially signature violations, while ExeCoder shows more syntax and binding-related failures.The distribution suggests stronger models reduce syntax and runtime errors, leaving failures more concentrated in static semantic and logical errors.Source: 2) RQ4.2 Class-Level Error Analysis · p. 12
  • As translation granularity increases, the challenge shifts from local statements to consistency across class members, methods, and properties, especially under stricter target-language type and structural constraints.Longer contexts and greater semantic complexity further increase the consistency requirements.Source: 2) RQ4.2 Class-Level Error Analysis · p. 12

VI. IMPLICATIONS

The implications are that code-translation research must move beyond isolated methods, evaluate executable correctness alongside similarity, and account for language direction, error patterns, and global class consistency.

1) Move beyond isolated method-level translation

  • 24.82% average CA versus 89.39% average CSR for the best class-level method shows that class translation remains difficult despite stronger method-level results.LLMs and LLM-based methods generally outperform learning-based methods on method-level correctness, but future work should target class- and repository-level settings that preserve inter-method dependencies, class structure, and global semantics.Source: §VI · p. 12

2) Evaluate with correctness and similarity metrics

  • Similarity metrics alone cannot reliably indicate compilation success or functional correctness, so evaluations should pair them with correctness-oriented measures.Learning-based methods may achieve CodeBLEU scores comparable to LLM-based methods despite substantially lower CA and CSR; correct LLM-based translations may also differ from references at the surface level.Source: §VI · p. 12

3) Account for direction and benchmark characteristics

  • Translating from information-rich languages such as Java and C++ to information-sparse languages such as Python and JavaScript is generally easier than the reverse direction.The reverse direction requires additional type inference and structural completion; future methods should account for type inference, type mapping, and target-benchmark type and logical complexity.Source: §VI · p. 12

4) Address static-semantic and logical errors

  • 49.37% static-semantic errors and 27.46% logical errors account for most sampled method-level failures, with signature violations comprising 70.18% of static-semantic errors.At class level, static-semantic errors comprise 70.67% of failures and signature violations alone 53.33% of all failures; stronger type mapping, interface checks, symbol binding, semantic reasoning, control-flow preservation, and logic are needed.Source: §VI · pp. 12–13

5) Preserve global consistency in class-level translation

  • Class-level translation requires consistency across fields, signatures, method bodies, inheritance, and cross-method references rather than independent local code generation.Language-specific type systems, object models, declarations, and member dependencies amplify difficulty; C++ targets are considerably harder than Python or Java in several settings, motivating structure-aware, dependency-aware, and incrementally verified approaches.Source: §VI · p. 13

VII. THREATS TO VALIDITY · VIII. CONCLUSION

The study finds that LLMs and LLM-based methods generally outperform learning-based methods in correctness, while direction and granularity affect translation difficulty; its conclusions are qualified by benchmark and method scope. It calls for type-, structure-, and context-aware systems and evaluation combining correctness with similarity.

VII. THREATS TO VALIDITY

  • Expert judgment may bias error-cause classification despite independent annotation, reconciliation, inter-rater agreement measurement, and inductive open coding.Three evaluators independently annotated cases, used Fleiss’ kappa, and resolved disagreements through discussion after examining error cases.Source: §VII · p. 13
  • The authors consider data-leakage effects limited after cleaning overlapping training samples and testing commercial LLMs on post-cutoff problems.The additional dataset contained 50 C++ snippets with test suites from Luogu; observed result changes were limited, though its context length and difficulty differed from the original function-level dataset.Source: §VII · p. 13
  • Findings may not generalize beyond the selected benchmarks, languages, and methods to all real-world scenarios, language pairs, or future models.The study mitigates this threat by evaluating multiple benchmarks with different properties and comparing representative methods under a unified setting.Source: §VII · p. 13

VIII. CONCLUSION

  • The study evaluates 11 representative translation techniques under unified settings across two method-level multilingual benchmarks and one class-level benchmark.It includes learning-based methods, LLM-based methods, and general-purpose LLMs, using correctness and similarity metrics.Source: §VIII · p. 13
  • LLMs and LLM-based methods generally outperform learning-based methods in correctness, but direction affects performance and class-level translation is harder than method-level translation.Static semantic and logical errors dominate failure patterns, with function signature violations particularly common; the correctness advantage is less evident on similarity metrics.Source: §VIII · p. 13
  • The authors identify type-aware, structure-aware, and context-aware translation, alongside combined correctness- and similarity-oriented evaluation, as priorities for future systems.Source: §VIII · p. 13

Related papers

Loading 2608.20776v1…