PaperlayerView exact revision on arXiv

Source-linked AI summary

An Extensive Empirical Study on Code Translation Technique

Ruihang Fan, Jiajun Jiang, Xinpeng Wang, Jiateng Fu, Fengjie Li, Jiasi Shen

arXiv:2608.20776v1cs.SEcs.PL

TL;DR

Evidence remains limited on how code translation techniques compare across methodological paradigms and translation granularities, despite automation’s importance for software evolution. This study evaluates representative methods and LLMs across multilingual method- and class-level benchmarks, finding that LLM-based approaches generally achieve better method-level correctness while class-level translation remains substantially harder.

  • Problem

    Evidence remains limited on the comparative strengths of automated code translation techniques across methodological paradigms and translation granularities, despite automation’s importance for software evolution.

  • Method

    The study compares 11 representative learning-based methods, LLM-based methods, and general-purpose LLMs on multilingual method- and class-level benchmarks using correctness and similarity metrics.

  • Results

    LLMs and LLM-based methods generally outperform learning-based methods in correctness, while class-level translation is substantially harder and static semantic and logical errors dominate failures.

  • Takeaways & Limitations

    Future code translation systems should become more type-aware, structure-aware, and context-aware while combining correctness-oriented and similarity-oriented evaluation.

  • Takeaways & Limitations

    The findings may not fully generalize beyond the selected benchmarks, programming languages, methods, real-world scenarios, language pairs, and future models.

Abstract

from arXiv · show

Automated code translation is increasingly important for software evolution, yet the relative strengths and limitations of learning-based and large language model (LLM)-based techniques remain insufficiently understood. To address this gap, we conduct a large-scale empirical study comparing representative code translation techniques across methodological paradigms and translation granularities. We evaluate learning-based methods, LLM-based methods, and general-purpose LLMs on multilingual method-level and class-level benchmarks involving multiple programming languages. Our analysis considers executable correctness, code similarity, translation direction, translation granularity, and failure patterns. The results show that LLMs and LLM-based methods generally outperform learning-based methods in method-level correctness, although similarity metrics alone do not reliably reflect functional correctness. Translation direction substantially affects performance, particularly when translating between languages with different type-system characteristics. Class-level translation remains considerably more difficult than method-level translation because it requires preserving global semantics, interfaces, member relationships, and cross-method dependencies. Our error analysis further shows that static semantic errors and logical errors are the primary challenges in existing code translation systems. These findings provide empirical evidence and practical guidance for developing more robust, type-aware, structure-aware, and context-aware code translation techniques.

I. INTRODUCTION

This study systematically compares code translation techniques across methodological paradigms, languages, translation directions, and granularities. It finds that LLM-based approaches generally improve method-level correctness, while direction, language, granularity, and semantic failures substantially shape performance.

  • Key findings: LLMs and LLM-based methods generally outperform learning-based methods in method-level correctness, although gains over base LLMs depend on method design, model capability, and task.Their advantage is less pronounced under CodeBLEU than under correctness metrics such as CA and CSR.
  • Key findings: Translation direction substantially affects correctness, with statically typed to dynamically typed translation generally outperforming the reverse direction.Benchmark characteristics also influence this directional effect.
  • Key findings: Class-level translation is substantially harder than method-level translation because it must preserve class semantics, member relationships, interfaces, and behavioral equivalence.On ClassEval-T, methods achieve relatively high CSR but much lower CA, indicating that compilability is easier than semantic preservation.
  • Key findings: Class-level performance varies strongly by target language, with Python generally achieving higher CA than Java or C++, while C++ is particularly difficult in several directions.Larger contexts and cross-member dependencies amplify language-specific type and structural constraints.
  • Failure analysis: Static semantic and logical errors dominate failures, comprising 76.83% of sampled method-level failures, while static semantic errors account for 70.67% at class level.Function signature violations are the most common static semantic error, underscoring challenges in type mapping, interface preservation, symbol binding, and logic.
  • Study scope: The study evaluates 11 representative techniques spanning learning-based methods, LLM-based methods, and general-purpose LLMs under unified experimental settings.The evaluation covers multiple methodological paradigms and translation granularities.

II. RELATED WORK · A. Code Translation Methods

Code translation methods span rule-based, learning-based, and LLM-based paradigms, with recent work increasingly using neural models and LLM feedback to improve translation quality. The related work also organizes representative techniques by publication year, supported languages, evaluation benchmarks, and source-code availability.

  • A. Code Translation Methods: Existing code translation approaches are broadly categorized as rule-based, learning-based, and LLM-based methods.
  • A. Code Translation Methods: Table I summarizes representative techniques by publication year, supported languages, evaluation benchmarks, and whether their source code is open-sourced.
  • A. Code Translation Methods: Rule-based systems parse code into structured intermediate forms and apply handcrafted mappings, but are typically language-specific and labor-intensive.
  • A. Code Translation Methods: Learning-based methods infer translation patterns from data, with early work adapting statistical machine translation techniques from natural-language processing.
  • A. Code Translation Methods: Supervised neural translation models expanded this paradigm, but their effectiveness is constrained by the availability and quality of parallel corpora.
  • A. Code Translation Methods: Unsupervised neural transcompilers such as TransCoder addressed limited parallel data, while later work added unit-test filtering and intermediate representations.
  • A. Code Translation Methods: LLM-based methods leverage reasoning and iterative refinement, including reinforcement learning guided by compiler and symbolic-execution feedback to optimize compilation correctness.
  • A. Code Translation Methods: Recent LLM-based systems extend translation with multilingual evaluation, automated test-driven feedback, multi-agent error correction, and intermediate languages for bridging syntactic gaps.

B. Code Translation Benchmarks · C. Empirical Studies on Code Translation

The paper reviews code translation benchmarks and empirical studies assessing translation methods across languages, task types, and evaluation settings. It emphasizes the shift toward executable-test benchmarks and systematic analyses of translation effectiveness, efficiency, and bugs.

  • B. Code Translation Benchmarks: CodeNet covers 55 programming languages using large-scale parallel programs collected from online coding platforms.
  • B. Code Translation Benchmarks: Avatar targets challenging Python–Java translation tasks, while CodeXGLUE benchmarks code-to-code translation using parallel Java–C# programs.
  • B. Code Translation Benchmarks: Many early benchmarks lack executable test cases, limiting reliable evaluation of translated programs’ functional correctness.
  • B. Code Translation Benchmarks: TransCoder-Uni includes test suites for 568 of 948 Python, C++, and Java translation cases.
  • B. Code Translation Benchmarks: G-TransEval provides 400 parallel translation instances across five languages, with difficulty labels and complete test suites for every language pair.
  • C. Empirical Studies on Code Translation: Jiao et al. proposed a four-type taxonomy and systematically evaluated state-of-the-art neural translation models across the resulting translation types.
  • C. Empirical Studies on Code Translation: Prior studies examined LLM-based translation execution efficiency, LLM effectiveness for automated translation, and bugs introduced by LLMs.

III. METHODOLOGY … A. Research Questions

The methodology establishes a unified comparison of representative code translation approaches, benchmarks, and research questions. It evaluates multilingual translation quality using correctness, compilation, functional correctness, and CodeBLEU analyses.

  • A. Code Translation Method Selection: The study addresses the underexplored need to compare different code translation paradigms under a unified experimental setting.
  • A. Code Translation Method Selection: The evaluation investigates 11 representative code translation approaches, excluding the general-purpose LLMs from the marked non-LLM methods.
  • A. Code Translation Method Selection: Selected methods include VIM-PT, which learns shared and language-specific semantics, and StructCoder, which incorporates abstract syntax trees and data flow.
  • B. Benchmark Selection: The benchmark selection applies criteria including multilingual coverage, enabling comparison across diverse programming languages for the same translation task.
  • A. Research Questions: RQ1 examines overall method-level translation performance using two large-scale benchmarks, compilation and functional correctness, and CodeBLEU for multidimensional quality analysis.

B. Evaluation Metrics … A. RQ1: Method-Level Code Translation Performance

The study evaluates code translation using correctness and similarity metrics, with controlled configurations across learning-based and LLM-based methods. On method-level benchmarks, LLM-based methods generally achieve stronger correctness, while CodeBLEU is less indicative of functional success.

  • B. Evaluation Metrics: CA, CSR, and CodeBLEU measure passed tests, successful compilation, and generated-reference similarity, respectively.CA and CSR are correctness metrics; CodeBLEU combines n-gram, keyword, abstract syntax tree, and data flow matching.
  • C. Implementation and Configuration: Learning-based methods used released or XLCoST-trained checkpoints, with benchmark-overlapping training examples removed where applicable.Models were evaluated using checkpoints with the best validation performance, while training followed open-source repository defaults.
  • C. Implementation and Configuration: LLM-based methods used temperature = 0.8, top p = 0.9, and n = 1, with three independent runs averaged.The experiments ran on a machine with dual Intel Xeon 6388 CPUs, 512GB RAM, and two NVIDIA A800-SXM4-80GB GPUs.
  • V. RESULTS ANALYSIS: The method-level evaluation reports CA@1, CSR@1, and CodeBLEU across language pairs on G-TransEval and TransCoder-Uni.The first output from each method is used by default for CA@1 and CSR@1.
  • A. RQ1: Method-Level Code Translation Performance: UniTrans achieves average CSR (CA) scores of 96.01% (94.36%) on G-TransEval and 96.59% (91.66%) on TransCoder-Uni.UniTrans outperforms learning-based methods and other LLM-based approaches; it also improves over GPT-3.5-turbo by reported percentage-point margins.
  • A. RQ1: Method-Level Code Translation Performance: LLM-based methods generally outperform learning-based methods on correctness, while enhancement gains over base LLMs depend on design, model capability, and task.TransCoder-ST, the strongest learning-based method, achieves CA = 56.96% and CSR = 70.50%.

B. RQ2: Performance on Different Translation Tasks

Translation performance depends strongly on direction and the source–target language pair. Static-to-dynamic translation is generally more favorable, while dataset characteristics can moderate this effect.

  • Translation Direction: 99.25% CA for C++ to Python with DeepSeek-V4-Flash on G-TransEval falls to 76.33% for the reverse direction.Other LLMs and LLM-based methods show similar directional differences.
  • Language Pair Effects: On G-TransEval, CA is significantly higher when Python or JavaScript is the target than when Java or C++ is the target.The results confirm that source and target language choices clearly affect performance.
  • Dataset Effects: On TransCoder-Uni, the performance change associated with Python as source or target is much less pronounced than on G-TransEval.The study speculates that fewer type annotations and more complex logic in TransCoder-Uni weaken Python-related disadvantages.
  • Type-System Direction: Translation generally performs better from statically typed languages to dynamically typed languages than in the opposite direction.Using a statically typed source and dynamically typed target is usually more favorable for practical code translation.

C. RQ3: Class-Level Code Translation Performance

Class-level code translation is substantially harder than method-level translation: existing techniques often generate compilable code without preserving semantic correctness. Performance varies by language direction, with C++ targets particularly difficult because larger class contexts intensify type and structural constraints.

  • Overall Performance: 24.82% average CA and 89.39% average CSR make UniTrans the strongest overall method, yet it correctly translates only a limited portion of class-level code.General-purpose LLMs achieve average CA values between 18.82% and 22.87% and average CSR values between 72.15% and 83.51%.
  • Overall Performance: Class-level translation causes clear CA and CSR declines, but the much larger CA drop shows semantic correctness—not compilation—is the primary challenge.Compilable class-level code is considerably easier to generate than semantically correct translations under longer contexts and richer inter-method dependencies.
  • Translation Direction: The CA–CSR gap widens across nearly all methods and directions, with Java→C++ and Python→C++ often near-zero CA despite moderate or high CSR.For C++→Java and Python→Java, CSR generally exceeds 73.40%, while CA remains limited to ≤25.53%.
  • Translation Direction: Translations targeting Python achieve relatively higher CA, whereas C++ targets are consistently hardest, including Java→C++ results where all methods achieve CA ≤12.77%.Python→C++ performs only marginally better than Java→C++, indicating stronger language effects at class level.
  • Challenges and Implications: Larger class structures and more complex type interactions make cross-language type-system understanding a major bottleneck, especially when translating into statically typed languages.Practical systems should therefore emphasize type-aware translation to satisfy strict type and structural constraints.

D. RQ4: Case Study of Translation Errors · 1) RQ4.1 Method-Level Error Analysis:

The study manually classifies failed translations to identify dominant method-level error patterns and their causes. Static semantic and logical errors predominate, with type inference, type mapping, and code complexity shaping benchmark- and direction-specific failures.

  • D. RQ4: Case Study of Translation Errors: The study manually analyzes failed outputs using a two-step classification process to identify error categories and major translation challenges.Thirty failed cases establish initial categories, followed by independent evaluation by three experienced developers.
  • 1) RQ4.1 Method-Level Error Analysis:: 49.37% of 570 method-level failed outputs are static semantic errors, while 27.46% are logical errors.Together, these categories account for the vast majority of failed cases.
  • 1) RQ4.1 Method-Level Error Analysis:: 70.18% of static semantic errors are function signature violations caused by difficult cross-language type inference and type mapping.The challenge is especially pronounced when translating from information-sparse languages into information-rich, statically typed targets.
  • 1) RQ4.1 Method-Level Error Analysis:: Logical errors often arise from misplaced statements that compile successfully but produce incorrect outputs, making them difficult to detect in complex algorithms.One example places an update to ans outside the loop during Python-to-C++ translation.
  • 1) RQ4.1 Method-Level Error Analysis:: G-TransEval’s complex compound STL types increase type-mapping difficulty, whereas TransCoder-Uni’s simpler types but higher code complexity increase logical-error challenges.Medium-or-higher difficulty tasks comprise nearly 30% of TransCoder-Uni cases versus 20.20% of G-TransEval cases.
  • 1) RQ4.1 Method-Level Error Analysis:: Datasets with more diverse and complex types are more susceptible to translation-direction effects because static-to-dynamic and dynamic-to-static translation differ in type inference and mapping difficulty.This finding links language effects to the complexity of the source and target type systems.

2) RQ4.2 Class-Level Error Analysis:

Class-level translation has a similar broad error profile to method-level translation but is more constrained by semantic consistency. Function signature violations, symbol bindings, and cross-member dependencies become primary bottlenecks as translation granularity increases.

  • Class-Level Error Distribution: 70.67% of class-level failed cases are static semantic errors, while logical, syntax, and runtime errors account for 19.33%, 8.00%, and 2.00%, respectively.Compared with method-level translation, class-level translation is less affected by syntactic problems but more constrained by semantic consistency.
  • Static Semantic Errors: Function signature violations comprise 75.47% of static semantic errors and 53.33% of all class-level failed cases.Other static semantic failures mainly involve declaration or binding violations and undefined or undeclared identifiers.
  • Structural Consistency: Class-level translation must preserve consistency across fields, methods, class-level scopes, and structural relationships among class members.The larger translation unit requires preserving global structure in addition to local statement correctness.
  • Method-Wise Error Patterns: UniTrans and commercial models primarily produce static semantic errors, whereas ExeCoder produces more syntax and binding-related failures.Function signature violations dominate the static semantic errors of the stronger models, while ExeCoder shows a more diverse error distribution.
  • Comparison with Method-Level Translation: Class-level and method-level translation share dominant static-semantic and logical error categories, but class-level translation amplifies type consistency, symbol binding, interface consistency, and cross-member dependencies.The main challenge shifts from local statement translation to maintaining syntactic and semantic consistency across the entire class.

VI. IMPLICATIONS

The implications call for moving beyond isolated method translation and evaluating code translation with both correctness and similarity metrics. They also emphasize direction- and benchmark-aware methods, stronger semantic and logical reasoning, and global consistency for class-level translation.

  • Beyond Method-Level Translation: Method-level translation is not solved, as class-level translation remains substantially more difficult despite high compilability.The best-performing method achieves average CA of 24.82% and average CSR of 89.39% at class level.
  • Evaluation Metrics: Code translation should be evaluated with correctness and similarity metrics because similarity scores alone cannot reliably indicate compilation success or functional correctness.Learning-based methods can achieve CodeBLEU scores comparable to LLM-based methods while having substantially lower CA and CSR.
  • Direction and Benchmark Effects: Translation difficulty varies with language direction and benchmark characteristics, especially when reverse translation requires additional type inference and structural completion.Translating from information-rich languages such as Java and C++ to information-sparse languages such as Python and JavaScript is generally easier than the reverse.
  • Failure Patterns: Static semantic consistency and logical equivalence are the main bottlenecks in existing translation systems.Static semantic errors account for 49.37% and logical errors for 27.46% of sampled method-level failures; at class level, static semantic errors account for 70.67%.
  • Global Consistency: Class-level translation requires global consistency across language-specific type systems, object models, declarations, interfaces, and member dependencies.The findings motivate stronger type-mapping constraints, interface checking, symbol-binding analysis, semantic reasoning, and preservation of control-flow and algorithmic structure.

VII. THREATS TO VALIDITY

The study faces internal-validity threats from subjective error classification and possible data leakage, alongside external-validity limits from its selected benchmarks, languages, and methods. The authors mitigate these risks through inter-rater agreement checks, dataset cleaning, post-cutoff evaluation data, and comparisons across diverse benchmarks.

  • Internal Validity: Expert judgment may bias error-cause classification despite independent annotation and reconciliation by three evaluators.Fleiss’ kappa was used to measure inter-rater agreement and assess classification consistency.
  • Internal Validity: Data leakage was mitigated by manually cleaning overlapping training samples and selecting test benchmarks from a different source than the models’ primary training data.The learning-based training samples were cleaned, while StarCoder2 and Magicoder primarily use GitHub data and the benchmarks come from GeeksforGeeks.
  • Internal Validity: A manually constructed dataset of 50 C++ snippets with test suites, drawn from post-cutoff Luogu problems, supported leakage-resistant evaluation of three commercial LLMs in both translation directions.The problems appeared after the commercial LLMs’ knowledge cutoff dates.
  • External Validity: Findings may not generalize beyond the selected benchmarks, programming languages, methods, real-world scenarios, language pairs, or future models.The study mitigated this threat by evaluating diverse benchmarks and comparing representative methods under a unified setting.

VIII. CONCLUSION

This large-scale study evaluates 11 code translation techniques across method- and class-level multilingual benchmarks using correctness and similarity metrics. It finds that LLM-based approaches generally improve correctness, while translation direction and granularity remain important challenges.

  • Study scope: The study evaluates 11 representative techniques across two method-level multilingual benchmarks and one class-level benchmark using correctness and similarity metrics.The techniques include learning-based methods, LLM-based methods, and general-purpose LLMs under unified settings.
  • Key findings: LLMs and LLM-based methods generally outperform learning-based methods in correctness, but their advantage is less evident on similarity metrics.The results also show that translation direction affects performance, particularly on G-TransEval.
  • Key findings: Class-level translation remains substantially more difficult than method-level translation, especially for preserving semantic correctness and structural consistency.The conclusion identifies static semantic errors and logical errors as prominent failure patterns.
  • Implications: Future systems should improve type-aware, structure-aware, and context-aware translation while combining correctness-oriented and similarity-oriented evaluation.These directions follow from the study’s findings on correctness, similarity, translation direction, and translation granularity.
Loading 2608.20776v1…