PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation

Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, Wen Wang

arXiv:2310.04951v2cs.AIcs.PL

TL;DR

Existing code-translation datasets often focus on a single popular language pair, limiting coverage of diverse applications. CodeTransOcean brings together multilingual and cross-framework datasets and evaluation resources; the paper reports improvements from multilingual modeling and evaluates ChatGPT’s code translation and debugging.

  • Problem

    Many code-translation datasets focus on a single popular language pair, while existing metrics may not assess executability or functional equivalence reliably.

    Source: §1 · p. 1, §4.2 · p. 7

  • Method

    The authors construct CodeTransOcean’s multilingual and cross-framework datasets, develop multilingual modeling approaches, and introduce DSR@K to evaluate code execution and debugging.

    Source: §1 · pp. 2–3, §4.2 · p. 7

  • Results

    Multilingual modeling improves translation quality for high- and low-resource pairs and training efficiency; baseline ChatGPT achieves 48.57% DSR@0, with self-debugging and one-shot improving performance while chain-of-thought degrades accuracy.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    CodeTransOcean provides datasets, tools, and baselines for code-translation research, while fuzzy execution remains an area requiring further study.

    Source: §5 · p. 9

    AutoTransExecuter currently supports only Python as the target language because automating language-specific dependencies and runtime configuration is challenging.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent code translation techniques exploit neural machine translation models to translate source code from one programming language to another to satisfy production compatibility or to improve efficiency of codebase maintenance. Most existing code translation datasets only focus on a single pair of popular programming languages. To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. CodeTransOcean consists of three novel multilingual datasets, namely, MultilingualTrans supporting translations between multiple popular programming languages, NicheTrans for translating between niche programming languages and popular ones, and LLMTrans for evaluating executability of translated code by large language models (LLMs). CodeTransOcean also includes a novel cross-framework dataset, DLTrans, for translating deep learning code across different frameworks. We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boosting the training efficiency. We also propose a novel evaluation metric Debugging Success Rate@K for program-level code translation. Last but not least, we evaluate LLM ChatGPT on our datasets and investigate its potential for fuzzy execution predictions. We build baselines for CodeTransOcean and analyze challenges of code translation for guiding future research. The CodeTransOcean datasets and code are publicly available at https://github.com/WeixiangYAN/CodeTransOcean.

1 Introduction

Existing code-translation datasets have limited language coverage and often omit needs such as niche-language modernization and deep-learning framework migration. CodeTransOcean addresses these gaps with multilingual and cross-framework datasets, while the paper also studies multilingual modeling, ChatGPT translation, and evaluation approaches.

1 Introduction

  • Existing datasets provide insufficient language coverage, limited scale, and uneven distributions, while many focus on one popular language pair.These gaps constrain translation of niche-language legacy systems and code across deep-learning frameworks.Source: §1 · pp. 1–2
  • CodeTransOcean combines MultilingualTrans, NicheTrans, and LLMTrans with DLTrans for cross-framework code translation.The datasets cover eight popular languages, thirty-seven niche languages paired with the popular ones, and 350 LLMTrans samples with execution results.Source: §1 · p. 2
  • Multilingual modeling significantly improves translation quality for both high- and low-resource language pairs and improves training efficiency.The experiments use MultilingualTrans, NicheTrans, and DLTrans.Source: §1 · p. 2
  • On LLMTrans, baseline ChatGPT achieves 48.57% DSR@0; self-debugging and one-shot improve performance, while chain-of-thought degrades translation accuracy.DSR@K measures samples that execute successfully and produce expected functionality after K debugging rounds.Source: §1 · p. 2
  • The paper proposes fuzzy execution to address limitations of existing code-translation metrics, but preliminary ChatGPT results show inadequate prediction for arbitrary programming languages.The authors identify this as an area requiring future research.Source: §1 · p. 3

2 Related Work

Prior code-translation datasets vary in scope, but many are limited in language coverage, scale, or data balance. The section reviews representative datasets and notes that these constraints can limit practical applications and model generalizability.

Code Translation Datasets

  • Existing code-translation datasets commonly have insufficient language coverage, limited scale, or imbalanced data distributions.The section argues these shortcomings can contribute to overfitting, underfitting, and poor generalizability.Source: §2 · p. 3
  • CodeTrans consists of Java–C# pairs, AVATAR covers Java–Python, and XLCoST expands CoST by 7.3 times but retains limited coverage and imbalanced distribution.The section also notes that XTest is not open-sourced and about half of CodeNet has incorrect solutions.Source: §2 · p. 3

Code Translation Methods and Evaluation Metrics

  • The paper directs readers to Appendix A.1 for details on code-translation methods and evaluation metrics.No further method or metric details are given in this section.Source: §2 · p. 3

3 The CodeTransOcean Benchmark

CodeTransOcean is a program-level benchmark for code translation, with multilingual, niche-language, LLM-evaluation, and cross-framework datasets. It spans 270,507 samples, 45 programming languages, and four deep learning frameworks.

3 The CodeTransOcean Benchmark

  • 270,507 program-level samples cover 45 programming languages and four deep learning frameworks, with over 200K unit tests and no overlap with existing code translation datasets.The benchmark balances language and framework distributions.Source: §3 · p. 4

3.1 Multilingual Code Translation

  • 30,419 MultilingualTrans programs cover eight popular languages, with data described as balanced, practical, and reviewed by its source website.The eight languages are C, C++, C#, Java, Python, Go, PHP, and Visual Basic.Source: §3.1 · p. 4
  • 236,468 NicheTrans programs cover translation from 37 niche languages to eight popular languages, making it the first dataset for these language pairs.The dataset is intended to support modernization and evaluation of neural-model generalizability.Source: §3.1 · pp. 4–5
  • LLMTrans contains 350 programs translating seven popular languages to Python, with recorded execution results for evaluating LLM code translation.AutoTransExecuter automates translation, execution, debugging, and success-rate calculation, but supports translation only to Python.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 Cross-framework Code Translation

  • Automated cross-framework translation re-implements code across frameworks, producing readable code for secondary development and flexibility to combine framework advantages.The paper contrasts this with adaptive technologies that rely on agreed data formats or convert only final models through computation graphs.Source: §3.2 · p. 5
  • DLTrans defines cross-framework code translation across PyTorch, TensorFlow, MXNet, and Paddle in 12 directions, with coarse- and fine-grained subsets.The subsets contain 408 program-level and 3,270 function- or class-level samples; experiments use only the coarse-grained subset.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate multilingual training strategies, introduce Debugging Success Rate@K (DSR@K), and examine ChatGPT’s code-translation performance and prompting strategies.

4.1 Multilingual Modeling

  • One-to-Many ranks highest across all language pairs, while One-to-One is the baseline; One-to-Many also outperforms that baseline across the experiments.The four strategies rank One-to-Many > Many-to-Many > Many-to-One > One-to-One for all pairs; this ordering also holds for high- and low-resource pairs.Source: §4.1 · pp. 6–7
  • High-resource languages use shared information across source languages more effectively, while low-resource languages are less equipped to handle added uncertainty and noise from shared parameters.Low-resource languages often need more source-language data to realize the benefits of multilingual modeling.Source: §4.1 · p. 6
  • Swapping source and target languages for data augmentation and training a bidirectional model significantly improves translation accuracy on DLTrans and NicheTrans.The experiments use Many-to-Many; Table 4 reports EM, BLEU, and CodeBLEU across three random seeds, while Table 5 reports NicheTrans BLEU for one-way and two-way training.Source: §4.1 · p. 6, Table 4 · p. 7, Table 5 · p. 7

4.2 Debugging Success Rate@K

  • Existing EM, BLEU, and CodeBLEU metrics rely on surface-form matching and cannot reliably assess functional correctness for program-level translation.The authors motivate DSR@K as closer to software development practice, where developers test and debug completed code.Source: §4.2 · p. 7
  • DSR@K measures the percentage of samples that execute with expected results within K LLM generation and debugging attempts.DSR@0 also supports program-level code-translation evaluation without debugging rounds.Source: §4.2 · p. 7

4.3 ChatGPT for Code Translation

  • The straightforward baseline prompt outperforms tested prompt variants, and Divide-and-Conquer significantly degrades performance; varying temperature or top_p alone does not notably change it.The authors hypothesize that dividing code removes global context needed to account for structure and variable configurations.Source: §4.3 · p. 8
  • 3% absolute DSR gain follows the first self-debugging round; gains continue but begin to plateau after the second round.ChatGPT receives compiler feedback and retries translation until reaching the predefined debugging-round limit.Source: §4.3 · pp. 8–9
  • 1.72% absolute DSR@0 gain makes the same-source, same-target contextual example the strongest of the three one-shot strategies; Chain-of-Thought Experiment #2 instead declines by 6% absolutely.The one-shot strategies all improve over the zero-shot baseline, while applying Chain-of-Thought degrades executability.Source: §4.3 · p. 9, Table 8 · p. 9

5 Conclusion

CodeTransOcean provides a comprehensive benchmark for multilingual and cross-framework code translation. The paper reports benefits from multilingual modeling and examines ChatGPT’s code-translation and fuzzy-execution capabilities.

5 Conclusion

  • CodeTransOcean includes multilingual and cross-framework datasets for code translation.The conclusion presents the benchmark alongside resources, tools, and baselines for the field.Source: §5 · p. 9
  • Multilingual modeling improves translation quality for both high- and low-resource language pairs and improves training efficiency.The paper also reports that advanced ChatGPT strategies lead to significant performance gains, while fuzzy execution remains an area for future research.Source: §5 · p. 9

6 Limitations

The paper identifies limits in existing code-translation metrics and in automated execution-based evaluation. It also reports that ChatGPT remains weak at fuzzy-execution prediction.

6 Limitations

  • Match-based metrics overlook executability and functional equivalence across implementations, while execution-based metrics require costly tests and leave execution overhead and security risks unresolved.The paper argues that practical evaluation must address these limitations.Source: §6 · p. 10
  • AutoTransExecuter currently supports only Python as the target language, while language-specific dependencies and compiler configurations complicate broader automation.Fully automated execution also raises security concerns because malicious code could exploit the system.Source: §6 · p. 10
  • ExecuteStatus contains 300 executable MultilingualTrans samples and 300 non-executable samples from ChatGPT translation results, with execution status and, when executable, the output recorded.The dataset and AutoTransExecuter are used to evaluate ChatGPT’s predictions of executability and executed output.Source: §6 · p. 10
  • ChatGPT shows low accuracy, recall, and F1 for fuzzy execution, tends to predict code is executable about 88% of the time, and predicts execution results with 4% accuracy.The authors conclude that using ChatGPT for fuzzy execution is not yet practical, though they see potential for addressing deficiencies in current metrics.Source: §6 · pp. 10–11

A.1.1 Code Translation Methods

Code-translation methods have moved from manual rewriting to automatic compiler, rule-based, and neural approaches. The discussion describes trade-offs among these approaches and notes the growing role of LLMs.

A.1.1 Code Translation Methods

  • Manual rewriting is inefficient, costly, and error-prone, requiring developers to understand both languages and navigate programming structures and semantic challenges.It is presented as an early approach to code translation.Source: §A.1.1 · p. 14
  • Compilers and transpilers save time but may fail to preserve source-language features and behavior or capture code intent; rule-based methods depend heavily on complete rules and manual preprocessing.The paper describes rule-based translation as treating the task as program synthesis.Source: §A.1.1 · p. 14
  • Neural methods have become dominant, treating translation mainly as sequence-to-sequence generation; examples include pretrained models that use syntax, semantics, or code-intent objectives.The surveyed systems include CodeBERT, GraphCodeBERT, CodeT5, PLBART, and NatGen.Source: §A.1.1 · pp. 14–15
  • LLMs such as ChatGPT can understand and generate code sequences, and the paper describes their progress as advancing code translation.The passage notes ChatGPT’s instruction-following, in-context examples, and human-feedback optimization.Source: §A.1.1 · p. 15

A.1.2 Code Translation Metrics

The section distinguishes reference-matching metrics from execution-based metrics. It notes that both families have practical or evaluative limitations.

A.1.2 Code Translation Metrics

  • Match-based metrics compare translations with references, but they can miss executability and functional equivalence across different implementations.The section describes EM, BLEU, CodeBLEU, CrystalBLEU, CodeBERTScore, and Ruby, with some supporting only limited programming-language coverage or lacking an open-source codebase.Source: §A.1.2 · p. 15
  • Execution-based metrics compare generated-code results with expected results, but require executable test sets that can be costly to construct and run in isolated sandboxes.Examples include PASS@k, AvgPassRatio, computational accuracy, and CodeScore.Source: §A.1.2 · pp. 15–16

A.2.1 Data Sources & Licenses

CodeTransOcean draws multilingual and niche-language data from Rosetta Code and cross-framework deep-learning data from Dive into Deep Learning, with licensing constraints observed. The appendix describes dataset quality checks, translation challenges, modeling strategies, and ChatGPT prompt variants.

A.2.1 Data Sources & Licenses

  • Rosetta Code supplies MultilingualTrans and NicheTrans examples of solutions to identical tasks across programming languages, while DLTrans derives from Dive into Deep Learning.The authors state that they follow the respective CC BY-SA 4.0 and Apache-2.0 licenses.Source: §A.2.1 · p. 16

A.2.2 Data Processing

  • Data processing preserves original formats for compilability, removes duplicate files to reduce potential leakage, and manually filters cross-framework samples that fail requirements.For cross-framework Python examples, original formatting is retained.Source: §A.2.2 · p. 16

A.2.3 Data Quality

  • Manual assessment of 1K samples per dataset found compilability above 90% for MultilingualTrans samples with compilation requirements and verified functional equivalence of code pairs.The authors report these checks as evidence of dataset quality.Source: §A.2.3 · p. 16
  • The datasets span basic syntax, complex algorithms, and deep-learning construction, training, and inference, reflecting varied code styles and domains.The authors describe this diversity as covering a wide variety of real-world scenarios.Source: §A.2.3 · pp. 16–17

A.3 Specific Challenges in Implementing Cross-framework Translation

  • Cross-framework translation must address framework design differences, complex deep-learning components, inconsistent code structures, and compatibility across hardware and operating systems.Source: §A.3 · p. 17

A.6 Prompt Variations

  • ChatGPT prompt variants include role assignment, polite inquiry, and clarification that translated code should execute; one-shot examples vary whether languages match the requested direction.The one-shot experiments also include examples matching neither requested language.Source: §A.6 · p. 17, Paddle · p. 17, §A.7 · p. 18

A.4 Code Examples on Different Deep Learning Frameworks

  • The appendix illustrates masked softmax cross-entropy and Seq2Seq decoder implementations across PyTorch, TensorFlow, MXNet, and Paddle.The multilingual-modeling setup contrasts independent pairwise, one-to-many, many-to-one, and unified many-to-many models.Source: Figure 1 · p. 17, §A.5 · p. 17, Figure 2 · p. 19

A.8 Chain of Thought

  • Chain-of-thought variants ask ChatGPT to describe or understand source-code function, with later variants predicting outputs and requiring executable translations.A further variation combines output prediction with a same-direction one-shot example.Source: §A.8 · p. 18, Paddle · p. 19
Loading 2310.04951v2…