Source-linked AI summary
Exploring and Unleashing the Power of Large Language Models in Automated Code Translation
Zhen Yang, Fang Liu, Zhongxing Yu, Jacky Wai Keung, Jia Li, Shuo Liu, Yifan Hong, Xiaoxue Ma, Zhi Jin, Ge Li
TL;DR
Automated code translation remains costly and imperfect, while LLMs have not been exhaustively evaluated as alternatives to learning-based transpilers. The paper studies these models and introduces UniTrans, which uses generated test cases for augmentation, execution-based checking, and repair; experiments show substantial improvements across tested models and translation datasets.
Problem
Learning-based transpilers remain unsatisfactory for practical deployment and require expensive task-specific training, while LLMs’ potential for code translation has not been exhaustively explored.
Method
UniTrans generates test cases from source programs, uses them to augment translation and check execution, then repairs incorrect translations using execution feedback.
Results
UniTrans substantially improves GPT-3.5, LLaMA-13B, and LLaMA-7B across almost all six Python, Java, and C++ translation datasets.
Takeaways & Limitations
Test cases are critically important throughout UniTrans’s translation, checking, and repair lifecycle.
Takeaways & Limitations
The evaluation uses a limited choice of experimental models and acknowledges potential data leakage, especially for the closed-source GPT-3.5.
Abstract
from arXiv · showhide
Code translation tools (transpilers) are developed for automatic source-to-source translation. Although learning-based transpilers have shown impressive enhancement against rule-based counterparts, owing to their task-specific pre-training on extensive monolingual corpora. Their current performance still remains unsatisfactory for practical deployment, and the associated training resources are also prohibitively expensive. LLMs pre-trained on huge amounts of human-written code/text have shown remarkable performance in many code intelligence tasks due to their powerful generality, even without task-specific training. Thus, LLMs can potentially circumvent the above limitations, but they have not been exhaustively explored yet. This paper investigates diverse LLMs and learning-based transpilers for automated code translation tasks, finding that: although certain LLMs have outperformed current transpilers, they still have some accuracy issues, where most of the failures are induced by a lack of comprehension of source programs, missing clear instructions on I/O types in translation, and ignoring discrepancies between source and target programs. Enlightened by the above findings, we further propose UniTrans, a Unified code Translation framework, applicable to various LLMs, for unleashing their power in this field. Specifically, UniTrans first crafts a series of test cases for target programs with the assistance of source programs. Next, it harnesses the above auto-generated test cases to augment the code translation and then evaluate their correctness via execution. Afterward, UniTrans further (iteratively) repairs incorrectly translated programs prompted by test case execution results. Extensive experiments are conducted on six settings of translation datasets between Python, Java, and C++. Three recent LLMs of diverse sizes are tested with UniTrans, and all achieve substantial improvements.
1 INTRODUCTION
Automated code translation supports codebase migration but remains costly and error-prone with rule-based and learning-based transpilers. This paper studies LLMs and proposes UniTrans, which uses generated test cases to augment and repair translations.
- Source-to-source translation helps port software across programming languages for expanding business platforms.
- Rule-based transpilers require expertise in both languages and substantial time for designing rules, while producing poor readability and correctness.Migrating COBOL to Java reportedly required approximately $750 million and five years.
- The empirical study cleans a widely used translation dataset and compares five recent LLMs with three state-of-the-art learning-based transpilers.The compared transpilers are TransCoder, TransCoder-IR, and TransCoder-ST.
- UniTrans uses auto-generated test cases throughout translation to provide program information, check correctness, and support repair.Its three phases are test case generation, translation augmentation, and translation repair.
- 4.02% CA and 13.28% EM Acc are GPT-3.5’s average improvements with UniTrans; LLaMA-13B and LLaMA-7B also improve on both metrics.LLaMA-13B improves CA and EM Acc by 19.20% and 36.42%; LLaMA-7B improves them by 28.58% and 71.22%.
- The paper rigorously cleans the dataset, analyzes LLM prospects and limitations, and evaluates UniTrans quantitatively and qualitatively.The evaluation includes ablation studies, discussion experiments, and case studies.
2 BACKGROUND AND RELATED WORK
Prior automated code translation and testing approaches range from handcrafted rules and search heuristics to learning-based methods. These approaches commonly face language dependence, limited generality, or substantial training-resource demands.
- Rule-based translation approaches such as C2Rust and CxGo use manually crafted rules and are language-dependent and labor-intensive.Rules must cover functions, objects, and standard libraries for each language pair.
- Traditional test-generation tools use search-based heuristics but suffer limitations in test diversity and quantity.Examples include Randoop, EvoSuite, and MOSA.
- Learning-based test-generation and program-repair approaches improve expressiveness or diversity but require massive training resources.Learning-based repair tools also address limitations of handcrafted heuristics and templates.
3 MOTIVATION
The motivation study examines failed LLM translations and finds recurring logic, I/O, precision, and syntax problems. These findings motivate using test cases to clarify requirements, types, and translation errors.
- The study evaluates LLMs against learning-based transpilers on four translation directions using Computational Accuracy and Exact Match Accuracy.
- The researchers randomly sample 195 GPT-3.5 failed cases for a 95% confidence level and 5% confidence interval, retaining 174 after exclusions.The exclusions concern easily removable import statements or class wrappers.
- Logic failures comprise 20.41% to 51.92% of failures across datasets, indicating inadequate source-program comprehension as a primary weakness.
- Python-to-C++ translations have the most frequent I/O failures at 37.21%, reflecting difficulty inferring static types from dynamically typed code.
- Precision failures reach 28.85% when translating from statically typed to dynamically typed languages because data types and operations differ.
- Test cases are introduced as extra information to convey program requirements, annotate I/O types, and provide execution feedback for repair.
4 UNITRANS
UniTrans uses source-assisted test-case generation, test-case-augmented translation, execution-based checking, and repair of failed translations. Its pipeline connects generated inputs and target-language I/O information with execution results to guide correction.
- Overview: UniTrans has three phases: test-case generation, translation augmentation, and translation repair.The phases generate tests from source programs, use tests during translation and checking, then repair incorrectly translated programs.
- Test Case Generation: LLMs generate candidate inputs from source programs before prepared inputs are turned into target-language test cases.The input-generation prompt requests ten differentiated valid input groups, while target-language test cases include explicit variable types and output types.
- Translation Augmentation: Translation augmentation prompts combine source programs with prepared test cases and require target programs to pass all given tests.Programs passing every prepared test are returned; failures proceed to repair after preliminary execution.
- Translation Repair: Execution results distinguish compilation/runtime errors from logic discrepancies and provide error lines, messages, or output mismatches for repair.An Error Analyzer extracts this information, accounting for prefix lines added by testing templates when locating buggy lines.
- Translation Repair: Repair prompts target compilation/runtime failures with a specified buggy line, while logic-error prompts use failed tests and error messages without requiring a known line.Only the first failed test is fed for repair by default, and repaired programs may be repeatedly evaluated up to a maximum iteration.
5 EXPERIMENTAL SETTING
The experiments compare recent LLMs with learning-based transpilers on cleaned Python, Java, and C++ translation data using exact-match and execution-based metrics. Additional research questions test UniTrans across models and translation pairs, component contributions, test-case counts, and input generation.
- Dataset and Models: The study uses 568 translation samples with evaluation-purpose unit tests from a 948-function Python, Java, and C++ dataset.The retained tests cover 464 Python, 482 Java, and 467 C++ cases.
- Dataset and Models: Five LLMs and three learning-based transpilers are included in the empirical study.The LLMs are GPT-3.5, three LLaMA sizes, and CodeGen; the transpilers are TransCoder, TransCoder-IR, and TransCoder-ST.
- Evaluation Metrics: Computational Accuracy measures matching execution results, whereas Exact Match Accuracy measures literal equality with ground-truth translations.CA captures semantic equivalency under the test suite, while EM Acc is described as a lower bound on transpiler effectiveness.
- Implementation: LLM evaluation uses nucleus sampling with top_p=0.95, temperature=0.8, and 10 samples per translation.GPT-3.5 is accessed through OpenAI’s API, while LLaMA and CodeGen use replication packages and HuggingFace weights.
- Research Questions and Evaluation Methodology: UniTrans experiments use LLaMA-7B, LLaMA-13B, and GPT-3.5 across six translation datasets between Python, Java, and C++.The evaluation also studies component contributions, test-case quantity, and valid input generation, including previously unexamined Python–Java directions.
6 EXPERIMENTAL RESULTS
LLMs show strong code-translation performance, with GPT-3.5 outperforming alternatives on average, but they still have accuracy issues. UniTrans consistently improves diverse LLMs through test-case augmentation and iterative repair, while repair rounds and test-case counts require model-specific choices.
- RQ1: GPT-3.5 achieves 87.92% CA and 18.04% EM Acc on average across four translation datasets, consistently outperforming the compared models.LLaMA-33B also outperforms state-of-the-art learning-based transpilers on most datasets and metrics.
- RQ1: LLMs show promising translation performance against state-of-the-art transpilers, but even GPT-3.5 does not produce perfect translations.The empirical study therefore identifies both strong performance and remaining accuracy issues.
- RQ2: UniTrans consistently improves LLM code translation across diverse models and programming-language pairs, with statistically significant and non-negligible gains.The evaluation covers six translation datasets and includes the previously less-investigated Python–Java direction.
- RQ3: Both TAP and TRP improve translation ability, with smaller models receiving greater enhancement because they have more room for improvement.TAP generally contributes larger gains than TRP, especially for EM Acc.
- RQ4–RQ5: More test cases can provide broader information but also introduce redundancy and distraction, so larger models should receive more cases and smaller models fewer.LLMs generate valid inputs for over 90% of programs with few attempts, supporting the practical feasibility of UniTrans.
- RQ6: Iterative repair can fix more bugs but risks degeneration, so larger LLMs should use fewer repair rounds and smaller LLMs more rounds.Some repairs move programs farther from the ground truth and cause performance declines under the evaluation metrics.
7 DISCUSSION
UniTrans addresses several categorized LLM translation failures through test-case augmentation and iterative repair, while the study identifies validity threats involving data, dataset quality, model selection, and metrics.
- Failure Taxonomy Look Back: Test-case augmentation resolves 46.15% of I/O failures and 23 Logic failures, while also addressing Syntax and API mistakes.TAP uses test cases with I/O types to support program-logic and I/O-requirement comprehension.
- Failure Taxonomy Look Back: Translation repair addresses additional errors after augmentation, resolving 10 Syntax failures, or 47.62% of such mistakes.TRP repairs incorrectly translated programs based on the translated programs produced by TAP.
- Failure Taxonomy Look Back: Precision failures remain difficult: only 25% are solved in total by TAP and TRP.The authors report that both components still fail in certain cases, especially Precision failures.
- Threats to Validity: Potential data leakage threatens internal validity because LLM training sets may overlap with the testing set.This concern is especially difficult to assess for the closed-source GPT-3.5 model.
- Threats to Validity: External validity is constrained by possible residual noise or errors in the manually cleaned evaluation dataset.Four authors cross-checked cleaned results in pairs, and the cleaned dataset was released for public evaluation.
- Threats to Validity: The study’s limited model selection constrains generalization, despite covering multiple LLM families, sizes, and a closed-source model.The authors plan to include more relevant models in future experiments.
- Threats to Validity: Metric choice is a construct-validity threat, although CA, EM Acc, and PR measure lexical, semantic, and fine-grained correctness.The authors therefore characterize the evaluation as comprehensive.
8 CONCLUSION
The paper studies LLMs for codebase migration and proposes UniTrans, which combines test cases with translation and repair. It reports improved translation capabilities and identifies prompt design as a direction for further research.
- Conclusion: UniTrans combines test-case augmentation with bug repair to improve diverse LLMs’ code-translation capabilities.The framework is designed to enhance codebase-migration efficiency and reliability.
- Implications for practitioners: Practitioners can use prompts and auto-generated test-case execution without large-scale LLM fine-tuning.Users designate a few hyperparameters, including TAP’s test-case count and TRP’s repair rounds.
- Implications for researchers: Researchers can extend the three-step framework by exploring alternative prompt designs for test-case generation, translation augmentation, and translation repair.The paper states that prompts with potentially superior efficacy may exist.