PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code

Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, Reyhaneh Jabbarvand

arXiv:2308.03109v3cs.SE

TL;DR

Prior work has explored LLMs for code translation, but evidence cataloging their limitations is sparse despite the task requiring syntax and semantic preservation. This study evaluates LLMs, characterizes translation bugs, and tests context-rich prompting; translations succeed 11.94% on average, while prompt crafting raises success by 5.5% on average.

  • Problem

    Prior research has shown LLMs’ potential for code translation, but their limitations remain insufficiently understood for a task requiring both syntactic correctness and semantic preservation.

    Source: §1 · p. 2

  • Method

    The study evaluates general and code LLMs on benchmark and real-world translations, catalogs bugs, compares LLM and non-LLM approaches, and tests iterative prompts with failure context.

    Source: §1 · p. 2

  • Results

    11.94% of 43,379 translations succeeded on average, while prompt crafting increased success rate by 5.5% on average across the studied LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LLM- and non-LLM-based translation approaches have different strengths: transpilers retain broader application context and are deterministic, whereas LLMs are probabilistic.

    Source: §7 · p. 10

    Correctness was measured with provided test cases, so weak suites may let buggy translations pass as correct.

    Source: §8 · p. 11

Abstract

from arXiv · show

Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation -- with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset -- consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs -- can help drive research in this area.

1 INTRODUCTION

Code translation must preserve source functionality, but the limitations of LLM-based translation remain insufficiently understood. This study evaluates LLMs and alternative approaches, catalogs translation bugs, and tests whether added context can improve results.

1 INTRODUCTION

  • The study evaluates seven LLMs across 1,700 samples from three datasets and two real-world projects spanning C, C++, Go, Java, and Python.The models include general and code LLMs.Source: §1 · p. 2
  • 11.94% of translations succeeded on average (median 5.3%); GPT-4 reached 47.3% and StarCoder 14.5%, while real-world success was 8.1% for GPT-4 and 0% for other models.Success was measured against tests provided with the code samples.Source: §1 · p. 2
  • Manual analysis of 1,725 GPT-4 buggy translations produced a taxonomy of 15 categories in five groups and identified challenges involving data types, APIs, and language-specific features.The study also compares LLM- and non-LLM-based approaches, finding distinct advantages that suggest combining them.Source: §1 · p. 2
  • Prompt crafting adds context from failed executions or tests iteratively, increasing success by 5.5% on average and by 12% for GPT-4.The authors note considerable remaining scope for improvement, likely through combining program analysis techniques and LLMs.Source: §1 · p. 2
  • The study contributes a systematic bug taxonomy, real-world project evaluation, prompt-crafting investigation, and comparison with non-LLM approaches; its artifacts are publicly available.Some tools, including mppSMT, Tree2Tree [29], and Sharpen, were excluded from comparisons because their tools or artifacts were unavailable.Source: §1 · p. 2

2 EMPIRICAL SETUP

The empirical setup selects general and code LLMs and five programming languages, using both benchmark datasets and real-world projects. Translation correctness is evaluated with supplied tests, while the general-model shortlist reflects the leaderboard at experiment time.

2 EMPIRICAL SETUP

  • The study selected three code LLMs released in 2023, three general models ranked within the top models of 20B parameters or less, and GPT-4.The parameter limit reflected available computing resources, and leaderboard selection was based on its ranking at experimentation time.Source: §2 · p. 3
  • The five selected languages—C, C++, Go, Java, and Python—were chosen using popularity, paradigm diversity, and dataset availability criteria.The language count also kept manual taxonomy construction manageable.Source: §2 · p. 3
  • The dataset combines prior-study datasets, viewed as potentially less challenging, with real-world projects intended to assess translation in more complex settings.The real-world projects were Apache Commons CLI and Python Click, broken into classes and files to fit LLM context windows.Source: §2 · p. 3
  • CodeNet and Avatar use input-output tests, while EvalPlus and the real-world projects use unit tests; real-world translations did not reach test execution because generated code was syntactically incorrect.EvalPlus Python tests were manually translated and verified as JUnit tests.Source: §2 · p. 3

3 LLM-BASED CODE TRANSLATION

The study evaluates LLM translation across 31 language pairs and examines why translations fail. Most unsuccessful translations produce compilation errors, with outcomes varying by programming language.

3 LLM-BASED CODE TRANSLATION

  • The evaluation prompts provide natural-language instructions, source and target languages, and source code; success requires compilation, runtime checks, and passing existing tests.The study prompted LLMs on 6,197 problems across 31 translation pairs.Source: §3 · p. 3, §3.1 · p. 3

3.1 Effectiveness of LLMs in Code Translation

  • CodeGeeX’s pass@1 on HumanEval-X drops from 25.6% with its template to 0.02% with the study’s template, which omits translated-code information.That information includes imports, class declarations, and method signatures, which the authors say are typically unavailable and non-trivial to compute.Source: §3.1 · p. 3
  • Across all models, the average number of tests per sample correlates with unsuccessful translation, with correlation coefficients ranging from 0.64 to 0.85.Source: §3.1 · p. 4
  • LLMs struggle with real-world projects, which the authors attribute to complex dependencies and language features, and to translating files or methods in isolation.Go also produces more compilation errors, which the authors link to strict syntax constraints such as forbidding unused variables or imports.Source: §3.1 · p. 4

3.2 Outcome of Unsuccessful Translations

  • Unsuccessful translations are classified as compilation errors, runtime errors, functional errors, or non-terminating executions.Source: §3.2 · p. 4
  • 77.8% of unsuccessful translations produce compilation errors and 13.4% produce functional errors; Go and C++ are stricter syntactically, while Python is easier for LLMs to generate correctly.Functional errors occur when translated code runs but fails tests, indicating that behavior is not preserved.Source: Figure 2 · p. 4, Table 3 · p. 4, §3.2 · p. 4

4 LLM-BASED TRANSLATION BUGS

The section develops a taxonomy of LLM translation bugs from manual analysis of unsuccessful translations, grouping the errors by root cause. The categories cover language differences, missing or altered logic and dependencies, data handling, and model-specific constraints.

4 LLM-BASED TRANSLATION BUGS

  • The taxonomy’s analysis focuses on unsuccessful translations and aims to identify their root causes and characterize bug categories across language pairs.Source: §4 · p. 4, §4.1 · p. 5

4.1.1 Methodology.

  • The taxonomy was built by eight labelers in two phases across 31 translation pairs, taking about 630 person-hours and yielding 15 categories in five groups.The team consolidated an initial taxonomy from Java-to-Python translations, then examined the remaining pairs and added categories when needed.Source: §4.1.1 · p. 5

4.1.2 Syntactic and semantic differences between the source and the target languages.

  • Five categories capture failures to handle source–target language differences, including target-language rule violations, copied syntax, API mismatches, and incorrect operators.Examples include retaining C++’s atan2l in Go, mapping Java substring() to Go strings.IndexByte(), and translating Python floor division as Java division.Source: §4.1.2 · p. 5

4.1.3 Dependency and logic bugs in the translated code.

  • Dependency and logic bugs include missing imports or definitions, incorrect loops and conditions, added or removed logic, and mismatched behavior after replacing an API call.Examples include omitting a called method’s definition, an off-by-one array slice, and translating an immutable Java list into a mutable Python list.Source: §4.1.3 · pp. 5–6

4.1.4 Data-related Bugs.

  • Data-related bugs arise from wrong assumptions about input data, incorrect types, parsing errors, and output-format differences.The examples include reading too few input values and inserting a space into output that had none in the Java source.Source: §4.1.4 · p. 6

4.1.5 D: Model-Specific Bugs.

  • Model-specific and experiment-setup failures include interleaved natural language, token-limit overruns, and memory issues; these are excluded from the taxonomy as not LLM-introduced bugs.The reported symptoms include compilation errors or no output.Source: §4.1.5 · p. 6

4.2 Prevalence of LLM-based Translation Bugs

The prevalence analysis finds that data-related bugs make up the largest reported share, while syntax and semantics differences and dependency or logic errors are also substantial. Input parsing is especially prominent in the benchmark datasets that use input streams.

4.2 Prevalence of LLM-based Translation Bugs

  • 54% of data-related bugs, or 18.1% of all bugs, stem from incorrect input parsing; this bug was less frequent in EvalPlus and absent from the real-world projects.The authors attribute its prevalence in part to CodeNet and Avatar programs reading input streams.Source: §4.2 · p. 6
  • 34.3% of data-related bugs, or 11.5% of all bugs, involve incorrect target data types, including mismatches in type behavior or type systems across languages.A Python float translated to Java Float failed on large test data because Python float uses 64-bit precision and Java Float uses 32-bit precision.Source: §4.2 · pp. 6–7
  • 33.5% of all bugs are data-related, particularly in Python-to-C, C++, and Go translations, where they constitute over 43%, 33%, and 37% of bugs, respectively.The analysis covers attempted translations across 31 language pairs and includes real-world projects.Source: §4.2 · p. 6, Table 4 · p. 7
  • 30.5% of translation bugs arise from source–target syntactic and semantic differences; almost 80% of that group violate target-language requirements.Examples include using a reserved C++ keyword and preserving an incompatible method declaration order.Source: §4.2 · p. 7
  • 24.2% of translation bugs involve incorrect logic or missing target-language dependencies, with missing imports the dominant category.Missing imports account for 35.5% of bugs in this group and 8.6% of all bugs.Source: §4.2 · p. 7

4.3 Translation Bugs in Real-world Projects

Real-world projects exhibit the same broad translation-bug categories as crafted benchmarks, but their bug frequencies and challenges differ. Project translation also exposes issues involving language features, dependencies, and code complexity.

4.3 Translation Bugs in Real-world Projects

  • Observed project bugs include removed or added logic, missing imports, API-behavior mismatches, incorrect data types, broken inheritance, and incorrect exception handling.Annotations and decorators also pose challenges; Click's contextmanager decorator must be translated appropriately to preserve automatic resource release.Source: §4.3 · pp. 7–8
  • Real-world code is more complex than crafted benchmarks, and GPT-4 produced nine Click cases explaining translation infeasibility or difficulty and partial method translations.The study notes that even longer context windows may not let LLMs capture dependencies among methods implementing class logic.Source: §4.3 · p. 7
  • GPT-4 sometimes renames overloaded Java methods for Python without updating call sites, or preserves duplicate names so Python overrides earlier definitions.Both patterns can break functionality or call relations.Source: §4.3 · p. 8
  • GPT-4 translated Commons CLI to Python more successfully than Click to Java, but the authors caution that limited observations and project complexity constrain generalization.They also identify language features and API ecosystems as possible influences on translation success.Source: §4.3 · p. 8
  • Real-world projects share taxonomy categories with crafted benchmarks, but model-specific constraints (29.4%), missing imports (27.4%), and removed logic (15.7%) are most prevalent there.Data, syntactic, and semantic differences are more common in crafted benchmarks; limited context and missing application knowledge can contribute to project-specific bugs.Source: §4.3 · p. 8

5 LLM- VS. NON-LLM-BASED TRANSLATION

The comparison finds that relative performance depends on the language pair: LLMs outperform the non-LLM approach for C-to-Go, while C2Rust leads for C-to-Rust. The approaches also differ in bug profiles and generated-code safety and idiomaticity.

5 LLM- VS. NON-LLM-BASED TRANSLATION

  • For Java to C#, JavaToCSharp produced no compilable translations and requires heavy library-API rewriting, while GPT-4 and StarCoder achieved 54.1% and 18.45% success rates.The tool's developers confirmed it was not built for full code migration.Source: §5 · p. 8
  • For C-to-Go, GPT-4 achieves a 10% higher success rate than the non-LLM approach; for C-to-Rust, the non-LLM approach translates 95% of code, 34% better than the best-performing LLM.Source: §5 · p. 8
  • GPT-4's C-to-Go bugs were mostly target-language violations, while CxGo bugs more often involved input parsing or API-behavior mismatches.The reported shares were 71.8% for GPT-4 unused imports, variables, and other target-language violations; CxGo's leading categories were each 26.7%.Source: §5 · p. 9
  • For C to Rust, C2Rust achieved 95% success versus GPT-4's 61%; GPT-4 bugs included compilation, runtime, and test failures, while most C2Rust failures were compilation errors.The reported GPT-4 bug shares were 50%, 39.3%, and 10.7%, respectively.Source: §5 · p. 9
  • GPT-4's C-to-Rust translations were safer and more idiomatic than C2Rust's, whose generated code was all unsafe; only three GPT-4 translations were unsafe.The cited risks include bypassed Rust safety checks, external C functions, and unrestricted unsafe blocks.Source: §5 · p. 9

6 MITIGATING TRANSLATION BUGS

The paper iteratively prompts LLMs with context about prior failed translations and checks each patch for success. The approach increased success rates, although bug outcomes sometimes persisted, changed type, or worsened.

6 MITIGATING TRANSLATION BUGS

  • Revised prompts include the source and prior prompt, incorrect translation and error-specific details, translation instructions, and optionally expected input-output behavior for functional errors.Error details may include a stack trace, compiler log, incorrect output, or an infinite-loop message.Source: §6 · p. 9
  • Iterative prompt crafting increased success rates by 12.33% for GPT-4, 3.55% for StarCoder, 2.65% for CodeGen, and 1.97% for Llama 2 at iter1.GPT-4 improved by a further 1.7% at iter2; the evaluation excluded CodeGeeX because its prompt template was rigid and stopped iterations when overall gains fell below 5%.Source: Figure 6 · p. 10, §6 · p. 10
  • At iter1, GPT-4 fixed 58% of compilation errors completely, while 38% became other translation bugs; runtime, functional, and non-termination errors were fixed less often.The corresponding fixed shares were 45%, 48%, and 43%; at iter2 most outcomes remained unchanged, and some translations degraded or acquired new bugs.Source: Figure 7 · p. 10, §6 · p. 10
  • The iterative method updates each prompt with information from the previously failed translation, verifies the resulting patch, and can continue until a fixed iteration limit or success-gain threshold.The outcome at each iteration is called a translation patch.Source: §6 · p. 10

7 DISCUSSION

The discussion contrasts LLM and non-LLM translation strengths and identifies challenges in scaling translation to real-world projects. It suggests combining context-providing techniques with decomposition and further LLM improvements.

7 DISCUSSION

  • Transpilers can access broader application context and are generally deterministic, while LLMs are probabilistic and tend to produce more natural, human-like code.Non-LLM approaches may neglect target-language idioms, harming readability, maintainability, and sometimes security.Source: §7 · p. 10
  • Real-world translation needs techniques that scale while preserving relevant inter-file context, potentially through program analysis or dependency-aware decomposition and recomposition.Files in real-world projects are not standalone programs.Source: §7 · p. 10
  • Different target-language API ecosystems can leave no suitable equivalent for a source API, motivating investigation of code summarization combined with code synthesis.The discussion identifies library API calls as a significant challenge in real-world translation.Source: §7 · p. 10
  • The authors suggest coherent prompts informed by program analysis for closed-source models, and fine-tuning or specialized models for open-source translation improvements.The proposed specialized models would address distinct aspects of the translation process, particularly bug categories.Source: §7 · p. 11

8 THREATS TO VALIDITY

The study addresses threats to external, internal, and construct validity through choices of languages, models, datasets, repeated review, and test-based correctness measures.

External Validity

  • External validity is bounded by the selected programming languages, LLMs, datasets, and non-LLM tools, although the study covers five languages, multiple datasets, two real-world projects, and several tools.The selections were guided by language popularity and dataset characteristics.Source: §8 · p. 11

Internal Validity

  • Each translation task was run once, so LLM nondeterminism may change success rates, though the authors say this does not change their focus on translation-bug characteristics.Prompt-template sensitivity was mitigated by following practices from the respective artifacts, papers, and reports.Source: §8 · p. 11
  • Manual bug labels lack a formal inter-rater reliability metric, but two labelers resolved discrepancies by discussion and an author reviewed all labels for consistency.The authors also tested automation scripts, spot-checked results, and made artifacts available for review and replication.Source: §8 · p. 11

Construct Validity

  • Test-based correctness can misclassify buggy translations as correct when suites are weak, a particular concern for CodeNet’s single test case per sample.The authors describe the other datasets’ test suites as fairly rigorous.Source: §8 · p. 11

9 RELATED WORK

Prior code-translation work includes program-analysis tools and learning-based methods, while this study focuses on bugs introduced by LLM translation and compares LLM and non-LLM approaches.

Code Translation and Synthesis

  • Program-analysis transpilers translate between languages, while learning-based approaches range from statistical and neural methods to deep-learning, unsupervised, and phase-based methods.Examples include C2Rust, CxGo, Java-to-C# tools, and translation methods [29].Source: §9 · p. 11
  • Recent code-translation work uses LLMs such as GPT-4 and CodeGeeX [91], but the authors identify no prior study of bugs LLMs introduce during translation.The passage also lists StarCoder [52], PolyCoder [86], SantaCoder [24], CodeGen [58], BLOOM [67], CodeT5 [77], CodeX [27], and Llama 2.Source: §9 · p. 11

Bug and repair study

  • Unlike studies of software bugs, model-related bugs, or LLM-based bug fixing, this work studies translation-introduced bugs and compares translation effectiveness with non-LLM techniques.The passage cites software-bug and model-bug studies [40] [41], LLM repair [32] [45] [84] [85], and other repair approaches [83] [88].Source: §9 · p. 11

10 CONCLUDING REMARKS

The study evaluates LLM code translation across languages, benchmarks, and real-world projects, then analyzes translation failures and tests iterative prompt crafting. It concludes that current LLMs do not yet effectively automate code translation, particularly for complex real-world projects.

10 CONCLUDING REMARKS

  • State-of-the-art LLMs are not yet effective at automating code translation, particularly for complex real-world projects.The study also compares the strengths and weaknesses of existing non-LLM-based translation techniques.Source: §10 · p. 11
  • Manual analysis identified 15 root causes of unsuccessful translations, and the study presented and evaluated an iterative prompt-crafting technique to investigate the effect of added context.Source: §10 · p. 11
  • The authors aim to investigate how software-engineering and programming-language techniques, including added context, can mitigate translation bugs and advance real-world application translation.Source: §10 · p. 11

Related papers

Loading 2308.03109v3…