Source-linked AI summary
CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Execution
Prithwish Jana, Piyush Jha, Haoyang Ju, Gautham Kishore, Aryan Mahajan, Vijay Ganesh
TL;DR
Existing LLM-based translation methods lack a proactive approach to ensuring outputs compile reliably or are substantially functionally equivalent to their inputs. CoTran incorporates compiler and symexec-based testing feedback into reinforcement-learning fine-tuning of whole-program translators. For J2P, CoTran gets +9.62% FEqAcc compared to PPOCoder [38].
Problem
Existing LLM-based translation methods lack a proactive approach to ensuring outputs compile reliably or are substantially functionally equivalent to their inputs.
Source: §1 · p. 1Method
CoTran incorporates compiler and symexec-based testing feedback into reinforcement-learning fine-tuning of whole-program translators.
Source: §1 · p. 1Results
Takeaways & Limitations
The paper concludes that compiler and symbolic feedback, especially with interleaved RL+SFT, improve translation quality and produce more compilable, functionally equivalent code.
Source: §6 · p. 7Testing cannot guarantee functional equivalence, though sufficient testing can establish inequivalence.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
In this paper, we present an LLM-based code translation method and an associated tool called CoTran, that translates whole-programs from one high-level programming language to another. Existing LLM-based code translation methods lack training to ensure that the translated code reliably compiles or bears substantial functional equivalence to the input code. In our work, we fine-tune an LLM using reinforcement learning, incorporating compiler feedback, and symbolic execution (symexec)-based testing feedback to assess functional equivalence between the input and output programs. The idea is to guide an LLM during fine-tuning, via compiler and symexec-based testing feedback, by letting it know how far it is from producing perfect translations. We conduct extensive experiments comparing CoTran with 14 other code translation tools, including human-written transpilers, LLM-based translation tools, and ChatGPT. Using a benchmark of over \num{57000} code pairs in Java and Python, we demonstrate that CoTran outperforms the other tools on relevant metrics such as compilation accuracy (CompAcc) and functional equivalence accuracy (FEqAcc). For example, in Python-to-Java translation, CoTran achieves 48.68% FEqAcc and 76.98% CompAcc, whereas the nearest competing tool (PLBART-base) gets 38.26% and 75.77% respectively. Additionally, CoTran, built on top of CodeT5, improves FEqAcc by +14.89% and CompAcc by +8.14% for Python-to-Java (resp., +12.94% and +4.30% for Java-to-Python).
1 Introduction
CoTran targets whole-program translation, where function-level translation is insufficient and existing LLM methods do not proactively ensure compilability or functional equivalence. It uses compiler and symbolic-execution testing feedback during reinforcement-learning fine-tuning, and reports improved translation accuracy and contributions including a tokenizer and dataset.
1 Introduction
- Existing LLM translation methods lack a proactive approach to ensure translated code compiles reliably or is substantially functionally equivalent to its input.The paper identifies this as a barrier to wider adoption of LLM-based code translation.Source: §1 · p. 1
- CoTran uses compiler feedback for non-compiling outputs and symexec-based testing feedback for outputs that fail test-suite equivalence, incorporating both in reinforcement-learning fine-tuning.It jointly fine-tunes back-to-back source-to-target and target-to-source LLMs to support functional-equivalence checking.Source: §1 · p. 1, Figure 1 · p. 2
- +9.62% FEqAcc for J2P versus PPOCoder [38] and +10.42% for P2J versus PLBART-base [1] are the reported nearest-tool comparisons.The paper also reports that CoTran outperforms all tools of similar size for both translation directions.Source: §1 · p. 2
- Functional equivalence is limited to equivalent results with respect to a test suite, not a general proof of program equivalence.The paper notes that program equivalence is undecidable in general.Source: §1 · p. 2
- CoTran introduces a specialized keyword tokenizer, with reported FEqAcc gains of +3.57% for J2P and +6.62% for P2J.It also reports CompAcc gains of +3.28% for J2P and +4.79% for P2J.Source: §1 · p. 2
- AVATAR-TC contributes 57,000+ Java–Python code pairs with human-written test cases for evaluating functional equivalence.The dataset fixes faulty code pairs from AVATAR and adds problem-setter test cases collected from coding platforms.Source: §1 · p. 2
2 Related Work
Related work spans handcrafted transpilers, neural and transformer-based translation models, and LLM translation methods that use compiler or testing signals. CoTran distinguishes its feedback as fine-grained and uses back-to-back translation for symexec-based equivalence checking.
2 Related Work
- Rule-based transpilers rely on human-written rules and vary in which language constructs they handle; some disclaim that translations may not compile or run readily.Examples include java2python, py2java, and the commercial TSS code converter.Source: §2 · p. 2
- TransCoder-ST uses unit tests to create synthetic equivalent-code pairs, while PPOCoder [38] uses Boolean compiler feedback and CodeBLEU-inspired reward signals.Earlier back-to-back translation was not used for symexec-based functional-equivalence checking.Source: §2 · p. 3
- CoTran’s compiler feedback is fine-grained, capturing distance from compilable output rather than only whether compilation succeeds.The related-work discussion also distinguishes CoTran’s use of back-to-back translation for symexec-based equivalence checking.Source: §2 · p. 3
3 The CoTran Method
CoTran trains forward and backward translators with supervised learning followed by interleaved reinforcement-learning and supervised updates using compiler and symbolic-execution feedback. Its feedback scores compilability and round-trip test outcomes, while parameter-efficient fine-tuning limits the trainable parameters.
3 The CoTran Method
- The keyword tokenizer adds source- and target-language keywords, operators, and Python newline and indentation tokens to a modified RoBERTa vocabulary.This makes each such special token, keyword, or operator a single token ID.Source: §3 · p. 3
3.1 Definitions: CE Loss, CF, and SF
- Compiler feedback scores syntax using the first-error position, while symbolic-execution feedback tests whether generated round-trip code passes unit tests.Compiler feedback also accounts for translation length; symbolic execution generates tests for source functions and checks them on the reconstructed code.Source: §3.1 · pp. 3–4
- Testing can establish inequivalence with sufficient testing, but cannot guarantee functional equivalence between programs.The paper makes no claim to establish program equivalence.Source: §3.1 · p. 3
3.2 The CoTran Training Loop
- Separate supervised fine-tuning first trains the forward and backward models on their respective translation tasks before joint back-to-back fine-tuning.The supervised phase minimizes cross-entropy loss for each model.Source: §3.2 · p. 4
- CoTran jointly fine-tunes forward and backward translators with interleaved PPO reward updates and Adam-based cross-entropy updates.The reward combines symbolic-execution and compiler feedback, with a KL-divergence penalty; original model parameters are frozen except for LoRA layers.Source: §3.2 · p. 4
4 The AVATAR-TC Dataset
AVATAR-TC is a Java–Python whole-program dataset with human-written test cases, built from cleaned code pairs collected from programming platforms. Its 57,368 pairs are split by problem to support out-of-distribution testing.
4 The AVATAR-TC Dataset
- The dataset pairs whole programs with human-written test cases and, to the authors’ knowledge, is the first large-scale dataset of its kind with compilability ensured and thorough testing.The human-written test cases are used for evaluation, not for training CoTran variants.Source: §4 · pp. 4–5
- AVATAR-TC contains 57,368 Java–Python pairs across 9,198 problems, split into train, validation, and test sets at a 76:5:19 ratio.No problem overlaps across splits; validation and test have a one-to-one pair–problem mapping.Source: §4 · p. 5
- The authors collected code and test cases from competitive-programming websites and platforms, then manually corrected minor faults and discarded pairs with major issues.They retained the AVATAR split by problem statement, removing some pairs during cleaning.Source: §4 · p. 5
5 Experiments
The experiments compare CoTran and 14 other tools on Java–Python translation, using compilation, functional-equivalence, and additional code-quality metrics. The reported findings favor RL-based feedback training, while highlighting differences between translation directions and whole-program versus function-level translation.
5.1 Experimental Setup and Competing Tools
- The setup compares supervised tools and CoTran variants on the same 1,746 AVATAR-TC test programs, with supervised tools and CoTran variants also fine-tuned on its training set.The comparison includes human-written transpilers, unsupervised and supervised LLM tools, and ChatGPT.Source: §5.1 · p. 5
5.2 Evaluation Metrics
- Evaluation includes BLEU, CodeBLEU, exact match, CompAcc, FEqAcc, errPos1st, EpC, and f_crate, measuring overlap, compilation, test-based equivalence, error location, and errors per code.f_crate is the percentage ratio of FEqAcc to CompAcc.Source: §5.2 · p. 5
5.3 Analysis of Empirical Results and Ablation Study
- RL-based methods outperform the tried SFT schemes, and the proposed compiler reward detects small translation changes that Boolean, syntactic-match, and dataflow-match rewards may miss.For the Java integer-reversal example, several existing rewards remain constant across substantial portions of the truncation range.Source: §5.3 · p. 6, Figure 2 · p. 6
- +12.94% and +14.89% in FEqAcc, and +4.30% and +8.14% in CompAcc, are the J2P and P2J gains, respectively, for CoTran + CF + SF with interleaved RL+SFT.The experiments compare CoTran with 14 other tools; Table 2 covers J2P and P2J, but the supplied caption does not provide its cell values.Source: §5.3 · pp. 5–6, Table 2 · p. 6
- Interleaving RL with SFT improves translation performance over RL-only training, while adding kw-Tok increases FEqAcc and CompAcc for both translation directions.kw-Tok contributes FEqAcc gains of +3.57% for J2P and +6.62% for P2J, and CompAcc gains of +3.28% and +4.79%, respectively.Source: §5.3 · p. 6
- For P2J, CoTran + CF + SF (RL+SFT) lowers EpC to 0.71 ± 2.15, versus 0.95 ± 2.58 for CodeT5-base and 10.98±9.67 for ChatGPT.The reported measure is mean ± standard deviation of compilation errors per translated code.Source: §5.3 · p. 7, Figure 3 · p. 7
- For P2J, CoTran + CF + SF (RL+SFT) reaches 63.24% f_crate, compared with 29.69% for CodeGPT, 50.49% for PLBART-base, and 49.08% for CodeT5-base.The method ranks among the top two on f_crate across each of the seven AVATAR-TC sub-datasets; G-CodeJam is small and all tools perform similarly poorly there.Source: §5.3 · p. 7, Figure 4 · p. 7
- Function-level TransCoder-based tools perform poorly on whole-program translation: their J2P FEqAcc is 0.46%, and none of their P2J translations compile.The authors attribute this limitation to training on function-level translations.Source: §5.3 · p. 7
6 Conclusion and Future Work
CoTran uses compiler and symbolic-execution feedback during fine-tuning, with interleaved RL and SFT improving code-translation quality. It outperforms state-of-the-art tools on compilability and functional equivalence.
6 Conclusion and Future Work
- CoTran outperforms state-of-the-art tools in producing compilable and functionally equivalent code.The paper plans to extend CoTran to translate legacy code to modern languages.Source: §6 · p. 7
- Interleaved RL- and SFT-based training with compiler and symbolic feedback significantly improves code-translation quality.The paper highlights fine-grained feedback that indicates proximity to an ideal solution rather than Boolean yes/no directives.Source: §6 · p. 7
- The paper identifies symbolic feedback and fine-grained compiler feedback as effective components of its approach.Its conclusion emphasizes symbolic feedback and feedback that indicates proximity to an ideal solution.Source: §6 · p. 7
A.1 Reproducibility
The authors provide the CoTran code and AVATAR-TC dataset in a repository, along with translations from several tools and instructions for running the code.
A.1 Reproducibility
- The CoTran code and AVATAR-TC dataset are available in the project repository.The repository includes Python-to-Java and Java-to-Python translations.Source: §A.1 · p. 9
- The repository includes P2J and J2P translations generated by state-of-the-art translators and CoTran variants.Source: §A.1 · p. 9
- A README describes the repository structure, dependencies, and instructions for running the code.Source: §A.1 · p. 9
A.2 CoTran Implementation and Design Choices
CoTran’s implementation uses generated tests to compare translated methods and parameter-efficient training. AVATAR-TC supplies whole-program Java–Python pairs that are compiled and thoroughly tested.
A.2 CoTran Implementation and Design Choices
- For Java inputs, Symflower generates JUnit tests per method and checks them on the output method with the highest Jaccard similarity.This matching accommodates method names that differ while methods exhibit similar input-output behavior.Source: §A.2 · p. 9
- Symflower generated tests for 5,738 Java training codes, averaging 6.34 tests per code.Source: §A.2 · p. 9
- The training setup used LoRA with rank 16 and scaling factor 32, Adam at 10^-4 for CE optimization, and PPO at 1.41×10^-5 for RL.The pipeline ran on four NVIDIA V100 GPUs and six CPU cores per GPU.Source: §A.2 · p. 9
- The parameter-efficient fine-tuning approach reduces total training time by approximately 75% versus jointly training two LLMs without low-rank optimization.Source: §A.2 · p. 9
- AVATAR-TC contains whole-program Java–Python pairs, unlike datasets focused on equivalent snippets or functions and often similar languages.The paper describes Java and Python as statically typed and dynamically typed languages, respectively, with different syntactic styles.Source: §A.2 · p. 9
- AVATAR-TC is described as a large-scale dataset of compilable code pairs thoroughly tested with human-written test cases.Source: §A.2 · p. 9
A.3 Non-RL methods for integrating CF, SF in training
The appendix explores adding compiler and symbolic feedback to supervised fine-tuning instead of using them as reinforcement-learning rewards. The tested additive and multiplicative alternatives show only minimal gains over the baseline and do not match RL-based schemes.
A.3 Non-RL methods for integrating CF, SF in training
- CoTran+ adds compiler and symbolic feedback to cross-entropy loss through a linearly weighted combination.Its forward and backward models use hyperparameters αc and αs.Source: §A.3 · p. 9
- CoTran× weights cross-entropy loss using a reciprocal product of compiler and symbolic feedback, with weights normalized per mini-batch.Source: §A.3 · p. 9
- Compiler and symbolic feedback come from non-differentiable operations, so adding them to CE loss in CoTran+ does not affect optimization.Source: §A.3 · p. 9
- Both additive and multiplicative approaches yield only minimal improvement over the baseline and do not match RL-based schemes; CoTran+ is also sensitive to αc and αs.Source: §A.3 · p. 9
A.4 Results: Surprising Finding w.r.t. Transpilers
Human-written transpilers fall short of LLM-based methods on all listed metrics, and their limitations make them poor fits for whole-program translation that must compile readily.
A.4 Results: Surprising Finding w.r.t. Transpilers
- Human-written transpilers fall short of LLM-based methods on all listed metrics.The paper says they are not ideal for translating whole programs and generating readily compilable translations.Source: §A.4 · p. 9
- The java2python transpiler lacks support for the latest versions of Java and Python 3 after seven years without an update.Hand-crafted rule-based systems require manual updates after major language-version releases.Source: §A.4 · p. 9
- TSS CodeConv struggles when conversion rules are undefined and may copy source-language code verbatim into its translation.The paper describes these transpilers as a possible starting point for human developers, but not ideal for whole-program translations that compile readily.Source: §A.4 · p. 9
A.5 Correlation between CF and SF
The results suggest a correlation between compiler feedback (CF) and symbolic-execution feedback (SF): CF can improve functional equivalence even without direct SF fine-tuning, while combining both improves results in Python-to-Java translation.
A.5 Correlation between CF and SF
- In P2J, combining CF and SF via RL+SFT raises CompAcc from 73.63% to 76.98% and FEqAcc from 40.41% to 48.68%.Table 3 presents the full code-translation performance comparison for Java-to-Python and Python-to-Java.Source: Table 3 · p. 10, §A.5 · p. 10
- In J2P, CF through RL+SFT raises FEqAcc from 44.52% to 49.83% without direct SF fine-tuning, while CompAcc rises from 96.12% to 96.79%.The passage describes the CompAcc increase as slight and the FEqAcc increase as significant.Source: §A.5 · p. 10
- The results suggest an implicit correlation between CF and SF: improving compilability also improves functional correctness, and vice versa.The passage notes that CF improves functional equivalence even without direct SF fine-tuning.Source: §A.5 · pp. 9–10