Source-linked AI summary
Leveraging Automated Unit Tests for Unsupervised Code Translation
Baptiste Roziere, Jie M. Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, Guillaume Lample
TL;DR
Unsupervised code translation lacks reliable semantic supervision, while noisy back-translation is especially risky because small token changes can produce erroneous programs. The paper uses multilingual unit tests to filter valid pseudo-labels and reports substantial improvements over prior unsupervised methods across language pairs, including a 35.7% error-rate reduction for Python →C++.
Problem
Unsupervised code translation lacks supervised signal that distinguishes semantically different tokens occurring in similar contexts, without manually crafted parallel datasets.
Method
TransCoder-ST uses automated unit-test generation and multilingual test suites to filter invalid translations and create parallel training examples from monolingual data.
Results
12.6% points average computational-accuracy improvement over the baseline corresponds to a 25.5% average error-rate reduction, while Python →C++ improves by more than 24% points and reduces error by 35.7%.
Takeaways & Limitations
Test-filtered pseudo-labels substantially improve unsupervised code translation across language pairs, directions, and evaluation metrics.
Takeaways & Limitations
The validation criterion focuses on translation correctness based on semantics and does not yet incorporate requirements such as code formatting or linter outputs.
Abstract
from arXiv · showhide
With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation. However, the majority of unsupervised machine translation approaches rely on back-translation, a method developed in the context of natural language translation and one that inherently involves training on noisy inputs. Unfortunately, source code is highly sensitive to small changes; a single token can result in compilation failures or erroneous programs, unlike natural languages where small inaccuracies may not change the meaning of a sentence. To address this issue, we propose to leverage an automated unit-testing system to filter out invalid translations, thereby creating a fully tested parallel corpus. We found that fine-tuning an unsupervised model with this filtered data set significantly reduces the noise in the translations so-generated, comfortably outperforming the state-of-the-art for all language pairs studied. In particular, for Java $\to$ Python and Python $\to$ C++ we outperform the best previous methods by more than 16% and 24% respectively, reducing the error rate by more than 35%.
1 INTRODUCTION
Source-code translation matters for modernizing legacy and multilingual codebases, but unsupervised systems struggle because limited parallel data and noisy back-translation can produce semantically invalid programs. TransCoder-ST uses automated unit tests to filter invalid translations, build tested parallel data, and improve unsupervised translation.
- Motivation: Legacy codebases impose substantial modernization costs, while multilingual codebases hinder interoperability and programmer efficiency.Automatic translation could make migrations faster and cheaper and help programmers learn or understand code.
- Problem: Source-to-source translation remains open because rule-based systems are not exhaustive and little to no parallel source-code data prevents standard supervised training.
- Problem: Back-translation trains on potentially invalid input-output pairs, and programming-language errors can make an entire translation erroneous.Unlike natural-language inaccuracies, small code changes can cause compilation failures or incorrect programs.
- Approach: TransCoder-ST leverages automated unit tests to filter invalid translations and reduce noise from unsupervised back-translation.The approach creates tests for source functions, translates them into target languages, and selects translations matching the original semantics on tested inputs.
- Results: 12.6% average CA@1 improvement over previous models demonstrates stronger translation across Java, Python, and C++ language pairs.The corresponding average relative improvement is 25.5%.
- Contributions: The method creates a parallel dataset of 135,000 functions and is completely unsupervised, with potential generalization to other languages and test-generation tools.
2 RELATED WORK
Related work spans automated test generation, machine learning for programming languages, and code translation, motivating supervised semantic signals for unsupervised translation.
- Automated test generation: Automated test generation has decades of research and mature open-source and production tools.The paper uses this established software-engineering capability as a foundation for code translation.
- Automated test generation: EvoSuite is a widely used Java test-generation tool guided by coverage metrics and mutation scores.The paper applies EvoSuite to generate tests for code-translation data construction.
- Programming-language translation: Prior programming-language translation methods include statistical, tree-to-tree neural, and unsupervised approaches across limited language pairs.TransCoder translated Python, Java, and C++ without parallel data or expert knowledge, while DOBF later improved TransCoder with deobfuscation training.
- Machine learning for programming languages: Unsupervised code translation addresses limited parallel data but can confuse semantically different tokens in similar contexts.Such confusion is especially consequential in programming languages, where small inaccuracies can make an entire translation erroneous.
- Motivation: The paper targets noisy back-translation by using automated tests to provide supervision directly related to code semantics.This approach selects translations that match the original function on tested inputs, addressing the need for semantic signal without manually crafted parallel datasets.
3 METHOD
The method generates and translates tests for Java functions, retains candidate translations that pass sufficiently strong test suites, and uses the resulting parallel data for iterative training.
- Parallel data creation: EvoSuite automatically generates unit tests for Java functions, providing the starting point for parallel data creation.The generated tests can later be transformed into semantically equivalent Python and C++ tests.
- Parallel data creation: Only test suites with mutation scores above 90% are retained for building the parallel dataset.Mutation score is the ratio of killed mutants and serves as a strict filter for semantic coverage.
- Parallel data creation: Generated Java tests are transformed into Python and C++ tests with identical inputs, expected outputs, and side effects.The dataset is restricted to Java functions that compile and run in isolation with supported simple types.
- Translation filtering: For each Java function, the system generates 20 Python and C++ candidates and selects the first beam-search translation passing all unit tests.Passing tests provide the method’s assessed evidence that the translated functions share the tested semantics.
- Training method: The generated parallel examples fine-tune pre-existing unsupervised translation models through offline or online training.Offline training iterates dataset creation and fine-tuning, while online training creates examples during training but can become unstable without stabilization.
- Evaluation: Evaluation uses CA@N, which checks whether any of the model’s top-N solutions passes all corresponding unit tests.The models are evaluated on the validation and test sets associated with TransCoder.
4 EXPERIMENTS
Experiments compare the proposed offline and online self-training methods with established baselines using computational accuracy and ablations. Both methods improve translation broadly, while online training generally performs best but remains sensitive to test quality, cache initialization, and decoding choices.
- Results: Both offline and online self-training significantly improve computational accuracy over baselines for every language pair and direction.Online self-training outperforms offline self-training even after several iterations.
- Training details: The experiments use the TransCoder architecture, Java functions selected from a GitHub dataset, and 103,488 functions with mutation score > 0.9 and at least two asserts.The parallel examples are generated through alternating language-pair batches and online cache updates.
- Results: 25.5%: offline training reduces the Java →Python baseline error rate, while average CA@1 increases by 7.4% over previous models.Offline training already improves the baseline after one iteration.
- Ablation study: High mutation-score tests and a pre-filled cache substantially improve ablation performance, while beam size 20 improves generation over greedy decoding by about two percentage points on average.Starting without a cache slows training and harms generalization; removing the mutation-score requirement lowers scores further.
- Results: After three iterations, more unit-test-passing samples can lower TransCoder test accuracy when translations are incompatible with TransCoder’s expected types.Figure 4 illustrates an overflow-related disagreement between the multilingual tests and TransCoder’s tests.
- Results: 12.6 percentage points: online self-training’s average improvement over the baseline, corresponding to a 25.5% error-rate reduction.For Python → C++, the improvement exceeds 24 percentage points and reduces the error rate by 35.7%.
5 CONCLUSION
The paper grows a parallel corpus for code translation from monolingual data by using multilingual unit tests to filter pseudo-labels. Offline and online variants improve unsupervised translation substantially, while the approach remains dependent on test-generation quality and currently uses Java-only test generation.
- 5 CONCLUSION: Multilingual unit tests filter good pseudo-labels, enabling a parallel corpus for automated code translation from completely monolingual data.The method uses unit tests to improve candidate translations iteratively.
- 5 CONCLUSION: Both offline and online methods improve computational accuracy by an average of 12.6 percentage points, with up to 24 points for Python →C++.These gains correspond to error-rate reductions of 25.5% and 35.7%, respectively.
- 5 CONCLUSION: The method could benefit from better automated test generation and from test-generation tools or human-written tests for languages beyond Java.The paper also suggests supplementing semantic validation with formatting or linter requirements.
REPRODUCIBILITY
The authors emphasize reproducibility by matching prior source-code translation architectures and releasing code, models, and setup instructions.
- REPRODUCIBILITY: The study uses the same architecture and framework as previous source-code translation work to support comparable results.The submission includes code, a reproducible environment setup, and plans to release trained models.
ETHICAL CONSIDERATIONS
The paper connects source-code translation with codebase migration and developer efficiency, while noting that labor-market effects remain unclear.
- ETHICAL CONSIDERATIONS: The methods could facilitate codebase migration and interoperability, potentially improving software-developer efficiency.The paper says the effect of increased efficiency on the labor market is unclear because lower costs could also increase demand for software engineers.
A.1 GENERATED UNIT TESTS
The paper evaluates automatically generated unit tests as filters for translation data. High mutation scores identify useful tests, while some high-scoring single-assert suites and exception-dependent tests require filtering or exclusion.
- Test-suite selection: 95% mutation score qualified one generated test suite for pseudo-labelling, although one test may be too strict for Python int translations.The third test case could cause translations using Python’s int type to fail.
- Test-suite selection: 100% mutation score does not guarantee an informative test suite when a function returns only a constant.Removing such functions and tests from self-labelling improves model performance.
- Test-suite limitations: 5.6% of generated tests expect exceptions, and fewer than 2% of high-mutation-score tests do so, preventing parallel examples from being created.These tests cannot be translated successfully and therefore cannot validate candidate translations.
A.2 MUTATION SCORE
Mutation testing assesses whether generated tests detect syntactic faults by running them against altered programs. Mutation score is preferred to coverage because it measures fault-revealing behavior rather than code execution alone.
- Mutation testing: Mutation testing introduces syntactic faults through mutation operators and checks whether at least one test fails on each mutant.A mutant survives when the test output matches the original program’s output.
- Mutation testing: The example mutation replaces < with > in a return statement, but one input fails to distinguish the original and mutated programs.That input therefore does not kill the mutant.
- Mutation score: Mutation score is considered more effective than code coverage for assessing whether test suites reveal faults.Coverage can reach 100% without assertions and therefore without detecting incorrect behavior.
B TRANSLATION EXAMPLES
The examples show that TransCoder frequently makes semantic translation errors involving conditions, operators, functions, and types. TransCoder-ST more often corrects these errors, while unit-test training also encourages more general template-based solutions.
- Example comparisons: The translation examples compare input functions with outputs from TransCoder and TransCoder-ST across Python, C++, and Java language pairs.The displayed examples include Python-to-C++, Python-to-Java, C++-to-Python, C++-to-Java, Java-to-C++, and Java-to-Python.
- Semantic corrections: TransCoder often mistranslates conditions and operators across Python, C++, Java, and Java-to-Python examples, whereas TransCoder-ST generally recovers the intended semantics.Observed errors include extra clauses, incorrect comparisons, wrong operators, and confusion between /, //, * and **.
- Semantic corrections: TransCoder-ST corrects concrete semantic failures such as translating INT_MAX as Integer.MIN_VALUE and mishandling string-boundary logic.The examples cover C++-to-Java and Java-to-C++ translation.
- Generalization: Parallel unit tests favor template-based solutions that pass for several parameter types and appear more often in generated parallel data and model outputs.The online model generated templates three times more often under this effect.
C.1 BEAM REORDERING
Beam reordering uses generated unit tests at inference time to prioritize candidate translations that pass them. This produces only small gains, substantially below the paper’s self-training approaches.
- Method: Beam reordering computes unit-test results for every proposed C++ or Python translation and prioritizes candidates that pass.The method applies tests to the beam at test time rather than creating a filtered parallel corpus.
- Results: Up to 1.7% CA@1 improvement over the best baseline was observed for Java →Python, but gains remained small.The method remained far below CA@10, which asks whether any of ten beam elements is correct.
- Results: Beam reordering underperformed the offline and online self-training methods, while training from scratch on the generated dataset remained below TransCoder and TransCoder-ST.The authors attribute limited beam-reordering performance partly to low-mutation-score or incompatible tests.