Leveraging Automated Unit Tests for Unsupervised Code Translation

1 INTRODUCTION

• Arbitrary source-to-source translation remains open: rule-based systems are not exhaustive, parallel code data are scarce, and unsupervised TransCoder can confuse language-specific semantics.Examples include Python float division versus integer division in C++ and Java, and differing operator precedence; TransCoder is trained without supervised signal and learns token semantics from context.Source: §1 · pp. 1–2
• TransCoder’s back-translation trains on potentially invalid input-output pairs, and input noise can severely deteriorate neural translation performance.Source code is especially vulnerable because a small error can cause compilation failure or an erroneous program, unlike a minor inaccuracy in natural language.Source: §1 · p. 2
• EvoSuite, a Java test-generation tool using coverage metrics and mutation scores, is applied to code translation for the first time.The paper notes that mature automated testing tools and future advances in testing can support ongoing improvements in automated translation.Source: §1 · p. 2
• TransCoder-ST uses an automated unit-test pipeline to filter invalid translations and reduce noise from back-translation in unsupervised machine translation.Automatically generated source tests are translated with simple scripts, then used to select target-language functions that match the original semantics on tested inputs.Source: §1 · pp. 2–3
• 12.6% average Computational Accuracy (CA@1) improvement and 25.5% average relative improvement over previous state of the art; Python → C++ gains 24% CA@1, reducing error rate by 35.7%.The approach has online and offline implementations and significantly outperforms prior methods on all language pairs considered.Source: §1 · p. 3
• The method generates multilingual unit tests for hundreds of thousands of Java functions and creates 135,000 parallel functions across Java, Python, and C++.The approach is completely unsupervised and could be generalized to other programming languages and unit-test creation tools.Source: §1 · p. 3

Unit Test Generation

• Automated test generation has decades of research and includes symbolic, fitness-guided, hybrid, and neural approaches (Anand et al., 2013; Cadar & Sen, 2013; Harman et al., 2015; Baars et al., 2011; Tufano et al., 2020).Industrial-strength test-generation tools became widely available in the last decade.Source: §2 · p. 3
• EvoSuite uses search-based software engineering and coverage and mutation-score fitness functions; this work selects it for availability, active development, and adoption, while allowing substitution.The authors cite replication and interoperability as benefits of using an open-source, widely used framework.Source: §2 · p. 3
• Mutation score measures the proportion of mutants distinguished from the original program and serves as a proxy for fault-revealing power correlated with real fault revelation.Mutants are programs with deliberately inserted faults.Source: §2 · p. 3

Machine Learning for Programming Languages

• Deep learning has been applied to bug detection and repair, code completion, and code pretraining, with BART and code-specific objectives performing strongly on CodeXGLUE (Ahmad et al., 2021; Guo et al., 2020; Roziere et al., 2021; Lu et al., 2021).The cited work also includes BERT-based pretraining methods for code (Kanade et al., 2020; Feng et al., 2020).Source: §2 · pp. 3–4
• Code-generation studies evaluate programs produced from natural-language prompts, whereas code translation starts from code that is more precise and often more concise (Chen et al., 2021; Austin et al., 2021).Chen et al. (2021) report solving 28.8% of HumanEval problems with their model.Source: §2 · p. 4

Translation of Programming Languages

• Earlier statistical and neural translation methods targeted a few language pairs, relying on small manually collected parallel datasets or rule-based data creation.Examples include C#–Java, Python 2–Python 3, and CoffeeScript–JavaScript (Chen et al., 2018).Source: §2 · p. 4
• TransCoder translated Python, Java, and C++ without parallel data or expert knowledge, while DOBF later improved it with a deobfuscation objective (Roziere et al., 2021).TransCoder used masked language modeling, denoising auto-encoding, and back-translation; the paper refers to the later model as DOBF.Source: §2 · p. 4
• Unsupervised methods can miss semantic distinctions between tokens used in similar contexts, motivating semantic supervision without manually crafted parallel datasets.The paper identifies this as a need for code translation.Source: §2 · p. 4

3.1 PARALLEL DATA CREATION

• EvoSuite-generated Java test suites are retained only when their mutation score exceeds 90%, selecting tests intended to examine function semantics thoroughly.Mutation score is the ratio of killed mutants; a mutant is killed when a test produces a different result from the original program.Source: §3.1 · pp. 4–5
• Generated tests are translated to C++ and Python with matching inputs, expected outputs, and side effects, for isolated Java functions with supported simple types.Supported types include Java primitives, standard types, and arrays or lists of supported types.Source: §3.1 · p. 5
• Java-to-Python and Java-to-C++ translations are generated with DOBF and TransCoder, respectively, and the first of 20 beam candidates passing the generated tests is retained.The method uses DOBF (Roziere et al., 2021) for Java to Python and TransCoder for Java to C++; passing all tests indicates matching semantics as assessed by those tests.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 TRAINING METHOD

• Offline training iteratively fine-tunes on generated Java–Python, Java–C++, and Python–C++ data, regenerating data from selected validation checkpoints until improvements cease.The approach uses DOBF for Java-to-Python and TransCoder for Java-to-C++; generated examples can improve the model, which can then generate better parallel data.Source: §3.2 · pp. 5–6
• Online training generates examples during training and uses a cache of previously test-passing examples to stabilize updates and reduce forgetting.At each step, the model trains on cache samples or creates new examples; sampled examples are removed with a specified probability.Source: §3.2 · p. 6

3.3 EVALUATION

• Models are evaluated on TransCoder validation and test sets using CA@N, which measures whether any of the top N translations passes all associated unit tests.The passages note that BLEU and exact match do not capture code semantics well and typically correlate poorly with generated-function correctness.Source: §3.3 · p. 6

4.2 RESULTS AND DISCUSSION

• 12.6% points over the baseline on average, online self-training performs best except against third-iteration offline training for Java →C++; for Python →C++, it gains more than 24% points and reduces error by 35.7%.The average improvement corresponds to a 25.5% error-rate reduction.Source: Table 2 · p. 7, §4.2 · p. 8

4.1 TRAINING DETAILS

• From 333,542 compiled-isolated Java functions, EvoSuite produced high-quality tests for 103,488, each with mutation score > 0.9 and at least two asserts.Test generation used a 20-second budget and line, branch, cbranch, output, and mutation coverage criteria.Source: §4.1 · p. 7

4.2 RESULTS AND DISCUSSION

• 7.4% average CA@1 improvement over the best previous models corresponds to a 16.6% error-rate reduction; for Java →Python, offline training reduces the best baseline’s error rate by 25.5%.Offline training improves CA@1 at beam size 10 for every direction, with especially substantial gains for language pairs involving Python.Source: §4.2 · p. 7, Table 2 · p. 7
• More than 22% fewer examples have no correct translation in the beam of 20 with online self-training, and gains remain stable across CA@n metrics and beam sizes.This indicates the method helps find correct solutions that the baseline did not assign high probability, rather than merely reordering its beam.Source: Table 3 · p. 8, §4.2 · p. 8
• A prefilled cache and high-mutation-score test selection improve performance; no cache destabilizes training, while an empty cache slows learning and weakens generalization.Greedy example generation loses about two percentage points on average versus beam size 20; beam search is needed for best performance, though greedy decoding can still outperform the baseline.Source: Table 4 · p. 8, §4.2 · pp. 8–9
• CA@1 improves by 2% points on average from the first to third offline iteration, but the fourth iteration scores worse as more test-passing outputs become incompatible with TransCoder’s expected translations.An overflow example illustrates the mismatch: a translation that reproduces Java integer behavior can fail TransCoder’s tests.Source: §4.2 · p. 8, Figure 4 · p. 9

5 CONCLUSION

• 12.6 percentage points average improvement in computational accuracy and up to 24 percentage points for Python → C++ correspond to translation error-rate reductions of 25.5% and 35.7%, respectively.Both offline and online methods substantially improve the state of the art; the results use no unit-test generation tool for Python and C++, only for Java.Source: §5 · p. 9
• The method grows a parallel corpus from monolingual data by using multilingual unit tests to filter good pseudo-labels, improving the model and candidate translations.The conclusion says the approach uses no unit-test generation tool for Python and C++, and uses one exclusively for Java.Source: §5 · p. 9
• The dataset could grow by improving automatic test generation, using test tools for languages beyond Java, generating tests with EvoSuite, or mining human-written tests from open-source projects.The conclusion suggests these sources could produce larger, and possibly higher-quality, datasets.Source: §5 · p. 9
• Validation in this paper checks only semantics, but could also enforce code formatting, linter output, or other arbitrary criteria.The conclusion says parallel-example validation could be supplemented with these requirements.Source: §5 · p. 9
• The approach could transfer to natural-language translation by using grammar and syntax checkers to filter incorrect generations and reduce back-translation noise.The conclusion notes that neural machine translation is sensitive to noise in parallel data (Belinkov & Bisk, 2018; Khayrallah & Koehn, 2018), suggesting this could improve low-resource performance.Source: §5 · p. 9

REPRODUCIBILITY

• The study uses prior-work architecture and framework choices for comparability and provides code with reproduction instructions, with code and trained models slated for release.The ReadMe includes steps to reproduce results and a script to set up a suitable environment. Training used Tesla V100 GPUs and libraries including PyTorch and CUDA.Source: REPRODUCIBILITY · p. 10

ETHICAL CONSIDERATIONS

• Code translation could facilitate codebase migrations and interoperability while making software developers more efficient.The paper presents these as potential benefits of its methods.Source: ETHICAL CONSIDERATIONS · p. 10
• The effect of greater efficiency on developer employment is unclear because lower costs could enable more projects and increase demand for software engineers.The authors note that software engineering demand remains high despite earlier productivity improvements.Source: ETHICAL CONSIDERATIONS · p. 10
• The authors believe automatic translation tools would not drastically worsen software developers’ prospects, given sustained demand amid earlier efficiency gains.They cite tools, languages, libraries, and methodologies as prior sources of improved developer efficiency.Source: ETHICAL CONSIDERATIONS · p. 10
• Migrating antiquated codebases such as COBOL could harm specialists in those languages, who are currently in high demand.Source: ETHICAL CONSIDERATIONS · p. 10
• Such migration could benefit society by making software used in financial transactions and government services easier to debug and update.Source: ETHICAL CONSIDERATIONS · p. 10
• The authors also expect migration to increase demand for other software engineering skills.Source: ETHICAL CONSIDERATIONS · p. 10

A.1 GENERATED UNIT TESTS

• A selected pseudo-labelling suite achieves a 95% mutation score, though its third test may be overly strict because Python int translations fail it.The potentially too-strict case is the third test, labeled test2 in the figure.Source: Figure 5 · p. 15
• A one-assert suite with a 100% mutation score can still cover an uninteresting constant-return function; removing such functions and tests from self-labelling improves model performance.The figure notes that this suite tests the constant-return function's semantics properly, despite containing only one test and one assert.Source: Figure 6 · p. 15
• Tests target static functions with selected return and parameter types to simplify type mapping; translation failures affect 5.6% of tests and less than 2% of high-mutation-score tests, yielding no parallel examples.When test translation fails, candidate translations cannot pass the translated tests, so no parallel examples are created.Source: §A.1 · p. 15

A.2 MUTATION SCORE

• Mutation testing introduces minor syntactic faults using mutation operators; a test kills a mutant when its output differs from the original program’s output.Otherwise, the mutant survives; EvoSuite’s mutation operators are described by Fraser & Arcuri (2015).Source: §A.2 · p. 15
• A generated mutant replaces the < operator with >, but the test using input (-800, -800, -1) does not kill it because both programs return the same output.The mutation changes the comparison in the return statement.Source: §A.2 · p. 15, Figure 7
• Mutation score is considered the most effective criterion for assessing fault-revealing ability, whereas code coverage can reach 100% without assertions or fault detection.Coverage records execution but does not establish that the execution result is correct.Source: §A.2 · pp. 15–16
• About 40% of generated unit tests met the 0.9 mutation-score threshold, while many remaining tests had a score of 0.The histogram summarizes mutation scores for the generated tests.Source: Figure 8 · p. 16

• Self-training helps TransCoder avoid semantically incorrect Python-to-C++ translations and often fixes hallucinated clauses or incorrect comparisons in Python-to-Java translations.Source: Figure 9 · p. 17, Figure 10 · p. 17
• For C++-to-Python translation, TransCoder errors include incorrect conditions, operators, and functions, which TransCoder-ST generally resolves.Source: Figure 11 · p. 17
• In C++-to-Java examples, TransCoder-ST corrects an inappropriate minimum-value initializer and preserves a set-membership condition that TransCoder negates.Source: Figure 12 · p. 18
• For Java-to-C++ examples, TransCoder-ST fixes a failed logical expression and selects a min-priority queue where TransCoder selects a max-priority queue.Source: Figure 13 · p. 18
• Self-created labels often correct Java-to-Python confusions between division and exponentiation operators, while parallel unit tests favor reusable templates that pass across parameter types.Template-based solutions appear three times more often for the online model trained the longest.Source: Figure 14 · p. 19, Figure 15 · p. 19

C.1 BEAM REORDERING

• The method creates unit tests for Java test-set functions, runs them on each proposed C++ or Python translation, and prioritizes beam elements that pass.The tests reorder the beam at test time.Source: §C.1 · p. 20
• Up to 1.7% CA@1 for Java →Python: test-time beam reordering yields only small improvements over the best baseline and remains far below CA@10.Table 6 also characterizes the gains as small compared with the offline and online self-training methods.Source: §C.1 · p. 20, Table 6 · p. 20
• Generated tests may have low mutation scores and insufficiently test function semantics, while some are incompatible with the test-set tests.The passage points to Figure 4 for an example of incompatibility.Source: §C.1 · p. 20
• Training from scratch on the generated parallel dataset gives decent performance, but remains below TransCoder and TransCoder-ST.Table 6 reports CA@1 with beam size 10 for the baselines, offline and online methods, beam reordering, and scratch training.Source: Table 6 · p. 20