Multi-lingual Evaluation of Code Generation Models

• Execution-based evaluation is favored over n-gram matching because it accepts functionally correct implementations, but its reliance on carefully constructed tests has left many available datasets limited to Python.HumanEval (Chen et al., 2021) and MBPP helped establish execution-based evaluation as a primary approach.Source: §1 · p. 1
• The framework converts Python execution-based datasets into other languages by transforming prompts and test statements with a rule-based transpiler, without translating canonical function bodies.Static analysis enables reliable conversion for basic programming problems, while function completion can be evaluated without canonical solutions.Source: §1 · p. 2
• The resulting MBXP, Multilingual HumanEval, and MathQA-X benchmarks extend MBPP, HumanEval, and MathQA to Java, JavaScript, TypeScript, Go, Ruby, Kotlin, PHP, C#, Scala, C++, Swift, and Perl.The authors also release an execution package for all supported languages.Source: §1 · p. 2
• Large-scale solution bootstrapping equips the multilingual execution-based benchmarks with canonical solutions for translation, robustness, insertion, summarization, and other code-related evaluations.MBXP is processed into t-MBXP, r-MBXP, i-MBXP, and s-MBXP variants.Source: §1 · p. 2
• The study evaluates models from ∼100M to ∼10B parameters in multilingual and monolingual settings across in-domain and out-of-domain generation, few-shot prompting, translation, robustness, summarization, and insertion.These experiments examine multiple dimensions of code-generation ability using the newly constructed datasets.Source: §1 · p. 2

Model comparisons

• At sufficient capacity, multilingual models often outperform the best comparably sized monolingual models trained with equivalent resources, benefiting from knowledge across languages.Multilingual training can combine knowledge about language A found in data written in languages A, B, C, and others.Source: §2 · pp. 2–3

Cross-language generalization

• Large language models can generate syntactically correct code and pass unit tests in languages on which they were not intentionally trained, likely through cross-language data spillover.Code comments, co-occurrences, and other spillover data can embed knowledge of different languages within the main training language.Source: §2 · pp. 2–3

Few-shot prompting

• Few-shot prompting significantly improves out-of-domain code generation in unfamiliar languages by reducing compilation and parsing errors.These errors are identified as major sources of failure when the model is unfamiliar with a programming language.Source: §2 · p. 3

Code translation

• Language models can use reference code in one language to improve generation in another, making difficult problems easier through zero-shot code translation.This translation ability also appears in monolingual models, such as Java-only models translating Python references into Java despite limited Python understanding.Source: §2 · p. 3

Additional capabilities

• Multilingual models are more robust to prompt perturbations and better at summarizing code.Source: §2 · p. 3

Conversion process

• The framework converts each Python function prompt into a valid target-language prompt, inferring input and output types from heterogeneous original test cases for typed languages.For an argument containing int and float values, it assigns the most general type encountered.Source: §3 · p. 3
• Converted test cases are aligned with the target prompt’s defined class, method, and argument list so generated code can execute correctly.Figure 2 illustrates this coordination for a Java BinomialCoeff example.Source: §3 · p. 3

4.2.2 GENERALIZATION TO OUT-OF-DOMAIN LANGUAGES

• Non-zero out-of-domain pass@k scores and asymmetric cross-language performance show that models generate unseen-language code, likely through natural multilingual data spillover.Python models perform well on Java, whereas Java models have negligible performance on Python; cross-language code and text can occur within projects and comments.Source: §4.2.2 · pp. 4–5, Figure 3 · p. 5

4.2.3 MULTI-LINGUAL VERSUS MONO-LINGUAL MODELS

• At 2.7B and 13B parameters, multilingual models outperform the best monolingual models across all evaluation languages, with especially large gains on out-of-domain languages.At smaller sizes, monolingual models can outperform multilingual ones in some languages; multilingual models also have higher per-language validation losses despite better evaluation performance.Source: Figure 4 · p. 5, §4.2.3 · p. 6

4.3 ZERO-SHOT CODE TRANSLATION

• Translation raises a Java monolingual model’s pass@1 from 20% without translation to 36% with a Python solution, while target-language knowledge remains more important than source-language knowledge.Reference solutions preserve underlying logic and semantics while producing target-language syntax; source languages have unequal effects, and overly similar syntax can cause confusion.Source: Figure 7 · p. 7, §4.3 · pp. 8–9

4.4 FEW-SHOT PROMPTING

• Three-function few-shot prompts consistently improve execution accuracy, especially on out-of-domain evaluations, by disambiguating target languages and reducing compilation, syntax, and parsing errors.The improvement also appears on MathQA, suggesting prompt-based conditioning may help out-of-domain or scarce programming languages.Source: Figure 8 · p. 8, §4.4 · p. 9

4.5 MATHQA

• MathQA performance is typically 10 −20% for ordinary completion but exceeds 94% with translation, because simple mathematical solutions can be converted when Python references are provided.Translation can approach 100% pass@100 and remains strong for a 672M model, whereas multilingual models do not consistently outperform monolingual counterparts on MathQA.Source: §4.3 · p. 8, §4.5 · pp. 9–10, Table 1 · p. 11

4.6 ROBUSTNESS EVALUATION: R-MBXP

• Multilingual models are more robust on average, with performance drops of 7.80% versus 9.39% for monolingual models across perturbed r-MBXP datasets.They also achieve higher pass@1 scores on most perturbed datasets, motivating further research using this execution-based multilingual robustness benchmark.Source: §4.6 · pp. 10–11

4.7 CODE SUMMARIZATION: S-MBXP

• Multilingual models generally outperform monolingual models for code summarization in both zero-shot and few-shot settings, with few-shot gains larger for larger models.The section removes the docstring description and prompts models with a function signature and solution to generate the functionality description.Source: §4.7 · p. 11, Figure 11 · p. 11

4.8 CODE INSERTION: I-MBXP

• In i-MBXP, adding right context improves InCoder’s pass@1 performance by 23.2% on Python, 14.4% on JavaScript, and 37.6% on Java relative to left context alone.Performance increases positively with the number of right-context lines, demonstrating the benchmark’s execution-based multilingual code-insertion capability.Source: §4.8 · pp. 11–12, Table 2 · p. 12, Table 3 · p. 12

• The framework fills a gap by providing multilingual code evaluation with synthetic solutions, datasets beyond HumanEval, and multiple code-generation abilities.It includes MBPP and MathQA in addition to HumanEval.Source: §5 · p. 12
• Execution-based datasets such as APPS and CodeContest can also be converted into multilingual benchmarks because their algorithmic problems and tests are language-agnostic.These datasets contain challenging algorithmic competition problems and tests that can be converted to Python and other languages.Source: §5 · p. 12
• Concurrent work translates HumanEval into multiple languages, while BabelCode studies execution-based multilingual evaluation and language-distribution balancing in training data.Together, these efforts provide resources for evaluating multilingual code generation.Source: §5 · p. 12

6 DISCUSSION

• The released datasets support research on multilingual versus monolingual models, out-of-domain performance, zero-shot translation, and multilingual code insertion.These evaluations are presented as resources for advancing code-generation research.Source: §6 · p. 12
• Multilingual models show potential to generalize to new languages and tasks, although comprehensive real-world evaluation may require additional language-specific assessments.The discussion attributes multilingual advantages partly to data spillover across languages.Source: §6 · p. 12

A.1 IMPLICATION OF FINDINGS

• Models of 2.7B parameters and above clearly outperform all monolingual models in the controlled setting, likely because multilingual training enables cross-language data spillover.The required size may increase as more languages are added to training.Source: §A.1 · p. 20

A.2 IMPLICATION OF EVALUATION DATA AT SCALE

• Parallel datasets enable systematic study of code translation by evaluating generation in supported target languages against canonical solutions from a source language.They expose questions about cross-language generalization, translation success factors, and modeling strategies.Source: §A.2 · p. 20

A.3 POSSIBILITIES OF TRUE GENERALIZATION

• Out-of-domain performance may reflect both data spillover and true generalization, but synthetic languages would still resemble existing languages, leaving the distinction difficult to resolve.A constructed language could test completion without direct training-data exposure, while shared control-flow and data-structure concepts would preserve overlap.Source: §A.3 · p. 20

A.4 POTENTIAL PROXY FOR GENERAL CODING CAPABILITIES

• MBXP and HumanEval can proxy general coding ability only if zero-shot performance on their task slice correlates with broader coding performance without distribution-specific adaptation.The paper therefore avoids finetuning to reduce bias toward particular coding problems and better reflect underlying capabilities.Source: §A.4 · p. 20

A.5 LIMITATIONS

• MBXP conversions may misrepresent natural multilingual programming because tasks differ in language suitability and the framework assumes every problem transfers equally to each target language.For example, string manipulation may be more natural in Python or PHP than C++, so benchmark scores may not match real-world language-use distributions.Source: §A.5 · pp. 20–21
• MBXP omits language-specific functionality and supports only value-oriented tests, limiting transfer to real-world software and complex integration testing.The authors recommend complementing MBXP with language-specific evaluations where available.Source: §A.5 · p. 21

A.6 GENERATION TENDENCY VERSUS GENERATION ABILITY

• The guided prompt format measures compliance with a desired language and completion from a supplied function signature more directly than a model’s unconstrained language-generation tendency.Without explicit guidance, multilingual models may generate another language; even with guidance, closely related languages such as Ruby and Python can be confused.Source: §A.6 · p. 21
• Language-specific syntax nevertheless emerges in generations, such as Ruby’s list.select and Kotlin’s nums.filter, despite the benchmark targeting generic programming understanding.Sample generations for converted languages are provided in Section R.1.Source: §A.6 · p. 21

B OTHER RELATED WORK

• Prior code-generation research spans pretrained encoder-only and encoder-decoder models, large multilingual or conversational models, program synthesis and editing systems, and general language models capable of code completion.Representative systems include CodeBERT, GraphCodeBERT, CodeT5, PLBART, Codex, CodeGen, InCoder, CoditT5, PaLM, and BLOOM.Source: B OTHER RELATED WORK · p. 21
• Existing methods improve code generation through retrieval augmentation, constrained decoding, execution-based minimum-Bayes-risk decoding, and language-model-based code repair, while this work is model-agnostic and serves as a testbed for them.The benchmark can therefore support evaluation of complementary generation-improvement techniques.Source: B OTHER RELATED WORK · pp. 21–22
• Unlike prior code-completion resources that often use match-based metrics or language-specific formats, MBXP uses execution-based function completion across multiple programming languages in parallel.The authors adopt the HumanEval format for converted MBXP because it best matches typical coding environments, and convert Python MBPP to that format for consistency.Source: B OTHER RELATED WORK · p. 22
• Prior code-translation datasets provide parallel functions or programs across narrower language sets, whereas MBXP covers a wider range of languages with unit tests for functional-accuracy evaluation.The related work also distinguishes this programming-language multilinguality from MCoNaLa’s multilingual natural-language focus.Source: B OTHER RELATED WORK · p. 22

C.1 SAMPLE GENERATION

• Generation uses nucleus sampling with p = 0.95, a 1792-token input limit, and up to 256 generated tokens, truncating from the left when contexts exceed the limit.Left truncation is more frequent for few-shot prompting and translation settings.Source: §C.1 · p. 23

C.2 STOPPING CRITERIA

• Stopping criteria group languages by syntax: standalone curly-brace functions stop after }, class-wrapped Java and C# functions close the wrapper automatically, and Ruby stops at ‘end’.Automatically closing class wrappers avoids unnecessary additional functions and token-limit overruns.Source: §C.2 · p. 23
• The stopping procedure does not support multi-function evaluation, where an initial function may refer to subsequent functions.This extension is explicitly outside the paper’s scope.Source: §C.2 · p. 23

C.3 CODE EXECUTION

• The adapted human-eval framework executes MBXP programs natively across languages and computes unbiased pass@k, recording failures through exit codes and standard-error messages.Tasks can fail because generated code does not parse or compile, or because test execution raises exceptions.Source: §C.3 · p. 23

D.1 MULTI-LINGUAL MATHQA

• MathQA translation can fail when mathematical built-ins differ across languages, such as Python’s max and two-argument math.log versus Java’s Math.max and one-argument Math.log.The example produces Java code that does not resolve max and incorrectly preserves Python’s logarithm interface.Source: §D.1 · p. 24

D EVALUATION RESULTS ON ADDITIONAL DATASETS

• The additional-dataset evaluation extends the multilingual benchmark analysis from MBXP to MathQA and multilingual HumanEval, including both generation procedures and execution-based assessment.The supplied passages provide the dataset-specific findings and implementation details summarized above.Source: §D.1 · p. 24, §D.2 · p. 24

D.2 MULTI-LINGUAL HUMANEVAL

• Few-shot prompting and translation results on multilingual HumanEval are generally consistent with MBXP across the evaluated models.The section reports results using both the authors’ models and publicly available models; dataset-preparation details are deferred to Section R.2.Source: §D.2 · p. 24

E LANGUAGE “SPILLOVER” IN TRAINING DATA

• Code-generation models typically generalize to out-of-domain languages, which the paper hypothesizes may result from cross-lingual data spillover in files containing multiple languages.The section discusses cross-lingual code occurrences as a possible explanation for this generalization.Source: E LANGUAGE “SPILLOVER” IN TRAINING DATA · p. 25
• The spillover discussion motivates interpreting out-of-domain performance as potentially influenced by multilingual code exposure rather than language isolation alone.This qualification follows the paper’s hypothesis that cross-lingual files are common in code projects and may expose models to multiple languages.Source: E LANGUAGE “SPILLOVER” IN TRAINING DATA · p. 25

E.1 TYPES OF CROSS-LANGUAGE DATA SPILLOVER

• Cross-language spillover occurs through embedded guest languages, guest code stored in host-language strings, separate language-specific files within projects, and foreign-function interfaces.The paper notes that project-level mixtures do not apply to its models because languages are filtered per file rather than per project.Source: §E.1 · p. 25

E.2 EXAMPLE 1: EMBEDDED JAVASCRIPT IN PYTHON FILES

• A Python file can embed JavaScript in strings that are later written as a generated script, illustrating spillover through code generation.The example constructs JavaScript encoding and decoding tables inside Python string literals and writes them to a .js file.Source: §E.2 · pp. 25–26

E.3 EXAMPLE 2: JAVA AND PYTHON INTEGRATION AS JYTHON

• The Python–Java Jython example demonstrates cross-language integration within one project, combining Python classes with Java interfaces and application code.The Python implementation satisfies a Java interface, and Java invokes it through Jython’s object factory.Source: §E.3 · p. 26

F.1 PERFORMANCE TREND WITH RESPECT TO MODEL SIZE

• Across MBXP datasets, multilingual models clearly outperform monolingual models once model size is sufficiently large, consistently across pass@1, pass@10, and pass@100.The pass@k trends are consistent across k, differing mainly in score scale.Source: Figure 12 · p. 27, §F.1 · p. 27

F.2 COMPREHENSIVE SAMPLING RESULTS

• Pass@k trends are reported for 125M, 672M, 2.7B, and 13B monolingual and multilingual models across in-domain and out-of-domain languages.The comprehensive results are presented in Figures 13–16.Source: Figure 13 · p. 28, Figure 14 · p. 29, Figure 15 · p. 30

G FEW-SHOT PROMPTING

• Few-shot prompting prepends three correct functions from the corresponding MBXP language dataset to each evaluation prompt.Examples are selected once and fixed across uses rather than tuned on the evaluation tasks.Source: G FEW-SHOT PROMPTING · p. 32
• Few-shot prompting yields an automatic performance gain of roughly 0.3%, which the paper notes is small relative to the observed gains.The evaluation contains approximately 1000 cases for each language, and further prompt tuning remains possible.Source: G FEW-SHOT PROMPTING · p. 32

G.2 QUALITATIVE EXAMPLES

• The qualitative examples demonstrate prompts that precede function-completion tasks for selected languages.The examples instantiate the general procedure described for few-shot evaluation.Source: §G.2 · p. 32

G.2.1 PYTHON FEW-SHOT PROMPT

• The Python prompt demonstrates regular-expression word extraction, element squaring, and duplicate detection through completed functions and examples.These demonstrations provide multiple Python tasks before a subsequent completion.Source: §G.2.1 · p. 32

G.2.2 JAVASCRIPT FEW-SHOT PROMPT

• The JavaScript prompt demonstrates non-prime detection, digit removal from strings, and distinct-element testing with implementations and input-output examples.The examples use JavaScript function syntax and language-specific operators and methods.Source: §G.2.2 · p. 34

G.2.3 PHP FEW-SHOT PROMPT

• The PHP prompt demonstrates recursive nested-list summation, dictionary-value summation, and squaring list elements.Each task is framed as an expert PHP programming instruction with example calls and outputs.Source: §G.2.3 · pp. 34–35

G.2.4 RUBY FEW-SHOT PROMPT / G.2.5 KOTLIN FEW-SHOT PROMPT

• Ruby and Kotlin prompts provide language-specific demonstrations for list and numeric utilities, including removal, minimum or insertion-point search, word lengths, and filtering.The Ruby examples cover digit and even-number removal plus minimum selection, while Kotlin examples cover insertion points, longest words, and length-based filtering.Source: §G.2.4 · p. 35, §G.2.5 · p. 36

G.1 EVALUATION RESULTS

• Few-shot prompting raises 13B Python-model pass@1 on out-of-domain MBJP from 5.7% to 10.3%.The comparison uses pass@1 at temperature 0.2.Source: §G.1 · p. 32
• Few-shot prompting increases 13B multi-lingual-model pass@1 on MBJP from 5.9% to 12.2%.The passage describes these as large performance boosts for out-of-domain evaluation.Source: §G.1 · p. 32

H.1 TRANSLATION RESULTS FROM VARIOUS LANGUAGE SOURCES

• The translation setup prepends a canonical solution from Python, Java, or JavaScript when available, while otherwise using the usual target-language prompt.Java and JavaScript canonical solutions come from separately trained bootstrapping models.Source: §H.1 · p. 37

H.2 COMPARING TRANSLATION PERFORMANCE OF MULTI-LINGUAL AND MONO-LINGUAL MODELS

• Translation improves most settings across multilingual and Python-, JavaScript-, and Java-only models, while larger models typically perform better in both normal completion and translation.Monolingual models can show large boosts partly because their baseline performance is initially weaker, and target-language knowledge may matter more than source-language knowledge.Source: §H.1 · pp. 37–38, Figure 18 · p. 38, Figure 19 · p. 39

H.1 TRANSLATION RESULTS FROM VARIOUS LANGUAGE SOURCES

• 15.9% versus 5.9% pass@1 for Ruby and 46.5% versus 19.1% for PHP show large zero-shot translation gains over normal completion.The Ruby result uses JavaScript as the source language, while the PHP result uses Java.Source: §H.1 · p. 37
• Translation gains vary by source language: Java works best for MBPHP, whereas JavaScript is consistently best for MBKP and MBRBP, sometimes outperforming syntactically closer sources.For MBJSP, the best source varies by model type, with JavaScript favored by JavaScript and Java models and Python favored in many other settings.Source: §H.1 · p. 37, Table 4 · p. 37

H.3 GENERATED TRANSLATION EXAMPLES

• The generated examples demonstrate cross-language transfer from Python prompts to PHP and Kotlin implementations, preserving task descriptions, examples, and expected outputs.The PHP examples implement repeated-element sums and products, while the Kotlin example sorts matrix rows by their sums.Source: §H.3.1 · p. 42, §H.3.3 · p. 43

I ANALYSIS: EFFECTS OF FEW-SHOT AND TRANSLATION PROMPTS

• Few-shot prompting uses three same-language examples, whereas translation supplies the same problem’s solution in another language; translation helps difficult tasks, while few-shot prompting conditions target syntax.The two settings differ only in the prompt preceding the function-completion prompt.Source: I ANALYSIS: EFFECTS OF FEW-SHOT AND… · p. 45

I.2 SOLVE RATE PER PROBLEM DUE TO FEW-SHOT PROMPTING AND TRANSLATION

• Translation can raise low-baseline task solve rates to 1.0, while few-shot solve-rate accumulations remain closer to baseline across tasks.The analysis samples 100 generations per task at temperature 0.8 and ranks tasks by baseline solve rate.Source: §I.2 · p. 45, Figure 24 · p. 46
• Java helps solve many difficult MBPHP problems, reinforcing that source languages have unequal effects across evaluation targets.The high concentration of MBPHP points near solve rate 1.0 complements the source-language trends reported in Section H.Source: §I.2 · p. 46, Figure 24 · p. 46

I.1 TEST CASE ERROR VERSUS NON-ASSERTION ERROR

• Failures are categorized as assertion or test-based errors versus non-assertion errors, including unrelated compile, parsing, and runtime failures.Non-assertion error fractions use temperature 0.2 generations with 30 samples per problem.Source: §I.1 · p. 45
• Few-shot prompting lowers non-assertion errors for out-of-domain languages, suggesting more precise language-specific syntax generation.The effect is small on in-domain languages, where models are already fluent, but translation produces higher non-assertion errors in in-domain evaluation.Source: §I.1 · p. 45, Figure 22 · p. 45
• Figure 22 reports the percentage of non-assertion errors among all error cases for in-domain and out-of-domain datasets.The figure compares the two dataset regimes separately.Source: Figure 22 · p. 45

J.1 DATASET PREPARATION AND EVALUATION SETUP

• The robustness benchmark measures greedy-decoding average pass@1 after perturbing MBXP docstrings with back-translation paraphrases, character-case changes, or synonym substitutions.These transformations are selected to preserve naturalness and use default NL-Augmenter implementations.Source: §J.1 · p. 47
• The evaluation compares multilingual models trained on Python, Java, and JavaScript with monolingual models trained separately on each language.The benchmark applies the same robustness framework across trained models and MBXP datasets.Source: §J.1 · p. 47
• Ruby, PHP, and Kotlin are treated as out-of-domain datasets, whereas Python, Java, and JavaScript are treated as in-domain datasets for all models.This domain assignment simplifies cross-model robustness comparisons.Source: §J.1 · p. 47

J.2 EVALUATION RESULTS

• Perturbations consistently reduce pass@1, while multilingual models experience smaller percentage drops than monolingual models on both in-domain and out-of-domain datasets.Across model sizes, average pass@1 drops by 8.02%–9.40%; multilingual drops are 7.63% versus 9.81% in-domain and 7.98% versus 8.97% out-of-domain.Source: Figure 25 · p. 47, §J.2 · p. 47

J.3 QUALITATIVE EXAMPLES

• Qualitative MBPP examples show that back-translation paraphrasing, character-case changes, and synonym substitutions can turn previously successful completions into failures.The examples use completions from a 672M Python model and illustrate the perturbation types used for robustness evaluation.Source: §J.3 · pp. 48–49

K.1 DATASET PREPARATION

• The i-MBxP dataset masks consecutive lines from canonical solutions, generating multiple variants per problem while removing duplicate masks.Problems with fewer than two canonical-solution lines are skipped; remaining problems mask m = [1, 8] lines with one to three runs depending on solution length.Source: §K.1 · p. 50

K.2 EVALUATION SETUP

• InCoder-6.7B fills masked code between left and right contexts using sentinel prompts, stopping criteria, deduplication, and comparison with a left-to-right baseline.The model was pretrained on 159 GB of code across Python and 28 other languages, plus 57 GB of StackOverflow text.Source: §K.2 · p. 50

K.3 EVALUATION RESULTS

• Right context significantly boosts insertion accuracy across all evaluated languages, with additional right-context lines further increasing accuracy.InCoder-6.7B is evaluated using greedy search and execution accuracy against a left-to-right baseline.Source: §K.3 · p. 50, §K.2 · p. 50

K.4 QUALITATIVE EXAMPLES FOR I-MBXP

• Qualitative insertion examples show right context guiding implementations that match surrounding code, whereas left-to-right generation can produce inconsistent or incorrect alternatives.Examples cover Python nested loops, Java list construction, and JavaScript character counting.Source: §K.4 · p. 50

L.1 DATASET PREPARATION AND EVALUATION SETUP

• For S-MBXP, the original descriptions are removed, leaving function signatures and canonical solutions as inputs for zero-shot or few-shot code summarization.Few-shot prompts prepend three code-summary pairs, while zero-shot prompts append a language-specific comment prefix.Source: §L.1 · p. 54
• Summarization quality is measured with smoothed BLEU against ground-truth docstrings, following CodeXGLUE because MBXP summaries are short.The summaries typically contain one or two sentences.Source: §L.1 · p. 54

L.2 EVALUATION RESULTS

• On code summarization, larger models performed better, multilingual models surpassed monolingual models, and few-shot gains were larger for larger models; Python BLEU-4 rose from 3.20 to 6.07 zero-shot and from 20.75 to 34.10 few-shot.Monolingual models generalized well to other languages with few-shot examples, while smaller monolingual models were less stable than multilingual models under few-shot prompting.Source: §L.2 · p. 54, Figure 26 · p. 54

L.3 QUALITATIVE EXAMPLES

• Qualitative zero-shot examples in Python, JavaScript, and Java show generations that summarize code functionality, sometimes accurately matching tasks such as discriminant calculation, dictionary combination, and list swapping.The examples also include imperfect generations, such as describing date conversion as writing a .txt file.Source: §L.3 · pp. 54–55
• Few-shot examples demonstrate cross-language code summarization for operations including regex extraction, string rotations, squaring list elements, tuple checks, and recursive gcd computation.The examples span Python, Java, and JavaScript and pair code with summaries and ground-truth prompts.Source: §L.3 · pp. 55–57

M EVALUATING PUBLIC MODELS

• Public-model results followed the same log-linear size trends, with better execution accuracy in in-domain languages across OPT, BLOOM, and CodeGen families.The evaluation used MBXP pass@1 with greedy decoding, generating 256 tokens per example and truncating outputs to one function.Source: M EVALUATING PUBLIC MODELS · p. 59, Table 6 · p. 60, Table 7 · p. 61
• BLOOM models outperformed OPT models, consistent with BLOOM’s code-containing pretraining data and relatively similar performance across languages except Kotlin and Ruby.BLOOM used code from PHP, Java, Python, JavaScript, and Ruby among other languages, whereas OPT did not train specifically on code.Source: M EVALUATING PUBLIC MODELS · p. 59, Table 6 · p. 60
• CodeGen-mono 16B improved over CodeGen-multi by 6% on JavaScript and 8% on PHP, while larger models showed better-than-log-linear gains on out-of-domain languages.The out-of-domain languages highlighted are Kotlin, Java, and PHP, and the authors speculate that additional Python training improves other languages.Source: M EVALUATING PUBLIC MODELS · p. 59, Table 6 · p. 60
• Ruby accuracy increased from 3.5% to 16.46% with few-shot prompting on CodeGen-multi models, while translation mode improved significantly in all languages except Ruby.These results demonstrate substantial gains in out-of-domain languages from few-shot prompting and translation-based evaluation.Source: M EVALUATING PUBLIC MODELS · p. 59, Table 8 · p. 61, Table 10 · p. 62

N.1 MODEL ARCHITECTURE AND TRAINING DETAILS

• Training used 210B tokens for mono-lingual models and 630B tokens for multi-lingual models, with 2048-token sequences, AdamW, cosine annealing, bfloat16, and DeepSpeed.For 13B models, each language used batch size 1024 and 100,000 steps; multi-lingual training tripled the steps for three languages, and 0.1% of data was held out for validation.Source: §N.1 · p. 63, Figure 27 · p. 63

N.2 OBSERVATIONS ON VALIDATION LOSSES VERSUS PERFORMANCE

• Validation loss follows an approximate log-linear relationship with model size, while multilingual models consistently incur higher loss because they fit more diverse information.The comparison covers 128M, 672M, 2.7B, and 13B multilingual and monolingual models.Source: §N.2 · pp. 63–64, Figure 28 · p. 64
• As model size increases, multilingual models increasingly outperform monolingual models on execution pass@k, despite monolingual models having lower validation loss.Multilingual models benefit from cross-language knowledge sharing and transfer, particularly beyond 672M parameters.Source: §N.2 · pp. 63–64

O DATASET CONVERSION FRAMEWORK · O.1.1 FORMAT CHOICE

• The conversion framework targets execution-based function completion, where correctness is tested by executing converted test statements against generated function bodies.Each problem contains a prompt, test statement, and canonical solution; only prompts and tests need conversion because canonical solutions are not used for correctness.Source: O DATASET CONVERSION FRAMEWORK · p. 65

O.1.1 FORMAT CHOICE · O.1.2 INFERRING ARGUMENT AND RETURN TYPES · O.1.3 SUPPORTED TYPES OF OBJECT CONVERSION

• The framework standardizes datasets around function signatures, infers target-language argument and return types recursively, and supports common integers, strings, null values, lists, tuples, and nested containers.Python types are mapped to target-language types such as list to ArrayList and dict to HashMap, using common supertypes across test cases.Source: §O.1.1 · p. 65, §O.1.2 · p. 65, §O.1.3 · p. 65

O.1.3 SUPPORTED TYPES OF OBJECT CONVERSION · O.1.4 CONSTRUCTING REPRESENTATIONS OF CODE OBJECTS · O.1.5 CONVERTING TEST STATEMENT

• It recursively constructs target-language object representations and test statements with language-appropriate assertions and deep comparison for nested values.For example, Python [1,2,3] becomes Java Arrays.asList(1, 2, 3).Source: §O.1.3 · p. 66, §O.1.4 · p. 66, §O.1.5 · p. 66

O.1.6 PROMPT CONSTRUCTION · O.1.7 DOCSTRING AND NATURAL LANGUAGE CONVERSION · O.1.8 VALIDATION

• Prompt conversion preserves language conventions by adapting naming, formatting, reserved words, docstring placement, and language-specific natural-language terms such as Python and None.The framework validates that converted objects, tests, and signatures parse or compile in each target language.Source: §O.1.6 · p. 66, §O.1.7 · p. 66, §O.1.8 · p. 66

O.1.9 QUALITY CHECK VIA REVIEWERS

• Manual reviews of Java and JavaScript samples found no issues specifically attributable to language conversion, although some ambiguity came from the original datasets.Reviewer feedback was used to iteratively improve the conversion algorithm.Source: §O.1.9 · p. 66

O.1.10 COMPARISON TO OFF-THE-SHELF ML TRANSLATOR · O.1.11 SYNTHETIC CANONICAL SOLUTIONS

• The framework avoids machine translators’ limited language-pair coverage, format restrictions, and nondeterministic type or object-construction errors, while synthetic canonical solutions achieve 96% JavaScript, 93% Java, and 93% PHP coverage.Solutions are generated by sampling up to 10,000 versions per problem and filtering with converted tests, using function completion and zero-shot translation settings.Source: §O.1.10 · p. 67, §O.1.11 · p. 67

O.2 POTENTIAL USE OF TRANSCODER FOR DATASET CONSTRUCTION

• Transcoder supports only Python, Java, and C++, and type-inference errors can create false negatives, whereas the proposed static-analysis framework avoids these errors and scales to 10+ languages.Examples include incorrect boolean-to-int returns and list types mapped to strings; Transcoder can also produce nonstandard or inconsistent function names.Source: §O.2 · pp. 67–68, Table 11 · p. 72

P.1 MULTI-STAGE DATA BOOTSTRAPPING

• Synthetic canonical-solution generation combines normal, few-shot, and translation modes, escalating from 100 to 1,000 to 10,000 samples with temperatures 0.8, 1.0, and 1.2.Different generation modes are used because evaluation analyses indicate they can produce correct solutions for different problems.Source: §P.1 · p. 68

P.2 DISCUSSION: GROUND TRUTH ASSUMPTIONS OF TEST CASES

• Execution-based filtering assumes test cases are necessary and sufficient correctness conditions, but false positives and incomplete tests can undermine that ground-truth assumption.Additional tests generated from canonical solutions are proposed, but this also depends on canonical solutions being fully correct; the investigation is left for future work.Source: §P.2 · pp. 68–69

Q QUALITY CHECK OF CONVERTED DATASETS

• Manual review of converted Java and JavaScript problems found ambiguous statements and incorrect test cases, issues attributed almost entirely to the source dataset rather than the conversion process.The authors plan to correct the original MBPP and propagate changes automatically to converted datasets.Source: Q QUALITY CHECK OF CONVERTED DATASETS · pp. 70–71

R.1 MBXP

• MBXP converts MBPP into a HumanEval-style function-signature-and-docstring format and provides corresponding datasets across 10+ programming languages.The examples show parallel prompts, completions, and tests for languages including Python, Java, JavaScript, TypeScript, Go, C#, PHP, Ruby, Kotlin, C++, and Perl.Source: Table 11 · p. 72, §R.1.2 · p. 73, §R.1.3 · p. 73

R.1.12 MBSCP: SCALA

• Scala and Swift examples show generated minimum-cost-path completions paired with canonical test cases whose expected outputs are 8, 12, and 16.The examples use the same three matrix inputs and verify each result with assertions.Source: §R.1.12 · pp. 80–81, §R.1.13 · p. 81

R.2 MULTI-LINGUAL HUMANEVAL

• Multilingual HumanEval expands implicit Python loop tests into explicit assertions, retaining 161 of 164 cases after filtering user-defined-function cases.The converted benchmark format is made compatible with execution-based evaluation across languages.Source: §R.2 · p. 82

R.3 MULTI-LINGUAL MATHQA

• MathQA-Python is extended into MathQA-Java and MathQA-JavaScript by converting problems to a unified function-and-test format and applying rule-based language conversion.Each problem preserves its short mathematical description and answer-derived test case for execution-based evaluation.Source: §R.3 · p. 82
• MathQA multilingual tests use tolerance-based floating-point comparison, with the shown example accepting the answer 38.0 within 1e-8 in Python and 1e-7 in Java and JavaScript.The comparison logic avoids false negatives from floating-point representation while supporting execution-based checks.Source: §R.3 · pp. 82–84