PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs

Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Carolyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, Arjun Guha

arXiv:2308.09895v6cs.PLcs.LG

TL;DR

Code LLMs perform unevenly across programming languages, with weaker performance in low-resource languages. The paper introduces MultiPL-T, which creates test-validated training data by translating high-resource code, and reports stronger fine-tuned models across several low-resource languages.

  • Problem

    Code LLMs perform well on languages with abundant training data but struggle with low-resource languages, where evidence on their broader coding capabilities is also needed.

    Source: §1 · p. 2, §6 · p. 26

  • Method

    MultiPL-T translates high-resource functions into target-language training data and filters translations using validated source-language tests compiled for the target language.

    Source: §1 · p. 3

  • Results

    MultiPL-T fine-tunes outperform base models across model families, with the largest gains on the lowest-resource languages and small-to-mid-sized models, and achieve state-of-the-art results against same-base fine-tunes across five languages.

    Source: §5.2.3 · p. 18, §8 · p. 27

  • Takeaways & Limitations

    The paper concludes that MultiPL-T is more effective and efficient than training longer on existing data, and its data can be reused to train multiple models.

    Source: §8 · p. 27

    MultiPL-T training items are standalone functions without dependencies on other functions, classes, or third-party libraries, while conventional benchmarks largely test that same format.

    Source: §5.3.1 · p. 19

Abstract

from arXiv · show

Over the past few years, Large Language Models of Code (Code LLMs) have started to have a significant impact on programming practice. Code LLMs are also emerging as building blocks for research in programming languages and software engineering. However, Code LLMs produce impressive results on programming languages that are well represented in their training data (e.g., Java, Python, or JavaScript), but struggle with low-resource languages that have limited training data available. Low resource languages include OCaml, Racket, and several others. This paper presents an effective approach for boosting the performance of Code LLMs on low-resource languages using semi-synthetic data. Our approach, MultiPL-T, translates training data from high-resource languages into training data for low-resource languages in the following way. 1) We use a Code LLM to synthesize tests for commented code from a high-resource language, filtering out faulty tests and code with low test coverage. 2) We use a Code LLM to translate Python code to a target low-resource language, and use tests to validate the translation. We apply this approach to generate tens of thousands of validated training items for Julia, Lua, OCaml, R, and Racket. Furthermore, we use an open model (StarCoderBase) with open training data (The Stack), which allows us to decontaminate benchmarks, train models without violating licenses, and run experiments that could not otherwise be done. With MultiPL-T generated data, we present fine-tuned versions of StarCoderBase and Code Llama for Julia, Lua, OCaml, R, and Racket. On established benchmarks (MultiPL-E), these models outperform other open Code LLMs. The MultiPL-T approach is easy to apply to new languages, and is significantly more efficient and effective than alternatives such as training longer.

1 Introduction

MultiPL-T addresses uneven Code LLM performance across languages by transferring high-resource code into test-validated training data for low-resource languages. The paper reports improved fine-tuned models, datasets for five languages, and evaluation against alternative fine-tuning approaches.

Motivation

  • Code LLMs produce stronger code in high-resource languages than in low-resource languages, leaving programmers using languages such as Racket and OCaml with fewer benefits.The paper links this disparity to differences in the amount of language data available for training.Source: §1 · p. 2
  • Finding more low-resource-language data is difficult, while training longer can hurt performance and directly generated synthetic code is poor quality.The paper notes that The Stack already includes permissively licensed code for hundreds of languages.Source: §1 · p. 2

Our approach

  • MultiPL-T transfers high-resource code by translating functions, generating and validating tests in the source language, compiling the tests, and filtering target translations by test results.The approach relies on documented high-resource code, test generation, test compilation, and retries or prompting to obtain passing translations.Source: §1 · p. 3, Figure 2 · p. 4

Contributions

  • MultiPL-T fine-tunes achieve state-of-the-art performance on five low-resource languages and are reported as more efficient and effective than alternatives such as training longer.The evaluation includes comparisons with longer training and other fine-tuning approaches.Source: §1 · pp. 4–5
  • The paper releases tens of thousands of tested training items for Julia, Lua, OCaml, R, and Racket, plus 133,168 documented Python functions with validated, high-coverage tests.The Python functions are presented as a resource for generating fine-tuning sets for other languages.Source: §1 · p. 4
  • MultiPL-T fine-tunes outperform prior fine-tunes of StarCoderBase and Code Llama on low-resource languages, including Racket fine-tunes for newer model families.The paper also reports that data generated with StarCoderBase-15B improves larger Code Llama models.Source: §1 · pp. 4–5

2 Background

The background introduces Code LLM training and fine-tuning, the natural-language-to-code task and its evaluation, and the reasons for using StarCoder in much of the work.

2 Background

  • Code LLMs are trained on large code corpora, and StarCoderBase uses a filtered subset of The Stack spanning 86 programming languages.The Stack v1.2 contains 3TB of permissively licensed source code across 358 languages.Source: §2.1 · p. 5

2.1 Training and Fine-Tuning Large Language Models of Code

  • Fine-tuning uses fewer resources than pretraining, and the paper uses StarCoder models for most experiments while also testing transfer to larger Code Llama models.StarCoderBase-1B is used for most experiments, StarCoderBase-15B for translations, and StarCoder-15B for Python test generation.Source: §2.1 · pp. 5–6

2.2 Code LLM Tasks and Benchmarking Code LLMs

  • Natural-language-to-code benchmarks prompt a model with a function signature and comment, then test its generated function; the paper uses pass@1 and extends MultiPL-E to OCaml.Pass@1 measures whether a correct solution is produced in a single attempt.Source: §2.2 · pp. 6–7

2.3 Why StarCoder?

  • StarCoder was selected for open training data, permissive licensing, and a small 1B model suitable for budget experiments; open data also supports contamination checks and comparisons with longer training.The paper notes that Code Llama's undisclosed training data prevents the same exhaustive evaluation.Source: §2.1 · p. 5, §2.3 · p. 7

3 Alternatives to MultiPL-T

The section tests two alternatives to MultiPL-T for improving low-resource code generation: further training on natural code and self-instruction. Both yield poor results in the reported experiments.

3.1 Further Training on Natural Data

  • Fine-tuning on The Stack barely improves Racket and OCaml performance and decreases Lua performance; size-matched MultiPL-T data instead shows significant improvement.The full The Stack datasets vary greatly in size, so the size-matched comparison controls for training-data volume.Source: §3.1 · p. 8, Figure 4 · p. 8

3.2 Self-Instruction for Low-Resource Programming Languages

  • Four of five Racket programs generated by StarCoderBase-15B from five hand-written examples have bugs.The authors describe this as a higher error rate than is evident in self-instruct datasets for high-resource languages [Muennighoff et al. 2024].Source: §3.2 · p. 8, Figure 5 · p. 9
  • Self-instruction and further training on existing public data do not improve Code LLM performance on low-resource languages.The authors therefore turn to MultiPL-T.Source: §3.2 · p. 9

4 Our Approach

MultiPL-T creates semi-synthetic training data by filtering documented Python functions, generating and validating tests, translating functions into target languages, and deduplicating passing translations.

4.3 Translation from a High-Resource to a Low-Resource Language

  • The pipeline prompts a Code LLM to translate Python functions and validates candidate translations with compiled tests in the target-language runtime.Prompts include the translated docstring and signature plus the original Python code as a comment; 50 candidate translations are generated per function, and only solutions passing all tests are retained.Source: §4 · p. 10, §4.3 · p. 14

4.1 Filtering Data from a High-Resource Language for Translation and Test Generation

  • 432,361 Python functions remain after filtering The Stack’s source data for translation and test generation.The filtering is motivated by the GPU expense of processing all 22 million functions; it selects documented, typeable functions and removes incomplete or benchmark-matching examples.Source: §4.1 · p. 11

4.2 Generating Python Unit Tests

  • 133,668 Python functions remain after generated test suites are validated and functions below 90% line coverage are discarded.StarCoder-15B generates five independent test suites per function; generated tests are run individually, and functions with no passing tests are discarded.Source: §4.2 · p. 13
  • For typed targets such as OCaml and Julia, test-derived Python types guide target-language annotations and test-case compilation.Types are inferred from argument and expected-return types in tests, combining observed types into unions and simplifying Union[T, None] to Optional[T].Source: §4.2 · pp. 13–14

4.3 Translation from a High-Resource to a Low-Resource Language

  • MultiPL-T compiles Python tests to each target language, discarding items when tests cannot be compiled or none remain.The test compiler handles simple equality assertions over atomic values and collections.Source: §4.3 · pp. 14–15

4.4 Deduplication

  • ROUGE-L deduplication removes near-duplicate passing solutions to preserve code-form diversity in the training data.The method strips comments, uses a 0.6 similarity threshold, and repeats grouped comparisons after random regrouping.Source: §4.4 · pp. 15–16

5 Evaluation

The evaluation reports benchmark results, additional qualitative and quantitative analyses, and ablations of MultiPL-T design choices.

5 Evaluation

  • The evaluation reports state-of-the-art results on standard natural-language-to-code benchmarks.The section also presents novel qualitative and quantitative evaluation.Source: §5 · p. 16
  • The evaluation includes ablations assessing the significance of MultiPL-T design decisions.Source: §5 · p. 16
  • The evaluation fine-tunes a variety of models of different sizes with MultiPL-T.Source: §5 · p. 16

5.1 Experimental Setup and Implementation

The implementation uses different fine-tuning configurations by model size and relies on substantial GPU time, with dataset generation the largest reported resource use.

5.1 Experimental Setup and Implementation

  • StarCoderBase-1B is fine-tuned for seven epochs with batch size 8 and learning rate 3 × 10−5; larger listed models use ten epochs, batch size 32, and learning rate 2 × 10−5.All models use a sequence length of 2,048 tokens; the larger-model configuration applies to StarCoderBase-15B, CodeLlama-34B, and CodeLlama-70B.Source: §5.1 · p. 16
  • Dataset generation uses approximately 9,984 GPU hours, the largest listed resource expenditure within an estimated 550 A100 GPU days.The work also reports approximately 3,444 hours for training and 310 hours for evaluation.Source: §5.1 · p. 16
  • The final pipeline uses vLLM for inference, DeepSpeed ZeRO for larger-model fine-tuning, Transformers, and a modified MultiPL-E toolchain.The MultiPL-E modifications include OCaml support.Source: §5.1 · p. 16

5.2 Evaluation on Standard Benchmarks

MultiPL-T fine-tuning improves MultiPL-E performance across target languages and model families, with the strongest gains generally appearing for the lowest-resource languages. The results also show that data generated by a smaller model can improve larger models, while comparisons with newer models and larger datasets require care.

5.2.1 Fine-Tuning StarCoderBase and Code Llama.

  • MultiPL-T fine-tuning improves MultiPL-E performance across the target languages for StarCoderBase and Code Llama.The reported comparison uses peak performance across checkpoints.Source: Table 2 · p. 17, §5.2.1 · p. 17
  • Racket and OCaml show the largest relative gains, with both models’ fine-tuned versions scoring more than twice their base models.Fine-tuned 1B models perform comparably to base 15B models on these languages.Source: §5.2.1 · p. 17

5.2.2 Fine-Tuning StarCoder2 and DeepSeek Coder.

  • Fine-tuned DeepSeek Coder and StarCoder2 both improve on Racket, although their MultiPL-T datasets are included in StarCoder2’s pretraining corpus.The authors conclude only that the data did not hurt target-language performance and further fine-tuning still helps.Source: Table 3 · p. 18, §5.2.2 · p. 18

5.2.1 Fine-Tuning StarCoderBase and Code Llama.

  • Data generated by StarCoderBase-15B improves Code Llama models nearly 5x larger, reversing the usual larger-model-to-smaller-model direction of distillation.The paper also reports improvements for CodeLlama-34B and CodeLlama-70B.Source: §5.2.1 · p. 18

5.2.3 Summary.

  • Across four model families and sizes from 1B to 70B, fine-tuned models outperform their base models, with larger gains for the lowest-resource languages and small-to-mid-sized models.The 70B models show less improvement than the smallest models.Source: §5.2.3 · p. 18
  • At the time of writing, the fine-tuned DeepSeek Coder model outperforms every fine-tuned model on the BigCode Models Leaderboard for Racket.The compared leaderboard models used an order of magnitude more data or distilled proprietary models; the authors conclude MultiPL-T is more data efficient for low-resource languages.Source: §5.2.3 · p. 18

5.3 New Tasks and Qualitative Evaluation

The paper tests whether fine-tuning on single-function training items affects performance on tasks with broader context and the idiomatic style of generated Racket. Most fine-tuned models do as well or better on the new benchmark, while Racket style scores decline slightly.

5.3.1 Evaluating In-Context Learning.

  • The authors construct a 14-problem benchmark to test user-defined types, higher-order functions, helper functions, and external libraries, with hidden tests and prompts manually translated into OCaml, Racket, and Lua.Manual construction was intended to make problems similar across languages and absent from training data.Source: §5.3.1 · p. 19
  • Most fine-tuned models perform as well or better on the new benchmark, though the 15B OCaml-tuned model may have overfit to the MultiPL-T format.The authors speculate that mixing natural and MultiPL-T data could reduce overfitting likelihood.Source: §5.3.1 · p. 19

5.3.2 Evaluating Coding Style.

  • On 35 HumanEval problems solvable by both models, the Racket style score falls from 89.5% for the base model to 85.2% after fine-tuning.Mean grades were 13.4/15 for the base model and 12.8/15 for the fine-tuned model.Source: §5.3.2 · p. 20, Figure 11 · p. 22
  • Fine-tuned solutions more often use nested if expressions and iteration where recursion is available, while base solutions more often use car/cdr instead of Racket list abstractions.The inspected sets comprised 17 problems scored higher for the base model and eight scored higher for the fine-tuned model.Source: §5.3.2 · pp. 20–21
  • Fine-tuning slightly reduces idiomatic Racket style while significantly increasing correctness, and both models score well on the rubric, suggesting a minimal trade-off.The correctness increase is reported earlier in the paper; the rubric evaluation concerns Racket style.Source: §5.3.2 · pp. 21–22

5.4 MultiPL-T Ablations

The ablations examine the efficiency and components of MultiPL-T, including translation prompts, test validation, deduplication, and direct fine-tuning on source-language data. They show that outcomes vary by target language and that data validation and deduplication matter to the pipeline’s effectiveness and cost.

5.4.1 Fine-Tuning Efficiency.

  • After one epoch, performance increases substantially for low-resource Racket and OCaml, while higher-resource Lua needs more data for even modest gains.The efficiency comparison does not include the computing cost of generating the training data, though generated data can be reused across models.Source: §5.4.1 · p. 23

5.4.2 Translation Versus Generation.

  • Including original Python code in translation prompts substantially raises successful-translation likelihood for Racket and Lua but slightly lowers it for OCaml.The experiment compares pass@50 across 1,000 source functions and two prompt formats.Source: §5.4.2 · p. 23, Table 7 · p. 23

5.4.3 Testing Is Critical for MultiPL-T.

  • Without test-based validation, translated Racket training items are poor quality and fine-tuning is not effective.The table reports a slight decrease for the 15B model and a slight increase for the 1B model, but the 1B result remains far below fine-tuning with tested items.Source: Table 9 · p. 24, §5.4.3 · p. 24

5.4.4 The Impact of Deduplication.

  • 17.1 pass@1 on MultiPL-E and 12 hours to train without deduplication compare with 17.3 pass@1 and less than 30 minutes with deduplication.Deduplication reduces training time substantially while the two scores remain very close; the authors note larger models would cost even more to train without it.Source: §5.4.4 · p. 24

5.4.5 Targeting the Source Language.

  • 15.1 pass@1 on Python for the base model compares with 15.0 after fine-tuning directly on the filtered Python dataset, unlike gains from translating data to low-resource languages.The authors hypothesize that the base model had already trained on three epochs of this Python data, whereas translation creates novel data for it.Source: §5.4.5 · pp. 24–25

6 Discussion

The discussion considers extending MultiPL-T to more languages and models, while noting that its usefulness depends on language and data conditions. It also identifies gaps in low-resource-language evaluation and a boundary for applying the approach to languages absent from pretraining.

Generalizing to other programming languages.

  • Adapting MultiPL-T to another language involves translating comments and function signatures for prompts and writing a compiler for simple source assertions.The paper notes MultiPL-E already supports these steps for more than 20 languages, including D, Bash, MATLAB, Haskell, and Perl.Source: §6 · p. 25
  • Some source-target pairs may work better than others; the authors speculate that C++ may be a better source than Python for generating Rust data.Source: §6 · p. 25

Generalizing to other LLMs.

  • The authors expect the approach to generalize beyond the StarCoder and Code Llama families after demonstrating Racket fine-tuning with DeepSeek Coder and StarCoder2.Source: §6 · p. 25

Regenerating MultiPL-T data.

  • StarCoderBase-15B-generated data improves CodeLlama-70B, DeepSeekCoder-34B, and StarCoder2-15B, despite being produced by a weaker model.The paper says a more capable generator would improve conversion rates, but that using GPT-4 would violate its terms of use.Source: §6 · p. 25

“No-resource” languages.

  • MultiPL-T is unlikely to work as-is for languages on which StarCoderBase was not trained, though prompting with enough language information might bootstrap data generation.The authors note that doing so efficiently may be challenging.Source: §6 · p. 25

Composability with self-instruct.

  • MultiPL-T could translate high-quality instruction datasets from a high-resource language, potentially composing with self-instruct approaches effective on Python.Source: §6 · p. 25

Other kinds of benchmarks.

  • The authors call for specialized and population-specific benchmarks for low-resource languages to better understand Code LLM capabilities and limitations.Source: §6 · pp. 25–26

7 Related Work

Related work explores code translation, cross-language transfer, instruction tuning, and high-quality data as ways to improve code models. MultiPL-T distinguishes itself by using an off-the-shelf pretrained Code LLM for test generation and translation, with open models and data enabling comparison against training longer.

Code translation with language models and unit tests

  • MultiPL-T uses an off-the-shelf pretrained Code LLM for both test generation and low-resource code translation, unlike earlier translation-data approaches.The authors argue earlier open models were less capable, while commercial models were too costly at the required query scale.Source: §7 · p. 26

Multi-lingual language models of code

  • Multilingual transfer can improve low-resource code generation, but may help less for syntactically or semantically distant languages such as Racket.Prior studies also rely mainly on BLEU and other syntax-based metrics, which the authors describe as inadequate for evaluating code generation.Source: §7 · p. 26
  • Code LLMs show language-dependent performance across natural-language-to-code benchmarks, with similar patterns reported for fuzzing and translation.The cited benchmarks include [Athiwaratkun et al. 2022; Orlanski et al. 2023].Source: §7 · p. 26

Instruction tuning

  • Instruction-tuning datasets largely target high-resource languages, and the authors argue these techniques are unlikely to succeed for low-resource languages.Self-instruction and instruction variation are presented as cheaper alternatives to human-written instruction data.Source: §7 · pp. 26–27

Training on high-quality data

  • High-quality synthetic training data has supported strong Python results, but the authors argue this approach is less likely to succeed for low-resource languages.The cited work reports high HumanEval scores for a small model trained on a modest amount of “textbook quality” data [Gunasekar et al. 2023].Source: §7 · p. 27

Proprietary Code LLMs

  • Open models and data let MultiPL-T compare fine-tuning with the natural baseline of training longer, while proprietary models often restrict training or generated-data use.The authors state MultiPL-T data fine-tuning is better than training longer on existing data for StarCoderBase and its training set.Source: §7 · p. 27

8 Conclusion

Code LLM quality varies by programming language, and low-resource languages may fare worse as developer tools increasingly rely on these models. The paper presents open MultiPL-T data and reports that its fine-tunes outperform other fine-tunes of the same base models across five languages.

8 Conclusion

  • Code LLM quality is highly language-dependent, and the authors hope MultiPL-T datasets will help low-resource languages compete in developer tools that rely on LLMs.The paper describes MultiPL-T as a methodology for generating large-scale, high-quality fine-tuning datasets in low-resource languages.Source: §8 · p. 27
  • MultiPL-T data and code are open, built from The Stack and augmented by StarCoder rather than a more capable proprietary model.This choice allows comparison of MultiPL-T fine-tuning with training longer on existing data.Source: §8 · p. 27
  • MultiPL-T achieves state-of-the-art results for Julia, Lua, OCaml, R, and Racket among fine-tunes of the same base model.The paper also evaluates in-context learning with a new benchmark and coding style qualitatively.Source: §8 · p. 27

Data-Availability Statement

The paper makes its code, data, and models available under open licenses, with links to the code, datasets, and fine-tuned models.

Data-Availability Statement

Loading 2308.09895v6…