Enhancing Code Generation for Low-Resource Languages: No Silver Bullet

I. INTRODUCTION

• Low-resource languages have scarce public training data, which hampers LLMs’ ability to generalize to their constructs and edge cases.The paper contrasts these niche languages with high-resource languages such as Python and Java, for which training draws on extensive code examples.Source: §I · p. 1
• Prior work proposed fine-tuning and few-shot learning, but a comparative study across techniques and model sizes was missing.Fine-tuning is constrained by limited training data; earlier work reported benefits, though performance often remained below high-resource languages.Source: §I · p. 1
• The study compares fine-tuning, few-shot learning, and newly devised translation prompts across six LLMs, and tests pre-training for high-to-low-resource code translation.The evaluated models include DeepSeek Coder 1B, 7B, and 33B, Code Llama 7B and 13B, and GitHub Copilot.Source: §I · pp. 1–2
• The experiments focus on R and Racket after initial evaluation found stronger high-versus-low-resource performance gaps for them than for Julia and Lua.The study uses code-generation problems with descriptions and tests, and fine-tunes with public datasets containing 37,592 R functions and 40,489 Racket functions.Source: §I · p. 2

II. RELATED WORK

• Related work includes multilingual fine-tuning, semantic-similarity-based fine-tuning, balanced-language training, few-shot learning, and generated low-resource fine-tuning datasets.The study uses the MultiPL-T datasets from Cassano et al. as one state-of-the-art fine-tuning approach.Source: §II · p. 2
• The study addresses the reported scarcity of niche-language benchmarks by evaluating LLM performance and comparing strategies for improving low-resource code generation.A recent survey highlighted benchmark scarcity and the importance of experimenting with more advanced techniques [22].Source: §II · p. 3

III. ON THE GAP IN PERFORMANCE FOR LLM-BASED CODE GENERATION IN HIGH- AND LOW-RESOURCE LANGUAGES

• The study asks how state-of-the-art LLM performance differs when generating code in high- and low-resource programming languages.The question is motivated by prior evidence of a gap and the lack of earlier study on GitHub Copilot in this setting.Source: §III · p. 3

A. Context Selection: Programming Languages

• The language set comprises Java and Python as high-resource languages, and Julia, Lua, R, and Racket as low-resource languages.The paper reports 135k and 371k public repositories with at least 10 stars for Java and Python, versus 6k Julia, 16k Lua, 16k R, and 1k Racket repositories.Source: A. Context Selection: Programming… · p. 3

B. Context Selection: LLMs

• The model set comprises DeepSeek Coder, Code Llama, and GitHub Copilot, with open-source model sizes spanning 1B, 7B, 13B, and 33B parameters.The study uses instruct versions of the open-source models; larger Code Llama variants were excluded because inference was extremely expensive.Source: B. Context Selection: LLMs · p. 3

C. Models’ Evaluation

• The evaluation uses 157 HumanEval programs common across languages, checking whether generated implementations pass their unit tests.The original benchmark has 164 programs; only 157 translated programs are common to all languages considered.Source: C. Models’ Evaluation · p. 3
• Performance is measured with pass@1 over 50 repetitions, using a generation temperature of 0.2.A program counts as correct when a generation passes all provided unit tests.Source: C. Models’ Evaluation · p. 3

D. Data Analysis

• Language comparisons use McNemar’s test on paired correct/wrong outcomes, Benjamini-Hochberg-adjusted p-values, and odds ratios for effect size.The same code-generation problems across languages allow pairwise comparison of outcomes.Source: D. Data Analysis · p. 4

E. Results Discussion

• The language results form three clusters: Python and Java usually lead, Julia and Lua are often close to them, and R and Racket are weakest.DeepSeek Coder 1B on Julia is an exception to the generally strong Julia/Lua results; Copilot performs slightly better on Lua than Java.Source: E. Results Discussion · p. 4
• R and Racket pass@1 ranges from 7% to 33.1%, versus 19.2% to 61.4% for Julia and Lua and 30.6% to 74.9% for Java and Python.Average gaps against Java/Python are 24.3%/34.2% for R and 28.7%/38.6% for Racket, compared with 10.9%/20.8% for Julia and 3.6%/13.5% for Lua.Source: E. Results Discussion · p. 4
• Differences between high- and low-resource languages and between Julia/Lua and R/Racket are statistically significant for all evaluated models, except Copilot’s Python-versus-Lua comparison.The reported odds ratios indicate larger gaps for R/Racket than for Julia/Lua.Source: E. Results Discussion · p. 4, Table II · p. 5
• Sampled generations include unsupported constructs such as Julia’s `push` instead of `push!`, as well as failures to handle edge cases and translate requirements into the target language.In the R example, the generation returns null rather than an empty list and a vector rather than a list; Python and Lua generations pass the tests.Source: E. Results Discussion · pp. 4–5

IV. STUDYING TECHNIQUES TO BOOST CODE GENERATION ON LOW-RESOURCE LANGUAGES

• The research question is which techniques best improve state-of-the-art LLM code generation in low-resource programming languages.The question follows findings of a major performance gap, particularly for R and Racket.Source: §IV · p. 6
• The study targets R and Racket because these languages showed a major performance drop.Source: §IV · p. 6
• The comparison uses DeepSeek Coder 1B, 7B, and 33B, Code Llama 7B and 13B, and GitHub Copilot.Source: §IV · p. 6

A. Experimented Techniques

• The study groups five techniques into in-context learning, which avoids weight updates, and fine-tuning, which requires additional target-language training.In-context learning is cheaper in time and resources, while fine-tuning is more expensive and unavailable for some commercial tools such as Copilot.Source: A. Experimented Techniques · p. 6

1) In-context Learning – Translation Examples:

• Translation-example prompts give the model two Python-to-target-language function pairs, using Python for its popularity and succinctness.The short examples limit prompt space because inference cost increases with prompt size.Source: 1) In-context Learning – Translation… · p. 6
• The examples cover basic constructs such as conditionals and assignments, as well as data structures and function calls; the generation prompt matches RQ1.Examples come from MultiPL-T.Source: 1) In-context Learning – Translation… · p. 6

2) In-context Learning – Translation Rules:

• Translation-rule prompts provide predefined Python-to-target mapping rules spanning simple constructs and advanced features such as data structures and error handling.The rules were defined from official language documentation and avoid the need to find or manually craft translation examples.Source: 2) In-context Learning – Translation… · p. 6

3) In-context Learning – Few-shot:

• Few-shot prompting uses two low-resource-language examples pairing a function description with its implementation, matching examples across prompting approaches for comparison.Keeping the examples constant helps distinguish prompt-schema effects from differences in the demonstrated code.Source: 3) In-context Learning – Few-shot · pp. 6–7

4) Fine-tuning – Code Generation:

• 37,592 R functions and 40,489 Racket functions were available for fine-tuning, with docstring and signature as input and the function body as output.The data derives from translated functions released by Cassano et al..Source: 4) Fine-tuning – Code Generation · p. 7

5) Pre-training and Fine-tuning – Code Translation and Generation:

• The translation pre-training dataset is built by matching fine-tuning entries to original Python functions using function names and docstrings, excluding uncertain matches.Docstring matching can be complicated by translation-time edits, such as replacing “dictionary” with “hash table” in Racket.Source: 5) Pre-training and Fine-tuning – Code… · p. 7
• 22,796 R translation pairs and 25,390 Racket translation pairs were assembled for pre-training before code-generation fine-tuning.The pre-training stage uses already-trained released models and teaches Python-to-target translation; the authors call it pre-training because it precedes code-generation fine-tuning.Source: 5) Pre-training and Fine-tuning – Code… · p. 7

B. Training Procedure

• Fine-tuning was limited to DeepSeek Coder and Code Llama, since Copilot cannot be fine-tuned.Source: B. Training Procedure · p. 7
• 3 epochs were used for all models because loss showed no substantial benefit from the second to the third epoch.Longer training could slightly improve results, but training costs are extremely high for models such as DeepSeek Coder 33B.Source: B. Training Procedure · p. 7

• Fine-tuned models are evaluated at each epoch, but only the best model’s MultiPL-E results are reported because training on small datasets for many epochs may degrade performance.Results for all three fine-tuning epochs are available in the replication package.Source: C. Models’ Evaluation and Data Analysis · p. 8
• The evaluation covers all 161 MultiPL-E programs for both R and Racket, comparing six models across five techniques, except Copilot, which is tested only with in-context learning.Source: C. Models’ Evaluation and Data Analysis · p. 8
• Each technique is assessed using pass@1 rates over 50 repetitions, yielding 8,050 values per technique comparison.Source: C. Models’ Evaluation and Data Analysis · p. 8
• Fine-tuning uses a 2,048-token input limit, while in-context prompts use 3,072 tokens; differences are analyzed with McNemar’s test, Odds Ratios, and Benjamini-Hochberg-adjusted p-values.Source: C. Models’ Evaluation and Data Analysis · p. 8

D. Results Discussion

• 7.0% to 18.4%: pre-training plus fine-tuning raises DeepSeek Coder 1B's Racket pass@1, and fine-tuning techniques perform best for this model.Both fine-tuning-based techniques significantly outperform all in-context learning approaches for the 1B model (ORs 1.45 to 14.01).Source: D. Results Discussion · p. 8
• At 7B and 13B, results are mixed by language; for DeepSeek Coder 33B, the best in-context approach leads the best fine-tuning approach by +12.5% on R and +8.3% on Racket.The reported ORs for these 33B differences are 2.79 on R and 2.06 on Racket.Source: D. Results Discussion · p. 8
• DeepSeek Coder 33B's fine-tuning lowers performance against baseline on both R and Racket, which the authors suggest may reflect insufficient data to update its weights effectively.The reported ORs are 1.64 for R and 1.42 for Racket.Source: D. Results Discussion · p. 8
• GitHub Copilot also benefits from prompting: few-shot adds +8.4% pass@1 on R, while translation examples add +2.8% on Racket.Fine-tuning could not be compared for Copilot.Source: D. Results Discussion · p. 8
• The code-translation pre-training objective adds no relevant boost; translation-rule prompts perform worst, while few-shot and translation-example prompts are the stronger in-context options.Fine-tuning alone works better than pre-training plus fine-tuning for 3 of 5 models on R and 4 of 5 on Racket.Source: D. Results Discussion · pp. 8–9
• No specific technique is best across all model, size, and language combinations.Table III reports pass@1 rates by model, size, and technique, with bold values marking the best technique for each model, size, and language.Source: D. Results Discussion · p. 8, Table III · p. 9
• Translation-example prompting improves the baseline across all tested models and languages except the 1B model, with gains from +0.9% to +6.3%.The authors characterize it as a cheap, safe bet; few-shot learning worsens performance for Code Llama 13B on Racket.Source: D. Results Discussion · p. 9
• In a representative R example, translation-example prompting produces the required list output and reuses example patterns, while both fine-tuned versions fail.The baseline returns a vector instead of the requested list.Source: D. Results Discussion · p. 9

Construct validity

• pass@k may be biased by generation randomness or incomplete benchmark tests; the study mitigates this with k = 1, temperature 0.2, and n ≥20 repetitions.Source: §V · p. 10

Internal validity

• Default model configurations were used without hyperparameter tuning, and prompt choice can substantially affect in-context learning performance.The study partially mitigates prompt risk by testing three in-context learning techniques.Source: §V · p. 10
• Further prompt-tuning may improve results, while limiting fine-tuning to three epochs may cap performance in some cases.Moving from two to three epochs brought no substantial improvements; longer training on small datasets may decrease performance.Source: §V · p. 10

External validity

• Findings from four low-resource languages, one closed-source tool, two open-source model families, and four model sizes may not generalize to other settings.The authors describe the sample as representative because observed performance differed across languages and models.Source: §V · p. 10

Conclusion

• Fine-tuning may be best for small models, while in-context learning always improves performance and is especially useful for larger models.Modern LLMs handle Julia and Lua relatively well, but R and Racket still lag behind.Source: §VI · p. 10

Future work

• Future studies should include more low-resource languages and LLMs and investigate retrieval-augmented generation, chain-of-thought prompting, and agent-based solutions.Source: §VI · p. 10