PaperlayerView exact revision on arXiv

Source-linked AI summary

No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages

Alessandro Giagnorio, Alberto Martin-Lopez, Gabriele Bavota

arXiv:2606.16827v1cs.SE

TL;DR

No-resource programming languages remain largely unsupported by LLMs, limiting AI-assisted programming. This paper benchmarks such languages and evaluates prompting, fine-tuning, and further pre-training, finding that pre-training provides the strongest gains while instruction-following remains a challenge.

  • Problem

    No-resource programming languages lack training data and commercial-tool support, leaving organizations to develop custom AI-assisted programming solutions.

  • Method

    The paper releases three benchmarks for two no-resource languages and evaluates few-shot prompting, retrieval augmentation, fine-tuning, and further pre-training.

  • Results

    No-resource languages achieve below 20% pass@1 in most cases and near-zero performance on hard tasks, while further pre-training provides the strongest improvement.

  • Takeaways & Limitations

    Training-based techniques can make open models outperform commercial LLMs on no-resource languages, with pre-training the most promising approach.

Abstract

from arXiv · show

Large Language Models (LLMs) have significantly advanced the automation of software engineering tasks. One prominent example is code generation, where an LLM produces code in a specified programming language based on a natural language description. Most research in this area has focused on high-resource languages, such as Python or Java, which benefit from abundant training data. A smaller body of work has explored low-resource languages, which are underrepresented in training corpora. In contrast, no-resource languages for which LLMs have seen virtually no training data remain largely unstudied. These languages often emerge in industry, where organizations develop proprietary or domain-specific languages unsupported by commercial tools like GitHub Copilot. This results in the need for companies to deploy their own in-house code recommenders. To investigate possible solutions in this context, we build and release three code generation benchmarks for no-resource languages, based on two recently proposed programming languages for which very little training data is available. Using these benchmarks, we experiment several solutions to teach LLMs about no-resource languages, including prompt-based techniques as well as pre-training and fine-tuning exploiting the little data available. While further pre-training gives the largest performance gains for no-resource languages, applying it directly to instruction-tuned models harms their ability to follow instructions. To address this, we start from a base model, further pre-training it on the target language, and then inject instruction-following capabilities via weight diff transfer from an instruction model. Such an approach significantly improves code generation capabilities in no-resource settings, allowing companies to cheaply deploy a specialized instruct model without dealing with the computational cost of instruction fine-tuning.

I. INTRODUCTION · II. STUDY DESIGN

The paper studies code generation for no-resource programming languages, which lack pre-training data and are unsupported by commercial tools. It introduces benchmarks and evaluates language-popularity effects and specialization techniques across multiple languages and LLMs.

  • I. INTRODUCTION: No-resource languages fall outside LLM pre-training distributions, leaving tools such as Copilot and ChatGPT unable to support them.The study focuses on general-purpose languages that share syntactic and semantic characteristics with mainstream languages but lack training data.
  • I. INTRODUCTION: Commercial and open LLMs obtain very low pass@1 on the no-resource-language benchmarks.Pass@1 is the percentage of tasks for which the LLM produces a test-passing solution in one attempt.
  • I. INTRODUCTION: The study compares no-resource performance with results for high-resource languages and previously studied low-resource languages to establish reasonable performance bounds.The comparison includes Python, Java, Julia, Lua, R, Racket, and Haskell; approximately 50% pass@1 on low-resource languages is treated as an indicative upper bound for no-resource specialization.
  • I. INTRODUCTION: Further pre-training is the best-performing specialization technique, with models approaching ∼15% pass@1 on no-resource languages.The evaluated techniques include few-shot prompting, Retrieval-Augmented Generation, further pre-training, and fine-tuning.
  • I. INTRODUCTION: The paper releases three code-generation benchmarks translated into two no-resource languages, Gleam and MoonBit.The benchmarks are HumanEval, MBPP, and McEval-Hard.
  • II. STUDY DESIGN: The study evaluates specialization techniques across nine high-, low-, and no-resource languages and six commercial and open LLMs.This experimental context supports the study’s investigation of techniques for supporting no-resource languages.
  • II. STUDY DESIGN: RQ1 examines how programming-language popularity and available repository training data affect LLM code-generation performance.It also asks what upper bounds are reasonable for no-resource languages and how well modern LLMs handle languages previously considered low-resource.
  • II. STUDY DESIGN: RQ2 tests whether few-shot prompting, RAG, further pre-training, and fine-tuning improve LLM performance on no-resource languages.Few-shot prompting supplies concrete code-generation examples before a new task, while RAG injects additional information into the prompt.

A. Context Selection · 1) Languages: · 2) Benchmarks:

The study focuses on Gleam and MoonBit as representative no-resource languages and evaluates code generation using three function-level benchmarks, including the newly proposed McEval-Hard. HumanEval and MBPP are translated to the target languages, while translation quality is additionally checked through expert review for Gleam.

  • A. Context Selection: The context comprises selected languages, code-generation benchmarks, LLMs, and datasets for pre-training and fine-tuning.
  • 1) Languages:: The study focuses on Gleam and MoonBit, recently introduced languages designed for multi-thread scalability and cloud or edge computing, respectively.Gleam v1 was announced on 4 March 2024, while the MoonBit compiler became available relatively recently.
  • 1) Languages:: Gleam and MoonBit were selected because they were released after the evaluated LLMs’ cutoff, are sufficiently documented, and have community support.These criteria support high-quality benchmark creation while reducing the likelihood that the models saw substantial training data about the languages.
  • 1) Languages:: Gleam and MoonBit have at least one order of magnitude fewer public GitHub repositories than the other languages considered, supporting their no-resource classification.Repository counts were collected on 2 July 2025; Java and Python each have >10M repositories, while low-resource languages range from approximately 22k to 981k.
  • 2) Benchmarks:: The evaluation uses HumanEval, MBPP, and the newly proposed McEval-Hard, all testing function-level code generation from descriptions and signatures.Function-level benchmarks were chosen because the study tests whether LLMs can handle focused implementation tasks and enables fairer comparisons across resource levels.
  • 2) Benchmarks:: HumanEval and MBPP originate from MultiPL-E, whose language-translated tasks include natural-language function descriptions and tests for implementation correctness.For Gleam and MoonBit, 6 of 161 HumanEval tasks and 27 of 399 MBPP tasks were excluded because they were too Python-specific to translate.
  • 2) Benchmarks:: McEval-Hard addresses limitations of McEval, whose language-specific tasks differ across 40 languages and include only approximately 50 tasks per language, with fewer than 10 hard tasks.McEval-Hard was built from McEval to support the study’s no-resource-language evaluation.
  • 2) Benchmarks:: A random sample of 50 Gleam benchmark instances was double-checked by the language creators, who found no major translation issues.Their feedback mainly recommended stylistic improvements that did not alter code behavior.

3) LLMs:

The study evaluates six diverse LLMs, using instruction-following models for zero-shot and in-context learning, non-instruct models for pre-training, and open instruct models for fine-tuning. These choices reflect model availability and the risk of catastrophic forgetting after further pre-training.

  • Model selection: Six LLMs are selected across open and commercial models, varying in size, instruction-following, reasoning capabilities, and training-data cutoff dates.The models include o3-mini and GPT-4o; cutoff information is unavailable for some models.
  • RQ1 and RQ2: In RQ1, four instruction-following LLMs are evaluated zero-shot across all nine programming languages.The models are Qwen 2.5 Coder 32B Instruct, Qwen 3 32B Instruct, o3-mini, and GPT-4o.
  • In-context learning: Few-shot and RAG experiments use the four instruction-following models: Qwen 2.5 Coder 32B Instruct, Qwen 3 32B Instruct, o3-mini, and GPT-4o.These are the in-context learning techniques tested in the study.
  • Pre-training: Pre-training uses Qwen 2.5 Coder 32B Base and Qwen 3 8B Base because further pre-training on instruct models can cause catastrophic forgetting of instruction capabilities.Qwen 3 32B Base was unavailable, motivating the use of its 8B Base model.
  • Fine-tuning: Fine-tuning uses Qwen 2.5 Coder 32B Instruct and Qwen 3 32B Instruct because fine-tuning cannot be performed on closed models.Both selected models are open instruct models.

4) Datasets Used for Further Pre-Training and Fine-Tuning on Gleam and MoonBit: … 2) RQ2:

The study builds recent Gleam and MoonBit datasets for continued pre-training and fine-tuning, then evaluates four strategies for improving code generation in no-resource languages. RQ1 uses repeated zero-shot benchmark evaluations, while RQ2 compares few-shot, retrieval-augmented generation, pre-training, and fine-tuning.

  • 4) Datasets Used for Further Pre-Training and Fine-Tuning on Gleam and MoonBit:: Gleam repositories created after 5 March 2024 and MoonBit files created in 2025 were collected to capture up-to-date language grammar and code.The languages are recent, and MoonBit lacked an official first-release date.
  • B. Data Collection: Functions for fine-tuning were extracted with tree-sitter and filtered by documentation, character, description, body, TODO, and duplication criteria.The parsers were developed by the Gleam and MoonBit developers and were expected to be reliable.
  • 1) RQ1:: In RQ1, models generated code zero-shot from prompts containing each task’s natural-language description and function signature.Temperature was set to 0.2 for all models except o3-mini, which used its default because it does not allow temperature settings.
  • 1) RQ1:: Each model ran 10 times per benchmark and language, producing 66,240 generations per model and 264,960 generations across four models, all evaluated with test suites.Repeated runs accounted for the stochastic nature of LLMs.
  • 2) RQ2:: RQ2 compares few-shot retrieval, documentation-based RAG, further pre-training, and fine-tuning, using the same inference parameters as RQ1.The four strategies target improved performance on no-resource languages.
  • 2) RQ2:: Few-shot retrieval embeds the fine-tuning dataset with text-embedding-3-large, indexes it with FAISS, and prepends the top-five similar examples to each generation prompt.Benchmark prompts are also embedded before retrieval.
  • 2) RQ2:: RAG retrieves and summarizes documentation snippets through LLM-generated task plans and queries, using five relevant portions per query to augment generation.The implementation used gpt-4o-mini-2024-07-18 to plan, generate queries, and summarize retrieved documentation.
  • 2) RQ2:: Pre-training and fine-tuning use LoRA, with pre-training on 2,048-token sequences and fine-tuning on 4,096-token sequences for five epochs.Pre-training uses causal language modeling; fine-tuning uses learning rate 5 × 10−5, AdamW, and a cosine scheduler.

C. Data Analysis

The analysis evaluates code generation primarily with pass@1 and passed%, comparing four LLMs across nine languages and testing methods for improving no-resource performance. Statistical comparisons use McNemar’s test with Benjamini–Hochberg correction and Odds Ratio effect sizes.

  • Evaluation metrics: pass@1 is the reference metric, scoring 1 when generated code passes all unit tests and 0 otherwise.The analysis also reports passed% to capture partial correctness when some, but not all, tests are satisfied.
  • Evaluation metrics: The study reports passed% alongside pass@1 to provide a more fine-grained view of model performance.passed% measures the percentage of unit tests passed for each coding task.
  • Research questions: RQ1 compares four LLMs across nine languages, while RQ2 evaluates few-shot, RAG, pre-training, and fine-tuning against zero-shot performance in no-resource languages.These analyses examine language popularity and methods for boosting performance in no-resource settings.
  • Statistical analysis: McNemar’s test compares paired dichotomous outcomes using 1,540 pass@1 values for HumanEval-Gleam.The distributions contain 154 coding tasks × 10 repetitions.
  • Statistical analysis: Benjamini–Hochberg adjusts p-values for multiple comparisons, while Odds Ratio quantifies differences between methodologies.The comparisons include zero-shot versus few-shot and RAG for the same model.

D. Replication Package · III. RESULTS DISCUSSION

The replication package provides the benchmarks, generations, prompts, scripts, and additional results needed to reproduce the experiments. Results show a sharp performance gap by language-resource level and indicate that training-based methods, especially pre-training, are most effective for no-resource code generation.

  • D. Replication Package: The replication package includes JSONL benchmarks, model generations, experimental prompts, replication scripts, and additional results.The scripts cover the workflow from data collection through implementation of all experimented techniques.
  • III. RESULTS DISCUSSION: High-resource languages achieve 59%–97% pass@1, averaging 79% across models and benchmarks.The range spans Java on McEval-Hard with GPT-4o at 59% and Python on HumanEval with o3-mini at 97%.
  • III. RESULTS DISCUSSION: Low-resource languages average 62% pass@1, exceeding 50% in 49 of 60 evaluated cases despite popularity not fully determining performance.Scores range from 27% for Haskell on McEval-Hard with Qwen 3 to 87% for Haskell on HumanEval with o3-mini; Lua consistently outperforms R despite fewer GitHub repositories.
  • III. RESULTS DISCUSSION: No-resource languages remain at 0%–20% pass@1, averaging 9%, with apparent successes on simple benchmarks attributable to trivial tasks and reusable code patterns.For MoonBit, the prompt supplies the function signature, while the body `return l * l * l;` is common across high-resource languages such as Java.
  • III. RESULTS DISCUSSION: No-resource failures are predominantly syntactic: roughly two-thirds for GPT-4o and up to 90% for o3-mini on Gleam, unlike typically below 10% elsewhere.Java is an exception among high- and low-resource languages, with syntactic failures around 30%, plausibly due to syntactic verbosity.
  • III. RESULTS DISCUSSION: Few-shot prompting outperforms RAG in 7 of 12 Gleam cases and 8 of 12 MoonBit cases, with larger gains on MoonBit and simpler benchmarks.Both in-context methods provide substantially smaller gains on McEval-Hard, suggesting examples or documentation alone are insufficient for challenging tasks.
  • III. RESULTS DISCUSSION: Fine-tuning outperforms zero-shot and in-context learning on the same models, while fine-tuned open models can surpass commercial models.On Gleam, fine-tuned Qwen 3 32B Instruct reaches 23.57% on HumanEval, 37.32% on MBPP, and 3.88% on McEval-Hard, versus GPT-4o 5-shot at 15.45%, 30.37%, and 1.23%.
  • III. RESULTS DISCUSSION: Further pre-training is more effective than fine-tuning because it exploits entire code files and language documentation, while training-based methods reduce syntactic failures below 20%.For MoonBit, Qwen 2.5 Coder 32B Base reaches 41.62 versus 34.74 on HumanEval, 44.76 versus 37.38 on MBPP, and 25.86 versus 10.93 on McEval-Hard; these differences are statistically significant.

IV. INSTRUCTION TRANSFERRING

Instruction transferring addresses the loss of instruction-following caused by further pre-training on scarce no-resource-language data without requiring costly instruction fine-tuning. It substantially improves code generation, enabling smaller models to outperform larger baselines.

  • Motivation: Further pre-training is the strongest performance booster, but it removes instruction-following capabilities needed for coding assistants.Prompt variability makes instruction following crucial, while instruction-tuning datasets are typically unavailable and the process is extremely costly.
  • Method: Instruction transferring computes a weight difference between same-architecture instruct and base models, then applies it to a target-language further-pre-trained base model.The experiments use Qwen 2.5 Coder 32B and Qwen 3 8B instruct/base pairs; Qwen 3 8B additionally transfers reasoning capabilities.
  • Results: 33%: pass@1 increases by up to 33% over further pre-training, with a 12% average increase across benchmarks, models, and languages.All improvements are statistically significant, with adjusted p-value < 0.05 and odds ratios ranging from 1.21 to 10.95.
  • Results: Instruction transferring is the best approach across all evaluated LLMs and techniques, and Qwen 3 8B consistently outperforms Qwen 3 32B Instruct.Compared with fine-tuned Qwen 3 32B Instruct, Qwen 3 8B gains up to +28% in pass@1 and averages +12% across benchmarks and languages.
  • Error analysis: Instruction transferring reduces syntactic errors for Qwen 2.5 32B by 24.1% on Gleam and 73.5% on MoonBit, while 8B models trade more syntactic for fewer semantic errors.The divergent error shifts are attributed to differences in model capacity, yet the smaller models still improve overall performance.

V. VALIDITY DISCUSSION · A. Experimental Procedure and Evaluated Techniques

The evaluation used default model configurations and selectively designed in-context procedures, while acknowledging limits from computational constraints, context coverage, and constrained decoding. Manual documentation selection produced strong in-context results, but instruction transfer remained superior, and syntactic errors remained the main failure mode.

  • V. VALIDITY DISCUSSION: No hyperparameter tuning was performed because it would have required substantial computational resources; default author-recommended configurations were used.Training lasted five epochs based on loss monitoring, with first- and third-epoch checks producing worse results than the fifth epoch.
  • A. Experimental Procedure and Evaluated Techniques: Few-shot prompting and RAG dynamically selected relevant code examples and documentation, but retrieved context might omit language features needed for a task.The prompts and scripts used for these in-context techniques were publicly released.
  • A. Experimental Procedure and Evaluated Techniques: The manual documentation context was built by identifying task-relevant files, manually verifying them, and asking GPT-4.1 to summarize required concepts with explanations and code examples.This reduced the full documentation to a representative subset of files relevant to at least one benchmark task.
  • A. Experimental Procedure and Evaluated Techniques: The manual-context strategy was evaluated on Qwen 2.5 Coder 32B Instruct, the only model reported across all experimental settings, using 10 runs to capture stochastic output variation.The methodology was described as mainly useful for identifying an approximate upper bound on performance.
  • A. Experimental Procedure and Evaluated Techniques: 15.58% on HumanEval, 25.07% on MBPP, and 1.76% on McEval-Hard were obtained for Gleam, versus 32.47%, 39.44%, and 6.61% for MoonBit.The first two benchmark results exceeded other in-context techniques and nearly matched fine-tuning, but remained below instruction transferring, especially on McEval-Hard.
  • V. VALIDITY DISCUSSION: Syntactical errors were the main reason LLMs failed on no-resource languages, motivating constrained decoding as a possible mitigation.Although constrained decoding can improve open-weight models without additional training, it does not address language APIs, semantic correctness, or language-specific coding conventions.

B. Experimental Assumptions

The study assumes the evaluated LLMs had not encountered Gleam or MoonBit during training, but this absence cannot be verified from publicly available information. Their omission from Qwen documentation indicates only that the languages are not advertised as supported.

  • Assumption: The study assumes that evaluated LLMs were not exposed to Gleam or MoonBit code during training.The authors identify this as a key experimental assumption.
  • Assumption: The open models’ training corpora are not publicly characterized well enough to verify that Gleam or MoonBit code was absent.Therefore, the assumption cannot be confirmed for the open models evaluated.
  • Model documentation: Neither Gleam nor MoonBit appears in the 92 languages listed for Qwen 2.5 Coder or the 358 languages listed for Qwen 3.The authors interpret these omissions only as evidence that the languages are not advertised as supported, not as evidence that they were absent from training data.

C. Data Quality

The study evaluates the quality of automatically collected fine-tuning pairs and translated benchmark tasks. Human ratings were generally high, while manual inspection identified several categories of translation problems.

  • Fine-tuning data quality: Fine-tuning effectiveness depends critically on the quality of automatically collected ⟨doc, function⟩ pairs.Here, doc is the natural-language implementation description and function is the target code generation.
  • Human evaluation: Krippendorff α was 0.69 for content adequacy, 0.48 for conciseness, and 0.65 for fluency & understandability.These correspond to substantial, moderate, and substantial agreement, respectively.
  • Human evaluation: Median quality ratings were always 5 overall and for each evaluator, while average values never fell below 4.3.The distributions covered content adequacy, conciseness, and fluency & understandability.
  • Translation quality: A statistically significant 95%±5% sample of translated coding tasks was manually inspected across the nine languages.The population comprised 3,061 translated coding tasks.
  • Translation quality: Of 14 flagged prompts, five contained inherited benchmark issues, four had textual deviations, three lacked type information, and two had type-related errors.Examples included incorrect doctest return values, missing sentences, missing type hints, and an incorrect expected Boolean value in MoonBit.

D. Generalizability · VI. RELATED WORK

The study’s findings may not generalize beyond its nine languages, six LLMs, three benchmarks, and selected tasks, but its focus on no-resource settings remains relevant beyond Gleam and MoonBit. Related work spans multilingual benchmarks, synthetic or aligned training data, in-context methods, and specialized DSL assistance, while some approaches cannot support the target languages.

  • D. Generalizability: The findings may not generalize beyond the nine languages, six LLMs, three benchmarks, and selected tasks evaluated.The selected tasks may also fail to reflect real usage scenarios, including domain-specific uses such as high-performance computing, game development, or IoT.
  • D. Generalizability: The study targets the gap between no-resource and higher-resource languages rather than Gleam and MoonBit specifically.Its implications remain relevant even if future LLMs are trained on Gleam and MoonBit.
  • VI. RELATED WORK: Most existing code-generation techniques and benchmarks target popular languages such as Python and Java.Related work therefore focuses on benchmarks and methods for low-resource programming languages.
  • VI. RELATED WORK: MULTIPL-E translated HUMANEVAL and MBPP to 24 languages, and this study used those translations to assess LLMs on low-resource languages.The authors also translated the same benchmarks further, although the supplied passage truncates that description.
  • VI. RELATED WORK: The authors revised MCEVAL, a benchmark with 2k humanwritten code-generation tasks across 40 programming languages, into a more challenging version for low- and no-resource languages.MCEVAL provides about 50 code instances per language and was motivated by concerns about training-data contamination in popular benchmarks.
  • VI. RELATED WORK: Prior low-resource approaches include language-specific intermediate DSLs, multilingual fine-tuning, synthetic translation, shared intermediate representations, and in-context learning.Examples include JSON-based task DSLs, training on similar programming languages, MULTIPL-T, LLVM alignment, few-shot prompting, and retrieval-augmented grammar and example selection.
  • VI. RELATED WORK: MULTIPL-T and LLVM-based alignment were excluded as baselines because they cannot be applied to the target no-resource languages.MULTIPL-T depends on automated translation that LLMs cannot support for high- to no-resource languages, while LLVM lacks support for Gleam and MoonBit.
  • VI. RELATED WORK: ModelMate fine-tunes pre-trained models for textual DSL assistance by converting semantically compatible modeling-language models into the target DSL.Its approach increases training data through a model-to-text transformation, but the supplied passage truncates its underlying assumption.

VII. CONCLUSION AND FUTURE WORK

The paper evaluates six state-of-the-art LLMs across high-, low-, and no-resource programming languages using newly created benchmarks, finding severe no-resource performance gaps. Pre-training base models on limited target-language data is most effective, while future work will broaden benchmarks to additional real-world use cases.

  • VII. CONCLUSION AND FUTURE WORK: The study evaluates six state-of-the-art LLMs across high-, low-, and no-resource programming languages using benchmarks for Gleam, MoonBit, and McEval-Hard.Creating these benchmarks required major effort.
  • VII. CONCLUSION AND FUTURE WORK: Future work will expand the benchmarks to more diverse real-world no-resource use cases, including debugging, refactoring, and code generation for entire change requests.The proposed extensions target tasks beyond the current benchmark scope.
Loading 2606.16827v1…