PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo

Kilian Kier, Alessandro Giagnorio, Omar AbedelKader, Oleksandr Zaitsev, Robert Peharz, Romain Robbes, Gabriele Bavota, Stéphane Ducasse

arXiv:2607.04939v1cs.SE

TL;DR

Low-resource languages such as Pharo lack strong LLM-based in-IDE completion because scarce data and language-specific integration challenges limit existing approaches. This paper develops a Pharo-specialization pipeline and benchmarks, showing that specialized models outperform their base checkpoints and substantially larger code LLMs while remaining small enough for real-time IDE use.

  • Problem

    Pharo lacks LLM-based in-IDE completion that addresses its extreme data scarcity, Tonel-aware data curation, and real-time integration constraints.

  • Method

    The paper combines Pharo-specific data curation, continued pre-training, fine-tuning, and benchmarks for syntactic and realistic repository-level completion.

  • Results

    Pharo-specialized models outperform their base counterparts and surpass substantially larger code LLMs on Pharo completion while remaining small enough for real-time in-IDE usage.

  • Takeaways & Limitations

    Specialization can compensate for model scale in low-resource programming languages while yielding models compact enough for interactive IDE use.

Abstract

from arXiv · show

Large Language Models (LLMs) unlocked new possibilities in automated code writing, becoming the backbone of most code completion tools. While LLMs excel in mainstream languages, they often lack support for the so-called low-resource languages where training data is scarce. As a result, these languages lag behind in the quality of code completion tooling available to their communities. A concrete example is Pharo, a Smalltalk-inspired language whose IDE currently offers only single-token completion. In this work, we report on our experience bringing LLM-based code completion to Pharo. First, we describe an end-to-end pipeline that combines Pharo-specific data curation, continued pre-training and fine-tuning of open code LLMs. Second, we introduce a set of Pharo code completion benchmarks designed to evaluate whether models (i) learn Pharo's syntax and (ii) accurately complete masked Pharo code from real-world GitHub repositories. Third, we show empirically that Pharo-specialized models substantially outperform their original base checkpoints and also exceed the accuracy of substantially larger code LLMs on Pharo completion. Overall, our case study demonstrates the feasibility of bringing strong LLM-based code completion to low-resource programming languages, with models small enough to provide ``real-time'' in-IDE support.

I. INTRODUCTION

The paper addresses Pharo’s limited, single-token IDE completion by developing an end-to-end specialization pipeline for small open code LLMs. It introduces Pharo-specific benchmarks and shows that specialized models outperform base and substantially larger models while supporting real-time in-IDE use.

  • Motivation: Pharo’s IDE provides only single-token completion because effective LLM-based tooling is lacking, limiting multi-token, context-aware assistance.Pharo is a very low-resource language, and its Tonel format and Smalltalk-derived syntax create additional training challenges.
  • Task requirements: In-IDE completion requires low-latency, incremental, syntactically valid predictions conditioned on partially written programs, distinguishing it from offline code generation.The paper notes that techniques designed for code generation, including RAG-based solutions, are not directly transferable to this setting.
  • Approach: The authors present an end-to-end pipeline combining Pharo-specific data curation, continued pre-training, and fine-tuning to adapt open code LLMs for Pharo statement completion.The pipeline includes tools for collecting data from public Pharo projects and addresses repository peculiarities such as Tonel.
  • Evaluation: The work introduces benchmarks that assess both Pharo syntactic competence and usefulness in realistic coding scenarios.The syntax-focused benchmarks translate HumanEval+ to Pharo and collect exercises from Exercism.
  • Findings: Pharo-specialized models outperform their base counterparts and surpass substantially larger code LLMs on Pharo completion while remaining suitable for real-time in-IDE usage.The targeted models are small open checkpoints that are subsequently quantized for fast inference on developer machines.

II. BACKGROUND AND RELATED WORKS … 2) Pharo’s Syntax:

Pharo is an open-source, dynamic, reflective Smalltalk implementation built around a live programming environment, dedicated VM, and image-based development model. Its compact, message-oriented syntax uses methods for control flow, keyword-based calls, English-like statements, streamlined iterators, and typically small methods.

  • A. Pharo and its completion engine: Pharo is an open-source Smalltalk implementation combining a dynamic, reflective language with a live programming environment and tools.The environment emphasizes simplicity, immediacy, and direct interaction with live objects.
  • A. Pharo and its completion engine: Pharo’s live environment lets developers modify and inspect running systems without a traditional compile-run cycle.This capability follows from its emphasis on direct interaction with live objects.
  • 1) The Pharo environment:: Pharo runs on a dedicated virtual machine, similar to Java’s execution model.The VM is one component of Pharo’s execution environment.
  • 1) The Pharo environment:: Developers define classes and methods in the IDE rather than explicitly manipulating files.The IDE transparently manages publication of code to git or databases such as ENVY.
  • 1) The Pharo environment:: Pharo provides an image mechanism that saves and loads frozen memory containing code entities.The image functions as a docker-like snapshot mechanism.
  • 2) Pharo’s Syntax:: Pharo expresses control-flow constructs such as if, for, and while as simple methods rather than syntax elements.This makes language constructs message-based.
  • 2) Pharo’s Syntax:: Pharo uses keyword-based method signatures that interleave arguments with method-name parts, forming readable keyword messages.An example is at: aSymbol ifAbsentPut: aBlock.
  • 2) Pharo’s Syntax:: Pharo’s syntax is compact and English-like, uses periods as statement separators, offers streamlined iterators, and has small methods.Its core syntax fits on half of a postcard, and about 50% of methods have three or fewer lines of code.

3) Pharo and code completion:

Pharo’s completion evolved from noisy prefix-based suggestions toward semantic, ranked, and package-aware approaches that manage its large search space. Its current engine, COMPLISHON, improves responsiveness through modularity and lazy computation but still recommends only one token at a time.

  • History: Early Pharo completion matched typed prefixes and used lightweight parsing and heuristics, but often produced noisy suggestions.The heuristics attempted to infer possible variable types from previously sent messages.
  • History: Semantic completion parses the source, identifies the AST node at the cursor, and filters candidates appropriate to that node type, improving accuracy despite large candidate sets.
  • Proposed ranking improvements: Ranking research used method-usage heuristics, Pharo-trained n-gram models, and popularity metrics, but these approaches were not integrated into standard Pharo because of deployment or performance concerns.
  • Proposed ranking improvements: Package-aware completion narrowed Pharo’s global class namespace through the current package, related repository packages, and global namespace.
  • COMPLISHON: COMPLISHON, introduced in Pharo 9, uses a modular, lazy architecture with prefix filtering, deduplication, and caching to improve extensibility and responsiveness, but recommends only one token at a time.COMPLISHON is Pharo’s current completion engine.

B. LLMs and low-resource languages · A. Selected Large Language Models · B. Training Datasets

The paper situates Pharo completion within research on low-resource programming languages, then selects compact fill-in-the-middle models and describes a curated dataset pipeline for Pharo specialization.

  • B. LLMs and low-resource languages: Low-resource programming languages face persistent code-generation performance gaps linked to scarce training data, imbalance, and limited cross-lingual transfer.Prior work addresses these gaps through language-specific adaptation, multilingual rebalancing, retrieval augmentation, compiler representations, and broader evaluation suites.
  • B. LLMs and low-resource languages: Domain-specialized models can outperform substantially larger general-purpose models when trained on focused corpora.MonoCoder and MPIrigen demonstrate this pattern for High Performance Computing code generation.
  • B. LLMs and low-resource languages: This work targets Pharo code completion under strict latency and incremental syntactic-validity constraints, rather than offline code generation.Pharo is characterized by a specific Tonel repository structure and IDE constraints.
  • A. Selected Large Language Models: The study selects Qwen2.5 Coder Base models at 0.5B, 1.5B, 3B, and 7B parameters, plus the 4B Mellum-base model.Both model families support fill-in-the-middle generation, enabling use of code before and after the cursor.
  • A. Selected Large Language Models: The selected model families use different fill-in-the-middle strategies, whose influence on low-resource-language specialization is examined.The difference concerns the order in which each model processes the prefix and suffix.
  • A. Selected Large Language Models: Models range from 0.5B to 7B parameters to facilitate deployment on consumer-grade machines.The quantized 7B model uses 4.3GB of memory at 4-bit precision, and the paper reports that quantization does not negatively affect completion performance.
  • A. Selected Large Language Models: The models are fine-tuned on a curated Pharo dataset and compared with base versions, Qwen3 Coder 480B A35B Instruct, and Claude Sonnet 4.5.The larger comparison models represent an open-weight code model and a commercial model behind Claude Code.
  • B. Training Datasets: The training-dataset section describes step by step how the Pharo data was collected, filtered, and prepared for teaching the selected LLMs.The dataset pipeline is explicitly organized around collection, filtering, and preparation.

1) Data Collection and Filtering: · 2) Training Instances Construction:

The pipeline filters Pharo repositories by licensing, version compatibility, and repository age before constructing separate pre-training and fine-tuning datasets. It uses Pharo-aware lexical and syntactic extraction, then realistic AST-guided masking with rehearsal training for code completion.

  • 1) Data Collection and Filtering:: 748 MIT-licensed GitHub repositories tagged “Pharo” were initially selected, while repositories without explicit license files were excluded.MIT was chosen because it is permissive and suitable for training-data inclusion.
  • 1) Data Collection and Filtering:: Repositories were filtered for compatibility with Pharo 10 or newer by importing each into the five newest environments, Pharo 10 through Pharo 14.The filter targets current Pharo syntax and APIs.
  • 1) Data Collection and Filtering:: Repositories created before June 1, 2024, were used for training, while later repositories supported repository-level evaluation.The cutoff was intended to reduce evaluation contamination from model pre-training data.
  • 1) Data Collection and Filtering:: The contamination-control procedure also removed training methods with overlapping 8grams against the canonical syntactic-evaluation material.The June 1 cutoff could not guarantee non-contamination for all baselines, Qwen3 Coder, and Claude 4.5 Sonnet.
  • 2) Training Instances Construction:: The filtered repositories produced separate pre-training and fine-tuning datasets targeting Pharo knowledge and realistic completion scenarios, respectively.Pre-training teaches syntax, structure, and main features; fine-tuning aligns models with code completion use.
  • 2) Training Instances Construction:: A Pygments lexer tokenized Pharo code, while an integrated tree-sitter grammar parsed Tonel methods and code into abstract syntax trees.Both tools were released in the replication package for future Pharo code-analysis research.
  • 2) Training Instances Construction:: Random-AST fine-tuning masks from a randomly selected token or fragment through the enclosing statement AST node, combining realistic cursor positions with syntactic structure.The strategy avoids restricting completion targets to pre-selected AST boundaries.
  • 2) Training Instances Construction:: 20% rehearsal mixing preserved AST-aware pre-training knowledge, yielding a final fine-tuning dataset of 324,725 instances.The rehearsal sample was drawn randomly from the AST-aware dataset.

C. Syntax Evaluation Benchmarks … 4) Task verification:

The authors construct two Pharo method-level syntax benchmarks from HumanEval+ and Exercism, each supporting AST-Aware and Random-AST completion tasks. They verify benchmark and completion correctness through manual checks, compilation and automated test execution in Pharo 13.

  • C. Syntax Evaluation Benchmarks: The study creates two method-level Pharo benchmarks to evaluate syntax understanding: a translated HumanEval+ suite and Pharo exercises from Exercism.Each benchmark is divided into AST-Aware completion and Random-AST completion, with a tool evaluating result correctness.
  • 1) HumanEval+:: HumanEval+ requires synthesizing a function body from a signature and docstring and provides roughly 80× additional test cases per task.The benchmark includes 164 tasks, each with a reference solution.
  • 1) HumanEval+:: The translated HumanEval+ benchmark contains 164 Pharo tasks manually checked for errors in docstrings, solutions and test cases.The resulting code was checked to compile in Pharo 13 and pass all translated tests.
  • 2) Exercism:: The Exercism benchmark starts from 50 Pharo-track exercises, excludes trivial ones and retains 47 processed exercises.Each exercise was made self-contained and associated with a single Pharo method representing its reference solution.
  • 3) Task creation:: The authors process both benchmarks into AST-aware and Random-AST code completion tasks using the masking strategies and settings from Section III-B2.AST-aware masking hides valid AST nodes, whereas Random-AST masking hides random code spans.
  • 3) Task creation:: AST-aware masking evaluates completion of valid AST nodes, while Random-AST masking evaluates completion when random code spans are masked.The two masking strategies are applied to both the HumanEval+ and Exercism instances.
  • 4) Task verification:: A publicly available tool automatically executes auto-completed code in a Pharo 13 image and reports pass/fail results, error messages and failure stack traces.The tool is extensible to additional benchmarks for future Pharo code completion and generation research.

D. Repository-level benchmark … 2) Baselines:

The repository-level benchmark models realistic developer changes by masking newly introduced Pharo AST nodes and evaluates completion under multiple repository-context policies. Specialized models are trained through continued pre-training and fine-tuning, then compared with their base checkpoints and substantially larger LLMs.

  • 1) Instance construction:: The benchmark mines commits from 22 Pharo test repositories and masks AST nodes newly introduced in added lines of impacted methods.For each commit, the procedure extracts newly introduced AST nodes, selects one, and masks it to construct a completion task.
  • 2) Context Modeling:: Repository context is varied because insufficient context can limit project information, whereas excessive context can slow generation and obscure relevant code.The benchmark explicitly studies how the amount and relevance of prepended repository context affect completion.
  • 2) Context Modeling:: The four context policies are no context, class signature, package signature, and impacted methods.These policies range from only the masked target method to broader signatures or full bodies of other modified methods.
  • 2) Context Modeling:: 8,000 tokens cap total context, yielding 2,185 repository-level completion tasks evaluated under each context policy.Tasks exceeding the threshold for at least one context were excluded, reflecting reduced effectiveness beyond 8k tokens for Mellum.
  • 1) Training:: Training uses continued pre-training to teach Pharo syntax and APIs, followed by fine-tuning for realistic code completion scenarios.LoRA is used in both stages with alpha 32, r 16, and dropout 0.05.
  • 2) Baselines:: The evaluation covers Pharo syntax comprehension and repository-level completion.These two dimensions correspond to learning the language and completing masked code in repository contexts.
  • 2) Baselines:: Models are compared with their base versions and with Qwen3 Coder 480B A35B Instruct and Claude Sonnet 4.5.The base versions are the same LLMs before the additional pre-training and fine-tuning.

3) Syntactic evaluation and metrics: … 2) Improvement over the base models:

The study evaluates Pharo code completion using test-suite pass@1 for method-level tasks and semantic-similarity metrics for repository-level tasks. It finds that continued pre-training and supervised fine-tuning improve Pharo performance, with fine-tuning especially addressing realistic masked-completion settings.

  • 3) Syntactic evaluation and metrics:: Method-level performance is measured with pass@1, the probability of producing a correct solution on a single attempt.The reported pass@1 aggregates results across the 20 repetitions.
  • 4) Repository-level evaluation and metrics:: Repository-level completions are evaluated without test suites using ChrF and CrystalBLEU to measure semantic similarity with developer-written references.ChrF measures character n-gram overlap and computes an F-score from precision and recall.
  • IV. RESULTS: Results are organized around method-level completion, repository-level contextual influence, and inference time as a first-class code-completion requirement.The method-level evaluation uses HumanEval+ and Exercism, while repository-level evaluation starts from real code changes.
  • A. Method-level Completion: Table II compares base, continued-pretraining, and supervised-fine-tuning checkpoints on HumanEval+ and Exercism under AST-aware and Random(r)-AST masking.Statistical significance is indicated for changes relative to each base checkpoint.
  • 1) Reading the results:: 12.25 percentage points: Qwen2.5 Coder 3B - SFT raises HumanEval+ AST-aware pass@1 from 71.48% to 83.73% over its base model.The comparison is statistically significant, and Odds Ratio complements the statistical analysis.
  • 2) Improvement over the base models:: Fine-tuning closes the realistic r-AST performance gap while preserving continued-pretraining gains on AST-aware completion.Continued pre-training improves AST-aware performance but is insufficient for strong r-AST results and often harms that setting.

3) Small specialized LLMs vs large general-purpose models: · 4) Reasons behind the failures: · B. Repository-level evaluation

Specialized Pharo training enables small models to compete with or surpass much larger general-purpose LLMs, while repository-level performance improves most with relevant recent-method context. Base-model failures are primarily syntactic, and the training procedure reduces these errors while avoiding impractical latency from excessive generation and post-processing.

  • 3) Small specialized LLMs vs large general-purpose models:: Specialized Qwen2.5 Coder 3B and 7B models achieve higher pass@1 than much larger models on realistic r-AST benchmarks, despite large models leading on the AST-aware benchmark.This demonstrates that accurate Pharo completion is feasible with small specialized models.
  • 4) Reasons behind the failures:: 65.6% of base-model failed completions are syntax errors, compared with 17.9% unexpected exceptions and 16.5% assertion failures.The distribution confirms that base LLMs mainly struggle with Pharo syntax.
  • 4) Reasons behind the failures:: Two-step training reduces syntax errors by 33% on average across the five evaluated LLMs.A TripleSumToZero example shows Qwen2.5 Coder 7B - SFT consistently reconstructing the correct Pharo fragment “(aCollection at: i)”.
  • B. Repository-level evaluation: Repository-level evaluation compares average ChrF and CrystalBLEU across four benchmarks using no context, class signatures, package signatures, impacted methods, and random methods.The results distinguish base models, SFT models, and larger general-purpose baselines.
  • B. Repository-level evaluation: +15.91% ChrF and +23.03% CrystalBLEU result when Qwen2.5 Coder 7B - SFT receives last modified methods instead of no context.ChrF rises from 60.05% to 75.96%, while CrystalBLEU rises from 35.96% to 58.99%.
  • B. Repository-level evaluation: Base models often generate extra FIM tokens or code, slowing inference and requiring error-prone post-processing that is impractical for low-latency completion.Post-processing narrows the performance gap but does not make the approach practical because of latency.
  • B. Repository-level evaluation: Impacted-method context significantly boosts ChrF and CrystalBLEU, whereas class and package signatures provide limited, often non-significant gains.Across the same models, the reported average improvements are 14.54% for ChrF and 20.21% for CrystalBLEU.
  • B. Repository-level evaluation: +7.52% ChrF and +8.63% CrystalBLEU are the average advantages of specialized 3B and 7B models over Qwen3 Coder 480B A35B Instruct with impacted methods context.The 1.5B SFT model also outperforms that large baseline; Claude 4.5 Sonnet remains best at 83.02% ChrF and 70.52% CrystalBLEU, though benchmark contamination cannot be excluded.

C. Analysis of Models Latency

The study evaluates whether the best 3B and 7B Pharo models are practical for local deployment on consumer hardware. Q4_K_M quantization substantially reduces the 7B model’s memory footprint without major performance loss, while the non-quantized 3B model achieves lower inference latency than the quantized 7B model.

  • Deployment practicality: The best 3B and 7B models were assessed for deployment on consumer hardware.Inference time was measured during the repository-level benchmark with impacted methods as context, across 10 iterations and multiple consumer-grade processors and a gaming GPU.
  • Memory footprint: 70%: Q4_K_M quantization reduces the 7B model’s memory footprint from 14.19 GiB to 4.36 GiB.The quantization method was applied using llama.cpp and selected for reducing memory while preserving performance.
  • Performance preservation: The quantized 7B model incurs no major loss in performance.The passage introduces this conclusion in the context of method-level evaluation, although the supplied excerpt does not include the detailed results.
  • Inference latency: The non-quantized 3B model achieves lower inference latency than the quantized 7B model.Average generation time was computed on Apple M3 Max, Apple M4 Max, and an RX 7800XT GPU.

V. DISCUSSION … B. Limitations

The discussion identifies Pharo completion’s combined data, format, and syntax challenges, while showing that specialization enables compact models for real-time IDE use. It also emphasizes that language-specific parsing, datasets, benchmarks, and context policies require substantial but reusable engineering effort.

  • 1) What makes Pharo completion difficult?:: Pharo completion is difficult because extremely low-resource conditions combine with Tonel’s mixed code-and-metadata format and Smalltalk-specific syntax.Pharo has fewer public repositories than mainstream languages, and its syntax hinders straightforward transfer.
  • 2) On the value of specialization:: Specialized 3B and 7B models outperform much larger general-purpose models on method-level Pharo benchmarks, with the specialized 7B model also competitive at repository level.These results support compact models for in-IDE completion, where latency and deployability are first-order concerns.
  • 3) On the engineering effort:: The specialization pipeline required selecting training data, building Pharo tooling, constructing datasets and benchmarks, and developing language-specific context-selection policies.The engineering effort substantially exceeded the effort required to train the models themselves.
  • 3) On the engineering effort:: Parsing Pharo code was critical for defining training objectives, benchmark tasks, and context selection.The authors identify parsing strategies as potentially applicable to other low-resource languages despite their different idiosyncrasies.
  • 3) On the engineering effort:: Language-specific benchmarks were labor-intensive but critical, and the authors consider their benchmark strategies applicable to other low-resource languages.The same passage also emphasizes the importance of parsing for multiple pipeline stages.
  • 4) Comparison with “low-resource” literature:: This work contributes evidence on improving LLMs for low-resource programming languages, where prior findings include effective pre-training for small models but mixed or harmful effects for larger ones.The passage situates the study alongside transfer-learning work on code generation, translation, and program repair, while noting limited evidence for low-resource code completion.

1) Data contamination: … VI. CONCLUSION & FUTURE WORK

The study addresses contamination and evaluation limitations while concluding that Pharo-specific data curation, continued pre-training, and fine-tuning can make strong LLM-based completion feasible for a severely low-resource language.

  • 1) Data contamination:: A June 1, 2024 cutoff and removal of methods overlapping benchmark solutions reduce, but cannot eliminate, training-test contamination.The authors note that contamination especially cannot be fully ruled out for larger-model baselines.
  • 2) Method-level evaluation:: Masked spans from benchmark implementations enable executable tests but may not fully represent interactive coding situations.The evaluation partially addresses this by sampling arbitrary cursor positions and using repository-level evaluation.
  • 2) Method-level evaluation:: The study evaluates continuous pre-trained models and their fine-tuned versions, excluding a fine-tuned-only baseline.Prior work is cited as suggesting that fill-in-the-middle capabilities are acquired more effectively during pre-training than fine-tuning alone.
  • 3) Repository-level evaluation:: Repository-level evaluation lacks executable tests and uses ChrF and CrystalBLEU to measure similarity to developer-written completions rather than functional correctness.These metrics reflect similarity with real edits but do not establish semantic equivalence.
  • 4) Inference evaluation:: Inference latency is measured on only a small selection of high-specification machines, leaving lower-end usage scenarios uncovered.Cloud deployment may benefit lower-end machines, while the reported latency is pessimistic because prompt-caching context selection was not used.
  • VI. CONCLUSION & FUTURE WORK: Pharo-specific data curation, continued pre-training, and fine-tuning substantially improve LLMs on syntactic and repository-level benchmarks.The conclusion presents this pipeline as feasible for a severely low-resource language such as Pharo.
  • VI. CONCLUSION & FUTURE WORK: In several settings, specialized Pharo models match or exceed the accuracy of substantially larger general-purpose systems.This result is reported as part of the study’s conclusion about low-resource code completion.
Loading 2607.04939v1…