PaperlayerView exact revision on arXiv

Source-linked AI summary

LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language

Luca Ballore

arXiv:2605.09015v1cs.CLcs.LG

TL;DR

Sardinian has little representation in NLP despite roughly one million speakers, motivating LLiMba’s adaptation of a 3B model using Sardinian data, continued pretraining, and supervised fine-tuning on one consumer GPU. The strongest configuration, rsLoRA r256, reaches 28.5 BLEU from English into Sardinian, while qualitative evaluation shows that translation metrics can miss factual and morphological failures.

  • Problem

    Sardinian has minimal NLP coverage because commercial systems do not support it and existing training resources contain little usable Sardinian text.

  • Method

    LLiMba adapts Qwen2.5-3B-Instruct through continued pretraining and supervised fine-tuning using Sardinian and related Romance replay text on a single 24 GB consumer GPU.

  • Results

    rsLoRA r256 reaches 28.5 BLEU from English into Sardinian, versus 17.3 after continued pretraining and 21.0 with full fine-tuning, while ranking qualitative behavior differently from translation metrics.

  • Takeaways & Limitations

    Adapter capacity matters more than the choice among LoRA variants in this setting, and qualitative factuality checks are needed because loss and translation metrics do not measure external truth.

  • Takeaways & Limitations

    All tested SFT methods still fabricate on factual queries about content absent from training data.

Abstract

from arXiv · show

Sardinian, a Romance language with roughly one million speakers, has minimal presence in modern NLP. Commercial services do not support it, and current language models do not produce it reliably. We present LLiMba, a 3B parameter Sardinian-ready model adapted from Qwen2.5-3B-Instruct through continued pretraining (CPT) and supervised fine-tuning (SFT) on a single 24 GB consumer GPU. The corpus contains 11.5 million tokens of Sardinian spanning LSC, Logudorese, and Campidanese, augmented with 2.4 million tokens of related Romance text as replay against register blurring. After CPT the model reaches a perplexity of 6.76 on held out Sardinian and outperforms the base across all six FLORES-200 directions. We compare five SFT configurations under matched conditions: full fine-tuning, LoRA r64, rsLoRA r128, rsLoRA r256, and DoRA r256. rsLoRA r256 wins on every direction into Sardinian, reaching 28.5 BLEU from English against 17.3 after CPT and 21.0 with full fine-tuning. The rank ablation places r128 between LoRA r64 and rsLoRA r256 on BLEU but reveals failure modes invisible to the metric, including leakage across scripts no other variant produces. LoRA r64 retains less factual content from SFT than configurations at higher rank and produces more confident fabrications, though all methods fabricate on content absent from training. DoRA r256 yields the smallest gap between training and evaluation but the worst factual accuracy. The findings indicate that adapter capacity matters more than the choice among LoRA variants for adapting a Romance pretrained base to a low resource Romance target, that stronger regularization is not uniformly beneficial, and that translation metrics smoothly order configurations whose qualitative behavior differs categorically. Perplexity comparisons across scripts must account for byte fallback tokenization, which deflates the metric for scripts other than Latin.

1 Introduction

LLiMba addresses Sardinian’s severe underrepresentation in NLP by adapting a Romance-pretrained model with a reproducible, single-GPU pipeline. The study compares five SFT configurations while documenting failure modes and evaluation caveats.

  • Motivation: Sardinian has roughly one million speakers but little usable training data and no major commercial NLP support.Its invisibility reflects sparse corpora and insufficient commercial incentives for data acquisition.
  • Motivation: Romance-family similarity makes adaptation focus on Sardinian-specific lexicon, orthography, and idiom rather than learning the language family from scratch.The base model can provide linguistic scaffolding from related Romance languages.
  • Approach: LLiMba adapts Qwen2.5-3B-Instruct using approximately 11.5 million Sardinian tokens, related Romance replay text, CPT, and SFT on one 24 GB GPU.The corpus begins with approximately 13.5 million Sardinian tokens and retains 11.5 million after filtering and deduplication.
  • Approach: The study compares full fine-tuning, LoRA r64, rsLoRA r128, rsLoRA r256, and DoRA r256 under matched conditions.The comparison is designed to separate adapter capacity and method choice.
  • Evaluation: The paper combines translation metrics with qualitative analysis because cleaning, prompt phrasing, temperature, and tokenization produce failure modes that metrics may miss.It also documents a reproducible pipeline intended for individual researchers and small labs.

2 Background

Prior work establishes two-stage adaptation and parameter-efficient tuning as relevant approaches, while Sardinian provides a distinct Romance-language setting with a closer pretrained starting point than Tibetan. The evaluation uses FLORES-200 translation directions and reports both BLEU and chrF.

  • Related adaptation work: Tibetan adaptation studies use continued pretraining followed by supervised fine-tuning, but differ from Sardinian in typology, script, and tokenization behavior.Sardinian instead benefits from substantial Romance-language pretraining in Qwen2.5-3B-Instruct.
  • Adapter methods: LoRA replaces full weight updates with low-rank adapters, while rsLoRA uses α/√r scaling to stabilize higher ranks.DoRA separately adapts weight magnitude and direction to preserve directional structure.
  • Adapter methods: LoRA can underperform full fine-tuning when the target domain is far from pretraining, while preserving capabilities outside the target domain more effectively.This finding motivates full fine-tuning during CPT in the present study.
  • Evaluation: FLORES-200 supplies parallel sentences for six Sardinian translation directions, evaluated through lm-evaluation-harness with BLEU and chrF.chrF is included because morphological richness and dialectal variation can make exact-match BLEU less robust.

3 Data

The data pipeline combines heterogeneous Sardinian text, related Romance replay, machine-translated and native-curated instruction data, and held-out FLORES-200 evaluation. It spans Sardinian’s principal written variants while targeting LSC output.

  • Dataset structure: The training data comprises Sardinian pretraining text with Romance replay, instruction-pair SFT data, and a FLORES-200 held-out evaluation set.These groups support language adaptation, instruction following, and translation assessment.
  • Pretraining corpus: 11.5 million Sardinian tokens remain after filtering and deduplication from approximately 13.5 million collected tokens.The prepared corpus composition is summarized in Table 1.
  • Pretraining corpus: The corpus covers LSC, Logudorese, and Campidanese, exposing the model to all variants while targeting LSC for output.This reflects their distribution across news, institutional, and literary sources.
  • Pretraining corpus: Approximately 2.4 million tokens of Italian, Spanish, Portuguese, and Catalan replay text are mixed into pretraining to mitigate forgetting and distinguish Sardinian from Italian.The replay carries no language tag, so the model learns language distinctions from the text itself.
  • SFT data: The SFT pool combines machine-translated Capybara instructions with native-curated translation, synthesized, and song-related examples.Residual translation calques remain a documented limitation of the machine-translated pool.
  • SFT data: 14,404 SFT pairs and approximately 12.8 million tokens remain after deduplication and fivefold upsampling of the synthesized bucket.The synthesized bucket contributes 422 unique pairs before upsampling.
  • SFT data: About six percent of SFT examples use non-Sardinian system prompts or no system prompt to expose the model to varied deployment conditions.The intended behavior is for response language to follow the user’s request rather than the system prompt language.
  • Evaluation data: FLORES-200 evaluation uses 997 parallel sentences across Sardinian, Italian, English, Spanish, French, and Portuguese in six translation directions.A qualitative probe set supplements BLEU and chrF by testing conversation, factuality, continuation, creativity, and grammar.

4 Method

LLiMba uses full-parameter CPT followed by independently compared SFT methods, with memory-saving techniques enabling 3B-parameter training on a 24 GB GPU. Matched SFT runs isolate adapter method and rank as the main variables.

  • Training pipeline: The two-stage pipeline performs full-fine-tuning CPT on Sardinian data, followed by SFT in five configurations starting from the same CPT model.Both stages use fixed random seeds and run on a single RTX 4090 with 24 GB of VRAM.
  • Training pipeline: Qwen2.5-3B-Instruct preserves an instruction-following scaffold that CPT can partially erase and SFT can re-anchor with limited data.Its 3B size also fits the available hardware with memory-saving measures.
  • Continued pretraining: CPT updates all model parameters because teaching a new language is treated as a large domain shift better served by the full parameter set.This choice follows prior evidence that LoRA can underperform full fine-tuning for distant target domains.
  • Continued pretraining: Unpacked training prevents attention leakage across document boundaries, while overlapping 4096-token windows preserve context in long documents.Documents exceeding the sequence length use a 128-token overlap rather than truncation.
  • Hardware: Paged 8-bit AdamW, bfloat16 weights, gradient checkpointing, and accumulation reduce peak VRAM use to 22–23 GB for full fine-tuning.Without the 8-bit optimizer, full fine-tuning would not fit the 24 GB hardware budget.
  • Supervised fine-tuning: The SFT comparison includes full fine-tuning, LoRA r64, rsLoRA r128, rsLoRA r256, and DoRA r256.rsLoRA r256 uses α/√r scaling, while DoRA separately adapts magnitude and direction.
  • Supervised fine-tuning: All five SFT runs share data, templates, loss formulation, effective batch size, and hardware, isolating adapter method and rank.The loss is restricted to assistant completions, with system prompts and user turns masked out.
  • Model selection: The rsLoRA r256 model is selected for deployment based on the reported results, while the other four configurations serve as comparison points.This deployment choice follows the matched-condition SFT comparison.

5 Results

CPT supplies most of the translation improvement, while SFT configurations differ more sharply in their qualitative factual and lexical behavior than translation metrics reveal. rsLoRA r256 is strongest for translation into Sardinian, whereas other directions saturate near CPT performance.

  • Translation benchmarks: CPT improves BLEU across all six directions by four to seven times over the base, with SC-to-EN rising from 11.73 to 33.52 BLEU.EN-to-SC shows the largest relative gain, moving from 2.75 to 17.26 BLEU.
  • Translation benchmarks: 28.47 BLEU makes rsLoRA r256 the strongest SFT configuration on EN-to-SC, ahead of LoRA r64 at 23.60 and full fine-tuning at 21.04.The same ordering repeats on IT-to-SC and ES-to-SC.
  • Translation benchmarks: CPT wins SC-to-ES at 19.31 BLEU and 47.76 chrF, while full fine-tuning narrowly wins SC-to-IT at 18.12 BLEU but trails CPT on chrF.SFT adds little for translation from Sardinian into closer Romance languages.
  • Translation benchmarks: DoRA r256 trails LoRA r64 on four of six directions despite four times the capacity, with only marginal edges on SC-to-IT and SC-to-ES.Those two edges occur where SFT methods cluster tightly and differences remain within noise.
  • Qualitative findings: rsLoRA r128 lies between LoRA r64 and rsLoRA r256 by BLEU on every into-Sardinian direction, but this smooth ranking misses qualitative failures.The ablation shows leakage across scripts that no other variant produces, based on a single training run without seed or data-order replication.
  • Qualitative findings: Full fine-tuning stays closest to attested vocabulary, whereas rsLoRA r256 and DoRA r256 produce localized clusters of unattested Sardinian-shaped forms.The paper characterizes these forms as morphological hallucination, distinct from factual fabrication and most visible in long unconstrained descriptions.
  • Qualitative findings: rsLoRA r256 is the only SFT variant whose focused factual responses consistently withstand verification, while LoRA r64, DoRA r256, and rsLoRA r128 fabricate increasingly severe details.All variants remain fluent after CPT, but factual grounding and lexical fidelity diverge substantially.
  • Qualitative findings: DoRA r256 produces the smallest train-evaluation gap, but the reported qualitative probes still show the worst factual accuracy among the configurations.The train-evaluation gap is roughly constant across methods, between 0.08 and 0.12, with evaluation loss slightly below training loss.

6 Discussion

The discussion finds that adapter capacity and parameterization jointly shape translation quality, while qualitative factuality can diverge sharply from loss- and translation-based metrics. It also argues that tokenization artifacts and external factual checks are necessary for interpreting evaluation results.

  • 6.1 Capacity and adapter choice: rsLoRA r128 beats LoRA r64 on every translation direction yet produces worse factuality and the only observed cross-script leakage.The authors report that the usable-versus-unusable rank threshold for their SFT pool lies somewhere between 128 and 256, while acknowledging that the proposed explanation is not directly tested.
  • 6.1 Capacity and adapter choice: At rank 256, unconstrained rsLoRA outperforms DoRA on translation, showing that adapter parameterization matters alongside capacity.The authors attribute the comparison to rsLoRA’s unconstrained low-rank update versus DoRA’s magnitude-direction decomposition, while noting that they lack ablations to disentangle the mechanism.
  • 6.2 The performance-factuality paradox: DoRA r256 produces confident fabrications despite training data containing the relevant correct facts, whereas rsLoRA r256 produces accurate outputs on the same probes.The authors speculate that DoRA’s decomposition makes overwriting strong base-model associations harder, but explicitly do not test this hypothesis directly.
  • 6.3 The limits of loss-based evaluation: Loss-based metrics track BLEU but not factuality, so falling training or evaluation loss can coexist with outputs that are not grounded in external truth.The qualitative probes distinguish model behaviors that BLEU, chrF, training loss, and evaluation loss separate only modestly.

7 Limitations

The study’s limitations concern data reproducibility, evaluation coverage, mechanistic uncertainty, and residual hallucination, while some behaviors remain prompt-sensitive.

  • Data and reproducibility: The pretraining corpus cannot be redistributed in raw form because it includes sources whose copyright status was not exhaustively verified.Reproducibility therefore depends on rerunning the collection pipeline against original sources that may change or disappear.
  • Scope and generalization: The findings about adapter behavior may not generalize beyond Qwen2.5-3B-Instruct, Sardinian, or targets lacking comparable Romance-language support.The experimental design fixes the base model, CPT corpus, and target language, and the authors describe mechanistic differences from typologically distant settings.
  • Evaluation and analysis: BLEU and chrF are imperfect proxies, while the qualitative probe is small, hand-curated, single-speaker reviewed, and prompt-specific.The authors call for broader dialect coverage, systematic factuality evaluation, and systematic prompt variation.
  • Evaluation and analysis: The explanations for DoRA underperformance and rsLoRA r256’s advantage remain hypotheses because the paper does not isolate scaling, decomposition, or rank effects.The empirical claim is limited to matched-data, matched-hardware, and matched-hyperparameter comparisons among tested methods.
  • Residual model behavior: All tested SFT methods still fabricate on factual queries outside training data, and broad prompts can elicit unattested Sardinian-like words.The morphological hallucination is most visible for one extended cultural-description prompt and is largely absent from focused question answering and translation.

8 Conclusion

The paper shows that a 3B model can be adapted to Sardinian on a single consumer GPU, then compares SFT configurations and identifies both performance gains and qualitative failure modes. It also points toward preference tuning, larger models, and transfer to other low-resource Romance languages.

  • 8 Conclusion: A 3B parameter model can be adapted to Sardinian end-to-end on a single consumer GPU using continued pretraining followed by parameter-efficient supervised fine-tuning.The strongest configuration is rsLoRA at rank 256.
  • 8 Conclusion: rsLoRA r256 outperforms standard LoRA r64 and full fine-tuning, while DoRA at the same rank has the worst factual grounding among tested methods.The comparison uses matched data and hardware.
  • 8 Conclusion: The r128 ablation falls between LoRA r64 and rsLoRA r256 on translation metrics but introduces script leakage and biographical errors that those metrics do not reveal.The qualitative comparison also identifies factual fabrication and morphological hallucination as distinct failure modes.
  • Future work: Preference tuning, scaling to 7B parameters, and applying the pipeline to other low-resource Romance languages are proposed extensions.The suggested languages include Corsican, Friulian, Romansh, and Sicilian.

A Translation evaluation results

The translation evaluation uses BLEU and chrF on a held-out subset of FLORES-200 with a fixed decoding and prompting setup.

  • Translation evaluation results: 997 sentences from FLORES-200 are evaluated with BLEU and chrF using the same held-out subset, greedy decoding, and chat-template prompts.The evaluations run through lm-evaluation-harness 0.4.11.

A.1 BLEU with standard errors

Table 6 reports BLEU scores with standard errors for all evaluated translation directions and highlights the best score in each direction.

  • A.1 BLEU with standard errors: Table 6 organizes BLEU scores with standard errors by translation direction and marks the best score per direction in bold.Use the standard errors to judge the uncertainty around each reported score.

A.2 chrF with standard errors

Table 7 reports chrF scores with standard errors, identifying the best score in each translation direction.

  • Table 7 reports chrF scores with standard errors and marks the best score per direction in bold.

B.1 Paragraph about Sardinia

The generated Sardinia paragraph describes the island’s geography, history, culture, language, traditions, and contemporary life, while some outputs contain factual or linguistic errors.

  • Sardinian is presented as an ancient Romance language with a continuing role in Sardinian identity and literature.
  • Sardinia is described as a Mediterranean island with coasts, mountains, forests, plains, beaches, caves, and nuraghes.
  • The passages portray Sardinian culture through pastoral life, traditional music, gastronomy, festivals, crafts, villages, and local traditions.
  • The LSC passages describe a written reference norm mediating Logudorese and Campidanese varieties without replacing local forms.

B.9 Gigi Riva

The outputs variably describe Gigi Riva as a major Sardinian football figure, but they also contain contradictory biographies, fabricated details, and a non-Sardinian greeting response.

  • The responses also give incompatible death dates and ages, reporting both 2023 at 78 and 2024 at 79.
  • Several outputs identify Gigi Riva as a famous Italian footballer and a major figure in Sardinian football history.
  • The more detailed account places Riva’s birth in Leggiuno in 1944 and links his career to Cagliari, including the 1969–70 Serie A title.
  • Other outputs contradict this account by placing his birth in Casteddu or Tergu and assigning him careers at Juventus, Milan, or Inter.
  • Greeting responses range from Sardinian replies to an Italian response that explicitly says it does not understand the situation well.

B.11 Grammatical analysis

The grammatical-analysis outputs consistently identify the subject and verb, while varying in how they classify and explain the sentence’s complements.

  • They treat “chi si chistionat in Sardigna” as a further complement or clause connected with the language’s being spoken in Sardinia.
  • The analyses identify “Sa limba sarda” as the subject and “est” as the verb.
  • They classify “una limba romànica” as a complement describing or identifying the Sardinian language.
  • One explanation separately interprets “in Sardigna” as indicating the location where Sardinian is spoken.
Loading 2605.09015v1…