PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language

Mullosharaf K. Arabov, Svetlana S. Khaybullina

arXiv:2610.31397v1cs.CL

TL;DR

Bashkir remains poorly represented in modern language models, and systematic evidence on adaptation methods for the language is limited. This paper releases an open corpus and classification datasets and benchmarks adaptation approaches. Results vary by task and model: YandexGPT-5-Lite-8B performs strongly on several comparisons, while CPT effects differ across models.

  • Problem

    Bashkir has limited digital resources, and systematic comparisons of large-language-model adaptation strategies for the language have been lacking.

    Source: §1 · p. 2

  • Method

    The study releases a Bashkir corpus and labelled datasets and benchmarks adaptation strategies across eight architectures, including CPT, with analyses of tokenisers and LLM judges.

    Source: §1 · p. 2, §3.5 · p. 9

  • Results

    YandexGPT-5-Lite-8B with CPT reaches binary-classification macro-F1 0.748 versus 0.653 for XLM-R and 0.650 for mBERT; CPT effects on perplexity vary by model.

    Source: §7 · p. 26

  • Takeaways & Limitations

    Fertility is the only tokenisation metric reported to directly order downstream quality, while MorphAcc and OOV distinguish models less strongly.

    Source: §4.8 · p. 19

    The tokeniser analysis excludes DeepSeek-7B because its fertility value is physically impossible and indicates an input-normalisation artefact.

    Source: §4.8 · p. 19

Abstract

from arXiv · show

Bashkir is a low-resource agglutinative language written in Cyrillic and remains poorly represented in modern LLMs. We present an open resource suite and a systematic benchmark of adaptation methods: the Bashkir Web Corpus (71,567 documents, 46.9M tokens, 16 sources) and three labelled datasets for binary, multiclass and multilabel text classification; the benchmark covers the first two. Experiments use eight architectures (82M-8B: DistilGPT2, GPT-2 base/medium, Phi-3-mini, Mistral-7B, YandexGPT-5-Lite-8B, mBERT, XLM-R) and three adaptation groups (full fine-tuning, QLoRA, in-context learning), plus continual pretraining (CPT). (1) QLoRA on 3.8-8B models outperforms encoder-only fine-tuning on binary classification: YandexGPT-5-Lite-8B with CPT reaches macro-F1 0.748 vs 0.653 (XLM-R) and 0.650 (mBERT); pairwise differences are significant after Holm correction (p<0.001). (2) YandexGPT-5-Lite-8B significantly outperforms Mistral-7B and Phi-3-mini on multiclass classification, BPB and native-speaker ratings (mean 3.28 vs 2.52 and 2.22; n=100; alpha=0.69-0.77; p<0.05); on binary classification its QLoRA advantage is not significant (p>=0.084). (3) CPT effects depend on model and task: perplexity improves for Phi-3-mini (+0.58) and YandexGPT-5-Lite-8B (+0.33) but worsens for Mistral-7B (-1.15, p<0.001); on binary classification CPT helps Mistral-7B most (+4.0 macro-F1). (4) Tokeniser fertility on Bashkir varies 2.67x (2.92-7.80) and tracks adaptation quality, while perplexity across tokenisers is not comparable. Cyrillic tokenisers with Russian pretraining are preferable; LLM judges reproduce native ratings for grammar and coherence (rho=0.33-0.46) but not language correctness (rho=0.16, p=0.11); native evaluation remains necessary. All artefacts are open-licensed.

1 Introduction

Bashkir’s limited digital presence leaves it poorly served by modern language technologies, while existing linguistic resources and adaptation methods have not been systematically benchmarked for the language. This study contributes an open corpus and labelled datasets alongside experiments comparing adaptation strategies and analyses of tokenisation and generation evaluation.

1 Introduction

  • Bashkir has substantially fewer machine-readable texts, annotated corpora, and pretrained models than high-digital-presence languages and several related Turkic languages.The Machine Fund contains more than 75 linguistic databases, but its Oracle-based architecture is designed for information search and aggregation rather than neural-network training.Source: §1 · p. 2
  • The paper situates Bashkir adaptation within broader digital inequality, noting that English-dominated training leaves low-presence languages such as Bashkir excluded from modern language-technology ecosystems.It motivates reproducible benchmarks and open datasets by the needs of applications including translation, speech processing, and text classification.Source: §1 · p. 2
  • Bashkir’s digital inequality reflects tokeniser fragmentation of agglutinative forms, scarce large open corpora, and resources oriented toward search rather than machine learning.The paper notes that these constraints limit training opportunities and that systematic comparisons of LLM adaptation strategies for Bashkir were lacking.Source: §1 · p. 2
  • The study combines an open Bashkir corpus and three labelled datasets with adaptation benchmarks, tokeniser analysis, and validation of LLM judges against native-speaker ratings.It evaluates eight architectures spanning 82M to 8B parameters and releases experiment code, preprocessing scripts, trained adapters, and the corpus under open licences.Source: §1 · pp. 2–3
  • The study’s four questions examine QLoRA versus encoder fine-tuning, model-dependent CPT effects, tokeniser fertility, and whether LLM judges reproduce native-speaker ratings.The questions also ask which generation-quality criteria remain inaccessible to automatic evaluation.Source: §1 · p. 3

2 Related Work

Related work spans Bashkir digitisation and linguistic resources, language technology for Turkic languages, and parameter-efficient adaptation. It establishes both the existing resource base and the gaps motivating a systematic Bashkir benchmark.

2.1 Digitisation of the Bashkir Language and the Formation of Linguistic Resources

  • The Machine Fund’s digital infrastructure grew from early digitisation and lexicographic work into databases for language resources, while Bashkir corpus research also covers literary, folklore, dialectal, and media data.Related studies address corpus annotation, regional variation, code-switching, audio corpora, and speech segmentation.Source: §2.1 · p. 3, §2.2 · pp. 3–4

2.3 Lexicography, Orthography and Formal Grammar

  • Bashkir work on dictionaries, orthography, grammar, and applied platforms describes resources and tools for text normalisation, parsing, and language services.The Machine Fund and Bashkortsoft literature includes dictionary interfaces, spell-checking, and processing of noisy input.Source: §2.3 · p. 4, §2.4 · p. 4

2.5 Morphology of Agglutinative Languages

  • Morphology research includes formal two-level and graph-based descriptions, as well as a neuro-symbolic pipeline combining a language model with a finite-state automaton.The latter improved translation BLEU from 16.14 to 25.71 without retraining the translator.Source: §2.5 · p. 5

2.6 Turkic Languages: Corpora and Machine Translation

  • Turkic-language NLP studies develop multilingual corpora, transfer learning, and data augmentation for translation, alongside Russian–Bashkir parallel resources and LoRA adaptation.Reported work includes a 9.7-million-pair Russian–Bashkir corpus and LoRA-based Bashkir translation systems.Source: §2.6 · p. 5

2.7 Speech Processing for Turkic Languages

  • Turkic speech studies cover Bashkir speech datasets, multilingual recognition, and speech synthesis, with transfer learning often helping but one multilingual system reporting no Bashkir improvement.The review also describes Whisper adaptation and a Bashkir synthesis dataset of 62,852 audio files.Source: §2.7 · pp. 5–6

2.8 Sentiment Analysis and Text Classification

  • Earlier Bashkir sentiment work proposed dictionary and machine-learning methods, with an SVM using stochastic gradient descent reporting the highest precision, recall, and F-measure.The cited work frames sentiment analysis of Bashkir news texts.Source: §2.8 · p. 6

2.9 Parameter-Efficient Fine-Tuning

  • The review describes LoRA, QLoRA, DoRA, and QDoRA, while this study uses QLoRA because unquantised LoRA does not fit the available 24 GB GPU for 7–8B models.DoRA and QDoRA are left for future validation with 4-bit quantisation.Source: §2.9 · p. 6

2.10 Application of PEFT to Turkic Languages

  • Prior Turkic benchmarks cover PEFT and classification, and Tatar studies report that BPE tokenisation can reduce topic-classification quality for an agglutinative language.Related work also compares LoRA/QLoRA adaptation and classical and transformer classification approaches.Source: §2.10 · p. 6

3 Methodology

The methodology builds a Bashkir corpus and classification datasets, then compares model-adaptation strategies using language-modelling, classification, efficiency, and generation-quality measures. It documents compute constraints, evaluation procedures, and released artefacts to support reproducibility.

3.1 Overall Research Design

  • The six-stage design runs from corpus construction and stratified data splitting through dataset creation, model selection, adaptation, and multi-component evaluation.Figure 1 presents the pipeline from source data to evaluation.Source: §3.1 · p. 7, Figure 1 · p. 8

3.2 The Bashkir Web Corpus

  • The Bashkir Web Corpus contains 71,567 documents and about 46.9 million tokens from 16 sources, with Wikipedia providing 59.0% of documents.Source imbalance, missing publication dates, residual web noise, and domain unevenness constrain interpretation.Source: §3.2 · p. 7

3.3 Datasets for Downstream Tasks

  • Three news-based datasets support binary, multiclass, and multilabel classification, with 16,994, 17,897, and 22,318 documents respectively.The binary labels use automatically mapped source categories and undersampling; the multiclass labels are imbalanced, while multilabel annotations use keywords and may be biased.Source: §3.3 · p. 7

3.4 Models

  • The model suite spans 82M to 8B parameters, including small GPT-2-family models, three medium-scale generative models, and mBERT and XLM-R encoder baselines.The medium-scale models are loaded in 4-bit NF4 quantisation; the compute environment uses an NVIDIA L4 GPU with 24 GB.Source: §3.4 · pp. 8–9

3.5 Adaptation Strategies

  • The study compares PEFT, in-context learning, and full fine-tuning, and tests CPT before downstream QLoRA adaptation on three medium-scale generative models.CPT uses 10,000 corpus documents for one epoch, comparing QLoRA alone with CPT followed by QLoRA.Source: §3.5 · pp. 9–10

3.6 Hyperparameters and Metrics

  • Evaluation includes perplexity and BPB, classification F1, computational cost, target-language token ratio, and tokeniser fertility, MorphAcc, and OOV rate.Paired comparisons use McNemar’s exact test or paired bootstrap, with Holm–Bonferroni correction; PEFT and CPT used one seed, limiting between-seed inference.Source: §3.6 · p. 10

3.7 Reproducibility

  • The paper releases its code, preprocessing scripts, trained adapters, corpus, and classification datasets with fixed 80/10/10 splits.The corpus is available in full and 1,000-document sample versions; PEFT uses one seed, limiting statistical power for method comparisons.Source: §3.7 · p. 11

4 Results

BPB favors QLoRA adaptation of Mistral-7B and YandexGPT-5-Lite-8B over full fine-tuning of the GPT-2 family, while perplexity rankings are affected by tokenizer differences. Full fine-tuning also degrades English and Russian performance, with larger degradation on English.

4.1 Language Modelling

  • 0.462 and 0.549 BPB for YandexGPT-5-Lite-8B and Mistral-7B outperform the GPT-2 family’s 0.784–0.917, with 0.91–1.24% of parameters trainable.Full fine-tuning updates 100% of parameters; QLoRA peak memory is 2.5–6.5 GB.Source: Table 2 · p. 11, §4.1 · p. 11
  • GPT-2 medium has the lowest PPL (2.33), but YandexGPT-5-Lite-8B has the lowest BPB (0.462); PPL depends on tokenizer and is not directly comparable across vocabularies.GPT-2’s fertility is 7.80 versus 3.80–4.81 for Mistral-7B and YandexGPT-5-Lite-8B, helping explain the ranking discrepancy.Source: §4.1 · p. 11
  • All six pairwise comparisons of PPL and BPB among the three QLoRA models are significant after Holm correction (pHolm < 0.001; n=2,000).The relative model ordering by PPL and BPB is consistent with the cited comparison study, although absolute PPL differs with evaluation procedures.Source: §4.1 · p. 11
  • 1.83–2.08× English BPB degradation after full fine-tuning exceeds the 1.06–1.21× Russian degradation; model size does not reduce forgetting in the 82M–355M range.The passage attributes the language difference to overlap between Bashkir and Russian Cyrillic vocabularies; BPB is used to assess retention.Source: §4.1 · p. 12
  • QLoRA models retain a 0.59–0.60 Bashkir generation share and were not subject to the observed forgetting, but forgetting was not directly compared with full fine-tuning.The QLoRA experiments used one seed and did not save per-example English/Russian NLL.Source: §4.1 · p. 12

4.2 Continual Pretraining

Continual pretraining (CPT) has model- and task-dependent effects: it improves language-model perplexity for Phi-3-mini and YandexGPT-5-Lite-8B but worsens it for Mistral-7B. Classification effects differ by model and task, and language-model improvements do not consistently predict classification improvements.

4.2 Continual Pretraining

  • +0.58 and +0.33 PPL reduction for Phi-3-mini and YandexGPT-5-Lite-8B are significant (p < 0.001), while Mistral-7B’s PPL increases by 1.15 (p < 0.001).The reported PPL is computed on the test set after CPT and downstream adaptation, and is not comparable with the separate language-modeling results in Table 2.Source: §4.2 · pp. 12–13
  • +4.0 macro-F1 points for Mistral-7B and +2.7 for Phi-3-mini follow CPT on binary classification, while YandexGPT-5-Lite-8B remains at 0.748.For Mistral-7B, the PPL effect is opposite in direction: PPL increases by 1.15 while binary macro-F1 increases by 4.0 points.Source: §4.2 · p. 13
  • +0.034 and +0.012 macro-F1 for YandexGPT-5-Lite-8B and Phi-3-mini on multiclass classification contrast with Mistral-7B’s −0.015.The directions match the respective CPT effects on perplexity for these models.Source: §4.2 · p. 13
  • CPT’s effects depend on model and task, and improved language modeling does not unambiguously translate into improved classification.The study did not isolate CPT from downstream training by evaluating classification after CPT alone.Source: §4.2 · p. 13

4.3 Binary Classification

On binary classification, QLoRA models outperform encoder-only baselines, while YandexGPT-5-Lite-8B’s point advantage over other QLoRA models is not statistically confirmed. CPT helps Mistral-7B most on this task, though its language-modeling effect is in the opposite direction.

4.3 Binary Classification

  • pHolm = 0.084 for YandexGPT-5-Lite-8B versus Phi-3-mini and 0.183 versus Mistral-7B, so the QLoRA models’ within-group differences are not significant.The comparisons use paired bootstrap and McNemar tests with Holm correction across 15 comparisons.Source: §4.3 · p. 13
  • 0.693–0.699 macro-F1 from full fine-tuning of 82M–355M GPT-2 models is comparable to Phi-3-mini QLoRA’s 0.702 with 3.8B parameters.The passage notes consistency with observations for Tatar classification tasks [2026a].Source: §4.3 · p. 13
  • 0.748 macro-F1 for YandexGPT-5-Lite-8B with CPT exceeds XLM-R (0.653) and mBERT (0.650); all three QLoRA models significantly outperform both encoder baselines.The comparison includes QLoRA configurations with and without CPT and the majority baseline (0.342).Source: §4.3 · p. 13, Table 6 · p. 14
  • +4.0 macro-F1 points for Mistral-7B and +2.7 for Phi-3-mini follow CPT, while YandexGPT-5-Lite-8B remains at 0.748.Mistral-7B’s CPT effect on language-modeling quality is opposite: PPL increases by 1.15.Source: §4.3 · p. 14
  • YandexGPT-5-Lite-8B’s rounded binary metrics are unchanged with CPT, but the underlying model state and per-example NLL differ.The passage limits the interpretation to no classification-quality change on this task.Source: §4.3 · p. 14
  • 0.012 macro-F1 separates YandexGPT-5-Lite-8B’s rank-8 and rank-16 configurations (0.736 versus 0.748), while rank 16 is lower for Mistral-7B and Phi-3-mini.The CPT + QLoRA configuration is comparable to or higher than the no-CPT configurations across the three models.Source: §4.3 · p. 14

4.4 Multiclass Classification

On the 19-category task, YandexGPT-5-Lite-8B has the highest reported macro-F1 and significantly outperforms the other QLoRA models. Encoder-only baselines are near or at the majority baseline, and CPT effects vary across models.

4.4 Multiclass Classification

  • 0.368 macro-F1 is the maximum for YandexGPT-5-Lite-8B, significantly above Mistral-7B (0.283–0.298) and Phi-3-mini (0.119–0.166; pHolm < 0.001).The multiclass task has 19 categories, with 47.5% of documents in the dominant class.Source: Table 8 · p. 15, §4.4 · p. 15
  • 0.034–0.053 macro-F1 for encoder-only baselines is close to the 0.034 majority baseline; XLM-R also matches majority-baseline accuracy at 0.470.mBERT’s accuracy is 0.498 versus 0.470 for the majority baseline, a small but significant difference (pHolm = 0.006).Source: §4.4 · p. 15
  • +0.034 and +0.012 macro-F1 from CPT for YandexGPT-5-Lite-8B and Phi-3-mini contrast with Mistral-7B’s −0.015.For these three models, the multiclass CPT effect directions coincide with those on perplexity.Source: §4.4 · p. 15

4.5 In-Context Learning

Few-shot prompting produced unstable binary-task results, with no significant comparisons after correction; YandexGPT-5-Lite-8B remained strongest in zero-shot. Multiclass ICL also fell below QLoRA.

Binary task

  • YandexGPT-5-Lite-8B reached macro-F1 0.679 zero-shot in English and 0.634 with 8 Bashkir examples, while adding 4 examples yielded no gain.Few-shot effects varied across models and prompt languages: Phi-3-mini improved in English but declined in Bashkir, while Mistral-7B also varied by setting.Source: §4.5 · p. 16, Table 10 · p. 16
  • After Holm correction, none of 18 few-shot comparisons was significant, and average accuracy gain across models and languages was +0.001.Effects varied by language; YandexGPT-5-Lite-8B's zero-shot results in Russian and English remained highest.Source: §4.5 · p. 16, Table 11 · p. 16
  • Binary-task ICL macro-F1 values of 0.36–0.68 were comparable to QLoRA's 0.70–0.75, with a 7–12-point macro-F1 gap despite zero trainable parameters.The table reports macro-F1 by instruction language and number of prompt examples.Source: Table 10 · p. 16, §4.5 · p. 17

Multiclass task

  • On multiclass classification, YandexGPT-5-Lite-8B's best ICL macro-F1 was 0.251, below its QLoRA range of 0.332–0.368.These multiclass ICL results were excluded from the main analysis and provided in the replication package.Source: §4.5 · p. 17

4.6 Generation Quality: LLM-as-Judge

LLM judges produced different leaders across direct scoring and pairwise ranking. Pairwise results are an additional signal because uncontrolled text length may bias comparisons.

Direct scoring

  • Direct scoring named YandexGPT-5-Lite-8B highest on grammar (2.89), Mistral-7B on coherence (2.44), and GPT-2 base on language correctness (2.66).The table reports direct scores as means over three judges on a 1–5 scale, alongside pairwise win rates.Source: §4.6 · p. 17, Table 12 · p. 17

Pairwise ranking

  • Mistral-7B led pairwise ranking with win rate 0.529, ahead of YandexGPT-5-Lite-8B at 0.444, while ties were frequent across models.Mistral-7B had 18 wins and 0 losses out of 34 comparisons; YandexGPT-5-Lite-8B had 16 wins and 0 losses out of 36.Source: §4.6 · p. 17, Table 12 · p. 17
  • Uncontrolled length may bias pairwise comparisons between Mistral-7B and YandexGPT-5-Lite-8B toward longer texts, so pairwise ranking is treated as an additional signal.Pairwise ranking favored Mistral-7B, while direct scoring and native-speaker ratings favored YandexGPT-5-Lite-8B.Source: §4.6 · p. 17

Granular evaluation

  • Granular morphology scores correlated with direct grammar ratings (ρ=0.60), while lexicon-language and syntax-coherence correlations were each ρ=0.22.Granular scores were not aggregated for cross-model comparison and were provided as auxiliary results in the replication package.Source: §4.6 · p. 17

4.7 Validation by Native Bashkir Speakers

Native Bashkir speakers rated YandexGPT-5-Lite-8B highest across grammar, coherence, and language correctness. LLM-judge ratings correlated with native ratings for grammar and coherence, but not language correctness.

Native-speaker ratings

  • YandexGPT-5-Lite-8B led native ratings on grammar (2.94), coherence (3.09), and language correctness (3.82).Ratings were on a 1–5 scale; Table 13 reports means over three annotators with bootstrap confidence intervals.Source: §4.7 · p. 18, Table 13 · p. 18

Pairwise comparisons

  • YandexGPT-5-Lite-8B significantly outperformed Mistral-7B on grammar (Δ=+0.82), coherence (Δ=+0.91), and language correctness (Δ=+0.58).Mistral-7B and Phi-3-mini could not be ordered by native-speaker ratings because their differences were not significant after correction.Source: §4.7 · p. 18

LLM-judge validation

  • LLM-judge ratings correlated with native ratings for grammar (ρ=0.462) and coherence (ρ=0.334), but not language correctness (ρ=0.162, p=0.11).Native-speaker evaluation remains necessary for Bashkir-language correctness.Source: §4.7 · p. 18
  • LLM judges compressed rating ranges and sometimes changed model rankings, including placing GPT-2 base above YandexGPT-5-Lite-8B on language correctness.Native speakers rated those models 3.82 versus 2.33, respectively.Source: §4.7 · p. 18

4.8 Tokeniser Analysis

Bashkir tokeniser fertility varied substantially and ordered downstream quality across the three tested model groups. Morphological boundary accuracy and OOV rate distinguished models less strongly.

Fertility

  • Bashkir fertility ranged from 2.92 subtokens per word for T-lite-2.1 to 7.80 for the GPT-2 family and Phi-2, a 2.67-fold spread.The table compares fertility on Bashkir, English, and Turkish test sets.Source: §4.8 · p. 19, Table 15 · p. 19

Cross-language comparison

  • Bashkir fertility spread was 2.67-fold, versus 1.09-fold on English and 1.25-fold on Turkish; the paper attributes the larger Bashkir spread to its low-resource status.The Bashkir/Turkish fertility penalty was 2.18 times for the GPT-2 family and Phi-2, 1.28 for Mistral-7B, and 1.18 for YandexGPT-5-Lite-8B.Source: §4.8 · p. 19

Fertility and downstream quality

  • The Bashkir/Turkish fertility ratio ordered the three tested model groups identically to downstream quality: GPT-2 family, Mistral-7B, then YandexGPT-5-Lite-8B.Binary macro-F1 ranges were 0.699, 0.709–0.721, and 0.736–0.748, respectively; multiclass ranges were 0.122, 0.283–0.298, and 0.332–0.368.Source: §4.8 · p. 19

Other tokenisation metrics

  • MorphAcc ranged from 0.047 to 0.093 and OOV rate from 0.017 to 0.065, while only fertility directly ordered downstream quality.DeepSeek-7B was excluded from Tables 15 and 16 because an impossible fertility value indicated an input-normalisation artefact.Source: §4.8 · p. 19, Table 16 · p. 19

4.9 Variance Decomposition

The variance decomposition suggests adaptation strategy accounts for more variation in perplexity and BPB, while architecture accounts for more variation in language alignment. These results are preliminary, and model size and adaptation method are confounded in the current configuration.

4.9 Variance Decomposition

  • The method factor compares small-model Full FT with large-model QLoRA, not adaptation methods independently; separating their effects requires a crossed design.GPT-2 family models were trained only by Full FT, while 3.8–8B generative models were trained only by QLoRA.Source: §4.9 · p. 20
  • 73% of PPL variance is explained by method (η2 = 0.73, p= 0.002), 72% of BPB variance by method (p= 0.002), and 81% of target-token variance by architecture (p< 0.001).The decomposition indicates adaptation strategy dominates PPL and BPB, whereas architecture dominates the proportion of target tokens.Source: Table 17 · p. 20, §4.9 · p. 20
  • For PPL, architecture had a large but nonsignificant contribution (η2 = 0.54, p= 0.16), while seed was also nonsignificant (η2 = 0.13, p= 0.18).For BPB, architecture explained 36% (p= 0.46) and seed 5% (p= 0.43); neither was significant.Source: §4.9 · p. 20
  • With n= 15 PPL/BPB observations and n= 9 target-token observations, the analysis has limited power and is preliminary; further work should add configurations, especially random seeds.The large PPL architecture effect did not reach significance at this sample size.Source: §4.9 · p. 20

5 Discussion

YandexGPT-5-Lite-8B leads on multiclass classification, BPB, and native-speaker ratings, while the effects of continual pretraining vary by model and task. The discussion connects adaptation quality with tokenisation and cautions that LLM judges do not reproduce native ratings for Bashkir language correctness.

5.1 Factors Behind the Superiority of YandexGPT-5-Lite-8B

  • YandexGPT-5-Lite-8B significantly outperforms Mistral-7B and Phi-3-mini on multiclass classification, BPB, and native-speaker ratings, but its binary-classification advantage is not significant.On native-speaker ratings, the means are 3.28 versus 2.52 and 2.22; binary-classification comparisons have pHolm values of 0.183 and 0.084.Source: §5.1 · p. 20
  • A Cyrillic tokeniser and Russian-language pretraining are proposed as contributors to YandexGPT-5-Lite-8B’s advantage, not model size alone.Its fertility is 3.80 subtokens per Bashkir word versus 7.80 for GPT-2; the authors note that Phi-3-mini fertility was not measured, so its quality gap cannot be attributed directly to fertility.Source: §5.1 · pp. 20–21

5.2 The Selective Effect of Continual Pretraining

  • CPT’s benefit depends on both model and task: it gives Mistral-7B the largest binary macro-F1 gain, while its effects differ across perplexity and multiclass classification.On binary classification, Mistral-7B gains 4.0 points; PPL improves for Phi-3-mini and YandexGPT-5-Lite-8B but worsens for Mistral-7B, and multiclass effects are positive for the former two and negative for Mistral-7B.Source: §5.2 · p. 21

5.3 Divergence Between LLM-as-Judge Modes and Native Speakers

  • LLM judges’ pairwise ranking can reverse direct-scoring and native-speaker comparisons, and their ratings correlate with native ratings for grammar and coherence but not language correctness.The authors attribute the pairwise discrepancy most likely to missing length control and conclude that native-speaker evaluation remains necessary for language correctness.Source: §5.3 · pp. 21–22

5.4 Tokenisation as a Source of Digital Inequality

  • Bashkir tokeniser fertility ranges from 2.92 to 7.80, and higher fertility accompanies weaker GPT-2-family results on BPB and native-speaker grammar ratings.The authors caution that the relationship is not deterministic: only five models had both measured fertility and downstream results, and Phi-3-mini fertility was not measured.Source: §5.4 · p. 22

5.5 Practical Recommendations

  • The discussion favors Cyrillic tokenisers with Russian pretraining and QLoRA on 3.8–8B models over encoder-only fine-tuning for classification, while treating CPT as model- and task-dependent.For YandexGPT-5-Lite-8B, QLoRA reaches binary macro-F1 0.736 and 0.748 with CPT, versus 0.650 for mBERT and 0.653 for XLM-R; the recommendation also retains native-speaker evaluation for final generation-quality conclusions.Source: §5.5 · p. 23

6 Limitations

The limitations qualify the benchmark’s statistical strength, task coverage, evaluation evidence, and model scope. They also identify additional experiments needed to test close comparisons and broader conclusions.

6.1 A Single Random Seed for PEFT Experiments

  • PEFT and CPT experiments use one random seed, so comparisons lack between-seed variability estimates and close QLoRA differences need at least three seeds to confirm.Test-set bootstrap intervals capture uncertainty across examples, not model initialisation.Source: §6.1 · p. 23

6.2 Multiclass Classification

  • No multiclass configuration exceeds macro-F1 0.37 on the 19-category task, where 47.5% of documents belong to the dominant class.The QLoRA results are treated as a lower bound; weighted loss, resampling, or hierarchical classification remain possible approaches.Source: §6.2 · p. 23

6.3 Sample and Agreement in Native-Speaker Evaluation

  • Native-speaker evaluation covers 100 texts with uneven per-model samples, limiting comparisons between similarly performing models.Agreement ranges from α=0.694–0.772, below the stated 0.80 threshold for high agreement; the authors call for at least 30 texts per model.Source: §6.3 · p. 24

6.4 Divergence Between LLM-as-Judge and Native Speakers

  • LLM-judge scores correlate significantly with native ratings for grammar and coherence, but not language correctness (ρ=0.162, p=0.11).Judges compress rating ranges and can change model rankings; the authors treat them as an additional formal-grammatical signal, not a replacement for native evaluation.Source: §6.4 · p. 24

6.5 Scope of the Tokeniser Analysis

  • The fertility–adaptation relationship is correlational across five models, MorphAcc remains low (0.047–0.093), and fertility estimates lack confidence intervals.Controlled vocabulary changes are needed to test causality; the reported fertility point estimate is a ratio of total subtokens to total words, so per-document-ratio bootstrap intervals would not estimate its uncertainty.Source: §6.5 · p. 24
  • Perplexity uses sequences truncated to 128 tokens, which cover fewer Bashkir words for higher-fertility tokenisers and can mislead cross-model comparison.The authors use BPB as the primary cross-model metric and call for full-document-length evaluation at 512 or 1024 tokens.Source: §6.5 · p. 24

6.6 Scope of the Model and Excluded Configurations

  • The study tests models from 82M to 8B parameters, and its QLoRA-versus-full-fine-tuning comparison contrasts 7–8B models with fully fine-tuned 82M–355M models.Models above 14B were not tested, and full fine-tuning of 3–8B models could yield a different comparison; DoRA, QDoRA, and some ICL configurations were also excluded or incomplete.Source: §6.6 · p. 25

6.7 Directions for Further Work

  • Further work includes more PEFT seeds, methods for multiclass imbalance, larger native-speaker samples, controlled vocabulary experiments, and full-length perplexity evaluation.RAG and multi-agent-debate pilots were too small for final conclusions and remain outside the present study’s scope.Source: §6.7 · p. 25

7 Conclusion

The paper releases Bashkir language resources and benchmarks adaptation strategies across models and tasks. Results favor QLoRA over encoder baselines on binary classification, while model and continual-pretraining effects vary by task; tokeniser fertility is associated with adaptation quality.

Contribution

  • The corpus contains 71,567 documents and 46.9 million tokens from 16 sources; the corpus, datasets, code, adapters, and evaluation scripts are open-licensed.The authors present the resource suite and benchmark as resources for further work on Bashkir and other low-resource agglutinative languages with non-standard scripts.Source: §7 · pp. 25, 27
  • The benchmark tests eight architectures across full fine-tuning, QLoRA, and in-context learning, and evaluates continual pretraining before QLoRA.The paper provides the Bashkir Web Corpus, three labelled datasets, and trained adapters.Source: §7 · p. 25

Results

  • Macro-F1 reaches 0.748 for YandexGPT-5-Lite-8B with CPT, versus 0.653 for XLM-R and 0.650 for mBERT on binary classification.The QLoRA-versus-encoder baseline comparison is significant after Holm correction (pHolm < 0.001).Source: §7 · p. 26
  • YandexGPT-5-Lite-8B significantly outperforms Mistral-7B and Phi-3-mini on multiclass classification, BPB, and native-speaker ratings, but its binary-classification advantage is not significant.The reported comparisons include multiclass macro-F1 0.368 versus 0.283–0.298 and 0.119–0.166, and mean native-speaker ratings of 3.28 versus 2.52 and 2.22.Source: §7 · p. 26
  • CPT effects depend on model and task: perplexity improves for Phi-3-mini and YandexGPT-5-Lite-8B but worsens for Mistral-7B, which gains 4.0 macro-F1 points on binary classification.The paper concludes that CPT configuration should be chosen with the task in mind.Source: §7 · p. 26
  • Tokeniser fertility varies 2.67-fold, from 2.92 to 7.80, and tracks observed adaptation quality; perplexity is unsuitable for cross-model comparison under different tokenisers.GPT-2 medium has the lowest PPL (2.33) but the worst BPB (0.784); the authors use BPB as the primary cross-model metric.Source: §7 · p. 26
Loading 2610.31397v1…