GigaAM Multilingual: Foundation Model for Underrepresented Languages

• The listed contacts use Gmail addresses, with usernames and the @gmail.com domain shown across the supplied passages.Source: §1 · p. 1

1. Introduction

• Multilingual ASR has advanced through scaling and weakly or self-supervised objectives, yet low-resource and long-tail languages retain prohibitive error rates.The passage cites prior work for progress [2] [4] and the performance disparity.Source: §1 · p. 1
• Naive upsampling to equalize language data can overfit or degrade high-resource performance, motivating principled data mixing and weighting.The passage attributes the disparity partly to head-language-favoring empirical data distributions.Source: §1 · p. 1
• GigaAM extends the efficient self-supervised Russian ASR learner GigaAM to multilingual ASR, pre-training on 2M hours despite small target-language shares.The target Central Asian languages are Kyrgyz, Kazakh, and Uzbek.Source: §1 · p. 1
• Group-aware reweighting uses cluster-level sampling weights to increase low-share language groups’ contribution during SSL pre-training, alongside a data recipe for downstream adaptation.The paper identifies the interaction between pre-training reweighting and fine-tuning as a contribution.Source: §1 · p. 1
• Fine-tuning mixes Russian, English, Kyrgyz, Kazakh, and Uzbek data from open-source corpora, synthetic speech, weak supervision, and internal annotations.Source: §1 · p. 1
• Under matched fine-tuning data and decoding, GigaAM adapts better than Whisper and Omnilingual [4] on Kyrgyz, Kazakh, and Uzbek, and converges faster for minimally covered Bashkir and Georgian.The introduction also describes GigaAM Multilingual as an open-weight state-of-the-art ASR model for Kyrgyz, Kazakh, and Uzbek.Source: §1 · p. 1

2. Related Work

• Whisper demonstrates zero-shot generalization, USM and MMS improve coverage and transfer [2], and Omnilingual ASR targets extensibility [4], yet long-tail languages lag.Source: §2 · pp. 1–2

2.1. Multilingual speech representation learning

• Self-supervised learning builds transferable encoders from unlabeled audio, but multilingual sampling must account for imbalance; mHuBERT-147 scales HuBERT-style training to 147 languages.This motivates group-aware pre-training reweighting and examining how it interacts with downstream adaptation.Source: §2.1 · p. 2

2.2. Data pipelines for low-resource ASR

• Scarce, noisy transcripts are a key tail-language bottleneck, prompting automated corpus construction and refinement.GigaSpeech 2 uses Whisper transcription, MMS forced alignment, and iterative pseudo-label refinement; OWSM v3.2 studies heterogeneity and filtering, while OWSM v4 emphasizes data cleaning and web-data integration. The paper quantifies pipeline components and reports source-aware weighting as important for stable low-resource gains (Section 4.4).Source: §2.2 · p. 2

2.3. Practical encoder selection for low-resource adaptation

• Undisclosed data mixtures and limited ablations complicate encoder choice, motivating controlled evaluation of two approximately 3–5M-hour encoders under identical adaptation settings.The comparison uses the same fine-tuning recipe, weighting strategy, and decoding configuration on the target language set (Section 4.5).Source: §2.3 · p. 2

3.1. Pre-training

• GigaAM uses a 600M-parameter, 24-layer Conformer teacher/student and HuBERT-style masked-unit prediction, training on teacher-derived K=1000 labels from 40%-masked spans.Both encoders use 1024-dimensional hidden states, Rotary Position Embeddings [14], and a 25 Hz frame rate.Source: §3.1 · p. 2

3.2. Data Preparation and Weighting

• To balance 70+ languages, the method clusters a normalized language co-occurrence graph into five groups and weights audio at the cluster level.It assigns recording language using majority-vote and mean-confidence thresholds of 0.7, treating recordings below either threshold as uncertain.Source: §3.2 · p. 2

3.3. Fine-tuning

• Fine-tuning uses a CTC objective, a shared character vocabulary for English, Russian, Kazakh, Kyrgyz, and Uzbek, and open, internal, synthetic, and weakly supervised data.The multi-domain training data is intended to address Central Asian language data scarcity.Source: §3.3 · p. 2

3.3.1. Open-Source Data

• Public read and conversational speech sources include Common Voice, FLEURS, Kazakh Speech Corpus 2, and Yodas, spanning varied conditions and speaking styles.The listed sources are part of a broader collection of publicly available corpora across target languages.Source: §3.3.1 · pp. 2–3

3.3.2. Crowdsourcing and Quality Control

• Crowdsourced Russian, Kyrgyz, Kazakh, and Uzbek utterances receive five independent transcripts, with golden-set checks and reliability-weighted ROVER aggregation [33] supporting quality control.Annotators are removed when golden-set performance conflicts with agreement against the ROVER aggregate on non-gold tasks.Source: §3.3.2 · p. 3

3.3.3. Synthetic Data

• For Kazakh and Kyrgyz coverage, synthetic speech uses mC4 prompts and an in-house TTS system with over 100 voices, retaining utterances only below a CER threshold.A preliminary ASR checkpoint verifies prompt-to-hypothesis agreement and filters mismatches and low-quality generations.Source: §3.3.3 · p. 3

3.3.4. Weakly Supervised Data

• Weakly supervised segments come from long recordings aligned with MMS transcripts, then are filtered using preliminary-ASR CER against the segment transcript.Segments above the CER threshold are discarded as likely misaligned or noisy.Source: §3.3.4 · p. 3

4.1. Experimental Setup

• Pre-training runs for 300k steps with 2048 × 32 s of audio per update; default sampling uses empirical cluster proportions for pre-training and uniform language probabilities with domain stratification for fine-tuning.Sampling weights are applied at the epoch level by reconstructing the training pool.Source: §4.1 · p. 3

4.3. Pretrain Data Mixture

• Shifting pre-training weight toward Central Asian cluster C3 improves Kyrgyz WER from 9.4 to 8.5 and Uzbek from 10.5 to 9.7, with E2 selected for release.Russian WER changes from 4.6 to 4.7 and English from 14.4 to 15.4; E3 recovers English to 14.6% but compromises target-language gains.Source: Table 3 · p. 3, §4.3 · p. 4

4.2. Evaluation

• Evaluation uses Common Voice, FLEURS, and crowdsourced in-the-wild datasets, with greedy decoding and normalized references and outputs for WER.Whisper Large v3 does not support Kyrgyz and is run without a language token; Seamless-M4T v2 is explicitly language-forced, and Omnilingual ASR uses language specified per utterance.Source: §4.2 · p. 3

4.4. Fine-tuning Data Ablation

• Domain-aware fine-tuning most improves Kazakh and Kyrgyz, particularly on the Internal spontaneous-speech split; Uzbek shows no clear benefit over language balancing.Unlike language balancing alone, domain-aware sampling limits domination by large synthetic subsets; the released model combines E2 pre-training with this fine-tuning strategy.Source: Table 4 · p. 4, §4.4 · p. 4
• Language-balanced fine-tuning improves low-resource Kyrgyz, Kazakh, and Uzbek with marginal English regression, whereas unbalanced sampling favors English and degrades tail-language performance.The comparison evaluates natural-distribution, language-balanced, and domain-aware sampling strategies.Source: §4.4 · p. 4

4.5. Encoder Ablation

• Under matched CTC fine-tuning, both GigaAM variants outperform Whisper and Omnilingual; the 240M model achieves 12.2% average WER versus 14.1% for Whisper and 16.6% for Omnilingual, except on English.The 240M encoder is substantially smaller and faster than the larger baselines.Source: Table 5 · p. 4, §4.5 · p. 4

4.6. Tail languages fine-tuning

• GigaAM achieves the lowest WER on Bashkir and Georgian after per-language CTC fine-tuning, despite each language having less than 500 hours of pre-training coverage.Each separate model uses only Common Voice training data: 143 hours for Bashkir and 93 hours for Georgian.Source: Table 6 · p. 4, §4.6 · p. 4

5. Conclusion

• Under matched CTC fine-tuning, the compact 240M model surpasses the larger Whisper Large v3 and Omnilingual 1B encoders, particularly on spontaneous speech.The study reports significant gains on Kazakh, Kyrgyz, and Uzbek.Source: §5 · p. 4
• GigaAM Multilingual is a Conformer encoder pre-trained on 2M hours of audio with cluster-aware sampling and balanced data to prevent head-language dominance.Domain-aware sampling during fine-tuning is also critical to prevent head-language dominance.Source: §5 · p. 4
• The resulting encoder enables rapid adaptation with limited supervision, supports data curation, and is released for future research and production in low-resource settings.Source: §5 · p. 4