Source-linked AI summary
GigaAM Multilingual: Foundation Model for Underrepresented Languages
Andrei Kuzmenko, Alexandr Maximenko, Aleksandr Kutsakov, Georgii Gospodinov, Dmitrii Bolotov, Oleg Kutuzov, Pavel Bogomolov, Fyodor Minkin
TL;DR
Long-tail languages still have uneven multilingual ASR quality because of severe data scarcity. This paper introduces balanced pre-training and domain-aware fine-tuning for GigaAM Multilingual, which outperforms strong open encoders on Central Asian languages, especially spontaneous speech.
Problem
Multilingual ASR remains highly uneven, with long-tail languages exhibiting prohibitive error rates despite strong performance on high-resource languages.
Method
GigaAM Multilingual uses cluster-aware group reweighting during self-supervised pre-training and domain-aware sampling when fine-tuning on multilingual data.
Results
12.2% average WER versus 14.1% for Whisper and 16.6% for Omnilingual: the compact GigaAM encoder outperforms both larger baselines, particularly on target-language spontaneous speech.
Takeaways & Limitations
The released encoder and ASR model provide an efficient recipe for adapting multilingual ASR to underrepresented languages under realistic data imbalance.
Abstract
from arXiv · showhide
Despite recent scaling successes, multilingual ASR performance remains highly uneven, with long-tail languages suffering from severe data scarcity. This work addresses the challenge of building robust foundation models for underrepresented Central Asian languages (Kazakh, Kyrgyz, Uzbek). We present GigaAM Multilingual, a Conformer encoder pre-trained on 2M hours of audio using a HuBERT-style objective. Crucially, we introduce a cluster-level data balancing strategy during pre-training and a domain-aware sampling method during fine-tuning to mitigate head-language dominance. In controlled comparisons, our approach outperforms strong open pretrained encoders (Whisper Large v3, Omnilingual-1B) on target languages, achieving significant gains on spontaneous speech while maintaining efficiency. We release the foundation encoder and ASR model, offering a proven recipe for effective multilingual adaptation under realistic data imbalance.
1 SaluteDevices, Russia {andrey.kuzmenko2907, ae.maximenko, askutsakov, georgygospodinov, bolotovdm,
This section provides contact information for the SaluteDevices, Russia authors.
- Authors and affiliation: The listed contact addresses use gmail.com email accounts.The addresses are associated with olegkutuzov01, bobrosoft98, minkin.fyodor, and other listed handles.
1. Introduction
Multilingual ASR progress has not eliminated severe quality disparities caused largely by skewed data distributions. This work extends GigaAM to underrepresented Central Asian languages using group-aware pre-training and multilingual fine-tuning strategies.
- Motivation: Multilingual ASR remains highly uneven: high-resource languages perform strongly, while many low-resource and long-tail languages have prohibitive downstream error rates.The disparity persists despite scaling data, model capacity, and weakly supervised or self-supervised objectives.
- Motivation: Data imbalance favors head languages, while naive equalization can cause overfitting or degrade high-resource-language performance.The introduction motivates principled data mixing and weighting during pre-training and adaptation.
- Approach: GigaAM Multilingual extends an efficient Russian self-supervised learner to multilingual ASR by pre-training on a 2M-hour corpus with highly skewed language-group proportions.Kyrgyz, Kazakh, and Uzbek are included but occupy small shares, and general-purpose ASR models often struggle in this low-resource regime.
- Approach: The method uses group-aware reweighting during self-supervised pre-training to increase low-share language groups’ contributions through cluster-level sampling weights.Fine-tuning mixes Russian, English, Kyrgyz, Kazakh, and Uzbek using open-source, synthetic, weakly supervised, and internally annotated data.
- Results and contributions: Under matched fine-tuning data and decoding, GigaAM adapts better than Whisper and Omnilingual pretrained encoders on Kyrgyz, Kazakh, and Uzbek, while converging faster on minimally covered languages.The work evaluates pre-training and fine-tuning sampling through ablations and reports cross-lingual transfer to tail languages.
2. Related Work
Related work shows that large-scale multilingual ASR and self-supervised encoders improve coverage and transfer, but long-tail languages remain unevenly served because scaling alone cannot overcome language, dataset, and transcription-data imbalance. Recent work therefore emphasizes balanced sampling, automated corpus refinement, heterogeneous-data management, and controlled encoder comparisons.
- Large-scale multilingual ASR: Large-scale systems such as Whisper, USM, MMS, and Omnilingual ASR improve multilingual robustness, coverage, transfer, and rapid adaptation through scaling and pre-training.Whisper uses large-scale weak supervision for zero-shot generalization, while USM and MMS rely on multilingual pre-training and fine-tuning; Omnilingual ASR emphasizes extensibility and minimal-data adaptation.
- Long-tail limitations: Improvements remain uneven across languages and varieties, with long-tail languages still lagging behind.
- Balanced self-supervised learning: Multilingual self-supervised learning shows that scaling alone is insufficient, requiring language- and dataset-aware sampling to prevent head-language dominance.mHuBERT-147 applies multilingual up-sampling and batching across 147 languages, motivating group-aware reweighting during pre-training and downstream adaptation.
- Low-resource data: Tail-language adaptation is constrained by scarce and noisy transcribed speech, motivating automated corpus construction and refinement pipelines.
- Corpus construction and data quality: GigaSpeech 2 combines Whisper transcription, MMS forced alignment, and iterative refinement to improve pseudo-labels, while OWSM studies heterogeneous sources and data cleaning.These efforts address both low-resource corpus creation and the effects of data heterogeneity on speech-to-text foundation models.
- Controlled encoder comparison: Selecting an open multilingual encoder is non-trivial because training-data mixtures are undisclosed and ablations are limited, motivating controlled comparisons under identical adaptation settings.The comparison uses two recent encoders trained on approximately 3–5M hours and holds the fine-tuning recipe, weighting strategy, and decoding configuration constant.
3. Method
GigaAM Multilingual uses a HuBERT-style masked-unit objective with paired 600M-parameter Conformer teacher and student encoders. Its pipeline balances multilingual pre-training data and fine-tunes with shared-vocabulary CTC on diverse data for Central Asian languages.
- Pre-training objective: The model uses HuBERT-style masked unit prediction with 600M-parameter, 24-layer Conformer teacher and student encoders operating at 25 Hz.Both encoders use 1024-dimensional hidden states and Rotary Position Embeddings.
- Pre-training objective: Teacher representations are clustered into 1,000 K-means centroids, whose nearest-centroid assignments provide discrete prediction targets.The student predicts these teacher-derived labels after contiguous spans covering 40% of input frames are masked.
- Data balancing: The 2-million-hour corpus is segmented into speech-only fragments, language-identified with MMS LID 4017, and organized using clustered language co-occurrence patterns.The balancing strategy avoids directly estimating weights for all 70 languages, which is difficult for low-resource languages.
- Data balancing: Five language clusters support pre-training balancing, while recordings below either 0.7 majority-vote or 0.7 mean-segment-confidence threshold are treated as uncertain.Low-weight edges and low-degree vertices are pruned before clustering.
- Fine-tuning: Fine-tuning uses CTC with a shared character vocabulary across English, Russian, Kazakh, Kyrgyz, and Uzbek, combining open-source, internal, synthetic, and weakly supervised data.Synthetic Kazakh and Kyrgyz speech uses over 100 TTS voices and ASR-based CER filtering; forced-alignment segments receive analogous ASR-based filtering.
4. Experiments
Experiments evaluate GigaAM across public and in-the-wild datasets, ablate pre-training and fine-tuning sampling, and compare encoders under matched CTC adaptation. Cluster balancing improves target-language WER, domain-aware fine-tuning benefits spontaneous speech, and GigaAM outperforms larger multilingual baselines in controlled comparisons.
- Evaluation setup: Evaluation uses Common Voice, FLEURS, and internal crowdsourced in-the-wild datasets, with utterances longer than 30 seconds and digit-containing references excluded.References and outputs are normalized before WER computation; English additionally canonicalizes British/American spelling variants.
- Pre-training sampling ablation: From E0 to E2, Kyrgyz WER improves from 9.4 to 8.5 and Uzbek WER from 10.5 to 9.7, while Russian changes from 4.6 to 4.7 and English from 14.4 to 15.4.E1 and E2 progressively shift pre-training weight toward the Central Asian cluster; E2 is adopted for the final model and public release.
- Final recipe: The final released model combines the E2 pre-trained encoder with domain-aware sampling for fine-tuning.This recipe follows the reported pre-training and fine-tuning sampling findings.
- Fine-tuning sampling ablation: Language-balanced sampling improves tail-language performance with only marginal English regression, while domain-aware sampling most benefits Kazakh and Kyrgyz on the Internal spontaneous-speech split.Domain-aware sampling regulates sub-domain composition and prevents large synthetic subsets from dominating the training curriculum.
- Encoder comparison: 12.2% average WER is achieved by the 240M GigaAM encoder versus 14.1% for Whisper and 16.6% for Omnilingual under matched CTC fine-tuning, despite its smaller size.Both GigaAM variants outperform Whisper and Omnilingual in the controlled setup; the 240M model trails the larger baselines only on English.
5. Conclusion
The study presents GigaAM Multilingual for high-quality ASR in underrepresented languages under severe data imbalance. It finds that balancing pre-training data and domain-aware fine-tuning are critical for preventing head-language dominance.
- The study targets high-quality ASR for underrepresented languages facing severe data imbalance.
- GigaAM Multilingual is a Conformer encoder pre-trained on 2M hours of audio with a cluster-aware sampling strategy.
- Explicit data balancing during pre-training and domain-aware sampling during fine-tuning are critical to prevent head-language dominance.