PaperlayerView exact revision on arXiv

Source-linked AI summary

BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models

Shivam Singh, Aditya Yadavalli, Catherine Arnett, Alex Warstadt

arXiv:2609.09554v1cs.CL

TL;DR

BuzzASR addresses poor ASR performance in languages disadvantaged by highly multilingual models. It adapts Whisper with language-specific fine-tuning, tokenizer replacement, and text-only training across 102 languages. The models outperform Whisper on most evaluated languages and provide strong open-source performance, while monolingual deployment has narrower coverage and higher multi-language resource costs.

  • Problem

    Multilingual ASR models often perform poorly for languages that are underrepresented in their training data, despite available language resources.

  • Method

    BuzzASR creates language-specific Whisper models using simple speech fine-tuning or a fuller pipeline with tokenizer replacement, text-only data, and speech–text multitask fine-tuning.

  • Results

    BuzzASR generally outperforms Whisper-large-v3 and offers competitive open-source performance for most covered languages when larger Omni-7B models are unsuitable to deploy.

  • Takeaways & Limitations

    Language-specialized fine-tuning can provide strong monolingual ASR performance and lower sequence lengths, inference latency, and compute for covered languages.

  • Takeaways & Limitations

    The evaluation does not cover different domains or dialects, and supplementary analysis indicates that out-of-domain generalization may be challenging.

Abstract

from arXiv · show

We introduce BuzzASR, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages. Large end-to-end Transformer-based ASR models such as Whisper have revolutionized ASR, but most prominent models are highly multilingual. As a result, these models often perform poorly on languages less well-represented in their training set. While it has long been known that effective language adaptation can be achieved through simple fine-tuning on monolingual data, this strategy has only been applied to a small number of languages. We massively scale up this simple approach to 102 languages covered in the FLEURS dataset, while also implementing a more complex language adaptation strategy that integrates monolingual tokenizer replacement and data augmentation using text-only fine-tuning. BuzzASR models outperform Whisper-large-v3 on 77 out of 102 languages, reducing character error rates (CER) by a factor of over 2.8 on average. Our models achieve state-of-the-art CER among open-source systems on 27 of 102 languages on the combined FLEURS and Common Voice test set. Our tokenizer replacement strategy yields an average 3.3x improvement in compression rate (characters per token) over Whisper's multilingual BPE, with gains of up to 21.7x. We release all models, code, and detailed results: https://lemn-lab.github.io/buzz-asr

1 Introduction

ASR remains inaccurate for most widely spoken languages despite available multilingual models and substantial speech resources. BuzzASR addresses this gap by adapting Whisper to individual languages.

  • Motivation: ASR models are highly inaccurate for all but a small fraction of the 200 most widely spoken languages.Datasets with at least 10 hours of transcribed speech exist for at least 1,000 languages.
  • Motivation: At least 1,000 languages have ASR datasets containing 10 or more hours of transcribed speech.The paper argues that limited performance is not explained solely by a lack of data.
  • Study scope: Figure 1 compares the best BuzzASR model for each language against Whisper across 102 languages, excluding some outliers for presentation.The comparison evaluates language-specific adaptation at broad multilingual scale.
  • Prior evidence: Monolingual domain adaptation can substantially improve low-resource ASR with less than 10 hours of monolingual data.Earlier work cut word error rates by half against contemporary state-of-the-art systems.

1 Tokenizer Replacement

BuzzASR scales language-specific Whisper adaptation to 102 languages and combines simple fine-tuning with tokenizer and text-data adaptations. The resulting models improve substantially over Whisper, especially where the base model is weakest.

  • Results: 102 language-specific Whisper models are evaluated, with BuzzASR outperforming Whisper-large-v3 on normalized CER for 77 languages.The median CER reduction is 2.18× and the mean reduction is 2.84×.
  • Results: 3.45× is the median CER reduction in Whisper’s 51 worst languages.For ten languages, the best fine-tuned model reduces CER by more than 6×.
  • Adaptation: Tokenizer replacement and monolingual text fine-tuning extend the basic adaptation strategy beyond speech-only fine-tuning.The tokenizer addresses inefficient multilingual segmentation, while text fine-tuning adapts the decoder using additional monolingual text.
  • Results: 27 languages achieve state-of-the-art CER among open-source systems on the combined FLEURS and Common Voice evaluation.The released suite is intended to provide language-specific open-source alternatives at scale.

2 Related Work

ASR research has progressed through multilingual models and expanded speech corpora, but multilingual scaling can disadvantage individual languages. BuzzASR builds on this literature by emphasizing language-specific adaptation.

  • Research trajectory: Earlier low-resource ASR work often developed monolingual or limited-multilingual systems for individual languages before multilingual pretraining became prominent.This progression motivates combining broad pretrained models with language-specific adaptation.
  • Multilingual ASR: Whisper and related self-supervised and multilingual systems expanded ASR coverage, scale, and performance across languages and conditions.These developments also motivated fine-tuning, parameter-efficient, prompt-based, and multitask adaptation strategies.
  • Datasets: FLEURS covers 102 languages, while Common Voice and newer initiatives broaden speech resources for underrepresented languages and regions.Omnilingual provides transcribed data for 348 traditionally under-resourced languages.
  • Limitations of multilingualism: Multilingual training can benefit from crosslingual transfer but may also suffer from the curse of multilinguality.The cited literature associates degraded performance with limited capacity and interference from unrelated languages.

3 Adaptation strategies

BuzzASR adapts Whisper through either speech-only fine-tuning or a fuller pipeline combining tokenizer replacement, text-only learning, and speech recognition fine-tuning. The design targets language-specific vocabulary, syntax, and limited speech–text data.

  • Simple fine-tuning: Simple fine-tuning updates the entire Whisper model on aligned monolingual speech–text data using the ASR objective.This pipeline consists only of ASR fine-tuning.
  • Full fine-tuning: Full fine-tuning replaces Whisper’s multilingual tokenizer and trains on a mixture of monolingual text-only and speech–text data.The pipeline combines tokenizer replacement, multitask learning, and final speech recognition fine-tuning.
  • Tokenizer replacement: Whisper’s English-skewed 51,865-token BPE is poorly suited to underrepresented and non-Latin-script languages.Replacement is intended to improve language-specific adaptation efficiency.
  • Text fine-tuning: Text-only fine-tuning trains the decoder with next-token prediction while random encoder embeddings are supplied and the encoder is frozen.This uses abundant monolingual text to adapt the decoder without requiring aligned speech.
  • Motivation: Text-only training may improve language-specific syntactic priors and prevent newly introduced tokenizer tokens from being under-trained.These motivations concern decoder specialization and vocabulary adaptation rather than acoustic encoder training.
  • Multitask learning: Multitask fine-tuning mixes aligned ASR and text-only examples, with α controlling the ASR proportion.α = 1.0 gives pure ASR fine-tuning, while α = 0 gives pure text fine-tuning.

4 Experiments

BuzzASR evaluates two language-adaptation strategies across 102 languages using FLEURS and Common Voice data, with CER as the primary metric. The experiments compare fine-tuning configurations and systems against Whisper and other multilingual ASR baselines.

  • Experimental design: 612 fine-tuned models are produced from three hyperparameter runs for each language and fine-tuning strategy.Learning-rate configurations are selected per language using FLEURS development-set WER.
  • Datasets: 102 languages are evaluated using FLEURS and Common Voice, with Common Voice available for 77 languages.FLEURS supplies approximately 10 hours per language; Common Voice training data is capped at 90 hours per language.
  • Fine-tuning strategies: Simple fine-tuning trains the full Whisper encoder-decoder on aligned speech-text data, whereas full fine-tuning adds tokenizer replacement and text-only data.Multitask experiments interleave speech-text and text-only samples in one fine-tuning stage.
  • Evaluation: CER is the primary metric because word length varies widely across languages and makes WER less comparable.Both CER and WER are reported, and scores are computed with jiwer.
  • Evaluation: The evaluation combines FLEURS and Common Voice test data, while the 25 languages absent from Common Voice use FLEURS results only.Languages with fewer than 2,000 Common Voice test utterances use all available samples.
  • Baselines: The comparison includes Whisper-large-v3, Omnilingual 1B and 7B, MMS, Qwen3-ASR 1.7B, Cohere Transcribe 2B, and other baselines across 102 languages.Whisper supports 83 target languages directly; the remaining 19 use prompts based on closely related supported languages.

5 Results

Across 102 languages, BuzzASR generally improves CER over Whisper and reaches state-of-the-art open-source performance on 27 languages. Tokenizer adaptation substantially increases compression rates, while supplementary experiments identify method-specific trade-offs and domain limitations.

  • 5.1 ASR Results: 77 languages outperform Whisper-large-v3, with CER reduced by a factor of 2.8 on average.Improvements are strongest where Whisper’s baseline CER is high, while several high-resource languages show little or no improvement.
  • 5.1 ASR Results: The median CER reduction factor is 2.2 across 102 languages, with reductions exceeding 6× in the strongest cases.The reduction factor is defined as Whisper zero-shot CER divided by the lower CER from simple or full fine-tuning.
  • 5.1 ASR Results: Simple fine-tuning improves CER for 65 languages, whereas full fine-tuning is better for 69 languages.The results indicate that neither strategy is optimal for every language; full fine-tuning is more suitable when base-model error is high.
  • 5.2 Tokenizer-quality metrics: Tokenizer compression rises from 2.38 to 5.04 characters per token at the median, a 2.18× gain over Whisper.The relationship between compression gains and CER reductions is positive but moderate, with larger gains generally observed for non-Latin scripts.
  • 5.3 Supplementary Results: Higher proportions of ASR training are better for multitask fine-tuning, with 50% ASR data optimal among the compared settings.Increasing text-only data yields modest improvements.
  • 5.3 Supplementary Results: Warm-start tokenizer replacement performs best among the compared tokenizer-adaptation strategies.The comparison includes random initialization of language-specific tokens and appending new tokens to the original merge list.
  • 5.3 Supplementary Results: Out-of-domain evaluation hurts performance slightly, suggesting that some observed gains reflect improved train–test domain alignment.The authors qualify this as only part of the performance difference, rather than attributing all gains to domain alignment.

6 Discussion and Conclusion

BuzzASR provides language-specialized ASR for dozens of languages and demonstrates that large-scale language-specialized fine-tuning can improve monolingual performance. The discussion highlights tokenizer adaptation, deployment trade-offs, and several directions for extending the approach.

  • BuzzASR can be deployed in existing Whisper pipelines and generally outperforms the similarly sized Omni-1B, though Omni-7B performs better on most languages.
  • Language-specialized fine-tuning is effective at scale across the BuzzASR model suite.The authors motivate similar approaches for other ASR, text, and multimodal architectures.
  • Language-adapted tokenizers improve ASR disproportionately when multilingual tokenizers are ill-suited and reduce inference latency and compute through shorter sequences.
  • Future Work: Further hyperparameter tuning and multitask learning could improve performance and better exploit abundant text-only data.
  • Future Work: The approach could be extended with established ASR optimizations, including noise or text-to-speech augmentation, monolingual language-model interpolation, and joint training on related languages.
  • Future Work: The suite could expand beyond its current language coverage, particularly to under-represented and under-studied languages.

Limitations

The limitations concern model coverage, training resources, evaluation scope, deployment trade-offs, and reliance on Whisper as the starting architecture.

  • The experiments fine-tune only Whisper, while future work could explore other starting models.
  • The study uses only a small fraction of existing ASR data, limiting potential performance gains from additional training data.The authors capped training data at 100 hours primarily because of computational resource limitations.
  • Evaluation uses FLEURS and Common Voice test splits and does not assess different domains or dialects.The paper notes that out-of-domain generalization may challenge adapted ASR models.
  • Monolingual models may be unsuitable when the language is unknown or multiple languages are required, because deploying several models requires substantially more compute than one multilingual model.
  • The implementation uses related-language prompting for 19 languages that Whisper does not natively support.
  • Training varies across configurations, random seeds, checkpoint criteria, and early-stopping settings.
  • All experiments use 8×A100-80GB GPUs, reflecting substantial computational requirements.

B.2 Hyperparameter Sweep Results

The experiments compare training-data mixtures, tokenizer replacement strategies, domain settings, bilingual versus monolingual adaptation, and multitask ablations. Across these comparisons, the selected main configuration uses 50% ASR batches, 500k text lines, warm-start tokenizer replacement, and monolingual fine-tuning.

  • Multitask data mixture: 50% ASR batches was optimal among the evaluated mixing ratios, while increasing text data from 500k to 2.5M lines yielded little or no improvement.The experiments also tested 10% and 20% ASR proportions.
  • Tokenizer replacement: Warm-start tokenizer replacement achieved median CER 7.4, versus 15.1 for random initialization and 15.7 for the extended strategy.All three strategies first train a new monolingual BPE tokenizer.
  • Domain evaluation: On the combined test set, models trained on FLEURS alone and models trained on FLEURS plus Common Voice showed little difference.FLEURS-only training had a 1.15× disadvantage out of domain but a 1.33× advantage in domain.
  • Ablations: Full fine-tuning was more effective than simple fine-tuning or multitask fine-tuning without tokenizer replacement, while simple and multitask-only fine-tuning performed similarly.The ablation used additional multitask runs without tokenizer adaptation.
  • Bilingual adaptation: Monolingual simple fine-tuning nearly always outperformed bilingual fine-tuning, with median improvements of 1.25× for unrelated and 1.24× for related language pairs.The comparison covered 10 language pairs, including five related and five unrelated pairs.

G Ablation: Multitask Learning Without Tokenizer Replacement

This ablation separates the effects of multitask fine-tuning and tokenizer replacement. Full fine-tuning performs best in aggregate, while multitask-only training is similar to simple fine-tuning.

  • Ablation result: Full fine-tuning was more effective in aggregate than either simple fine-tuning or multitask fine-tuning without tokenizer replacement.The multitask-only condition omits tokenizer adaptation.
  • Scope: Tokenizer adaptation without multitask fine-tuning was not investigated because the authors considered it likely to fail without additional text-only data.This is an explicitly stated scope boundary of the ablation.
  • Ablation result: Simple fine-tuning and multitask-only fine-tuning showed little difference in performance.The comparison included all simple and full fine-tuning runs against the multitask-only approach.
  • Latency analysis: Latency analysis measures wall-clock latency, tokens per second, and generated-token counts across Whisper zero-shot, simple fine-tuning, and full fine-tuning.The measurements use sampled FLEURS and, where available, Common Voice test utterances.
  • Latency analysis: Full fine-tuning produced a global median wall-clock speedup of approximately 1.6×, while median per-token throughput remained approximately 73 tokens per second.Speedups were attributed to token-count compression rather than faster per-token decoding.

I Per-Language Tokenizer Metrics

BuzzASR’s per-language tokenizers generally improve compression and vocabulary use over Whisper’s multilingual tokenizer, especially for non-Latin scripts. These changes correspond to lower sequence lengths and broad latency gains, with Sino-Japanese scripts as an exception.

  • Compression: BuzzASR tokenizers generally achieve around 5 characters per token, whereas Whisper achieves less than 2 for non-Latin and Cyrillic scripts.Sino-Japanese scripts are the main exception because they are largely logographic.
  • Vocabulary utilization: BuzzASR vocabulary utilization is generally above 20%, while Whisper’s is below 10% and often below 3% for Arabic, Brahmic, and rarer scripts.The table compares per-language BPE metrics against Whisper-large-v3’s multilingual BPE.
  • Boundary behavior: BuzzASR boundary crossing is below 0.1% for most languages, whereas Whisper reaches that threshold mainly for Latin and Cyrillic scripts.Boundary crossing measures tokens containing an internal whitespace character across a word boundary.

J Full Evaluation Results

The evaluation reports full CER and WER results across languages using combined Common Voice and FLEURS tests when available. The tables compare BuzzASR configurations with Whisper-based and other open-source systems, using lower error as better.

  • CER results: Table 7 reports test-set CER percentages for all languages, using combined Common Voice and FLEURS data when available and FLEURS alone otherwise.Bold marks the best score within the Whisper group, and underlining marks the overall best score.
  • Systems compared: The compared systems include full fine-tuning, simple fine-tuning, multitask fine-tuning without tokenizer replacement, Whisper, and several other open-source ASR systems.The table headers list FFT, SFT, SFTMTL, Whisper, Omni, MMS, Cohere, and Qwen3 conditions.
  • WER results: Table 8 reports test-set WER percentages under the same language-dependent test-set construction.Lower WER is better, with bold and underline indicating within-group and overall best scores.
Loading 2609.09554v1…