Source-linked AI summary
Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST
Monica Sekoyan, Nithin Rao Koluguri, Nune Tadevosyan, Piotr Zelasko, Travis Bartley, Nikolay Karpov, Jagadeesh Balam, Boris Ginsburg
TL;DR
The paper addresses the need for multilingual ASR and AST systems that balance accuracy, efficiency, and scale. It develops Canary-1B-v2 and Parakeet-TDT-0.6B-v3 through large-scale data curation, multi-stage training, efficient encoder designs, and timestamp alignment. The resulting models achieve competitive or superior ASR and AST performance with high efficiency, while experiments reveal distinct scaling behavior for FastConformer and nGPT encoders.
Problem
Multilingual ASR and speech translation systems can require large resources and run slowly, creating a trade-off between accuracy, size, and speed.
Method
The paper develops multilingual models using a 1.7M-hour dataset, dynamic data balancing, FastConformer and nGPT encoder studies, and NFA-based timestamp generation.
Results
Canary-1B-v2 achieves state-of-the-art ASR and AST performance with high efficiency, while FastConformer excels after multi-stage training and nGPT generalizes strongly in large-scale single-stage training.
Takeaways & Limitations
The released models extend efficient multilingual ASR and AST to 25 languages, with Parakeet-TDT-0.6B-v3 targeting high-quality ASR in resource-limited settings.
Takeaways & Limitations
The training data are imbalanced, with ASR and En→X dominating while X→En translation data are comparatively limited.
Abstract
from arXiv · showhide
This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 languages primarily European. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10x faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.
1 INTRODUCTION
The report targets the trade-off between accuracy, model size, and inference speed in multilingual ASR and speech translation. It introduces Canary-1B-v2 and Parakeet-TDT-0.6B-v3 with large-scale data, multi-stage training, efficient architecture, timestamp generation, and competitive performance.
- The models address the resource and speed demands of large encoder-decoder speech systems while retaining strong performance.
- Canary-1B-v2 is a multilingual, multi-task model designed for robust ASR and AST with high efficiency.
- A two-stage pre-training regimen and high-quality fine-tuning stage use dynamic weight scheduling to address multilingual data imbalance.
- 1.7 million hours of data cover 25 languages through pseudo-labeled and human-annotated speech, with non-speech audio added for robustness.
- FastConformer, a unified BPE tokenizer, and NFA with an auxiliary CTC model provide efficient processing and segment-level timestamps for ASR and AST.
- Canary-1B-v2 achieves competitive or superior benchmark results to much larger models while providing significantly faster inference.
2 ARCHITECTURE
The architecture combines speech-specific efficiency with experiments in normalized GPT encoders and positional encodings. FastConformer reduces encoder cost, while nGPT introduces normalized Transformer components and alternative positional-bias designs for speech.
- 2 ARCHITECTURE: The broader encoder-decoder design supports speech recognition, speech translation, and decoder strategies ranging from specialized Transformers to LLM-based systems.
- 2.1 ENCODER: Speech encoder design governs scalability and efficiency because full self-attention is a major bottleneck for long audio sequences.
- 2.1.1 FASTCONFORMER: FastConformer uses aggressive subsampling and depthwise separable, lightweight convolutions to shorten sequences and reduce operations.
- 2.1.1 FASTCONFORMER: 2–3× inference speedup and greater memory efficiency are achieved while maintaining accuracy comparable to or better than the original Conformer.
- 2.1.2 NGPT: The nGPT encoder adapts normalized GPT layers with a conventional Transformer decoder to isolate the acoustic-domain effect of the encoder design.
- 2.1.2 NGPT: nGPT layers combine subsampling, RoPE-based multi-head attention, gated feed-forward modules, hyperspherical normalization, and residual connections.
- 2.1.3 POSITIONAL ENCODINGS: RoPE showed performance degradation in very long-context ASR and ST, motivating a symmetric ALiBi variant that treats left and right context equally.
3 DATA
The models use a 1.7M-hour multilingual dataset assembled from pseudo-labeled and human-annotated sources, with balancing strategies addressing task, language, and corpus imbalance. The data also includes non-speech audio and supports 25 European languages, while VoxPopuli remains necessary for low-resource-language coverage despite domain limitations.
- Training data sources: 227,000 hours of NeMo ASR Set 3.0 provide human-labeled coverage for ASR, AST X→En, and En→X across all 25 languages.NeMo represents 13% of the total training dataset and supplements the largely pseudo-labeled Granary data.
- Task and language imbalance: ASR and En→X dominate the training set, leaving X→En comparatively limited and motivating mitigation methods for task imbalance.The imbalance is summarized in Figure 4 and addressed through the paper’s balancing procedures.
- Robustness data: 36,000 hours of non-speech audio paired with empty targets teach the models not to generate transcriptions for non-speech content across language-pair tasks.Source-target pairs are randomly assigned to provide coverage for X→X, En→X, and X→En combinations.
- Training data sources: 1.7M hours combine Granary, NeMo ASR Set 3.0, and supplementary En→X data across 25 languages and ASR/AST tasks.Granary supplies pseudo-labeled multilingual data, NeMo contributes human-labeled data, and the supplementary set adds about 480,000 hours of English-to-non-English pairs.
- Data limitations: VoxPopuli exceeds 95% of ASR data for most non-high-resource languages, but its parliamentary register and mostly 30-second segments limit diversity.Removing VoxPopuli would eliminate 15 languages from the 25-language set, so the authors reserve resegmentation for future work.
- Task and language imbalance: The two-tier sampling policy first balances corpora within each language, then balances languages across training entries, including distinct ASR and AST directions.Corpus weights use α to control high-resource versus low-resource corpus trade-offs, while language weights use β for cross-language balancing.
4 TRAINING
The models use staged multilingual training, unified tokenization, and data-balancing strategies to manage scale, task diversity, and corpus imbalance. Fine-tuning and weight scheduling improve selected ASR and X→En results, while the two-stage setup supports more efficient experimentation.
- Training strategy: Two-stage pre-training separates broad foundation learning from task-oriented fine-tuning, which adapts the model to practical tasks and data challenges.The paper describes pre-training on broad data followed by fine-tuning on smaller, task-oriented labeled data.
- Tokenizer training: A unified tokenizer trained across 25 languages outperformed concatenated tokenizers and supports natural code-switching.The tokenizer experiments also found that larger vocabularies improved downstream ASR and AST performance.
- Pre-training comparison: The two-stage checkpoint required around 100,000 additional full-dataset steps after foundation training, versus 250,000 steps for each single-stage trial.This reduced experimentation cost by allowing weighting schemes and data compositions to vary during the second stage.
- Fine-tuning results: 6 absolute COMET points improved X→En performance from 73.23 to 79.30, while ASR WER fell 25% relatively from 11.03% to 8.40% after rebalanced fine-tuning.En→X performance remained stable because its original pre-training distribution was already well balanced.
- Fine-tuning results: Weight scheduling produced a 1.5% absolute WER gain on spontaneous CoVoST2 ASR and was selected for the released primary model.The authors associate the gain with preserving pre-trained knowledge during fine-tuning on cleaner data.
5 TIMESTAMPS
The timestamp pipeline uses NeMo Forced Aligner with an auxiliary CTC model rather than relying solely on attention alignments or predictive timestamp tokens. It produces reliable segment-level timestamps, especially for the European-language evaluation scope, while word-level AST timestamps remain constrained by non-monotonic translation alignment.
- Alignment approaches: Attention-based encoder-decoder alignments are soft and can be non-monotonic, motivating post-processing methods such as Dynamic Time Warping.DTW finds a high-scoring alignment path between output tokens and input audio frames.
- Predictive timestamp generation: Predictive timestamp generation shifts alignment into model training but requires costly timestamp data curation and an additional task-balancing burden.These challenges can negatively affect transcription accuracy if the training regimen is not carefully tuned.
- Forced alignment: NeMo Forced Aligner uses CTC log-probabilities and Viterbi decoding to generate token-, word-, and segment-level timestamps.Canary-1B-v2 pairs NFA with an auxiliary multilingual Parakeet CTC model.
- ASR versus AST: ASR alignment is monotonic, whereas AST alignment is cross-lingual and non-monotonic because translated words lack direct sequential correspondence.The Parakeet CTC model is trained only for ASR and therefore lacks AST knowledge.
- Evaluation: The NFA pipeline produced high-quality segment-level AST timestamps, so the authors recommend segment-level rather than word-level timestamps for translation outputs.Word-level timestamps can be inaccurate because speech translation may reorder words and phrases.
- Limitations: Reliable timestamping was observed within the primarily European-language evaluation scope, while languages with more divergent syntax may introduce substantial word-order shifts.The authors identify more comprehensive quantitative AST timestamp evaluation as future work.
6 EVALUATION
Evaluation shows Canary-1B-v2 and Parakeet-TDT-0.6B-v3 combine strong multilingual speech performance with substantially higher inference efficiency than larger baselines. Results also reveal task- and training-dependent differences between nGPT and FastConformer, plus context-length and positional-encoding trade-offs.
- English ASR: RTFx 749 makes Canary-1B-v2 approximately 7–10 times faster than evaluated models while outperforming Whisper-large-v3 on official average WER.The comparison uses Hugging Face Open ASR Leaderboard datasets and the official evaluation repository.
- English ASR: RTFx = 3332.74 gives Parakeet-TDT-0.6B-v3 the highest throughput, with 6.32% average WER within 0.18 absolute points of Phi-4-multimodal-instruct.This corresponds to approximately 54× faster inference than Phi-4-multimodal-instruct.
- Multilingual ASR: On the full 24-language evaluation, Canary-1B-v2 reaches around 8.1% WER, beating Whisper-large-v3 at 9.9% and remaining slightly behind seamless-m4t-v2-large at 7.2%.On the common six-language subset, Canary-1B-v2 reaches 5.2% WER versus 5.8% for Whisper-large-v3 and 5.3% for seamless-m4t-v2-large.
- Multilingual ASR: Parakeet-TDT-0.6B-v3 averages 9.7% WER across 24 languages, edging past Whisper-large-v3 at 9.9% and matching seamless-m4t-v2-large at 5.3% on common languages.The common-language result is 5.3% WER for Parakeet-TDT-0.6B-v3 and seamless-m4t-v2-large, versus 5.8% for Whisper-large-v3.
- Robustness: Canary-1B-v2 improves from 2.18% to 2.01% WER as SNR decreases from 100 to 25, while Parakeet-TDT-0.6B-v3 reaches 12.21% versus 19.38% for Canary-1B-v2 at −5 dB.Parakeet-TDT-0.6B-v3 remains between 1.92–1.96% WER from 100 to 25 dB.
- X→En performance: Canary-1B-v2 achieves 79.28 COMET on FLEURS and 78.14 on CoVoST2, outperforming Whisper-large-v3 and seamless-m4t-medium while remaining close to seamless-m4t-v2-large.On the common-language setting, it reaches 82.41 COMET on FLEURS and 79.07 on CoVoST.
- Encoder comparison: Scaling nGPT from 1B to 3B parameters improves generalization, especially on X→EN translation, whereas FastConformer surpasses nGPT across tasks after sufficient multi-stage fine-tuning.The results characterize nGPT as data-hungry and FastConformer as more responsive to fine-tuning and low-data scenarios.
- Long-context evaluation: ALiBi is more effective than RoPE for long-context ASR, while RoPE is more effective for AST, reflecting task-dependent positional-encoding behavior.Longer contexts degrade stability and accuracy, motivating positional-encoding changes and parallel chunking; parallel chunking lowers WER and raises RTFx over sequential chunking.
7 CONCLUSIONS
The report concludes that Canary-1B-v2 and Parakeet-TDT-0.6B-v3 deliver efficient multilingual ASR and AST through large-scale data curation, training strategies, and architectural choices. Its comparisons identify complementary strengths of FastConformer and nGPT, while NFA supplies segment-level timestamps and Parakeet extends ASR to resource-limited settings.
- Overall conclusion: Canary-1B-v2 and Parakeet-TDT-0.6B-v3 achieve state-of-the-art ASR and AST performance with high efficiency.The development combines architecture, data, and training-strategy exploration.
- Encoder comparison: FastConformer excels after multi-stage training and fine-tuning, while nGPT generalizes faster in large-scale single-stage training, especially on challenging translation tasks.The comparison is presented as a trade-off between fine-tuning gains and data-hungry scaling.
- Positional encodings: ALiBi is more effective for long-form ASR, whereas RoPE is more suitable for AST.The conclusion reports task-specific positional-encoding preferences.
- Additional contributions: NeMo Forced Aligner generates accurate segment-level timestamps, and Parakeet-TDT-0.6B-v3 extends high-quality ASR to resource-limited settings.The timestamp method is described as robust and the companion model supports the same multilingual scope.
APPENDIX
The appendix identifies Table 9 as a summary of training-data hours organized by corpus and task.
- Training data: Table 9 reports training-data hours by corpus and the tasks involved.The table summarizes corpus-level data allocation across tasks.
A TRAINING DATA LANGUAGE, CORPORA AND TASK DECOMPOSITION
The appendix identifies Table 10 as a language-, corpus-, and task-level breakdown of training data, with NeMo denoting NeMo ASR Set 3.0.
- Training data: Table 10 organizes training-data hours by language, corpus, and task.The table legend defines NeMo as NeMo ASR Set 3.0.
B EVALUATION: ASR PERFORMANCE
ASR performance is evaluated with per-language WER on the FLEURS, MLS, and CoVoST2 benchmarks.
- Per-language WER (%) results are reported on the FLEURS benchmark.
- Per-language WER (%) results are reported on the MLS benchmark.
- Per-language WER (%) results are reported on the CoVoST2 benchmark.
C EVALUATION: AST (X→EN) PERFORMANCE
AST performance for X→EN is evaluated using per-language COMET and BLEU scores on the FLEURS and CoVoST2 benchmarks.
- Per-language COMET (C) and BLEU (B) scores (%) are reported on the FLEURS benchmark.
- Per-language COMET (C) and BLEU (B) scores (%) are reported on the CoVoST2 benchmark.
D EVALUATION: AST (EN→X) PERFORMANCE
AST performance for EN→X is evaluated using per-language COMET and BLEU scores on FLEURS and CoVoST2, including a new FLEURS evaluation.
- Per-language COMET (C) and BLEU (B) scores (%) are reported on the FLEURS benchmark in a new evaluation.
- Per-language COMET (C) and BLEU (B) scores (%) are reported on the CoVoST2 benchmark.