Source-linked AI summary
OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models
Han Zhu, Lingxuan Ye, Wei Kang, Zengwei Yao, Liyong Guo, Fangjun Kuang, Zhifeng Han, Weiji Zhuang, Long Lin, Daniel Povey
TL;DR
Existing zero-shot TTS systems have limited language coverage, and discrete NAR systems can suffer from cascaded-pipeline drawbacks. OmniVoice directly maps text to multi-codebook acoustic tokens using a single-stage diffusion language model-style NAR architecture with full-codebook masking and LLM initialization. Trained on 581k hours across more than 600 languages, it reports state-of-the-art intelligibility, speaker similarity, and naturalness across Chinese, English, and multilingual benchmarks.
Problem
Zero-shot TTS systems generally support limited language sets, while cascaded discrete NAR pipelines introduce error propagation and acoustic information bottlenecks.
Method
OmniVoice directly maps text to multi-codebook acoustic tokens with a single-stage discrete diffusion NAR architecture, full-codebook random masking, and pre-trained LLM initialization.
Results
581k hours of open-source data support more than 600 languages, with state-of-the-art intelligibility, speaker similarity, and naturalness across benchmarks covering up to 102 languages.
Takeaways & Limitations
OmniVoice combines extensive multilingual coverage with multi-dimensional control for prompt denoising, voice design, and fine-grained paralinguistic and phonetic control.
Takeaways & Limitations
Open-source training data have inconsistent annotation and acoustic quality, and the model is not yet optimized for complex numeric sequences or mathematical patterns.
Abstract
from arXiv · showhide
We present OmniVoice, a massively multilingual zero-shot text-to-speech (TTS) model that scales to over 600 languages. At its core is a novel diffusion language model-style discrete non-autoregressive (NAR) architecture. Unlike conventional discrete NAR models that suffer from performance bottlenecks in complex two-stage (text-to-semantic-to-acoustic) pipelines, OmniVoice directly maps text to multi-codebook acoustic tokens. This simplified approach is facilitated by two key technical innovations: (1) a full-codebook random masking strategy for efficient training, and (2) initialization from a pre-trained LLM to ensure superior intelligibility. By leveraging a 581k-hour multilingual dataset curated entirely from open-source data, OmniVoice achieves the broadest language coverage to date and delivers state-of-the-art performance across Chinese, English, and diverse multilingual benchmarks. Our code and pre-trained models are publicly available at https://github.com/k2-fsa/OmniVoice.
1 Introduction
OmniVoice targets the limited language coverage and pipeline drawbacks of existing zero-shot TTS with a single-stage discrete NAR architecture. Its masking, LLM initialization, multilingual data, and controllability support broad coverage and strong multilingual performance.
- Motivation: Existing zero-shot TTS models often support only a limited set of languages, leaving many low-resource languages underserved.The paper frames broader language coverage as a technical and global speech-technology goal.
- Motivation: Discrete-token NAR systems typically use cascaded text-to-semantic-to-acoustic pipelines that introduce error propagation and acoustic information bottlenecks.Single-stage alternatives are presented as a way to bypass these limitations.
- OmniVoice: OmniVoice directly maps text to multi-codebook acoustic tokens with a discrete masked-diffusion objective and bidirectional Transformer.This streamlined single-stage design avoids the complexity of cascaded pipelines.
- Technical innovations: Full-codebook random masking stochastically masks all codebook layers, improving training efficiency and generative quality over per-layer masking.Per-layer masking optimizes only a sparse subset of the token matrix at each iteration.
- Technical innovations: LLM initialization transfers linguistic knowledge from pre-trained autoregressive LLMs to address intelligibility in single-stage discrete NAR TTS.The paper describes OmniVoice as the first NAR TTS model to successfully benefit from this initialization.
- Scale and evaluation: 581k hours of open-source multilingual data cover more than 600 languages, while evaluations report state-of-the-art intelligibility, speaker similarity, and naturalness across benchmarks covering up to 102 languages.The model also supports prompt denoising, speaker attribute-based voice design, and fine-grained paralinguistic and phonetic control.
2 Proposed Method
OmniVoice is a single-stage diffusion language model-style discrete NAR TTS system that directly maps text to multi-codebook acoustic tokens. Its design combines full-codebook masking, LLM initialization, multilingual data scaling, and controllability mechanisms.
- 2.1 Architecture: OmniVoice directly maps text to multi-codebook acoustic tokens with a bidirectional Transformer and discrete diffusion objective.The single-stage design avoids the conventional text-to-semantic-to-acoustic cascade.
- 2.1 Architecture: The model conditions on text tokens, prompt acoustic context, and partially masked target tokens to recover masked acoustic tokens.Text and acoustic embeddings are processed jointly, with separate prediction heads for each codebook.
- 2.1.1 Full-Codebook Random Masking for Training Efficiency: Full-codebook random masking independently masks tokens across all codebook layers, using an instance-level masking ratio sampled uniformly from 0 to 1.Unlike per-layer masking, the strategy computes loss across a larger portion of the token matrix.
- 2.1.1 Full-Codebook Random Masking for Training Efficiency: 50% of tokens are used for loss computation on average, C times more than with per-layer masking, accelerating convergence and improving generative quality.The increase follows from independently masking entries across the T × C acoustic token matrix.
- 2.1.2 LLM Initialization for Intelligibility: Pre-trained AR LLM weights initialize the bidirectional backbone, transferring linguistic knowledge to improve intelligibility in the single-stage NAR model.The backbone is structurally identical to standard AR LLMs, enabling direct weight transfer.
- 2.2 Multilingual Scaling: The model scales through a 581k-hour open-source corpus spanning more than 600 languages, language-level resampling, LLM subword tokenization, and multidimensional controllability.Resampling uses repetition factors for languages with less audio, while controllability covers acoustic, identity, and linguistic aspects.
3 Experimental Setup
OmniVoice is evaluated under bilingual and multilingual data configurations using iterative unmasking, script-aware duration estimation, four benchmarks, and objective and subjective speech metrics.
- Data configurations: OmniVoice is trained in bilingual and multilingual configurations, with the latter using a self-built 581k-hour dataset spanning more than 600 languages.The bilingual variant uses Chinese and English Emilia subsets for comparison, while prompt denoising is omitted to isolate architectural effects.
- Inference protocol: 32-step iterative unmasking uses a time-shifted schedule, with N = 32, τ = 0.1, and newly unmasked proportions defined by k_n = r_n − r_n−1.The schedule controls cumulative and per-step token unmasking during inference.
- Inference protocol: Unmasking positions are sampled from temperature-adjusted confidence scores, while token identities are assigned deterministically by argmax.A temperature T = 5 introduces stochasticity during position selection; layer penalties and classifier-free guidance are also applied.
- Inference protocol: Target duration is estimated from prompt duration using script-dependent character weights that reflect differences between writing systems such as CJK and Latin.The target duration scales with the ratio of total character weights.
- Evaluation benchmarks: Evaluation covers LibriSpeech-PC, Seed-TTS, MiniMax-Multilingual-24, and FLEURS-Multilingual-102, spanning English, Chinese, and multilingual settings up to 102 languages.FLEURS-Multilingual-102 is constructed from FLEURS development and test splits.
- Evaluation metrics: Metrics combine speaker similarity, intelligibility, naturalness, and human judgments through SIM-o, WER or CER, UTMOS, CMOS, and SMOS.ASR systems vary by language and benchmark, while CMOS measures relative quality and SMOS measures absolute speaker similarity.
4 Experimental Results
OmniVoice performs strongly across Chinese, English, and multilingual benchmarks, while ablations validate its masking, LLM initialization, and prompt-denoising designs. It also demonstrates low-resource intelligibility and efficient inference.
- Chinese and English Evaluation: OmniVoice-Emilia surpasses all NAR baselines trained on the same Emilia corpus.
- Chinese and English Evaluation: OmniVoice achieves competitive overall performance against unconstrained-data baselines, with particular advantages in speaker similarity and intelligibility.
- Multilingual Benchmarks: OmniVoice outperforms ElevenLabs Multilingual v2 and MiniMax-Speech in average SIM-o and WER on MiniMax-Multilingual-24.The reported Cantonese WER is attributed to Whisper ASR limitations; SenseVoice-Small yields 2.273%.
- Multilingual Benchmarks: 4.00% average CER on FLEURS-Multilingual-102 is comparable to ground truth, while many languages with less than 10 hours of training data achieve CER < 5%.The authors state that OmniVoice’s performance exceeds the measurement capability of existing ASR models.
- Key Design Ablations: Full-codebook random masking consistently outperforms MaskGCT and SoundStorm masking, while single-codebook loss causes significant performance degradation.
- Key Design Ablations: LLM initialization lowers WER relative to models without initialization, whereas prompt denoising raises UTMOS from 4.23 to 4.32 but lowers SIM-o from 0.697 to 0.668.
- Inference Speed: 0.0319 RTF with 16 inference steps and batch size 1 outperforms ZipVoice’s 0.0557 under the same configuration.Batch inference further reduces OmniVoice’s RTF to 0.022.
5 Conclusions
OmniVoice is a massively multilingual zero-shot TTS model supporting over 600 languages. It combines a single-stage discrete-token NAR architecture with full-codebook masking and LLM initialization, trained on 581k hours of open-source multilingual data.
- OmniVoice supports over 600 languages and achieves state-of-the-art performance on Chinese, English, and diverse multilingual benchmarks.
- The model directly maps text to multi-codebook acoustic tokens through a diffusion language model-style single-stage discrete-token NAR framework.
- Full-codebook random masking improves training efficiency, while LLM weight initialization addresses intelligibility in the streamlined architecture.
- Training uses a 581k-hour multilingual dataset curated exclusively from open-source resources.
A Complete List of Multilingual Training Data
OmniVoice was trained using a broad collection of multilingual speech datasets. The listed sources include general, language-specific, demographic, and speech-characteristic resources.
- The training data combines datasets including Emilia, LibriTTS, Common Voice, VoxBox, and GigaSpeech 2.
- The collection includes multilingual resources such as the Meta Omnilingual ASR Corpus, FLEURS, EuroSpeech, and IndicVoices-R.
- Additional datasets cover language- and speech-specific material, including Wenetspeech variants, AISHELL6-whisper, NonVerbalSpeech-38K, SeniorTalk, and ChildMandarin.
B Results of Different Inference Steps
The multilingual OmniVoice model is evaluated with different inference-step settings on English and Chinese benchmarks. Table 9 reports the corresponding objective evaluation results.
- The multilingual OmniVoice model is evaluated using different numbers of inference steps.
- English and Chinese benchmarks are used for the inference-step evaluation.
- Table 9 presents objective evaluation results for OmniVoice across inference-step settings.
C Detailed Results on FLEURS-Multilingual-102
OmniVoice is assessed on FLEURS-Multilingual-102 using per-language character error rates. Figure 4 compares ground-truth and OmniVoice CERs across languages.
- Per-language character error rates are reported for FLEURS-Multilingual-102.
- Figure 4 compares ground-truth CERs with OmniVoice CERs across languages.
- Table 10 is titled as the per-language CER comparison between ground-truth speech and OmniVoice on FLEURS-Multilingual-102.
D Supported Languages of OmniVoice
OmniVoice supports 646 languages using 581k hours of training data. The supported-language table lists each language’s OmniVoice ID, ISO 639-3 code, and training duration.
- Supported language coverage: 646 languages are supported with 581k hours of training data.The complete language inventory and per-language training durations are provided in Table 11.
- Language inventory: Table 11 records each language’s OmniVoice language ID, ISO 639-3 code, and training-data duration.
- Language inventory: The table includes languages such as Kalabari, Kannada, Kazakh, Khmer, and Khowar with their corresponding identifiers and hours.
- Language inventory: The inventory also includes Uighur, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, and other languages with recorded training durations.
E Limitations and Future Works
OmniVoice’s performance and flexibility remain constrained by the quality and diversity of its open-source training data. Future work includes better instruction data, text normalization, and faster inference for discrete-space NAR TTS.
- Data quality: Open-source training data has inconsistent annotation and acoustic quality, leaving room for improvement with more curated data.
- Instruction following: Instruction-following flexibility is constrained by the diversity and quality of existing instruction-tuning data.Higher-quality instruction datasets could improve voice design and customization capabilities.
- Text processing: OmniVoice is not yet optimized for complex numeric sequences or mathematical patterns.An external text-normalization front-end is identified as a way to strengthen performance in these scenarios.
- Inference efficiency: Discrete-space NAR TTS lacks an approach comparable to continuous-space flow-distillation methods for drastically reducing inference steps.Developing inference-step reduction strategies is proposed as future work.
F Ethics Statements
The work is intended for academic research only, while the model’s high speaker-similarity synthesis capability creates potential misuse risks. Illegal use is strictly prohibited.
- Research use: The work is intended only for academic research purposes.
- Misuse risks: High speaker-similarity speech synthesis creates potential risks of misuse.
- Prohibited use: Any illegal use of the model is strictly prohibited.