Source-linked AI summary
Voxtral TTS
Mistral-AI, :, Alexander H. Liu, Alexis Tacnet, Andy Ehrenberg, Andy Lo, Chen-Yo Sun, Guillaume Lample, Henry Lagarde, Jean-Malo Delignon, Jaeyoung Kim, John Harvill, Khyathi Raghavi Chandu, Lorenzo Signoretti, Margaret Jennings, Patrick von Platen, Pavankumar Reddy Muddireddy, Rohin Arora, Sanchit Gandhi, Samuel Humeau, Soham Ghosh, Srijan Mishra, Van Phung, Abdelaziz Bounhar, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andrew Bai, Andrew Zhao, Angele Lenglemetz, Anmol Agarwal, Anton Eliseev, Antonia Calvi, Arjun Majumdar, Arthur Fournier, Artjom Joosen, Avi Sooriyarachchi, Aysenur Karaduman Utkur, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Benjamin Tibi, Bowen Yang, Charlotte Cronjäger, Clémence Lanfranchi, Connor Chen, Corentin Barreau, Corentin Sautier, Cyprien Courtot, Darius Dabert, Diego de las Casas, Elizaveta Demyanenko, Elliot Chane-Sane, Emmanuel Gottlob, Enguerrand Paquin, Etienne Goffinet, Fabien Niel, Faruk Ahmed, Federico Baldassarre, Gabrielle Berrada, Gaëtan Ecrepont, Gauthier Guinet, Genevieve Hayes, Georgii Novikov, Giada Pistilli, Guillaume Kunsch, Guillaume Martin, Guillaume Raille, Gunjan Dhanuka, Gunshi Gupta, Han Zhou, Harshil Shah, Hope McGovern, Hugo Thimonier, Indraneel Mukherjee, Irene Zhang, Jacques Sun, Jan Ludziejewski, Jason Rute, Jérémie Dentan, Joachim Studnia, Jonas Amar, Joséphine Delas, Josselin Somerville Roberts, Julien Tauran, Karmesh Yadav, Kartik Khandelwal, Kilian Tep, Kush Jain, Laurence Aitchison, Laurent Fainsin, Léonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Maarten Buyl, Manan Sharma, Marie Pellat, Mark Prins, Martin Alexandre, Mathieu Poirée, Mathieu Schmitt, Mathilde Guillaumin, Matthieu Dinot, Matthieu Futeral, Maxime Darrin, Maximilian Augustin, Mert Unsal, Mia Chiquier, Mikhail Biriuchinskii, Minh-Quang Pham, Mircea Lica, Morgane Rivière, Nathan Grinsztajn, Neha Gupta, Olivier Bousquet, Olivier Duchenne, Patricia Wang, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Philippe Pinel, Philomène Chagniot, Pierre Stock, Piotr Miłoś, Prateek Gupta, Pravesh Agrawal, Quentin Torroba, Ram Ramrakhya, Randall Isenhour, Rishi Shah, Romain Sauvestre, Roman Soletskyi, Rosalie Millner, Rupert Menneer, Sagar Vaze, Samuel Barry, Samuel Belkadi, Sandeep Subramanian, Sean Cha, Shashwat Verma, Siddhant Waghjale, Siddharth Gandhi, Simon Lepage, Sumukh Aithal, Szymon Antoniak, Tarun Kumar Vangani, Teven Le Scao, Théo Cachet, Theo Simon Sorg, Thibaut Lavril, Thomas Chabal, Thomas Foubert, Thomas Robert, Thomas Wang, Tim Lawson, Tom Bewley, Tom Edwards, Tyler Wang, Umar Jamil, Umberto Tomasini, Valeriia Nemychnikova, Vedant Nanda, Victor Jouault, Vincent Maladière, Vincent Pfister, Virgile Richard, Vladislav Bataev, Wassim Bouaziz, Wen-Ding Li, William Havard, William Marshall, Xinghui Li, Xingran Guo, Xinyu Yang, Yannic Neuhaus, Yassine El Ouahidi, Yassir Bendou, Yihan Wang, Yimu Pan, Zaccharie Ramzi, Zhenlin Xu
TL;DR
Voxtral TTS addresses the challenge of producing expressive multilingual speech from short reference clips for zero-shot voice cloning. It uses a hybrid semantic-autoregressive and acoustic-flow architecture built on Voxtral Codec, and human evaluations prefer it over ElevenLabs Flash v2.5 with a 68.4% win rate. The model is released under a CC BY-NC license.
Problem
Expressive natural speech remains difficult for zero-shot voice cloning, despite strong intelligibility from recent neural TTS systems.
Method
Voxtral TTS uses Voxtral Codec with ASR-distilled semantic and FSQ acoustic tokens, autoregressively generating semantics and flow-matching acoustics.
Results
68.4% win rate over ElevenLabs Flash v2.5 in human evaluation shows preference for Voxtral TTS in multilingual zero-shot voice cloning.
Takeaways & Limitations
Voxtral TTS generates expressive voice-cloned speech from as little as 3 seconds of reference audio and is preferred to API baselines in human evaluations.
Abstract
from arXiv · showhide
We introduce Voxtral TTS, an expressive multilingual text-to-speech model that generates natural speech from as little as 3 seconds of reference audio. Voxtral TTS adopts a hybrid architecture that combines auto-regressive generation of semantic speech tokens with flow-matching for acoustic tokens. These tokens are encoded and decoded with Voxtral Codec, a speech tokenizer trained from scratch with a hybrid VQ-FSQ quantization scheme. In human evaluations conducted by native speakers, Voxtral TTS is preferred for multilingual voice cloning due to its naturalness and expressivity, achieving a 68.4\% win rate over ElevenLabs Flash v2.5. We release the model weights under a CC BY-NC license.
1 Introduction
Voxtral TTS targets expressive, natural speech in zero-shot voice cloning, where capturing human vocal nuance remains challenging. It combines semantic autoregressive generation with flow-based acoustic modeling and reports strong multilingual results.
- Zero-shot TTS must capture human speech nuance and expressivity beyond intelligibility, especially when cloning unseen voices.
- Voxtral TTS combines autoregressive semantic-token generation with flow-matching acoustic prediction in a factorized speech representation.The design pairs long-range consistency from autoregressive modeling with rich acoustic detail from flow matching.
- Voxtral TTS supports 9 languages, voice prompts as short as 3 seconds, and low-latency streaming inference.
- 68.4% win rate over ElevenLabs Flash v2.5 demonstrates a human-evaluation preference for Voxtral TTS in multilingual zero-shot voice cloning.
2 Modeling
Voxtral TTS uses Voxtral Codec to represent speech as low-rate semantic and higher-rate acoustic tokens, then generates these streams with complementary discrete and continuous models. The codec combines convolutional-transformer encoding, hybrid quantization, ASR-distilled semantics, and adversarially trained reconstruction.
- Voxtral TTS architecture: The TTS decoder conditions on reference audio and text tokens, autoregressively generates semantic tokens, and uses flow matching to predict acoustic tokens before waveform decoding.
- Voxtral Codec: Voxtral Codec compresses 24 kHz mono waveforms into 12.5 Hz frames containing one semantic and 36 acoustic tokens at 2.14 kbps.
- Representation quantization: The codec splits its latent into independently quantized semantic and acoustic components, using VQ for semantics and 21-level FSQ across 36 acoustic dimensions.
- Semantic token learning: ASR distillation aligns semantic tokens with supervised Whisper representations, providing text-aligned supervision without an external forced aligner or paired transcripts.
- Training and configuration: The codec is a roughly 300M-parameter model whose final configuration achieves stable optimization with the best audio quality among its ablated settings.
- Architectural alternatives: MaskGIT and Depth Transformer alternatives perform reasonably but are inferior to flow matching in human evaluations, especially for expressivity.
3 Training
Voxtral TTS is trained on paired voice-reference, transcript, and target-audio samples, using separate semantic and acoustic objectives. Post-training applies DPO with flow-matching-specific handling and stability measures.
- Pretraining: Training samples pair a voice reference A1 with transcript T2 and target audio A2 from the same speaker.A1 and A2 are single-speaker segments that need not be temporally adjacent; prompts work best at 3–25 seconds.
- Pretraining: The model combines cross-entropy loss on semantic tokens with flow-matching loss on acoustic tokens, computing loss only on A2.
- Pretraining: Training initializes the decoder from Ministral 3B, freezes text embeddings, downweights nonspeech frames, and rewrites transcripts for text-format robustness.Extremely long silences receive zero loss weight, based on voice-activity detection.
- Direct Preference Optimization: Direct Preference Optimization targets word error rate and speaker similarity, using the standard objective for semantic tokens and an adapted objective for flow-matched acoustic codebooks.
- Direct Preference Optimization: DPO uses uniform weighting with βsemantic = 0.1 and βacoustic = 0.5, while a learning rate of 8e−8 improves training stability.The sampled t and x0 remain consistent between policy and reference models; length normalization by winner length causes instability.
- Direct Preference Optimization: DPO data come from rejection sampling over held-out voice samples and diverse synthetic conversational text prompts.Mistral Small Creative 1 generates continuations or replies conditioned on voice-prompt transcripts and randomly chosen personas.
4 Results
Voxtral TTS is evaluated with codec comparisons, automated metrics, and native-speaker human evaluations across multilingual and voice-cloning settings. Results show strong codec performance and speaker similarity, while human evaluation reveals advantages that automated metrics do not consistently capture.
- 4.1 Voxtral Codec: At similar bitrates, Voxtral Codec outperforms Mimi on all objective metrics and is comparable or better in subjective assessment for speech audio.The comparison uses the Expresso dataset and a 16-codebook Mimi configuration.
- 4.2 Automatic Evaluations: Automated evaluation measures intelligibility with WER, predicted quality with UTMOS-v2, and voice matching with speaker similarity.Speaker similarity is computed from ECAPA-TDNN speaker embeddings against the reference embedding.
- 4.2 Automatic Evaluations: Voxtral TTS significantly outperforms ElevenLabs models on speaker similarity, although Flash v2.5 performs better on most automated metrics.The evaluation compares Voxtral TTS, ElevenLabs v3, and ElevenLabs Flash v2.5 on SEED-TTS and MiniMax-TTS.
- 4.3.1 Flagship voices: In explicit emotion steering, Voxtral TTS is competitive with ElevenLabs v3, while in implicit steering it consistently outperforms both ElevenLabs models.Voxtral uses an emotion-bearing voice prompt for explicit steering and a neutral prompt for implicit steering.
- 4.3.2 Zero-Shot Voice Cloning: 68.4% is Voxtral TTS’s overall win rate against ElevenLabs Flash v2.5 in multilingual zero-shot voice cloning.Annotators rated likeness to the voice prompt, naturalness, and expressivity across languages.
5 Analysis
The analysis examines DPO checkpoints and inference parameters, finding broad metric improvements from DPO and trade-offs when increasing NFEs or CFG α.
- Multilingual evaluation: Voxtral TTS matches or outperforms ElevenLabs Flash v2.5 on every language, with an overall micro-average win rate of 68.4%.
- DPO improvements: DPO improves WER and UTMOS across languages, with the largest gains in German and French and regressions only on Hindi.Qualitatively, DPO reduces hallucinations and skipped words, while minimally affecting speaker similarity.
- DPO improvements: DPO reduces hallucinations, skipped words, and volume tapering, while speaker similarity remains within ±0.01 of the pretrained checkpoint.
- Inference Parameters: 8 NFEs is the default because increasing NFEs from 2 to 8 improves automatic metrics, whereas further increases marginally improve speaker similarity and slightly degrade WER.
- Inference Parameters: Higher CFG α nearly monotonically improves most automatic metrics but can cause over-adherence to the voice prompt and reduce implicit emotion steering in human evaluations.Lower α = 1.2 works best for higher-quality recordings, while in-the-wild recordings may benefit from higher α.
6 Inference and Serving in vLLM-Omni
Voxtral TTS uses a two-stage, asynchronously streamed serving pipeline in vLLM-Omni. CUDA graphs reduce decoding cost, while concurrency scaling maintains sub-second latency and real-time streaming performance.
- Serving architecture: The serving pipeline separates audio-token generation from codec waveform decoding and connects them through asynchronous chunked streaming over shared memory.The stages run in separate scheduling loops, and overlapping them supports streaming before the complete waveform is available.
- CUDA Graph Acceleration: The flow-matching transformer is the generation bottleneck, requiring 2 × N forward passes per generated frame when classifier-free guidance uses N function evaluations.
- CUDA Graph Acceleration: CUDA graphs use bucketed batch sizes with zero-padding and fall back to eager execution when a batch exceeds the largest captured bucket.
- CUDA Graph Acceleration: 47% lower latency and 2.5x lower real-time factor result from enabling CUDA graphs for flow-matching decoding.The comparison uses a 500-character input, 10-second reference audio, concurrency 1, and a single H200.
- Inference Throughput: Throughput rises from 119 to 1,431 characters per second per GPU as concurrency increases from 1 to 32, a 12x increase while latency remains sub-second.
7 Conclusion
The conclusion presents Voxtral TTS as a multilingual model combining autoregressive semantic-token generation with flow-matching acoustic-token generation. It generates expressive voice-cloned speech from short reference audio and is released as open weights under CC BY-NC.
- Conclusion: Voxtral TTS combines autoregressive generation of semantic tokens with flow-matching generation of acoustic tokens.Both token types come from Voxtral Codec, which combines ASR-distilled semantic tokens with FSQ acoustic tokens.
- Conclusion: Voxtral TTS generates expressive, voice-cloned speech from as little as 3 seconds of reference audio and is preferred to API baselines in human evaluations.
- Conclusion: The model weights are released under a CC BY-NC license to support further research and development of expressive TTS systems.