Source-linked AI summary
MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder
Bowen Zhang, Congchao Guo, Geng Yang, Hang Yu, Haozhe Zhang, Heidi Lei, Jialong Mai, Junjie Yan, Kaiyue Yang, Mingqi Yang, Peikai Huang, Ruiyang Jin, Sitan Jiang, Weihua Cheng, Yawei Li, Yichen Xiao, Yiying Zhou, Yongmao Zhang, Yuan Lu, Yucen He
TL;DR
MiniMax-Speech addresses limitations in transcription-dependent voice cloning and speech-generation quality. It combines an autoregressive Transformer with a learnable speaker encoder and Flow-VAE, achieving high-fidelity multilingual synthesis and robust zero-shot cloning while supporting downstream voice applications. Its emotion-control application depends substantially on the form of the training data.
Problem
Existing TTS methods can depend on transcribed reference audio or have limited audio quality and speaker similarity, constraining cross-lingual and expressive voice cloning.
Method
MiniMax-Speech combines an autoregressive Transformer, a jointly trained speaker encoder, and a Flow-VAE-based flow matching component.
Results
MiniMax-Speech supports synthesis in 32 languages and achieves state-of-the-art results on multiple objective and subjective evaluation metrics.
Takeaways & Limitations
The speaker encoder enables transcription-free zero-shot cloning and downstream extensions including emotion control, text-to-voice, and professional voice cloning.
Takeaways & Limitations
Emotion control depends substantially on training data formatted as reference audio, text, and target emotive audio.
Abstract
from arXiv · showhide
We introduce MiniMax-Speech, an autoregressive Transformer-based Text-to-Speech (TTS) model that generates high-quality speech. A key innovation is our learnable speaker encoder, which extracts timbre features from a reference audio without requiring its transcription. This enables MiniMax-Speech to produce highly expressive speech with timbre consistent with the reference in a zero-shot manner, while also supporting one-shot voice cloning with exceptionally high similarity to the reference voice. In addition, the overall quality of the synthesized audio is enhanced through the proposed Flow-VAE. Our model supports 32 languages and demonstrates excellent performance across multiple objective and subjective evaluations metrics. Notably, it achieves state-of-the-art (SOTA) results on objective voice cloning metrics (Word Error Rate and Speaker Similarity) and has secured the top position on the public TTS Arena leaderboard. Another key strength of MiniMax-Speech, granted by the robust and disentangled representations from the speaker encoder, is its extensibility without modifying the base model, enabling various applications such as: arbitrary voice emotion control via LoRA; text to voice (T2V) by synthesizing timbre features directly from text description; and professional voice cloning (PVC) by fine-tuning timbre features with additional data. We encourage readers to visit https://minimax-ai.github.io/tts_tech_report for more examples.
1. Introduction
MiniMax-Speech combines a learnable speaker encoder with an autoregressive TTS model and Flow-VAE to support transcription-free voice cloning, high-quality speech, and multiple downstream applications.
- MiniMax-Speech uses a learnable speaker encoder to enable zero-shot voice cloning from speech prompts without reference transcription.This removes text-speech mismatch issues and supports cross-lingual and multilingual synthesis.
- Jointly training the speaker encoder with the autoregressive model tailors speaker representations to TTS and improves speaker similarity and intelligibility.
- Flow-VAE strengthens the information representation power of flow matching, improving generated audio quality and speaker similarity.
- MiniMax-Speech supports 32 languages and achieves state-of-the-art results across multiple objective and subjective evaluation metrics.
- The model extends to emotion control, text-to-voice voice-library construction, and professional voice cloning through downstream adaptation methods.
2. Method
MiniMax-Speech combines an autoregressive Transformer with a jointly trained speaker encoder for intrinsic zero-shot voice cloning from untranscribed audio, while Flow-VAE and flow matching model continuous speech features for waveform generation. The architecture also supports optional one-shot prompting and downstream control applications without fundamentally altering the base model.
- Autoregressive Transformer: MiniMax-Speech generates discrete audio tokens with an autoregressive Transformer and targets high-fidelity cloning from a single untranscribed reference segment.The model is designed for zero-shot synthesis, where the reference audio provides speaker characteristics without paired text.
- Speaker Encoder: The learnable speaker encoder jointly trains with the autoregressive Transformer, extracting vocal timbre and prosodic style into a fixed-size conditioning vector.This vector guides generation toward the desired speaker identity, while joint optimization tailors the encoder to speech synthesis.
- Zero-Shot and One-Shot Voice Cloning: Intrinsic zero-shot cloning uses only an untranscribed reference waveform, whereas one-shot prompting can optionally provide paired text-audio information or finer-grained stylistic cues.The system’s architecture is fundamentally designed for zero-shot synthesis, while supporting both modalities through its conditioning encoder.
- Extensibility: The speaker representation supports downstream emotion control, text-to-voice, and professional voice cloning without fundamentally altering the base model.These applications use the core speaker identity representation for additional control or adaptation.
- Flow-VAE and Flow Matching: The latent flow matching model operates on continuous speech features extracted by an encoder-decoder module rather than mel-spectrograms.The Flow-VAE combines an encoder, decoder, and flow model; the flow transforms latent distributions through reversible mappings and supports a more expressive posterior.
- Flow-VAE and Flow Matching: At inference, the flow matching model conditions on global timbre information and optional prompt information, supporting both zero-shot and one-shot synthesis.Global timbre is extracted from mel-spectrogram features with the speaker encoder, while prompt information can come from the beginning of the current sentence.
- Flow-VAE and Flow Matching: Flow-VAE uses waveform input for posterior encoding and applies reversible flow transformations before calculating KL divergence against a standard normal distribution.This constrains the encoder output toward a normal distribution while enhancing its information expression ability.
3. Experiments
MiniMax-Speech is evaluated for voice cloning, naturalness, multilingual synthesis, cross-lingual performance, speaker conditioning, and audio quality. Across these evaluations, the model shows strong zero-shot cloning, leading subjective preference, multilingual speaker similarity, and advantages from its learnable speaker encoder and Flow-VAE.
- Evaluation scope: MiniMax-Speech was evaluated across voice cloning, human preference, multilingual, cross-lingual, and ablation settings.The evaluation also examined the model’s audio quality and speaker conditioning components.
- Voice cloning: Zero-shot cloning achieved lower WER than one-shot cloning and was judged more natural and realistic by listeners.The speaker encoder preserves vocal identity while allowing the autoregressive model greater freedom in generating text-faithful prosody.
- Subjective evaluation: MiniMax-Speech secured the leading position on the Artificial Arena leaderboard, with ELO scores indicating user preference for its naturalness and expressiveness over leading competitors.All submitted samples used the model’s zero-shot speaker-cloning capability.
- Multilingual evaluation: Across 24 evaluated languages, SIM was higher than Multilingual v2, while WER was comparable overall and better where Multilingual v2 exceeded 10%.The strongest WER advantages were reported for Chinese, Cantonese, Thai, Vietnamese, and Japanese.
- Cross-lingual evaluation: Zero-shot cross-lingual synthesis achieved lower WER than one-shot synthesis across all tested languages, whereas one-shot synthesis produced higher SIM but poorer pronunciation accuracy.These results support the speaker encoder’s flexibility across cloning paradigms and its advantage for zero-shot cross-lingual synthesis.
- Speaker conditioning: In speaker-conditioning ablations, the learnable speaker encoder balanced WER and SIM, achieving one-shot WER 1.243 and SIM 0.746.In zero-shot synthesis it achieved SIM 0.730 and WER 1.252; identical reference and target audio during training can cause semantic leakage and degrade performance.
- Audio quality: Flow-VAE demonstrated significant advantages over VAE across all evaluated metrics and showed better overall stability in listening tests.The comparison covered synthesized audio against the original audio.
4. Extensions
MiniMax-Speech’s disentangled speaker representations support extensions beyond cloning without modifying the base model. The paper demonstrates emotion control, text-driven timbre generation, and efficient professional voice cloning.
- Overview: The integrated speaker encoder captures vocal identity without transcription, providing a stable foundation for downstream extensions.The paper develops three extensions: emotion control, text-to-voice, and professional voice cloning.
- Emotion control: LoRA modules trained for discrete emotion categories enable precise, stable emotional control during inference without changing the pretrained core architecture.Neutral references improved expressiveness, while random-emotion references supported natural speech and stable speaker similarity.
- Emotion control: The emotion-control training format uses multiple emotive renditions of identical text to decouple emotional rendering from lexical content.This trains the model to express the same content with varied emotional inflections.
- Text-to-voice: The T2V framework combines natural-language descriptions with structured tags to generate desired vocal characteristics without requiring a reference sample of the target timbre.The structured attributes include speech rate, gender, language, pitch, and volume, while compressed timbre representations are used for prediction.
- Professional voice cloning: Professional voice cloning fine-tunes a target speaker’s conditional embedding while keeping the autoregressive Transformer frozen.Optimizing only a speaker-specific embedding supports adaptation for potentially thousands of speakers without deploying separate complete models.
5. Conclusion
MiniMax-Speech addresses limits in transcribed-prompt cloning and generative audio quality with a learnable speaker encoder and Flow-VAE. It supports 32 languages, achieves strong objective and subjective results, leads the public TTS Arena, and enables several extensions.
- Conclusion: MiniMax-Speech targets challenges from transcribed reference requirements, limited cross-lingual capability, and generative-component limits on audio quality and speaker similarity.Its two central innovations are a learnable speaker encoder and Flow-VAE integrated with flow matching.
- Conclusion: The model supports 32 languages and demonstrates SOTA performance on objective and subjective evaluations, including voice-cloning metrics and the public TTS Arena leaderboard.The speaker encoder also supports LoRA emotion control, text-driven timbre generation, and efficient professional voice cloning.
A. Contributors
The report lists its contributors in alphabetical order.
- Contributors: The contributors are listed alphabetically.The listed contributors are Bowen Zhang, Congchao Guo, Geng Yang, Hang Yu, Haozhe Zhang, Heidi Lei, Jialong Mai, Junjie Yan, Kaiyue Yang, Mingqi Yang, Peikai Huang, Ruiyang Jin, Sitan Jiang, Weihua Cheng, Yawei Li, Yichen Xiao, Yiying Zhou, Yongmao Zhang, Yuan Lu, and Yucen He.