VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning

1 Introduction

• Codec-token pipelines improve scalability and in-context learning but lose acoustic detail through quantization and separate abstract language modeling from local diffusion refinement.The authors argue this semantic-acoustic divide impedes end-to-end optimization and holistic, expressive, context-aware synthesis.Source: §1 · p. 3
• Continuous autoregressive models avoid quantization loss, yet conflate semantic-prosodic planning and acoustic rendering, a task entanglement the authors link to instability and long-sequence error accumulation.Source: §1 · p. 3
• VoxCPM uses a differentiable FSQ bottleneck to stabilize semantic-prosodic plans in the TSLM while the RALM recovers fine-grained acoustic details.Both models guide a local diffusion decoder, and the hierarchy is trained end-to-end without pre-trained speech tokenizers.Source: §1 · p. 3
• Residual learning and the bottleneck provide functional separation without fragmented stages, simplifying training and removing dependence on external speech tokenizers.The authors also report ablations validating semi-discrete residual representations and say codes and models are publicly released.Source: §1 · p. 4
• 1.8 million hours of bilingual speech training yields VoxCPM-0.5B, which achieves state-of-the-art zero-shot TTS performance among open-source systems with RTF as low as 0.17 on a consumer-grade NVIDIA RTX 4090 GPU.Source: §1 · p. 4

2.1 Discrete Token-Based TTS

• Discrete-token TTS uses codec-derived representations for sequence generation, while later systems extend the paradigm toward zero-shot adaptation, controllability, efficiency, and dialogue.Source: §2.1 · p. 4

2.2 Continuous Representation TTS

• Continuous-representation TTS models mel-spectrograms or audio latents, with diffusion, flow matching, and patch-based methods pursuing greater detail, diversity, prosody, or streaming.Continuous models can still entangle semantic planning with acoustic rendering, contributing to long-sequence instability.Source: §2.2 · pp. 4–5

2.3 Hierarchical and Residual Modeling in TTS

• Hierarchical and residual methods decompose TTS into layered tasks, but few combine explicit residual designs with semi-discrete bottlenecks in a unified framework without external dependencies.Source: §2.3 · p. 5

3 Methodology

• LocEnc first processes audio latents as the input to the hierarchical speech-generation path.Source: Figure 1 · p. 5
• TSLM and FSQ produce a semi-discrete speech skeleton, which serves as the intermediate representation in the generation sequence.Source: Figure 1 · p. 5
• RALM refines acoustic details before LocDiT generates the high-fidelity latent output.Source: Figure 1 · p. 5

3.1 Core Design Motivation

• Discrete tokenization stabilizes generation but discards fine-grained acoustic detail, whereas continuous approaches preserve fidelity but can accumulate errors in long sequences.Source: §3.1 · p. 6
• Direct FSQ or VQ codebooks face a scalability challenge because richer acoustic representations require exponentially larger, sparse vocabularies that language models struggle to predict.Source: §3.1 · p. 6
• The proposed differentiable scalar-quantization bottleneck separates a discrete-like stable content skeleton from continuous residual components for expressive detail.Source: §3.1 · p. 6
• Quantization acts as hidden-state regularization rather than a discrete prediction target, avoiding vocabulary explosion while retaining the stabilizing effect of discrete representations.Source: §3.1 · p. 6

3.2 Model Overview

• VoxCPM generates continuous speech-latent patches autoregressively from input text, with each patch representing P frames of D-dimensional VAE latents.The generated sequence is Z = {z1, ..., zM}, conditioned on text tokens T = {t1, ..., tN}.Source: §3.2 · p. 6
• LocEnc, TSLM, RALM, and LocDiT form the hierarchical conditioning architecture, with a stop predictor determining when generation ends.Historical audio context is aggregated by LocEnc; the combined semantic and acoustic signal guides LocDiT in generating the current latent patch through diffusion.Source: §3.2 · p. 6
• The RALM adds residual acoustic details to the hierarchical representation that guides speech-latent generation.The resulting signal combines TSLM-derived semantic content with RALM-derived acoustic details, and gradients flow through all components during end-to-end training.Source: §3.2 · p. 6

3.3 Hierarchical Semantic-Acoustic Modeling

• The architecture is designed to separate semantic-prosodic planning from fine-grained acoustic synthesis through structured representation learning.This is the stated approach to the stability-expressivity trade-off.Source: §3.3 · p. 6

3.3.1 Text-Semantic Language Model (TSLM)

• The TSLM processes text and historical audio context to generate continuous representations encoding both speech content and its prosodic realization.It uses a pre-trained text language model backbone and learns contextually appropriate speech patterns from raw text rather than phoneme sequences.Source: §3.3.1 · p. 7

3.3.2 Semi-Discrete Representation Learning via FSQ

• FSQ maps TSLM hidden states to a differentiable semi-discrete bottleneck that captures a coarse semantic-prosodic skeleton.As an intermediate inductive bias, it encourages preservation of stable high-level components and reduces the TSLM's modeling burden.Source: §3.3.2 · p. 7

3.3.3 Residual Acoustic Modeling

• RALM conditions residual acoustic modeling on TSLM states, semi-discrete speech representations, and historical embeddings to recover speaker identity and fine acoustic variation.The TSLM+FSQ pathway focuses on content stability and prosodic coherence, while RALM specializes in acoustic expressivity and speaker characteristics.Source: §3.3.3 · p. 7

3.3.4 Local Diffusion Transformer Decoder

• LocDiT generates continuous latent patches conditioned on the combined hierarchical representation, using bidirectional within-patch context and the previous patch for consistency.The previous patch frames generation as outpainting; the decoder also masks LM guidance during training to enable classifier-free guidance at inference.Source: §3.3.3 · p. 7, §3.3.4 · pp. 7–8

3.4 Training Objective

• Conditional flow matching trains the model end-to-end to optimize generated speech-latent quality, using a noisy latent and the LocDiT-predicted velocity field.The authors adopt this formulation for training stability and sampling efficiency.Source: §3.4 · p. 8
• A binary classification loss trains the model to predict the end of a speech sequence.The objective combines flow matching and stop prediction as L = LFM + λLStop.Source: §3.4 · p. 8
• Gradients from the combined objective propagate through the autoregressive hierarchy, including FSQ via straight-through estimation, to coordinate component learning.The components learn semantic planning, stabilization, and acoustic refinement under the unified objective.Source: §3.4 · p. 8

3.5 Causal Audio VAE

• The causal VAE supports streaming encoding and decoding, targeting low-latency real-time use while reducing computation through a latent space that preserves perceptual quality.It is trained separately from the main model.Source: §3.5 · p. 8
• For 16 kHz single-channel audio, the VAE downsamples by 640x to a 25 Hz latent representation using strided convolutions.Its encoder and decoder use stacked causal convolutional networks, and the decoder upsamples to reconstruct the waveform.Source: §3.5 · p. 8
• VAE training combines adversarial GAN, Mel-spectrogram, and KL-divergence losses, with the KL weight set to 5e-5.Source: §3.5 · p. 8

4.1 Experimental Setup

• Training used a 1.8-million-hour internal bilingual corpus and 95,000 hours of Emilia; the respective VoxCPM variants trained for 500K and 200K iterations.The internal corpus mainly comprises Chinese and English speech.Source: §4.1 · pp. 8–9

4.2 Main Results: Comparison with State-of-the-Art TTS

• 1.85% English WER and 0.93% Chinese CER establish open-source state-of-the-art on SEED-TTS-EVAL; CV3 reports 3.40% ZH-CER, 4.04% EN-WER, and 7.89% Hard-Test EN-WER, beating CosyVoice 3.The reported CV3 Hard-Test comparison is against close-sourced CosyVoice 3.Source: Table 3 · p. 10, §4.2 · p. 10, Table 4 · p. 11
• VoxCPM leads English speaker similarity and slightly exceeds IndexTTS 2 in Chinese speaker similarity, while trailing it in Chinese naturalness; VoxCPM-Emilia has relatively lower naturalness.The subjective evaluations cover naturalness and speaker similarity.Source: §4.2 · pp. 10–11, Table 5 · p. 11

4.3 Ablation Study: Effect of the Semi-discrete Bottleneck

• Without FSQ, ZH-CER on hard cases rises to 24.92%; FSQ-d256 balances discretization with retained prosodic and speaker information, while d4 and d1024 underperform for different reasons.Lower dimensions constrain prosodic capacity; higher dimensions provide insufficient discretization to prevent task entanglement.Source: §4.3 · pp. 11–12, Table 6 · p. 11

4.4 Ablation Study: Effect of Residual Acoustic Modeling

• Ablations show degraded challenging-case performance without RALM or its original acoustic embeddings; the default residual connection performs best by combining TSLM and RALM hidden states.The reported division of labor assigns semantic-prosodic planning to TSLM and acoustic refinement to RALM.Source: §4.4 · p. 12, Table 7 · p. 12

4.5 Effect of Training Phase on Performance

• 8.87% ZH-Hard CER in Decay, down from 13.22% in Stable, accompanies a 4.4-point SIM improvement; the Decay phase improves all reported metrics.The WSD schedule first stabilizes convergence, then uses Decay to refine performance, especially zero-shot speaker similarity.Source: Table 8 · p. 12, §4.5 · p. 12

4.6 Effect of LM Guidance on LocDiT

• Moderate CFG at 2.0 gives the best reported balance of voice similarity and intelligibility; scales of 3.0 or higher significantly degrade intelligibility.Without CFG, the model has high error rates and low similarity scores.Source: §4.6 · p. 13, Table 9 · p. 13

Visual Analysis of Hierarchical Representations

• Figures 2–3 show TSLM-FSQ representations tracking text content and genre, while RALM residuals vary with speaker and add finer acoustic nuance; style can be inferred without prompt speech.The visual analysis describes distinct text-related clusters for genres such as news, storytelling, and rap lyrics.Source: Figure 2 · p. 13, Visual Analysis of Hierarchical… · pp. 13–14, Figure 3 · p. 14

• VoxCPM uses hierarchical semantic-acoustic modeling without external speech tokenizers and achieves state-of-the-art open-source zero-shot TTS performance in intelligibility and speaker similarity.Source: §5 · p. 14
• The model is primarily optimized for Chinese and English, offers constrained fine-grained prosody and emotion control, and its AudioVAE supports only 16kHz generation.The conclusion notes uncertain generalization to other languages and says 16kHz falls short of the 24kHz or 44.1kHz rates typically required for high-fidelity applications.Source: §5 · p. 14
• Voice-mimicking synthesis risks spoofing authentication or nonconsensual impersonation; experiments assume authorized reference voices, and the authors advocate detection methods and deployment guidelines.Source: §5 · p. 15

• VoxCPM is a collaborative release by THUHCSI, THUNLP, and ModelBest.Source: §6 · p. 15
• The listed core contributors are Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Ziyang Wang, Runchuan Ye, Weiyue Sun, Jiancheng Gui, Kehan Li, Zhiyong Wu, and Zhiyuan Liu.Source: §6 · p. 15
• The listed other contributors, in alphabetical order, are Biyuan Lin, Chao Jia, Chenzhe Jing, Hongyu Liu, Jie Cai, Jie Zhou, Junshao Guo, Lei Chen, Rongting Tang, Rui Li, Ruiqi Shao, Qundong Shi, Shuo Wang, Siyuan Huang, Shun Lei, Wenxi Yang, Xiaoshuang Wang, Yihang He, and Zichao Nie.Source: §6 · p. 15