VoxCPM2 Technical Report

1.1 Background and VoxCPM Foundation

• Token-based multi-stage systems can restore acoustic detail but separate semantic planning from rendering, while continuous-latent models can struggle to jointly optimize structure and texture.The report identifies dependence on discrete-tokenizer capacity and error accumulation in long-form or expressive generation as limitations of these approaches.Source: §1.1 · p. 3
• VoxCPM (Zhou et al., 2025) uses a hierarchical TSLM, FSQ bottleneck, RALM, and Local Diffusion Transformer to jointly optimize semantic planning and acoustic rendering over continuous latents.Its differentiable semi-discrete bottleneck provides a stable skeleton while the RALM recovers fine acoustic detail, without an external discrete speech tokenizer.Source: §1.1 · p. 3

1.2 VoxCPM2: From Foundation to Full-Featured System

• VoxCPM2 unifies multilingual and cross-lingual TTS, reference-free voice design, style-controllable cloning, and transcript-conditioned continuation cloning in one backbone.The modes share parameters, training objective, and inference pipeline, differing only in input sequence organization.Source: §1.2 · p. 4
• AudioVAE V2 encodes at 16 kHz and reconstructs at 48 kHz, keeping latent sequences compact while enabling implicit super-resolution and high-quality output.The model maintains a compact 6.25 Hz token rate.Source: §1.2 · p. 4
• VoxCPM2 scales the framework to 2B parameters and over 2 million hours of multilingual speech covering 30 languages and 9 Chinese dialects.It preserves end-to-end training and avoids external discrete speech tokenizers.Source: §1.2 · p. 4
• Architectural refinements include improved semantic-acoustic fusion, multi-token LocDiT conditioning, and an isolated reference-audio pathway for multilingual and controllable generation.The report presents these refinements as supporting large-scale generation.Source: §1.2 · p. 4
• 1.68% average WER on the internal 30-language test set; VoxCPM2 also achieves competitive or state-of-the-art results on multiple public benchmarks and supports efficient streaming inference.The report presents these results as evidence of strong empirical performance and practical deployability.Source: §1.2 · p. 4

2.1 Large-Scale TTS Foundation Models

• Discrete-codec TTS inherits LLM-style scaling and in-context learning, spanning autoregressive token prediction, masked parallel generation, and semantic-to-acoustic staged prediction.The literature also explores RVQ prediction patterns and simplified or semantically aligned codec interfaces.Source: §2.1 · p. 5
• Hybrid systems pair an autoregressive model for semantic or coarse-acoustic tokens with a diffusion or flow-matching decoder for high-perceptual-quality rendering.Examples add emotion and duration control, reference-based speaker encoding, or hybrid discrete-continuous codec interfaces.Source: §2.1 · p. 5
• Continuous-latent methods aim to preserve acoustic detail; diffusion-autoregressive hybrids combine language-model long-range planning with local diffusion synthesis, as in ARDiT and DiTAR (Li et al., 2024; Jia et al., 2025).Other approaches use non-autoregressive diffusion, sparse alignment, or end-to-end generation without explicit alignment and duration modules.Source: §2.1 · p. 5
• Hierarchical semantic-acoustic decomposition often uses separate stages or discrete codecs, whereas VoxCPM (Zhou et al., 2025) embeds hierarchy in one continuous-latent backbone without external discrete tokenizers.VoxCPM2 scales this hierarchical continuous-latent paradigm into a multilingual and controllable foundation model.Source: §2.1 · p. 6

2.2 Controllable and Expressive Speech Generation

• Controllable TTS has moved from categorical labels and fixed styles toward natural-language prompts, captioned speech data, open-ended instructions, attribute editing, and reference-free voice design.Recent foundation systems integrate natural-language voice generation as a native capability.Source: §2.2 · p. 6
• Evaluation now spans instruction adherence, human-likeness, timing and pause fidelity, robustness under complex conditions, and aggregated acoustic and perceptual measures.The passage cites InstructTTSEval, MINT-Bench, the Audio Turing Test, Timing-control Benchmark, EmergentTTS-Eval, and TTSDS.Source: §2.2 · p. 6
• VoxCPM2 treats voice and style descriptions as text prefixes to the same TSLM, supporting voice design, reference-based cloning, controllable cloning, and continuation cloning in one backbone.This contrasts with systems fragmented across dedicated style encoders, adapters, or per-mode routing.Source: §2.2 · p. 6

3.1 Overview

• 16 kHz waveforms are encoded as 64-dimensional latent frames at 25 Hz, then grouped in patches of P=4 to form a 6.25 Hz autoregressive sequence with 160 ms per step.Speech is modeled entirely in AudioVAE V2’s continuous latent space.Source: §3.1 · p. 7

3 Methodology

• The LocEnc, TSLM, RALM, and LocDiT predict the next continuous latent patch step by step.Figure 1 presents the overall VoxCPM2 architecture.Source: §3.1 · p. 7, Figure 1 · p. 7

3.1 Overview

• The FSQ layer applies per-dimension scalar quantization to TSLM hidden states, while the RALM recovers fine-grained acoustic detail using text-side states and fused audio-history representations.The fused history combines FSQ-quantized audio-side history with Local Encoder embeddings.Source: §3.1 · p. 7
• A stop predictor on TSLM-FSQ hidden states determines when generation ends, and the full pipeline is trained end-to-end.The Local Encoder embeddings and FSQ-quantized history provide causal access to the full sequence history.Source: §3.1 · p. 7
• Compared with VoxCPM, VoxCPM2 conditions LocDiT on hFSQ_i as separate tokens and replaces the pre-RALM fusion operator with learnable concatenation-projection.The separate conditioning tokens replace a single summed vector.Source: §3.1 · p. 7
• VoxCPM2 adds AudioVAE V2 for 48 kHz output without lengthening the autoregressive sequence, a refined and scaled backbone, and shared-backbone input layouts for five speech-generation modes.The layouts cover basic TTS, voice design, reference cloning, controllable cloning, and continuation cloning; the backbone adds wider information pathways and isolated reference-audio input.Source: §3.1 · pp. 7–8

3.2 AudioVAE V2

• 16 kHz encoding and 48 kHz decoding improve waveform fidelity without increasing autoregressive-loop cost, while reusing 16 kHz training data and reducing latent mismatch and sequence length.The 16 kHz input rate aligns the backbone’s operational latent space across source sample rates.Source: §3.2 · p. 8
• The encoder downsamples by 640× to produce 64-dimensional latent frames at 25 Hz; with patch size P = 4, the backbone uses a 6.25 Hz autoregressive sequence.The causal CNN encoder uses downsampling rates [2, 5, 8, 8], while the decoder uses a deeper stack with upsampling rates [8, 6, 5, 2, 2, 2].Source: §3.2 · p. 8
• Following the streaming-friendly causal-convolutional design of the original AudioVAE (Zhou et al., 2025), V2 modifies rate-related modules and optionally conditions the decoder on target sample rate.The optional condition lets the same latent be rendered at multiple actual output rates for downstream deployment.Source: §3.2 · p. 8

3.3.1 Internal Architecture Refinements

• VoxCPM2 replaces element-wise fusion of semantic and local-encoder states with learnable concatenation-projection, preserving richer information and learning combination weights.Source: §3.3.1 · p. 8
• LocDiT receives semantic, residual, and timestep signals as separate prefix tokens rather than one summed token, avoiding early collapse and increasing conditioning bandwidth.It attends over the sequence with full attention and predicts velocity at noisy-patch positions; the sequence includes latent frames from the previous clean and current noisy patches.Source: §3.3.1 · pp. 8–9
• The FSQ bottleneck expands from 256 to 512 dimensions; VoxCPM2 retains TSLM RoPE but removes RALM positional encodings following NoPE (Kazemnejad et al., 2023).The bottleneck retains 9 quantization levels per dimension; removing RALM positional encodings reduces overfitting to training lengths and improves long-utterance stability.Source: §3.3.1 · p. 9

3.3.2 Reference Audio Pathway

• A delimited reference-audio prefix supplies speaker identity without transcript alignment or temporal-prefix constraints, while remaining optional at inference and excluded from training loss.The reference clip is encoded into latent patches and serves only as conditioning context, separating speaker identity from style-control instructions.Source: §3.3.2 · p. 9

3.3.3 Configuration Scaling

• Scaling increases depth, width, and context length; patch size 4, up from 2, cuts token rate from 12.5 Hz to 6.25 Hz, reducing inference cost and improving long-form stability.The retained patch-size change was first introduced in VoxCPM1.5 and preserves a compact token rate and streaming-friendly causal structure.Source: §3.3.3 · p. 9

3.4 Unified Sequence Organization

• Five generation configurations use one unified sequence organization rather than mode-specific modules, with layouts summarized in Table 2.The configurations span basic TTS, voice design, reference-based cloning with or without style control, and continuation cloning.Source: §3.4 · pp. 9–10, Table 2 · p. 10
• Each position carries a text token and audio latent, while a binary modality indicator selects the input embedding.Sequences combine synthesis text, optional natural-language voice or style descriptions, delimited reference audio for speaker identity, and target audio to generate.Source: §3.4 · p. 10
• Training applies loss only to target audio; at inference, prompt audio and its transcript can provide observed context for autoregressive continuation.The prompt is structurally treated as the initial prefix of the target-audio segment used in training.Source: §3.4 · p. 10
• Natural-language descriptions concatenate with synthesis text for voice design and controllable cloning, enabling one TSLM to handle content and control instructions without extra modules.Continuation cloning uses a paired transcript for higher fidelity; adding an isolated reference segment provides temporal alignment and speaker-identity evidence in the “Reference + Continuation” recipe.Source: §3.4 · p. 10

Training objective

• Training combines target-patch conditional flow matching with stop prediction on TSLM-FSQ hidden states, masking both losses to target audio.LocDiT conditioning is randomly dropped with 10% probability for classifier-free guidance; optimization uses AdamW, cosine learning-rate decay, and linear warmup.Source: §3.5 · p. 10

Three-stage progressive curriculum

• Stage 1 trains on multilingual transcription–audio pairs, limiting segments to 60 seconds and LM sequences to 4096 tokens.This stage establishes pronunciation and prosody across all 30 target languages while supporting basic TTS and continuation cloning.Source: §3.5 · p. 10
• Stage 2 gradually increases controllable data while retaining substantial plain TTS data, adding voice/style descriptions and reference-audio cloning examples.The maximum sequence length extends to 8192 tokens and audio duration to 3 minutes; Stages 1 and 2 account for most training compute.Source: §3.5 · pp. 10–11
• Stage 3 anneals learning on curated expressive speech and precisely annotated controllable data, with more controllable and reference-based cloning samples than Stage 2.It retains the 8192-token context, samples audio from 2 seconds to 5 minutes, balances language-level sampling, and applies learning-rate annealing.Source: §3.5 · p. 11

3.6 Data Construction and Annotation for Controllability

• The training corpus exceeds 2 million hours of multilingual speech, with Chinese and English the majority and the other 28 languages ranging from roughly 1 K to 50 K hours each.Base TTS data undergoes source separation, voice activity detection, ASR-based transcript alignment, and quality filtering.Source: §3.6 · p. 11
• Controllable-generation training combines tens of thousands of hours of open-source expressive speech with several thousand internally curated and annotated hours.Open-source data broadens emotion, speaking-style, and speaker coverage; internal data prioritizes annotation precision and richer natural-language descriptions.Source: §3.6 · p. 11
• To select expressive samples for annotation, the pipeline collects diverse scenarios and pre-screens large unlabeled corpora with lightweight emotion classifiers, retaining sufficiently expressive speech.This addresses acoustically flat public controllable corpora, which can cap controllability.Source: §3.6 · p. 11
• Selected utterances receive free-form natural-language descriptions of voice design attributes and style controls, generated by audio-understanding models and checked with expert classifiers.Voice design covers attributes such as age, gender, accent, vocal texture, and scenario; style control covers emotion, rate, pitch, energy, and emphasis. Step-Audio R1 (Tian et al., 2025) and Gemini 2.5 Pro generate descriptions, with dedicated classifiers verifying gender, age, and emotion; descriptions are used directly as text prefixes without extra embedding modules.Source: §3.6 · p. 11
• For reference-based cloning, same-session clips with speaker-embedding cosine similarity above 0.7 are retained, excluding clips that directly precede the target utterance.References may differ in fine acoustic details, so reference-based cloning naturally achieves lower similarity than continuation-based cloning; the pool supports both reference and controllable cloning.Source: §3.6 · p. 11
• To reduce style leakage from transcript content, VoxCPM2 clones an annotated voice and style onto semantically unrelated text, then mixes the resulting examples into training.These <description, another text, audio> pairs are used primarily in stage 2; stage 3 annealing is restricted to natively recorded high-quality speech to minimize self-synthesis artifacts. The method also extends controllable training to long-tail languages with limited native expressive data.Source: §3.6 · p. 11

3.7 Inference

• Inference generates speech autoregressively, one latent patch at a time, using three techniques to balance speed and quality.The techniques are classifier-free guidance, sway sampling, and CFG-Zero*.Source: §3.7 · p. 12
• CFG combines conditional and unconditional LocDiT velocity predictions at each denoising step, with α = 2.0 by default and α ∈ [1.5, 3.0] as a practical range.LM-side conditioning is randomly dropped during training, and inference evaluates LocDiT twice per denoising step.Source: §3.7 · p. 12
• Sway sampling allocates more solver steps to high-noise regimes, while CFG-Zero* reduces early-step artifacts; both are enabled by default and add no learnable parameters.The passage attributes sway sampling to Chen et al. (2025b) and CFG-Zero* to Fan et al. (2025).Source: §3.7 · p. 12
• Causal TSLM and RALM structure plus patch-local LocEnc and LocDiT enable streaming, as each generated latent patch is immediately decoded by a stateful AudioVAE V2 decoder.In continuation mode, the last few prompt patches are retained as the decoder’s initial context for smooth transitions.Source: §3.7 · p. 12

4.1 Experimental Setup

• Evaluations span zero-shot cloning, multilingual synthesis, natural-language controllability, reconstruction, and inference efficiency on public benchmarks and internal test sets.The setup includes Seed-TTS-Eval, CV3-Eval, MiniMax-MLS-Test, InstructTTSEval, and an internal benchmark of 500 utterances per language across 30 languages; intelligibility uses WER or CER, while cloning similarity uses speaker-embedding cosine similarity (SIM).Source: §4 · p. 12, §4.1 · pp. 12–13

4.2 Zero-Shot Voice Cloning on Seed-TTS-Eval

• 1.84/75.3 (WER/SIM) on test-EN, 0.97/79.5 on test-ZH, and 8.13/75.3 on test-ZH-Hard show competitive 2B-parameter zero-shot cloning.Combining reference audio as both an isolated speaker reference and a continuation prefix yields the highest SIM across subsets; reference-only gives the best intelligibility on hard Chinese. Listening tests also find superior cloning naturalness and prosody.Source: §4.2 · pp. 13–14, Table 3 · p. 13, Table 4 · p. 14

4.3 Multilingual Capability

• 22 of 24 MiniMax-MLS-Test languages have the highest SIM for VoxCPM2, while CV3-Eval results show competitive intelligibility across nine languages, including its hard subsets.On CV3-Eval, hard-zh WER is 8.55 and hard-en WER is 8.48. MiniMax intelligibility is strong on several languages, but Arabic and Hindi are weaker; the authors caution that Whisper-large-v3 may contribute to higher WER. Czech and Romanian, unseen in training, show partial intelligibility and promising speaker preservation.Source: §4.3 · pp. 14–15, Table 5 · p. 14, Table 6 · p. 15
• 1.68% average WER/CER across 30 languages, with errors below 3% in 28 and below 1% in six, demonstrates balanced performance on the internal benchmark.The 30-language test set contains 500 utterances per language and uses Gemini 3.1 Flash Lite for ASR; Arabic and Hindi results improve substantially relative to the MiniMax evaluation using Whisper-large-v3.Source: §4.3 · p. 15, Table 8 · p. 16

4.4 Voice Design and Controllable Generation

• 84.2 / 83.2 / 71.4 on APS, DSD, and RP make VoxCPM2 the best overall system on the English InstructTTSEval subset.On Chinese, it ties the top APS score at 85.2, with DSD 71.5 and RP 60.8; it also supports voice design and style-controllable cloning in 30 languages and nine Chinese dialects, with preliminary singing ability.Source: §4.4 · pp. 16–17, Table 9 · p. 17

4.5 AudioVAE V2 Reconstruction Quality

• AudioVAE V2 provides competitive reconstruction across low- and full-band metrics while encoding at 16 kHz and reconstructing at 48 kHz.VoxCPM’s Audio VAE remains competitive on 16 kHz-band speech metrics, while VoxCPM1.5 achieves the best full-band Mel-distance; V2 uses a compact low-rate latent space for high-sample-rate waveform generation.Source: §4.5 · pp. 17–18, Table 10 · p. 17

4.6 Inference Efficiency and Deployment

• RTF improves from 0.30 in standard PyTorch to 0.13 with Nano-vLLM on a single RTX 4090, with about 8 GB peak VRAM under PyTorch.The optimized path delivers over 7× real-time generation; despite 2.5× more parameters than VoxCPM1.5, VoxCPM2 maintains competitive or better RTF under optimized serving, aided by a 6.25 Hz language-model token rate. Its causal design supports streaming, and it is compatible with vLLM-Omni for batched serving.Source: §4.6 · p. 18, Table 11 · p. 18

4.7 Subjective Listening Tests

• 4.78 N-MOS in zero-shot cloning and multilingual synthesis, 4.74 S-MOS in cloning, and 4.50 I-MOS in controllable generation show strong perceived quality and instruction adherence.For controllable generation, N-MOS was 4.48; multilingual S-MOS was 4.66 across eight languages. The authors note that embedding-based SIM does not fully reflect perceived cloning quality, which is sensitive to personal characteristics and speaking-style consistency.Source: Table 12 · p. 18, §4.7 · p. 19, Table 13 · p. 19

5 Conclusion and Future Work

• VoxCPM2 uses one 2B-parameter backbone for 48 kHz synthesis in 30 languages and 9 Chinese dialects, supporting voice cloning, voice design, and style control without task-specific variants or external discrete tokenizers.Experiments on public and internal benchmarks show competitive or state-of-the-art performance, balancing scalability, voice fidelity, and controllability.Source: §5 · p. 19
• Cross-lingual quality varies on some low-resource languages because of data-distribution imbalance; complex instructions remain difficult to follow consistently, and serving overhead needs reduction.Source: §5 · pp. 19–20
• Future work will expand high-quality long-tail corpora, refine continuous-latent representations, and improve instruction-following robustness across speaking styles and languages.Source: §5 · p. 20
• Preliminary singing generation extends controllable synthesis, but its quality needs substantial improvement; longer-term work will explore a shared framework for speech, singing, and other modalities.Source: §5 · p. 20
• Future iterations will add provenance tracking, digital watermarking, and voice-cloning detection; the Apache 2.0 release includes training code and adaptation tools.Source: §5 · p. 20

• VoxCPM2 is a collaborative release by THUHCSI, THUNLP, and ModelBest, with thanks to the OpenBMB community.Source: §6 · p. 20
• The report names 18 core contributors.Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Jiancheng Gui, Jiaheng Wu, Ziyang Wang, Xudong Shen, Runchuan Ye, Zhisheng Zhang, Jiuyang Zhou, Bingsong Bai, Weiyue Sun, Mengyuan Deng, Qundong Shi, Zhiyong Wu, and Zhiyuan Liu.Source: §6 · p. 20
• The report also lists 20 other contributors in alphabetical order.Biyuan Lin, Caixian Chen, Chao Jia, Chenzhe Jing, Daixi Zeng, Jiayi Zhang, Jie Zhou, Jilong Ma, Jie Sun, Ling Zheng, Minmin Fan, Siyuan Huang, Shuo Wang, Susu Bai, Wenxi Yang, YingJiao Wang, Yitong Wang, Zhen Luo, Zhizheng Yang, and Zhong Zhuang.Source: §6 · p. 20