PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxCPM2 Technical Report

Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Jiancheng Gui, Jiaheng Wu, Ziyang Wang, Xudong Shen, Runchuan Ye, Zhisheng Zhang, Jiuyang Zhou, Bingsong Bai, Weiyue Sun, Mengyuan Deng, Qundong Shi, Zhiyong Wu, Zhiyuan Liu

arXiv:2606.06928v1cs.SDeess.AS

TL;DR

VoxCPM2 addresses the challenge of building a multilingual, controllable speech generator that preserves acoustic quality without external discrete speech tokenizers. It uses a unified hierarchical continuous-latent backbone with shared sequence organization, and reports competitive or state-of-the-art benchmark performance including 1.68% average WER on an internal 30-language set. The paper concludes that this paradigm provides a viable foundation for large-scale multilingual and controllable speech generation.

  • Problem

    Discrete-token and continuous-latent TTS approaches respectively lose fine acoustic detail or must jointly optimize semantic-prosodic structure and acoustic texture, motivating a unified alternative.

  • Method

    VoxCPM2 uses a single hierarchical continuous-latent backbone that unifies multiple generation modes through shared parameters, sequence organization, and an asymmetric AudioVAE.

  • Results

    VoxCPM2 achieves competitive or state-of-the-art performance on public benchmarks and 1.68% average WER on its internal 30-language evaluation set.

  • Takeaways & Limitations

    The results support hierarchical continuous-latent modeling without external discrete speech tokenizers as a viable foundation for multilingual and controllable speech generation.

  • Takeaways & Limitations

    Cross-lingual quality remains affected by data-distribution imbalance, while complex cross-language instructions and computational overhead remain challenging.

Abstract

from arXiv · show

We present VoxCPM2, a https://info.arxiv.org/help/prep#abstractsfully open-source multilingual and controllable speech generation foundation model that extends the hierarchical diffusion-autoregressive modeling paradigm of VoxCPM. VoxCPM2 advances the framework in three key dimensions: (i) capability, by unifying 30 languages, 9 Chinese dialects, natural-language voice design, style-controllable voice cloning, and high-fidelity continuation cloning within a single backbone; (ii) quality, through an asymmetric AudioVAE that encodes at 16 kHz and reconstructs at 48 kHz, enabling implicit super-resolution with high encoding efficiency; and (iii) scale, by jointly scaling the model to 2B parameters and the training data to over 2 million hours of multilingual speech. To support these diverse capabilities within one model, we introduce a unified sequence organization that expresses all generation modes through different arrangements of the same input building blocks, allowing joint training under a single set of parameters and objective. VoxCPM2 achieves state-of-the-art or competitive performance on public zero-shot and instruction-following TTS benchmarks. On our internal 30-language evaluation set, it attains an average WER of 1.68%. These results demonstrate that hierarchical continuous-latent modeling, without relying on any external discrete speech tokenizer, offers a viable and powerful foundation for large-scale multilingual and controllable speech generation. The model weights, fine-tuning code, and inference tools are publicly released under the Apache 2.0 license to foster community research and development.

1 Introduction

VoxCPM2 extends hierarchical continuous-latent speech generation into a unified multilingual and controllable foundation model. It combines one backbone, 48 kHz synthesis, broad generation capabilities, and competitive empirical performance.

  • Capability: VoxCPM2 unifies basic TTS, natural-language voice design, controllable cloning, and continuation-based cloning within one backbone.These modes share parameters, training objective, and inference pipeline, differing only in input sequence organization.
  • Quality: An asymmetric AudioVAE encodes at 16 kHz and reconstructs at 48 kHz, enabling implicit super-resolution without lengthening the autoregressive sequence.The design maintains compact latent sequences while producing high-quality output.
  • Scale: VoxCPM2 scales to 2B parameters and over 2 million hours of multilingual speech covering 30 languages and 9 Chinese dialects.It maintains a compact 6.25 Hz token rate while scaling model and data size.
  • Foundation: The model preserves hierarchical end-to-end continuous-latent training without external discrete speech tokenizers.Its architecture combines semantic and acoustic modeling while retaining end-to-end optimization.
  • Results: 1.68% average WER on the internal 30-language test set accompanies competitive or state-of-the-art results on multiple public benchmarks.The reported contributions also include efficient streaming inference.

2 Related Work

Related TTS research spans discrete-token, continuous-latent, and hierarchical semantic-acoustic paradigms, alongside increasingly flexible controllable speech generation. VoxCPM2 addresses fragmentation by placing multiple control modes in one hierarchical continuous-latent backbone.

  • Large-scale TTS paradigms: Discrete-token systems inherit LLM-style scaling and in-context learning, but multi-stage designs separate semantic planning from acoustic rendering.Their performance also depends heavily on the intermediate discrete speech tokenizer.
  • Large-scale TTS paradigms: Continuous-latent methods preserve fine acoustic details but must jointly optimize semantic-prosodic structure and local acoustic texture.This creates a shared optimization challenge within one representation space and training objective.
  • Large-scale TTS paradigms: Hierarchical semantic-acoustic decomposition appears across discrete and continuous systems, either through separate stages or explicit internal hierarchy.The surveyed approaches differ in where and how semantic and acoustic representations are separated.
  • Controllable speech generation: Controllable speech generation has progressed from categorical labels and fixed attributes toward natural-language interfaces and fine-grained expressive control.Recent work also expands captioned data construction and instruction-focused evaluation.
  • VoxCPM2 positioning: VoxCPM2 replaces fragmented style encoders, adapters, and per-mode routing with ordinary text prefixes and a unified sequence organization.This supports voice design, reference-based cloning, controllable cloning, and continuation cloning within one backbone.

3 Methodology

VoxCPM2 organizes text, optional reference audio, continuous speech latents, and termination control within one overall architecture. The figure presents these shared input and generation building blocks.

  • Latent generation: Continuous speech latent tokens and patch-level latent generation represent the target audio within the model.Residual acoustic hidden states are also shown as part of the generation pipeline.
  • Unified organization: A unified sequence organization expresses the generation modes through shared sequence components.The architecture includes a continue-or-stop decision for terminating generation.
  • Inputs: Text tokens and optional reference audio provide the conditioning inputs for VoxCPM2 generation.The figure labels text with description and an optional reference-audio pathway.
  • Architecture overview: The architecture diagram summarizes VoxCPM2 as a shared pipeline rather than separate task-specific systems.Its overall architecture is presented as the organizing view for the methodology.

3.1 Overview

VoxCPM2 generates continuous latent patches autoregressively through a Local Encoder, TSLM, RALM, and LocDiT. Its refinements provide richer conditioning while retaining end-to-end training and compact high-rate output.

  • Backbone overview: The backbone predicts the next continuous latent patch step by step using LocEnc, TSLM, RALM, and LocDiT.A stop predictor on TSLM-FSQ hidden states determines when generation terminates.
  • Hierarchical modeling: The TSLM produces a semi-discrete semantic skeleton, while the RALM recovers fine-grained acoustic details from text-side and audio-history conditioning.The RALM uses fused FSQ-quantized history and Local Encoder embeddings for causal access to sequence history.
  • Architecture refinements: VoxCPM2 supplies LocDiT with FSQ representations as separate conditioning tokens rather than one summed vector.This is one of the stated VoxCPM2-specific modifications to the formulation.
  • Architecture refinements: The fusion operator before the RALM is replaced by a learnable concatenation-projection.This refinement accompanies the revised LocDiT conditioning scheme.
  • System-level changes: AudioVAE V2 raises output sampling to 48 kHz without lengthening the autoregressive sequence.The overview also highlights wider information pathways, isolated reference-audio input, and unified sequence organization.

3.2 AudioVAE V2

VoxCPM2 refines its hierarchical backbone with an asymmetric AudioVAE, richer semantic-acoustic fusion, higher-bandwidth diffusion conditioning, and an isolated reference-audio pathway. These changes support efficient, stable multilingual and controllable generation.

  • AudioVAE V2: 16 kHz encoding and 48 kHz reconstruction provide efficient latent encoding alongside higher-fidelity waveform output.The asymmetric design improves decoder-side fidelity without increasing autoregressive generation cost while reusing 16 kHz training data.
  • AudioVAE V2: A 640× temporal reduction produces 64-dimensional latent frames at 25 Hz, while the decoder uses deeper causal convolutions and higher-bandwidth upsampling.The encoder downsamples with rates [2, 5, 8, 8], and the decoder upsamples with rates [8, 6, 5, 2, 2, 2].
  • Backbone refinements: Learnable concatenation-projection fusion replaces element-wise summation, preserving richer information from the semantic and local-encoder streams.The projection learns how to combine the two streams before they enter the RALM.
  • Diffusion conditioning: LocDiT receives semantic, residual, and timestep signals as separate prefix tokens, avoiding early information collapse and increasing conditioning bandwidth.The diffusion transformer attends over these prefixes and patch latents before predicting the velocity field at noisy-patch positions.
  • Backbone refinements: The FSQ bottleneck expands from 256 to 512 dimensions while retaining 9 quantization levels per dimension.This increases capacity for the larger model and broader linguistic coverage without changing quantization granularity.
  • Reference conditioning: An isolated reference-audio pathway inserts a target-speaker clip as a delimited voice-identity prefix without requiring its transcript.Causal attention lets subsequent positions use this segment for speaker-identity conditioning.
  • Scaling: Increasing patch size from P = 2 to P = 4 lowers the language-model token rate from 12.5 Hz to 6.25 Hz, reducing inference cost and improving long-form stability.The change is retained from VoxCPM1.5 and preserves the compact token rate and streaming-friendly causal structure.

3.4 Unified Sequence Organization

VoxCPM2 expresses five generation configurations through one unified sequence organization. The same text, reference-audio, and target-audio building blocks are rearranged to support multiple capabilities under shared modeling components.

  • Unified organization: Five configurations share one sequence organization and reduce to four capabilities: basic TTS, voice design, reference-based cloning, and continuation cloning.Reference-based cloning includes variants with or without additional style control.
  • Input building blocks: Each input position carries a text token, an audio latent, and a binary modality indicator that determines its embedding.Sequences are assembled from synthesis text, optional reference audio, and the target-audio segment to generate.
  • Training and inference: Only target-audio tokens contribute to training loss, while preceding tokens provide conditioning context.At inference, prompt audio and its transcript can serve as the initial observed prefix of the target segment for autoregressive continuation.
  • Control modes: Natural-language voice and style descriptions are concatenated with synthesis text, enabling control through the same TSLM without additional modules.Continuation cloning benefits from paired transcripts, and isolated reference segments can add speaker identity and temporal alignment.

3.5 Training Strategy

VoxCPM2 uses a fixed two-term objective within a three-stage progressive curriculum. Training progressively expands language, controllability, context length, and data quality while protecting base synthesis quality.

  • Objective: The objective combines patch-level conditional flow matching on target latents with binary stop prediction on TSLM-FSQ hidden states.Both losses are masked to the target-audio segment, and 10% conditioning dropout enables classifier-free guidance at inference.
  • Curriculum: A three-stage curriculum varies data composition, mixing ratio, and context length while keeping the loss formulation fixed.The curriculum is designed to avoid destabilizing base synthesis quality while adding the model’s target capabilities.
  • Stage 1: Stage 1 pretrains multilingual TTS and continuation cloning on transcription-audio pairs, using 60-second segments and 4096-token maximum sequences.This stage establishes pronunciation and stable optimization before broader controllability training.
  • Stage 2: Stage 2 adds voice-design and cloning data, extending sequences to 8192 tokens and audio duration to 3 minutes.The stage includes natural-language descriptions and reference-transcription-target triplets for controllable generation.
  • Stage 3: Stage 3 performs high-quality annealing SFT on curated expressive speech with a larger proportion of precisely annotated controllable data.The final mixture emphasizes natively recorded high-quality speech and richer controllable annotations.

3.6 Data Construction and Annotation for Controllability

VoxCPM2 constructs controllability data by combining broad expressive speech with precise annotations, same-speaker references, and content-decoupled cloned examples. A staged mixture limits artifacts from self-synthesized training data.

  • Data scale: The corpus contains over 2 million hours of multilingual speech, with Chinese and English comprising the majority and 28 other languages ranging from roughly 1 K to 50 K hours each.Base TTS data is prepared through source separation, voice activity detection, ASR transcript alignment, and quality filtering.
  • Controllable data: Controllable-generation data combines tens of thousands of hours of open-source expressive speech with several thousand hours of internally curated annotations.The open-source portion broadens coverage, while internal data emphasizes annotation precision and richer descriptions.
  • Audio selection: Expressive samples are pre-screened with lightweight emotion classifiers after collection across diverse scenarios, avoiding acoustically flat utterances for annotation.The selection process targets speech with sufficient expressiveness to support controllability.
  • Annotation: Annotations describe voice-design attributes and style-control attributes using free-form natural-language descriptions generated by audio-understanding models.Voice attributes include age, gender, accent, vocal texture, and scenario; style attributes include emotion, rate, pitch, energy, and emphasis.
  • Reference mining: Same-speaker reference clips are mined from the recording session using speaker-embedding cosine similarity above 0.7, while clips directly preceding targets are excluded.This pool supports both reference cloning and controllable cloning.
  • Content decoupling: Cloned synthesis pairs an annotated control description with an unrelated transcript and generated audio to reduce style leakage from textual content.The synthetic data is used mainly in stage 2, while stage 3 restricts annealing to natively recorded high-quality speech to minimize artifacts.

3.7 Inference

VoxCPM2 generates speech autoregressively in latent patches and combines sampling, guidance, and streaming techniques to balance quality, speed, and continuity.

  • VoxCPM2 generates speech autoregressively, one latent patch at a time.
  • Classifier-free guidance: Classifier-free guidance combines conditional and unconditional LocDiT velocity fields, using α = 2.0 by default and a practical range of 1.5–3.0.
  • Sampling: Sway sampling allocates more solver steps to high-noise regimes, while CFG-Zero* reduces early-step artifacts without adding learnable parameters.
  • Streaming: Causal language models and patch-local diffusion enable chunk-based streaming, with each latent patch immediately decoded by a stateful AudioVAE V2 decoder.Continuation mode retains the last few prompt patches as decoder context for smoother transitions.

4 Experiments and Results

VoxCPM2 is evaluated across cloning, multilingual synthesis, controllability, reconstruction, efficiency, and subjective quality, showing competitive objective and perceptual performance across these capabilities.

  • Evaluation scope: The evaluation covers zero-shot cloning, multilingual synthesis, natural-language controllability, reconstruction quality, inference efficiency, and subjective listening tests.
  • Zero-shot voice cloning: 1.84/75.3 WER/SIM on test-EN, 0.97/79.5 on test-ZH, and 8.13/75.3 on test-ZH-Hard demonstrate competitive Seed-TTS-Eval cloning performance.The results reflect the reported WER/SIM pairs for the 2B-parameter model.
  • Zero-shot voice cloning: VoxCPM2 combines strong speaker similarity with good intelligibility and surpasses most token-based autoregressive systems in similarity.Subjective tests further report superior naturalness and prosody for cloning.
  • Multilingual capability: VoxCPM2 remains competitive across nine CV3-Eval languages, including hard-zh: 8.55 and hard-en: 8.48, despite its smaller 2B scale.The comparison is against Fish Audio S2, which uses larger scale and additional RL post-training.
  • Multilingual capability: VoxCPM2 achieves the highest SIM on 22 of 24 MiniMax-MLS-Test languages and strong intelligibility on several languages, while Arabic and Hindi remain weaker.The passage attributes the latter weakness partly to limited training data and evaluation effects from Whisper-large-v3.
  • Multilingual capability: On the internal 30-language benchmark, VoxCPM2 achieves an average WER/CER of 1.68%, with rates below 3% on 28 languages and below 1% on six.The results support a competitive and balanced multilingual TTS model within one unified hierarchical continuous-latent system.
  • Controllable generation: Voice design and controllable cloning use the same unified sequence organization without dedicated control heads or style encoders.The model supports natural-language voice and style control alongside 30 languages and 9 Chinese dialects.
  • Controllable generation: 84.2 / 83.2 / 71.4 on English APS, DSD, and RP respectively gives VoxCPM2 the best overall InstructTTSEval scores on that subset.On Chinese, it ties the top APS score at 85.2 and remains competitive on DSD and RP.

5 Conclusion and Future Work

VoxCPM2 provides a unified multilingual and controllable speech-generation foundation with competitive or state-of-the-art performance, while several quality, robustness, efficiency, and responsible-deployment challenges remain.

  • VoxCPM2 supports 48 kHz synthesis across 30 languages and 9 Chinese dialects, with zero-shot cloning, voice design, and fine-grained style control on one backbone.
  • Cross-lingual quality varies on some low-resource languages because of data-distribution imbalances.
  • Highly abstract or complex instructions remain difficult to capture consistently across languages, while large-scale serving still carries computational and deployment overhead.
  • Singing quality remains substantially improvable despite preliminary singing-voice generation capability.
  • Responsible deployment motivates stronger safeguards, including content provenance tracking, digital watermarking, and voice-cloning detection.

6 Contributors

VoxCPM2 is a collaborative release involving research labs at Tsinghua University and ModelBest, with additional support from the OpenBMB community.

  • VoxCPM2 was collaboratively released by THUHCSI, THUNLP, and ModelBest.
  • The authors acknowledge support from the OpenBMB community.
  • Core contributors include Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, and other listed researchers.
  • Other contributors are listed alphabetically, including Biyuan Lin, Caixian Chen, Chao Jia, and additional researchers.
Loading 2606.06928v1…