Source-linked AI summary
JoyAI-Voice 2.0: A Full-Continuous Autoregressive Speech Generation Model with Semantic-Acoustic Joint Representation
Yafeng Chen, Boya Dong, Yankun Huang, Hao Li, Jingdong Li, Xiangyu Liang, Hao Ni, Wenchao Wang, Yuxuan Wang, Zhangyu Xiao, Wei Deng, Nan Duan, Yu Gu, Wenhao Guan, Weisheng Han, Yabin Li, Yuan Liu, Jiaxin Ye, Fan Yu, Lin Zhu
TL;DR
Continuous autoregressive speech models can be unstable over long rollouts, while discrete speech tokenization can discard acoustic detail. JoyAI-Voice 2.0 combines continuous latents and semantic-acoustic dual encoding with autoregressive planning and local diffusion rendering, and reports strong voice-cloning and voice-design results.
Problem
Continuous-representation autoregressive models can accumulate deviations across long rollouts, while discrete-token systems can lose timbre micro-textures and paralinguistic details through quantization.
Source: §1 · p. 2Method
The model uses continuous AudioVAE latents, complementary semantic and acoustic patch representations, autoregressive planning, and local diffusion rendering.
Source: §1 · p. 3, §2.1 · p. 3, §2.3 · p. 4Results
Takeaways & Limitations
The model supports zero-shot voice cloning, instruction-based voice design, controllable paralinguistic tags, and streaming inference at low first-packet latency.
Source: §5 · p. 9
Abstract
from arXiv · showhide
We present JoyAI-Voice~2.0, an end-to-end anthropomorphic speech generation model built upon a fully continuous, dual-encoder architecture. Raw speech is encoded into continuous latents and partitioned into patches. Each patch is decomposed by a semantic-acoustic dual encoder into a semantically purified representation and an acoustic representation, which are fused and jointly fed to a causal autoregressive Transformer for planning. The Transformer predicts the conditioning for the next patch, and a local diffusion Transformer renders its full latents for 48\,kHz synthesis. The model is trained with a joint flow-matching and stop-prediction objective, followed by supervised fine-tuning and reinforcement learning with DiffusionNFT to improve model performance. It achieves the lowest average word error rate of 2.51\% on Seed-TTS, a 14.9\% relative reduction over the strongest baseline, and state-of-the-art attribute fidelity on InstructTTSEval in both Chinese and English, leading on 5 of 10 perceptual dimensions with the highest overall score of 0.893 on MDVD-Eval.
1 Introduction
JoyAI-Voice 2.0 adopts fully continuous autoregressive speech generation, using a semantic-acoustic representation to address rollout instability while retaining acoustic detail. It reports strong results for voice cloning and instruction-based voice design.
1 Introduction
- Continuous representations avoid the quantization bottleneck that can discard timbre micro-textures and paralinguistic detail, unlike discrete-token systems.The authors also report that JoyVoice 1.0's expressiveness was bounded by its codec's representation capacity.Source: §1 · p. 2
- Autoregressive generation is chosen to support incremental, prompt-conditioned streaming as text is emitted, targeting millisecond-level first-packet latency.The authors contrast this with non-autoregressive diffusion systems' strength in single-pass offline synthesis.Source: §1 · p. 2
- A frozen, ASR-distilled semantic representation anchors planning against acoustic nuisances and long-range error feedback, while a learned acoustic representation restores fine-grained details.The authors present this separation as improving training stability and model performance.Source: §1 · p. 2
Fully Continuous Autoregressive Modeling with a Dual-Encoder Architecture
- The model combines fully continuous autoregressive planning with a per-patch flow-matching diffusion head over AudioVAE latents, eliminating discrete speech tokenization.The fused semantic and acoustic representations are jointly fed to the language model for planning.Source: §1 · pp. 2–3
Zero-Shot Voice Cloning; Instruction-Based Voice Design
- 2.51% average word error rate on Seed-TTS is a 14.9% relative reduction over the strongest baseline; voice design leads on 5 of 10 MDVD-Eval dimensions with a 0.893 overall score.The voice design variant also attains state-of-the-art attribute fidelity on InstructTTSEval in both Chinese and English.Source: §1 · p. 3
2 Full-Continuous Dual-Encoder Architecture
JoyAI-Voice 2.0 keeps speech generation in continuous latent space: dual encoders provide complementary patch representations for autoregressive planning, followed by local diffusion rendering.
2 Full-Continuous Dual-Encoder Architecture
- Speech is compressed into continuous AudioVAE latents, divided into fixed-length patches, jointly planned by a causal Transformer, then rendered patchwise and decoded as 48 kHz waveforms.No discrete tokenization occurs in the pipeline.Source: §2 · p. 3
2.1 Semantic-Acoustic Joint Representation
- Each patch's semantic summary filters noise for stable long-rollout planning, while its acoustic representation retains spectral texture, speaker timbre, and micro-prosody for high-fidelity rendering.The two representations are fused and jointly consumed by the language model.Source: §2.1 · p. 3
2.1.1 Semantic Encoder
- Whisper-large-v3 supplies the semantic encoder backbone, trained only on ASR transcription supervision to exclude acoustic nuisances.Its decoder head is discarded at inference.Source: §2.1.1 · pp. 3–4
2.1.2 Acoustic Encoder
- A lightweight Transformer acoustic encoder pools each patch's latent frames into one vector, preserving fine-grained details omitted by the semantic representation.It is trained jointly with the rest of the model.Source: §2.1.2 · p. 4
2.2 Causal Autoregressive Transformer
- A decoder-only causal Transformer predicts each next patch's conditioning from the text prompt and preceding patches, supporting long-range coherence and incremental streaming.The backbone is built on Qwen2.5-1.5B [19].Source: §2.2 · p. 4
2.3 Local Diffusion Transformer
- A local diffusion Transformer renders the next patch's full latents from planned conditioning, using prior generated latents as prefix context for contextual outpainting.Flow matching trains the renderer; the frozen VAE decoder produces 48 kHz waveforms.Source: §2.3 · p. 4
3 Training
Training proceeds from autoregressive acoustic-patch pretraining to task-specific supervised fine-tuning and DiffusionNFT reinforcement learning. The objectives combine flow matching with termination prediction, while reinforcement learning balances text accuracy and speaker similarity.
3.1 Pre-Training & Supervised Fine-Tuning
- Under teacher forcing, the autoregressive backbone conditions each next patch on fused representations of preceding ground-truth patches, with text as a prefix.The AudioVAE, speaker encoder, and semantic encoder backbone remain frozen during speech-generation training.Source: §3.1 · p. 5
- Pretraining jointly uses patch-level flow matching to generate acoustic patches and an auxiliary classifier to predict utterance termination.The diffusion Transformer predicts target velocity from autoregressive states, preceding acoustic patches, and speaker embeddings; the EOS loss updates only its classifier.Source: §3.1 · p. 5
- Supervised fine-tuning creates separate base and instruction-conditioned variants from a shared checkpoint, using speech-text pairs and, for instruction tuning, speaker, style, and emotion annotations.The instruction is conditioning rather than a text-generation target.Source: §3.1 · p. 5
3.2 Reinforcement Learning
- After SFT, DiffusionNFT reinforcement learning targets synthesis accuracy on long, challenging text and speaker similarity in zero-shot voice cloning.Training adds word-, phrase-, and clause-level repetition to broaden the text patterns encountered during RL.Source: §3.2 · p. 6
- RL uses continuation-based zero-shot cloning: CFG-guided samples prompted by reference speech and transcript provide acoustic-latent targets for the fully conditioned text-and-speaker branch.Multiple continuations are sampled with different noise initializations.Source: §3.2 · p. 6
- The reward combines negative WER and speaker-embedding cosine similarity, shifting from a 5:1 ASR-to-speaker weight early to 1:5 as optimization turns toward speaker fidelity.The ASR reward becomes less informative about further progress in later RL stages.Source: §3.2 · p. 6
4 Experiments
Experiments prepare continuous-latent training records and evaluate zero-shot voice cloning and voice design. JoyAI-Voice 2.0 leads on average recognition error and several voice-design measures, while English role-play and speaker similarity remain less strong.
4.1 Data
- A distributed DAG pipeline converts large-scale audio into structured training records through audio cleanup, transcription validation, encoding, and metadata alignment.Parallel ASR outputs are cross-validated by pairwise WER, while a speaker-embedding model and frozen AudioVAE produce speaker representations and continuous latents.Source: §4.1 · pp. 6–7
4.2 Evaluation
- Evaluation covers Seed-TTS voice cloning and InstructTTSEval voice design, with MDVD-Eval assessing whether multiple instructed attributes are realized together.MDVD-Eval uses 1,000 Chinese cases combining 2–4 attributes; Gemini-2.5-Pro is selected as judge after achieving the highest Atomic Agreement with nine researchers (79.9%).Source: §4.2 · p. 7, Table 1 · p. 7
4.3.1 Voice Cloning
- 2.51% average recognition error gives JoyAI-Voice2.0-Base the lowest Seed-TTS result, a 14.9% reduction versus the 2.95% dots.tts baseline.It also has the lowest CER on test-zh (0.74%) and test-hard (5.53%), while Qwen3-TTS-12Hz-1.7B-Base leads test-en WER at 1.23% versus 1.27%; JoyAI's speaker similarity is competitive but not top-ranking.Source: §4.3.1 · p. 7, Table 2 · p. 8
4.3.2 Voice Design
- JoyAI-Voice2.0-VoiceDesign leads or ties on most InstructTTSEval subsets, but English role-play remains a weakness at 63.4% versus FireRedTTS3-Instruct’s 72.0%.It leads APS on ZH (90.1) and EN (84.4), leads ZH-DSD (83.3), ties EN-DSD at 82.3, and leads ZH-RP at 69.8; baseline results were re-evaluated under Gemini-2.5-Pro.Source: Table 3 · p. 8, §4.3.2 · p. 8
- On MDVD-Eval, JoyAI-Voice2.0-VoiceDesign scores 0.893 overall, leads on 5 of 10 dimensions, and has particularly strong age and gender results.Gender is near-perfect across most systems, while timbre is the most challenging dimension overall; commercial systems were evaluated through their official APIs.Source: Table 4 · p. 9, §4.3.2 · p. 9
5 Conclusion
JoyAI-Voice 2.0 combines continuous latents with semantic-acoustic separation for autoregressive planning and patchwise waveform rendering. The report concludes that it performs strongly across voice cloning and voice-design benchmarks and supports streaming and controllable speech generation.
5 Conclusion
- A 1.5B causal language model plans over the semantic stream, while a local flow-matching diffusion Transformer renders full continuous latents patch by patch into 48 kHz waveforms.The semantic-acoustic separation is described as shielding planning from acoustic-noise propagation and closing the error-feedback loop of open-loop continuous rollouts.Source: §5 · p. 9
- Seed-TTS average recognition error is 2.51%, a 14.9% reduction over the strongest baseline; voice design is state-of-the-art on InstructTTSEval and leads on 5 of 10 MDVD-Eval dimensions.The voice-design variant scores 0.893 overall on MDVD-Eval; the report also highlights robustness to noisy references and paralinguistic tags such as [Laughter] and [Sigh].Source: §5 · p. 9
- The model also supports instruction-based voice design, zero-shot cloning, paralinguistic tags, and streaming inference for low first-packet latency.The conclusion frames these capabilities as suited to anthropomorphic, empathetic, real-time conversational deployment.Source: §5 · p. 9