Source-linked AI summary
VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS
Wenbing Yang, Qihang Lu, Zihan Sun, Peilei Jia, Yingming Gao, Ya Li, Jun Gao
TL;DR
Text-driven TTS often leaves target voice design implicit, and text-based representations must capture both speaker identity and acoustic nuance. VoTSpeech predicts a hybrid-supervised voice latent that conditions language and acoustic generation; evaluations report improved instruction adherence and expressiveness while maintaining speech quality.
Problem
Instruction-following TTS often leaves voice design implicit, while text-driven voice representations must capture speaker identity and dynamic acoustic nuances.
Source: §1 · p. 1Method
VoTSpeech predicts a continuous voice latent from instructions, supervises it with speaker embeddings and pooled audio VAE features, and uses it to condition both the LLM and Acoustic DiT.
Source: §1 · p. 1, §2.2 · pp. 2–3Results
VoTSpeech improves instruction adherence and expressiveness while maintaining high speech quality; against direct fine-tuning, DSD accuracy rises by 3.5 percentage points.
Source: §4 · p. 4, §3.2 · p. 4Takeaways & Limitations
The evaluations support explicit voice modeling as a viable pathway for controllable speech synthesis.
Source: §4 · p. 4The voice latent is an utterance-level global descriptor; fine-grained temporal dynamics remain with the LLM’s sequence-modeling capacity.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .
1. INTRODUCTION
Text-driven TTS often leaves voice design implicit, while text-based voice representations must capture both speaker identity and expressive acoustic nuance. VoTSpeech separates voice design from speech generation with an explicit latent and reports competitive InstructTTSEval-ZH performance.
1. INTRODUCTION
- Existing instruction-following TTS often treats voice instructions as textual context, leaving target-voice creation implicit within utterance generation.The paper argues this coupling obscures explicit control over what voice to create versus what it says.Source: §1 · p. 1
- Text-driven voice representations must capture both speaker identity and dynamic acoustic nuances, a challenge not solved by static identity-focused representations alone.The paper identifies prediction and integration of such a multi-faceted latent within a unified architecture as non-trivial.Source: §1 · p. 1
- VoTSpeech predicts an explicit continuous voice latent from instructions and uses it to guide both LLM prosody planning and Acoustic DiT rendering.The latent is trained against a hybrid target combining speaker embeddings and pooled audio VAE features.Source: §1 · p. 1, Figure 1 · p. 2
- VoTSpeech uses a consistency-controlled annotation pipeline and reports competitive InstructTTSEval-ZH performance, with ablations validating core design choices.The introduction highlights descriptive-style and role-play tasks.Source: §1 · p. 1
2. METHOD
VoTSpeech separates instruction-guided voice design from speech generation using a predicted continuous voice latent. Its method constructs complementary voice annotations, predicts a hybrid-supervised latent, conditions both language and acoustic generation, and integrates prediction into a two-stage inference workflow.
2.1. Instruction Data Construction
- Instruction–speech pairs distinguish persistent speaker traits from utterance-level delivery dynamics.Speaker-level descriptions are assigned across a speaker’s recordings; utterance-level instructions describe localized attributes such as pitch, stress, tempo, and emotional trajectory.Source: §2.1 · p. 2
2.2. Model Architecture
- VoTSpeech inserts a voice-generation stage before synthesis text, predicting a continuous voice embedding from instruction context without access to the target text.The Voice Patch Encoder combines speaker identity with temporally pooled acoustic features, and the flow-matching Voice DiT predicts the latent from Gaussian noise.Source: §2.2 · pp. 2–3
- The predicted voice latent conditions both LLM sequence modeling and Acoustic DiT rendering, unlike acoustic-only voice conditioning.Learned projections form an LLM prefix embedding and an Acoustic DiT condition; zero-initialized residual paths provide direct instruction context.Source: §2.2 · p. 3
2.3. Training and Inference
- Voice DiT and speech generation are jointly trained with speaker and audio-VAE encoders frozen, while stop-gradient prevents the voice loss from altering Causal LLM representations.Scheduled sampling progressively increases conditioning on generated voice latents across both pathways; inference first predicts the voice latent, then reuses cached instruction states to generate speech latents autoregressively.Source: §2.3 · p. 3
3. EXPERIMENTS
VoTSpeech performs strongly against established systems and direct fine-tuning; ablations favor dual-path conditioning and hybrid acoustic targets, while the discussion notes limits of its global voice latent.
3.1. Experimental Setup
- Evaluation compares VoTSpeech with four instruction-guided systems and same-backbone direct fine-tuning, using uniform Gemini judging and listener ratings.Training uses 1,543.52 hours of unique Chinese audio for 20k steps.Source: §3.1 · pp. 3–4
3.2. Experimental Results
- VoTSpeech leads DSD (78.5%) and RP (66.9%), with competitive APS (84.8%), low CER (2.58%), and top NMOS (4.12), EMOS (4.30), and AMOS (4.26).Compared with direct fine-tuning, DSD rises by 3.5 percentage points, RP by 2.6 percentage points, and instruction adherence by 0.30.Source: Table 1 · p. 4, §3.2 · p. 4
- Removing LLM-side voice conditioning degrades all tasks, including a 3.0 percentage-point RP drop, despite retaining Acoustic DiT conditioning.Source: Table 2 · p. 4, §3.2 · p. 4
- Without VAE target features, APS gains 0.3 percentage points, but DSD and RP fall by 0.8 and 3.3 percentage points, respectively.This variant retains the dual-path interface but trains the target using only static speaker embeddings.Source: Table 2 · p. 4, §3.2 · p. 4
- The utterance-level global latent omits frame-level details, and pooled VAE features can sometimes encode recording traits alongside target timbre.Fine-grained pitch and pacing remain dependent on the LLM sequence model.Source: §3.2 · p. 4
4. CONCLUSION
VoTSpeech combines a hybrid target latent with dual-path conditioning to unify LLM prosody planning and DiT acoustic rendering. Evaluations report improved instruction adherence and expressiveness while maintaining high speech quality.
- Evaluations show improved instruction adherence and expressiveness while maintaining high speech quality, supporting explicit voice modeling as a viable pathway for controllable speech synthesis.Source: §4 · p. 4
5. COMPLIANCE WITH ETHICAL STANDARDS
The voluntary anonymous listening test obtained informed consent, collected no personally identifiable information, and reports only aggregate ratings.
- The voluntary anonymous listening test obtained informed consent, collected no personally identifiable information, and reports only aggregate ratings.Source: §5 · p. 5