Source-linked AI summary
OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech
Yong Ren, Jiangyan Yi, Jianhua Tao, Haiyang Sun, Zhengqi Wen, Hao Gu, Le Xu, Ye Bai
TL;DR
Existing InstructTTS systems are constrained by instructions built from low-level acoustic attributes, limiting flexible control of expressive speech. OV-InstructTTS introduces the OV-Speech dataset and a reasoning-driven framework that infers emotional, acoustic, and paralinguistic information before synthesis. The approach substantially improves instruction following, naturalness, and speaker similarity over baselines.
Problem
Existing InstructTTS methods rely on combinations or rephrasings of acoustic labels, making flexible high-level instructions difficult to support.
Method
OV-InstructTTS combines the OV-Speech dataset with OV-InstructTTS-TEP, which reasons from open-vocabulary instructions to enriched text and speech generation.
Results
OV-InstructTTS-TEP achieves the best overall objective and subjective performance, including Gemini Score 70.42, MOS 4.28, ICMOS 3.91, and SIM 0.722.
Takeaways & Limitations
Reasoning and enriched transcriptions improve alignment between open-vocabulary instructions and expressive synthesized speech.
Abstract
from arXiv · showhide
Instruct Text-to-Speech (InstructTTS) leverages natural language descriptions as style prompts to guide speech synthesis. However, existing InstructTTS methods mainly rely on a direct combination of audio-related labels or their diverse rephrasings, making it difficult to handle flexible, high-level instructions. Such rigid control is insufficient for users such as content creators who wish to steer generation with descriptive instructions. To address these constraints, we introduce OV-InstructTTS, a new paradigm for open-vocabulary InstructTTS. We propose a comprehensive solution comprising a newly curated dataset, OV-Speech, and a novel reasoning-driven framework. The OV-Speech dataset pairs speech with open-vocabulary instructions, each augmented with a reasoning process that connects high-level instructions to acoustic features. The reasoning-driven framework infers emotional, acoustic, and paralinguistic information from open-vocabulary instructions before synthesizing speech. Evaluations show that this reasoning-driven approach significantly improves instruction-following fidelity and speech expressiveness. We believe this work can inspire the next user-friendly InstructTTS systems with stronger generalization and real-world applicability. The dataset and demos are publicly available on our project page.
1. INTRODUCTION
Existing InstructTTS systems improve controllability through natural-language prompts but remain tied to rephrased acoustic attributes. OV-InstructTTS addresses this gap with open-vocabulary instructions, a reasoning-driven framework, and the OV-Speech dataset.
- Motivation: Existing InstructTTS methods derive instructions mainly from predefined acoustic attributes, limiting flexible expression of high-level user intent.These systems control factors such as pitch, speaking rate, and emotion, while InstructTTSEval’s role-play task remained limited and offered no solution.
- Contributions: OV-Speech pairs speech with narrative-context-derived instructions, reasoning chains linking instructions to acoustics, and paralinguistic-aware transcriptions.Its instructions are not merely reformulations of acoustic or emotional labels, enabling diverse and unconstrained expression while preserving consistency with target speech.
- Proposed paradigm: OV-InstructTTS shifts controllable speech synthesis toward direct alignment with users’ intentions through open-vocabulary instructions.The paradigm is intended to make InstructTTS more user-friendly and flexible for real-world applications.
- Contributions: OV-InstructTTS-TEP uses a reasoning process to interpret open-ended instructions and synthesize expressive speech consistent with user intent.The framework is based on a large audio language model and explicitly bridges high-level instructions with low-level acoustic information.
- Contributions: LLM-based and subjective evaluations confirm consistency between synthesized speech and open-ended instructions, while experiments and ablations demonstrate the value of the dataset and framework.The supplied introduction reports the evaluation conclusions without specifying numerical results.
2. THE OV-SPEECH DATASET CONSTRUCTION
OV-Speech addresses the lack of paired speech and open-vocabulary instructions by extending ContextSpeech through a five-stage, LLM-assisted construction pipeline. The resulting dataset combines contextual instructions, consistency filtering, reasoning annotations, and paralinguistic tags.
- Dataset overview: OV-Speech extends the 476.8-hour ContextSpeech corpus with open-vocabulary instruction–speech pairs through a five-stage construction pipeline.The source contains multi-speaker audiobook speech, novels, single-sentence utterances, transcriptions, emotional labels, and acoustic descriptions.
- Contextual Information Extraction: Contextual information extraction uses 1000 words before and after each utterance to derive environment, event, personality, interlocutor state, and speaker intent.Qwen3-32B distills the surrounding narrative into structured contextual elements relevant to the speech performance.
- Open-Vocabulary Instruction Generation: Open-vocabulary instructions are generated by randomly selecting 2–5 contextual elements and prompting Qwen3-32B to produce director-like descriptions.Random selection combined with creative generation yields varied, unconstrained instructions grounded in narrative context.
- Consistency Filtering: Consistency filtering compares LLM-predicted emotional and acoustic attributes with ContextSpeech ground-truth labels before retaining instruction–speech examples.The process uses speaker identity, transcript, contextual information, Deepseek-R1 predictions, and Qwen3-32B judging.
- Reasoning and transcription annotation: Passing samples receive two-stage reasoning chains that deconstruct instructions and infer acoustic attributes, alongside transcriptions enriched with 18 paralinguistic tags.These annotations connect high-level instructions to low-level acoustics and provide more informative TTS prediction targets.
- Dataset outcome: The completed OV-Speech dataset provides open-vocabulary instructions, reasoning chains, and fine-grained paralinguistic transcriptions as a foundation for OV-InstructTTS.The pipeline is designed to supply comprehensive supervision for the proposed task.
3. THE OV-INSTRUCTTTS-TEP FRAMEWORK
OV-InstructTTS-TEP replaces a direct instruction-to-speech mapping with explicit reasoning followed by enriched text and audio generation. The reasoning step infers emotion, acoustic descriptions, and paralinguistic cues before synthesis.
- Framework design: OV-InstructTTS-TEP explicitly generates a reasoning chain before synthesizing speech, using LALMs’ comprehension, reasoning, and generation capabilities.This structured two-step design is intended to bridge the semantic gap between high-level instructions and low-level acoustic realizations.
- Inference and synthesis: For a complex situational instruction, the model infers emotion labels, acoustic descriptions, and paralinguistic tags, then predicts an enriched transcript and corresponding audio.The example includes inferred emotions such as doubt, contempt, and displeasure, plus a breathing tag before speech synthesis.
4. EXPERIMENTS
Experiments evaluate OV-InstructTTS-TEP against strong TTS and LALM baselines using objective and subjective measures, then isolate the contributions of OV-Speech, reasoning, and enriched transcription tokens.
- Experimental Setup: The evaluation compares against GPT4o, Higgs-Audio-V2, CosyVoice2, and Step-Audio-2-mini, selected for strong naturalness or instruction-following capabilities.Metrics cover instruction following, intelligibility, speaker similarity, naturalness, and instruction consistency through automated and human evaluations.
- Main Results: OV-InstructTTS-TEP achieves the best overall performance across objective and subjective metrics, including Gemini Score 70.42, MOS 4.28, ICMOS 3.91, SIM 0.722, and CER 3.61%.The model also attains Gemini Rank 3.39/6, and its MOS exceeds GroundTruth recordings.
- Ablation Study: Fine-tuning on OV-Speech raises Gemini Score from 61.49 to 66.34 without instructions and from 63.18 to 67.70 with instructions, while MOS and ICMOS also improve.The instruction-based fine-tuned variant consistently outperforms its no-instruction counterpart.
- Ablation Study: Adding explicit reasoning increases Gemini Score from 67.70 to 68.71 and improves MOS from 4.23 to 4.27 and ICMOS from 3.74 to 3.90.The study links this reasoning step to bridging high-level instructions and low-level acoustic attributes.
- Ablation Study: Enhanced transcription tokens alone slightly reduce Gemini Score from 67.70 to 66.98, but combining them with reasoning produces Gemini Score 71.57 and Gemini Rank 3.89/8.The combined configuration also improves MOS and ICMOS, supporting synergy between reasoning and fine-grained transcription.
5. CONCLUSION
The paper introduces OV-InstructTTS, OV-Speech, and OV-InstructTTS-TEP to support flexible speech control from open-vocabulary instructions. Experiments show gains in instruction following, naturalness, and speaker similarity, with reasoning and enriched text prediction being crucial.
- 5. CONCLUSION: OV-InstructTTS addresses the limitation of instructions confined to low-level acoustic attributes by enabling more intuitive and flexible controllable speech synthesis.The paradigm is supported by OV-Speech, which includes open-vocabulary instructions, reasoning annotations, and paralinguistic-aware transcriptions.
- 5. CONCLUSION: OV-InstructTTS-TEP first reasons over an instruction and then interleaves enriched text-token and audio-token generation before synthesizing speech.The conclusion identifies reasoning and enriched text token prediction as crucial components.
- 5. CONCLUSION: Extensive experiments show that the approach substantially outperforms baselines in instruction following, naturalness, and speaker similarity.The paper presents the method as a step toward intuitive, flexible, and user-centric controllable TTS.