Source-linked AI summary
Poly-InstructTTS: Learning In-the-Wild Expressive Speech Synthesis from Open-Ended Instructions
Junhui Zhang, Qianhui Xu, Qingxiang Guo, Dawei Yang, Ling Miao, Qiangqiang Wang, Yang Song
TL;DR
Fine-grained natural-language control remains challenging for expressive TTS because existing systems face instruction-interpretation, prompt-conflict, and data-coverage limitations. Poly-InstructTTS addresses these issues with in-the-wild instruction data, a prompt-free GPT-FM architecture, and speaker fine-tuning. It achieves strong objective and subjective performance, while evaluations identify acoustic-condition weaknesses and a stability–expressiveness trade-off.
Problem
Expressive TTS systems struggle with complex natural-language instructions, conflicting prompt audio, and training data that underrepresent subtle emotions, styles, and paralinguistic behaviors.
Method
Poly-InstructTTS combines a 1,000-hour in-the-wild instruction corpus with a prompt-free GPT using attribute-based thinking tokens, FM timbre injection, and instruction-conditioned speaker fine-tuning.
Results
Poly-InstructTTS achieves strong objective and subjective performance, with consistently high APS, DSD, and RP scores and top-1 I-MOS on the base testset and top-2 on the expanded set.
Takeaways & Limitations
The framework supports expressive instruction following across fine-grained emotions, styles, and paralinguistic behaviors while extending evaluation coverage through a broader testset.
Takeaways & Limitations
Performance weakens under noise and echo, while increasing expressiveness improves APS, DSD, and RP but raises WER; the system also still relies on reference-conditioned timbre.
Abstract
from arXiv · showhide
While recent text-to-speech (TTS) models achieve high naturalness, controlling fine-grained expression via natural-language instructions remains challenging. We introduce Poly- InstructTTS, which learns expressive speech from open-ended instructions using in-the-wild audiovisual data. We build a scalable multi-modal pipeline to construct a 1,000-hour instruction-annotated corpus covering 1,000+ fine-grained emotions and styles. The framework uses a prompt-free GPT with attribute-based thinking tokens, followed by a flow-matching module that injects timbre from a reference audio. We also present a speaker fine-tuning procedure to transfer instruction control to specific speakers while preserving persona. We further extend InstructTTSEval with broader tasks. Experiments show that Poly-InstructTTS delivers strong performance in instruction adherence and expressiveness. Audio demos and the expanded testset are available on our project page.
1. Introduction
Poly-InstructTTS addresses limited fine-grained instruction control by learning from diverse expressive audiovisual data. Its framework combines a scalable instruction-data pipeline with prompt-free GPT-based synthesis and attribute-guided acoustic generation.
- Natural-language control remains difficult because models must interpret complex instructions, while prompt audio can conflict with requested styles and degrade controllability.
- Expressive speech data are scarce: common datasets are dominated by neutral reading, while emotion-labeled datasets cover few categories and lack natural-language descriptions.
- Poly-InstructTTS learns expressive speech synthesis from diverse natural-language instructions for fine-grained emotions, styles, and paralinguistic behaviors.
- The system constructs diverse instruction–audio pairs from cinematic and television audiovisual media through a multi-modal data-processing pipeline.
- Its framework uses a prompt-free GPT with attribute-based thinking tokens, an FM module for timbre injection, and speaker fine-tuning for specific-speaker control.
- The authors extend InstructTTSEval with more diverse evaluation tasks for the open-source community.
2. Related Works
Prior instruction-TTS work relies heavily on limited or neutral speech data and, in some systems, audio prompts that risk style leakage. Poly-InstructTTS instead targets in-the-wild expressive media and uses attribute-based thinking tokens within a GPT-FM design.
- Instruction-controllable TTS depends on high-quality instruction–text–audio pairs, progressing from acoustic tags and rules toward LLM-generated multi-attribute descriptions.
- Most existing instruction datasets use audiobooks, podcasts, open-source corpora, or synthesized speech, limiting coverage of paralinguistic phenomena and extreme emotions.
- Poly-InstructTTS targets cinematic and television data to obtain expressive, context-driven speech beyond standardized reading tones.
- Some prior GPT-based systems still require audio prompts, creating potential style leakage when prompt acoustics conflict with the intended instruction.
- Unlike free-form thinking-text supervision, the proposed architecture uses compact attribute-based thinking tokens to steer style generation within a GPT-FM framework.
3. Method
The method builds instruction–audio pairs from cinematic and television media, then synthesizes speech with a prompt-free GPT-FM architecture. Attribute-based thinking tokens guide prosody and style, while reference audio supplies timbre and speaker fine-tuning transfers control to target speakers.
- Data Source: The data source is a 1,000-hour in-the-wild cinematic corpus containing expressive, context-driven interactions, emotions, paralinguistic behaviors, and accents.
- Data Source: Commercial cinematic media cannot be publicly distributed because of copyright restrictions, although the pipeline and LLM prompts are released for reproduction on other corpora.
- Data Processing: The preprocessing pipeline segments videos, extracts and denoises center-channel audio, and uses ASR, speaker diarization, and subtitle matching to obtain clean utterances.
- Instruction Annotation: A multi-modal LLM summarizes context, annotates transcripts with gender, accent, emotion, and style, then generates vivid natural-language instructions for each utterance.
- Poly-InstructTTS Framework: The GPT-FM architecture maps instructions and text to speech tokens, converts them to mel-spectrograms, and reconstructs waveforms while injecting reference timbre only in the acoustic module.
- Poly-InstructTTS Framework: Attribute-based thinking tokens bridge instructions and acoustic tokens using gender, emotion intensity, style, and accent attributes.
- Speaker Fine-Tuning: Instruction-Conditioned Speaker Fine-Tuning prepends a Speaker ID to transfer expressive variance to target speakers while maintaining their intrinsic persona.
4.1. Experimental Setup
The experiments evaluate Poly-InstructTTS on base and expanded instruction-following testsets using objective stability and adherence metrics alongside human judgments. The setup broadens coverage to accents, emotions, styles, and role-play scenarios.
- Dataset: Approximately 2,500 raw hours were processed into 1,000 hours with over 1.1M utterances, covering over 200 accents, 800 emotions, and 400 stylistic variations.
- Dataset: The instruction–audio match rate exceeds 95% according to human ratings, and the corpus is randomly split into 99% training and 1% validation data.
- Evaluation Testset: Evaluation uses the original InstructTTSEval benchmark together with an expanded 200-sample testset targeting underrepresented accents, subtle and extreme emotions, and non-mainstream styles.
- Metrics: Objective evaluation measures WER for stability and APS, DSD, and RP for acoustic instructions, descriptive styles, and role-play ability.
- Human Evaluation: Subjective evaluation uses 20 native listeners rating instruction-following accuracy and speech naturalness and stability on 5-point I-MOS and N-MOS scales.
- Implementation: The model is trained for 30 epochs on 8 NVIDIA A800 GPUs with AdamW at a learning rate of 1e-4 and uses CosyVoice components during inference.
4.2. Evaluation Results
Poly-InstructTTS performs strongly on instruction-following and expressive-speech evaluation, while revealing a stability–expressiveness trade-off and weaknesses in challenging acoustic conditions.
- Poly-InstructTTS achieves consistently high APS/DSD/RP scores against open- and closed-source instruction-following TTS systems, with the largest gains on RP.The authors associate the RP gains with in-the-wild training data and attribute-based thinking tokens.
- The expanded InstructTTSEval comparison covers closed-source APIs, open-source models, and proposed-system ablations on base and expanded testsets.The table caption identifies the evaluated system categories and both testset variants.
- As training epochs increase, APS/DSD/RP steadily improve while WER rises, exposing a trade-off between expressiveness and stability.The authors hypothesize that extreme emotions and paralinguistic behaviors weaken monotonic text–token alignment.
- The system is weaker under challenging acoustic conditions such as noise and echo, with the weakness appearing more related to the FM than the GPT pathway.
4.3. Ablation Studies
Ablations support attribute-based thinking tokens as useful instruction-to-acoustic priors, while additional pretrained instruction encoders do not improve the encoder-free design.
- Removing attribute-based thinking tokens lowers subjective scores, indicating that they help align instructions with acoustic tokens.The tokens complement free-form instruction text and are associated with better expressiveness than direct generation.
- Frozen FlanT5, instructor, and GTR-base instruction representations do not yield positive gains over the encoder-free default.Under the reported training setup, raw instruction text is sufficient for generating styled speech.
4.4. Instruction-Conditioned SFT Analysis
Instruction-conditioned speaker fine-tuning improves persona preservation and naturalness, but the base setup retains higher instruction adherence.
- The study compares instruction-conditioned SFT with the base Poly-InstructTTS using only FM-stage timbre injection and the same timbre reference.The comparison evaluates whether speaker-specific instruction control can be added while preserving speaker identity.
- The subjective evaluation covers 10 fine-tuned speakers and reports I-MOS, N-MOS, and P-MOS, with P-MOS measuring alignment to the speaker persona.
- SFT attains higher P-MOS and N-MOS, whereas the base setup achieves higher I-MOS.The authors interpret SFT as learning a speaker-specific mapping to a suitable stylistic range, with a trade-off in instruction adherence.
- Table 3 presents a subjective comparison between the proposed Instruction-Conditioned SFT and the base method.
5. Conclusion
Poly-InstructTTS combines in-the-wild expressive-speech data with prompt-free instruction control and speaker fine-tuning. The authors identify future work on balancing expressiveness and stability, improving FM robustness, and removing reference-audio dependence.
- The paper constructs a 1,000-hour in-the-wild cinematic corpus covering 1,000+ fine-grained emotions and styles.
- Poly-InstructTTS uses a prompt-free GPT with attribute-based thinking tokens, an FM framework, and an expanded InstructTTSEval testset.
- Instruction-conditioned SFT gives target speakers instruction controllability while preserving their personas.
- Future work targets better expressiveness–stability balance, improved FM performance under complex conditions, and text-only reference-free voice generation.The reference-free goal would remove the need for a timbre reference audio while preserving controllability.
7. Generative AI Use Disclosure
OpenAI’s GPT-5 and Google’s Gemini 3.0 Pro were used only to polish English and improve readability during manuscript preparation. The authors state that all technical content was human-authored, reviewed, and verified.
- OpenAI’s GPT-5 and Google’s Gemini 3.0 Pro were used solely for English-language polishing and readability improvements.
- The tools were not used to generate scientific ideas, methods, or experimental results.
- The human authors authored the technical content, analyses, claims, and conclusions.
- The authors reviewed and verified all AI-assisted edits and take responsibility for the final text.