Poly-InstructTTS: Learning In-the-Wild Expressive Speech Synthesis from Open-Ended Instructions

• Natural-language control remains difficult because models must interpret complex instructions, while prompt audio can conflict with requested styles and degrade controllability.Source: §1 · p. 1
• Expressive speech data are scarce: common datasets are dominated by neutral reading, while emotion-labeled datasets cover few categories and lack natural-language descriptions.Source: §1 · p. 1
• Poly-InstructTTS learns expressive speech synthesis from diverse natural-language instructions for fine-grained emotions, styles, and paralinguistic behaviors.Source: §1 · p. 1
• The system constructs diverse instruction–audio pairs from cinematic and television audiovisual media through a multi-modal data-processing pipeline.Source: §1 · p. 1
• Its framework uses a prompt-free GPT with attribute-based thinking tokens, an FM module for timbre injection, and speaker fine-tuning for specific-speaker control.Source: §1 · p. 1
• The authors extend InstructTTSEval with more diverse evaluation tasks for the open-source community.Source: §1 · p. 1

• Instruction-controllable TTS depends on high-quality instruction–text–audio pairs, progressing from acoustic tags and rules toward LLM-generated multi-attribute descriptions.Source: §2.1 · p. 1
• Most existing instruction datasets use audiobooks, podcasts, open-source corpora, or synthesized speech, limiting coverage of paralinguistic phenomena and extreme emotions.Source: §2.1 · p. 1
• Poly-InstructTTS targets cinematic and television data to obtain expressive, context-driven speech beyond standardized reading tones.Source: §2.1 · p. 1
• Some prior GPT-based systems still require audio prompts, creating potential style leakage when prompt acoustics conflict with the intended instruction.Source: §2.2 · p. 1
• Unlike free-form thinking-text supervision, the proposed architecture uses compact attribute-based thinking tokens to steer style generation within a GPT-FM framework.Source: §2.2 · p. 2

Data Source

• The data source is a 1,000-hour in-the-wild cinematic corpus containing expressive, context-driven interactions, emotions, paralinguistic behaviors, and accents.Source: §3.1.1 · p. 2
• Commercial cinematic media cannot be publicly distributed because of copyright restrictions, although the pipeline and LLM prompts are released for reproduction on other corpora.Source: §3.1.1 · p. 2

Data Processing

• The preprocessing pipeline segments videos, extracts and denoises center-channel audio, and uses ASR, speaker diarization, and subtitle matching to obtain clean utterances.Source: §3.1.2 · p. 2

Instruction Annotation

• A multi-modal LLM summarizes context, annotates transcripts with gender, accent, emotion, and style, then generates vivid natural-language instructions for each utterance.Source: §3.1.3 · p. 2

Poly-InstructTTS Framework

• The GPT-FM architecture maps instructions and text to speech tokens, converts them to mel-spectrograms, and reconstructs waveforms while injecting reference timbre only in the acoustic module.Source: §3.2 · p. 2
• Attribute-based thinking tokens bridge instructions and acoustic tokens using gender, emotion intensity, style, and accent attributes.Source: §3.2 · pp. 2–3

Speaker Fine-Tuning

• Instruction-Conditioned Speaker Fine-Tuning prepends a Speaker ID to transfer expressive variance to target speakers while maintaining their intrinsic persona.Source: §3.3 · p. 3

Dataset

• Approximately 2,500 raw hours were processed into 1,000 hours with over 1.1M utterances, covering over 200 accents, 800 emotions, and 400 stylistic variations.Source: §4.1.1 · p. 3
• The instruction–audio match rate exceeds 95% according to human ratings, and the corpus is randomly split into 99% training and 1% validation data.Source: §4.1.1 · p. 3

Evaluation Testset

• Evaluation uses the original InstructTTSEval benchmark together with an expanded 200-sample testset targeting underrepresented accents, subtle and extreme emotions, and non-mainstream styles.Source: §4.1.2 · p. 3

Metrics

• Objective evaluation measures WER for stability and APS, DSD, and RP for acoustic instructions, descriptive styles, and role-play ability.Source: §4.1.3 · p. 3

Human Evaluation

• Subjective evaluation uses 20 native listeners rating instruction-following accuracy and speech naturalness and stability on 5-point I-MOS and N-MOS scales.Source: §4.1.3 · p. 3

Implementation

• The model is trained for 30 epochs on 8 NVIDIA A800 GPUs with AdamW at a learning rate of 1e-4 and uses CosyVoice components during inference.Source: §4.1.4 · p. 3

• Poly-InstructTTS achieves consistently high APS/DSD/RP scores against open- and closed-source instruction-following TTS systems, with the largest gains on RP.The authors associate the RP gains with in-the-wild training data and attribute-based thinking tokens.Source: §4.2 · p. 3
• The expanded InstructTTSEval comparison covers closed-source APIs, open-source models, and proposed-system ablations on base and expanded testsets.The table caption identifies the evaluated system categories and both testset variants.Source: Table 2 · p. 4
• As training epochs increase, APS/DSD/RP steadily improve while WER rises, exposing a trade-off between expressiveness and stability.The authors hypothesize that extreme emotions and paralinguistic behaviors weaken monotonic text–token alignment.Source: §4.2 · p. 4
• The system is weaker under challenging acoustic conditions such as noise and echo, with the weakness appearing more related to the FM than the GPT pathway.Source: §4.2 · p. 4

• Removing attribute-based thinking tokens lowers subjective scores, indicating that they help align instructions with acoustic tokens.The tokens complement free-form instruction text and are associated with better expressiveness than direct generation.Source: §4.3 · p. 4
• Frozen FlanT5, instructor, and GTR-base instruction representations do not yield positive gains over the encoder-free default.Under the reported training setup, raw instruction text is sufficient for generating styled speech.Source: §4.3 · p. 4

• The study compares instruction-conditioned SFT with the base Poly-InstructTTS using only FM-stage timbre injection and the same timbre reference.The comparison evaluates whether speaker-specific instruction control can be added while preserving speaker identity.Source: §4.4 · p. 4
• The subjective evaluation covers 10 fine-tuned speakers and reports I-MOS, N-MOS, and P-MOS, with P-MOS measuring alignment to the speaker persona.Source: §4.4 · p. 4
• SFT attains higher P-MOS and N-MOS, whereas the base setup achieves higher I-MOS.The authors interpret SFT as learning a speaker-specific mapping to a suitable stylistic range, with a trade-off in instruction adherence.Source: §4.4 · p. 4
• Table 3 presents a subjective comparison between the proposed Instruction-Conditioned SFT and the base method.Source: Table 3 · p. 4

• The paper constructs a 1,000-hour in-the-wild cinematic corpus covering 1,000+ fine-grained emotions and styles.Source: §5 · p. 4
• Poly-InstructTTS uses a prompt-free GPT with attribute-based thinking tokens, an FM framework, and an expanded InstructTTSEval testset.Source: §5 · p. 4
• Instruction-conditioned SFT gives target speakers instruction controllability while preserving their personas.Source: §5 · p. 4
• Future work targets better expressiveness–stability balance, improved FM performance under complex conditions, and text-only reference-free voice generation.The reference-free goal would remove the need for a timbre reference audio while preserving controllability.Source: §5 · p. 4

• OpenAI’s GPT-5 and Google’s Gemini 3.0 Pro were used solely for English-language polishing and readability improvements.Source: §7 · p. 5
• The tools were not used to generate scientific ideas, methods, or experimental results.Source: §7 · p. 5
• The human authors authored the technical content, analyses, claims, and conclusions.Source: §7 · p. 5
• The authors reviewed and verified all AI-assisted edits and take responsibility for the final text.Source: §7 · p. 5