Source-linked AI summary
Stabilizing Instruction Supervision for Instruct-TTS via Controllable Diversification and Drift Filtering
Yizhong Geng, Kecan Mao, Qifei Li, Cong Wang, Yingming Gao, Ruimin Wang, Chunfeng Wang, Hao Li, Ya Li
TL;DR
Instruct-TTS instruction rewrites can be semantically unreliable and limited in coverage, undermining supervision. The paper combines constrained diversification, drift filtering, and acoustic perturbation-based supervision; on Chinese InstructTTSEval, the full recipe reaches 56.4% average accuracy, while constrained rewriting reduces drift from 40.4% to 15.4%.
Problem
Label-to-instruction pipelines can produce homogeneous phrasing and semantic drift, weakening the coverage and fidelity of Instruct-TTS supervision.
Source: §1 · p. 1Method
Results
56.4% average accuracy with the Full Recipe, versus 48.9% without drift filtering and 51.0% for Naive SFT; constrained rewriting reduces drift from 40.4% to 15.4%.
Source: §4.2 · p. 4, §5 · p. 4, §4.3 · p. 4Takeaways & Limitations
The findings frame supervision instability as a data-quality problem and indicate that coverage and fidelity need to be addressed together.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Instruct-TTS systems expand structured style labels into natural-language training instructions through LLM rewriting, yet we find that over 40% of unconstrained rewrites contain semantic drift that corrupts supervision and weakens generalization. We formalize this problem as instruction supervision instability and propose a data-centric stabilization recipe that jointly improves coverage and fidelity through three mechanisms: controllable instruction diversification for systematic expansion, LLM-based drift filtering for quality control, and attribute-aligned supervision that grounds prosody control in acoustic perturbations. On the Chinese split of InstructTTSEval, our recipe raises instruction-following from 34.5% without fine-tuning and 51.0% with naive fine-tuning to 56.4%, while constrained rewriting reduces drift from 40.4% to 15.4%. Ablations confirm the three mechanisms are complementary, and the drift taxonomy may generalize to instruction-driven generation beyond TTS.
1. Introduction
The introduction frames unreliable label-to-instruction data as a supervision-quality problem: rewrites can be narrow in coverage and drift from their intended meaning. It proposes addressing both weaknesses through a combined data-centric recipe.
Motivation
- Homogeneous phrasing limits coverage, while semantic drift can corrupt training signals and may explain why naive fine-tuning underperforms a no-SFT baseline.Examples include turning a control instruction into an execution-like utterance, adopting a persona, or altering seed attributes.Source: §1 · p. 1
Problem characterization
- Over 40% of unconstrained rewrites corrupt supervision, with instruction-to-execution, role-assumption, and entity/label drift identified as recurring patterns.The paper names this problem instruction supervision instability and argues that coverage and fidelity must be improved jointly.Source: §1 · p. 1
Data-centric framing
- The paper treats instruction supervision instability as a data-quality problem rather than an architecture limitation, motivating improvements to both coverage and fidelity.It reports that drift filtering is the most impactful single component.Source: §1 · pp. 1–2
Stabilization recipe
- The proposed recipe combines constrained instruction diversification, LLM-verifier drift filtering, and attribute-aligned supervision from pitch, speed, and volume perturbations.The mechanisms target coverage and fidelity together; ablations report that they are complementary.Source: §1 · pp. 1–2
2. Related Work
Prior Instruct-TTS work extends natural-language control across speaking attributes, while this paper shifts attention to the quality of the upstream instruction data and grounded control of low-level prosody.
Instruct-TTS and instruction data quality
- PromptTTS, PromptTTS 2 [2], InstructTTS, and VoxInstruct extend natural-language control from speaking style to emotion, accent, and prosody.ParlerTTS scales the paradigm with large annotated corpora [17].Source: §2 · p. 2
- These efforts emphasize architectures and training objectives while treating upstream instruction data as given; this work instead targets label-to-instruction coverage and fidelity.It characterizes domain-specific semantic drift that standard NLP instruction pipelines [6] do not address.Source: §2 · p. 2
Controllable speech synthesis
- Attribute-aligned supervision pairs parameterized acoustic perturbations with natural-language prompts to ground low-level pitch, speed, and volume control.Conventional controllable TTS instead commonly uses reference-audio embeddings, discrete style tags, or signal-processing and model-internal controls [20] [21].Source: §2 · p. 2
3. Method
The method combines acoustic perturbations, constrained instruction diversification, and verifier-based drift filtering to build stabilized instruction–speech pairs for fine-tuning.
3.1. Attribute-Aligned Supervision
- Pitch, speaking-rate, and volume perturbations are paired with concise prompts to ground low-level prosody control in acoustic manipulations rather than potentially drifted rewrites.The starting clips carry structured seed attributes such as accent, emotion, gender, and speaking style.Source: §3.1 · p. 2
3.2. Controllable Instruction Diversification
- Persona, syntactic-pattern, and nondroppable attribute-slot constraints generate systematic variants while preserving seed attributes.Constrained rewriting reduces drift from 40.4% to 15.4%, but residual drift motivates filtering.Source: §3.2 · p. 2
3.3. Drift Filtering
- A verifier from a different model family scores candidate instructions for drift, and majority no-drift voting resolves borderline cases before retained examples form stabilized fine-tuning pairs.The verifier assigns a drift decision, category, and confidence score; confidence thresholds and voting expose a coverage–fidelity trade-off.Source: §3.3 · p. 3
4. Experiments
Experiments on Chinese InstructTTSEval test the recipe against drift, instruction-following, controllability, and data retention. The Full Recipe achieves 56.4% average accuracy, while attribute-level and filtering comparisons show complementary gains and a coverage–fidelity trade-off.
4.1. Experimental Setup
- The setup fine-tunes CosyVoice 2.0-0.5B on about 90 hours of Chinese speech and evaluates APS, DSD, and RP on unseen instructions.Human evaluation uses 20 native Mandarin listeners rating naturalness and controllability; attribute-aligned supervision creates 17 variants per clip through pitch, speed, and volume perturbations. DeepSeek-R1 generates candidates and GPT-4o verifies them, retaining about 61%.Source: §4.1 · p. 3
4.2. Drift Prevalence and Downstream Impact
- 40.4% total drift under unconstrained rewriting falls to 15.4% with constrained generation; post-filtering reduces drift to roughly 5%, alongside 56.4% average accuracy.The reported unconstrained categories include instruction-to-execution (18.3%), role-assumption (12.7%), and entity/label drift (9.4%); the residual 15.4% motivates filtering.Source: §4.2 · pp. 3–4
4.3. Main Results and Ablation
- 56.4% average accuracy for Full Recipe, versus 51.0% for Naive SFT and 48.9% without drift filtering; its NMOS and CMOS are both 4.16.Full Recipe's NMOS and CMOS exceed other systems by over 0.7 on both scales.Source: §4.2 · p. 4, §4.3 · p. 4
4.4. Effect of Attribute-Aligned Supervision
- Full Recipe gains about +32 each for Pitch and Speed and +31.8 for Volume over Base; Emotion gains +21.4, while Naive SFT scores 50.0 versus 42.6.The table reports per-attribute APS accuracy and absolute gains of Full Recipe over Base.Source: §4.4 · p. 4, Table 4 · p. 4
4.5. Drift Filtering Strategy Comparison
- 65.4% DSD accuracy with 61.5% retention is the best result among the filtering strategies, trading greater fidelity for reduced data diversity.Confidence thresholding alone reaches 63.8% accuracy at 72.3% retention; self-consistency voting alone reaches 63.5% at 68.1% retention.Source: §4.5 · p. 4, Table 5 · p. 4
5. Conclusion
The paper frames semantic drift in LLM rewrites as instruction-supervision instability that can corrupt Instruct-TTS training. It presents a three-part data-centric recipe and reports reduced drift and improved instruction-following on Chinese InstructTTSEval.
- The proposed recipe combines controllable diversification, LLM-verifier drift filtering, and attribute-aligned supervision from acoustic perturbations paired with prompts.On Chinese InstructTTSEval, constrained rewriting reduces drift from 40.4% to 15.4%, and the Full Recipe achieves the best instruction-following accuracy with particularly strong low-level controllability gains.Source: §5 · p. 4
7. Generative AI Use Disclosure
The authors disclose generative AI use for language refinement and minor editing, and LLM use in experimental methods. They state that the authors developed and verified the experimental design, analysis, and conclusions and take responsibility for the paper.
- Generative AI assisted with language refinement and minor editing; LLMs were also used for instruction generation, drift verification, and automatic evaluation.The authors state that they developed and verified the experimental design, analysis, and conclusions and take responsibility for the paper's content.Source: §7 · p. 5