PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexiVoice: Enabling Flexible Style Control in Zero-Shot TTS with Natural Language Instructions

Dekun Chen, Xueyao Zhang, Yuancheng Wang, Kenan Dai, Li Ma, Zhizheng Wu

arXiv:2601.04656v1cs.SD

TL;DR

Flexible style control in zero-shot TTS requires separating natural-language style instructions, speaker timbre, and textual content. FlexiVoice addresses this with an LLM-based TTS system, an instruction–speech dataset, and Progressive Post-Training; experiments report stronger disentanglement and instruction following than competing baselines, with human evaluations confirming naturalness, controllability, and robustness.

  • Problem

    Zero-shot TTS can clone timbre from reference speech, but style instructions can conflict with reference acoustics and textual prosody, creating a Style-Timbre-Content Conflict.

  • Method

    FlexiVoice uses optional natural-language style instructions, optional reference speech for timbre, and Progressive Post-Training combining DPO with staged multi-objective GRPO.

  • Results

    FlexiVoice reports large ACC-I gains, reaching 97.4% in English and 99.8% in Chinese on TO-easy, while maintaining 89.4% and 81.8% on TR-easy.

  • Takeaways & Limitations

    FlexiVoice decouples speaking style from speaker identity, improves instruction adherence and robustness over baselines, and produces natural speech in human evaluations.

Abstract

from arXiv · show

This study proposes FlexiVoice, a text-to-speech (TTS) synthesis system capable of flexible style control with zero-shot voice cloning. The speaking style is controlled by a natural-language instruction and the voice timbre is provided by a speech reference in zero-shot manner. FlexiVoice is built with an LLM core, which takes text as input, and also takes an optional natural language instruction and an optional speech reference to control style and timbre, respectively. FlexiVoice is equipped with a novel Progressive Post-Training (PPT) scheme that progressively unlocks accurate and flexible controllability. In particular, it first employs Direct Preference Optimization (DPO) to enable FlexiVoice to accurately follow both natural language instruction and speech reference simultaneously. It then uses a multi-objective Group Relative Policy Optimization (GRPO) to disentangle style instruction, reference timbre, and textual content. Finally, it adapts instruction GRPO for more advanced instruction following. Experimental results show that FlexiVoice surpasses competing baselines and demonstrates strong capability in decoupling control factors. Human evaluations further confirm its naturalness, controllability, and robustness. Audio samples are available at https://flexi-voice.github.io.

1 INTRODUCTION

FlexiVoice targets the Style-Timbre-Content Conflict in zero-shot TTS by combining natural-language style instructions with optional reference speech for timbre control. Its dataset and Progressive Post-Training framework support flexible control, while evaluations report stronger instruction adherence, robustness, and naturalness.

  • Motivation: The Style-Timbre-Content Conflict arises when acoustic reference cues or textual prosody override explicit style instructions.The paper motivates active disentanglement rather than instruction conditioning alone.
  • Results: Evaluations show decoupled speaking style and speaker identity, larger instruction-adherence and robustness gains than baselines, and natural, robust speech in human evaluations.The study uses emotion datasets and InstructTTSEval to assess flexible controllability and instruction following.
  • System and contributions: FlexiVoice controls speaking style with optional natural-language instructions and timbre with optional reference speech, enabling combinations of style and speaker identity.The system is built on a pretrained LLM and supports either control input alone or both together.
  • Data and contributions: FlexiVoice-Instruct is a large-scale, diverse speech dataset annotated with LLM-generated natural-language instructions covering human-like and expressive scenarios.It supplements pretraining with instruction–speech data for flexible style control.
  • Training framework: Progressive Post-Training sequences multi-modality DPO, decoupling GRPO, and instruction GRPO to address modality conflicts and improve controllability.The curriculum is designed to establish alignment, disentangle control factors, and extend instruction following.

2 RELATED WORK

Prior controllable speech-synthesis work has separately pursued zero-shot timbre cloning and instruction-based style control. Reinforcement-learning approaches improve targeted TTS properties, but FlexiVoice is positioned around broader multimodal controllability.

  • Controllable speech synthesis: Zero-shot TTS primarily clones speaker timbre from short reference speech, while instruction-based TTS specifies style through natural-language prompts.Existing systems often provide partial style control or operate within limited style spaces.
  • Controllable speech synthesis: Some systems use separate speech references to control timbre and style, illustrating multimodal approaches to disentangled speech control.The cited examples include Vevo and IndexTTS2.
  • Instruction-speech datasets: Instruction–speech datasets broaden style conditioning through prompt–speech pairs, weak labels, and multimodal audio annotations.Prior resources differ in scale, style coverage, and public accessibility.
  • Reinforcement learning: Recent reinforcement-learning methods apply preference alignment or multi-objective post-training to targeted intelligibility, emotion, prosody, and speech–singing control.These studies motivate reinforcement-learning alignment for controllable speech synthesis.

3 OVERVIEW OF FLEXIVOICE

FlexiVoice combines an LLM-based speech-token pipeline with a staged Progressive Post-Training curriculum. The stages move from multimodal alignment to disentanglement and then complex instruction following, using preference and reward-based optimization.

  • Architecture: FlexiVoice’s LLM processes text, optional instructions, and reference-speech tokens to generate discrete speech tokens, which are converted to waveform audio.A flow-matching module produces Mel-spectrogram features before waveform synthesis.
  • Pre-training: FlexiVoice-Base is pretrained on Emilia and diverse instruction-speech resources, while the LLM core is trained without reference speech during pretraining.The broader corpus includes FlexiVoice-Instruct and datasets covering emotion, age, debate, and paralinguistic expression.
  • Progressive Post-Training: PPT follows a curriculum in which S1 aligns instruction and reference control, S2 disentangles reference and text styles, and S3 handles harder real-world instructions.The stages progress from controlled emotion-centric tasks to ambiguous complex directives.
  • S1: Multi-modality controllability: S1 uses DPO preference tuples whose winner matches the instruction, text, and reference, with policy and reference models initialized from FlexiVoice-Base.This aligns emotional output without requiring an explicit reward model.
  • S2: Decoupling: S2 uses multi-objective rewards for instructed emotion and speaker verification to suppress style leakage while preserving timbre consistency.The style reward comes from Emotion2vec-Large, and the speaker-verification reward comes from CAM++.
  • S3: Complex instruction following: S3 applies GRPO to complex instructions without reference speech, mixes some S2 data to mitigate catastrophic forgetting, and performs multi-task optimization.References are discarded because they may conflict with open-ended constraints and destabilize training.

4 FLEXIVOICE-INSTRUCT DATASET

FlexiVoice-Instruct is trained on a diverse 4,316-hour instruction–speech dataset assembled from Emilia and game voice acting data. An LLM annotator converts transcriptions and metadata into natural-language style and scenario descriptions while filtering noisy samples.

  • The dataset contains 4,316 hours of instruction–speech data covering diverse styles and scenarios.Speech-related textual metadata is processed with an LLM-based annotator to generate natural, high-level instructions.
  • Emilia and game voice acting data are processed with a unified strategy for instruction annotation.The two sources provide complementary speech and style information for pre-training.
  • For Emilia, transcriptions and source metadata support open-ended style and scene descriptions, while an LLM filters noisy or conflicting samples.Deepseek-V3 evaluates metadata usefulness for style and scenario inference before annotation is retained.
  • Game voice acting data enrich stylistic coverage by linking character names, personalities, and speaking styles to transcriptions.The LLM identifies recognizable characters and refines inferred descriptions using the corresponding transcription.
  • For both sources, speech transcriptions and related metadata prompt an LLM to generate natural, human-like descriptions used as pre-training instructions.

5 EXPERIMENTS

Experiments evaluate FlexiVoice on multi-modal disentanglement, perceptual quality, and complex instruction following across English and Chinese. FlexiVoice shows strong style–timbre–content separation, preserves naturalness, and benefits from progressive post-training.

  • Experimental setup: The evaluation combines emotion-centric multi-modal control tasks with the InstructTTSEval benchmark for complex instruction following.The emotion tasks test instruction-based emotional TTS and disentanglement, while InstructTTSEval covers broader speech attributes and instruction levels.
  • Multi-modality control: 97.4% ACC-I in English and 99.8% in Chinese on TO-easy demonstrate strong instruction-based emotion control with text-only conditioning.On TR-easy with reference speech, FlexiVoice reaches 89.4% ACC-I in English and 81.8% in Chinese while preserving speaker consistency.
  • Target text disentanglement: 89.4% ACC-I with 6.6% ACC-T in English and 98.4% with 0.8% in Chinese show effective disentanglement from conflicting emotional text.The Chinese easy-to-hard performance gap is only 1.4%, indicating stable instruction-controlled style under textual conflict.
  • Reference speech disentanglement: 78.2% ACC-I in English and 75.8% in Chinese, with ACC-R of 10.6% and 13.2%, show separation of reference timbre from conflicting reference style.The model preserves the reference speaker while following the instructed emotion in hard TR settings.
  • Intelligibility and subjective evaluation: FlexiVoice maintains perceptual quality despite marginally higher WER/CER, with Q-MOS reaching 4.08 versus 3.72 in EN-TO-Easy and CMOS up to +0.9.Human evaluations report stronger perceptual clarity, expressiveness, and instruction adherence than comparison systems.
  • Instruction following and ablations: Progressive post-training reaches 79.3 English InstructTTSEval average accuracy, while joint training reaches 75.5 and the S3-first order reaches 72.3.The staged curriculum also raises Decoupling Avg. to 88.5 and adds 7.6 points on InstructTTSEval Avg. over the preceding stage.

6 CONCLUSION

FlexiVoice combines natural-language style instructions with reference-speech timbre control through a Progressive Post-Training paradigm. Experiments report improved disentanglement and instruction following while maintaining naturalness and robustness.

  • FlexiVoice uses natural-language instructions for style and reference speech for timbre in multi-modality TTS control.The system is supported by an instruction–speech dataset and a Progressive Post-Training paradigm.
  • Progressive Post-Training first strengthens controllability and disentanglement through emotion-centric DPO/GRPO, then targets complex instruction following with an ALM-based reward.The staged training is reported to yield stable optimization and broad generalization.
  • FlexiVoice achieves consistent disentanglement gains, strong InstructTTSEval performance, and maintained naturalness and robustness in human evaluations.The conclusion reports large ACC-I improvements with low interference from text and reference inputs, and performance surpassing competing baselines.

7 ETHICS STATEMENT

The study uses publicly available or licensed resources and filters the instruction–speech dataset for offensive, biased, or harmful content. It reports no collection or processing of personal or sensitive user data.

  • The instruction–speech dataset was created from publicly available or licensed resources and filtered for offensive, biased, or harmful content.
  • The study did not collect or process personal or sensitive user data.
  • The authors acknowledge potential misuse of speech synthesis for deceptive or harmful audio while framing the study around controllability, robustness, and transparency.

8 REPRODUCIBILITY STATEMENT

The authors describe reproducibility measures covering model design, training, evaluation, data processing, and release materials. They plan to release the dataset, checkpoints, code, settings, and scripts.

  • The paper provides detailed descriptions of the model architecture, training objectives, and evaluation protocols in the main text and appendix.
  • The authors will release the instruction–speech dataset, model checkpoints, and training and inference code.
  • Release materials will also include hyperparameter settings, data processing procedures, and evaluation scripts.

A.1 MODEL STRUCTURE

FlexiVoice uses an autoregressive LLM stage followed by flow matching and vocoding. The first stage processes text, instructions, and reference speech, while the second converts generated codes into audio representations.

  • FlexiVoice contains two main stages: an autoregressive LLM and a flow-matching module.The model structure is illustrated in Figure 3.
  • The LLM stage receives text, an optional instruction, and reference speech, formatting text and instructions through the LLM input template.Reference speech is represented using discrete semantic codes, with its transcription concatenated before the text.
  • The flow-matching module converts generated codes into a mel-spectrum using reference speech codes as conditioning, followed by a Vocos vocoder that produces target audio.

A.2 DATA PROCESS FOR PRE-TRAINING

The pre-training stage uses instruction-speech datasets spanning diverse speech properties and instruction formats, with dataset-specific processing for single-label, multi-label, and paralinguistic data.

  • Table 6 lists the instruction-speech datasets used during pre-training.
  • For ParaSpeechCaps, detailed acoustic feature dictionaries are concatenated and descriptions are randomly sampled when both resources are available.
  • Single-label datasets receive randomly selected templates filled with labels such as speaker identity, scene, or emotion.
  • Multi-label datasets use LLM-generated Chinese and English instructions for combinations of attributes such as age, gender, and accent.
  • NVSpeech contributes paralinguistic tags without added instructions, while its tags enlarge the LLM vocabulary for expressive generation.

A.3 PROMPTS IN INSTRUCTION DATA CONSTRUCTION

The appendix details prompt construction, GRPO training data, DPO emotion pairs, reward selection, evaluator validation, and non-emotional style-control results.

  • A.3 PROMPTS IN INSTRUCTION DATA CONSTRUCTION: Prompt construction instructs an LLM to generate natural-language voice descriptions from transcripts and metadata, while prioritizing text when metadata conflicts.
  • A.3 PROMPTS IN INSTRUCTION DATA CONSTRUCTION: Generated descriptions use colloquial, vivid phrasing, avoid character names or irrelevant titles, and include an information-value label when specified.
  • A.3 PROMPTS IN INSTRUCTION DATA CONSTRUCTION: Prompt inputs and outputs are structured around speech transcripts, metadata, natural-language style descriptions, and high/medium/low information values.
  • A.4 DATA CONSTRUCTION FOR GRPO TRAINING: Decoupling GRPO data pairs NCSSD speech with randomly assigned emotions to create aligned and conflicting instruction-transcription examples.
  • A.4 DATA CONSTRUCTION FOR GRPO TRAINING: Complex-instruction GRPO data combines sampled instruction-text inputs with 14,000 generated instructions covering acoustic dictionaries, multiple explicit features, and arbitrary scenarios.
  • A.5 DETAILS IN EMOTIONAL CONTROL DPO: Emotion DPO pairs use an instruction such as an angry-tone request, with same-speaker preferred and mismatched-emotion rejected recordings.
  • A.6 REWARD SELECTION IN DECOUPLING GRPO: Speaker verification outperforms speaker similarity as a reward because embedding similarity also reflects pitch and emotion, hindering multi-objective optimization.
  • A.7 VALIDATION OF REWARD MODELS: Kimi-Audio-7B agrees with Gemini across task types with Macro-F1 > 0.60, while FlexiVoice achieves the highest cross-language, cross-attribute correlations in non-emotional style control.
Loading 2601.04656v1…