PaperlayerView exact revision on arXiv

Source-linked AI summary

VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation

Jiarui Hai, Karan Thakkar, Ke Chen, Yunyun Wang, Jiaqi Su, Rithesh Kumar, Mounya Elhilali, Zeyu Jin

arXiv:2608.13613v1eess.AScs.LG

TL;DR

Existing text-to-voice systems struggle with diverse fictional voices and flexible, quality-preserving editing. VoiceDesigner unifies generation and editing through hybrid data construction and a modality-aware diffusion architecture, achieving stronger prompt alignment while maintaining competitive usability and perceptual quality.

  • Problem

    Existing systems have limited coverage of fictional voices and weak, quality-preserving capabilities for cloning and instruction-based voice editing.

  • Method

    VoiceDesigner combines DSP and generative data simulation with an MM-DiT architecture to jointly support voice generation, cloning, and instruction-guided editing.

  • Results

    VoiceDesigner achieves strongest prompt–style alignment and lowest word error rate while maintaining competitive transcription accuracy, usability, and perceptual quality.

  • Takeaways & Limitations

    The unified framework supports diverse voice types and flexible editing instructions within one text-to-voice system.

Abstract

from arXiv · show

Recent breakthroughs in generative models have made text-to-voice generation (TTV) possible, enabling the synthesis of speech directly from textual voice descriptions. However, existing systems face two key challenges. First, they struggle to generate a diverse range of voices, spanning real-world human speakers and fictional characters. Second, they lack robust and flexible voice editing capabilities, such as voice cloning and the ability to modify attributes like emotion and tone. In this paper, we propose VoiceDesigner, a unified framework for text-to-voice generation and editing that supports diverse and controllable voice design. To tackle the above challenges, we propose solutions from two perspectives. First, we develop a hybrid data pipeline that leverages digital signal processing techniques and speech generation models to construct a diverse voice dataset covering both real-world and fictional voices. Second, we introduce a diffusion transformer with architectural improvements to better handle complex conditioning and enhance multi-task performance, enabling unified voice generation and editing. Through subjective and objective evaluations, VoiceDesigner achieves superior prompt alignment with both voice descriptions and editing instructions, while maintaining competitive perceptual quality and voice usability compared to state-of-the-art TTV models.

I. INTRODUCTION

VoiceDesigner addresses existing text-to-voice limitations in voice diversity and flexible editing through a unified framework. It combines hybrid data simulation with an enhanced diffusion-transformer architecture and achieves strong prompt alignment while retaining competitive perceptual quality and usability.

  • Motivation: Existing TTV models struggle with uncommon speaking styles and fictional human or non-human characters because trait-based prompts poorly represent subjective character descriptions.Examples include nervous or unconventional styles and voices such as pirates, queens, monsters, demons, and dragons.
  • Motivation: Existing cloning and instruction-based editing systems are less robust for unconventional voices, provide limited editing capability, and may fail to preserve voice quality.These limitations hinder reusing and refining designed voices for different creative needs.
  • Approach: VoiceDesigner introduces a hybrid data pipeline combining DSP techniques and generative models to support diverse human/non-human voices, rich character specifications, and editing pairs.The DSP pipeline synthesizes voices humans cannot naturally produce, while complementary simulation provides stylistic variations and paired editing data.
  • Approach: VoiceDesigner extends MM-DiT with optimized conditioning modules to jointly model text-to-voice generation and editing using textual instructions, transcripts, and audio conditions.The architecture is designed to coordinate heterogeneous conditioning signals while enabling high-fidelity generation and precise control.
  • Results: Subjective and objective experiments show superior alignment with voice descriptions and edit instructions while maintaining competitive perceptual quality and voice usability against open-source TTV baselines.The reported evaluation covers both prompt alignment and practical voice quality and usability.

II. RELATED WORK … A. Task Formulation and Voice Coverage

Prior work expanded text-guided voice generation from basic demographic attributes toward detailed speaker and style descriptions, while voice editing remained comparatively underexplored. VoiceDesigner unifies generation, cloning, and editing through broad voice coverage and controllable modifications that preserve speaker identity.

  • A. Text-Guided Voice Generation: Caption-based systems progressed from gender and age guidance toward detailed timbre and diverse speaking-style descriptions.PromptTTS and PromptTTS++ focused primarily on basic attributes, while LibriTTS-P, DreamVoice, and ParaSpeechCaps explored finer-grained characteristics.
  • B. Text-Guided Voice Editing: Voice editing remains relatively underexplored compared with text-instruction-guided voice generation.The related work identifies voice editing as a less developed area.
  • B. Text-Guided Voice Editing: Existing editing approaches include simulated editing data and emotion-controlled voice cloning, but limitations in data quality or indirect control constrain flexibility.ISSE uses data simulation and a model framework, whereas IndexTTS-2 controls emotion through vectors and can convert instructions into parameter settings via an LLM.
  • C. Unified Generation and Editing Models: Recent visual-generation research increasingly unifies generation and editing by jointly modeling targets and conditioning inputs in a shared token space.This direction replaces earlier task-specific conditional-generation pipelines with shared representations for images and conditions.
  • III. METHODOLOGY: The methodology covers task formulation, VoiceDesigner architecture, and a data pipeline for diverse voice types and corresponding editing pairs.These components address voice coverage, prompt design, editing control, flexible generation, and controllable editing.
  • A. Task Formulation and Voice Coverage: VoiceDesigner supports voice generation, zero-shot voice cloning, and voice editing as three distinct operating modes.Generation uses a textual voice description and transcript; cloning uses an audio reference and transcript; editing additionally uses an editing instruction.
  • A. Task Formulation and Voice Coverage: Its two-level voice prompt schema combines speaker and style traits with character specifications spanning human roles and non-human entities.Traits include gender, age, speaking rate, pitch, expressiveness, emotion, and accent; examples include princess, wizard, monster, and robot.
  • A. Task Formulation and Voice Coverage: Voice editing treats the reference recording as an editing reference, enabling attribute changes while preserving the original speaker identity.Supported changes include emotion, speaking style, pitch, formants, and special audio effects, such as making a voice darker.

B. Model Architecture · 1) Overall Framework: · 2) Diffusion Transformer:

VoiceDesigner unifies voice generation, cloning, and editing in a flow-matching diffusion model operating in DAC-VAE audio latent space and conditioned on transcript, instruction, and audio reference. Its single-stream multimodal transformer introduces token-level AdaLN and 3D-RoPE to handle heterogeneous conditioning and modality-aware interactions.

  • 1) Overall Framework:: VoiceDesigner uses a flow-matching diffusion model in DAC-VAE audio latent space, conditioned on transcript, instruction, and audio reference.The diffusion model generates audio latent representations that are decoded into speech.
  • 1) Overall Framework:: Voice cloning combines transcript and reference speech to preserve vocal characteristics, while voice editing additionally uses an editing instruction to modify attributes while preserving linguistic content.Unifying these modes in one diffusion model reduces redundant training costs and enables knowledge transfer across tasks.
  • 1) Overall Framework:: During inference, a sentence-level duration predictor estimates target speech duration from instruction and transcript for generation, and additionally from reference audio for editing.The predicted duration sets the target latent sequence length, which starts as Gaussian noise and is denoised by the diffusion model.
  • 2) Diffusion Transformer:: Unlike prior systems supporting primarily one-way conditioning, VoiceDesigner handles instruction, transcript, and audio-reference signals across generation, cloning, and editing.This multimodal conditioning is implemented with a single-stream MM-DiT.
  • 2) Diffusion Transformer:: The single-stream MM-DiT concatenates modality tokens in a shared-parameter transformer after independent preprocessing by small modality-specific transformer modules.This backbone supports unified processing of heterogeneous inputs.
  • 2) Diffusion Transformer:: Token-level AdaLN assigns continuous time embeddings to audio tokens because diffusion noise affects generation targets while conditioning tokens remain noise-free.Under flow matching, t = 0 denotes a fully noisy latent and t = 1 a clean latent; target-token timesteps are sampled from [0, 1].
  • 2) Diffusion Transformer:: Token-level AdaLN also supports voice cloning and speech continuation by keeping prefix speech tokens noise-free while adding diffusion noise to subsequent tokens.This enables coherent continuation conditioned on the speech prefix.
  • 2) Diffusion Transformer:: 3D-RoPE extends rotary positional encoding into three dimensions to preserve modality-specific structure for instruction prompts, transcripts, and audio references.The design addresses the limitations of sharing standard 1D positional embeddings across heterogeneous conditions.

C. Data Simulation and Augmentation · 1) DSP-based Voice Simulation:

VoiceDesigner constructs diverse-voice speech data through two simulation pipelines, including a DSP-based pipeline that transforms ordinary recordings into human and non-human voices. Structured audio effects and pitch-contour manipulation expand character-voice coverage without dedicated recordings for every character.

  • C. Data Simulation and Augmentation: VoiceDesigner introduces two data simulation pipelines to construct a speech dataset containing diverse voices.
  • 1) DSP-based Voice Simulation:: Non-human voices are difficult to obtain from existing corpora because they require acoustic transformations beyond natural human phonation.Examples include monsters, ghosts, and miniature creatures whose voices are commonly produced through post-processing of human recordings.
  • 1) DSP-based Voice Simulation:: The DSP pipeline transforms standard speech recordings using pitch shifting, formant shifting, reverberation, equalization, band-pass filtering, and dynamic range compression.The transformations use Pedalboard, librosa, and Parselmouth.
  • 1) DSP-based Voice Simulation:: SiFi-GAN enables pitch-contour manipulation, including flattening intonation toward monotonic speech or adjusting pitch characteristics.
  • 1) DSP-based Voice Simulation:: Composing effects into structured pipelines transforms ordinary speech into character-style voices such as dragons, demons, possessed entities, robots, astronauts, and miniature creatures.
  • 1) DSP-based Voice Simulation:: The pipeline substantially expands non-human voice coverage while avoiding dedicated recordings for each character.

2) Generative Simulation via Voice Cloning and Conversion: … 1) Data for Pretraining Stage:

VoiceDesigner addresses limited stylistic speech diversity through generative simulation and prepares large-scale English pretraining data with speaker, vocal, and quality annotations. The framework also uses filtering and controlled caption generation to improve data reliability and description diversity.

  • 2) Generative Simulation via Voice Cloning and Conversion:: Generative simulation augments limited emotional, accented, and character-driven recordings with zero-shot text-to-speech, targeting broader linguistic and voice diversity.The passage motivates simulation by noting that stylistic recordings are limited in linguistic and voice diversity, which restricts generation range and generalization.
  • 2) Generative Simulation via Voice Cloning and Conversion:: Generated samples undergo multi-stage filtering for linguistic accuracy, speaker consistency, and emotion consistency against transcript, timbre, and style references.Whisper-based WER filtering discards samples above 0.1 and determines whether the original transcript is retained or replaced.
  • 1) Data for Pretraining Stage:: VoiceDesigner generates speech at a 48 kHz sampling rate.The passage attributes this implementation choice to the system’s speech-generation setup and character-style data limitations during fine-tuning.
  • 1) Data for Pretraining Stage:: For large-scale pretraining, four English speech corpora provide 56,165 hours annotated with speaker and vocal attributes.The corpora are Emilia, Common Voice, LibriTTS-R, and a subset of HiFiTTS-2-44.1k; attributes include age, gender, pitch, vocal expressiveness, speaking rate, and audio quality.
  • 1) Data for Pretraining Stage:: PENN-derived pitch statistics represent pitch level and tonal expressiveness, while speaking rate is computed from phoneme count divided by silence-removed utterance duration.Silence and unvoiced frames are excluded for pitch estimation, and pitch values are converted to the logarithmic scale before quantization.
  • 1) Data for Pretraining Stage:: Caption templates combined with LLM-generated synonym variations create voice descriptions that remain diverse yet controlled.This approach avoids directly generating captions from attribute tags alone.

2) Supervised Finetuning Sets for Voice Generation: · 3) Supervised Finetuning Sets for Voice Editing:

VoiceDesigner’s supervised finetuning sets combine richly annotated real and simulated data for diverse voice generation with instruction-guided editing pairs built from real recordings and synthetic speech. The generation data covers emotion, speaking style, accent, timbre, and character identity, while editing pairs model style changes within the same speaker.

  • 2) Supervised Finetuning Sets for Voice Generation:: The voice-generation sets are annotated with fine-grained attributes including emotion, accent, voice timbre, and character identity.These annotations support diverse voice generation.
  • 2) Supervised Finetuning Sets for Voice Generation:: E1 combines ESD English, RAVDESS, and SAVEE recordings covering common emotional categories such as happiness, anger, and sadness.
  • 2) Supervised Finetuning Sets for Voice Generation:: E2 combines Expresso, EARS, and CapSpeech-Agent recordings with fine-grained speaking styles including curiosity, confusion, whispering, boredom, and sarcasm.
  • 2) Supervised Finetuning Sets for Voice Generation:: VCTK and Common Voice contribute recordings with regional accents, while DreamVoice provides timbre labels on LibriTTS and VCTK.
  • 2) Supervised Finetuning Sets for Voice Generation:: The DSP-based simulation pipeline extends LibriTTS-R and VCTK with additional non-human character voice variations to support character prompts.
  • 2) Supervised Finetuning Sets for Voice Generation:: The internal character dataset contains 16 hours of recordings, 20 character identities, 12 professional voice actors, and 360 distinct voice variations.
  • 2) Supervised Finetuning Sets for Voice Generation:: All generation datasets receive extended voice descriptions conditioned on the voice traits of each clip.
  • 3) Supervised Finetuning Sets for Voice Editing:: Voice-editing pairs use real recordings and synthetic speech, with speaking-style edits created from the same speaker performing different styles.Either real or synthetic speech can serve as the reference or target; real targets also receive randomly added quality-improvement and artifact-reduction notes in the instruction prompt.

B. Training Setup … V. RESULTS AND ANALYSIS

VoiceDesigner trains its audio codec, duration predictor, and 1.0B-parameter diffusion transformer through staged procedures designed for generalization, alignment, generation, editing, and quality refinement. The evaluation covers natural and character voice generation, zero-shot cloning, editing, ablations, and qualitative analyses.

  • 1) DAC-VAE and Duration Predictor Training:: DAC-VAE training uses Emilia, Common Voice, VCTK, EARS, and AudioSet, with 48 kHz audio and a 25 Hz latent frame rate.The duration predictor is initialized from Qwen3-0.6B and encodes reference audio with a Dasheng encoder.
  • 2) Diffusion Transformer Training:: The diffusion transformer is a 1.0B-parameter MM-DiT using T5Gemma-XL for textual instructions and the F5-TTS tokenizer for transcripts.Training uses three progressive stages and the AdamW optimizer.
  • 2) Diffusion Transformer Training:: Stage 1 pretraining learns voice continuation, transcript–speech alignment, and prompt–voice alignment through voice cloning and generation on 64 A100 GPUs.The maximum audio duration is 20 seconds.
  • 2) Diffusion Transformer Training:: Stage 2 task adaptation enhances text-to-voice generation and editing while retaining voice cloning on HiFiTTS-2-44.1k, real recordings, and DSP-simulated character voices.Training uses 64 A100 GPUs with a maximum audio duration of 30 seconds.
  • 2) Diffusion Transformer Training:: Stage 3 quality refinement fine-tunes the model for 10k steps using real recordings and a small set of high-quality DSP-simulated data.Training uses 16 A100 GPUs with a maximum audio duration of 30 seconds.
  • V. RESULTS AND ANALYSIS: The results evaluate natural and character voice generation, then assess zero-shot voice cloning and voice editing, followed by ablation studies and qualitative analyses.These experiments examine the contributions of individual components.

A. Evaluation on Voice Generation

VoiceDesigner is evaluated on objective linguistic and style accuracy, plus subjective prompt–voice alignment, usability, and perceptual quality. It achieves the strongest Style-ACC and lowest word error rate, while leading open-source systems in MOS-C and remaining competitive in usability and quality.

  • Subjective evaluation: The subjective evaluation uses TTV-Traits for diverse human voices and TTV-Character for human and non-human voices with character and voice-attribute prompts.TTV-Traits contains 75 prompt-voice pairs, while TTV-Character contains 150 prompt–voice samples covering 50 characters.
  • Objective evaluation: VoiceDesigner achieves the strongest prompt–style alignment (Style-ACC) among all systems and the lowest word error rate, demonstrating strong style alignment with competitive transcription accuracy.ElevenLabs-TTV API and CapSpeech-NAR remain competitive on Style-ACC, while Qwen3TTS lags behind.
  • Subjective evaluation: VoiceDesigner demonstrates strong prompt–voice alignment (MOS-C), outperforming all open-source baselines on both benchmarks while remaining below ElevenLabs-TTV API.Overall, its MOS-C indicates superior prompt–voice coherence among open-source systems and encouraging proximity to the leading commercial system.
  • Subjective evaluation: VoiceDesigner is competitive with Qwen3TTS-VoiceDesign and surpasses CapSpeech-NAR on MOS-U, while Qwen3TTS-VoiceDesign delivers particularly clean MOS-Q outputs.The enhancement module significantly improves VoiceDesigner’s audio quality, narrowing the perceptual-quality gap.

B. Evaluation on Voice Cloning · C. Evaluation on Voice Editing

VoiceDesigner demonstrates strong zero-shot cloning of natural human and special-character voices, achieving competitive speaker preservation and superior similarity judgments. It also outperforms editing baselines in voice similarity and instruction following for emotional and stylistic transformations.

  • B. Evaluation on Voice Cloning: Voice cloning evaluation follows zero-shot TTS protocols on SeedTTS test-en, using SIM-o for speaker preservation and WER for intelligibility.The evaluation covers human voices and compares VoiceDesigner with recent state-of-the-art systems.
  • B. Evaluation on Voice Cloning: A dedicated test set contains 20 voice–utterance pairs spanning challenging character voices such as dragons, monsters, robots, and other non-human characters.VoiceDesigner is compared with CosyVoice-3 and IndexTTS-2 for special-character cloning.
  • B. Evaluation on Voice Cloning: VoiceDesigner achieves the lowest WER among compared systems, while its speaker similarity surpasses most recent zero-shot TTS models and remains close to SeedTTS.These results indicate accurate speech generation and strong cloning of natural human voices.
  • B. Evaluation on Voice Cloning: Subjective evaluations show that VoiceDesigner performs best in both voice timbre and speaking-style similarity, surpassing CosyVoice-3 and IndexTTS-2.This further supports its ability to clone special character voices.
  • C. Evaluation on Voice Editing: Voice editing is evaluated on an unseen internal expressive speech dataset of professional voice actors covering diverse speaking styles and emotional expressions.The dataset was never used during any of VoiceDesigner’s three training stages.
  • C. Evaluation on Voice Editing: The editing benchmark comprises 100 pairs from 50 source utterances spoken by 10 speakers, with each recording transformed into two target styles.VoiceDesigner is compared with Step-Audio-EditX and IndexTTS-2’s text-prompt emotion control mode.
  • C. Evaluation on Voice Editing: VoiceDesigner surpasses Step-Audio-EditX and IndexTTS-2 in voice similarity and instruction-following accuracy for fine-grained voice style editing.Step-Audio-EditX sometimes introduces audible artifacts, while both baselines are inconsistent on certain styles.

D. Ablation Studies and Additional Analysis · 1) Architecture Comparison:

The architecture comparison evaluates CapSpeech-NAR alongside three VoiceDesigner diffusion-transformer variants under voice cloning and TTV settings. Results indicate that token-level conditioning and 3D positional encoding improve convergence and final performance, while VD-DiT-v1 converges faster than CapSpeech-NAR.

  • 1) Architecture Comparison:: The comparison includes CapSpeech-NAR, which extends F5-TTS with cross-attention for TTV modeling, and three proposed VoiceDesigner architectures.VD-DiT-v1 uses the original single-stream MM-DiT; VD-DiT-v2 adds token-level AdaLN; VD-DiT-v3 further adds 3D-RoPE.
  • 1) Architecture Comparison:: The ablation uses 5,000 hours of HiFiTTS-2, excludes 200 speakers for evaluation, and tests both zero-shot TTS voice cloning and TTV.The evaluation constructs 400 reference–utterance pairs from excluded speakers for voice cloning.
  • 1) Architecture Comparison:: Evaluation compares convergence at Early (50K steps), Early-Mid (100K), Late-Mid (150K), and Late (200K) training stages.These stages visualize metric values across different training steps to assess convergence efficiency.
  • 1) Architecture Comparison:: VD-DiT-v1 converges faster and achieves significantly lower WER than CapSpeech-NAR.The paper attributes this advantage to CapSpeech-NAR sharing a unified positional space for audio and text tokens, requiring more training to disentangle and align modalities.
  • 1) Architecture Comparison:: VD-DiT-v1 performs better early for speaker similarity and voice generation but becomes slightly weaker than CapSpeech-NAR later in training.This indicates that the relative advantage changes across training stages for these voice-cloning metrics.
  • 1) Architecture Comparison:: Token-level AdaLN in VD-DiT-v2 explicitly distinguishes conditional tokens, substantially improving convergence speed and final speaker similarity and voice-generation performance with only a minor WER increase.VD-DiT-v3 introduces 3D-RoPE to assign distinct positional patterns to instruction and speech-content representations, with the supplied passage ending before its full result is stated.

2) Generative Simulation Data Augmentation: · 3) Runtime Analysis: · VI. CONCLUSION

VoiceDesigner’s generative augmentation improves evaluation performance, while its runtime remains competitive for generation and editing. The framework unifies diverse voice generation and flexible editing through hybrid data construction and modality-aware modeling.

  • 2) Generative Simulation Data Augmentation:: The augmentation study evaluates voice cloning and conversion on emotional speech corpora E1 and E2, which are limited in duration and linguistic diversity.The ablation follows the augmentation procedure described in Section IV-A2.
  • 2) Generative Simulation Data Augmentation:: Models trained with real data augmented by generative samples are compared against models trained with real data only using Style-Acc and WER.Evaluation uses a simplified objective dataset excluding accent-related attributes.
  • 2) Generative Simulation Data Augmentation:: Generative data augmentation consistently improves performance over real-data-only training.The comparison initializes models from a Stage-1 checkpoint and fine-tunes them on voice generation.
  • 3) Runtime Analysis:: 0.36 RTF for voice generation is comparable to CapSpeech’s 0.33 and faster than Qwen3-TTS’s 1.21 on a single NVIDIA RTX 4090 GPU.RTF is averaged over 200 test utterances averaging approximately 15 s.
  • 3) Runtime Analysis:: 0.42 RTF for voice editing outperforms Step-Audio-EditX’s 0.80 and IndexTTS2’s 0.72.Runtime is measured using the Real-Time Factor on a single NVIDIA RTX 4090 GPU.
  • VI. CONCLUSION: VoiceDesigner is presented as a unified framework supporting diverse voice types and flexible editing instructions.The framework addresses limited voice diversity and weak editing capabilities in existing TTV systems.
  • VI. CONCLUSION: The proposed hybrid data construction pipeline expands coverage of both human and non-human voices.The conclusion identifies this pipeline as a response to limited voice diversity in existing systems.
  • VI. CONCLUSION: An MM-DiT-based architecture with modality-aware conditioning is introduced to jointly model voice generation, cloning, and instruction-based editing.The passage presents this architecture as addressing weak editing capability in existing TTV systems.
Loading 2608.13613v1…