VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation

I. INTRODUCTION

• Existing TTV models handle common human voice traits but struggle with uncommon styles and fictional human or non-human characters described through identity-based, subjective prompts.Their training data predominantly contains natural human speech, leaving diverse fictional and character voices poorly covered.Source: §I · p. 1
• Existing cloning and instruction-based editing systems struggle with unconventional voices, limited editing, and voice-quality preservation, while separate generation and editing systems raise costs.These limitations also restrict applicability in low-resource settings.Source: §I · p. 1
• VoiceDesigner’s hybrid data pipeline combines DSP-generated non-human voices with generative stylistic variations and paired editing data to broaden generation and editing supervision.The generative simulation pipeline uses zero-shot TTS and voice conversion models with fine-grained voice-trait descriptions.Source: §I · pp. 1–2
• VoiceDesigner unifies voice generation and editing with MM-DiT, adding token-level AdaLN and 3D-RoPE to coordinate text instructions, transcripts, and audio conditions.The architecture is intended to support multimodal conditioning across both tasks.Source: §I · p. 2
• Subjective and objective experiments show superior alignment with voice descriptions and editing instructions while maintaining competitive perceptual quality and voice usability against open-source TTV baselines.Source: §I · p. 2

A. Text-Guided Voice Generation

• Early PromptTTS, [20] and PromptTTS++ [5] used captions mainly for gender and age, while LibriTTS-P, DreamVoice, and ParaSpeechCaps broadened speaker and style descriptions.Source: A. Text-Guided Voice Generation · p. 2
• SpeechCraft and VoxInstruct add emotion signals but face quality and variation limits; AudioBox [25] scales labels but is closed-source, while CapSpeech separates pretraining and fine-tuning.Source: A. Text-Guided Voice Generation · p. 2
• ElevenLabs offers natural-language voice design, and Qwen3-TTS [17] extends character voice design within instruction-based speech generation.Source: A. Text-Guided Voice Generation · p. 2

B. Text-Guided Voice Editing

• Voice editing remains underexplored, with ISSE [12] limited by data and instruction quality, IndexTTS-2 [2] enabling indirect style edits, and Step-Audio-EditX [13] targeting diverse emotions and speaking styles.IndexTTS-2 uses emotion vectors in voice cloning, with an official release integrating an LLM to convert text instructions into parameter settings.Source: B. Text-Guided Voice Editing · p. 2

C. Unified Generation and Editing Models

• Unified visual-generation models moved from task-specific editing pipelines, toward shared-token modeling of generation targets and conditions –, supporting generation, editing, and style transfer.The shared representation also reduces the need for task-specific architectures.Source: C. Unified Generation and Editing Models · p. 2

III. METHODOLOGY

• The methodology formulates text-to-voice generation and editing around voice coverage, prompt design, and editing control.Source: §III · p. 2
• It describes the VoiceDesigner architecture and the design choices intended to enable flexible voice generation and controllable editing.Source: §III · p. 2
• It introduces a data pipeline to construct diverse voice types and corresponding editing pairs for framework training and evaluation.Source: §III · p. 2

A. Task Formulation and Voice Coverage

• The two-level voice prompt schema specifies speaker and style traits inspired by prior work, [17], then adds character roles such as princess, wizard, monster, or robot.Traits include gender, age, speaking rate, pitch, expressiveness, emotion, and accent; character descriptors cover human roles and non-human entities.Source: A. Task Formulation and Voice Coverage · p. 2
• Voice cloning follows zero-shot TTS, supporting speech inpainting and continuation from a reference audio.Source: A. Task Formulation and Voice Coverage · p. 2
• Voice editing treats the reference recording as an editing reference, changing emotion, speaking style, pitch, formants, or audio effects while preserving speaker identity.For example, it can make a speaker sound happy or make the voice darker.Source: A. Task Formulation and Voice Coverage · p. 3

1) Overall Framework:

• A flow-matching diffusion model in DAC-VAE audio latent space conditions on transcripts, instructions, and reference audio for unified voice generation and editing.DAC-VAE, serves as the encoder and decoder.Source: 1) Overall Framework · p. 3
• Generation uses a transcript and voice description, cloning uses a transcript and reference speech, and editing adds an instruction to change voice attributes while preserving linguistic content.The three modes arise from different combinations of transcript, instruction, and audio-reference inputs.Source: A. Task Formulation and Voice Coverage · p. 2, 1) Overall Framework · p. 3
• Unifying the modes in one diffusion model reduces redundant training costs and enables knowledge transfer across tasks.Source: 1) Overall Framework · p. 3
• At inference, a sentence-level duration predictor, [35] estimates target speech duration from instruction and transcript, adding reference audio for editing; its estimate sets the latent sequence length before noise denoising.Source: 1) Overall Framework · p. 3

2) Diffusion Transformer:

• VoiceDesigner uses a single-stream MM-DiT, [30] to concatenate modality tokens and process them with shared transformer parameters after independent modality-specific modules.The framework supports instruction, transcript, and audio-reference conditioning across generation, cloning, and editing, unlike prior systems, [25] that primarily support one-way conditioning.Source: 2) Diffusion Transformer · pp. 3–4
• Token-level AdaLN, inspired by diffusion forcing, [38], assigns target audio tokens sampled timesteps and fixes reference tokens at t = 1 to distinguish noisy targets from clean conditions.A dedicated [C] token marks instruction and transcript conditions; noise-free speech prefixes also condition subsequent tokens for cloning and continuation.Source: 2) Diffusion Transformer · p. 4
• 3D-RoPE encodes instruction, transcript, and audio tokens on separate axes, with aligned transcript–audio pairs sharing an x-coordinate to preserve temporal correspondence.This preserves alignment-aware and modality-aware positional structure while enabling unified self-attention across tokens.Source: 2) Diffusion Transformer · p. 4

C. Data Simulation and Augmentation

• VoiceDesigner introduces two data-simulation pipelines to construct a speech dataset covering diverse voices.The supplied passages describe the DSP-based pipeline for synthesizing non-human voices.Source: C. Data Simulation and Augmentation · p. 4

1) DSP-based Voice Simulation:

• The DSP pipeline addresses the scarcity of non-human voices in speech corpora by synthesizing acoustic transformations beyond natural human phonation.Creative-media character voices are often made by recording human performers and applying post-processing, layering, and audio effects.Source: 1) DSP-based Voice Simulation · p. 4
• The pipeline combines pitch and formant shifting, reverberation, equalization, band-pass filtering, and dynamic-range compression, plus SiFi-GAN pitch-contour manipulation.The effects use Pedalboard, librosa, and Parselmouth; SiFi-GAN can flatten intonation toward monotonic speech or adjust pitch-variation ranges.Source: 1) DSP-based Voice Simulation · p. 5
• The DSP pipeline expands non-human voice coverage by transforming ordinary speech into character-style voices without dedicated recordings for each character.Examples include dragons, demons, possessed entities, robots, astronauts, and miniature creatures.Source: 1) DSP-based Voice Simulation · p. 5

2) Generative Simulation via Voice Cloning and Conversion:

• Voice cloning adds transcript content while preserving reference timbre and style, whereas voice conversion preserves content and style while replacing speaker timbre.The generative simulation pipeline augments stylistic speech data that is limited in linguistic and voice diversity.Source: 2) Generative Simulation via Voice… · p. 5
• Samples with WER > 0.1 are discarded; for remaining samples, the original transcript is retained when WER < 0.05, otherwise Whisper transcription replaces it.WER is computed using Whisper.Source: 2) Generative Simulation via Voice… · p. 5
• For each prompt, the higher speaker-similarity cloning candidate or emotion-similarity conversion candidate is retained, and samples below 0.6 speaker similarity are discarded.Speaker and emotion embedding similarities assess identity preservation and emotional consistency against timbre and style references.Source: 2) Generative Simulation via Voice… · p. 5

1) Data for Pretraining Stage:

• For large-scale pretraining, four speech corpora—Emilia, Common Voice, LibriTTS-R [9], and a subset of HiFiTTS-2-44.1k [48]—total 56,165 hours; each clip is annotated with speaker and vocal attributes.Attributes include age, gender, pitch, vocal expressiveness, speaking rate, and audio quality.Source: 1) Data for Pretraining Stage · p. 5
• When demographic metadata is unavailable, a pretrained age-and-gender estimator supplies it, and age is assigned to five defined categories.The categories are child (1–12), teen (13–19), young adult (20–39), adult (40–64), and elderly (65+).Source: 1) Data for Pretraining Stage · p. 5
• Pitch level and tonal expressiveness are derived from speaker-level mean pitch and utterance-level pitch standard deviation, while speaking rate uses phonemes per silence-removed duration.PENN [49] computes pitch; silence and unvoiced frames are excluded, pitch is log-scaled before quantization, and percentile bins for pitch, expressiveness, and audio quality use Emilia and HiFiTTS-2-44.1k; AudioBox-Aesthetics [50] annotates quality and cleanness.Source: 1) Data for Pretraining Stage · p. 5
• Caption templates plus LLM-generated synonym variations produce voice descriptions that are diverse yet controlled rather than directly generated from attribute tags.Source: 1) Data for Pretraining Stage · p. 5

• The collected datasets provide fine-grained annotations for emotion, accent, voice timbre, and character identity.Source: 2) Supervised Finetuning Sets for… · p. 5
• ESD English, RAVDESS, and SAVEE provide recordings labeled with common emotions such as happiness, anger, and sadness.Source: 2) Supervised Finetuning Sets for… · p. 5
• Expresso [54], EARS [55], and CapSpeech-Agent cover fine-grained styles including curiosity, confusion, whispering, boredom, and sarcasm.Source: 2) Supervised Finetuning Sets for… · p. 5
• VCTK and Common Voice provide regional accents, while DreamVoice supplies timbre labels on LibriTTS/VCTK.Source: 2) Supervised Finetuning Sets for… · p. 5
• DSP-based simulation on bandwidth-extended LibriTTS-R and VCTK adds non-human character variants, alongside an internal 16-hour dataset of 20 identities performed by 12 actors, yielding 360 variations.Source: 2) Supervised Finetuning Sets for… · p. 5
• For datasets with few recordings per voice, cloning and pitch-controlled conversion models expand scale, linguistic coverage, and emotional diversity, then Qwen3-30B-A3B-Instruct8 generates extended voice descriptions.The models generate stylistically consistent speech with varied content and controlled attributes, following designs similar to,,, and.Source: 2) Supervised Finetuning Sets for… · pp. 5–6

3) Supervised Finetuning Sets for Voice Editing:

• Editing pairs combine real and synthetic speech, using same-speaker style variations and DSP-simulated transformations; pitch and formant pairs reverse processed-reference and original-target roles to mitigate inference artifacts.Real targets may receive quality-improvement or artifact-reduction notes, and curated instruction templates with synonymous variations increase linguistic diversity and training robustness.Source: 3) Supervised Finetuning Sets for… · p. 6

1) DAC-VAE and Duration Predictor Training:

• DAC-VAE training uses five audio corpora, resampled to 48 kHz, and operates at a 25 Hz latent frame rate; duration prediction uses a Qwen3-0.6B initialization, Dasheng reference-audio encoder, and diffusion head.Training follows the Stable Audio configuration and uses both pretraining and finetuning datasets; AudioSet is included to improve generalization to general audio.Source: 1) DAC-VAE and Duration Predictor… · p. 6

2) Diffusion Transformer Training:

• A 1.0B-parameter MM-DiT uses T5Gemma-XL for textual instructions and the F5-TTS tokenizer for transcripts, and is trained in three stages with AdamW.The stages are pretraining, task adaptation, and quality refinement.Source: 2) Diffusion Transformer Training · p. 6
• Pretraining teaches voice continuation, transcript–speech alignment, and prompt–voice alignment through voice-cloning and voice-generation tasks on 64 A100 GPUs, with audio up to 20 seconds.Source: 2) Diffusion Transformer Training · p. 6
• Task adaptation uses the SFT and editing datasets to strengthen generation and editing while retaining voice-cloning training on HiFiTTS-2-44.1k, real SFT recordings, and DSP-simulated character voices.It runs on 64 A100 GPUs with audio up to 30 seconds.Source: 2) Diffusion Transformer Training · p. 6
• Quality refinement fine-tunes the model for 10k steps using real recordings and a small set of high-quality DSP-simulated data, on 16 A100 GPUs with audio up to 30 seconds.Source: 2) Diffusion Transformer Training · p. 6

V. RESULTS AND ANALYSIS

• The results section covers generation for natural and character voices, then evaluates voice reuse and editing through zero-shot cloning and voice editing.It also presents ablations and qualitative analyses of component contributions.Source: §V · p. 6

A. Evaluation on Voice Generation

• Subjective evaluation uses TTV-Traits for diverse human voices and TTV-Character for human and non-human voices across 50 characters.TTV-Character rates prompt consistency, voice usability, and audio quality on 5-point MOS scales; 100 trained listeners evaluated recordings, with each rated by at least four listeners.Source: A. Evaluation on Voice Generation · pp. 6–7
• VoiceDesigner has the strongest Style-ACC and lowest WER, and its MOS-C exceeds open-source baselines on both benchmarks but remains below the leading commercial system.Overall, it achieves superior prompt–voice coherence among open-source systems while maintaining competitive usability and perceptual quality.Source: Table III · p. 7, A. Evaluation on Voice Generation · p. 7
• VoiceDesigner’s MOS-U is competitive with Qwen3TTS-VoiceDesign and exceeds CapSpeech-NAR, while enhancement improves its MOS-Q and narrows the quality gap.Qwen3TTS-VoiceDesign produces particularly clean outputs; VoiceDesigner’s usability reflects stable prosody, appropriate pausing, and expressive delivery across human and non-human voices.Source: Table III · p. 7, A. Evaluation on Voice Generation · p. 7
• VoiceDesigner’s raw outputs can retain ambience or background noise, so results also include an externally enhanced variant; cloning and editing need no additional enhancement.The enhancement estimates attainable perceptual quality without changing the underlying generation model; ElevenLabs and Qwen3TTS appear to use proprietary post-processing.Source: A. Evaluation on Voice Generation · p. 7

B. Evaluation on Voice Cloning

• Voice cloning is evaluated under a commonly used zero-shot TTS protocol on the SeedTTS test-en benchmark for human voices [64].The supplied passage ends before specifying the comparison.Source: B. Evaluation on Voice Cloning · p. 7

C. Evaluation on Voice Editing

• VoiceDesigner surpasses Step-Audio-EditX and IndexTTS-2 in voice similarity and instruction-following accuracy for fine-grained voice style editing.Step-Audio-EditX occasionally introduces audible artifacts, and both baselines perform inconsistently on certain styles.Source: Table VI · p. 8, C. Evaluation on Voice Editing · p. 9
• Voice editing experiments assess emotion and speaking-style modifications using professional voice actors’ expressive speech, which was never used in VoiceDesigner’s three training stages.The evaluation focuses on these modifications because existing editing approaches primarily target them.Source: C. Evaluation on Voice Editing · p. 8
• The editing evaluation constructs 100 pairs across emotional and stylistic transformations, excluding ambiguous or weakly defined styles.The protocol follows Step-Audio-EditX.Source: C. Evaluation on Voice Editing · p. 8
• Each source recording from 10 speakers is edited into two target styles, with VoiceDesigner compared against Step-Audio-EditX and IndexTTS-2’s text-prompt emotion control..Source: C. Evaluation on Voice Editing · p. 8
• Objective editing similarity is measured as SIM-t against the target-style recording using WeSpeaker ResNet-34 embeddings.The passage states this embedding method is more robust and effective than ECAPA and X-vector [67].Source: C. Evaluation on Voice Editing · p. 8
• A 5-point MOS-E listening test measures how accurately edits reflect instructions relative to the original recording.110 trained listeners each rated 25 recordings, and every recording received at least four ratings.Source: C. Evaluation on Voice Editing · p. 9

1) Architecture Comparison:

• The compared designs are CapSpeech-NAR, which adds cross-attention to F5-TTS for TTV, and VD-DiT-v1, -v2, and -v3.VD-DiT-v1 uses original single-stream MM-DiT, [30]; v2 adds token-level AdaLN, and v3 adds 3D-RoPE.Source: 1) Architecture Comparison · p. 9
• For voice cloning, CapSpeech-NAR and VD-DiT-v1 use the F5-TTS and E2-TTS speaker-conditioning strategy, while v1 and v2 keep speaker-condition tokens noise-free.Speaker embeddings are concatenated with noisy latents, masked regions receive empty tokens, and speaker-condition tokens use flow-matching time step 1.Source: 1) Architecture Comparison · p. 9
• The ablation uses approximately 0.3B-parameter models trained on a 5,000-hour HiFiTTS-2 subset, evaluating zero-shot voice cloning and TTV on held-out speakers.It reserves 200 speakers for evaluation and constructs 400 reference–utterance pairs for cloning and 400 prompt–utterance pairs for TTV; metrics are WER and SIM-o for cloning and Style-Acc for TTV.Source: 1) Architecture Comparison · p. 9
• VD-DiT-v1 converges faster and achieves lower WER than CapSpeech-NAR, but its early voice-cloning advantage over CapSpeech-NAR weakens later in training.The paper suggests CapSpeech-NAR needs more training because its unified positional space must disentangle and align audio and text tokens.Source: 1) Architecture Comparison · p. 9

2) Generative Simulation Data Augmentation:

• The augmentation ablation uses E1 and E2 emotional speech corpora, adding voice-cloned and voice-converted samples to address limited duration and linguistic diversity.Models are initialized from a Stage-1 checkpoint and evaluated on a simplified objective dataset without accent attributes using Style-Acc and WER.Source: 2) Generative Simulation Data… · p. 9
• Generative data augmentation consistently improves voice-generation linguistic fidelity and instruction adherence over real-data-only training, as shown in Figure 5.The authors say these results demonstrate improvements in both content accuracy and style controllability.Source: 2) Generative Simulation Data… · p. 9

3) Runtime Analysis:

• VoiceDesigner’s RTF is 0.36 for voice generation versus CapSpeech (0.33) and Qwen3-TTS (1.21), and 0.42 for editing versus Step-Audio-EditX (0.80) and IndexTTS2 (0.72).RTF was averaged over 200 test utterances of approximately 15 seconds on one NVIDIA RTX 4090 GPU.Source: 3) Runtime Analysis · p. 9

VI. CONCLUSION

• VoiceDesigner unifies text-to-voice generation and editing for diverse voice types and flexible editing instructions.The conclusion frames the work as addressing limited voice diversity and weak editing capability in existing TTV systems.Source: §VI · p. 9
• Its hybrid data-construction pipeline expands coverage to human and non-human voices.Source: §VI · p. 9
• A MM-DiT-based architecture with modality-aware conditioning jointly models voice generation, cloning, and instruction-guided editing.Source: §VI · p. 9
• Future work targets larger real-world character-style datasets, higher-fidelity modeling, and narrowing the gap with commercial systems through more diverse data and model scaling.Source: §VI · p. 9