PaperlayerView exact revision on arXiv

Source-linked AI summary

DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors

Keon Lee, Dong Won Kim, Jaehyeon Kim, Seungjun Chung, Jaewoong Cho

arXiv:2406.11427v2eess.AScs.AIcs.CLcs.LGcs.SD

TL;DR

Prior latent diffusion TTS systems rely on phonemes and durations for alignment, and factor-free approaches have not yet matched their performance. DiTTo-TTS combines a Diffusion Transformer with total-length prediction and text-speech latent alignment; it reports competitive zero-shot performance without those domain-specific factors.

  • Problem

    Prior LDM-based TTS models rely on phonemes and durations for text-speech alignment, while recent factor-free approaches have shown potential but suboptimal performance.

    Source: §1 · p. 1

  • Method

    DiTTo-TTS uses a Diffusion Transformer with variable-length speech generation and text-speech latent alignment, without phonemes or durations.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    DiTTo-TTS matches or surpasses state-of-the-art models in zero-shot naturalness, intelligibility, and speaker similarity; its base model is 4.6 times faster than CLaM-TTS.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The reported findings show that LDM-based TTS can scale across data and model sizes without relying on phonemes or durations.

    Source: §1 · p. 2, §7 · p. 10

    Generating voices, including those of real individuals, from minimal input presents potential risks of voice spoofing, impersonation, or deepfake audio.

    Source: §8 · p. 11

Abstract

from arXiv · show

Large-scale latent diffusion models (LDMs) excel in content generation across various modalities, but their reliance on phonemes and durations in text-to-speech (TTS) limits scalability and access from other fields. While recent studies show potential in removing these domain-specific factors, performance remains suboptimal. In this work, we introduce DiTTo-TTS, a Diffusion Transformer (DiT)-based TTS model, to investigate whether LDM-based TTS can achieve state-of-the-art performance without domain-specific factors. Through rigorous analysis and empirical exploration, we find that (1) DiT with minimal modifications outperforms U-Net, (2) variable-length modeling with a speech length predictor significantly improves results over fixed-length approaches, and (3) conditions like semantic alignment in speech latent representations are key to further enhancement. By scaling our training data to 82K hours and the model size to 790M parameters, we achieve superior or comparable zero-shot performance to state-of-the-art TTS models in naturalness, intelligibility, and speaker similarity, all without relying on domain-specific factors. Speech samples are available at https://ditto-tts.github.io.

1 INTRODUCTION

DiTTo-TTS investigates whether latent diffusion can achieve strong TTS performance without phonemes or durations, which prior LDM-based TTS systems use for alignment. The paper reports competitive results and examines architecture, variable-length modeling, and latent alignment.

Key findings

  • DiTTo-TTS achieves superior or comparable zero-shot performance to state-of-the-art TTS models in naturalness, intelligibility, and speaker similarity, without phonemes or durations.The base-sized model is 4.6 times faster than CLaM-TTS while using 3.84 times less model size.Source: §1 · p. 2
  • DiT with long skip connections and global AdaLN is more suitable for TTS than U-Net, according to the paper’s architecture comparisons.The paper describes these as minimal modifications to the DiT architecture.Source: §1 · p. 2
  • Predicting total speech length at inference improves performance over fixed-length modeling, which prior work used.The Speech Length Predictor uses the text and speech prompt to predict the total length.Source: §1 · p. 2
  • Aligning text and speech latents is important for effective cross-attention, using either a jointly trained text encoder or a speech autoencoder with an auxiliary language-modeling objective.Source: §1 · p. 2

2 RELATED WORK

Prior LDM-based TTS models rely on phonemes and durations for precise alignment, while newer factor-free approaches simplify the framework but retain fixed-length modeling or other distinct design choices.

LDMs for TTS

  • LDM-based TTS has used phonemes and phoneme-level durations to meet the demands of precise text-speech alignment and audio fidelity.Source: §2 · p. 2

TTS without domain-specific factors

  • Simple-TTS and E3 TTS remove phonemization and phone-level duration prediction but impose fixed target lengths, with padding that can restrict length flexibility and reduce quality.Source: §2 · p. 2

Concurrent work

  • SimpleSpeech predicts speech length with an LLM and uses a scalar-quantized codec, whereas E2 TTS simplifies text conditioning and removes explicit alignment.Source: §2 · p. 2

3 METHOD

DiTTo-TTS uses latent diffusion conditioned on text, with span masking for prompted speech generation, total-length prediction, and a DiT architecture designed for text-speech conditioning.

3.1 PRELIMINARY

  • The model trains a latent diffusion process to predict speech representations from noisy latents and text, using random span masking to support zero-shot audio prompting.Masking also identifies the portion to generate; the loss is modified to account for the mask.Source: §3.1 · pp. 3–4

3.2 MODEL AND TRAINING

  • Speech latents are aligned with linguistic latents by fine-tuning a neural audio codec with an auxiliary language-modeling objective.Gradients update the latent projection while the language-model decoder remains fixed.Source: §3.2 · p. 4
  • The speech length predictor estimates total output length from text and a speech prompt, while the diffusion model denoises a speech latent conditioned on text.Its causal decoder predicts the remaining token count, and the predictor is trained separately with cross-entropy loss.Source: §3.2 · p. 4

3.3 MODEL ARCHITECTURE

  • DiT uses cross-attention and global AdaLN, while Mel-VAE compresses audio sequences about 7–8 times more than EnCodec into a 10.76 Hz code.The architecture search selected DiT; modeling overall length leaves detailed token durations to cross-attention.Source: §3.3 · pp. 4–5

4 EXPERIMENTAL SETUP

Experiments use English and multilingual speech data, several DiTTo model sizes, and objective and subjective measures across continuation and cross-sentence tasks.

Dataset

  • Training data comprise 82K hours from over 12K speakers across nine languages, with separate English-only and multilingual models.DiTTo-en uses 55K English hours; DiTTo-multi uses the 82K-hour multilingual set.Source: §4 · p. 5

Training

  • The study trains four English model sizes and one multilingual XL model, with 739.97M parameters in DiTTo-en-XL and 790.44M in DiTTo-multi.Source: §4 · p. 5

Metrics

  • Evaluation uses CER and WER for intelligibility and robustness, SIM for speaker identity, SMOS for speaker similarity, and CMOS for relative naturalness and audio quality.Source: §4 · p. 5

Baselines and tasks

  • Baselines include autoregressive, non-autoregressive, and diffusion-based TTS models; continuation and cross-sentence tasks use a three-second speech prompt.Some baseline scores come from published papers or demo samples because those models are not publicly available.Source: §4 · p. 6

5 EXPERIMENTAL RESULTS ADDRESSING OUR RESEARCH QUESTIONS

The experiments assess whether LDM-based TTS can perform well without domain-specific factors and which design choices support that performance. DiTTo performs competitively against TTS baselines, while variable-length modeling and closer text-speech representation alignment improve results.

5.1 WE CAN ACHIEVE STATE-OF-THE-ART PERFORMANCE WITHOUT DOMAIN-SPECIFIC FACTORS (RQ1)

  • DiTTo-en ranks first or second across continuation and cross-sentence measures; its base model beats CLaM-TTS across quality dimensions while running 4.6 times faster with 3.84 times smaller model size.DiTTo-en also performs comparably to more complex non-autoregressive models and surpasses Simple-TTS.Source: Table 1 · p. 6, §5.1 · pp. 6–7, Table 2 · p. 7
  • Human evaluations find DiTTo-en significantly outperforms Simple-TTS and CLaM-en, approaching ground truth in speaker similarity, naturalness, quality, and intelligibility.The evaluation uses 40 LibriSpeech test-clean samples per speaker.Source: Table 3 · p. 7, §5.1 · p. 7
  • Performance improves with model size, and separate experiments show model performance improving as training data scale increases.The dataset subsets range from 0.5K to 50.5K hours.Source: §5.1 · pp. 7–8

5.2 KEY ASPECTS BEHIND THE SCENE (RQ2)

  • DiT is more suitable for TTS than U-Net, especially when domain-specific factors are removed; Mel-VAE compression causes information loss and poor U-Net output quality.Flat-U-Net improves on U-Net but remains short of DiTTo-mls.Source: §5.2 · p. 8, Table 4 · p. 8
  • Variable-length modeling significantly outperforms fixed-length approaches, whose performance degrades as padding increases; changing latent length also enables speech-rate control.The speech length predictor estimates total speech length rather than phoneme durations.Source: §5.2 · p. 8, Table 5 · p. 8, Figure 2 · p. 9
  • SpeechT5 text conditioning outperforms ByT5-base in speech accuracy, and jointly modeling text or speech improves results; the best results occur when both encoders are jointly trained.Fine-tuning only the audio codec is also reported to improve alignment and TTS performance efficiently.Source: §5.2 · pp. 8–9, Table 6 · p. 9

6 ABLATION STUDY

The ablations identify architectural choices and latent compression as consequential for DiTTo, with the final design favoring a block-spanning skip connection and Mel-VAE.

6 ABLATION STUDY

  • The final DiTTo-mls architecture uses a residual connection spanning the transformer block, and Table 7's ablation results confirm the design's effectiveness.Connecting hidden layers inside the block or omitting skip connections is less effective for convergence.Source: §6 · p. 9, Table 7 · p. 10
  • Sharing AdaLN reduces XL-model parameters by 30.5% and improves performance, while omitting pooled-text slightly declines performance.The final design adopts a long skip connection between transformer blocks rather than U-ViT-style internal skips or no skip.Source: §6 · pp. 9–10
  • Removing quantization at decoding produces minimal performance differences, while experiments across Mel-VAE compression ratios find better performance with shorter latents.The codec comparison also reports higher PESQ and ViSQOL for DAC, despite Mel-VAE's stronger WER and SIM-r results.Source: §6 · p. 10
  • Mel-VAE performs best in WER and SIM-r among the compared codecs, while its shorter latents support more efficient training and inference.DAC has longer, higher-dimensional latents; EnCodec also has longer latents and poorer codec performance.Source: §6 · p. 10

7 CONCLUSION

DiTTo-TTS compares to or surpasses state-of-the-art models in zero-shot naturalness, intelligibility, and speaker similarity without phonemes or durations. The conclusion identifies architecture search, total-length prediction, and text-speech latent alignment as central to this result.

7 CONCLUSION

  • DiTTo-TTS compares to or surpasses state-of-the-art models in zero-shot naturalness, intelligibility, and speaker similarity without phonemes or durations.The paper reports that the model scales with data and model size.Source: §7 · p. 10
  • The conclusion attributes the outcome to architecture search, variable-length modeling through total-length prediction, and text-speech latent alignment for cross-attention learning.The authors identify applying advances from other LDM modalities and cross-domain knowledge from pretrained DiTs as future directions.Source: §7 · p. 10

8 ETHICS STATEMENTS

The ethics discussion identifies misuse risks from zero-shot voice synthesis and calls for responsible deployment. It also describes privacy- and licensing-related constraints on the speech data used.

8 ETHICS STATEMENTS

  • Generating real individuals' voices with minimal input presents potential risks of voice spoofing, impersonation, and deepfake audio.The authors state that these risks make ethical considerations and responsible deployment important.Source: §8 · p. 11
  • The authors call for synthetic-audio detection, responsible-use and voice-cloning consent guidelines, and regulatory frameworks to reduce potential harm.These are proposed measures, not reported evaluations of mitigation effectiveness.Source: §8 · p. 11
  • Training and evaluation use publicly available, research-permitted speech datasets without PII, metadata, or speaker IDs, relying on text-audio pairs.The passage lists datasets including MLS, GigaSpeech, LibriTTS-R, VCTK, LJSpeech, Expresso, and AI-Hub datasets.Source: §8 · p. 11

9 REPRODUCIBILITY STATEMENTS

The reproducibility statement points to documentation of the architecture, data, preprocessing, training configurations, and evaluation methods, and describes a conditional plan for code and model release.

9 REPRODUCIBILITY STATEMENTS

  • Figure 1 presents the architecture overview, with detailed components in Section 3.3 and hyperparameters in Appendix A.5.The statement says these materials provide insight into model configurations.Source: §9 · p. 11
  • The paper directs readers to Appendix A.2 for training-data lists and statistics, Section 4 for preprocessing, and Section 4 and Appendix A.3 for training and evaluation details.Source: §9 · p. 11
  • Inference code, pretrained weights, and eventually the full training implementation are planned for gradual release if legal concerns can be addressed.Source: §9 · p. 11

A.1 RELATED WORK

Prior work explores large-scale autoregressive and non-autoregressive TTS, including diffusion models that avoid phonemes and phone-level durations. DiTTo-TTS builds on that potential while addressing the need for broader baseline comparisons and analysis of what enables performance.

A.1 RELATED WORK

  • Seed-TTSDiT shows diffusion TTS can avoid phonemes and phoneme-level durations, but lacks broad baseline comparisons and analysis of the factors behind success.DiTTo-TTS frames those comparisons and factor analysis as its central focus.Source: §A.1 · p. 19
  • LLM-based TTS includes VALL-E’s zero-shot speech-token modeling and CLaM-TTS’s compressed tokens, while resource-intensive inference limits such applications.VALL-E uses EnCodec; CLaM-TTS uses Mel-VAE to predict multiple tokens simultaneously.Source: §A.1 · p. 19
  • Non-autoregressive approaches include Voicebox’s speech infilling and NaturalSpeech’s auxiliary controls, but supplementary data and modules can hinder efficiency and scalability.Voicebox uses flow matching; NaturalSpeech builds on LDMs to control content, prosody, and timbre.Source: §A.1 · p. 19
  • EnCodec maps 24,000 Hz mono audio to 75 Hz latents, while speech-focused Mel-VAE compresses mel-spectrograms to approximately 10.76 Hz latents.The shorter latent sequence lowers the speech-generation module’s computational cost.Source: §A.1 · p. 19

A.2 DATASET DETAILS

The datasets span English, Korean, and seven additional MLS languages, with varied speaker, domain, and expressive-speech coverage. The speech-length predictor also uses LibriSpeech to reduce its dependence on text normalization, and preprocessing is designed to reduce data-read overhead.

A.2 DATASET DETAILS

  • English training combines MLS, GigaSpeech, LibriTTS-R, VCTK, and LJSpeech, with 40 hours of expressive Expresso speech added to DiTTo-en.GigaSpeech has multi-domain transcribed speech but lacks speaker information; VCTK is multi-speaker and LJSpeech single-speaker.Source: §A.2 · p. 20
  • LibriSpeech is added to speech-length-predictor training because its varied speech lengths reduce dependence on text normalization beyond MLS’s roughly 10–20-second recordings.With normalized input text, the predictor otherwise tends to produce lengths in the range associated with MLS.Source: §A.2 · p. 20
  • Korean training includes everyday reading, expressive professional voices, conversational speech, and broadcast speech, including 1,000 hours of expressive recordings.AIHub-expressive covers 89 speakers, seven speech styles, and four emotional tones.Source: §A.2 · p. 20
  • The other MLS language subsets are German, Dutch, French, Spanish, Italian, Portuguese, and Polish.Source: §A.2 · p. 20
  • Audio and metadata are stored in compressed TAR files containing parquet batches, then decompressed and deserialized during training to reduce data-read overhead.Each parquet batch contains 10K audio-metadata pairs.Source: §A.2 · p. 20

A.3 EXPERIMENTAL SETUP DETAILS

This section describes DiTTo’s evaluation protocols and experimental configurations, and examines how data scale, variable-length modeling, conditioning, and latent choices affect performance across model comparisons and evaluation settings.

A.3 EXPERIMENTAL SETUP DETAILS / A.4 BASELINE DETAILS

  • English evaluations use HuBERT-Large transcripts and multilingual evaluations Whisper; WavLM-TDCNN measures SIM-o and SIM-r, and MTurk SMOS uses 1–5 while CMOS uses −3 to 3; samples are 16 kHz except subjective demo samples.StyleTTS 2 and XTTS-v2 are open-source models in objective evaluation; RAVDESS demo comparisons include Mega-TTS 2, StyleTTS 2, HierSpeech++, SimpleSpeech, and E2 TTS.Source: §A.3 · p. 21, §A.4 · p. 21

A.5 MEL-VAE AND MODEL CONFIGURATION

  • DiTTo uses pretrained Mel-VAE latents and a modified DiT with text-conditioned cross-attention and global AdaLN, RoPE, gated MLPs, and a final long skip connection.Across four model scales, SMOS improves, the CMOS gap with ground truth decreases, and the gap with the baseline widens.Source: §A.5 · pp. 24–25, Table 10 · p. 25

A.6 NOISE SCHEDULE AND CLASSIFIER-FREE GUIDANCE

  • The selected sampling configuration uses scale-shift 0.3, CFG scale 5.0, and 25 diffusion steps; 25 steps suffice for high-fidelity outputs.Training drops text input with probability p = 0.1; inference removes text conditioning by masking cross-attention embeddings and zeroing the mean-pooled text embedding.Source: §A.6 · pp. 25–26, Figure 6 · p. 26

A.7 TRAINING AND INFERENCE OF SPEECH LENGTH PREDICTOR

  • Speech-length predictors use the corresponding DiTTo text encoder, train for 600K steps, and at inference use top-k 20 with a maximum length of 2048.For evaluation, the expected test-sample length is used; causal masking trains the predictor to determine total length from a prompt, while a single inference pass predicts remaining lengths simultaneously.Source: §A.7 · pp. 26–27

A.8 COMPARISON WITH OPEN-SOURCE MODELS

  • In the English-only cross-sentence evaluation, DiTTo significantly outperforms StyleTTS 2 in SIM-o, while StyleTTS 2 has slightly higher WER and CER; XTTS-v2 also performs below DiTTo-en-S.Source: §A.8 · p. 27, Table 11 · p. 27

A.9 HUMAN EVALUATIONS ON COMPARISON OF DEMO SAMPLES / A.10 HUMAN EVALUATIONS ON COMPARISON WITH CONCURRENT WORKS

A.11 HUMAN EVALUATIONS ON COMPARISON OF OUT-OF-DOMAIN RAVDESS DEMO SAMPLES

  • On the RAVDESS benchmark, DiTTo outperforms all baselines with MOS comparable to NaturalSpeech 3, and has superior CMOS in additional zero-shot emotion settings.The authors report that this demonstrates performance with out-of-domain audio prompts.Source: Table 14 · p. 28, Table 15 · p. 28, §A.11 · p. 29

A.13 SCALING DATA

  • At 0.5K training hours, WER=3.30 and SIM-r=0.5782; at 5.5K, WER improves to 3.14, and larger datasets yield further gains, though 0.5K has lower accuracy than P-Flow.Speaker similarity remains similar or improves incrementally as data scale increases.Source: §A.13 · p. 30, Table 18 · p. 30
  • WER benefits become more apparent as dataset size increases, although performance drops on smaller datasets, especially at 0.5K, despite improved Codec PESQ and ViSQOL.With approximately 50 hours of data, the model overfit and early stopping at 40K steps yielded WER 9.25 and SIM-r 0.3336; the authors caution against direct comparison with Voicebox’s setting.Source: §A.13 · p. 30

A.14 SPEECH LENGTH MODELING COMPARISON

  • Variable-length modeling improves performance over fixed-length approaches, and excluding padding from the target distribution is essential.Including a larger proportion of padding in the loss worsened performance; evaluation used five randomly selected samples per speaker from 42 MLS English test speakers.Source: §A.14 · p. 30

A.15 MEL-VAE++

  • Semantic content injection improves Mel-VAE++ metrics, particularly with ByT5-base, while HuBERT feature matching further improves Speaker Similarity.CAD analysis found less fluctuation and sample variation with semantic injection; text-encoder interaction occurs mainly in early reverse-process stages, leaving later DiT steps to improve acoustic quality.Source: §A.15 · pp. 31–32, Table 20 · p. 31, Figure 8 · p. 31

A.17 MEANING OF LATENT QUANTIZATION IN LDM-BASED TTS

  • Quantization is crucial for producing a good autoencoder latent, but quantization itself is irrelevant when DiTTo uses that latent as its target.The authors describe probabilistic RVQ as enabling Mel-VAE’s time-domain length compression while retaining high-quality speech reconstruction.Source: §A.17 · p. 32

A.18 IMPACT OF TRAINING STEPS ON PERFORMANCE

  • DiTTo-mls achieves reasonable WER and SIM performance by 100K training steps.The key-aspects experiments trained for 200K steps on the MLS English subset in the base configuration, using SpeechT5, original Mel-VAE, span masking, L1 loss, and a 20-second data limit; total training time was one day.Source: Table 21 · p. 32, §A.18 · p. 32, §A.16 · p. 32

A.19 TRADE-OFF BETWEEN WER AND SIM PERFORMANCE IN DITTO-DAC

  • WER 7.21→4.70 as DiTTo-dac-24k scale-shift falls from 0.3 to 0.2, but SIM-o drops 0.5478→0.5302; even optimized, it does not beat DiTTo-mls on either metric.At scale-shift 0.1, most diffusion timesteps become excessively noisy, negatively affecting both WER and SIM.Source: Table 22 · p. 33, §A.19 · p. 33

A.20 PERFORMANCE OF MEL-VAE VARIANTS WITH DIFFERENT COMPRESSION RATIOS

  • Mel-VAE performance improves as time-domain compression increases from 2× to 8×, corresponding to shorter latent sequences.The 8× ratio is the original Mel-VAE configuration used in the paper.Source: Table 23 · p. 33, §A.20 · p. 33

A.21 TARGETING MEL-SPECTROGRAMS WITHOUT LATENT MODELINGS

  • Greater latent compression (DiTTo-mls > DiTTo-encodec > DiTTo-mel) improves WER; DiTTo-mls balances similarity and intelligibility, and subjective SMOS/CMOS evaluations favor it.DiTTo-mel has the highest SIM-o, while DiTTo-mls remains faster than Voicebox at the same number of diffusion steps.Source: Table 24 · p. 34, §A.21 · p. 34
Loading 2406.11427v2…