Source-linked AI summary
DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, Yuxuan Wang
TL;DR
Existing approaches to autoregressively generating continuous speech representations can be computationally demanding or produce suboptimal results. DiTAR combines a language model for inter-patch prediction with a diffusion transformer for localized patch generation, and reports state-of-the-art zero-shot speech quality with substantially lower computational requirements.
Problem
Autoregressive generation of continuous speech representations remains challenging because existing diffusion-based approaches can require excessive computation or yield suboptimal outcomes.
Method
DiTAR uses a divide-and-conquer patch-based framework in which a language model predicts across aggregated patches and a diffusion transformer predicts within the next patch.
Results
DiTAR achieves state-of-the-art robustness, speaker similarity, and naturalness in zero-shot speech generation while requiring substantially less computational power.
Takeaways & Limitations
DiTAR combines autoregressive language-modeling capabilities with diffusion-based continuous representation generation and reduced computational demands.
Takeaways & Limitations
The zero-shot TTS evaluation standardizes prompt audio, texts, and tools to support fair comparison between systems.
Abstract
from arXiv · showhide
Several recent studies have attempted to autoregressively generate continuous speech representations without discrete speech tokens by combining diffusion and autoregressive models, yet they often face challenges with excessive computational loads or suboptimal outcomes. In this work, we propose Diffusion Transformer Autoregressive Modeling (DiTAR), a patch-based autoregressive framework combining a language model with a diffusion transformer. This approach significantly enhances the efficacy of autoregressive models for continuous tokens and reduces computational demands. DiTAR utilizes a divide-and-conquer strategy for patch generation, where the language model processes aggregated patch embeddings and the diffusion transformer subsequently generates the next patch based on the output of the language model. For inference, we propose defining temperature as the time point of introducing noise during the reverse diffusion ODE to balance diversity and determinism. We also show in the extensive scaling analysis that DiTAR has superb scalability. In zero-shot speech generation, DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness.
1 Introduction
DiTAR addresses the difficulty of autoregressively generating high-quality continuous speech representations under practical computational constraints. It combines language-model inter-patch prediction with diffusion-based intra-patch generation and reports state-of-the-art zero-shot speech results.
- Continuous representations can preserve complex multimodal information better than discrete tokens, whose fidelity is limited by bitrate constraints.
- Causal-attention diffusion language-model combinations can perform poorly, while approaches that repurpose language-model parameters for diffusion impose substantial computational demands.
- DiTAR divides continuous tokens into patches, assigning inter-patch prediction to a language model and intra-patch prediction to a bidirectional diffusion transformer.Patchification also shortens the language model’s context sequence, reducing computational load.
- DiTAR defines temperature as the noise-introduction point along the reverse diffusion ODE trajectory to balance diversity and determinism.The method is designed for faster ODE solvers rather than the numerous steps required by prior SDE-based sampling.
- DiTAR achieves state-of-the-art robustness, speaker similarity, and naturalness in zero-shot text-to-speech with much lower computational power than competing models.
2 Related Work
Prior work combines autoregressive modeling and diffusion for multimodal or speech generation, but often incurs high computation or remains poorly suited to continuous autoregressive prediction. DiTAR uses patchification to reduce sequence length while enabling bidirectional local modeling.
- Integrating Autoregressive Language Model and Diffusion: Existing integrations of autoregressive language models and diffusion either increase computational demands as sequences and models grow or perform poorly with causal attention.
- Patchification in Generative Modeling: Patchification is widely used in speech, image, and video generation primarily to shorten sequences and reduce computational load.
- Patchification in Generative Modeling: In DiTAR, patchification additionally enables bidirectional modeling within patches, improving modeling effectiveness in the autoregressive framework.
- Zero-Shot Text-to-Speech: Zero-shot TTS systems use either multi-stage coarse-to-fine pipelines or single-stage generation of high-information continuous representations.Multi-stage systems commonly predict low-information discrete semantic or prosodic representations before refinement.
3 Approach
DiTAR combines causal autoregressive modeling across patches with bidirectional diffusion within each patch of continuous tokens. Its design also includes context-aware generation, LM guidance, and ODE-compatible temperature sampling for zero-shot TTS.
- 3.1 Formulation: DiTAR factorizes continuous-token generation autoregressively, aggregating local tokens into patches for long-context modeling and bidirectional intra-patch prediction.The autoregressive model uses θa for long-context learning, while θb is a bidirectional diffusion transformer for next-patch prediction.
- 3.2 Zero-shot TTS: For zero-shot TTS, prompting text, target text, and prompting speech form prefix context, after which the model autoregressively generates target speech.The speech representation uses a VAE-based pipeline that compresses 24000hz waveforms into 40Hz latents with dimension 64.
- 3.1.2 Overall architecture: The language model predicts inter-patch structure, while LocDiT generates the following localized patch conditioned on the language model’s output.Historical patches can also serve as prefix inputs to LocDiT, aligning generation with an outpainting-style task.
- 3.3 LM Guidance: DiTAR uses LM guidance by randomly replacing language-model conditioning with a dummy embedding during training and combining conditional and unconditional diffusion outputs at inference.The method requires two diffusion-head computations and one language-model computation.
- 3.4 Temperature for Continuous-Valued LMs: Temperature τ is defined as the time point where noise enters the reverse diffusion ODE, interpolating between deterministic sampling at τ = 0 and standard ODE sampling at τ = 1.Intermediate values introduce random noise during reverse integration to balance diversity and stability.
4 Experiments
Experiments evaluate DiTAR’s quality, efficiency, scaling, sampling, and architectural choices. DiTAR achieves strong robustness, speaker similarity, naturalness, and audio quality while reducing computational demands.
- Experimental results: DiTAR consistently delivers the best WER across two training-data configurations and test sets, matching NAR systems with phone-level duration models.
- Experimental results: DiTAR matches NAR systems objectively in speaker similarity and surpasses leading NAR systems subjectively in S-MOS.
- Experimental results: DiTAR achieves the highest naturalness scores among leading NAR systems in subjective evaluation.
- Experimental results: DiTAR achieves strong UTMOS, leads on subset B, and exceeds ground truth in subjective Q-MOS.On LibriSpeech subset A, DiTAR scores just behind NaturalSpeech3 on UTMOS.
- Inference efficiency: DiTAR reduces computational demands by approximately 3 ∼43× compared with other NAR systems, while hybrid systems use less computation but produce lower-quality outputs.
- Scaling: WER and SIM consistently improve as training data increases from 20k to 280k hours or model size increases from 0.1B to 1B parameters.
- Scaling: Enlarging the language model and LocDiT improves performance, whereas increasing encoder size has little effect.
- Ablation study: Performance declines with excessively small or large LocDiT patches, while historical context markedly improves synthesis and one historical patch balances efficiency and performance.
5 Conclusion
DiTAR combines a diffusion transformer for localized patch generation with a language model’s autoregressive structure. It introduces ODE-based temperature sampling and achieves strong zero-shot speech synthesis with lower computational requirements.
- DiTAR uses a diffusion transformer to generate localized patches while retaining the autoregressive features of language models.
- DiTAR defines inference temperature as the noise-introduction time during reverse diffusion ODE solving.
- In zero-shot speech synthesis, DiTAR achieves state-of-the-art robustness, speaker similarity, and naturalness with substantially lower computational requirements.
A.1 Training Details
Training uses speech datasets processed into phonetic text representations and large-scale GPU optimization.
- LibriLight is transcribed with an internal ASR system and converted to phonemes, while Emilia uses G2P conversion on official transcriptions.
- DiTAR is trained with AdamW at a constant learning rate of 1e-4 for 0.5M steps on 16 A100 GPUs, while the 1B model uses 32 A100 GPUs.
A.2 Model Configuration for Scaling
Scaling experiments evaluate DiTAR models ranging from 0.1B to 1B parameters using configurations detailed in Table 8.
- Scaling validation trains four DiTAR model sizes ranging from 0.1 billion to 1 billion parameters.
- Table 8 provides the configurations for DiTAR models with different sizes.
A.3 Derivation of xθ in temperature sampling for different parameterized diffusion
The section derives xθ, the predicted clean data, under diffusion parameterizations based on xt = α_tx0 + σ_tε. In x0-prediction it is directly the model output, while v-prediction requires rearranging v and x0.
- xθ denotes the predicted data under different diffusion parameterizations.The diffusion process is defined as xt = α_tx0 + σ_tε, with x0 drawn from q(x0) and ε standard Gaussian noise.
- In x0-prediction mode, xθ(xt, t) is exactly the model’s prediction.
- In v-prediction mode, also called flow-matching, the derivation proceeds by rearranging v and x0.
A.4 Calculation of FLOPs
The FLOPs analysis decomposes computational cost by architecture and layer type, using explicit assumptions for a 3-second prompt and 10-second synthesis. It separately accounts for causal-transformer prefix and autoregressive computation, non-causal transformers, and convolutional layers.
- FLOPs are evaluated for synthesizing 10 seconds of audio from a 3-second prompt at a 7 Hz text frame rate.This gives a prompt text length of 21 and target text length of 70; causal-transformer calculations include KV caching.
- Convolutional layers: A one-dimensional convolution network’s FLOPs depend on its layers, hidden channel size, kernel size, and input length.
- Causal transformers: For a causal transformer, FLOPs are computed from QKV, attention, V, FC, and feed-forward components multiplied by the number of layers.The formulation uses prefix input length Tpre and input length T, with the component expressions specified separately.
- Causal transformers: AR_Transformer_FLOPs is the sum of prefix FLOPs and autoregressive FLOPs.
B Subjective Evaluation
The subjective evaluation compares generated audios using pairwise and individual ratings, while the framework comparison highlights different placements of diffusion computation. The paper presents the rating interfaces and a comparison figure for these analyses.
- Subjective evaluation: Raters compare two audios against one reference and assign CMOS, N-MOS, Q-MOS, and S-MOS ratings.CMOS is scored after comparing the two audios, while the three MOS measures rate each audio individually.
- Subjective evaluation: Figures 7 and 8 show the user interface, rating interface, and questions presented to raters.
- Framework comparison: MAR and DiTAR place diffusion computation in a diffusion head, whereas ARDiT applies diffusion throughout the entire model.DiTAR structurally resembles a causal language model and becomes a continuous-valued LLM when scaled.
- Framework comparison: Figure 9 presents a comparison of different frameworks.