PaperlayerView exact revision on arXiv

Source-linked AI summary

DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis

Ziyu Zhang, Tianlun Zuo, Hanzhao Li, Haoyu Zhang, Lei Xie

arXiv:2609.13909v1cs.SDcs.AIeess.AS

TL;DR

Continuous-latent AR-DiT models remain unstable on long utterances and difficult linguistic structures. DiTAR+ combines Dilated Context Sampling with Hierarchical Acoustic Masking; it outperforms state-of-the-art models and improves robustness on challenging sentences while maintaining stable speaker similarity across long utterances.

  • Problem

    Continuous-latent AR-DiT models remain unstable on long utterances and linguistically difficult sentences, with restricted history and acoustic inertia weakening semantic alignment.

    Source: §1 · p. 2

  • Method

    DiTAR+ samples remote history while preserving temporal order, then masks acoustic history in shallow layers before deeper layers restore it for rendering.

    Source: §1 · pp. 2–3, §2.3 · pp. 6–7

  • Results

    DiTAR+ substantially outperforms state-of-the-art models and improves robustness on complex sentences while maintaining stable speaker similarity across long utterances.

    Source: §1 · p. 3, §4 · p. 14

  • Takeaways & Limitations

    Temporal phase coherence and semantic-acoustic decoupling are essential design principles for robust continuous-latent speech synthesis.

    Source: §4 · p. 14

    Standard DiTAR yields slightly higher Q-MOS than DiTAR+, which trades some micro-level acoustic smoothness for macro-level naturalness, semantic robustness, and long-term speaker stability.

    Source: Analysis of the Q-MOS Trade-off · p. 10

Abstract

from arXiv · show

Continuous-latent Autoregressive Diffusion Transformer (AR-DiT) models have demonstrated immense potential in zero-shot speech generation. However, they still suffer from limited decoding stability when synthesizing long utterances or complex linguistic structures. This instability primarily stems from a restricted historical receptive field and an acoustic inertia dependency within the diffusion decoder, which causes the model to ignore semantic conditions. To address these challenges, we propose DiTAR+, a dual-optimization framework. First, we introduce Dilated Context Sampling to expand the macro-level historical receptive field without violating physical temporal continuity, thereby preventing cumulative error propagation. Second, we propose Hierarchical Acoustic Masking to prevent shallow layers from attending to acoustic pre-context, explicitly decoupling semantic alignment from acoustic detail reconstruction. Extensive experiments show that our framework effectively mitigates pronunciation errors and semantic hallucinations, enhances generation robustness on challenging sentences, and maintains exceptionally high speaker similarity throughout the entirety of long-form utterances. On the linguistically challenging ZH-Hard set, DiTAR+ reduces the word error rate from 12.478% to 9.893%, and on extended utterances of 25 to 35 seconds it improves speaker similarity from 0.741 to 0.759 while simultaneously lowering the word error rate from 2.778% to 2.173%, outperforming both discrete-token and pure flow-matching baselines.

1 Introduction

Continuous-latent AR-DiT models face stability problems on long or linguistically difficult utterances. DiTAR+ combines Dilated Context Sampling and Hierarchical Acoustic Masking to address restricted historical context and acoustic inertia, improving robustness in challenging scenarios.

1 Introduction

  • Continuous-latent models retain richer acoustic detail than discrete codec-token methods, which can lose speaker characteristics, prosody, and expressive nuance through discretization.Source: §1 · pp. 1–2
  • AR-DiT instability on long or difficult utterances includes speaker-similarity degradation, repetition, pronunciation errors, and semantic misalignment.Source: §1 · p. 2
  • Restricted local history hides remote speaker and prosodic information, while acoustic inertia can weaken semantic guidance and cause decoding errors.Richer history may reinforce acoustic continuation over semantic alignment.Source: §1 · p. 2
  • Dilated Context Sampling expands the historical receptive field by sampling remote patches while preserving the physical order of continuous speech latents.Source: §1 · pp. 2–3
  • Hierarchical Acoustic Masking limits shallow layers' access to acoustic history, decoupling semantic alignment from acoustic-detail reconstruction.Source: §1 · p. 3
  • DiTAR+ outperforms baselines in challenging scenarios, generating complex sentences robustly while maintaining stable speaker similarity across long utterances.Source: §1 · p. 3

2 Method

DiTAR+ extends a two-stage autoregressive language-model/local-diffusion pipeline with DCS for wider ordered acoustic history and HAM for staged semantic and acoustic conditioning.

2.1 DiTAR Baseline

  • The baseline uses an autoregressive Transformer to produce semantic conditions for each patch, then a local diffusion decoder reconstructs fine-grained speech using the preceding patch as context.The language model conditions on text and reference audio; the decoder uses the semantic vector, diffusion timestep, noisy patch, and immediate acoustic history.Source: §2.1 · p. 3, Inter-Patch Semantic Generation · p. 3, Intra-Patch Acoustic Generation · p. 4

2.2 Dilated Context Sampling

  • DCS retains adjacent patches and adds stride-sampled distant patches, widening LocDiT’s context without reordering speech and mitigating long-range error accumulation and speaker drift.The dense local window maintains boundary smoothness while sparse historical patches extend the macro-receptive field.Source: §2.2 · p. 5

2.3 Hierarchical Acoustic Masking

  • HAM hides acoustic history from shallow layers so LocDiT first aligns to LM semantics, then restores history deeper for refinement, improving robustness on challenging long-form synthesis.This progressive conditioning is intended to reduce reliance on local acoustic shortcuts while preserving acoustic refinement using historical context.Source: §2.3 · pp. 6–7

3 Experiments

The experiments evaluate DiTAR+ on standard and challenging zero-shot speech tasks, testing both its overall performance and the distinct roles of DCS and HAM. Results indicate stronger robustness on difficult and extended utterances, alongside a trade-off in local acoustic smoothness.

3.1 Experimental Setup; 3.2 Compared Methods

  • Evaluation covers Seed-EN, Seed-ZH, ZH-Hard, and ZH-Long; DiTAR+ is benchmarked against DiTAR and CosyVoice 2, with objective speech metrics WER and SIM.ZH-Hard comprises 400 challenging utterances; ZH-Long comprises 400 extended sentences with expected acoustic durations of 25 to 35 seconds.Source: Evaluation Dataset · p. 8, Evaluation Metrics · p. 8, §3.2 · p. 8

3.3 Objective Evaluation

3.4 Subjective Evaluation

3.5 Ablation Study

  • DCS improves long-form stability: DiTAR + DCS reaches 2.165% WER and 0.757 speaker similarity on ZH-Long, while the baseline’s similarity falls to about 0.655 in the tail.The baseline degrades sharply after roughly 15–18 seconds; Fig. 2 shows DCS slowing that decline.Source: Effectiveness of DCS · p. 11, Table 3 · p. 11, Figure 2 · p. 12
  • HAM improves linguistic-complexity results on ZH-Hard, reaching 9.889% WER and 0.700 SIM, but alone does not extend the historical acoustic context window.Its SIM trajectory on extended generation tracks the baseline and converges to a similarly low tail value.Source: Effectiveness of HAM · p. 11, Table 3 · p. 11
  • Naive nonlinear reordering degrades performance, with WER rising to 16.231% on ZH-Hard and 5.105% on ZH-Long; the full framework has the most stable, highest similarity trajectory.The authors attribute the reordering failure to disruption of the absolute temporal order and learned temporal prior; the full model combines DCS and HAM.Source: Failure of Naive Reordering · pp. 11–12, The Full Framework · p. 12, Figure 2 · p. 12

3.6 Spectrogram Analysis

  • In the challenging-utterance example, the baseline loops for 25.5 seconds and reaches 52% WER, whereas HAM-equipped DiTAR+ terminates at 14.0 seconds with 6% WER.The spectrogram comparison associates the baseline loop with copying prior acoustic patterns and describes DiTAR+ as suppressing looping artifacts.Source: §3.6 · pp. 12–13, Figure 3 · p. 13

4 Conclusion

DiTAR+ is a parameter-free framework for decoding instability in continuous-latent AR-DiT speech generation. It combines DCS and HAM, and evaluations report improved robustness across linguistically complex and long-form synthesis.

4 Conclusion

  • DiTAR+ addresses restricted receptive fields and acoustic inertia with DCS, which expands the macro-level receptive field while preserving the absolute timeline, and HAM, which decouples semantic alignment from acoustic reconstruction.The conclusion describes the two mechanisms as complementary.Source: §4 · p. 13
  • Combined DCS and HAM evaluations show substantial improvements over state-of-the-art models, supporting temporal phase coherence and semantic-acoustic decoupling as design principles for robust continuous-latent speech synthesis.The conclusion reports mitigation of semantic hallucinations in linguistically complex sentences.Source: §4 · pp. 13–14
Loading 2609.13909v1…