PaperlayerView exact revision on arXiv

Source-linked AI summary

dots.tts Technical Report

Shi Lian, Changtao Li, Bohan Li, Hankun Wang, Da Zheng, Junfeng Tian, Yufeng Ma, Colin Zhang, Kai Yu

arXiv:2606.07080v2cs.SDcs.AIeess.AS

TL;DR

Modern TTS must extend beyond intelligible read speech toward expressive, controllable, real-time, and broad audio coverage. dots.tts addresses this with a fully continuous autoregressive architecture and stability- and efficiency-oriented training, achieving leading benchmark performance and low-latency generation.

  • Problem

    Modern TTS systems need expressive, controllable, real-time output covering diverse speech and audio settings beyond standard read-speech intelligibility.

  • Method

    dots.tts combines a semantic, prediction-friendly AudioVAE with semantic planning, full-context autoregressive flow-matching acoustic rendering, self-corrective alignment, and CFG-aware MeanFlow distillation.

  • Results

    dots.tts achieves state-of-the-art average performance across reported benchmarks, including average WER 2.92 and SIM 79.2 on Seed-TTS-Eval, while reaching 54 ms TTFB at RTF 0.245 on a single H800.

  • Takeaways & Limitations

    The released system supports real-time and conversational use cases, with training and inference code plus multiple checkpoints available under the Apache 2.0 license.

  • Takeaways & Limitations

    Raw BPE text conditioning increases data demands and leaves a low-resource WER gap for several script-divergent or under-represented languages and difficult pronunciation scenarios.

Abstract

from arXiv · show

We present dots$.$tts, a 2B-parameter continuous autoregressive text-to-speech (TTS) foundation model that models speech in a continuous latent space. Compared with existing continuous autoregressive models, our key innovations are threefold. First, we train an AudioVAE with multiple objectives to build a semantically structured and prediction-friendly continuous speech space. Second, we use full-history conditioning in the flow-matching head to preserve long-range consistency and reduce drift during generation. Third, we apply reward-free self-corrective post-training to the flow-matching head to further improve robustness and acoustic quality. After being trained on a large-scale multilingual corpus, dots$.$tts achieves the best average performance on Seed-TTS-Eval, with WERs of 0.94%/1.30%/6.60% and SIM scores of 81.0/77.1/79.5 on the zh/en/zh-hard test sets, respectively. Across other benchmarks, dots$.$tts also consistently demonstrates open-source state-of-the-art performance, exhibiting strong generation stability, voice cloning ability, and emotional expressiveness. For efficient inference, we further apply CFG-aware MeanFlow distillation, enabling low-latency speech generation with first-packet latencies of 85/54 ms in output streaming and dual-streaming modes, respectively. To facilitate reproducible research and practical deployment, we release the training and inference code, together with the pretrained, post-trained, and MeanFlow-distilled checkpoints, under the Apache 2.0 license.

1 Introduction

dots.tts targets the remaining gap between discrete-token and continuous autoregressive TTS: expressive, controllable, real-time speech without long-range drift. It combines a structured AudioVAE, decomposed semantic/acoustic generation, self-correction, and efficient streaming, achieving strong benchmark and latency results.

  • Motivation: Discrete-token systems benefit from mature language-model tooling but struggle to represent speech, emotional paralinguistics, singing, and ambient sound within one distribution.
  • Motivation: Continuous autoregressive TTS remains limited by long-range error accumulation because prediction errors are reconstructed and fed back without quantization.This contrasts with discrete codecs, which can snap imperfect samples back to valid acoustic configurations.
  • Contributions: dots.tts combines a semantic AudioVAE, semantic-planning and acoustic-rendering modules, and reward-free self-correction for continuous autoregressive generation.The AudioVAE is trained with downstream objectives and WavLM representation alignment; the flow-matching head is exposed to its own off-trajectory errors.
  • Results: Trained on 1.5M hours of speech, dots.tts achieves state-of-the-art stability and voice-cloning quality, leads multilingual speaker similarity, and remains competitive across broader benchmarks.
  • Contributions: dots.tts is a 2B-parameter fully continuous end-to-end autoregressive TTS system with state-of-the-art stability and zero-shot voice-cloning quality on Seed-TTS-Eval.
  • Contributions: 54 ms at RTF 0.245 in interleaved streaming mode and 85 ms at RTF 0.231 in plain mode provide low first-packet latency for real-time deployment.CFG-aware MeanFlow distillation reduces the flow-matching ODE to as few as 2 to 4 function evaluations.

2.1 Overview

dots.tts separates continuous speech representation from autoregressive prediction. Its backbone uses a semantic encoder, an LLM for semantic generation, and an autoregressive flow-matching head for acoustic rendering.

  • Architecture: The system consists of a frozen AudioVAE and an autoregressive backbone that predicts continuous latent representations one patch at a time.The AudioVAE encodes 48 kHz mono speech into 128-dimensional latents at 25 Hz and decodes them with a BigVGAN-style decoder.
  • Architecture: The backbone comprises a semantic encoder, an LLM, and an autoregressive flow-matching head that divide semantic generation from acoustic generation.The LLM consumes BPE text and 6.25 Hz audio-semantic embeddings, while the flow-matching head predicts the next acoustic latent patch.
  • Architecture: The LLM can use text-prefix sequences for standard TTS or interleaved text-audio sequences for low-latency streaming.

2.2 AudioVAE

The AudioVAE is designed to preserve high-fidelity speech while producing a continuous latent space that downstream autoregressive modeling can learn effectively. It combines causal architecture, reconstruction objectives, semantic alignment, and multitask supervision.

  • Training: The first training stage combines adversarial, mel-spectral reconstruction, feature-matching, KL, and flow-regularization losses to optimize reconstruction quality.
  • Training: The second stage adds frozen-WavLM frame alignment and ASR, emotion, and speaker classification objectives to make the latent space semantically structured and learnable.The downstream LLM is discarded, while its encoder is retained as the backbone’s semantic frontend.
  • Results: The second-stage objectives improve downstream diffusion learnability while preserving reconstruction quality.The full reconstruction comparison is reported in Table 1.

2.3 LLM backbone

The LLM backbone directly consumes BPE text and compact audio-semantic history, then conditions continuous acoustic prediction. It supports both conventional text-prefix synthesis and causal 1T1A streaming for conversational use.

  • Inputs and prediction: The backbone uses BPE text directly and processes audio at 6.25 Hz, with each step producing a four-frame VAE latent patch.The semantic encoder compresses each 25 Hz VAE latent patch into one audio-semantic embedding for the next LLM step.
  • Sequence layouts: Plain mode places the full text before the audio span, while 1T1A mode interleaves one text token with one audio step until text ends.The per-step interface between the LLM, semantic encoder, and flow-matching head remains identical across layouts.
  • Design trade-offs: Starting from a text LLM improves prosody, text normalization, and natural-language style prompting but requires substantially more speech-text data than phoneme-based input.
  • Sequence layouts: 1T1A streaming lets speech begin within a single text token of generation and continue incrementally without buffering a full utterance.
  • Sequence layouts: Plain mode is preferred when the complete text is available, whereas 1T1A mode is intended for real-time dialogue systems.

2.4 Semantic encoder

The semantic encoder converts generated VAE-latent patches into compact, causal embeddings aligned with the LLM’s text-semantic space. This lets the LLM condition on summarized history while omitting high-variance acoustic detail.

  • Semantic representation: The semantic encoder converts each 25 Hz VAE-latent patch into a single 6.25 Hz embedding for the LLM.It strips out high-variance acoustic detail during conversion.
  • Architecture: A strided causal projector and Transformer produce 4× temporal downsampling from VAE latent frames to LLM tokens.Strict causality permits one-patch-at-a-time unrolling during streaming inference.
  • Role in generation: The encoder’s pretrained semantic features align latent patches with the LLM’s text-semantic space.The LLM can therefore ignore acoustic detail and condition on a compact history summary during autoregressive rollout.

2.5 Autoregressive flow-matching head

The autoregressive flow-matching head predicts continuous acoustic patches using semantic hidden states, prior clean patches, and speaker conditioning. Block-causal training reproduces per-step inference context while enabling parallel prediction across patches.

  • Per-step latent generation: The DiT velocity predictor uses rectified-flow dynamics to transform Gaussian noise into clean latent patches with a few Euler ODE steps.The head is conditioned by the diffusion timestep and a speaker-embedding side input.
  • Per-step latent generation: The head conditions each step on the current LLM state, all earlier clean patches, and the current noisy patch.These streams are projected into a common hidden space before being interleaved.
  • Per-step latent generation: A frozen CAM++ speaker x-vector supplies global conditioning, while LLM and speaker streams are independently dropped with probability 0.5.The sequence interleaves one hidden-state token per audio step with four-token patch blocks.
  • Block-causal training attention: Parallel training across all patches reproduces the exact autoregressive context available at inference through a block-causal mask.The cause and generation halves share per-block positional layouts, with generation positions reset to match inference RoPE phases.
  • Block-causal training attention: The AR-FM prefix already contains every LLM hidden state, making the head a complete text-conditioned speech generator without additional acoustic feedback.The authors observe that this pushes the LLM toward semantic rather than acoustic information.

2.6 Training objectives

Training combines a semantically structured AudioVAE objective, backbone flow-matching and stop losses, reward-free self-corrective alignment, and CFG-aware MeanFlow distillation. The post-training stages target inference-time robustness and efficient single-pass generation.

  • AudioVAE objectives: The AudioVAE is pretrained and frozen before backbone training, using reconstruction, adversarial, feature-matching, KL, flow-prior, WavLM, and downstream supervision objectives.Downstream supervision covers ASR, emotion, and speaker heads.
  • Backbone pretraining: Backbone pretraining uses per-patch flow-matching regression and a balanced EOS stop-prediction loss.The flow-matching loss backpropagates through the LLM and semantic encoder, while the detached stop head avoids interfering with rollout dynamics.
  • Backbone pretraining: The flow-matching setup interpolates Gaussian noise and a clean VAE patch while conditioning on acoustic, speaker, and semantic context.Classifier-free guidance combines conditioned and dropped-conditioning velocity predictions using a guidance scale.
  • Post-training objectives: Self-corrective alignment performs a detached Euler rollout so the acoustic DiT trains on states reflecting its own inference-time errors.The off-trajectory state is re-noised and used in an auxiliary loss alongside ordinary on-trajectory flow matching.
  • Post-training objectives: This reward-free self-correction is applied directly to the acoustic DiT rather than the full TTS stack.The speaker encoder, semantic encoder, and LLM provide fixed conditioning during this stage.
  • Post-training objectives: CFG-aware MeanFlow distillation trains a student to match a frozen teacher’s CFG-guided mean velocity over sampled time intervals.The student uses one conditional forward pass at inference instead of separate conditional and unconditional evaluations.

3 Experiments

The experiments evaluate dots.tts across reconstruction, zero-shot cloning, multilinguality, cross-lingual cloning, and expressiveness, using large-scale multilingual training and multiple post-training variants. Results show strong overall benchmark performance, while low-resource languages and rare lexical items remain clear weaknesses.

  • Experimental setup: 1.5M hours of multilingual audio support evaluation across foundational quality, multilingual coverage, cross-lingual cloning, and expressiveness.The training mixture includes in-house, open-source, and caption-paired audio sources.
  • AudioVAE reconstruction: WER 4.14 and SIM 0.969 indicate that the AudioVAE latent adds almost nothing to end-to-end error.The AudioVAE’s reconstruction metrics are broadly competitive with continuous representations, while discrete tokenizers trail them on several metrics.
  • Seed-TTS-Eval: dots.tts achieves the best average on both WER and SIM in Seed-TTS-Eval, with WER 2.92% below every reported baseline.SOAR and MF NFE = 4 also remain below all reported baselines on WER; evaluation uses unseen approximately 3-second reference prompts.
  • Multilingual evaluation: 83.9 average SIM leads the MiniMax multilingual benchmark by 1.6 points, with a variant leading or tying on 21 of 24 languages.The WER picture is mixed because a few low-resource outliers raise the average; the authors associate this gap with insufficient BPE token coverage.
  • Cross-lingual cloning: 4.37% hard-en WER leads CV3-Eval, while SOAR leads cross-lingual SIM at 75.0 en→zh and 72.8 zh→en but trails CosyVoice 3 on cross-lingual WER.The largest observed SOAR-stage gain is on hard-en, improving WER from 5.99 to 4.49 before MF4 inherits the gain.
  • Expressiveness: 65.7% on Syntactic Complexity is the top score across open- and closed-source systems, but SOAR reduces expressiveness on Emotions and Paralinguistics.Pretrain leads open-source Emotions at 72.7%, while Complex Pronunciation and Foreign Words are the weakest scenarios at 16–18% and 36–40%.

3.4 Efficiency

dots.tts is optimized for real-time generation through causal latent-patch processing, interleaved text-audio inference, and MeanFlow distillation. These choices reduce first-packet latency while maintaining real-time generation on a single H800 GPU.

  • Efficiency measurements use vllm-omni with continuous batching, paged-KV attention, and compiled AR-FM and semantic-encoder components.The reported measurements are obtained on a single NVIDIA H800 GPU.
  • 85.4 ms plain-mode and 54.4 ms interleaved first-packet latency are achieved with RTFs of 0.231 and 0.245, respectively.Interleaving consumes the upstream LLM token stream as it is decoded, allowing audio generation to start earlier.

4 Conclusion

The conclusion presents dots.tts as a continuous end-to-end autoregressive TTS system addressing long-range error accumulation and post-training limitations. It reports leading benchmark results and real-time deployment performance from the same released backbone.

  • dots.tts targets long-range error accumulation and an immature post-training stack in continuous autoregressive TTS.Its architecture uses a semantic encoder, LLM, and autoregressive flow-matching head.
  • 54 ms TTFB at RTF 0.245 on a single H800 supports real-time and conversational use cases.This result combines CFG-aware MeanFlow distillation with 1-text-1-audio interleaved streaming.
  • 2.92 average WER and 79.2 average SIM on Seed-TTS-Eval are reported alongside 83.9 average speaker similarity on the 24-language MiniMax benchmark.The system also reports leading hard-subset and cross-lingual results on CV3-Eval.
  • 65.7% on Syntactic Complexity is the top table score, while 72.7% on Emotions is the strongest open-source result on EmergentTTS-Eval.The benchmarks cover stability, speaker similarity, naturalness, prosody, paralinguistics, and multilingual coverage.
  • The authors release training and inference code and pretrained, self-corrective-aligned, and MeanFlow-distilled checkpoints under Apache 2.0.The release is intended as a reproducible reference stack for continuous-AR TTS.

5 Limitations

The released system has multilingual coverage gaps, limited control and modality scope, and deployment risks associated with high-fidelity voice cloning. The authors identify data, phoneme-side, instruction-tuning, and policy-oriented directions for addressing these boundaries.

  • Raw BPE input drives a low-resource WER gap for Arabic, Hindi, Turkish, and Vietnamese because these languages are script-divergent or under-represented.The same coverage limit affects loanwords, technical terms, and proper names in Foreign Words and Complex Pronunciation scenarios.
  • The system is evaluated only under canonical zero-shot conditions, without explicit style or instruction control.An instruction-tuned variant using caption-paired data is proposed as a next step.
  • Singing and unified speech-and-sound generation are not covered because the backbone uses a speech-heavy training mixture.The AudioVAE is described as modality-agnostic in principle, but the released backbone is not trained for these modalities.
  • High-fidelity zero-shot voice cloning carries misuse risks, motivating consent-aware reference-audio policies, synthetic-speech detection, and content watermarking.The released checkpoints are intended for research and authorized deployment.
Loading 2606.07080v2…