PaperlayerView exact revision on arXiv

Source-linked AI summary

LeanTTS: Efficient Full-Stack Bilingual TTS for Zero-Shot Cloning and Voice Customization

Yuheng Lu, Qiulin Li, Xi Wang, Yang Wang, Shengpei Jiang, Liyin Liang, Yifeng Qian, Longbiao Wang

arXiv:2610.34408v1cs.SD

TL;DR

Personalized TTS systems often combine flexible speaker adaptation with substantial computational cost, while compact systems commonly support more constrained settings. LeanTTS compresses a bilingual synthesis stack for zero-shot cloning and token-conditioned voice customization. Its 18.9M-parameter system reduces the ZipVoice–Vocos path by 86.12% and reports four-step CPU RTFs of 0.049/0.061.

  • Problem

    Compact TTS often supports fixed-speaker or constrained synthesis, whereas zero-shot systems typically retain larger computational footprints, leaving a gap for efficient personalized TTS.

    Source: §1 · p. 1

  • Method

    LeanTTS compresses the bilingual pronunciation, flow-matching acoustic, and waveform-decoding components for reference-prompt cloning and token-conditioned customization.

    Source: §1 · p. 1

  • Results

    18.9M learned parameters reduce the complete ZipVoice–Vocos path by 86.12%, with four-step mean/P90 CPU RTFs of 0.049/0.061.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results establish a practical sub-20M operating point across speaker coverage, voice stability, model size, and latency.

    Source: §4 · p. 4

    Evaluation is limited to an Apple M5 CPU and eight customized voices.

    Source: §4 · p. 4

Abstract

from arXiv · show

While recent advances in text-to-speech (TTS) synthesis have substantially improved speech quality and modeling flexibility, the development of lightweight and computationally efficient systems remains comparatively underexplored. We present LeanTTS, a bilingual complete synthesis stack that supports two complementary regimes: reference-prompt zero-shot cloning for unseen speakers and token-conditioned customization for recurring voices. LeanTTS combines a compact Mandarin polyphone front end, a structurally compressed flow-matching acoustic model, and a lightweight Vocos decoder with four-step MeanFlow generation. The resulting system contains 18.9M learned parameters, 86.12% fewer than the 136.2M-parameter ZipVoice-Vocos pipeline. On Seed-TTS-Eval, its zero-shot mode obtains 2.02 CER/62.3 SIM on Mandarin and 2.32 WER/48.3 SIM on English, exposing the quality-size trade-off under aggressive compression. With token-conditioned customized voices, the same system reaches 1.57 CER/77.2 SIM on Mandarin and 1.93 WER/76.6 SIM on English, with UTMOS scores of 3.03 and 4.33, respectively. It further achieves mean/P90 CPU RTFs of 0.049/0.061, establishing a sub-20M operating point for bilingual personalization and real-time CPU deployment. Audio demos are available at https://github.com/notlu/leantts.

1. INTRODUCTION

LeanTTS addresses the gap between compact synthesis and personalized TTS by compressing a bilingual pipeline for both unseen-speaker cloning and preset-voice customization. Its 18.9M-parameter configuration reduces the ZipVoice–Vocos path by 86.12%, with reported quality and CPU-speed results.

1. INTRODUCTION

  • Large modern zero-shot TTS systems can require hundreds of millions or billions of parameters, challenging memory-, storage-, and latency-constrained devices.Local synthesis is especially relevant to personalized in-vehicle and embedded applications, where it can reduce network dependency and avoid transmitting reference audio.Source: §1 · p. 1
  • Prior lightweight TTS work explores compact architectures, architecture search, streaming synthesis, and efficient waveform generation.The passage frames this as existing work toward efficient TTS.Source: §1 · p. 1
  • Compact TTS often targets fixed-speaker or constrained synthesis, while zero-shot systems retain larger footprints, leaving a gap for compact personalized edge synthesis.Deployment cost spans pronunciation modeling, acoustic generation, and waveform reconstruction, not only the acoustic model.Source: §1 · p. 1
  • LeanTTS jointly compresses the Mandarin pronunciation front end, flow-matching acoustic generator, and neural waveform decoder for bilingual personalized synthesis.It supports reference-prompt cloning for unseen speakers and token-conditioned customization for recurring voices.Source: §1 · p. 1
  • 18.9M learned parameters reduce the complete ZipVoice–Vocos synthesis path by 86.12% while retaining Mandarin–English zero-shot synthesis.The four-step system also reaches mean/P90 CPU RTFs of 0.049/0.061.Source: §1 · p. 1

2. METHOD

LeanTTS combines compact bilingual front ends, compressed flow-matching acoustic models, and a lightweight waveform decoder. Its method uses candidate-limited Mandarin pronunciation prediction, teacher-guided recovery, and fixed-grid MeanFlow continuation for few-step synthesis.

2.1. Full-stack overview

  • The full stack converts Mandarin and English text into pronunciation tokens, predicts a Mel spectrogram, and uses Vocos to reconstruct a 24-kHz waveform.A leading speaker tag selects a preset voice; a separate few-step checkpoint is obtained through MeanFlow continuation.Source: §2.1 · pp. 1–2

2.2. Compact Mandarin pronunciation front-end

  • Mandarin contextual modeling runs only for ambiguous polyphones, using deterministic G2P candidates and retaining rule-based predictions for other characters.A two-layer encoder processes a target-centered window of at most 42 characters; candidate-slot prediction gathers at most six pronunciations rather than materializing all 659 logits.Source: §2.2 · p. 2

2.3. Structured acoustic compression and recovery

  • The acoustic students reduce depth and channel width from a 122.7M-parameter teacher, preserving stage-wise downsampling while producing 25.1M- and 13.3M-parameter configurations.L2-norm channel selection retains high-scoring channels across coupled projections; recovery combines teacher guidance with ground-truth flow matching, then continues with ground-truth flow matching alone.Source: §2.3 · p. 2

2.4. Fixed-grid rollout distillation

  • Fixed-grid rollout distillation trains the recovered 13M model on four intervals from noise to data, with one average-velocity prediction replacing the teacher trajectory over each interval.Training mixes prompt-preserving clean states with student-rolled states and minimizes masked MSE on valid generated frames while keeping prompt and padding frames fixed.Source: §2.4 · p. 2

2.5. Compact Vocos

  • Compact Vocos retains its eight-block architecture and 100-bin Mel and 24-kHz waveform interfaces while reducing backbone and intermediate widths to 192 and 576.The resulting decoder has 2.1M parameters and is retrained with adversarial, feature-matching, and multi-resolution spectral objectives.Source: §2.5 · p. 3

3. EXPERIMENTS

Experiments compare LeanTTS under matched conditioning settings and assess deployment efficiency, component compression, and teacher-guided recovery.

3.1. Experimental setup

  • Experiments use Emilia bilingual acoustic data, Seed-TTS-Eval for zero-shot synthesis, and separate pronunciation, vocoder, intelligibility, similarity, naturalness, and efficiency metrics.Preset speakers have approximately 300 hours each of Mandarin and English training speech; comparisons use zero-shot and preset/fixed-voice baselines suited to their conditioning settings.Source: §3.1 · p. 3, §3.2 · p. 3

3.2. Overall system comparison

  • 0.0490/0.0610 mean/P90 CPU RTF for LeanTTS-13M contrasts with 1.6975/2.2617 for ZipVoice-distill; the larger reductions chiefly come from acoustic-model compression.Replacing Vocos with Compact Vocos alone reduces ZipVoice-distill RTF to 1.6190/2.1540; the CPU protocol uses one PyTorch thread on an Apple M5.Source: §3.2 · pp. 3–4, Table 2 · p. 4
  • At four NFEs, 18.9M LeanTTS-13M obtains 2.02 CER/62.3 SIM on Mandarin and 2.32 WER/48.3 SIM on English zero-shot, with lower similarity than 136.2M ZipVoice.The reported 86.12% parameter reduction and 0.049 mean CPU RTF define its quality–size–latency operating point.Source: §3.2 · p. 4
  • 1.57 CER/77.2 SIM/3.03 UTMOS on Mandarin and 1.93 WER/76.6 SIM/4.33 UTMOS on English are achieved with token-conditioned customized voices.The same four-step model raises SIM by 14.9 and 28.3 points over reference-prompt zero-shot synthesis, respectively, without requiring a reference recording at inference.Source: §3.2 · p. 4

3.3. Component analysis

  • 98.0631% versus 98.1527% overall accuracy shows candidate-slot G2PW performs similarly to the compact global-head model while scoring at most six valid candidates.Reducing the contextual encoder from 8L/512 to 2L/128 causes a moderate accuracy decrease; the table compares front ends under a common protocol.Source: §3.3 · p. 4, Table 3 · p. 4
  • For FP32 Vocos, reducing 13.5M to 2M parameters lowers UTMOS from 3.7466 to 3.5116, ViSQOL from 4.6584 to 4.5365, and PESQ from 3.6675 to 3.3723.INT8 quantization further lowers reconstruction scores while substantially reducing the 2M model's deployment footprint; Table 4 evaluates reconstruction on LibriTTS test-clean.Source: §3.3 · p. 4, Table 4 · p. 4

3.4. Impact of Teacher-Guided Recovery

  • At matched architecture, data, and inference steps, teacher-guided recovery yields stronger later checkpoints, with simultaneous error-rate and SIM gains in both languages at 300k and 500k updates.The early recovery stage is comparable to ground-truth FM continuation; the reported difference emerges with subsequent training.Source: §3.4 · p. 4, Table 5 · p. 4

4. CONCLUSION

LeanTTS combines reference-prompt zero-shot cloning and token-conditioned customization in a sub-20M bilingual TTS stack. Its four-step system offers a practical operating point across speaker coverage, voice stability, model size, and latency, though evaluation remains limited in hardware and voice-set scope.

4. CONCLUSION

  • At four NFEs, zero-shot obtains 2.02 CER/62.3 SIM on Mandarin and 2.32 WER/48.3 SIM on English; customization obtains 1.57 Mandarin CER/77.2 SIM and 1.93 English WER/76.6 SIM.Customization uses recurring voices available during training; zero-shot retains cloning for unseen speakers.Source: §4 · p. 4
  • The 18.9M-parameter system reaches mean/P90 CPU RTFs of 0.049/0.061, while an MNN-based Android-emulator deployment achieves 0.0327 end-to-end RTF.Validation on physical mobile SoCs remains future work.Source: §4 · p. 4
  • Current evaluation is limited to an Apple M5 CPU and eight customized voices; broader multilingual voice sets remain future work.Source: §4 · p. 4
Loading 2610.34408v1…