PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Neural Speech Synthesis

Xu Tan, Tao Qin, Frank Soong, Tie-Yan Liu

arXiv:2106.15561v3eess.AScs.CLcs.LGcs.MMcs.SD

TL;DR

Neural TTS has advanced rapidly, but its diverse and quickly evolving research landscape still needs a comprehensive synthesis. This survey organizes core components and advanced topics, summarizes resources, and identifies future directions, while noting scope boundaries in its treatment of robustness.

  • Problem

    Neural TTS research is diverse and evolves quickly, creating a need for a comprehensive survey of its basics and recent developments.

  • Method

    The paper surveys neural TTS components, advanced topics, related resources, and future research directions.

  • Results

    The survey synthesizes research on text analysis, acoustic models, vocoders, fast, low-resource, robust, expressive, and adaptive TTS.

  • Takeaways & Limitations

    The survey provides a consolidated reference for academic researchers and industry practitioners working on TTS.

  • Takeaways & Limitations

    The survey mainly reviews robust issues in acoustic models because vocoder issues are less severe and their causes are unclear.

Abstract

from arXiv · show

Text to speech (TTS), or speech synthesis, which aims to synthesize intelligible and natural speech given text, is a hot research topic in speech, language, and machine learning communities and has broad applications in the industry. As the development of deep learning and artificial intelligence, neural network-based TTS has significantly improved the quality of synthesized speech in recent years. In this paper, we conduct a comprehensive survey on neural TTS, aiming to provide a good understanding of current research and future trends. We focus on the key components in neural TTS, including text analysis, acoustic models and vocoders, and several advanced topics, including fast TTS, low-resource TTS, robust TTS, expressive TTS, and adaptive TTS, etc. We further summarize resources related to TTS (e.g., datasets, opensource implementations) and discuss future research directions. This survey can serve both academic researchers and industry practitioners working on TTS.

1 Introduction

Neural TTS has improved substantially with deep learning, motivating a comprehensive survey of its foundations, developments, resources, and future directions.

  • TTS synthesizes intelligible, natural speech from text and draws on linguistics, acoustics, signal processing, and machine learning.
  • Neural TTS quality has improved substantially, but the field’s diverse and rapidly evolving topics still require a comprehensive survey.
  • 1.1 History of TTS Technology: Traditional TTS progressed from articulatory and formant synthesis to concatenative and statistical parametric synthesis.
  • 1.1 History of TTS Technology: Neural TTS introduced neural networks into speech synthesis, including models that generate acoustic features or waveforms directly.
  • 1.2 Organization of This Survey: The survey covers key TTS components, advanced topics, related resources, and future research directions.

2 Key Components in TTS

The survey organizes neural TTS research around its core components and representative model families, including acoustic models, vocoders, and fully end-to-end systems.

  • The survey reviews text analysis, acoustic models, vocoders, and fully end-to-end models as the principal areas of neural TTS.
  • Representative text-analysis and acoustic-model works span normalization, grapheme-to-phoneme conversion, prosody prediction, autoregressive, and non-autoregressive systems.
  • Representative vocoder research includes autoregressive, waveform-generating, flow-based, GAN-based, and diffusion-based models.
  • Fully end-to-end systems directly connect text or phonemes to speech generation and include models such as Char2Wav, FastSpeech 2s, and VITS.

2.1 Main Taxonomy

Neural TTS can be classified by its components and by the data representations flowing from text to waveform.

  • The main taxonomy groups neural TTS into text analysis, acoustic models, vocoders, and fully end-to-end models.
  • The data flow progresses from characters to linguistic features, acoustic features, and waveform, with fully end-to-end models bypassing intermediate representations.

2.2 Text Analysis

Text analysis transforms written text into linguistic information that supports pronunciation and prosody, using tasks such as normalization, segmentation, tagging, and phoneme conversion.

  • Text analysis converts input text into linguistic features containing pronunciation and prosody information for speech synthesis.
  • Typical frontend tasks include text normalization, word segmentation, part-of-speech tagging, prosody prediction, and grapheme-to-phoneme conversion.
  • Text normalization converts non-standard written forms into spoken forms, while neural models can frame this conversion as sequence-to-sequence prediction.
  • Prosody prediction models rhythm, stress, and intonation through variations in duration, loudness, and pitch, using encoders or reference representations.
  • Linguistic features aggregate text-analysis results across phoneme, syllable, word, phrase, and sentence levels for downstream TTS components.

2.3 Acoustic Models

Acoustic models map linguistic features, phonemes, or characters to acoustic features, with architectures evolving from SPSS models to neural sequence-to-sequence and non-autoregressive designs.

  • Acoustic models generate acoustic features from linguistic features or directly from phonemes or characters for subsequent waveform synthesis.These features include representations such as MCC, MGC, BAP, F0, V/UV, and mel-spectrograms.
  • Acoustic Models in SPSS: SPSS models use HMM, DNN, or RNN predictors to map linguistic features to speech parameters before vocoder synthesis.Their development targets richer context modeling, output-frame correlations, and improved acoustic quality.
  • Acoustic Models in SPSS: HMM-based SPSS produces over-smoothed features and limited detail, while DNN-based models improve synthesized quality and LSTM models better model long-span context.The limitations are attributed to HMM modeling capacity and vocoding quality.
  • RNN-based Models: Tacotron introduced encoder-attention-decoder acoustic modeling from characters to spectrograms, while later systems improved expressiveness, duration modeling, and parallel generation.Tacotron 2 predicts mel-spectrograms and uses WaveNet for waveform synthesis; later work includes style tokens, duration predictors, and non-autoregressive models.
  • The survey’s taxonomy separates acoustic models by network structure while noting that acoustic and vocoder taxonomies can also be organized by generative model.The cited table defines input, output, and NAR* characteristics used to describe acoustic models.
  • Transformer-based Models: TransformerTTS addresses recurrent models’ lack of parallelism and difficulty modeling long dependencies, while FastSpeech-style models target faster and more robust generation.Autoregressive models can be slow and may produce word skipping or repetition; FastSpeech uses explicit duration prediction for non-autoregressive expansion.

2.4 Vocoders

Neural vocoders convert linguistic or acoustic features into waveforms, progressing from autoregressive models toward flow, GAN, VAE, and diffusion approaches with different trade-offs.

  • Early neural vocoders directly generated waveform from linguistic features, while later systems commonly conditioned waveform generation on mel-spectrograms.Waveform generation is needed after acoustic modeling in many neural TTS pipelines.
  • Autoregressive Vocoders: WaveNet generates waveform points autoregressively and achieves good voice quality but suffers from slow inference speed.Subsequent work investigated lightweight and fast vocoders, including SampleRNN and WaveRNN.
  • Flow-based Vocoders: Flow-based vocoders use invertible mappings between probability distributions and can provide high voice quality with fast inference.Autoregressive transforms trade greater expressiveness for complicated teacher-distillation training, whereas bipartite transforms simplify training but often require more parameters.
  • GAN-based Vocoders: GAN-based vocoders combine generators and discriminators, using dilated convolutions, upsampling, multi-scale discrimination, and auxiliary losses.STFT and feature-matching losses can improve adversarial-training stability, efficiency, and perceptual audio quality.
  • Diffusion-based Vocoders: Diffusion vocoders formulate data-to-latent mapping through gradual noising and reverse denoising processes.Representative systems include DiffWave, WaveGrad, and PriorGrad.
  • Discussions: AR models are mathematically simpler, while non-AR generative models generally support parallel generation and latent manipulation; GANs cannot estimate data likelihood.The survey compares these properties across AR, VAE, flow, diffusion, and GAN vocoders.

2.5 Towards Fully End-to-End TTS

Neural TTS has progressed toward fully end-to-end text-to-waveform synthesis, motivated by simpler optimization and lower feature-development costs but constrained by modality and sequence-length mismatches.

  • Fully end-to-end TTS directly generates speech waveform from character or phoneme sequences, reducing annotation, feature-development, and deployment costs.Joint optimization can avoid error propagation between cascaded text-analysis, acoustic-model, and vocoder stages.
  • End-to-end training is difficult because text and waveform are different modalities and their sequence lengths differ substantially.A five-second utterance may contain about 100 phonemes but 80k waveform samples at 16kHz.
  • The progression toward end-to-end TTS moves from three-module SPSS pipelines to integrated acoustic modeling, waveform generation, simplified spectrogram prediction, and direct text-to-waveform systems.The stages correspond to SPSS, WaveNet, Tacotron-style models, and fully end-to-end architectures.

2.6 Other Taxonomies

Beyond component and data-flow organization, neural TTS research is classified by generation mode, generative model, network structure, and publication timeline.

  • Neural TTS works can be categorized as autoregressive or non-autoregressive according to their sequence-generation mode.
  • A generative-model taxonomy distinguishes normal sequence-generation, flow, GAN, VAE, and diffusion models.
  • A network-structure taxonomy distinguishes CNN, RNN, self-attention, and hybrid architectures.Hybrid structures combine more than one type, such as CNN+RNN or CNN+self-attention.
  • The evolution figure organizes representative research works by the time they became publicly available rather than by later formal publication date.The survey uses early public availability to recognize early knowledge sharing.
  • The evolution overview includes representative systems such as WaveNet, Tacotron, FastSpeech, WaveGlow, MelGAN, HiFi-GAN, and VITS.The survey notes that many additional works appear in Table 18 because the research literature is abundant.

3 Advanced Topics in TTS

The survey organizes advanced neural TTS research around practical challenges including speed, resource efficiency, robustness, and deployment constraints. Fast synthesis is addressed through parallel generation, lightweight architectures, and speech-domain knowledge.

  • Advanced Topics in TTS: Advanced neural TTS topics target practical product challenges, including fast synthesis, low-resource training, robustness, naturalness, and voice adaptation.The survey presents these topics as extensions beyond the basic TTS components.
  • Fast TTS: Long autoregressive speech sequences make neural TTS slow because one second can contain about 500 mel-spectrograms and 24,000 waveform points.The cited example assumes a 10 ms hop size and a 24 kHz sampling rate.
  • Fast TTS: Fast TTS methods use parallel spectrogram or waveform generation, lightweight model structures, and speech-domain knowledge.The survey identifies these as three broad techniques for accelerating synthesis.
  • Parallel Generation: RNN autoregressive models require O(N) computation, whereas FastSpeech uses feed-forward self-attention for parallel training and inference with O(1) computation.Most GAN-based models also provide O(1) computation in training and inference, while some flow models require teacher distillation or iterative steps.
  • Lightweight Model: Non-autoregressive generation improves inference parallelism but does not necessarily reduce parameter count or total computation, motivating lightweight models for mobile and embedded deployment.Limited parallel-computation capability on such devices makes computational efficiency important even when generation is parallel.
  • Speedup with Domain Knowledge: Speech-domain techniques such as linear prediction, multiband modeling, subscale prediction, multi-frame prediction, and streaming synthesis accelerate inference.LPCNet combines linear prediction coefficients with a lightweight neural residual model for autoregressive waveform generation.

3.3 Low-Resource TTS

Low-resource TTS addresses the scarcity of paired text–speech data across the world’s languages. The survey groups solutions around self-supervision, cross-speaker transfer, speech–text duality, and mining noisy web data.

  • Motivation: Most of the world’s more than 7,000 languages lack sufficient training data for high-quality TTS, while commercial services support only dozens of languages.The survey frames low-resource TTS as both a business concern and a social-good opportunity.
  • Self-supervised training: Self-supervised pretraining uses unpaired text or speech to enhance TTS language understanding or speech generation.Examples include BERT-enhanced text encoders and pretrained speech decoders.
  • Cross-lingual transfer: Cross-lingual transfer supports low-resource languages through shared phoneme or byte representations and language-similarity information.The survey mentions IPA and byte representations for arbitrary texts in multiple languages.
  • Cross-speaker transfer: Cross-speaker transfer improves a data-limited target speaker by converting other voices into the target voice or adapting pretrained voices.The surveyed approaches include voice conversion, voice adaptation, and voice cloning.
  • Speech chain/Back transformation: Speech chain and back transformation exploit the duality of TTS and ASR to use additional unpaired text and speech data.These methods aim to improve both TTS and ASR together.
  • Dataset mining in the wild: Dataset mining in the wild collects low-quality paired web data and applies enhancement, denoising, or disentangling to improve its usefulness for TTS.The surveyed methods are designed for training from noisy web speech resources.

3.4 Robust TTS

Robust TTS seeks to prevent word skipping, repetition, and attention collapse in acoustic models. The survey categorizes remedies by alignment learning, autoregressive error propagation, explicit duration prediction, and attention design.

  • Robustness Problems: Neural acoustic models can produce word skipping, repetition, and attention collapse because text–speech alignment is difficult and autoregressive errors propagate.Attention collapse produces unintelligible gibberish when attention fails to focus on a single input token.
  • Alignment Learning: Robustness methods enhance attention or remove it by explicitly predicting character or phoneme durations.Duration expansion matches the text representation to the mel-spectrogram sequence before generation.
  • Autoregressive Generation: Autoregressive approaches mitigate exposure bias and error propagation, while non-autoregressive generation removes autoregressive generation altogether.The survey treats these as two complementary solution categories.
  • Scope: The survey mainly addresses acoustic-model robustness because vocoder glitches are less severe, have unclear causes, and may be repaired by universal or specialized vocoders.Robustness can also fail when test domains are insufficiently covered by training domains.
  • Attention Design: Attention mechanisms for TTS should satisfy local, monotonic, and complete alignment properties between text tokens and mel-spectrogram frames.The survey reviews techniques according to whether they provide these properties.
  • Attention Design: Location-based attention uses positional information, while hybrid attention incorporates previous alignments to stabilize monotonic alignment.The survey reports that properly handled location-based attention can ensure monotonicity.
  • Attention Design: Monotonic attention can avoid skipping and repetition, but basic forms do not guarantee complete alignment.Stepwise monotonic attention is introduced to address this limitation.
  • Attention Design: Windowing and off-diagonal penalties restrict attention toward diagonal alignments, reducing learning flexibility and difficulty.These methods exploit the expected monotonic structure of text–speech alignment.

3.5 Expressive TTS

Expressive TTS models variation in content, speaker timbre, prosody, style, emotion, and recording conditions to synthesize more natural speech. The survey reviews explicit and implicit modeling across multiple granularities, alongside disentangling, control, and transfer.

  • Motivation: Expressive TTS addresses one-to-many text-to-speech mappings in duration, pitch, volume, speaker style, and emotion.Using insufficient input information with regular L1 loss can produce over-smoothed mel-spectrogram predictions.
  • Variation Information: The survey organizes expressive information into text content, speaker or timbre, prosody/style/emotion, and recording-device or noise conditions.These correspond respectively to what to say, who says it, how it is said, and the transmission channel.
  • Information Type: Expressive TTS research models variation information through explicit labels or implicit representations learned from data.Explicit labeling can require substantial human effort and may miss fine-grained variation.
  • Implicit Modeling: Reference encoders extract prosody embeddings from reference audio without requiring explicit annotations.Training uses ground-truth reference audio, while inference can use another reference to synthesize similar prosody.
  • Information Granularity: Variation can be modeled from language and speaker levels through paragraph, utterance, word, syllable, phoneme, and frame granularities.Hierarchical structures can combine information across multiple time resolutions.
  • Disentangling, Control, and Transfer: The survey covers disentangling, controlling, and transferring variation information in expressive speech synthesis.Representative approaches include adversarial training, cycle-consistency or feedback losses, semi-supervised learning, and changing variation representations.

3.6 Adaptive TTS

Adaptive TTS synthesizes voices for new users and is studied through generalization and efficient adaptation. The field balances voice quality against the costs of adaptation data and parameters.

  • Adaptive TTS synthesizes a voice for a target user and is also called voice adaptation, voice cloning, or custom voice.
  • The survey separates adaptive TTS into general adaptation and efficient adaptation settings.General adaptation improves source-model generalization to new speakers or domains; efficient adaptation reduces adaptation data and parameters.
  • General adaptation: Source TTS models can generalize poorly because source text lacks acoustic information about prosody, speaker timbre, and recording environments.
  • General adaptation: Cross-domain adaptation addresses differing acoustic conditions or styles between adaptation speech and source training data.Examples include recording devices, environmental noise, accents, and speaker rates.
  • Efficient adaptation: More adaptation data and fine-tuned parameters generally improve voice quality but increase data-collection, memory, and deployment costs.Adaptation may fine-tune the whole model, part of it, or only speaker embeddings.
  • Efficient adaptation: Few-shot adaptation uses paired text and speech ranging from several minutes to several seconds.

4 Resources

The survey gathers TTS resources spanning software, tutorials, challenges, and corpora. These resources are organized in a dedicated resource summary and accompanying table.

  • The survey collects open-source implementations, TTS tutorials and keynotes, TTS challenges, and TTS corpora.
  • Table 17 presents the collected TTS resources.
  • The listed resources include audio software, historical TTS tutorials, and generative-model speech materials.
  • The resource list includes the Blizzard, Zero Resource Speech, and Voice Conversion challenges.
  • The corpus table is organized by hours, speakers, sampling rate, and language.

5 Future Directions

The survey’s future directions prioritize higher-quality and more efficient speech synthesis. It highlights stronger generative and learned representations, robust and data-efficient systems, and lower computational cost.

  • The survey groups future neural TTS directions according to the end goals of speech synthesis.
  • High-quality speech synthesis: High-quality synthesis remains a major goal because speech quality involves intelligibility, naturalness, expressiveness, prosody, emotion, style, robustness, and controllability.
  • High-quality speech synthesis: More powerful and efficient generative models are proposed to further improve waveform and acoustic-feature generation.The survey identifies VAE, GAN, flow, and diffusion models as existing approaches in TTS components.
  • High-quality speech synthesis: Better text and speech representations are expected to support neural TTS quality, while unsupervised or self-supervised speech representation learning remains challenging.
  • Efficient speech synthesis: Efficient synthesis targets the costs of collecting data, labeling data, training models, and serving models.
  • Efficient speech synthesis: Data-efficient TTS is especially relevant for low-resource languages, with unsupervised, semi-supervised, and cross-lingual learning identified as directions.
  • Efficient speech synthesis: Reducing FLOPs in TTS training and inference could broaden access to advanced TTS while reducing carbon emissions.
Loading 2106.15561v3…