PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Neural Speech Synthesis

Xu Tan, Tao Qin, Frank Soong, Tie-Yan Liu

arXiv:2106.15561v3eess.AScs.CLcs.LGcs.MMcs.SD

TL;DR

Neural TTS research spans diverse, rapidly evolving topics, making a comprehensive overview useful. This survey organizes core components and advanced topics, summarizes resources, and identifies future directions; it focuses mainly on voice adaptation rather than adaptation to languages, styles, or domains.

  • Problem

    Because neural TTS topics are diverse and evolving quickly, a comprehensive survey of its basics and recent developments remains necessary.

    Source: §1 · p. 1

  • Method

    The survey reviews text analysis, acoustic models, vocoders, end-to-end models, advanced TTS topics, and related resources.

    Source: §2 · p. 4, §1.2 · p. 4, §5 · p. 33

  • Results

    The survey lists representative TTS algorithms and organizes future research directions into two categories according to TTS end goals.

    Source: §5 · pp. 33–34

  • Takeaways & Limitations

    The overview can help readers identify related TTS problems and applications beyond the core algorithms reviewed.

    Source: §5 · p. 33

    The adaptive TTS discussion focuses on different voices rather than languages, styles, or domains.

    Source: §3.6 · p. 30

Abstract

from arXiv · show

Text to speech (TTS), or speech synthesis, which aims to synthesize intelligible and natural speech given text, is a hot research topic in speech, language, and machine learning communities and has broad applications in the industry. As the development of deep learning and artificial intelligence, neural network-based TTS has significantly improved the quality of synthesized speech in recent years. In this paper, we conduct a comprehensive survey on neural TTS, aiming to provide a good understanding of current research and future trends. We focus on the key components in neural TTS, including text analysis, acoustic models and vocoders, and several advanced topics, including fast TTS, low-resource TTS, robust TTS, expressive TTS, and adaptive TTS, etc. We further summarize resources related to TTS (e.g., datasets, opensource implementations) and discuss future research directions. This survey can serve both academic researchers and industry practitioners working on TTS.

1 Introduction

TTS aims to synthesize intelligible, natural speech from text [346] and draws on multiple disciplines; as neural approaches have expanded and improved speech quality, this paper surveys their foundations and developments.

1 Introduction

  • Deep learning has driven rapid growth in neural TTS and improved synthesized-speech quality, motivating a comprehensive survey of this diverse, fast-evolving field.The survey aims to help readers understand current research and unsolved problems.Source: §1 · p. 1

1.1 History of TTS Technology

  • TTS progressed from articulatory, formant, and concatenative synthesis to SPSS, then neural systems that improve voice quality and increasingly map text directly to speech [254].SPSS predicts acoustic parameters from linguistic features and uses a vocoder; it offers flexibility and lower data costs than concatenative synthesis but can sound robotic or contain artifacts.Source: §1.1 · pp. 2–3

1.2 Organization of This Survey

  • The survey organizes neural TTS around text analysis, acoustic models, and vocoders, then covers advanced practical challenges, related resources, and future research directions.Advanced topics include faster synthesis, low-resource and robust TTS, expressive speech, and voice adaptation.Source: §1.2 · pp. 3–4

2 Key Components in TTS

This section organizes neural TTS around text analysis, acoustic models, and vocoders, while also covering fully end-to-end systems and complementary taxonomies.

Fully end-to-end TTS and additional taxonomies

  • The survey also reviews fully end-to-end TTS and adds taxonomies for autoregressive/non-autoregressive generation, generative models, network structures, and representative research timelines.Source: §2 · p. 4

Text analysis

  • Text-analysis work spans TN [316], G2P, prosody prediction, unified models [444], and DeepVoice 1/2.Source: §2 · p. 5

Acoustic models

  • Acoustic-model work is grouped as HMM-based, DNN-based, RNN-based, and emphasis.Source: §2 · p. 5

Vocoders

  • Vocoder coverage distinguishes SPSS tools STRAIGHT and WORLD from linguistic-to-wave methods including WaveNet [254], WaveRNN, WaveGlow, HiFi-GAN, and DiffWave.Source: §2 · p. 5

2 Key Components in TTS

  • The survey maps neural TTS by key components and data flows, then reviews text analysis, acoustic models, vocoders, and fully end-to-end systems.It also introduces classifications by autoregressive versus non-autoregressive generation, generative model, network structure, and research timeline.Source: §2 · p. 4, Figure 3 · p. 5

2.1 Main Taxonomy

The survey organizes neural TTS around its core components and the data-conversion flow from text to waveform, while recognizing multiple possible paths through that flow.

2.1 Main Taxonomy

  • The taxonomy groups neural TTS into text analysis, acoustic models, vocoders, and fully end-to-end models, following the conversion from text representations to waveform.Text analysis produces phonemes or linguistic features; acoustic models produce acoustic features, and vocoders produce waveforms, while end-to-end models map characters or phonemes directly to waveform.Source: §2.1 · p. 6
  • The conversion flow uses characters, linguistic features, acoustic features, and waveform, with phonemes commonly representing text and spectrograms serving as acoustic features in neural end-to-end TTS.Linguistic features carry pronunciation and prosody context; mel- or linear-spectrograms are converted to waveform by neural vocoders.Source: §2.1 · p. 6
  • TTS systems can follow several routes, including character-to-linguistic-feature-to-acoustic-feature-to-waveform, character-to-phoneme-to-waveform, and direct character-to-waveform conversion.The survey also describes routes that bypass linguistic features or acoustic features.Source: §2.1 · p. 6

2.2 Text Analysis

Text analysis turns raw text into pronunciation- and prosody-relevant information for speech synthesis. Neural TTS often simplifies this frontend, but practical systems still need text normalization and may incorporate pronunciation and prosody processing.

2.2 Text Analysis

  • Text analysis (the frontend) extracts linguistic features from text; these can aggregate results across phoneme, syllable, word, phrase, and sentence levels for later TTS stages.End-to-end neural TTS often uses character or phoneme sequences directly, but raw text still requires normalization.Source: §2.2 · pp. 6–7
  • Text normalization converts non-standard text into spoken-form words; early rule-based methods were followed by neural sequence-to-sequence [316, 229, 437] and hybrid rule/neural approaches.Examples include normalizing “1989” to “nineteen eighty nine” and “Jan. 24” to “Janunary twenty-fourth”.Source: §2.2 · pp. 6–7
  • For character-based languages such as Chinese, word segmentation identifies boundaries needed for later analysis, while POS tags support grapheme-to-phoneme conversion and prosody prediction.Source: §2.2 · p. 7
  • Prosody prediction models rhythm, stress, and intonation through language-specific tags: English ToBI marks tones and breaks, while Chinese uses hierarchical PW, PPH, and IPH boundaries.Chinese prosody prediction has used CRF, RNN, and self-attention structures [3] [64].Source: §2.2 · p. 7
  • Grapheme-to-phoneme conversion generates pronunciations for English out-of-vocabulary words, whereas in Chinese it mainly disambiguates polyphones using context [29].Source: §2.2 · p. 7
  • Neural TTS incorporates text analysis through unified multi-task frontend models [444] and prosody features predicted from text encoders, reference speech, pre-training, or syntax-aware graph networks [208].Text encoders may predict pitch, duration, phrase breaks, breath, or filled pauses; reference encoders learn prosody representations from speech.Source: §2.2 · pp. 7–8

2.3 Acoustic Models

Acoustic models transform linguistic, phoneme, or character inputs into acoustic features for vocoding, with neural end-to-end TTS shifting toward mel-spectrogram prediction. The section traces the progression from SPSS models to sequence-to-sequence and parallel neural architectures.

2.3 Acoustic Models

  • Acoustic models predict features such as MGC, BAP, and F0 from linguistic inputs in SPSS, or mel-spectrograms from phonemes or characters in neural end-to-end TTS.The predicted features are converted into waveforms using vocoders; mel-spectrograms are the most widely used acoustic feature described.Source: §2.3 · p. 8

2.3.1 Acoustic Models in SPSS

  • HMM-based SPSS offered flexible speaker, emotion, and style changes but produced over-smoothed, low-detail features; DNNs and later contextual models addressed acoustic-model limitations.SPSS models generate speech parameters from linguistic features and rely on vocoders such as STRAIGHT and WORLD; overall quality was also constrained by vocoding techniques.Source: §2.3.1 · p. 8

2.3.2 Acoustic Models in End-to-End TTS

  • Neural end-to-end models learn alignments through attention or predict duration jointly, simplifying inputs to characters or phonemes and using higher-dimensional mel- or linear-spectrograms.Tacotron uses an encoder-attention-decoder, while Tacotron 2 predicts mel-spectrograms and adds a WaveNet vocoder.Source: §2.3.2 · p. 10
  • 270x inference speedup on mel-spectrogram generation and 38x speedup on waveform generation accompany FastSpeech’s parallel, duration-based synthesis, which avoids attention-related word skips and repeats while retaining on-par voice quality.A duration predictor and length regulator expand phoneme representations to match mel-spectrogram sequence length for parallel generation.Source: §2.3.2 · p. 11
  • FastSpeech 2 uses ground-truth mel-spectrogram targets and adds pitch, duration, and energy inputs to improve voice quality while retaining fast, robust, controllable synthesis.Using ground-truth targets removes the teacher-student distillation stage and avoids information loss from distillation.Source: §2.3.2 · p. 11
  • End-to-end acoustic models also use flow-based mel-spectrogram generation and VAE, GAN, and diffusion approaches, including non-autoregressive flow models [366] [186] [141] [276] [185].Examples include autoregressive Flowtron, non-autoregressive Flow-TTS and Glow-TTS, and diffusion-based Diff-TTS, Grad-TTS, and PriorGrad.Source: §2.3.2 · p. 12

2.4 Vocoders

The survey traces vocoders from statistical parametric systems to neural waveform generators and organizes neural approaches by generative model. It compares their modeling, training, and inference tradeoffs.

Vocoder categories

  • Neural vocoders span autoregressive, flow-, GAN-, VAE-, and diffusion-based families; early systems used linguistic features, while later systems often used mel-spectrograms.The survey focuses on neural vocoders because of their high voice quality.Source: §2.4 · pp. 12–13

Autoregressive Vocoders

  • WaveNet [254] autoregressively generates waveform points with dilated convolution but is slow, motivating efficient designs such as WaveRNN and LPCNet.WaveRNN uses a recurrent network with dual softmax, pruning, and subscaling; LPCNet combines linear prediction with a lightweight RNN residual model and can condition on BFCC or mel-spectrogram features.Source: §2.4 · p. 13

Flow-based Vocoders

  • Flow vocoders include autoregressive inverse autoregressive flow (IAF) and bipartite-transform variants; WaveGlow and FloWaveNet, based on bipartite transforms, achieve high voice quality and fast inference speed.Source: §2.4 · pp. 13–14

GAN-based Vocoders

  • GAN-based vocoders, including MelGAN, Parallel WaveGAN, and HiFi-GAN, use generators and discriminators to synthesize audio.Designs include staged or multi-scale upsampling, multi-receptive-field fusion, and random-window, multi-scale, multi-period, or hierarchical discriminators; STFT and feature-matching losses can improve training stability and perceptual quality.Source: §2.4 · pp. 14–15

Diffusion-based Vocoders

  • Diffusion vocoders such as DiffWave, WaveGrad [41], and PriorGrad [185] iteratively add noise to waveform data and reverse the process to generate speech.They can achieve very high voice quality, but their long iterative inference is slow, motivating work to reduce generation time while maintaining quality.Source: §2.4 · p. 15

Other Vocoders

  • Neural source-filter vocoders aim to combine high voice quality with controllable speech generation.Other work surveys vocoder types and evaluates common vocoders for robustness.Source: §2.4 · p. 15

Discussions

  • Autoregressive models are mathematically simpler, but only non-autoregressive generative models support parallel speech generation; GANs cannot estimate sample likelihoods.Non-autoregressive models can also support latent manipulation, except GAN vocoders that do not take random Gaussian noise as input.Source: §2.4 · p. 15

2.5 Towards Fully End-to-End TTS

Fully end-to-end TTS aims to generate waveforms directly from text, reducing annotation and feature work, error propagation, and cost, but training is difficult because text and speech differ in modality and sequence length. Neural TTS progresses toward direct text-to-waveform synthesis by simplifying representations and combining modules.

Motivation

  • Fully end-to-end TTS requires less human annotation and feature development, can avoid cascaded-model error propagation through joint optimization, and can reduce costs.The cited costs include training, development, and deployment.Source: §2.5 · p. 16

Challenges

  • 80k waveform samples for a 5-second, roughly 20-word utterance versus about 100 phonemes make memory-limited training and short-clip context difficult.The example assumes a 16 kHz sample rate; the waveform is too long to fit into training, while short audio clips make it hard to capture context representations.Source: §2.5 · p. 16

Stages 0–2

  • The progression starts with SPSS's three-module pipeline, then merges text analysis with the acoustic model and acoustic modeling with vocoding, while WaveNet still needs text analysis.Stage 0 uses text analysis to produce linguistic features, an acoustic model to generate acoustic features, and a vocoder to synthesize waveform. Wang et al. combine text analysis and the acoustic model; WaveNet [254] combines acoustic modeling and vocoding, with related models [254] still requiring text analysis.Source: §2.5 · pp. 16–17

Stage 3

  • Stage 3 simplifies text and acoustic representations: Tacotron predicts linear spectrograms from characters or phonemes, and later systems predict mel-spectrograms for neural vocoders.Tacotron converts its linear spectrograms to waveform with Griffin-Lim. DeepVoice 3, Tacotron 2, TransformerTTS, and FastSpeech 1/2 predict mel-spectrograms and use neural vocoders including WaveNet [254], WaveRNN, WaveGlow, FloWaveNet, and Parallel WaveGAN.Source: §2.5 · pp. 16–17

Stage 4

  • Stage 4 systems move toward direct text-to-waveform synthesis through jointly tuned, parallel, alignment-learning, and flow-based designs.Char2Wav jointly tunes an encoder-attention-decoder with SampleRNN; ClariNet jointly tunes an autoregressive acoustic model and non-autoregressive vocoder; FastSpeech 2s uses a fully parallel structure and auxiliary mel decoder. EATS [69] uses duration interpolation and soft dynamic time wrapping loss for alignment, while Wave-Tacotron combines a flow-based parallel waveform decoder with an autoregressive Tacotron part.Source: §2.5 · p. 17, Table 7 · p. 17

2.6 Other Taxonomies

Neural TTS works can be classified by generation direction, generative model, and network structure; the survey also charts model evolution using public-release dates and selects representative works.

2.6 Other Taxonomies

  • The model-evolution chart orders works by when papers became public, such as on arXiv, rather than by later formal publication.The survey uses early public-release dates to recognize researchers’ efforts to share work early.Source: §2.6 · p. 17
  • The survey classifies neural TTS by autoregressive versus non-autoregressive generation, generative model, and network structure.Generative-model categories include normal sequence generation, flow, GAN, VAE, and diffusion; structures include CNN, RNN, self-attention, and hybrids.Source: §2.6 · p. 17, Figure 5 · p. 18
  • Because neural TTS research is abundant, the evolution chart includes representative works, while Table 18 lists more works.Source: §2.6 · p. 18

3 Advanced Topics in TTS · 3.1 Background and Taxonomy · 3.2 Fast TTS

Section 3 surveys advanced neural TTS topics aimed at practical use, including faster synthesis, low-resource and robust systems, expressive speech, and voice adaptation. Section 3.2 reviews parallel generation, lightweight models, and speech-domain techniques for speeding inference.

3.1 Background and Taxonomy

  • Advanced TTS research targets faster synthesis, data efficiency in low-resource settings, robustness against skipped or repeated words, expressive prosody, and efficient adaptation to target speakers.The taxonomy also covers modeling, controlling, and transferring speech style to improve naturalness.Source: §3.1 · pp. 18–19

3.2 Fast TTS

  • A 1-second utterance may contain 500 mel-spectrogram frames or 24k waveform samples, making early autoregressive systems slow for cloud and embedded deployment.Fast TTS methods address this with parallel generation, lightweight structures, or speech-domain knowledge.Source: §3.2 · pp. 19, 21
  • RNN autoregressive models require O(N) computation, whereas FastSpeech 1/2 and most GAN-based models [69] use O(1) computation.DeepVoice 3 and TransformerTTS parallelize training but retain autoregressive inference; inverse autoregressive flow models parallelize inference but need teacher distillation, while flow and diffusion models [41] [185] [141] [276] use T iterations.Source: Table 8 · p. 21, §3.2 · p. 21
  • Parallel generation does not reduce model size or total computation, so lightweight design remains important on mobile and embedded devices with limited parallel-computing capability.Pruning, quantization, knowledge distillation [111], and neural architecture search are among the techniques used; semi-autoregressive mel generation combines autoregression within phonemes with parallel generation across phonemes.Source: §3.2 · p. 21
  • LightSpeech uses neural architecture search to speed up FastSpeech 2 by 6.5x while maintaining voice quality.LVCNet [431] speeds up Parallel WaveGAN by 4x without sound-quality degradation; SqueezeWave [433] reduces computation while achieving similar audio quality.Source: §3.2 · p. 21
  • Speech-domain methods speed inference through linear prediction, multiband or subband modeling, sample bunching, multi-frame prediction, and streaming synthesis.LPCNet combines linear prediction with a lightweight residual predictor; Bunched LPCNet achieves more than 2x speedup through sample and bit bunching.Source: §3.2 · pp. 21–22
  • Streaming TTS [323] can synthesize as input tokens arrive, while FFTNet generates audio in real time.Okamoto et al. improve FFTNet voice quality with noise shaping and subband techniques; Popov et al. use frame splitting and cross-fading for low-end devices, and Kang et al. [152] enable real-time DCTTS synthesis on a single CPU thread.Source: §3.2 · p. 22

3.3 Low-Resource TTS

High-quality TTS typically requires abundant paired text and speech, yet most of the world’s more than 7,000 languages lack training data. The section surveys methods for improving TTS when paired data is scarce.

Self-supervised training

  • Self-supervised pretraining uses unpaired text or speech to strengthen TTS models, while quantized speech tokens can provide pseudo-paired data for fine-tuning on limited true pairs.Approaches include BERT-enhanced text encoders, pretrained speech decoders, and speech quantization [52, 81, 143, 440, 352, 201, 358, 436].Source: §3.3 · p. 22

Cross-lingual transfer

  • Cross-lingual transfer uses rich-resource languages to support low-resource TTS, addressing different phoneme sets through embedding mapping, reinitialization, IPA [109], or byte representations [108].Language similarity can also guide transfer.Source: §3.3 · pp. 22–23

Cross-speaker transfer

  • Cross-speaker transfer improves synthesis for a data-limited speaker by converting other speakers’ voices into the target voice [128] or adapting or cloning models trained on other voices [44] [40].Source: §3.3 · p. 23

Speech chain/Back transformation

  • Speech chain and back transformation exploit the duality of TTS and ASR to use unpaired text and speech data to improve both tasks.Source: §3.3 · p. 23

Dataset mining in the wild

  • Dataset mining in the wild finds low-quality paired text and speech on the Web for TTS training, with speech enhancement, denoising, and disentangling improving mined audio.Cooper and Hu et al. propose mining such data and developing training techniques.Source: §3.3 · p. 23

3.4 Robust TTS

Robust TTS addresses word skipping, repetition, and unintelligible attention collapse by improving text–speech alignment and mitigating autoregressive exposure bias and error propagation. The survey focuses mainly on acoustic models, where these issues are more severe than in vocoders.

3.4 Robust TTS

  • Neural TTS robustness problems include alignment-related skipping, repetition, and attention collapse, plus autoregressive error propagation; vocoder glitches are less severe, and unseen test domains pose an additional challenge.Attention collapse produces unintelligible gibberish when attention fails to focus on one input token. Vocoder glitches include hoarseness, metallic noise, jitter, or pitch breaking; proposed remedies include universal vocoder modeling [137] [144] and sophisticated designs [35]. Broader training-domain coverage [130] and relative position encoding [17] can help with unseen domains.Source: §3.4 · p. 23

3.4.1 Enhancing Attention

  • Attention robustness methods leverage local, monotonic, and complete text–speech alignments through location-sensitive and stepwise-monotonic attention or stronger encoder–decoder connections.Location-based attention can ensure monotonicity when properly handled; stepwise monotonic attention advances at most one input position per decoding step, preventing skips. Multi-frame prediction, prenet dropout or bottlenecks, positional connections, and an ASR-based cycle loss can strengthen encoder–decoder alignment.Source: §3.4.1 · pp. 24–25

3.4.2 Replacing Attention with Duration Prediction

  • Attention-free duration prediction bridges text and mel-spectrogram lengths, using external or internally learned alignments and either non-end-to-end training or end-to-end duration optimization.External alignments include encoder–decoder attention, CTC, and HMM tools; internal methods include dynamic programming, monotonic alignment search, and soft DTW. EATS [69] and Parallel Tacotron 2 [75] optimize duration end-to-end, while Non-Attentive Tacotron [304] supports semi-supervised duration prediction when labels are unavailable.Source: §3.4.2 · pp. 25–26

3.4.3 Enhancing AR Generation

  • Autoregressive mitigation methods target the training–inference mismatch with professor forcing, teacher–student distillation, bidirectional generation, and spectrogram-noise or frame-replacement augmentation.Teacher forcing uses ground-truth previous values in training, unlike inference, where the model uses its own predictions; accumulated prediction errors can therefore worsen later frames. Distillation aligns teacher and student hidden states, while augmentations simulate prediction errors.Source: §3.4.3 · pp. 26–27

3.4.4 Replacing AR Generation with NAR Generation

  • Non-autoregressive generation avoids autoregressive exposure bias and error propagation, using positional attention for alignment or duration prediction to bridge text and speech lengths.The survey groups TTS by alignment and generation type: autoregressive or non-autoregressive, each with attention or duration-based alignment; examples include ParaNet, Flow-TTS, FastSpeech, and EATS [69].Source: §3.4.4 · p. 27

3.5 Expressive TTS

Expressive TTS addresses the variation behind multiple plausible realizations of the same text by representing, modeling, disentangling, controlling, and transferring speech attributes.

3.5 Expressive TTS

  • Because one text can map to many durations, pitches, volumes, styles, and emotions, insufficiently conditioned L1 training over-smooths predictions; modeling variation enables expressive control, transfer, and disentanglement.Source: §3.5 · p. 27

3.5.1 Categorization of Variation Information

  • Variation information comprises text content, speaker/timbre, prosody/style/emotion, and recording channel/noise; prosody, style, and emotion are central targets for improving expressiveness.Source: §3.5.1 · p. 28

3.5.2 Modeling Variation Information

  • Explicit inputs use labeled language, speaker, style, or prosody, or extracted pitch, energy, and duration; implicit approaches include reference encoders, style tokens, VAEs, generative models, and text pretraining.Variation can be modeled from language and speaker level down to frames; hierarchical structures combine granularities such as utterance, phrase, word, and phoneme levels.Source: §3.5.2 · pp. 28–29

3.5.3 Disentangling, Controlling and Transferring

  • Disentanglement separates entangled content, prosody, timbre, and noise, while feedback or semi-supervised/adversarial learning helps ensure input attributes control synthesis.Style transfer can use labeled tags or obtain variation information from reference speech, predict it from text, or sample it from a latent space.Source: §3.5.3 · pp. 29–30

3.6 Adaptive TTS

Adaptive TTS adapts a source model—usually trained on multiple speakers—to synthesize a target user’s voice from limited adaptation data. This section focuses on voice adaptation and surveys generalization and efficient-adaptation methods.

3.6.1 General Adaptation

  • General adaptation improves source-model generalization through acoustic-condition modeling [40], speaker augmentation, multilingual training, and support for spontaneous/style and cross-language speech [212] [109].Acoustic-condition modeling addresses mismatches in recording conditions, accents, speaking rates, and timbre; AdaSpeech 3 adapts reading-style synthesis to spontaneous speech.Source: §3.6.1 · p. 31

3.6.2 Efficient Adaptation

  • Few-shot methods use paired data ranging from minutes to seconds; Chen et al. [40] and Arık et al. find quality improves quickly below 20 sentences, then more slowly.More adaptation data can improve voice quality but increases data-collection cost.Source: §3.6.2 · pp. 31–32
  • AdaSpeech [40] reduces tuned parameters with conditional layer normalization; untranscribed adaptation uses reconstruction and latent alignments or ASR transcription.Adaptation can tune the full model, part of it, or only speaker embeddings; tuning fewer parameters reduces memory and deployment cost.Source: §3.6.2 · pp. 31–32
  • Zero-shot methods [44] [56] [32] extract speaker embeddings from reference audio, requiring no adaptation data or parameters, but quality can suffer for speakers unlike the source speakers.Source: §3.6.2 · p. 32

4 Resources

The survey collects TTS resources spanning open-source implementations, tutorials and keynotes, challenges, and corpora. It organizes these resources in a catalog and describes corpora by hours, speakers, sampling rate, and language.

4 Resources

  • The resource survey covers open-source implementations, tutorials and keynotes, TTS challenges, and corpora.These categories are presented together in Table 17.Source: §4 · p. 32, Table 17 · p. 32
  • Tutorial and keynote resources include offerings from ISCSLP, IEICE, INTERSPEECH, IJCAI, and other venues.Examples include ISCSLP 2014, ISCSLP 2016, IEICE, and INTERSPEECH.Source: §4 · p. 33
  • The listed challenges include the Blizzard Challenge, Zero Resource Speech Challenge, and Voice Conversion Challenge.Source: §4 · p. 33
  • The corpus catalog is organized by hours, number of speakers, sampling rate, and language.Source: §4 · pp. 32–33

5 Future Directions

The survey reviews core neural TTS models and advanced topics, then frames future directions around high-quality and efficient speech synthesis. Together, these directions address speech quality and the costs of data collection, training, and serving.

High-quality speech synthesis

  • High-quality TTS remains the primary goal, with further improvement needed across intelligibility, naturalness, expressiveness, prosody, emotion, style, robustness, and controllability.Neural methods have improved synthesized speech quality, but substantial room for progress remains.Source: §5 · p. 34
  • More powerful and efficient generative models for waveforms or acoustic features could further improve speech quality.The survey identifies VAE, GAN, flow, and diffusion models as already adopted in acoustic models, vocoders, and fully end-to-end systems.Source: §5 · p. 34
  • Better text and speech representations, especially learned through unsupervised or self-supervised pre-training, could improve TTS quality and prosody.Initial text-pretraining explorations indicate that better text representations can improve speech prosody.Source: §5 · p. 34
  • Robust synthesis requires generalizing beyond training data to corner cases such as longer texts and different domains.Current models address word skipping and repetition from incorrect attention alignments but still face these robustness issues.Source: §5 · p. 34
  • Expressive, controllable, and transferable TTS requires variation modeling that balances precise control with inference-time availability.Explicit reference or prosody features support control and transfer but can be unavailable at inference; implicit latent sampling supports expressiveness but not precise feature control.Source: §5 · p. 34
  • More human-like synthesis depends on modeling casual, emotional, and spontaneous speech rather than only formal reading styles.Conversational speech can include pauses, repeats, changing speeds, varying emotions, and errors.Source: §5 · p. 34

Efficient speech synthesis

  • Data-efficient TTS should leverage unsupervised or semi-supervised learning and cross-lingual transfer for low-resource languages and limited-data speaker adaptation.ZeroSpeech Challenge explores learning from speech alone, without text or linguistic knowledge.Source: §5 · p. 34
  • Efficient TTS also requires compact models for constrained devices and lower-energy training and inference to reduce costs and carbon emissions.The survey highlights limited memory and power on mobile, IoT, and other low-end devices, and proposes reducing FLOPs as one route to energy efficiency.Source: §5 · pp. 34–35

Related papers

Loading 2106.15561v3…