Review of end-to-end speech synthesis technology based on deep learning

• Speech synthesis is the most complex human-computer interaction speech task because high-fidelity, stylistically varied output requires substantial training data and complex models.Source: §1 · p. 1
• Deep learning improves modeling of linguistic–acoustic dependencies, while recurrent models address continuity across adjacent speech frames that feedforward DNNs treat independently.Source: §1 · p. 2
• Traditional SPSS uses multiple manually engineered modules, whereas end-to-end TTS learns directly from paired text–speech data and can synthesize speech close to human voice.Source: §1 · p. 2
• The end-to-end pipeline contains a text front-end, acoustic model, and vocoder that convert text into standardized input, acoustic features, and waveform samples.Source: §1 · p. 2
• Earlier reviews mainly covered baseline systems such as WaveNet, Tacotron, and SampleRNN, leaving newer deep learning TTS methods insufficiently surveyed.Source: §1 · p. 2
• The review classifies recent methods for all three TTS components, summarizes multilingual speech corpora and subjective/objective evaluation methods, and identifies future research directions.Source: §1 · p. 3

• Text front-ends supply pronunciation and syntactic information needed for correct pronunciation, segmentation, pauses, and prosody, especially in languages such as Mandarin.Source: §2 · p. 3
• Traditional front-ends cascade normalization, word segmentation, part-of-speech tagging, grapheme-to-phoneme conversion, and prosodic structure prediction.Source: §2 · p. 3
• Deep learning NLP models including BLSTM-CRF, Transformer, Word2Vec, and BERT extract context that improves pronunciation, ambiguity resolution, and prosodic-boundary prediction.Source: §2 · p. 4

Text normalization

• Neural text normalization converts non-standard Mandarin words into spoken forms with higher prediction accuracy than a rule-based system.Source: §2 · p. 4

Prosodic structure prediction

• Multi-task learning models interrelated Mandarin prosodic levels, including prosodic words, prosodic phrases, and intonation phrases.Source: §2 · p. 4

Pronunciation prediction

• Polyphonic-character G2P is context-sensitive; Transformer and BERT improve prediction, while TinyBERT reduces model size and multilingual Transformer front-ends support uncommon and long texts.Source: §2 · p. 4

Style and unified front-ends

• Front-ends can add style and breath information, while unified models jointly process text and predict phoneme, tone, and prosody labels.Source: §2 · pp. 4–5

3 Acoustic model

• Tacotron maps text to mel-spectrograms with an attention-based encoder-decoder, but autoregression, alignment errors, limited controllability, and high data demands constrain deployment.Its CBHG encoder and recurrent decoder generate multiple mel-spectrogram frames per decoding step, yet sequential dependence prevents parallel computation.Source: §3 · p. 5

3.1.1 Non-RNN acoustic model

• Fast acoustic models replace recurrent components with CNNs, Transformers, or DFSMNs to enable parallel computation while preserving or reducing model complexity.Tacotron 2, Deep Voice 3, DCTTS, Transformer-based models, and DFSMN-based models exemplify these substitutions.Source: §3.1.1 · pp. 5–6

3.1.2 Non-autoregressive acoustic model

• Non-autoregressive models use teacher-guided alignment, lightweight replacements, normalizing flows, or GANs to generate acoustic features in parallel and simplify or accelerate synthesis.FastSpeech uses a length regulator and autoregressive teacher, while Glow-TTS, Flow-TTS, and EATS avoid autoregressive generation through alternative generative models.Source: §3.1.2 · p. 6

3.1.3 Streaming acoustic model

• Streaming incremental TTS outputs speech while text arrives, improving interaction latency, but limited context makes prosody modeling difficult and motivates future-word conditioning and buffered inference.Applications include simultaneous translation, dialog generation, and assistive technologies.Source: §3.1.3 · pp. 6–7

3.2 Robust acoustic model

• Robust acoustic modeling addresses rare but severe autoregressive and attention failures, including premature stopping, repetitions, skipped words, long pauses, and nonsense.Such errors may be missed in small listening tests but substantially harm customer-facing user experience.Source: §3.2 · p. 7

3.2.1 Stable autoregressive generation process

• Stable autoregressive generation methods reduce exposure bias and error propagation through scheduled sampling, adversarial or iterative training, distillation, bidirectional modeling, augmentation, and auxiliary text recognition.An auxiliary CTC recognizer maximizes mutual information between predicted acoustic features and input text, strengthening text dependence.Source: §3.2.1 · pp. 7, 9

3.2.2 Accurate alignment

• Accurate alignment methods enforce locality, monotonicity, and completeness because content-based attention commonly produces alignment errors in long or complex utterances.GMM attention improves long-sentence generalization at some cost to naturalness, while hybrid location-sensitive attention combines content and location information.Source: §3.2.2 · p. 10
• Monotonic attention variants improve stability: DCA is more stable and generalizable, MA guarantees forward single-unit focus with linear complexity, and SMA additionally prevents skipped input units.SMA ensures locality, monotonicity, and completeness, although hard attention requires specialized training approximations rather than standard back-propagation.Source: §3.2.2 · p. 11
• Monotonic attention methods constrain alignment to proceed left-to-right, improving stability or avoiding muffling, skipping, and repetition errors, although hard monotonic attention requires long training.Zhang et al. recursively compute monotonic attention probabilities with a forward algorithm and transition agent, while Yasuda et al. factorize generation into alignment-transition and emission probabilities.Source: §3.2.2 · p. 11
• Guided attention loss concentrates attention weights near the diagonal, while forced increment attention limits each acoustic frame’s text-position advance to at most one.These methods directly incorporate text–speech correspondence to enforce monotonic alignment during training.Source: §3.2.2 · p. 11
• Deep Voice 3 and ParaNet promote monotonic alignment by adding positional encoding to Transformer content-based dot-product attention.Source: §3.2.2 · p. 11

SA SMA

• Duration predictors avoid attention-induced skipping, repetition, and irregular stops by aligning input phonemes with output acoustic features, using external aligners or internal alignment modules.External methods include knowledge distillation, ASR or forced-alignment tools, and HMM-based aligners, while internal methods include dynamic programming, MAS, monotonic length prediction, and positional attention.Source: SA SMA · pp. 12–13

3.3 Expressive acoustic model

• Expressive synthesis models separately address “what to say,” “who to say,” and “how to say,” with speaker identity and prosodic attributes treated collectively as style features.Prosodic style includes tone, speech rate, and emotion, while reference encoders can control style parameters directly or transfer them from reference audio.Source: §3.3 · p. 13, §3.3.1 · p. 13

3.3.1 Acoustic model with reference encoder

• Reference-encoder systems disentangle and control prosody, pitch, loudness, emotion, rhythm, content, and timbre, using embeddings, classifiers, GST tokens, multiple encoders, or VAE latent variables.GST-Tacotron can suffer blurry synthesis and text leakage with unpaired inputs, while multi-reference and VAE methods target more independent style control; some approaches extend modeling to word- and phoneme-level prosody.Source: §3.3.1 · pp. 13, 15
• Alignment results reported for models including DurIAN, FastSpeech, ParaNet, Non-Attentive Tacotron, AlignTTS, Glow-TTS, DeviceTTS, and RobuTrans are generally stable, while FastSpeech 2 can err on long sentences.Glow-TTS is also reported to control the synthesized pitch contour, and DeviceTTS combines high synthesized-speech quality with fast training and inference.Source: §3.3.1 · p. 14
• Reference encoders model speech style with VAE, normalizing-flow, and GAN approaches that improve interpretability, style control, disentanglement, or content-style separation.Parallel Tacotron uses separate VAE models for global and phoneme-level features; Flowtron and Glow-TTS control style through latent variables, while Ma et al. combine adversarial and collaborative games.Source: §3.3.1 · p. 16

3.3.2 Acoustic model of explicit modeling style features

• Explicit style models control prosody through waveform constraints or interpretable duration, pitch, energy, and style inputs aligned with linguistic structure.Morrison et al. accept partial f0 contours, CHiVE models hierarchical f0, energy, and duration, and FastSpeech 2 predicts variance features for specified styles.Source: §3.3.2 · p. 16

3.3.3 Multi-speaker acoustic model

• Multi-speaker synthesis commonly uses speaker embeddings or fine-tuning, while learnable encoders, contrastive and cyclic losses, and identity feedback improve synthesis for unseen speakers.Speaker embeddings can be extracted from limited reference fragments, and adaptation fine-tunes embeddings, model components, or the whole model using few text-speech pairs.Source: §3.3.3 · pp. 16–17

3.4 Low-resource acoustic model

• Low-resource TTS methods address vocabulary gaps, scarce language data, noise, and limited paired data through phoneme modeling, transfer learning, semi-supervised pre-training, and dual learning.Text and phoneme inputs support unseen-word pronunciation, while cross-language transfer, VQ-VAE pre-training, and dual TTS-ASR learning reduce paired-data requirements.Source: §3.4 · pp. 17–18
• Low-resource systems also augment target-speaker data through voice conversion and reduce noise by denoising acoustic features or learning noise-related style variables.Voice-converted data can be used before target-speaker fine-tuning, while GST-Tacotron and mixed-Gaussian representations enable clean-speech synthesis from noisy references.Source: §3.4 · p. 18

4 Vocoder

• WaveNet provides conditional, long-range waveform modeling but its autoregressive structure makes training and inference slow.It uses dilated causal convolutions with speaker and linguistic conditioning.Source: §4 · p. 18

4.1.1 Small size vocoder

• Smaller vocoders reduce computational cost through simpler networks, multiscale processing, parameter pruning, quantization, or linear prediction.WaveRNN is described as suitable for resource-constrained devices, while Multi-Band WaveRNN and LPCNet further improve efficiency.Source: §4.1 · p. 18, §4.1.1 · p. 19

4.1.2 Non-autoregressive vocoder

• Normalizing-flow vocoders enable parallel synthesis through inverse autoregressive or bipartite transformations, but expressiveness and inference speed involve trade-offs.Parallel WaveNet and ClariNet use distillation for parallel inference; WaveGlow and FloWaveNet use bipartite transformations that may require deeper networks for comparable quality.Source: §4.1.2 · pp. 19–20
• WaveVAE and WaveFlow simplify or accelerate flow-based training, while GAN vocoders use adversarial and multiresolution losses to generate speech in parallel.WaveVAE trains from scratch at the expense of sound quality, whereas WaveFlow avoids complex distillation and targets fast, high-fidelity synthesis with small memory use.Source: §4.1.2 · pp. 20, 22

4.2 High-quality vocoder

• Diffusion vocoders generate waveforms non-autoregressively with fast decoding, but their inference is slightly slower than flow-based vocoders.GAN-based approaches improve naturalness using multiscale discriminators, feature matching, multiresolution STFT losses, and periodic-pattern modeling.Source: §4.1.2 · p. 22, §4.2 · p. 23, Table 5 · p. 24
• Few-shot multispeaker vocoding can adapt through speaker embeddings, parameter fine-tuning, or an encoder, with full-model fine-tuning yielding the highest naturalness under high-quality training data.The reported method works only when the training speech data quality is high.Source: §4.2 · pp. 23–24

5 Speech corpus

• High-quality labeled speech data are essential for TTS, but comparatively few open-source corpora meet the requirements for deep-learning systems.The review introduces available corpora to support reproducible TTS experiments.Source: §5 · p. 24

5.1 English speech corpus

• English offers many corpora, including multispeaker VCTK, single-speaker LJ Speech, and processed LibriTTS designed to address ASR-oriented data limitations.LibriTTS provides 585 hours at 24 kHz from 2,456 speakers with standardized text and reduced background noise.Source: §5.1 · pp. 24–25

5.2 Mandarin speech corpus

• Mandarin corpus construction is more costly because characters require phoneme or pinyin transcription, while tone and prosodic structure complicate TTS.CSMSC, AISHELL-3, and DiDiSpeech provide single-speaker, multispeaker, and large-scale Mandarin resources, respectively.Source: §5.2 · pp. 25–26

5.3 Multilingual speech corpus

• Multilingual resources broaden language coverage, but Common Voice is noisy and requires denoising before TTS use.CSS10 contains one speaker across ten languages, whereas Common Voice contains nearly 9,283 hours across 60 languages.Source: §5.3 · p. 26

5.4 Emotional speech corpus

• Public emotional-speech resources remain limited because available datasets are often small or noisy and generally lack sufficient training data for deep-learning emotional TTS.EmoV-DB addresses part of this gap with five emotions across English and French.Source: §5.4 · p. 26

6 Evaluation method

• Speech quality is assessed through clarity, intelligibility, and naturalness, but no uniform evaluation criterion currently exists.The review therefore separates subjective listening tests from objective quantitative metrics.Source: §6 · p. 27

6.1 Subjective evaluation method

• MOS rates naturalness on a five-point scale, whereas MUSHRA uses hidden references and anchors on a 0–100 scale with finer discrimination.MUSHRA requires fewer participants for statistically significant results than MOS, according to the review.Source: §6.1 · p. 27
• CMOS, AB, and ABX tests compare models directly, with ABX evaluating which output more closely matches a target speaker’s characteristics.These relative methods complement absolute ratings when model comparison is the goal.Source: §6.1 · p. 27

6.2 Objective evaluation method

• Objective metrics quantify acoustic, prosodic, intelligibility, spectral, and distributional differences, including F0 errors, CER/WER/UER, MCD, PESQ, FDSD, and KDSD.MCD measures mel-cepstral spectral distance, while related MSD captures harmonic content from logarithmic mel-spectral amplitude.Source: §6.2 · pp. 27–28
• GPE and VDE measure pitch and voicing errors, while FFE combines them into the percentage of frames containing either error.Lower GPE, VDE, and FFE values indicate better performance.Source: §6.2 · p. 28

7 Future development direction

• Stable real-time synthesis and reliable quality remain unresolved, while research is concentrated in common languages because paired text–speech data are scarce for minor languages and dialects.These limitations motivate broader future work beyond current waveform-cascade systems and common-language settings.Source: §7 · p. 28
• Fine-grained control of emotion, intonation, and rhythm at word and phrase levels requires better use of limited, imperfectly labeled emotional-speech data.The paper identifies both controllability and emotional-data scarcity as urgent problems.Source: §7 · p. 28
• Unsupervised representation learning and meta-learning could improve speech-data modeling while reducing the labeled speech required for TTS training.The proposed direction draws on learning acoustic units or internal phonetic representations from speech signals.Source: §7 · pp. 28–29
• A fully end-to-end model should map original text or phonemes directly to waveforms, avoiding separately trained modules and accumulated intermediate errors.Existing systems commonly retain text front-end, acoustic-model, and vocoder stages despite being called end-to-end.Source: §7 · p. 29
• TTS may benefit from image-generation methods, NMT and ASR alignment techniques, and dual training that combines synthesis with recognition tasks.Examples include pairing TTS with ASR, speaker recognition, or speech-emotion recognition.Source: §7 · p. 29

8 Conclusion

• The review details technologies used in each TTS module and classifies methods by their characteristics while comparing their advantages and disadvantages.This provides a structured overview of the current end-to-end TTS paradigm.Source: §8 · p. 29
• It summarizes public speech corpora for TTS tasks and commonly used subjective and objective speech-quality evaluation methods.The scope includes resources and assessment practices alongside model technologies.Source: §8 · p. 29
• The review proposes suggestions for future development directions in deep-learning end-to-end TTS.These directions extend the paper’s survey of current methods and resources.Source: §8 · p. 29