A Review of Deep Learning Techniques for Speech Processing

• Speech processing includes automatic speech recognition (ASR), speaker recognition (SR) [31], and speech synthesis, with HMMs enabling major advances.Source: §1 · p. 3
• Deep learning automatically extracts meaningful features from raw speech, improving accuracy, robustness, and generalization in noise, diverse accents, and dialects.Source: §1 · p. 3
• Despite substantial progress, deep-learning speech systems still require large labeled datasets and improved interpretability and robustness across environmental conditions.Source: §1 · p. 4
• The review examines RNNs, CNNs, Transformers, GNNs, diffusion models, and representation learning techniques for speech processing.Source: §1 · pp. 3–4
• It analyzes speech recognition, synthesis, speaker recognition, and speech-to-speech translation, then covers domain adaptation, meta-learning, and parameter-efficient transfer learning.Source: §1 · p. 4
• The review addresses a broad survey gap by organizing current techniques and applications for researchers, practitioners, and beginners, while identifying future research directions.Source: §1 · pp. 4–5

2.1 Speech Signals

• Speech signals are human-generated air-pressure variations that are converted into electrical signals and digitized by sampling amplitude at consistent time intervals.Periodic signals have a fixed period whose reciprocal is frequency, while waveform shape influences perceived timbre.Source: §2.1 · p. 5

2.2 Speech Features

• Speech features comprise time-domain and frequency-domain representations, which capture different information and can be combined for speech-processing tasks.Time-domain features include energy, zero-crossing rate, pitch, and LPC; frequency-domain features use spectra or spectrograms and generally require time-frequency transforms.Source: §2.2 · pp. 5–7
• MFCCs and Mel spectrograms provide compact or perceptually meaningful time-frequency representations widely used in speech recognition and other audio-processing tasks.MFCC extraction includes framing, Fourier transformation, Mel filtering, logarithmic scaling, discrete cosine transformation, and liftering; frequency-based representations are described as more noise-robust than temporal variations.Source: §2.2 · p. 7

2.3 Traditional models for speech processing

• Traditional shallow models extract features through mathematical operations and pass them to classifiers or regressors, including GMMs, SVMs, HMMs, KNN, and decision trees.These methods support speech recognition, speaker and language recognition, phoneme recognition, and speech synthesis, but have been progressively superseded by deep neural networks with enhanced capabilities.Source: §2.3 · pp. 7–8

3.1 Recurrent Neural Networks (RNNs)

• RNNs model sequential speech patterns and support recognition, synthesis, and language understanding, but recurrence slows training and limits accurate long-range dependency modeling.End-to-end RNN systems replaced HMM hybrids for sequence transduction, while CTC and attention-based encoder-decoders address input-output alignment; AED models can perform poorly on lengthy utterances.Source: §3.1 · p. 9, §3.1.1 · p. 10, §3.1.2 · p. 11

3.1.1 RNN Models

• RNN variants address specific sequence-modeling limitations: BRNNs combine past and future context, LSTMs retain or forget long-range information, and GRUs provide a lighter gated alternative.LSTMs use separate long- and short-term memory cells with three gates, whereas GRUs use two gates and one memory cell.Source: §3.1.1 · pp. 9–10

3.1.2 Application

• RNN-based applications include audiovisual speech activity detection, ASR, speech synthesis, post-filtering, and neural vocoding, with multimodal BRNNs improving recognition in noisy conditions.CNN-LSTM hybrids extract local frame features before temporal processing, while WaveNet, SampleRNN, and Tacotron generate synthesized speech from acoustic features without intermediate vocoding steps.Source: §3.1.2 · p. 11

3.2 Convolutional Neural Networks

• CNNs suit speech spectrograms because locality in time and frequency captures structural features while weight sharing reduces model complexity and improves robustness to signal variation.CNNs use alternating convolution and pooling layers to extract local patterns from grid-like inputs.Source: §3.2 · pp. 11–12, §3.2.1 · p. 13

3.2.1 CNN Model Variants

• 1D CNNs operate along the temporal dimension and are computationally and memory efficient, while 2D CNNs can be more expensive for large speech inputs.A cited emotion-prediction study found 2D CNN performance lower than 1D CNN performance; 1D convolution and pooling can also mitigate noise effects.Source: §3.2.1 · pp. 12–13

3.2.2 Application

• CNNs support ASR, speaker recognition, speech enhancement, text-to-speech, emotion recognition, health-condition detection, and language identification, including wav2vec2.0 and VGGVox state-of-the-art systems.CNNs extract speech features for downstream classification, while wav2vec2.0 feeds CNN-learned raw-speech representations into a transformer language model.Source: §3.2.2 · pp. 13–14

3.2.3 Temporal Convolution Neural Networks

• Temporal convolutional networks provide a parallelizable alternative to recurrent models, achieving comparable performance to LSTMs and GRUs while avoiding recurrent gradient and long-memory limitations.TCNNs use causal convolutions to prevent future leakage and dilated convolutions to expand the receptive field without increasing computational complexity; shared filters improve memory efficiency.Source: §3.2.3 · p. 14, §3.2.4 · p. 14, Figure 2 · p. 15

3.2.5 Application

• TCNNs support real-time speech enhancement, speech and music detection, and ASR feature extraction, with fewer trainable parameters and improved efficiency for long or memory-constrained inputs.Their applications include time-domain enhancement, radio-broadcast detection, single-channel enhancement, and waveform filter banks for ASR.Source: §3.2.5 · pp. 15–16

3.3.1 Basic Architecture

• Transformers use stacked encoder-decoder blocks with multi-head attention, feedforward networks, residual connections, and normalization to capture long-range sequence dependencies.Attention weights values according to query-key compatibility, while multiple heads attend to different representational spaces and dependency ranges.Source: §3.3.1 · pp. 16–18

3.3.2 Application

• Transformer applications span ASR, speech translation, TTS, enhancement, representation learning, and multimodal learning, including wav2vec, Data2vec, Whisper, and VALL-E frameworks.Whisper supports multilingual recognition, translation, and language identification, while VALL-E generates personalized speech from a 3-second acoustic prompt and VALL-E X enables cross-lingual synthesis.Source: §3.3.2 · pp. 18–19
• Transformers outperform RNNs in most of 15 evaluated ASR, multilingual ASR, speech-translation, and TTS applications, but speech deployment requires complex configurations and substantial computation.Speech variability in pitch, accent, speed, and phrasing further requires robustness beyond text-oriented modeling.Source: §3.3.2 · pp. 18–19, §3.3.1 · p. 18

3.4 Conformer

• Conformer combines convolutional layers for local dependencies with self-attention for global dependencies, enabling strong performance in speech recognition, speaker recognition, and language identification.It has outperformed RNNs, earlier Transformers, and CNN-based models, and set benchmarks on LibriSpeech and AISHELL-1.Source: §3.4 · pp. 20–21, §3.4.1 · p. 22

3.4.1 Application

• Conformer extensions address robustness, adaptation, and real-time processing through noisy-student training, SpecAugment, wav2vec 2.0 pre-training, LHUC speaker adaptation, multimodal processing, and transducer models.Conformer-LHUC shows promise for elderly speech recognition and Alzheimer’s disease diagnosis and treatment, while transducer-based models process audio faster than conventional RNNs.Source: §3.4.1 · p. 22

3.5 Sequence to Sequence Models

• Sequence-to-sequence models use encoder-decoder networks to map variable-length speech, text, or spectrogram inputs to output sequences for ASR, speech translation, TTS, and voice conversion.They can translate speech spectrograms directly between languages without an intermediate text representation, and may use RNN, Transformer, or Conformer components.Source: §3.5 · p. 22, §3.5.1 · p. 24

3.5.1 Application

• Attention-based and CTC-based sequence-to-sequence methods achieve competitive speech-processing results, with hybrid attention-CTC systems remaining an active research direction.Attention provides soft alignment using decoder states and encoder representations, whereas CTC considers all possible monotonic alignments between input and target sequences.Source: §3.5.1 · p. 24

3.6 Reinforcement Learning

• Deep reinforcement learning applies value-, policy-, or model-based methods to speech recognition, diarization, enhancement, and emotion modeling, often using preprocessed audio features such as MFCCs or spectrograms.In ASR, states may represent speech features, actions phoneme choices, and rewards phoneme correctness or edit-distance similarity; learning directly from raw audio can reduce reliance on hand-engineered features.Source: §3.6 · p. 24, §3.6.1 · pp. 24–25, §3.6.2 · p. 25

3.6.2 Application

• Deep reinforcement learning learns from raw data and feedback without hand-engineered features, improving flexibility, adaptability, and robustness in noisy environments across speech-related applications.Applications include speech enhancement for ASR, speech emotion recognition, speaker identification, and speech synthesis or coding.Source: §3.6.2 · pp. 25–26
• Deep reinforcement learning requires substantial training data and computation, while reward-function design must be carefully selected to produce the desired behavior.This constraint is especially relevant when optimizing speech enhancement against recognition outcomes in complex, non-differentiable ASR systems.Source: §3.6.2 · pp. 25–26

3.7.1 Basic Models

• GNNs represent speech as graphs of frames connected by adjacent-frame edges, then aggregate neighborhood information through graph convolution, attention, recurrent, and task-specific output layers.The standard pipeline can use graph convolutional, graph attention, or graph transformer layers, followed by classification, regression, or graph pooling outputs.Source: §3.7.1 · pp. 26–27

3.7.2 Application

• GNNs model complex dependencies for diarization, verification, synthesis, separation, and recognition, including multichannel enhancement that combines GCN-based separation with MVDR beamforming and SGJMAP gain estimation.GNNs can also model relationships among phonemes, words, pitch, duration, and amplitude.Source: §3.7.2 · p. 27

3.8 Diffusion Probabilistic Model

• Diffusion models progressively add noise until speech approaches a Gaussian distribution, then learn a reverse noise-estimation process to reconstruct data samples from noise.Unlike VAE and flow models, diffusion models keep latent dimensionality fixed and use a Markov chain of diffusion steps.Source: §3.8 · pp. 27–28

3.8.1 Application

• Diffusion models generate high-quality speech non-autoregressively; FastDiff runs 58 times faster than real time on a V100 GPU and outperforms competing end-to-end text-to-speech methods, while DiffWave matches WaveNet quality faster.FastDiff has achieved practical deployment speed, and DiffWave supports neural vocoding, class-conditional generation, and unconditional generation.Source: §3.8.1 · p. 28
• Diffusion-based speech enhancement incorporates noisy-signal characteristics into forward and reverse processes, adapts to non-Gaussian real noise, and supports stochastic refinement through SRTNet’s enhance-and-refine paradigm.SRTNet combines deterministic and stochastic modules and includes theoretical and experimental support for its effectiveness.Source: §3.8.1 · pp. 28–29

4.1 Supervised Learning

• Supervised representation learning optimizes labeled input-output mappings; CNNs learn speech features from spectrograms, whereas RNNs can extract representations directly from raw signals.For speaker recognition, GE2E and prototypical networks outperform other models on the VoxCeleb1 test set.Source: §4.1 · p. 29

4.1.1 Deep speaker representations

• Deep speaker embeddings support verification, identification, diarization, voice conversion, multi-speaker TTS, and speaker adaptation, with d-vectors and x-vectors widely adopted [552, 506, 507].d-vectors average frame-level embeddings, while x-vectors use temporal modeling and statistical pooling to produce segment-level speaker representations; augmentation improves x-vector performance.Source: §4.1.1 · pp. 30–31

4.2 Unsupervised learning

• Unsupervised representation learning pre-trains on abundant unlabeled data before fine-tuning or extracting representations for tasks with limited data.Probabilistic latent variable models capture relationships between observed and unobserved speech factors, with VAEs commonly used for speech processing.Source: §4.2 · p. 32

4.3 Semi-supervised Learning

• Semi-supervised learning combines supervised and unsupervised loss terms, leveraging unlabeled data to improve generalization and downstream performance when annotations are scarce.It is increasingly applied to ASR, TTS, speech quality assessment, and end-to-end speech translation, including extensions to low-resource languages [34] [658].Source: §4.3 · pp. 33–34

4.4 Self-supervised representation learning (SSRL)

• Self-supervised representation learning generates pseudo-labels from unlabeled speech, reducing annotation requirements while increasingly approaching or surpassing fully supervised methods.Pretext tasks enable robust feature learning for downstream speech-processing tasks without relying heavily on human-annotated datasets.Source: §4.4 · p. 34

4.4.1 Generative Models

• Generative and contrastive SSRL learn speech representations through reconstruction, autoregressive prediction, masking, or similarity objectives; Wav2Vec 2.0 pre-trains on unlabeled audio before task-specific fine-tuning [26].Autoencoders, VAEs, VQ-VAEs, WaveNet, masked reconstruction, and contrastive methods provide complementary mechanisms for extracting contextual or discrete representations.Source: §4.4.1 · pp. 35–36, §4.4.2 · p. 36

4.4.3 Predictive Models

• Predictive SSRL methods address continuous speech through discrete codes or masked-region prediction, with Discrete BERT reaching 25% WER using only 10 minutes of fine-tuning.HuBERT uses offline clustering and masked prediction, whereas TERA reconstructs acoustically altered frames across time, frequency, and task dimensions.Source: §4.4.3 · p. 38
• WavLM currently holds the top position on the SUPERB leaderboard, which evaluates the reusability of speech representations across downstream tasks.WavLM and related models enhance speaker representations through HuBERT-style training and data augmentation during pre-training.Source: §4.4.3 · pp. 38–39

5 Speech Processing Tasks

• ASR evaluation depends on standardized datasets and task-specific metrics, with Tables 5 and 6 summarizing datasets, applications, score ranges, and commonly used measures.The reviewed datasets include Common Voice, LibriSpeech, VoxCeleb, TIMIT, CHiME-5, Speech Commands, Wall Street Journal, and TED-LIUM [17] [92] [33] [589].Source: §5 · p. 39, Table 5 · p. 40, Table 6 · p. 41

5.1.1 Task Description

• ASR converts speech into text or commands through acoustic-feature analysis, acoustic and language modeling, and integration of their outputs into a transcription.Deep learning techniques have been adopted to enhance ASR accuracy [26] [443].Source: §5.1.1 · pp. 40–41
• End-to-end ASR replaces hand-designed HMM and GMM components with DNNs that learn acoustic and linguistic representations directly from speech.Attention-based encoder-decoder models map audio to hidden representations and selectively focus on input regions during decoding.Source: §5.1.1 · p. 41

5.1.3 Models

• RNNs model long-term temporal dependencies and variable-length audio, supporting end-to-end recognition and low-latency applications such as mobile and smart-home speech recognition [117].CTC trains variable-length input-output mappings without explicit alignment, while transfer learning can improve end-to-end ASR performance [106].Source: §5.1.1 · p. 41, §5.1.3 · pp. 42–43
• Transformer ASR uses architectural changes such as relative positional embeddings, pooling, or trainable convolution layers to address long-sequence degradation and improve optimization stability [672] [383].The initial CNN-enhanced Transformer achieved a 10.9% WER on the Wall Street Journal dataset.Source: §5.1.3 · pp. 42–43
• Whisper consistently outperforms wav2vec2.0 Large across several datasets, while both frameworks represent state-of-the-art ASR approaches.The comparison is summarized in Table 8; Whisper is trained on more than 680,000 hours of web audio and supports multilingual transcription and translation.Source: Table 8 · p. 44, §5.1.3 · p. 45
• Transformer-based ASR has improved performance over RNN-based systems, particularly for long sentences and noisy environments, with Deep Speech 2 achieving state-of-the-art results on several benchmarks.Deep Speech 2 combines CNN feature extraction, recurrent temporal modeling, and a Transformer decoder; reported benchmarks include LibriSpeech, VoxForge, and WSJeval92.Source: §5.1.3 · pp. 42, 44
• Conformer, USM, QuartzNet, Citrinet, pretrained language models, Vosk, and large-scale self-supervised or supervised pretraining extend ASR across languages and deployment conditions.USM uses 2 billion parameters and training data covering more than 12 million hours of speech, while Conformer-15 reportedly produces up to 43% fewer errors on noisy data.Source: §5.1.3 · pp. 44–45

5.2.1 Task Description; 5.2.3 Models

• Neural TTS systems use autoregressive or non-autoregressive architectures, with CNNs, RNNs, transformers, diffusion models, and vocoders converting linguistic features into speech.Transformer TTS was 4.25 times faster than Tacotron 2 with similar MOS, while FastSpeech generates mel-spectrograms in parallel and FastSpeech 2 conditions on energy, pitch, and duration.Source: §5.2.1 · p. 45, §5.2.3 · pp. 47–48

5.2.2 Datasets

• LJ Speech provides about 24 hours of single-speaker recordings with transcripts and benchmarks models including Tacotron, WaveNet, and DeepVoice.CMU Arctic, L2 Arctic, LibriSpeech, VoxCeleb, TIMIT, and Common Voice also support training and evaluation of text-to-speech models.Source: §5.2.2 · p. 46

5.2.4 Alignment

• EfficientTTS outperforms Tacotron 2 and Glow-TTS in speech quality, training efficiency, and synthesis speed while using multi-head attention for alignment and fewer parameters.Glow-TTS provides an order-of-magnitude synthesis speed-up over Tacotron 2 with comparable speech quality and requires no external aligner.Source: §5.2.4 · pp. 49–50

5.2.5 Speech Resynthesis

• Speech resynthesis uses self-supervised discrete representations to disentangle content, prosody, and speaker identity for controlled generation in enhancement, conversion, recognition, and synthesis.GSLM combines an unsupervised discrete speech encoder, unit-based generative language model, and speech decoder.Source: §5.2.5 · p. 50

5.2.6 Voice Conversion

• Voice conversion preserves linguistic content while changing speaker identity, but ASR–TTS pipelines overlook prosody; transformer, target-dependent prosody, CycleGAN, and VAE methods address this limitation.VTN uses a transformer with TTS pre-training, while CycleGAN-VC requires no extra data, modules, or alignment procedure.Source: §5.2.6 · pp. 50–51

5.2.7 Vocoders

• Neural vocoders include autoregressive, GAN, diffusion, and flow-based models; GAN vocoders can synthesize mel-conditioned audio several hundred times faster than real time on one GPU.Universal neural vocoding remains challenging, with limited success despite proposed WaveRNN, flow-based, and GAN-based approaches and promising scaling trends.Source: §5.2.7 · pp. 51–52

5.2.8 Controllable Speech Synthesis; 5.2.9 Disentangling and Transferring

• Controllable synthesis targets pitch, speed, emotion, and prosody, while disentangling methods separate content, style, speaker information, rhythm, pitch, timbre, and noise for greater control.Approaches use adversarial or collaborative games, VAE-based adversarial training, bottleneck reconstructions, and frame-level noise modeling.Source: §5.2.8 · p. 52, §5.2.9 · pp. 52–53

5.2.10 Robustness

• TTS robustness is challenged by unseen, atypical, extra-long, and noisy inputs; robust transformers, segment-level modeling, hybrid recurrent-transformer models, and diffusion-based prosody extraction improve generalization and fidelity.RobuTrans improves MOS, a segment-level s-Transformer remains robust for extra-long sentences, and a local recurrent-transformer model outperforms a transformer alone on a 20-hour Mandarin corpus.Source: §5.2.10 · p. 53

5.2.11 Low-Resource Neural Speech Synthesis

• Insufficient paired text and speech data limits high-quality TTS development and leaves many languages unsupported by commercial speech services.Researchers have therefore developed TTS systems for low-data scenarios [538].Source: §5.2.11 · p. 53
• Semi-supervised synthesis uses unpaired training data, while cascading pretrained ASR, MT, and TTS models increases data from unlabelled speech [328] [394].These techniques target improved efficiency in low-resource and zero-shot TTS systems.Source: §5.2.11 · pp. 53–54
• Crowdsourced acoustic-data collection supports TTS for low-resource languages, and zero-shot style transfer generates out-of-domain speech with new speaker identity, emotion, or prosody [50].The style-transfer approach produces speech samples exhibiting distinctive styles.Source: §5.2.11 · p. 54

5.3.1 Task Description

• Speaker identification determines an unknown speaker’s identity, whereas speaker verification tests whether a speaker matches a claimed identity using embedding-based comparisons.Deep learning models, including CNNs, RNNs, and combinations, support verification, identification, diarization, and robust recognition [247].Source: §5.3.1 · p. 54

5.3.2 Dataset

• VoxCeleb, TIMIT, SITW, RSR2015, RedDots, VOICES, CN-CELEB, BookTubeSpeech, Hi-MIA, FFSVC20, DIHARD, and LibriSpeech provide datasets or benchmarks spanning speaker recognition, verification, diarization, and far-field robustness.VoxCeleb is collected from public media through automated visual and active-speaker verification pipelines [92], while SITW and RSR2015 support text-independent and realistic-environment evaluation.Source: §5.3.2 · pp. 54–55

5.3.3 Models

• x-vector systems improved speaker verification through ResNet residual connections and hybrid TDNN-LSTM architectures with multi-level pooling.Residual connections improve embeddings, accelerate back-propagation convergence, and mitigate vanishing gradients [634].Source: §5.3.3 · p. 55
• ECAPA-TDNN enhances x-vector architectures with Res2Net modules, Squeeze-and-Excitation blocks, and channel-attention propagation, achieving promising speaker-verification results [108].The architecture uses a multiscale channel-adaptive module and is presented as an effective solution for speaker-verification challenges.Source: §5.3.3 · pp. 55–56
• Transformer-based methods use self-attention to obtain discriminative utterance-level speaker features, with TESA outperforming PLDA on the same dataset [109].A separate approach retains the original Transformer structure while using serialized multi-layer, multi-head attention to produce fixed-dimensional speaker representations [675].Source: §5.3.3 · p. 56

5.4.1 Task Description

• Speaker diarization systems extract acoustic features, segment speech, generate speaker embeddings, cluster or classify speakers, and re-segment boundaries to refine assignments.Acoustic features include pitch, energy, and MFCCs; CNNs or RNNs can transform them into low-dimensional speaker representations, while clustering commonly uses k-means and classification can use SVMs or MLPs.Source: §5.4.1 · pp. 56–57, Figure 16 · p. 57

5.4.2 Dataset

• Evaluation resources span telephone, meeting, broadcast, and noisy conversational speech, including CALLHOME, DIHARD, AMI, VoxConverse, LibriCSS, RT-03F, and CHiME-5/6.They include conditions such as overlapping speech, background noise, distant or reverberant microphones, and varied overlap ratios, with metrics including DER and JER.Source: §5.4.2 · pp. 57, 59

5.4.3 Models

• Segmentation choices trade off boundary accuracy, speaker-information duration, and labeling requirements across speaker-change detection, uniform segmentation, ASR-based segmentation, and supervised turn detection.End-to-end neural diarization instead directly predicts joint speaker activities with self-attention, avoiding clustering and supporting overlapped speech handling.Source: §5.4.3 · pp. 59–60
• A VBx diarization approach combining Bayesian HMM clustering of x-vectors with a ResNet101 x-vector extractor achieves superior results on CALLHOME, AMI, and DIHARD II.Embedding-based methods also use attention, contextual similarity modeling, and time-depth separable convolutions for speaker identification and clustering.Source: §5.4.3 · p. 60

5.5.1 Task Description

• End-to-end speech translation uses an audio encoder and text decoder to avoid the error propagation and latency of cascaded ASR–MT systems [62] [166] [478] [639], while accent and dialect variation can reduce accuracy.Real-time translation can preserve tone, intonation, and rhythm, supporting more natural communication, but integrating recognition, language processing, and translation remains intricate.Source: §5.5.1 · p. 61

5.5.2 Dataset

• Multilingual resources such as MuST-C, IWSLT [481], CoVoST 2 [564], TED-LIUM, Common Voice, VoxForge9, LibriSpeech, and How2 support speech-translation model development across many languages.The datasets provide combinations of speech, transcripts, and—in How2—video and image data for speech recognition, speech-to-text, speech-to-speech, and multimodal translation.Source: §5.5.2 · p. 61

5.5.3 Models

• End-to-end models outperformed cascade models in 2017 [3], while Translatotron demonstrated direct speech-to-speech translation without intermediate text, although its BLEU results lagged behind a conventional cascade.Later systems incorporated architectures, CTC loss, pretrained Wav2Vec 2.0 and mBART components, coupling modules, and selective fine-tuning of 20% of parameters [27] [622].Source: §5.5.3 · pp. 61–62
• Although transcript-based pretraining is standard [563] [603] [639] [214] [660], Zhang et al. trained from scratch using only speech-translation pairs and outperformed previous studies across four benchmarks covering 23 languages.The work also modeled acoustic features directly from raw speech to simplify inductive biases and improve speech description.Source: §5.5.3 · p. 62

5.6.1 Task Description

• Speech enhancement removes ambient noise and signal distortion, often as preprocessing to improve ASR performance in noisy environments.Ambient noise can reduce speech-recognition accuracy by making spoken-language signals difficult to interpret.Source: §5.6.1 · p. 62

5.6.2 Dataset

• AISHELL-4 provides 120 hours of authentic Mandarin conference recordings from 211 sessions with 4–8 speakers and an 8-channel microphone array [144].Its realistic acoustics and varied speech qualities support multi-speaker processing, including diarization and speech recognition.Source: §5.6.2 · p. 62
• The DNS Challenge dataset contains over 10,000 hours of noisy speech and over 1,000 hours of clean speech, while VCTK provides clean and noisy recordings from 109 speakers.These datasets support deep-learning training with realistic acoustics, natural speech, and paired noisy-clean signals; the DNS dataset is associated with [457].Source: §5.6.2 · p. 62

5.6.3 Models

• Neural approaches, including fully connected, denoising autoencoder, CNN, LSTM, and Transformer models, outperform classical methods and handle diverse noisy conditions.Classical methods include spectral subtraction, Wiener and Kalman filtering, MMSE estimation, comb filtering, subspace methods, and phase spectrum compensation, but spectral-domain methods are limited to stationary or quasi-stationary noise.Source: §5.6.3 · p. 63
• Diffusion models such as CDiffuSE, StoRM [298], and cold diffusion [623] target high-quality speech recovery under non-Gaussian noise, adverse conditions, or arbitrary degradations.CDiffuSE conditions diffusion and reverse processing on noisy speech, StoRM removes vocalizing and breathing artifacts, and cold diffusion iteratively restores clean speech.Source: §5.6.3 · p. 63

5.7.1 Task Description

• Audio super-resolution predicts missing high-resolution components of low-resolution signals, commonly using fixed scale factors and regression in the time or frequency domain.Deep neural networks are trained on paired low- and high-quality audio, then predict missing samples during testing.Source: §5.7.1 · p. 64

5.7.2 Datasets

• Audio super-resolution is evaluated on music, environmental sounds, and speech using MUSDB18, UrbanSound8K, VoiceBank, LibriSpeech, and TED-LIUM.These datasets range from more than 150 songs and over 8,000 environmental sound files to more than 1,000 hours of speech and over 140 hours of TED recordings.Source: §5.7.2 · p. 64

5.7.3 Models

• Recent models combine convolution with self-attention, separate domain adaptation from resampling, or integrate time- and frequency-domain information for audio super-resolution [624].Rakotonirina uses Attention-based Feature-Wise Linear Modulation, while Yoneyama et al. [624] jointly optimize domain adaptation and resampling in CycleGAN for unpaired signals.Source: §5.7.3 · p. 64
• NU-Wave 1 and 2 diffusion models generate 48kHz waveforms from coarse 16kHz or 24kHz inputs, highlighting diffusion as a promising audio-upsampling direction [290] [168].Lee and Han [290] and Han and Lee [168] propose these diffusion probabilistic models for neural audio upsampling.Source: §5.7.3 · p. 64

5.8.1 Task Description

• VAD recognizes speech presence or absence, helping speech-controlled applications operate reliably by distinguishing human speech from noise.These applications include hands-free home-device control, telephony, and safer in-vehicle infotainment use.Source: §5.8.1 · p. 65

5.8.2 Datasets

• VAD evaluation uses datasets spanning clean, noisy, real-world, and speaker-diverse conditions, including TIMIT, CHiME-5, AURORA-4, DEMAND, and VoxCeleb.TIMIT provides 6,300 utterances from 630 speakers; CHiME-5 contains multichannel recordings of 20 speakers; AURORA-4 has over 10,000 noisy utterances; DEMAND includes over 1,200 noise signals; and VoxCeleb contains over 100,000 utterances from more than 6,000 speakers.Source: §5.8.2 · p. 65

5.8.3 Models

• Deep learning improves VAD in noisy environments, with NAS-VAD and self-attentive VAD outperforming other models on reported metrics and TIMIT, respectively [462].NAS-VAD uses neural architecture search to reduce manual network design, while self-attentive VAD captures long-term dependencies through self-attention.Source: §5.8.3 · p. 65

5.9.1 Task Description

• Speech quality assessment objectively measures intelligibility and comprehensibility; MOS is the most common reliable human-judgment method, despite evaluation costs and limited scalability.Assessment supports real-world applications including ASR, speech enhancement, and VoIP.Source: §5.9.1 · p. 65

5.9.2 Datasets

• Evaluation uses clean and artificially degraded speech from TIMIT, NOIZEUS, and ETSI Aurora, with DNS Challenge [457] and FSDK50 supporting training and noise degradation.The datasets include distortions such as noise, acoustic echo, and other degradation types.Source: §5.9.2 · p. 66

5.9.3 Models

• Neural speech-quality estimators avoid clean references and outperform traditional approaches, but limited robustness and generalization constrain real-world use [119].DESQ predicts perceived quality from phoneme posterior probabilities produced by a deep neural network.Source: §5.9.3 · p. 66
• PESQ and POLQA generally require corresponding clean references, making them unsuitable when ground-truth clean speech is unavailable.This limitation motivates reference-free neural approaches for real-world quality assessment.Source: §5.9.3 · p. 66
• NORESQA estimates quality with non-matching references, introducing NORESQA-score based on SI-SDR and NORESQA-MOS for MOS assessment [368].Its framework reflects the human ability to assess speech quality even when reference content does not match.Source: §5.9.3 · p. 66

5.10.1 Task Description

• Speech separation isolates speech, music, and noise from mixed audio, increasingly treating the cocktail-party problem as supervised learning.Deep neural networks improve performance by learning complex relationships between input features and output sources.Source: §5.10.1 · pp. 66–67

5.10.2 Datasets

• Benchmarks range from controlled two-speaker mixtures in WSJ0-2mix and TIMIT-2mix to realistic multi-speaker, noisy, and music-source datasets including CHiME-4, LibriMix, and MUSDB18.WSJ0-2mix contains 30,000 training and 5,000 test mixtures, while LibriMix contains up to four speakers with 100,000 training and 1,000 test mixtures.Source: §5.10.2 · p. 67

5.10.3 Models

• Models combine clustering, masking, CNNs, RNNs, attention, and diffusion; Separate And Diffuse [357] is reported as state-of-the-art across benchmarks, while architectures trade accuracy against computational cost.Deep Clustering++ [181], Chimera++ [587], TasNet v2, Conv-TasNet, Dual-path RNN, SepFormer, MossFormer [663], and diffusion models are evaluated across separation settings; Table 11 compares methods using SI-SDRi.Source: §5.10.3 · pp. 67–68, Table 11 · p. 68

5.11.1 Task Description

• SLU systems identify an utterance’s domain, speaker intent, and relevant slots to support natural human–machine communication.Typical subtasks include keyword spotting, intent classification, and slot filling; for example, a weather request requires identifying the weather domain, information-seeking intent, and location slot.Source: §5.11.1 · p. 69

5.11.2 Dataset

• SLU research uses varied datasets for intent classification and slot filling, including ATIS, SNIPS [101], Fluent Speech Commands [350], and MIT-Restaurant and MIT-Movie.ATIS contains spoken airline-travel queries annotated with intent and slot labels.Source: §5.11.2 · p. 70

5.11.3 Models

• Deep-learning SLU models include CNNs, transformers [37], Wav2Keyword, Keyword Transformer, DIET [48], prompt tuning [59], and wav2vec2.0 adapters [315].KWT reports 98.6% and 97.7% accuracy on 12- and 35-word keyword tasks, respectively, while DIET outperforms fine-tuned BERT and trains six times faster; prompt tuning achieves comparable results with fewer trainable parameters than full fine-tuning.Source: §5.11.3 · p. 70
• Performance analyses on Google Speech Commands and ATIS benchmark datasets, summarized in Table 12, reveal strengths and limitations across keyword-spotting and slot-filling approaches.These findings motivate continued use of advances in deep learning and speech recognition to expand SLU capabilities.Source: Table 12 · p. 71, §5.11.3 · p. 71
• Real-world SLU remains difficult because diverse accents, dialects, noisy and unstructured speech, and multi-action commands complicate accurate comprehension, especially for house-service robots.Current research therefore focuses on more robust and accurate speech comprehension across diverse real-life scenarios.Source: §5.11.3 · p. 71

5.12 Audio/visual multimodal speech processing

• Visual cues help distinguish acoustically similar sounds, support lip-reading for hearing-impaired listeners, and improve communication in noisy environments.These benefits motivate audiovisual processing and assistive technologies for challenging listening conditions.Source: §5.12 · p. 71

5.12.1 Task Description

• Audiovisual processing covers lip-reading, speech separation, and talking face generation using complementary facial, acoustic, and speech information.Lip-reading maps silent facial videos to speech audio or characters; separation isolates speakers from mixtures; talking face generation synchronizes facial motion with speech.Source: §5.12.1 · p. 71

5.12.2 Datasets

• Key datasets include TCD-TIMID, LRW, LRW-1000, LRS2 and LRS3 10, and GRID for audiovisual recognition, separation, and synthesis.TCD-TIMID contains 600 sentences from 60 participants, while LRW covers 500 words from more than 1000 speakers.Source: §5.12.2 · pp. 72–73

5.12.3 Models

• Deep multimodal models combine CNNs, LSTMs, attention, self-supervised learning, conditional VAEs, cross-modal embeddings, and adversarial generation across speech tasks.Son et al.’s hybrid model generates characters from lip videos and audio; Ephrat et al. [130] introduced CNN-based separation and AVspeech; DAVS and ATVGnet generate talking faces.Source: §5.12.3 · pp. 73–74

6.1.1 Task Description

• Domain adaptation reduces source-target performance gaps by aligning distributions and supports speech recognition, speaker verification [76], and speech synthesis.Reviewed methods include reconstruction-based adaptation and domain-adversarial networks that learn domain-invariant features using gradient reversal.Source: §6.1.1 · p. 74, §6.1.2 · p. 74

6.1.2 Models

• Speech-recognition adaptation uses prompt-tuning [112], adapter modules, and Machine Speech Chain synthetic speech generation to improve target-domain recognition with limited new-domain data.Adversarial domain training also learns speaker-independent features insensitive to recording-environment variation.Source: §6.1.2 · pp. 74–75
• Speaker- and domain-adaptive TTS, including AdaSpeech [66] [599], is especially important in low-resource settings where sufficient training data are difficult to collect.Domain adaptation techniques are also applied to speaker recognition and verification [76] [306].Source: §6.1.2 · pp. 74–75

6.2.2 Models

• MetaASR significantly outperformed state-of-the-art multitask pretraining across target languages, while proposed speaker-verification methods consistently improved an existing meta-learning framework [279].Meta-learning formulates languages as tasks and learns initialization parameters from multiple pretraining languages.Source: §6.2.2 · p. 75
• Meta-TTS synthesized high-speaker-similarity speech from a few enrollment samples with fewer adaptation steps, and language-agnostic meta-learning was applied to low-resource TTS [358].Meta-learning has also been used for low-resource ASR, speaker verification, and domain generalization for speaker recognition.Source: §6.2.2 · p. 75, §6.2.1 · p. 75

6.3 Parameter-Efficient Transfer Learning

• Parameter-efficient transfer learning optimizes only a fraction of pretrained-model parameters to reduce the computational cost of downstream adaptation without sacrificing performance.Knowledge distillation trains smaller models from larger-model outputs or hidden representations, while pruning, low-rank factorization, and quantization reduce model size and energy use.Source: §6.3 · p. 76, §6.3.2 · p. 79, §6.3.3 · p. 79

6.3.1 Adapters

• Adapter-based methods performed comparably to fine-tuning on SURE benchmark tasks, with no significant advantage for any particular adapter type.Adapters freeze pretrained parameters and optimize small modules, including bottleneck, prefix-tuning, LoRA, and convolutional adapters.Source: §6.3.1 · pp. 76, 79

6.3.3 Model Compression

• Model compression made wav2vec2.0 twice as fast and energy efficient and 4.8 times smaller, with a 7% WER increase; without knowledge distillation, it was 3.6 times smaller with 0.1% WER degradation.Low-rank transformers also halved model size and increased inference speed 1.35 times, while iterative pruning improved results over fine-tuned wav2vec2.0.Source: §6.3.3 · p. 79

Large Speech Models

• Larger speech models and datasets, supported by adversarial training, could improve ASR and produce more natural TTS with human-like prosody.Adversarial discriminators distinguish machine-generated from reference speech to improve realism and synthesis quality.Source: §7 · p. 80

Multilingual Models

• Multilingual self-supervised models such as XLS-R address low-resource speech recognition by scaling beyond 2 billion parameters across 128 languages.XLS-R surpasses its predecessor in language coverage, and larger multilingual models are expected to outperform single-language models.Source: §7 · p. 80

Multimodal Speech Models

• Large multimodal speech models could unify speech with text, images, and other modalities for comprehensive multimodal understanding and generation.GPT-4 and Kosmos-I demonstrate joint image-text processing, motivating unified architectures for speech and other modalities.Source: §7 · p. 80

In-Context Learning

• Mixed-modality models could enable in-context learning for speech tasks by specifying tasks and examples directly within the input.InstructGPT, FLAN-T5 [90], and LLaMA illustrate the effectiveness of context-driven learning in large language models.Source: §7 · p. 80

Controllable Speech Generation

• Controllable TTS could use in-context text guidance to personalize tone, accent, age, gender, and other synthesized-speech attributes.This approach aims to provide users with fine-grained customization of speech synthesis.Source: §7 · pp. 80–81

Parameter-efficient Learning

• Parameter-efficient adaptation and model compression could reduce update costs, computation, and memory while preserving performance in large mixed-modality models.Specialized adapters are needed for minimal parameter updates, while compression research [280] highlights task-specific sparsity.Source: §7 · p. 81

Explainability

• Explainability remains unresolved for large speech networks, motivating analysis of learning dynamics and architectures with built-in inductive biases.NTK-based analyses of text models have not yet been extended to speech, while brain-inspired architectures may improve interpretability.Source: §7 · p. 81

Neuroscience-inspired Architectures

• Neuroscience-inspired models align speech-model layers with the brain’s functional hierarchy and can achieve performance comparable to state-of-the-art systems.These models aim to improve understanding of speech perception and production while advancing performance.Source: §7 · p. 81