PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Deep Learning Techniques for Speech Processing

Ambuj Mehrish, Navonil Majumder, Rishabh Bhardwaj, Rada Mihalcea, Soujanya Poria

arXiv:2305.00359v3eess.AS

TL;DR

Speech processing needs models that can handle diverse tasks and rapidly changing deep-learning techniques, while the field still faces data, interpretability, and robustness challenges. This review synthesizes architectures, representations, tasks, transfer-learning methods, and future directions, reporting advances across speech applications and highlighting both specialized and scalable approaches. Its supported conclusions point toward continued exploration of architectures such as conformers, self-supervised multilingual models, and efficient or domain-specific systems.

  • Problem

    Speech processing lacks a current broad review covering its diverse tasks, rapidly evolving deep-learning techniques, applications, and associated challenges.

  • Method

    The paper comprehensively reviews speech representations, traditional models, deep-learning architectures, speech-processing tasks, datasets, transfer learning, and future research directions.

  • Results

    The review reports substantial advances across speech recognition, synthesis, speaker recognition, and related tasks, including strong results from conformer and self-supervised multilingual approaches.

  • Takeaways & Limitations

    Deep learning provides a broad set of architectures for speech processing, while model choice can depend on whether applications require general multilingual coverage or domain-specific performance.

  • Takeaways & Limitations

    Universal neural vocoding has achieved limited success to date, and deep reinforcement learning requires substantial data, computational resources, and careful reward-function selection.

Abstract

from arXiv · show

The field of speech processing has undergone a transformative shift with the advent of deep learning. The use of multiple processing layers has enabled the creation of models capable of extracting intricate features from speech data. This development has paved the way for unparalleled advancements in speech recognition, text-to-speech synthesis, automatic speech recognition, and emotion recognition, propelling the performance of these tasks to unprecedented heights. The power of deep learning techniques has opened up new avenues for research and innovation in the field of speech processing, with far-reaching implications for a range of industries and applications. This review paper provides a comprehensive overview of the key deep learning models and their applications in speech-processing tasks. We begin by tracing the evolution of speech processing research, from early approaches, such as MFCC and HMM, to more recent advances in deep learning architectures, such as CNNs, RNNs, transformers, conformers, and diffusion models. We categorize the approaches and compare their strengths and weaknesses for solving speech-processing tasks. Furthermore, we extensively cover various speech-processing tasks, datasets, and benchmarks used in the literature and describe how different deep-learning networks have been utilized to tackle these tasks. Additionally, we discuss the challenges and future directions of deep learning in speech processing, including the need for more parameter-efficient, interpretable models and the potential of deep learning for multimodal speech processing. By examining the field's evolution, comparing and contrasting different approaches, and highlighting future directions and challenges, we hope to inspire further research in this exciting and rapidly advancing field.

1 Introduction

Speech processing has evolved from traditional statistical and feature-engineering approaches toward deep learning architectures that learn representations from speech signals. This review surveys that evolution, the broad range of applications, and challenges shaping future research.

  • Motivation: Speech processing covers analysis, synthesis, and recognition tasks, including automatic speech recognition, speaker recognition, and speech synthesis.Its applications span telecommunications, healthcare, and entertainment.
  • Deep Learning Advances: Deep learning learns meaningful features from raw speech signals, reducing reliance on manual feature engineering and improving performance in noise, accents, and dialects.Deep neural networks can adapt and generalize more effectively across challenging conditions.
  • Challenges and Future Directions: The review identifies substantial labeled-data requirements, limited interpretability, and robustness to environmental conditions as continuing challenges.These challenges motivate examination of current approaches and future research directions.
  • Scope and Evolution: The review traces speech processing from early representation learning and HMM-based modeling to DNNs, CNNs, RNNs, Transformers, GNNs, and diffusion models.It also examines representation learning techniques tailored to speech processing.
  • Scope and Evolution: The paper surveys speech-processing tasks, datasets, and research that applies deep learning techniques to recognition, speaker identification, synthesis, and related applications.It aims to provide a broad view rather than focusing on a single speech-processing task.

2 Background

Speech processing represents and models speech signals using features derived in the time or frequency domain and traditional statistical models. These foundations support analysis, recognition, and synthesis before the discussion of deep neural architectures.

  • Speech Signals: A speech signal is a sound signal consisting of human-generated air-pressure variations used for spoken communication.Speech signals are commonly digitized into numerical values for processing.
  • Speech Features: Speech features are numerical representations used for speech analysis, recognition, and synthesis, broadly divided into time-domain and frequency-domain features.The two categories capture different information and may be used together.
  • Speech Features: Time-domain features derive from amplitude over time; energy summarizes signal strength within a window and supports recognition, segmentation, and speaker diarization.These features are simple to compute and can suit real-time processing.
  • Speech Features: Frequency-domain representations such as Mel spectrograms and MFCCs are widely used because they are more robust to noise than temporal sound variations.Mel spectrograms map short-term power spectra onto a perceptual mel scale.
  • Traditional Models: Traditional shallow representation-learning methods extract features through operations such as Fourier transforms, wavelet transforms, and linear predictive coding before classification or regression.GMMs model speech-feature distributions, while HMMs model temporal dependencies and likely sequences of speech sounds.

3 Deep Learning Architectures and Their Applications in Speech Processing Tasks

Deep learning architectures support speech-processing applications through learned representations, temporal modeling, attention, multimodal integration, and generative modeling. Across the surveyed approaches, CNNs, RNNs, Transformers, Conformers, and diffusion models achieve strong results while retaining distinct efficiency, dependency-modeling, and data requirements.

  • CNNs: CNNs support acoustic modeling, raw-speech representation learning, speaker recognition, speech enhancement, and text-to-speech synthesis.Examples include wav2vec 2.0, VGGVox, Deep Noise Suppression architectures, and Tacotron2.
  • TCNNs: TCNNs provide a convolutional alternative to recurrent architectures and demonstrate comparable performance to LSTMs and GRUs across speech-related tasks.Their causal, dilated design preserves temporal ordering while expanding the receptive field for longer dependencies.
  • Transformers: Transformers model long-range dependencies and have achieved state-of-the-art performance across ASR, speech translation, and TTS applications.Transformer TTS also improves training and inference efficiency over RNN-based models, with training reported as about 4.25 times faster.
  • Representation Learning: Speech representation frameworks such as wav2vec enable transfer across languages and domains, including strong results with limited labeled data.w2v-BERT combines contrastive learning with masked language modeling, while XLS-R uses multilingual augmentation and contrastive learning.
  • Conformers: Conformers combine convolution with self-attention to model local and global dependencies efficiently in speech sequences.They have been applied to speech recognition, speaker recognition, and language identification, with studies reporting performance beyond previous state-of-the-art models.

4 Speech Representation Learning

Speech representation learning has evolved from supervised embeddings to unsupervised, semi-supervised, and self-supervised methods for extracting useful speech characteristics. The reviewed approaches span speaker representations, probabilistic latent-variable models, and predictive models trained with limited annotation.

  • Speech representation learning extracts characteristics useful for speaker identification, speech recognition, and emotion recognition.
  • The review covers supervised, unsupervised, and self-supervised representation-learning techniques, including autoencoders, GANs, and predictive frameworks.
  • Deep speaker representations: d-vector models perform frame-level speaker classification, whereas x-vector models aggregate frame embeddings into utterance-level speaker representations.
  • Unsupervised learning: Probabilistic latent-variable models learn representations of relationships between observed and unobserved speech variables without explicit labels.
  • Semi-supervised learning: Semi-supervised learning combines labeled and unlabeled data by optimizing supervised and unsupervised objectives.
  • Predictive Models: Discrete BERT converts continuous speech into discrete codes and uses masked prediction for self-supervised representation learning.

5 Speech Processing Tasks

Speech-processing research spans diverse tasks and relies on task-appropriate datasets and evaluation metrics. Deep-learning ASR has evolved from recurrent architectures toward Transformers and newer pretrained frameworks, improving performance in long and noisy speech settings.

  • Standardized datasets and task-specific metrics enable objective comparison of speech-processing systems and help identify areas for improvement.
  • Speech processing covers tasks including ASR, speaker recognition, speech synthesis, phoneme recognition, emotion recognition, intent classification, voice conversion, translation, and separation.
  • End-to-end ASR models use DNNs to map speech signals toward text without the hand-designed HMM and GMM components required by traditional pipelines.
  • 5.1.3 Models: RNN-based ASR captures temporal dependencies and variable-length audio, while Transformer-based approaches address limitations in long-range modeling.
  • 5.1.3 Models: Transformer-based ASR improved performance for long sentences and noisy environments, with hybrid RNN–Transformer systems achieving state-of-the-art results on several benchmarks.
  • 5.1.3 Models: Wav2Vec 2.0 learns speech representations from unlabeled audio before fine-tuning, whereas Whisper uses more than 680,000 hours of supervised web audio and supports multilingual transcription and translation.

5.2 Neural Speech Synthesis

Neural speech synthesis uses deep-learning models to generate speech from text or other inputs, with evaluation, datasets, and architectures spanning autoregressive, transformer, flow, VAE, and diffusion approaches.

  • Overview: Neural speech synthesis generates speech from text or other inputs using CNNs, RNNs, transformers, and diffusion models.Applications include assistive technologies for people unable to communicate because of neurological impairments.
  • Evaluation: Synthesized speech is evaluated with statistical tests alongside Mean Opinion Score evaluations to measure similarity to natural speech.The statistical tests provide an objective complement to traditional subjective assessments.
  • Transformer models: Transformer-based TTS was 4.25 times faster than Tacotron 2 while achieving similar MOS performance.The model used multi-head attention to generate mel-spectrograms from phoneme sequences and WaveNet as a vocoder.
  • Non-autoregressive models: FastSpeech uses parallel mel-spectrogram generation and a duration-predictor-based length regulator to address sequence-length mismatch.FastSpeech 2 additionally conditions synthesis on energy, pitch, and more accurate duration, and trains directly on ground-truth targets.
  • Diffusion models: Diffusion models generate either raw waveforms, as in WaveGrad and DiffWave, or mel features, as in GradTTS and DiffTTS.
  • Flow-based models: EfficientTTS outperformed Tacotron 2 and Glow-TTS in speech quality, training efficiency, and synthesis speed using multi-head attention and fewer parameters.Glow-TTS itself achieved an order-of-magnitude synthesis speed-up over Tacotron 2 with comparable speech quality.

5.3 Speaker recognition

Speaker recognition covers identification and verification, supported by datasets collected across media, conversational, office, far-field, and noisy environments. Deep neural speaker-embedding systems, especially x-vector extensions, have substantially advanced verification.

  • Task description: Speaker recognition addresses speaker identification and verification using characteristics such as pitch, tone, accent, and other voice features.Applications include security, forensics, biometric authentication, and speaker diarization.
  • Datasets: VoxCeleb uses an automated YouTube pipeline with active-speaker verification and CNN-based facial recognition to confirm speaker identity.
  • Datasets: Speaker-recognition datasets include SITW, RSR2015, RedDots, VOICES, CN-CELEB, BookTubeSpeech, Hi-MIA, FFSVC20, DIHARD, and LibriSpeech.These resources cover office, furnished-room, far-field, challenge, and other recording conditions.
  • Early neural models: The d-vector approach verified short-duration utterances by averaging last-hidden-layer outputs across frames.
  • Deep speaker embeddings: The x-vector system was enhanced with ResNet residual connections, TDNN-LSTM hybrids, and multi-level pooling to capture complementary speaker information.Residual connections also improved convergence and mitigated vanishing gradients.
  • Advanced architectures: ECAPA-TDNN improves x-vector architectures with Res2Net modules, squeeze-and-excitation blocks, and channel-attention propagation and aggregation.The architecture achieved promising speaker-verification results and was used as the backbone for a multiscale channel-adaptive module.

5.4 Speaker Diarization

Speaker diarization determines who spoke when by combining acoustic features, segmentation, embedding extraction, clustering, classification, and resegmentation. Deep-learning systems and benchmark datasets address multi-speaker, far-field, overlapping, and conversational conditions.

  • Task description: Speaker diarization identifies and differentiates speakers in multi-speaker recordings through a pipeline of VAD, segmentation, clustering, and resegmentation.
  • System components: Diarization systems extract pitch, energy, and MFCC features, then use neural networks to produce speaker embeddings for clustering and classification.CNNs and RNNs are among the deep networks used for embedding extraction.
  • Datasets and metrics: CALLHOME provides conversational telephone speech from diverse speakers and is evaluated with diarization error rate, speaker-recognition accuracy, and speech-activity detection accuracy.
  • Datasets and metrics: The AMI database contains over 100 hours of audio and video from 112 office meetings involving four to seven participants.Recordings combine close-talk and room microphones with multiple-camera video and meeting annotations.
  • Segmentation and clustering: Diarization systems use alternative segmentation strategies, including speaker-change, uniform, ASR-based word, and supervised speaker-turn segmentation.Each strategy has distinct benefits and drawbacks for capturing speaker-turn boundaries and sufficient speaker information.
  • Deep-learning advances: A Bayesian HMM clustering x-vector system with a ResNet101 extractor achieved superior results on CALLHOME, AMI, and DIHARD II.Attentive aggregation is another approach that combines multiple frames into speaker embeddings for clustering.

5.5 Speech-to-speech translation

Speech-to-speech translation converts spoken language between languages and can use end-to-end systems instead of cascaded ASR and machine-translation pipelines. Research spans multilingual datasets, direct translation, pretrained components, and parameter-efficient fine-tuning.

  • Task description: Speech-to-text translation converts spoken language into another language’s text, traditionally through cascaded ASR and machine translation.End-to-end methods aim to avoid error propagation and high latency associated with cascaded systems.
  • Motivation: Real-time speech translation can preserve tone, intonation, and rhythm, supporting more natural and fluent communication.
  • Datasets: MuST-C, IWSLT, CoVoST 2, and TED-LIUM are widely used datasets spanning numerous languages and speech-translation tasks.
  • End-to-end models: End-to-end speech translation uses a single sequence-to-sequence model and was reported to outperform cascade models in 2017.Direct speech-to-speech translation was later demonstrated as feasible, though results lagged behind conventional cascade systems.
  • Pretraining and fine-tuning: A 2020 system combined Wav2Vec 2.0, mBART, encoder-decoder coupling modules, and selective fine-tuning of 20% of total parameters.
  • Pretraining and fine-tuning: Encoder or decoder pretraining with speech-recognition or text-translation transcripts has become standard for end-to-end speech translation.The importance of pretraining is less clear when transcripts are unavailable or systems are trained only on speech-translation pairs.

5.6 Speech enhancement

Speech enhancement addresses noise-induced recognition difficulty by improving speech-signal quality. The reviewed literature contrasts classical methods with neural and diffusion-based approaches across realistic datasets and benchmark metrics.

  • Ambient noise can reduce speech-recognition performance, motivating enhancement systems that remove noise and signal distortion before recognition.
  • Datasets: AISHELL-4 contains 211 conference meetings, 4–8 speakers per session, and 120 hours of authentic Mandarin recordings from an 8-channel microphone array.
  • Datasets: The DNS dataset provides over 10,000 hours of noisy speech and over 1,000 hours of clean speech for training speech-enhancement models.
  • Classical methods: Classical enhancement algorithms such as spectral subtraction and Wiener filtering operate in the spectral domain and are restricted to stationary or quasi-stationary noise.
  • Deep-learning methods: Neural approaches, including fully connected networks, denoising autoencoders, CNNs, LSTMs, and Transformers, have outperformed classical approaches across diverse noisy conditions.
  • Diffusion methods: Diffusion models have produced speech-enhancement methods such as CDiffuSE, which incorporates observed noisy-speech characteristics and adapts to non-Gaussian real noises.

5.7 Audio Super Resolution

Audio super-resolution reconstructs missing high-resolution components from low-resolution audio, commonly using deep networks trained on paired low- and high-quality examples. Research spans convolutional, self-attention, time-frequency, and diffusion-based approaches, evaluated with varied datasets.

  • Audio super-resolution predicts missing high-resolution components of low-resolution audio signals, commonly treating audio as discrete data at fixed scale factors.
  • Methods: Deep neural networks for audio super-resolution are trained using paired low- and high-quality audio examples.
  • Datasets: MUSDB18 contains more than 150 songs with separate instrument tracks, while UrbanSound8K contains over 8,000 environmental sound files across 10 categories.
  • Architectures: One proposed architecture combines convolution and self-attention, using Attention-based Feature-Wise Linear Modulation to modulate convolutional activations.
  • Architectures: TFNet models audio super-resolution as regression in the time or frequency domain and combines information from both domains.
  • Architectures: Recent diffusion models introduce additional approaches for neural audio upsampling.

5.8 Voice Activity Detection (VAD)

Voice activity detection supports speech-controlled applications by distinguishing human speech from noise. Deep learning has improved VAD performance in noisy environments, with datasets ranging from phonetically transcribed speech to real-world multichannel recordings.

  • Voice-controlled applications require accurate separation of human speech from noise for uninterrupted operation in homes, telephony, and vehicles.
  • Datasets: TIMIT provides 6,300 phonetically transcribed utterances from 630 speakers, whereas CHiME-5 contains multichannel recordings of 20 speakers in real-world locations.
  • Deep-learning methods: Deep learning has greatly improved VAD performance, particularly in noisy environments.
  • Deep-learning methods: NAS-VAD uses neural architecture search to reduce human effort in network design and reports superior AUC and F1-score performance compared with other models.

5.9 Speech Quality Assessment

Speech quality assessment aims to measure intelligibility and comprehensibility objectively, because human listening evaluations are costly and difficult to scale. Existing reference-based metrics face real-world limitations, motivating neural and non-matching-reference approaches.

  • Speech quality assessment objectively evaluates speech signals, primarily targeting intelligibility and comprehensibility for human listeners.
  • Assessment methods: Mean opinion score is widely used for speech-quality assessment, while human evaluation remains the gold standard but is time-consuming, expensive, and difficult to scale.
  • Datasets: TIMIT provides clean speech and artificially degraded versions, while NOIZEUS supports evaluation with clean speech and versions degraded by various noises.
  • Limitations: PESQ and POLQA mostly require corresponding clean references and therefore fail when ground-truth clean speech is unavailable in real-world scenarios.
  • Non-matching references: NORESQA uses non-matching references and introduces NORESQA-score based on SI-SDR and NORESQA-MOS for estimating a recording’s mean opinion score.

5.10 Speech Separation

Speech separation isolates speech, music, and background noise from mixed audio signals, a challenge addressed by increasingly diverse deep-learning architectures. The reviewed models span clustering, convolutional, recurrent, encoder-decoder, Transformer, and diffusion approaches, with accuracy often balanced against computational cost.

  • Speech separation extracts individual sources from mixtures containing speech, music, and background noise, commonly called the cocktail party problem.It is relevant to phone conversations, meetings, and live events.
  • Deep neural networks improve speech separation by learning complex relationships between input features and output sources.
  • Models: Deep Clustering++ uses latent-space feature clustering, spectral masking, and permutation-invariant training to handle multiple speakers, but has high computational cost.
  • Models: CNN-based TasNet v2 generates a time-frequency mask for each source and achieves state-of-the-art performance, but its computational cost is relatively high.
  • Models: Dual-path RNN captures low- and high-frequency information, while Gated DualPathRNN captures long-term dependencies; both incur high computational cost.
  • Models: DiffSep shows competitive performance on VoiceBank-DEMAND, while Separate And Diffuse represents state-of-the-art performance across various speech separation benchmarks.

5.11 Spoken Language Understanding

Spoken Language Understanding combines speech and language processing to identify domains, intents, and relevant slots in spoken utterances. The review covers its task categories, datasets, deep-learning models, benchmark findings, and continuing challenges from accent, dialect, and linguistic variation.

  • SLU aims to help machines comprehend human speech by identifying an utterance’s domain, speaker intent, and relevant slots.For example, a weather request can be labeled by domain, intent, and location.
  • Data-driven SLU commonly uses deep neural networks because they can handle complex and ambiguous speech data.
  • Tasks: SLU includes Keyword Spotting, Intent Classification, and Slot Filling tasks.
  • Datasets: Speech Commands provides over 100,000 English commands across 35 keywords, while ATIS contains airline-related queries annotated with intent and slot labels.
  • Models: CNNs and Transformers are state-of-the-art model families for keyword spotting; KWT reports 98.6% accuracy on 12-word tasks and 97.7% on 35-word tasks.
  • Challenges: Real-life SLU remains challenging because diverse accents, dialects, and linguistic variations complicate accurate speech comprehension.

5.12 Audio/visual multimodal speech processing

Audiovisual speech processing combines auditory and visual cues because facial articulation can help distinguish acoustically similar sounds. The review covers lip-reading, audiovisual speech separation, datasets, and deep-learning approaches for multimodal speech applications.

  • Audiovisual processing uses auditory and visual information because visual cues can distinguish acoustically similar sounds with different articulatory characteristics.
  • Lip-reading: Lip-reading derives spoken language from silent facial videos, with models producing corresponding speech audio or characters.Applications include videoconferencing and communication in noisy environments.
  • Audiovisual speech separation: Audiovisual speech separation aims to isolate individual speech sources by combining facial and acoustic information in noisy mixtures.
  • Datasets: VoxCeleb and TCD-TIMID are among the datasets used for audiovisual multimodal research, with TCD-TIMID containing 600 sentences spoken by 60 participants.
  • Models: LiRA uses lip-image sequences and audio waveforms for self-supervised pretraining and supports word-level and sentence-level lip-reading.
  • Models: CNNs, conditional VAEs, cross-modal embeddings, and still-image methods have been proposed for audiovisual speech separation.

6 Advanced Transfer Learning Techniques for Speech Processing

Advanced transfer learning techniques adapt speech models across domains, tasks, languages, speakers, and resource conditions while reducing adaptation cost. The review discusses domain adaptation, meta-learning, parameter-efficient tuning, adapters, and compression methods, including their reported benefits and trade-offs.

  • Domain adaptation: Domain adaptation reduces the performance gap between differing source and target domains by minimizing distribution differences.Applications include speech recognition, speaker verification, and text-to-speech adaptation.
  • Domain adaptation: Speech-processing domain adaptation includes reconstruction-based, adversarial, prompt-tuning, adapter-based, and speaker- or domain-specific TTS approaches.
  • Meta-learning: Meta-learning targets faster adaptation by learning parameter initialization, optimization strategies, architectures, or distance metrics.It is particularly advantageous for speech-processing tasks and low-resource settings.
  • Meta-learning: MetaASR significantly outperforms state-of-the-art multitask pretraining on unseen target languages, while improved speaker-verification methods consistently improve an existing meta-learning framework.
  • Meta-learning: Meta-TTS synthesizes high-speaker-similarity speech from a few enrollment samples with fewer adaptation steps.
  • Parameter-efficient transfer: Adapters, prefix tuning, and LoRA enable parameter-efficient transfer by modifying only limited components or adding learnable parameters to pretrained Transformers.
  • Parameter-efficient transfer: SURE benchmark results show adapter-based methods perform comparably to fine-tuning, without a significant advantage for any particular adapter type.
  • Model compression: Compression methods reduce model size or computation through pruning, low-rank factorization, and quantization, with task-dependent performance trade-offs.

7 Conclusion and Future Research Directions

The paper identifies several directions for speech processing, including larger, multilingual, multimodal, controllable, and parameter-efficient models. It also highlights interpretability, neuroscience-inspired architectures, and text-to-audio models as areas for continued development.

  • Large Speech Models: Future ASR and TTS systems are expected to use larger models and datasets, with adversarial training proposed to improve naturalness and prosody.The review connects increased scale with more human-like TTS output.
  • Multilingual Models: Self-supervised multilingual models such as XLS-R address low-resource speech recognition by training on diverse data at billion-parameter scale.XLS-R is described as having over 2 billion parameters and covering 128 languages.
  • Multimodal Speech Models and In-Context Learning: Multimodal speech models combine speech and text modalities, while in-context learning can specify speech-related tasks and examples within the input.These directions build on the growth of mixed-modality generative models and progress in large language models.
  • Controllable Speech Generation: Controllable TTS enables fine-grained adjustment of synthesized speech attributes such as tone, accent, age, and gender through text guidance.The review links this controllability to personalization and customization.
  • Parameter-efficient Learning: As speech and language models grow, parameter-efficient adapters and model compression are proposed to reduce the updates and resources required for these large models.The cited discussion also points to sparsity within large models as supporting compression approaches.
  • Interpretability and Emerging Architectures: Future research also includes explainability, neuroscience-inspired architectures, and text-to-audio models that perform well for both general audio and speech.The review notes correlations between speech-model layers and the brain’s functional hierarchy, alongside growing interest in speech-focused text-to-audio systems.
Loading 2305.00359v3…