Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey

Hamza Kheddar; Mustapha Hemis; Yassine Himeur

Source-linked AI summary

Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey

Hamza Kheddar, Mustapha Hemis, Yassine Himeur

arXiv:2403.01255v2Mar 2, 2024cs.SDcs.AIeess.ASeess.SP

TL;DR

ASR development is constrained by extensive data and resource requirements, confidentiality, and mismatched training and testing domains. This survey reviews DTL, FL, DRL, and Transformer-based ASR through a taxonomy and critical comparison, concluding that important challenges remain, especially for privacy, heterogeneous data, streaming, and long-sequence processing.

Problem
ASR deep-learning systems require extensive data and computational resources, while limited data, confidentiality, and domain mismatch constrain practical deployment.
Method
The survey reviews DTL, FL, DRL, and Transformer-based ASR frameworks using a taxonomy, critical analysis, and comparative study.
Results
The review identifies strengths, weaknesses, ongoing challenges, and future research directions across advanced deep-learning ASR frameworks.
Takeaways & Limitations
Advanced deep-learning approaches broaden ASR research, but further work is needed to address privacy, generalization, computational, and data-related challenges.
Takeaways & Limitations
Transformer-based ASR remains challenged by long audio sequences, speech variability, low-resource languages, and substantial memory and computational demands.

Abstract

from arXiv · show

Recent advancements in deep learning (DL) have posed a significant challenge for automatic speech recognition (ASR). ASR relies on extensive training datasets, including confidential ones, and demands substantial computational and storage resources. Enabling adaptive systems improves ASR performance in dynamic environments. DL techniques assume training and testing data originate from the same domain, which is not always true. Advanced DL techniques like deep transfer learning (DTL), federated learning (FL), and reinforcement learning (RL) address these issues. DTL allows high-performance models using small yet related datasets, FL enables training on confidential data without dataset possession, and RL optimizes decision-making in dynamic environments, reducing computation costs. This survey offers a comprehensive review of DTL, FL, and RL-based ASR frameworks, aiming to provide insights into the latest developments and aid researchers and professionals in understanding the current challenges. Additionally, transformers, which are advanced DL techniques heavily used in proposed ASR frameworks, are considered in this survey for their ability to capture extensive dependencies in the input ASR sequence. The paper starts by presenting the background of DTL, FL, RL, and Transformers and then adopts a well-designed taxonomy to outline the state-of-the-art approaches. Subsequently, a critical analysis is conducted to identify the strengths and weaknesses of each framework. Additionally, a comparative study is presented to highlight the existing challenges, paving the way for future research opportunities.

1. Introduction

ASR is a key technology for human-machine interaction, supporting speech-based interactions with AI systems. Deep learning has advanced ASR but introduces substantial data, computational, and domain-mismatch challenges.

ASR enables speech-based interaction with chatbots, digital assistants, and other AI platforms.
Transformers have become central to ASR because they capture long-range dependencies and complex sequential patterns.
Deep learning ASR requires extensive training data and significant computational resources.
Limited annotated data and mismatched training and testing domains can compromise ASR model performance.

Abbreviations

This section defines the survey’s technical abbreviations and outlines its review scope, contributions, methodology, and organization. The review covers advanced deep-learning ASR research and identifies challenges and future directions.

The survey covers DTL, DRL, FL, Transformers, and related ASR techniques and metrics.Examples include deep transfer learning, deep reinforcement learning, federated learning, and automatic speech recognition.
Its contributions include reviewing evaluation metrics and datasets, identifying challenges and gaps, and proposing future research directions.
The paper organizes ASR methodologies with a taxonomy based on acoustic-model and language-model domains.
The review searches Scopus and Web of Science using combined ASR, NLP, DTL, DRL, FL, and Transformer keywords.The search also evaluates innovation, study quality, contributions, and findings.
The article is structured into six sections covering introduction, background, advanced approaches, challenges, future directions, and conclusions.

2. Background

The background describes acoustic and language models, ASR evaluation metrics, datasets, and an end-to-end recognition process. Acoustic models represent speech characteristics, while language models provide linguistic sequence probabilities during decoding.

Acoustic models extract speech features from audio waveforms and support decoder-based transcription generation.Feature extraction may use LPC, cepstral analysis, filter banks, MFCCs, or wavelets.
Language models estimate word-sequence probabilities and provide linguistic context that complements acoustic models during decoding.The backoff n-gram model is identified as frequently used in ASR.
The n-gram language-model equation expresses a word’s conditional probability given the preceding n−1 words.
ASR methods are evaluated with general metrics such as accuracy, F1-score, recall, precision, and specificity, alongside ASR-specific metrics.
The literature uses diverse datasets for DTL-based ASR, primarily drawn from publicly accessible repositories.Some datasets have undergone multiple updates and improvements.

3. Advanced ASR methods and applications

Advanced ASR research applies Transformers, DTL, FL, and DRL to address limitations of conventional models, while reviewing related models, datasets, and practical implementations.

Traditional language models remain widely used for simplicity and reliability, while BERT provides stronger contextual understanding than single-direction language models.
DNN-HMM and CTC have advanced acoustic modeling, with CTC enabling end-to-end training without prealignment.CTC requires only input and output sequences, while S2S can operate without a language model or pronunciation dictionary.
Advanced methods such as DTL, domain adaptation, DRL, and FL address performance degradation, limited training data, knowledge transfer, and model generalization.
Figure 4 overviews current state-of-the-art advanced DL-based ASR approaches and related schemes for acoustic and language modeling.
The survey’s figures explain Transformers, DTL, FL, and DRL principles and illustrate practical applications including CNN-based Transformers and source separation.

3.1. Transformer-based ASR

Transformers support ASR through self-attention and multiple end-to-end architectures, while recent studies extend them across acoustic and language modeling. The reviewed approaches target accuracy, efficiency, streaming, sparsity, and limited labeled data.

Transformer architecture: Transformers use query–key–value self-attention to capture extensive dependencies in input sequences.The architecture also includes masked self-attention in decoders and cross-attention between decoder queries and encoder outputs.
Research scope: Transformer-based ASR research spans acoustic and language-modeling domains, with Table 4 summarizing recent techniques and their reported outcomes.The survey reviews approaches across both domains rather than treating Transformer ASR as a single architecture.
Acoustic domain: Transformer-based acoustic models report gains including 19% to 26% relative improvement over the best hybrid result with a standard n-gram LM.The cited comparison concerns a Transformer-based acoustic model and its hybrid baseline.
Acoustic domain: Self-supervised and adaptive Transformer methods improve robustness and data efficiency, including 60.9% improvement on speech classification tasks and over 15% relative error reduction for streaming methods.The reviewed work includes reduced reliance on labeled data and augmented memory self-attention for streaming acoustic modeling.
Efficiency and adaptation: Transformer systems are also adapted for efficient and collaborative ASR, with reported training-speed, inference-time, and relative accuracy improvements.Examples include memory-efficient architectures, collaborative acoustic encoders, and co-distillation for on-device ASR.
End-to-end architectures: End-to-end Transformer ASR commonly uses transducers or CTC to map variable-length acoustic inputs to transcriptions and support alignment-free or timing-aware training.Transducers address variable-length input and output sequences, while CTC introduces blank symbols and variable-length alignments.

3.2. DTL-based ASR

DTL transfers knowledge from a source domain or task to a related target domain or task, helping ASR address distribution differences and limited data. The surveyed approaches include self-supervised speech representations, data augmentation, multi-task dialect adaptation, and language-model adaptation.

DTL principle: DTL transfers knowledge from a pretrained source model to improve performance on a different but related target domain or task.A domain denotes a data distribution, while a task denotes a learning objective.
Acoustic-model adaptation: Unsupervised wav2vec pre-training on large unlabeled audio produced representations used to enhance acoustic-model training.The acoustic model used a simple CNN optimized through noise-contrastive binary classification.
Acoustic-model adaptation: XLS-R wav2vec 2.0 enabled an end-to-end acoustic model through self-supervised pre-training on extensive cross-lingual adult-speech corpora.
Acoustic-model adaptation: A multi-dialect Transformer acoustic model used soft-parameter-sharing multi-task learning and auxiliary cross-attentions to incorporate dialect information.The auxiliary task recognized dialect identity, while adaptive cross-entropy loss balanced multi-task learning.
Language-model adaptation: Language-model adaptation applies DTL to voice recognition to bridge source and target distributions, using pretrained language-model approaches such as BERT.
Hybrid architectures: A hybrid DTL scheme transferred CTC to a target attention-based model after multilingual feature extraction, showing superior performance on TIMIT but requiring high-resource testing.The scheme used non-negative matrix factorization and joint CTC-attention models; further optimization was needed for standard end-to-end training.

3.3. FL-based ASR

FL trains a shared global ASR model collaboratively while keeping training data localized, supporting privacy-preserving learning across decentralized clients. The section distinguishes horizontal and vertical FL and reviews secure aggregation, adaptive training, and communication-efficient approaches.

FL principle: FL collaboratively trains a global model across clients while keeping sensitive training data on edge devices or private servers.The global objective aggregates local loss functions from participating clients.
FL principle: FedAvg iteratively improves the global model by aggregating client-computed gradients or parameter updates based on local data.
Federated-learning types: Horizontal FL uses datasets with the same feature space but different sample spaces, aggregating encrypted local updates through a central server.Its process includes initialization, local training, gradient encryption, secure aggregation, and global parameter updates.
Federated-learning types: Vertical FL uses datasets with the same sample space but different feature spaces, coordinating through entity data alignment and encrypted model training.
Federated-learning types: HFL and VFL address different data-distribution scenarios while supporting decentralized learning and privacy preservation.
FL-based ASR applications: ASR applications include client-adaptive federated training for data heterogeneity, wav2vec 2.0-based federated modeling, and decentralized feature extraction.One approach communicates secure high-level features instead of model parameters to reduce communication costs for large models.

3.4. DRL-based ASR

DRL frames ASR optimization as sequential decision-making, where an agent learns from rewards while interacting with an environment. The surveyed work targets train–test discrepancies, user-feedback-based hypothesis selection, and resource-efficient model compression.

DRL principle: DRL learns decision-making policies through environment interaction, using rewards or penalties to maximize cumulative reward over time.Its framework includes an environment model, states, and observations.
DRL principle: Markov decision processes formalize dynamic stochastic decisions using state and action spaces, transition probabilities, and reward functions.Their objective is an optimal policy maximizing expected discounted total reward, with γ as the discount factor.
DRL-based ASR motivation: DRL-based ASR addresses mismatch between cross-entropy training and WER evaluation, as well as exposure differences caused by teacher forcing.
DRL-based ASR applications: Policy-gradient DRL has been used for speech recognition and sequence-to-sequence ASR, including user feedback through hypothesis selection.
Resource-efficient ASR: Over 5× ASR compression was achieved through iterative compression and AutoML-based rank searching without degrading WER.AutoML-based low-rank factorization also achieved up to 3.7× speedup.

4. Open Issues and Key challenges

The surveyed ASR approaches offer opportunities but face persistent challenges involving model complexity, domain variation, privacy, communication, rewards, and adaptation across speech conditions.

Transformer-based: Transformer-based ASR requires substantial memory and computation for long audio sequences while remaining robust to accents, disorders, and background noise.Generalization across these variations without substantial scenario-specific data remains open.
DTL and DA-based: Domain adaptation must address joint-distribution shifts, label-distribution shifts, catastrophic forgetting, feature discrepancy, sample-selection bias, and hyperparameter choices.Label shifts can occur even when feature distributions align, while fine-tuning may reduce performance on the original task.
DTL and DA-based: Speech-based DTL is complicated by mismatches between source and target databases involving language, speakers, age, ethnicity, and acoustic environments.
FL-based: Federated DTL must handle heterogeneous speech data, substantial model-update communication overhead, privacy risks from update inference, and personalization-generalization trade-offs.Bandwidth and energy use are particular concerns for mobile devices, while differential privacy or secure multi-party computation may mitigate leakage.
DRL-based: DRL-based ASR faces sparse and delayed rewards, exploration-exploitation trade-offs, large interaction-data requirements, privacy concerns, reward-design difficulty, and varied operating environments.Moving from supervised learning on annotated datasets to interaction-based training introduces additional technical and data-security challenges.

5. Future directions

Future directions emphasize inclusive personalization, multitask and federated multitask learning, model compression, continual adaptation, broader DRL methods, and transformer-based language modeling for ASR.

DTL and DA-based: Personalized data augmentation is proposed to improve recognition for individuals with dysarthria or older adults, whose extensive datasets are difficult to collect.
DTL and DA-based: Multitask learning can improve ASR generalization by learning shared representations across transcription and related tasks such as speaker identification or emotion recognition.Auxiliary tasks can provide additional data and help address accents, dialects, and noisy environments.
Federated multi-task learning and distillation for ASR: Federated multi-task learning gives clients task-specific personalized models while regularization encourages parameter sharing and collaboration across heterogeneous data and tasks.Raw speech data remains on user devices, and differential privacy can further anonymize shared updates.
Federated multi-task learning and distillation for ASR: Federated learning distillation should compress neural ASR models for edge devices while preserving performance metrics such as WER.The central research problem is balancing reduced model size against retained performance under storage and computation constraints.
DRL-based ASR: Incremental DRL could continuously adapt ASR to new data and changing conditions without full retraining, improving computational efficiency for noisy environments and varying accents.
DRL-based ASR: Applying broader DRL families, including DDQN, actor-critic, and DDPG, remains an underexplored direction for enhancing DTL methods.
Online DTL: Online DTL incrementally updates deep models as new target-domain data arrives, integrating feature extraction, fine-tuning, adaptation, and continual learning.Discrepancies in class distributions and feature representations between source and target domains complicate this process.
Transformers and LLMs-based ASR: Transformer-based ASR research can use large language models, domain adaptation, and fine-tuning to improve language handling, transcription accuracy, contextual appropriateness, and evaluation.The survey also calls for integrating DRL and FL into acoustic and language-model domains.

6. Conclusion

Advanced DL techniques address ASR’s data, privacy, domain-adaptation, and dynamic-decision challenges while potentially improving performance and reducing computational costs. The survey reviews these frameworks, analyzes their strengths and weaknesses, and identifies directions for future work.

Traditional ASR requires extensive, sometimes confidential datasets and substantial computational resources, motivating advanced DL techniques.
DTL, FL, and DRL-based techniques target domain adaptation, privacy preservation, and dynamic decision-making in ASR.
These techniques are described as enhancing ASR performance while reducing computational costs.
The survey provides a comprehensive review and structured taxonomy of advanced ASR frameworks, with critical analysis of their strengths and weaknesses.
Future research should address FL privacy concerns, improve DRL optimization, and develop more efficient and accurate Transformer-based speech recognition.