PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Progresses in Deep Learning based Acoustic Models (Updated)

Dong Yu, Jinyu Li

arXiv:1804.09298v2eess.AScs.CLcs.SD

TL;DR

The paper surveys how deep learning has advanced acoustic modeling for speech recognition, including contextual modeling, end-to-end optimization, robustness, and efficient decoding. It organizes motivations and core ideas across RNNs, CNNs, CTC, attention models, adaptation, enhancement, separation, robust training, and compression. The surveyed developments improve recognition modeling while exposing trade-offs such as computational cost, streaming limitations, and data requirements.

  • Problem

    Acoustic modeling must represent variable-length context, reduce manually designed components, handle robustness challenges, and support efficient decoding across speech-recognition settings.

  • Method

    The paper surveys recent deep-learning acoustic-model techniques and explains their motivations, core mechanisms, robustness strategies, efficiency methods, and future directions.

  • Results

    The surveyed approaches include models that exploit variable-length context, learn feature representations end-to-end, use CTC or attention, improve robustness, and reduce decoding cost.

  • Takeaways & Limitations

    Deep-learning acoustic-model research spans jointly learned representations, sequence criteria, adaptation, enhancement and separation, adversarial robustness, and more efficient decoding.

Abstract

from arXiv · show

In this paper, we summarize recent progresses made in deep learning based acoustic models and the motivation and insights behind the surveyed techniques. We first discuss acoustic models that can effectively exploit variable-length contextual information, such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), and their various combination with other models. We then describe acoustic models that are optimized end-to-end with emphasis on feature representations learned jointly with rest of the system, the connectionist temporal classification (CTC) criterion, and the attention-based sequence-to-sequence model. We further illustrate robustness issues in speech recognition systems, and discuss acoustic model adaptation, speech enhancement and separation, and robust training strategies. We also cover modeling techniques that lead to more efficient decoding and discuss possible future directions in acoustic model research.

I. INTRODUCTION

The paper surveys recent deep-learning acoustic-model developments, emphasizing variable-length contextual modeling and end-to-end optimization. It also updates earlier work through ICASSP2018 and links contextual modeling to improved recognition accuracy.

  • The survey covers recent acoustic-model developments, their motivations, and their core ideas, with emphasis on the preceding two years.
  • Deep RNNs and CNNs exploit variable-length contextual information more effectively than DNNs using fixed context windows.The paper connects these models with new state-of-the-art recognition accuracy.
  • The survey includes hybrid acoustic models, end-to-end models, robustness techniques, efficient decoding, and future research directions.
  • The updated version incorporates literature available through ICASSP2018.

A. Recurrent Neural Networks

RNN-based acoustic models address the limitations of fixed-context DNNs by representing sequence history, while gated and multidimensional designs improve long-range, spectro-temporal, and deployable modeling. These gains involve training and latency trade-offs.

  • RNNs encode sequence history and can use speech features observed up to the current frame, unlike fixed-window feed-forward DNNs.
  • Simple RNNs are difficult to train because gradients can increase or decrease exponentially, limiting their practical modeling of short-range effects.
  • LSTM-RNNs use input, output, and forget gates to stabilize gradient propagation over relatively longer time spans and outperform DNNs across ASR tasks.
  • Highway and residual LSTMs alleviate gradient vanishing in deep stacks through gated direct links or shortcut connections.
  • TF LSTM jointly scans time and frequency axes, producing normalized features that improve upper-layer time LSTMs and remain robust to distortion.Sequential processing along both axes increases computational complexity.
  • LC-BLSTM and RC-BLSTM bridge unidirectional and bidirectional LSTMs to reduce the real-time latency problem of whole-utterance bidirectional recognition.

B. Convolutional Neural Networks

CNN acoustic models apply convolution across speech representations to capture increasingly broad time-frequency patterns. Deep CNNs can approach BLSTM capability with lower latency, but computational cost remains important.

  • CNN acoustic models use convolution over tensor representations of speech, typically organized by row, column, and channel.
  • Each convolution output is a weighted sum of pixels across channels within a local input patch.
  • Shared kernels move across input patches, while strides and pooling subsample representations and reduce sensitivity to small pattern shifts.
  • CNN-based ASR evolved from TDNNs and frequency-variability models toward hybrid architectures exploiting variable-length contextual information along time and frequency.CLDNN and CDL achieved additional accuracy improvement over CNN-DNN models.
  • Deep CNN layers progress from local simple patterns to broader abstract patterns, with smaller kernels and more layers extending time-frequency dependencies.
  • Deep CNNs are compatible with BLSTMs but have limited latency and can better suit real-time systems when computation is controlled.
  • Treating an entire utterance as one input image can reuse intermediate computations and speed deep-CNN evaluation.

III. ACOUSTIC MODELS WITH END-TO-END OPTIMIZATION

The paper motivates end-to-end acoustic modeling as a way to jointly optimize sequential recognition components while removing manually designed elements. This section introduces the rationale for that optimization shift.

  • Hybrid DNN/HMM systems usually optimize their two components separately, whereas sequential recognition may benefit from joint optimization.
  • End-to-end models can remove manually designed components such as basic feature representations and lexicon design.

A. Automatically Learned Audio Feature Representation

The surveyed models learn acoustic representations jointly with recognition, extending from raw-waveform filtering to multichannel spatial and spectral processing. These approaches target phase invariance, robustness, and end-to-end far-field recognition.

  • Learned feature representations: Raw-waveform models replace manually designed Mel-filter-bank features by learning filters jointly with the recognizer.The CLDNN raw-waveform approach obtained a slight gain over log Mel-filter-bank features.
  • Learned feature representations: Pooling after time convolution reduces temporal variation in raw waveforms and provides phase-invariant representations.The paper motivates this design because small phase shifts can leave raw-waveform perception unchanged.
  • Robust far-field modeling: Far-field ASR models address additive noise and reverberation by jointly training beamforming or multichannel processing with the recognizer.Traditional beamforming remains a dominant baseline for multichannel speech recognition.
  • Robust far-field modeling: Multichannel time-convolution filters map microphone waveforms into a single time-frequency representation before upper-layer phoneme classification.Later factorization separates spatial filtering from spectral filtering and brings accuracy improvement.
  • Robust far-field modeling: Replacing CNN layers with 2D-LSTM layers improved robustness, while converting time-domain convolution into a frequency-domain product reduced computation.The resulting end-to-end system was used to build Google Home.

B. Connectionist Temporal Classification

CTC directly optimizes speech inputs against output label sequences by summing probabilities over valid alignments, supporting end-to-end acoustic modeling. The surveyed work addresses training difficulty, output-unit design, decoding efficiency, and CTC’s frame-independence limitation.

  • CTC formulation: CTC maps speech input frames to shorter output label sequences rather than optimizing traditional frame-by-frame labels.It introduces paths that align the longer input sequence with the output labels.
  • CTC formulation: The CTC loss sums negative log probabilities over all valid paths mapped to the target label sequence.The notation defines x as the speech input, l as the original labels, and B−1(l) as the mapped CTC paths.
  • CTC formulation: Under the conditional independence assumption, the sequence probability decomposes into a product of frame-level posterior probabilities.This assumption is the basis for the standard CTC probability factorization.
  • CTC results and efficiency: Context-dependent phone CTC outperforms monophone CTC and matches LSTM cross-entropy performance when training data is sufficiently large.Stacking three 10ms frames enables scoring and decoding every 30ms, three times faster than 10ms systems.
  • End-to-end output units: Character-based CTC removes lexicons and decision trees, providing a path toward end-to-end speech recognition with less expert-designed structure.The paper also reports greater robustness to accented speech because grapheme sequences are less affected by accents than phoneme pronunciations.
  • End-to-end output units: Gram-CTC learns variable-length character groups instead of using a fixed decomposition, improving modeling flexibility and final ASR accuracy.The method automatically learns a suitable decomposition of target sequences.
  • End-to-end output units: Word CTC can beat phoneme CTC with 100k output words and 125k hours of training data, but out-of-vocabulary words remain unmodeled or unrecognized.Hybrid word- and character-based systems provide fallback hypotheses for OOV segments, without substantially improving overall accuracy.
  • Training and decoding: CTC training is harder than cross-entropy training, motivating initialization strategies, large datasets, and curriculum learning such as SortaGrad.SortaGrad presents shorter utterances before longer ones during the first training epoch.

C. Attention-based Sequence-to-Sequence Models

Attention-based sequence-to-sequence models encode the full speech sequence and selectively weight hidden vectors for each output label. The surveyed improvements address training, representation, and decoding challenges, while the approach remains non-streaming and harder to train than CTC.

  • Model architecture: The encoder transforms the full speech input into a high-level hidden-vector sequence, and attention weights the most relevant vectors for each output label.This avoids the frame-independence assumption associated with CTC.
  • Model architecture: The attention decoder combines an output-probability generator, an RNN decoder, and an attention network that computes soft-alignment context vectors.The attention network produces weights over encoder states, which are used to form the context vector.
  • Attention mechanism: Attention scores can use content-based or hybrid forms computed from the previous decoder state and encoder hidden vectors.The hybrid form additionally incorporates location-related features through the attention parameters.
  • Training complexity: Windowing and pyramid encoders reduce attention-training complexity by limiting candidates or producing fewer high-level hidden vectors.These techniques address the high complexity and slow training of the vanilla model.
  • Limitations: Attention-based models are harder to train than CTC models and cannot operate in streaming fashion when attention uses the whole input utterance.The streaming limitation remains even when the encoder itself can operate in streaming mode.
  • Improvements: Word pieces, scheduled sampling, multi-head attention, label smoothing, external language models, and minimum word error rate training improved the surveyed attention-based systems.The paper reports that the resulting end-to-end system clearly outperformed the traditional hybrid system.

IV. ACOUSTIC MODEL ROBUSTNESS

Speech recognition systems can achieve remarkable accuracy when training and test conditions match, but performance degrades substantially in mismatched or complicated environments. The paper surveys adaptation, speech enhancement, and robust modeling as solution categories.

  • Matched conditions: Recognition accuracy is remarkable when training and test sets match, especially under quiet close-talk conditions.The paper frames matched quiet conditions as a favorable evaluation setting.
  • Mismatched conditions: Performance dramatically degrades under mismatched environments, including higher noise, music, interfering talkers, or strong accents.These conditions create robustness challenges for otherwise accurate systems.
  • Robustness approaches: The surveyed solution categories are acoustic model adaptation, speech enhancement, and robust modeling.These categories target recognition under environmental and speaker mismatches.

A. Acoustic Model Adaptation

Acoustic model adaptation addresses speaker and environment variation with limited speaker-dependent data. The surveyed methods constrain adaptation through regularization, compact parameterizations, or low-dimensional transformation subspaces.

  • Motivation: Speaker adaptation can significantly improve an individual speaker’s performance, but typically relies on limited and often unsupervised adaptation data.Speaker-independent models are trained on large datasets to work well across speakers.
  • Motivation: Unsupervised adaptation with limited speaker-dependent data motivates better criteria, model topologies, and smaller speaker-dependent model footprints.Compact models are important when adaptation must scale to millions of speakers.
  • Adaptation criteria: KLD regularization keeps an adapted model close to the speaker-independent model and is reported effective with limited adaptation data.The method constrains the adapted model rather than allowing it to stray far from the original model.
  • Compact parameterization: SVD bottleneck adaptation uses small low-rank speaker-dependent matrices, while LHUC and sigmoid adaptation reduce them further to diagonal matrices.These approaches trade modeling capacity against generalization while reducing deployment cost.
  • Subspace methods: Subspace methods represent transformations with few parameters by confining them to low-dimensional speaker-related subspaces.Auxiliary features such as i-vectors, speaker codes, and noise estimates can be concatenated with acoustic features to impose such constraints.
  • Subspace methods: CAT and FHL constrain transformations through basis models, with FHL using rank-1 bases to reduce per-basis training data and enable more bases.CAT bases are full-rank and therefore usually limited in number because they require substantial training data.

B. Speech Enhancement and Separation

Deep learning reframes speech enhancement and separation as supervised source-recovery problems, but speaker-independent multi-talker separation remains difficult because source labels are permutation-ambiguous. Deep clustering, attractor networks, and permutation invariant training address this ambiguity, with PIT avoiding separate evaluation-time tracing.

  • Supervised enhancement and separation: Deep learning casts enhancement and separation as supervised prediction of target sources from paired mixtures and source streams.The approach commonly predicts each target source for each time-frequency bin, improving substantially over earlier unsupervised methods but working best when sources differ clearly.
  • Speaker-independent separation: Speaker-independent multi-talker separation is difficult because symmetric mixtures provide no predetermined assignment between sources and output layers.This label-permutation problem prevents straightforward supervised training from reliably learning separated outputs.
  • Speaker-independent separation: Deep clustering maps time-frequency bins into an embedding space where bins from the same speaker are close and different-speaker bins are separated, then clusters them during evaluation.A subsequent network can estimate real masks from the resulting source partition.
  • Speaker-independent separation: Permutation invariant training assigns output streams to target sources using the minimum-error utterance-level assignment before minimizing the assigned loss.It resolves label permutation and speaker tracing jointly without a separate tracing step or evaluation-time assignment cost.
  • Multi-channel separation: Multi-channel separation improves on spectral-only methods by incorporating spatial information alongside acoustic embeddings.The surveyed methods combine spatial features with learned representations, though some require complex clustering algorithms.

C. Robust Training

Robust training addresses domain mismatch and limited target-domain transcription through adversarial adaptation, teacher/student learning, and knowledge distillation. These methods can exploit unlabeled or soft-labeled data while preserving different requirements depending on available target-domain knowledge.

  • Domain adaptation: Deep models lose performance on new domains, while conventional adaptation remains limited by the amount of transcribed target-domain data.Large-scale transcription for a new domain is described as time-consuming or expensive.
  • Domain adaptation: Adversarial training performs unsupervised domain adaptation with encoder, recognizer, and domain-discriminator networks, without requiring target-domain knowledge.It is considered for both noisy-environment robustness and speaker-invariant training.
  • Teacher/student learning: Teacher/student learning replaces hard transcription labels with posterior probabilities generated by a source-domain teacher model to train a target-domain student.This shifts adaptation toward generating suitable parallel data rather than transcribing thousands of hours of audio.
  • Teacher/student learning: Soft labels sometimes outperform hard-label cross-entropy training with the same data amount because they encode relative probabilities among plausible outputs.The paper gives noisy ASR as an example where these richer targets can make learning easier.
  • Practical choices: Adversarial adaptation is suited to unknown target domains, whereas teacher/student learning and knowledge distillation are favored when target-like data can be simulated.Teacher/student learning can use much larger unlabeled adaptation sets and was applied to a far-field speaker system using tens of thousands of hours of unlabeled data.

V. ACOUSTIC MODELS WITH EFFICIENT DECODING

Efficient decoding methods reduce the runtime and parameter costs of deep acoustic models through low-rank factorization, teacher/student compression, quantization, architectural simplification, and reduced evaluation frequency. These methods trade computational efficiency against retraining needs, accuracy, or model complexity.

  • Motivation: Deep networks improve WER through depth, but their computational cost is a concern for real-time industry deployment.The section motivates several approaches for reducing runtime cost.
  • Model compression: Singular value decomposition factorizes full-rank matrices into lower-rank matrices, reducing deep-model parameters without accuracy loss after retraining.The approach is described as applicable to any deep network structure, with related low-rank methods for compact LSTMs.
  • Model compression: Teacher/student learning and knowledge distillation compress a large model by training a smaller model against the teacher’s output distribution.The student objective is equivalent to cross-entropy training with the teacher’s soft labels.
  • Quantization: Heavy quantization can speed runtime evaluation without retraining, but very low-bit compression or complex structures can significantly reduce ASR accuracy.The text notes that refinement with quantization-aware methods is needed in such cases, although the passage truncates before detailing it.
  • Architectural simplification: LSTMP reduces recurrent computation by projecting LSTM outputs to a lower-dimensional vector used as recurrent input, and parameter reduction was also reported to reduce error rates.The reported error-rate benefit is presented as possibly reflecting improved generalization when the original LSTM is too strong.
  • Reduced evaluation frequency: Frame skipping reduces evaluation frequency by computing acoustic scores periodically and copying them to intervening frames, with LSTM systems requiring matching training and testing treatment.The strategy exploits correlation across frames during decoding.

VI. FUTURE DIRECTIONS

The paper identifies far-field recognition as a frontier requiring improvements across audio capture, acoustic modeling, and decoding. It highlights unresolved separation quality, end-to-end data and language-model integration, and robustness under unavoidable training-test mismatch.

  • Far-field recognition: Far-field microphones expose difficulties hidden by close-talk recording because speech-signal energy is very low at the distant microphone.The shift toward far-field interaction is driven by devices such as Amazon Echo and Google Home.
  • Far-field recognition: Solving distant speech recognition requires optimizing the full pipeline from microphone-array signal processing through acoustic modeling and decoding.Close-talk techniques can be applied directly but show inferior performance in distant recognition scenarios.
  • Separation: Current PIT systems separate mixtures of different-gender speakers well, but separation quality deteriorates when same-gender speakers speak simultaneously.The paper proposes investigating beamforming, multi-channel information, alternative separation models, and decoder feedback.
  • End-to-end modeling: End-to-end optimization is attractive for its simplicity and joint optimization when sufficient training data are available, but most tasks cannot obtain the hundreds of thousands of hours used by successful word-based CTC systems.The paper also raises integrating the separately trained language model into the end-to-end system as a possible accuracy direction.
  • Robustness: Even models trained with huge datasets lack robustness because training-test mismatch is unavoidable given data-collection costs.The paper questions whether models can adapt continuously within controlled limits beyond what simulated data can provide.
Loading 1804.09298v2…