PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Speech Recognition: A Survey

Rohit Prabhavalkar, Takaaki Hori, Tara N. Sainath, Ralf Schlüter, Shinji Watanabe

arXiv:2303.03329v1eess.AScs.CLcs.SD

TL;DR

The survey addresses how end-to-end ASR replaces classical HMM-based modular systems while handling alignment, training, decoding, and language-model integration. It develops a taxonomy and reviews model improvements, finding that E2E systems support compact on-device recognition but retain challenges including contextual biasing, external language-model integration, and lattice generation.

  • Problem

    E2E ASR must account for unknown alignments between acoustic frames and output labels while integrating modeling, training, decoding, and language knowledge.

  • Method

    The paper surveys and taxonomizes E2E ASR architectures and improvements, contrasting explicit and implicit alignment models with classical ASR.

  • Results

    RNN-T+AED yields a 17-22% relative improvement in word error rate over RNN-T alone on a Voice search task.

  • Takeaways & Limitations

    Integrated neural components enable compact, highly accurate on-device speech recognition models.

Abstract

from arXiv · show

In the last decade of automatic speech recognition (ASR) research, the introduction of deep learning brought considerable reductions in word error rate of more than 50% relative, compared to modeling without deep learning. In the wake of this transition, a number of all-neural ASR architectures were introduced. These so-called end-to-end (E2E) models provide highly integrated, completely neural ASR models, which rely strongly on general machine learning knowledge, learn more consistently from data, while depending less on ASR domain-specific experience. The success and enthusiastic adoption of deep learning accompanied by more generic model architectures lead to E2E models now becoming the prominent ASR approach. The goal of this survey is to provide a taxonomy of E2E ASR models and corresponding improvements, and to discuss their properties and their relation to the classical hidden Markov model (HMM) based ASR architecture. All relevant aspects of E2E ASR are covered in this work: modeling, training, decoding, and external language model integration, accompanied by discussions of performance and deployment opportunities, as well as an outlook into potential future developments.

I. INTRODUCTION

End-to-end ASR replaces much of the classical modular architecture with integrated neural modeling, training, and recognition. This survey defines E2E ASR, explains its motivations and scope, and organizes models by how they handle alignment.

  • Classical ASR decomposes recognition into feature extraction, acoustic modeling, language modeling, and search under Bayes decision theory.
  • Using only transcribed speech creates a challenge for exploiting text-only and audio-only data in joint E2E training.
  • E2E ASR jointly models and recognizes speech with an integrated neural structure, avoiding separately obtained knowledge sources.
  • E2E approaches aim to minimize expected word error rate while also reducing decoder complexity, memory, and development effort.
  • A single neural-network structure can simplify decoding, reduce memory footprint and power consumption, and support embedded ASR applications.
  • The survey covers E2E modeling, training, decoding, language-model integration, performance, deployment, and contrasts with classical ASR.
  • E2E model families are categorized by whether alignment between acoustic frames and output labels is modeled explicitly or implicitly.

A. Explicit Alignment E2E Approaches

Explicit-alignment E2E models represent frame-to-label alignments as latent variables and marginalize over valid paths. CTC, RNN-T, and related models differ in alignment semantics and conditional-independence assumptions.

  • Explicit-alignment models define latent alignments between encoder outputs and labels, then marginalize over valid alignment sequences to compute P(C|X).
  • Connectionist Temporal Classification (CTC): CTC augments the output vocabulary with a blank and collapses repeated labels before removing blanks to recover the target sequence.
  • Connectionist Temporal Classification (CTC): CTC assumes outputs at different timesteps are conditionally independent given the local encoder output.
  • Connectionist Temporal Classification (CTC): CTC predicts frame-level probabilities from an encoder representation through a softmax layer over labels and blank.
  • Recurrent Neural Network Transducer (RNN-T): RNN-T adds a prediction network for previously emitted non-blank labels and a joint network combining prediction and encoded-frame states.
  • Recurrent Neural Network Transducer (RNN-T): Unlike CTC, RNN-T allows repeated output labels without special treatment because blank outputs advance the model to the next frame.
  • Recurrent Neural Network Transducer (RNN-T): RNN-T reduces CTC’s independence assumptions by conditioning on previous non-blank predictions while remaining independent of their specific emission frames.

3) Recurrent Neural Aligner (RNA):

RNA modifies the transducer alignment structure by emitting at most one label per frame and conditioning on the entire preceding alignment. This increases dependence modeling but makes exact likelihood computation intractable, requiring an approximation.

  • RNA permits only one output per frame, either blank or non-blank, and advances to the next frame after that output.
  • Restricting RNA to one label per frame improves computational efficiency and simplifies decoding.
  • RNA’s prediction state summarizes the entire preceding alignment rather than only the previous non-blank label sequence.
  • RNA removes RNN-T’s remaining conditional-independence assumption by conditioning on prior labels and the frames that emitted them.
  • Exact RNA likelihoods and gradients are intractable, so the model uses decoder states from the most likely alignment path.

B. Implicit Alignment E2E Approaches

Implicit-alignment E2E approaches generate outputs without explicit alignment variables, using attention-derived context in AED models. They offer flexible output dependencies but are generally non-streaming and can be sensitive to long inputs.

  • Implicit Alignment: Explicit-alignment models can stream by producing hypotheses from encoded frames as speech arrives, whereas AED models process the entire acoustic sequence before output.CTC, RNN-T, and RNA use explicit latent alignment variables; AED models instead use an end-of-sentence symbol to indicate completion.
  • AED Models: AED models compute each output probability from the decoder state and an attention-derived context vector summarizing relevant encoder frames.The decoder state summarizes previous outputs, while the context vector summarizes encoder outputs according to attention.
  • AED Models: AED models avoid output-independence assumptions and explicit alignment marginalization, but previously generated context vectors are not revised during decoding.This creates a left-right asymmetry even when the encoder is bidirectional.
  • AED Models: Attention weights can be interpreted as alignment weights, indicating which encoded frames are relevant to each output symbol without defining an explicit alignment.For the example sequence (s, e, e, ⟨eos⟩), early frames correspond to s and later frames to the second e.
  • Attention Mechanisms: Dot-product and additive attention are common mechanisms, while multi-head attention concatenates independently computed context vectors.Other variants include Gaussian attention and stacked attention-based layers in Transformer decoders.

C. Attention-based E2E Approaches with Alignment Modeling

Attention-based models can be extended with explicit alignment mechanisms and combined with other E2E models to address streaming, long or noisy utterances, and contextual recognition. These approaches trade the flexibility of full attention against constraints or rescoring stages that improve deployment suitability.

  • Attention-based Models: Full AED models often outperform CTC or RNN-T, but they are fundamentally non-streaming and sensitive to acoustic-sequence length.Long-form audio therefore requires special processing.
  • Alignment Modeling: Streaming AED variants constrain attention to local or monotonic regions because speech typically has a local relationship between encoded frames and output units.Neural Transducer partitions frames into chunks, while monotonic attention scans frames left to right to identify emission points.
  • Alignment Modeling: Neural Transducer inference processes one chunk at a time and advances after emitting an end-of-chunk symbol.Training can use rough initial alignments followed by iterative refinement of likely output alignments.
  • Combinations of Models: 5-14% relative WER improvement over attention-based models was reported when CTC and attention models were trained jointly on WSJ and Chime tasks.The combination uses CTC’s left-to-right constraints to complement attention’s flexibility.
  • Combinations of Models: 17-22% relative WER improvement over RNN-T alone was reported for RNN-T first-pass decoding followed by AED rescoring on a Voice search task.The encoder can be shared to reduce computation, while AED contributes broader utterance context.
  • Contextual Biasing: Contextual biasing methods interpolate a biasing FST during beam search or inject embedded biasing phrases into an attention-based model.These methods target domain-specific phrases such as song names, app names, and contact names.

C. Encoder/Decoder Structure

E2E ASR encoder and decoder research targets parallel computation, compact decoding, alignment-aware training, and improved use of external language-model data. These developments span architecture, optimization, alignment handling, and discriminative objectives.

  • Encoder/Decoder Structure: LSTM encoders and decoders process frames sequentially, preventing multiple frames from being batched in parallel.This sequential dependency motivated architectures better suited to modern parallel hardware.
  • Encoder Structure: Convolutional, Transformer, Conformer, and multilayer-perceptron architectures replace or complement recurrence to exploit parallel computation.Transformer and Conformer models have shown competitive performance with LSTMs on many tasks.
  • Decoder Structure: A simple embedding lookup decoder attending to a few previous tokens can replace a large LSTM decoder in transducer models.This finding indicates that much of the E2E model’s power is concentrated in the encoder.
  • Alignment and Training: E2E training handles latent sequence alignment explicitly in CTC, RNN-T, and RNA, but implicitly in AED models.Explicit approaches marginalize valid alignments with forward-backward computation, while AED models avoid that explicit marginalization.
  • Language-Model Integration: External language models can be incorporated during training through MMI sequence-discriminative training when sequence normalization is included explicitly.This enables use of text-only data during E2E model training.
  • Discriminative Training: MWER and related discriminative objectives construct losses from word errors or beam hypotheses rather than only conditional likelihood.Some approaches first train P(C|X) to obtain useful beam hypotheses, while others train discriminatively from scratch.

D. Pretraining

E2E ASR optimization combines training schedules, stochastic optimization, regularization, and data augmentation. Reported improvements include benefits from longer training and strong performance from jointly applying augmentation and regularization methods.

  • Pretraining: E2E training can use dedicated schedules and initially generated alignments to guide optimization toward suitable alignment behavior.Hierarchical schedules may begin with fixed Viterbi-style alignments before returning to full-sum training.
  • Optimization: Learning-rate control includes decay, warmup, cosine annealing, fine-tuning, and population-based training.These methods can be global or parameter-wise.
  • Optimization: Curriculum learning may start with short sequences, increase subsampling over training, or order multi-speaker mixtures by signal conditions.The cited examples include sorting mixtures by SNR and speaker characteristics.
  • Optimization: Stochastic gradient descent with momentum and adaptive methods such as Adam is commonly used for optimization.The survey lists Adam and several variants among prominent adaptive approaches.
  • Optimization: More training epochs have been reported to improve results, and averaging across epochs has also been reported to help.The survey connects the epoch-improvement observation to Table 8 and the averaging observation to prior work.
  • Regularization and Augmentation: Data augmentation perturbs speech variation through speed, vocal-tract, frequency, noise, or semantic transformations, and can also synthesize speech from text.Examples include SpecAugment and text-to-speech augmentation at feature or signal level.
  • Regularization and Augmentation: Jointly applying many regularization and augmentation methods yielded state-of-the-art Switchboard performance for a single-headed AED model.The passage attributes this result to a recent study and its follow-up work.
  • Decoding: Greedy decoding selects alignment tokens independently across frames, enabling parallel and fast decoding but generally weaker performance than attention or RNN-T beam methods.Its degradation is reported as relatively small when the model is well trained under matched conditions.

B. Beam search

Beam search approximates ASR decoding by retaining a limited set of promising hypotheses rather than evaluating all possible output sequences. It supports label-synchronous and frame-synchronous variants with different pruning units and alignment requirements.

  • Search limitations: Attention-decoder performance does not consistently improve as the search space increases.A modified implicit length model can produce consistent behavior without degradation for increasing beam sizes.
  • Search principle: Beam search expands partial hypotheses and retains only the top Δ candidates by probability score.The unpruned expansion can reach |˜C1:i−1| × |U| hypotheses at output step i.
  • Label-synchronous search: Label-synchronous search prunes after output-token extensions and adds hypotheses that emit the end-of-sentence symbol to the completed set.The output length is not known in advance, so completed hypotheses must be retained during pruning.
  • Search limitations: Unlike frame-synchronous search, label-synchronous hypotheses of equal output length can cover different numbers of encoder frames because alignment is not explicit.This can make very short and long segment hypotheses receive scores in the same range.
  • Label-synchronous search: Label-synchronous CTC search marginalizes all possible alignments for each label hypothesis.This approach avoids explicitly enumerating a single alignment for each label sequence.

D. Frame synchronous beam search

Frame-synchronous beam search prunes hypotheses at each input frame and uses explicit alignment information to connect encoder frames with output tokens. It supports streaming, but combining frame- and label-dependent scores requires synchronization or additional dynamic programming.

  • Algorithm: Frame-synchronous search performs hypothesis pruning at every input frame t.Partial label sequences are obtained from alignments accumulated through the current frame.
  • Algorithm: Explicit alignment information enables frame-synchronous decoding to handle online and streaming recognition.The algorithm tracks transitions between input frames and output tokens.
  • Implementations: Classical HMM, CTC, and RNN-T beam-search methods commonly use WFSTs or lexical prefix trees and can combine N-gram or neural language models.These methods are categorized as frame-synchronous beam search.
  • Implementations: Fixed-length input blocks trade latency against accuracy while allowing future context from non-causal encoders.The approach can use BLSTM, output-delayed unidirectional LSTM, or transformer encoders.
  • Score fusion: Frame- and label-dependent score functions use different time indexes, creating a synchronization mismatch during fusion.Examples include CTC or RNN-T frame-dependent scores and attention or language-model label-dependent scores.
  • Score fusion: Label-synchronous fusion first marginalizes alignments to obtain label-dependent scores, as in CTC/attention joint decoding.The CTC forward algorithm eliminates wrong-alignment issues and difficulties finding sentence ends in label-synchronous search.

G. Lexical constraint during score fusion

Lexical and language-model constraints can be integrated during one-pass beam search or applied in multi-pass rescoring. E2E systems gain access to external text and domain adaptation, but model fusion introduces synchronization and computational challenges.

  • Unit mismatch: Word-based language models and lexical constraints mismatch the letter or token units commonly used by E2E ASR.WFSTs, delimiter-triggered scores, prefix trees, and lookahead probabilities provide synchronization strategies.
  • Fusion strategies: One-pass fusion combines module scores during beam search, whereas multi-pass methods rescore N-best lists or lattices after decoding.Multi-pass decoding avoids synchronization requirements by considering the entire utterance context.
  • Lattice generation: RNN-T and AED cannot directly generate lattices because their lack of Markov assumptions prevents hypothesis merging.Proposed remedies limit dependencies to finite history or merge similar RNN-T hypotheses during beam search.
  • Efficient decoding: Vectorizing multiple hypotheses replaces per-hypothesis score accumulation with matrix operations and enables parallel processing on CPUs and GPUs.This has been studied for RNN-T and attention-based decoding and can process multiple utterances simultaneously.
  • External language models: E2E models include an internal language-model component, but external language models remain useful during inference.RNN-T prediction networks and AED decoders perform the internal language-model role.
  • External language models: External language models can compensate for E2E generalization problems, exploit text-only data, and support domain adaptation.Text data is generally easier to collect than paired speech and text, while target-domain language models can improve domain-specific recognition.

1) N-gram LM:

N-gram language models offer simple, fast probability estimation but rely on finite-context assumptions, while neural alternatives improve generalization or long-range context. FNN-LMs use distributed representations, RNN-LMs use recurrence, and ConvLMs emphasize parallel efficiency.

  • N-gram LM: N-gram LMs are simple and fast because training mainly counts tuples and decoding can use table lookup or precompiled WFST probabilities.They underperform state-of-the-art neural language models.
  • FNN-LM: FNN-LMs map input tokens into embeddings, form a context vector from N−1 tokens, and predict the next-token distribution with a softmax layer.Distributed token and context representations provide better smoothing and generalization than count-based N-gram models.
  • RNN-LM: RNN-LMs use recurrent connections to retain contextual information without limiting context to N−1 tokens.Past information decays under simple recursion, while stacked LSTM layers use memory cells and gates to preserve longer-range context.
  • ConvLM: ConvLM replaces recurrent connections with gated temporal convolutions and residual convolutional blocks.A 14-block ConvLM has been applied to E2E ASR.
  • ConvLM: ConvLM achieves performance similar to RNN-LMs while offering a highly parallelizable architecture for training on large datasets.Its fixed context can consist of short units such as letters and word pieces.

5) Transformer LM:

Transformer language models use self-attention to exploit prior-token context, while E2E ASR can incorporate external LMs through several fusion strategies.

  • Transformer LM: Transformer LMs compute hidden representations with stacked multi-head attention and feed-forward blocks.The attention module uses prior tokens as context; the cited example stacks these blocks six times.
  • Transformer LM: Self-attention uses all previous tokens without compressing them into fixed-size recurrent memory, improving long-context utilization over RNN-LMs.Its computational complexity grows quadratically with sequence length, so context is typically limited.
  • LM fusion: Shallow fusion combines pretrained E2E and LM scores during inference using a log-linear combination.The LM contribution is controlled by a scaling factor γ.
  • LM fusion: Deep fusion jointly trains a network that combines E2E and LM states through a gate controlling the LM contribution.Cold fusion instead trains the E2E model with a pretrained LM whose parameters remain frozen.
  • Domain adaptation: Internal-LM compensation and related methods address language bias when source and target domains differ substantially.HAT can use the prediction network as the internal LM, avoiding an additional LM during inference.

C. Use of large-scale pretrained LMs

E2E ASR has approached or surpassed classical systems on major benchmarks through architectural, augmentation, and learning advances, while retaining important differences in modeling and training.

  • Architectural relationship: E2E systems integrate knowledge sources into joint models, contrasting with classical separation into acoustic and language models.Strict E2E systems are ideally trained from scratch with one global criterion and transcribed speech data.
  • Decoding: E2E and classical systems both use single-pass decoding, but E2E beam search typically uses much smaller beam sizes.Small beams can partly mask length bias in attention-based encoder-decoder models, trading model errors against search errors.
  • Modeling: E2E models commonly use direct character-level modeling, avoiding phoneme-based modeling and pronunciation lexica.Character-level modeling is also possible in classical hybrid HMM architectures.
  • Training resources: Attention-based encoder-decoder models need more training epochs and can suffer with limited resources, where regularization, augmentation, and multilingual training help.TTS considerably improved limited-resource attention models but did not reach other E2E or hybrid HMM results.
  • Benchmark performance: 1.8% (test-clean) and 3.7% (test-other) word error rates were reached by E2E systems on Librispeech, surpassing former state-of-the-art results.On Switchboard 300h, the best E2E result was 5.4% versus 6.6% for the best hybrid system.
  • Benchmark progress: SpecAugment produced a significant gain across both Librispeech and Switchboard, followed by improvements from Transformer, Conformer, and ContextNet architectures.The final Librispeech gain in 2021 involved self-supervised and semi-supervised learning.

IX. DEPLOYMENT OF E2E MODELS

E2E ASR deployment progressed from streaming RNN-T systems to TPU-optimized on-device models, while unresolved limits remain in data efficiency, modularity, robustness, and adaptation. The survey also identifies multichannel ASR as a promising E2E application.

  • Deployment progression: Google deployed a streaming RNN-T first-pass E2E system on the Pixel 4 in 2019, with AED rescoring of first-pass hypotheses.This deployment is presented as a typical case study for production-level on-device E2E ASR.
  • Deployment progression: Pixel 5 improvements reduced user-perceived latency through end-to-end endpointing and FastEmit, which encouraged faster microphone closing and earlier token emission.The cited mechanisms target different parts of the interaction latency.
  • Deployment progression: Pixel 6 used TPU-oriented conformer encoders, a small embedding prediction network, cascaded second-pass beam search, and neural LM rescoring for long-tail named entities.The survey reports this model as Google’s best released ASR system in both quality and latency, significantly exceeding its 80GB cloud model.
  • Deployment constraints: E2E models perform well with abundant training data but do not scale well to low-resource conditions, while domain change makes language-model exchange important.External language-model integration and internal language-model estimation remain active research areas.
  • Deployment constraints: Deployment is additionally constrained by longer training requirements, reduced modularity, rare-word challenges, speaker adaptation, recording mismatch, and limited text-only resource integration.The survey describes multichannel ASR as a promising E2E direction jointly addressing source separation, diarization, and recognition.
Loading 2303.03329v1…