PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in End-to-End Automatic Speech Recognition

Jinyu Li

arXiv:2111.01690v2eess.AScs.AIcs.CLcs.SD

TL;DR

E2E ASR has achieved state-of-the-art accuracy, but hybrid systems remain widely deployed because production also depends on factors such as streaming, latency, and adaptation. This paper surveys recent E2E techniques and practical technologies addressing those industry challenges. It concludes that E2E research is progressing toward broader industrial replacement of hybrid systems, with RNN-T, Transformer encoders, and adaptation and customization methods as prominent directions.

  • Problem

    E2E models often lead ASR benchmarks, yet practical production requirements still limit their commercialization relative to established hybrid systems.

  • Method

    The paper provides an industry-focused overview of E2E ASR models and technologies for streaming, latency, multilingual modeling, adaptation, customization, and other deployment challenges.

  • Results

    The overview identifies AED and RNN-T as the most popular E2E models, a shift from LSTM to Transformer encoders, and RNN-T as the dominant streaming choice.

  • Takeaways & Limitations

    E2E models may replace hybrid systems in industry when they provide strong solutions to production factors beyond recognition accuracy.

Abstract

from arXiv · show

Recently, the speech community is seeing a significant trend of moving from deep neural network based hybrid modeling to end-to-end (E2E) modeling for automatic speech recognition (ASR). While E2E models achieve the state-of-the-art results in most benchmarks in terms of ASR accuracy, hybrid models are still used in a large proportion of commercial ASR systems at the current time. There are lots of practical factors that affect the production model deployment decision. Traditional hybrid models, being optimized for production for decades, are usually good at these factors. Without providing excellent solutions to all these factors, it is hard for E2E models to be widely commercialized. In this paper, we will overview the recent advances in E2E models, focusing on technologies addressing those challenges from the industry's perspective.

JINYU LI

The paper surveys end-to-end ASR as an alternative to hybrid modeling, emphasizing its accuracy advantages and the practical barriers to commercial deployment. It introduces CTC, AED, and RNN-T as the main E2E techniques and outlines CTC’s path-based formulation.

  • Motivation: E2E models optimize the ASR network with a single objective and directly output labels, simplifying the pipeline compared with separately optimized hybrid components.The paper notes that E2E models have outperformed traditional hybrid models in academic and industry settings.
  • Motivation: Commercial adoption remains limited because streaming, latency, adaptation capability, and other production factors matter beyond benchmark ASR accuracy.Traditional hybrid systems have been optimized for these deployment factors for decades.
  • Popular E2E techniques: The three most popular E2E ASR techniques are CTC, attention-based encoder-decoder, and recurrent neural network Transducer.RNN-T is described as naturally supporting streaming with high accuracy and low latency.
  • CTC: CTC maps speech sequences to shorter label sequences by inserting blank labels and allowing repeated labels to form paths matching the input length.The output probability is computed over paths that map to the target label sequence.
  • CTC: Recent CTC advances use attention-enhanced Transformer encoders and self-supervised representations to relax conditional independence while retaining CTC’s simple objective.The paper identifies Transformer replacement of LSTM and self-supervised learning as important contributors to CTC’s renewed effectiveness.

B) Attention-based Encoder-Decoder

The paper presents AED and RNN-T as complementary E2E approaches, focusing on alignment, streaming, latency, and training-efficiency challenges. AED uses autoregressive attention, whereas RNN-T provides a more natural streaming formulation and has become dominant in industry streaming settings.

  • AED architecture: AED combines an encoder, attention module, and decoder; its autoregressive decoder conditions on previous labels without CTC’s conditional independence assumption.The decoder uses the previous output and a context vector to generate each label probability.
  • AED training: Joint AED-CTC training shares the encoder, improves convergence, mitigates alignment problems, and became a standard AED training recipe.Some systems also combine AED and CTC scores during decoding.
  • Streaming AED: Vanilla AED requires the full utterance for full-sequence attention, creating significant latency and making it impractical for continuous unsegmented streaming speech.Streaming AED methods instead apply attention to input chunks, differing in how those chunks are selected and used.
  • Streaming AED: Streaming AED methods can support streaming but generally do not enforce low latency; hard alignments and scout networks were proposed to address this limitation.These methods use alignment or predicted word boundaries to trigger low-latency output.
  • RNN-T: RNN-T conditions each output on preceding tokens and speech through the current frame, removing CTC’s conditional independence assumption and naturally supporting streaming.Its alignment paths traverse a T×U grid, with horizontal blank transitions and vertical non-blank emissions.
  • RNN-T: RNN-T alignment restrictions constrain training paths within a delay threshold, improving latency while reducing GPU memory use and accelerating training.The paper also describes FastEmit as a method for reducing late label prediction.

III. ENCODER

The encoder is the most important component in E2E ASR because it converts speech into high-level feature representations.

  • Encoder: The encoder converts speech input sequences into high-level feature representations used by E2E ASR models.

A) LSTM

Early E2E ASR encoders primarily used unidirectional or bidirectional LSTM networks, while later designs shifted toward Transformer-based encoders with configurable attention for streaming and latency.

  • A) LSTM: Early E2E encoders used multi-layer unidirectional or bidirectional LSTM-RNNs, with the final layer output serving as the encoder representation.The input to the first layer is the speech signal, while subsequent layers receive the preceding layer's hidden output.
  • A) LSTM: Unidirectional LSTM is more widely used commercially because it supports streaming, whereas bidirectional LSTM uses whole-utterance information and is non-streaming.With an LSTM encoder, CTC and RNN-T operate in streaming mode by default; AED requires streaming attention strategies.
  • A) LSTM: Latency-controlled and contextual LSTM methods add future context to reduce the accuracy gap between unidirectional and bidirectional encoders.Latency-controlled BLSTM operates on overlapping multi-frame chunks while retaining bidirectional processing within each chunk.
  • B) Transformer: Transformer encoders replace LSTM with stacked blocks containing multi-head self-attention and feed-forward networks, improving long-term dependency modeling.Residual connections and layer normalization connect the Transformer layers and blocks.
  • B) Transformer: Conformer combines Transformer self-attention with convolution to capture both global context and local patterns.Its block uses two half-step feed-forward networks around the self-attention and convolution modules.
  • B) Transformer: Streaming Transformer attention uses masks to restrict future or distant frames, trading receptive field, computational cost, memory, and latency.Strict streaming removes future-frame attention; limited-history context operations reduce runtime and memory growth while preserving access to longer history.

IV. OTHER TRAINING CRITERION

E2E ASR training can use objectives beyond the standard negative log-likelihood, including teacher-student learning and minimum word error rate training.

  • IV. OTHER TRAINING CRITERION: The standard E2E training loss is −lnP(y|x), but teacher-student and minimum word error rate criteria provide additional training objectives.These criteria are discussed as alternatives or complements to the standard loss.
  • IV. OTHER TRAINING CRITERION: Teacher-student learning transfers information between teacher and student networks, while minimum word error rate training targets the evaluation metric directly.The supplied overview introduces both criteria as major alternatives to standard likelihood training.
  • IV. OTHER TRAINING CRITERION: Figure 6 illustrates attention configurations that vary computational cost and latency when predicting the output for x10.Each subfigure places the reception field on the left and the attention mask matrix on the right.

A) Teacher-Student Learning

Teacher-student learning transfers output-distribution information between E2E models and supports model compression, streaming conversion, domain adaptation, and joint training.

  • A) Teacher-Student Learning: Teacher-student learning commonly minimizes the KL divergence between teacher and student output distributions, with knowledge distillation adding a temperature parameter.The approach was introduced before becoming widely used during the deep-learning era.
  • A) Teacher-Student Learning: In E2E ASR, a small student model can learn from a larger teacher E2E model.This is the most popular use of teacher-student learning in E2E modeling.
  • A) Teacher-Student Learning: Teacher-student learning can transfer knowledge from a non-streaming E2E model to a streaming model, despite delayed decisions and differing CTC spike patterns.Methods modify non-streaming output spikes to make them more compatible with streaming students.
  • A) Teacher-Student Learning: Paired source- and target-domain acoustic data enables teacher-student adaptation from a source environment to a target environment.The paired data may come from simultaneous recording or simulation.
  • A) Teacher-Student Learning: Co-learning trains teacher and student models simultaneously, simplifying training and improving student-model accuracy.This differs from the standard two-step process of training the teacher first and then the student.

B) Minimum Word Error Rate Training

Minimum word error rate training aligns optimization with word-level recognition accuracy, using beam-search hypotheses to estimate expected word errors.

  • B) Minimum Word Error Rate Training: MWER minimizes expected word errors instead of the negative log probability P(y|x), reducing the mismatch between training and evaluation criteria.Its risk function measures word-level edit distance between each hypothesis and the reference transcription.
  • B) Minimum Word Error Rate Training: In practice, MWER estimates the empirical posterior with an N-best hypothesis list generated by beam-search decoding.The exact posterior expectation is computationally intractable, so beam-search hypotheses provide the approximation.
  • B) Minimum Word Error Rate Training: MWER improves accuracy for AED, RNN-T, and hybrid autoregressive transducer models.The reported gain is less significant than in hybrid models, possibly because E2E models are already sequence-level optimized.

VI. ADAPTATION

Adaptation methods address speaker and domain mismatch in E2E ASR, with domain adaptation relying heavily on text-only resources because paired speech-text data is difficult to obtain. Recent approaches reduce external computation or preserve source-domain knowledge through text, synthesized, or spliced data.

  • Speaker Adaptation: Speaker adaptation targets a specific speaker by modifying E2E acoustic encoders or augmenting speech inputs with speaker embeddings.
  • Speaker Adaptation: Small amounts of target-speaker data motivate regularization and synthesized-speech approaches for speaker adaptation.Synthesized speech can be combined with original target-speaker speech during adaptation.
  • Domain Adaptation: Domain adaptation addresses content mismatch between source and target domains, but obtaining sufficient paired speech-text data in the new domain is difficult.Consequently, mainstream E2E domain-adaptation methods focus on new-domain text only.
  • Domain Adaptation: External-LM fusion adapts E2E models using new-domain text, with shallow fusion remaining the simplest and most popular approach.Other approaches include MAP decoding, density-ratio methods, ILME, ILMT, and MWER training with LM fusion.
  • Domain Adaptation: TTS-based adaptation avoids requiring real target-domain speech but increases computational cost and can mismatch real speech, reduce speaker variation, or impair recognition accuracy.A mapping network and spliced-data method were proposed to address these issues; spliced data concatenates source-training audio segments corresponding to new-domain text.
  • Domain Adaptation: Text-only adaptation updates an RNN-T prediction network as an LM, while lifelong-learning methods mix source and new-domain data or regularize updates to reduce forgetting.ILMT can make the internal RNN-T LM behave more like a standalone neural LM before text-only adaptation.

C) Customization

Customization uses user context to bias E2E ASR toward personally relevant terms such as contacts, locations, and playlists. Because large phrase lists challenge bias attention, fusion and contextual spelling-correction methods provide alternatives.

  • Customization: Customization leverages user-specific context to improve recognition of terms such as contacts, locations, and music playlists.Context activation phrases such as “call,” “email,” and “text” can make biasing more effective.
  • Customization: A context-bias encoder adds contextual phrase-list information alongside the audio encoder so decoder outputs depend on speech, prior labels, and context.
  • Customization: Bias attention becomes challenging when the contextual phrase list exceeds about 1000 phrases, motivating shallow fusion with a contextual biasing LM.Fusion-based biasing commonly benefits from a prefix, which may not always be available.
  • Customization: Contextual spelling correction uses ASR-hypothesis and contextual-phrase-list embeddings to generate a revised word sequence for very large bias lists.A filtering mechanism uses distance between the ASR hypothesis and contextual candidates.

A) Non-Autoregressive Models

Non-autoregressive E2E ASR predicts tokens in parallel or iteratively without depending on earlier predicted tokens, offering faster decoding than autoregressive models. Its main limitations are weaker sequence modeling and, for basic Mask CTC, inability to correct insertion and deletion errors.

  • A) Non-Autoregressive Models: Non-autoregressive models generate target tokens simultaneously or iteratively without relying on tokens predicted in earlier steps.They assume encoder features contain acoustic and some language-semantic information.
  • A) Non-Autoregressive Models: Parallel token prediction makes non-autoregressive decoding much faster than autoregressive decoding, but performance is generally worse.The weaker performance is linked to the limited strength of the language-semantic assumption.
  • A) Non-Autoregressive Models: A typical non-autoregressive decoder predicts or fixes the target length, assumes token independence, and selects the highest-probability token at each position.
  • A) Non-Autoregressive Models: Mask CTC initializes a sequence from CTC outputs, masks low-confidence tokens, and iteratively predicts them using unmasked tokens and the speech sequence.Its final sequence length initially matches the CTC output length.
  • A) Non-Autoregressive Models: Basic Mask CTC handles substitution errors but not insertion or deletion errors because its final sequence length equals the initial CTC output length.Predicting a partial target sequence was proposed to extend handling to deletion and insertion errors.
  • A) Non-Autoregressive Models: Unified E2E models can be configured for different streaming, latency, and computational-cost requirements at runtime.Dynamic sparsity, dynamic encoders, and variable-context training support these configurations.

C) Two-pass Models

Two-pass E2E ASR improves first-pass recognition by adding text-based or speech-aware second-pass processing, while cascaded encoders unify streaming and non-streaming inference. For overlapping speech, PIT-based systems face scalability and duplication issues, motivating serialized, streaming, and cost-reduced alternatives.

  • Two-pass models: Second-pass models improve a strong single-pass E2E system by correcting hypotheses or converting syllable sequences into characters with text-based language modeling.Text-only second-pass models can exploit more text data than speech-text training alone.
  • Two-pass models: A speech-aware second pass lets streaming RNN-T provide immediate results while an AED model attends to encoder outputs for better accuracy with small additional perceived latency.Deliberation models extend this approach with LSTM AED or Transformer AED decoders.
  • Two-pass models: Cascaded RNN-T encoders provide unified streaming and non-streaming solutions by routing causal outputs to the first pass and non-causal outputs to the second pass.The model is trained in one stage, unlike deliberation models trained in two stages.
  • Two-pass models: Second-pass processing adds latency and computational costs, so commercial systems require designs that hide or reduce these costs.
  • Multi-talker models: Overlapping-speech ASR remains difficult because symmetric mixtures create a permutation problem, prompting PIT-based direct ASR and later multi-talker E2E architectures.The initial PIT-based systems separate speech representations before generating recognition results through speaker branches.
  • Multi-talker models: PIT-ASR requires a fixed number of output branches, has O(S3) Hungarian-algorithm training cost, and may produce duplicated hypotheses; SOT simplifies training with one merged output sequence.SOT inserts a speaker-change symbol and orders reference sequences by start time.
  • Multi-talker models: Streaming multi-talker ASR is pursued with SURT, which uses mask-based unmixing to create recognition branches for RNN-T.HEAT reduces training cost without losing accuracy and is especially useful for continuous streaming conversations.

E) Multi-channel Models

Multi-channel E2E ASR replaces conventional beamforming with a multi-channel encoder that directly consumes microphone-array inputs. This approach preserves the benefits of neural joint optimization while simplifying front-end design.

  • Multi-channel Transformer ASR: Multi-channel Transformer ASR directly consumes microphone-array inputs through a multi-channel encoder instead of a conventional beamformer.The encoder uses cascaded within-channel self-attention and cross-channel attention blocks.
  • Results: E2E multi-channel models were reported to outperform standard beamforming followed by an ASR model.
  • Public evaluation: Librispeech evaluations show rapid WER reductions, with trends from CTC to AED/RNN-T and from LSTM to Transformer encoders.The reviewed works include both non-streaming and streaming E2E models.
  • Deployment considerations: Production comparisons must consider model size and latency because lower WER alone does not always indicate superiority.These factors especially affect streaming E2E models.

IX. CONCLUSIONS AND FUTURE DIRECTIONS

The paper surveys E2E ASR advances aimed at closing practical gaps with hybrid systems and summarizes emerging directions for broader deployment. It identifies unresolved challenges in data use, knowledge integration, customization, and low-resource training.

  • Overview: The overview covers technologies intended to help E2E models surpass hybrid models academically and potentially replace them in industry.
  • Emerging models: Advanced E2E directions include non-autoregressive, unified, and two-pass models targeting faster or more flexible inference.Non-autoregressive models perform one-shot inference; unified models support configurable runtime requirements; two-pass models combine streaming and non-streaming advantages.
  • Broader integration: E2E modeling is also being extended to unify speech separation, signal processing, speaker identification, diarization, and speech translation.
  • Data utilization: A major unresolved challenge is using large text-only corpora to improve E2E ASR without relying only on language-model fusion.Synthesizing TTS audio at industry scale is costly, and TTS audio can sometimes degrade recognition on real speech.
  • Knowledge integration: Standard E2E models may lack knowledge needed to normalize expressions such as “a quarter to six” into “5:45” without representative training examples.
  • Customization: E2E customization can bias recognition toward contextual phrase lists but does not fully solve recognition of arbitrary unseen words.
  • Low-resource languages: Low-resource E2E training remains challenging, while self-supervised learning is expected to become closely coupled with E2E models.
Loading 2111.01690v2…