PaperlayerView exact revision on arXiv

Source-linked AI summary

Recognizing long-form speech using streaming end-to-end models

Arun Narayanan, Rohit Prabhavalkar, Chung-Cheng Chiu, David Rybach, Tara N. Sainath, Trevor Strohman

arXiv:1910.11455v1eess.AScs.CLcs.SD

TL;DR

E2E ASR models can struggle to generalize across domains and recognize long-form speech when trained on short utterances. This work evaluates diverse acoustic training data and recurrent-state manipulation to simulate long-form training. On real long-form call-center speech, diverse training improves WER by 40% relative, with simulated long-form training adding a further 27% reduction.

  • Problem

    The paper examines E2E ASR generalization to unseen domains and the difficulty of recognizing long-form speech when training uses short utterances.

    Source: §1 · p. 1, §3 · p. 2

  • Method

    The authors test diverse acoustic-domain training and manipulation of internal recurrent states to simulate long-form audio during short-utterance training.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    On real long-form call-center speech, diverse training improves WER by 40% relative, and RSP with multidomain training improves WER by 27% relative over baseline.

    Source: §1 · p. 1, §5.3 · p. 5

  • Takeaways & Limitations

    The two approaches target related but distinct issues, and RSP combined with multidomain training further improves call-center performance.

    Source: §5.3 · p. 5, §7 · p. 6

    RSP does not prevent regression on YouTube, whose acoustic dissimilarities from search training extend beyond utterance length.

    Source: §6 · p. 6

Abstract

from arXiv · show

All-neural end-to-end (E2E) automatic speech recognition (ASR) systems that use a single neural network to transduce audio to word sequences have been shown to achieve state-of-the-art results on several tasks. In this work, we examine the ability of E2E models to generalize to unseen domains, where we find that models trained on short utterances fail to generalize to long-form speech. We propose two complementary solutions to address this: training on diverse acoustic data, and LSTM state manipulation to simulate long-form audio when training using short utterances. On a synthesized long-form test set, adding data diversity improves word error rate (WER) by 90% relative, while simulating long-form training improves it by 67% relative, though the combination doesn't improve over data diversity alone. On a real long-form call-center test set, adding data diversity improves WER by 40% relative. Simulating long-form training on top of data diversity improves performance by an additional 27% relative.

1. INTRODUCTION

End-to-end ASR replaces conventional recognition components with a single neural network, but its generalization across domains—and to long-form speech after short-utterance training—is limited. The paper examines RNN-T robustness and evaluates diverse-data training and recurrent-state manipulation as responses.

1. INTRODUCTION

  • E2E models replace the acoustic, pronunciation, and language models of conventional ASR with one neural network that maps acoustic frames to output units.The paper identifies CTC, RNN-T, and attention-based encoder-decoder models as E2E examples.Source: §1 · p. 1
  • On unseen real long-form call-center speech, diverse training improves WER by 40% relative, and simulating long-form training adds a further 27% WER reduction.The added reduction is reported without significant degradation on in-domain test sets.Source: §1 · p. 1
  • Because E2E models learn system components directly from training data, generalizing across domains is challenging, despite the goal of one ASR system performing equally well across them.The paper describes domains as groups of utterances with differing acoustic and linguistic characteristics.Source: §1 · p. 1

2. RECURRENT NEURAL NETWORK TRANSDUCER

RNN-T combines an acoustic encoder, a prediction network, and a joint network to model output sequences across possible alignments. Its streaming inference uses frame-synchronous beam search.

2. RECURRENT NEURAL NETWORK TRANSDUCER

  • The encoder processes acoustic frames and the prediction network processes prior non-blank labels; a feed-forward joint network combines their representations to produce logits.The encoder and prediction network use LSTMs, and word-pieces are the output units in this work.Source: §2 · pp. 1–2
  • RNN-T assigns output-sequence probability by marginalizing over valid alignments, which include blank symbols to account for the lack of frame-level label alignments.At each step, outputs are conditioned on an acoustic prefix and previous predictions.Source: §2 · pp. 1–2
  • Training maximizes log-likelihood over the training set, with gradients computed using dynamic programming.Source: §2 · p. 2

2.1. Inference

  • Frame-synchronous beam search expands hypotheses after each input frame until blank, prunes weaker candidates, and merges alignments of the same output sequence by summing probabilities.Source: §2.1 · p. 2

3. IMPROVING LONG-FORM MODELING

The section addresses acoustic-domain mismatch and poor generalization from short training utterances to long-form audio. It explores multidomain training and LSTM state manipulation to simulate longer contexts during training.

3.2. Simulating Long-Form Characteristics During Training

  • Long-form decoding is difficult to train directly because hardware memory limits utterance length, while recurrent models face vanishing gradients and implicit alignment over long segments.Training utterances are therefore typically segmented into chunks of a few seconds.Source: §3 · p. 2, §3.2 · p. 2

3.1. Increasing Data Diversity Through Multidomain Training

  • Multidomain training expands acoustic conditions and increases utterance-length diversity, addressing both acoustic mismatch and length variation.The paper explores adding a diverse set of training conditions to improve generalization.Source: §3.1 · p. 2

3.2. Simulating Long-Form Characteristics During Training

  • Training resets LSTM states between utterances, unlike long-form inference, which propagates states across time; the methods manipulate initial states to reduce this mismatch.The recurrent state carries information between steps, and models with explicit word-token conditioning must also propagate previous output tokens.Source: §3.2 · p. 3, §3.2.2 · p. 3

3.2.1. Random State Sampling (RSS)

  • Random state sampling (RSS) replaces the usual all-zero initial LSTM state with a sample from an assumed distribution, taken here to be a random normal distribution.At inference, LSTM states are initialized to zero, the expected value of the training initial states.Source: §3.2.1 · p. 3

3.2.2. Random State Passing (RSP)

  • Random state passing (RSP) initializes each training example from saved final states of prior mini-batch examples and carries their final output tokens forward, simulating concatenated utterances and transcripts.The figure illustrates passing the first utterance’s last LSTM state and output token to the second utterance.Source: §3.2.2 · p. 3, Figure 2 · p. 3
  • RSP propagates states forward across mini-batches without back-propagating gradients between them, and resets the states to zero with probability 0.5.This state propagation is described as similar in spirit to truncated back propagation through time.Source: §3.2.2 · p. 3

4. EXPERIMENTAL SETTINGS

The experiments evaluate streaming RNN-T models using different training sets and test conditions. The setup describes the model architecture and training, then compares datasets that vary in domain, acoustic conditions, and utterance length.

4.1. Model Architecture

  • The acoustic frontend uses 128-dimensional log-mel features, with four contiguous frames stacked and subsampled by three to provide inputs at a 30 msec frame rate.All experiments use multi-condition training and 8 kHz and 16 kHz versions of training data.Source: §4.1 · p. 3
  • The RNN-T uses an 8-layer unidirectional LSTM encoder and a 2-layer prediction network, with 2,048 units projected to 640 per layer and a 640-unit joint layer.Word-piece targets use a vocabulary of 4,096 tokens.Source: §4.1 · p. 3
  • The models have 120 million trainable parameters and are trained with 8 × 8 TPUs, mini-batches of 4,096, Adam, and synchronized stochastic gradient descent.The implementation uses TensorFlow and Lingvo.Source: §4.1 · p. 4

4.2. Datasets

  • Training data cover voice-search and far-field speech, segmented telephony, and semi-supervised YouTube segments; experiments train on either Search alone or all domains.The training-data table reports utterance-length statistics.Source: §4.2 · p. 4, Table 1 · p. 4
  • Evaluation includes matched-acoustic-condition sets of varying lengths, longer YouTube utterances, synthesized single-voice audiobook passages of at least 200 words, and unsegmented long call-center speech.The test-data table reports utterance-length statistics.Source: §4.2 · p. 4, Table 2 · p. 4

5. RESULTS

Search-only training generalizes poorly to long-form and acoustically distinct speech. The results show that data diversity and LSTM state manipulation improve performance in different settings, with further gains from combining them on multidomain training.

5.1. Generalization of Search Models on Long-Form Sets

  • Search-only training yields 69.1% WER on YouTube and 49.0% on TTS-Audiobook, with markedly high deletion rates on long or acoustically distinct sets.The analysis indicates the model often predicts blanks instead of valid words on these sets.Source: §5.1 · p. 4, Table 3 · p. 5

5.2. Multidomain Training: Impact of Data Diversity

  • Count-Weighted sampling performs best for E2E ASR; oversampling smaller domains or subdomains leads to rapid overfitting.The reported sampling-strategy results use a different word-piece vocabulary and are not directly comparable with other paper results.Source: §5.2 · p. 5, Table 4 · p. 5, §5.3 · p. 5
  • 40% relative WER improvement on the call-center set accompanies multidomain training; TTS-Audiobook WER falls from 49.0% to 4.7%.The multidomain model still has high call-center deletions, likely reflecting acoustic differences and longer pauses.Source: §5.2 · p. 5, Table 5 · p. 5

5.3. Simulating Long-Form Utterances During Training

  • RSP improves TTS-Audiobook WER by 67% relative with search-only training, but YouTube WER remains 61.7%.On TTS-Audiobook, state passing reduces deletion rates from 43.6 to 6.9 percentage points.Source: §5.3 · p. 5, Table 6 · p. 5
  • 27% relative WER improvement on call-center utterances comes from RSP over the multidomain baseline.RSP also reduces the call-center deletion rate from 26.5% to 17.1%.Source: §5.3 · p. 5

6. ANALYSIS: GENERALIZATION DURING TRAINING

The analysis tracks generalization as training proceeds across in-domain and out-of-domain test sets. In-domain improvement can coincide with worsening unseen-domain performance, while multidomain training and RSP alter those trends.

6. ANALYSIS: GENERALIZATION DURING TRAINING

  • After 50k steps, search-trained RSP reaches 34.5% WER on YouTube, then degrades severely; TTS-Audiobook performance is more stable but worsens slightly after 100k steps.The figure contrasts baseline and RSP models trained on search or multidomain data across four test sets.Source: §6 · p. 6, Figure 4 · p. 6
  • With multidomain training, RSP degrades more slowly than baseline on Call-center and achieves significantly better performance at convergence.Both models improve over training on Search, TTS-Audiobook, and YouTube.Source: §6 · p. 6, Figure 4 · p. 6
  • Multidomain-trained models regress less across diverse test conditions than search-trained models, including on Call-center.The analysis attributes the broader stability to training with a variety of data.Source: §6 · p. 6, Figure 4 · p. 6

7. CONCLUSIONS

Streaming RNN-T models are susceptible to domain mismatch and struggle with long-form speech when trained only on short segments. The paper evaluates greater training-data diversity and recurrent-state manipulation as remedies, reporting significant improvements on unseen domains with long utterances.

  • Both proposed techniques are complementary and yield significant performance improvements on unseen domains containing long utterances.Source: §7 · p. 6
  • RNN-T models are particularly susceptible to domain mismatches between training and inference, while exposure only to short training segments limits long-form recognition at inference.Source: §7 · p. 6
  • The paper evaluates increasing training-data diversity and simulating long-form audio by manipulating the model’s internal recurrent state.Source: §7 · p. 6
Loading 1910.11455v1…