Contextual RNN-T For Open Domain ASR

1. Introduction

• E2E RNN-T maps audio to text with one neural network, unlike hybrid systems whose acoustic, language, and pronunciation models are trained separately.The introduction frames this simpler training and inference pipeline against E2E difficulty with words rarely seen in paired audio-text training data.Source: §1 · p. 1
• Rare words that occur infrequently in training can be misrecognized by a vanilla RNN-T, as illustrated by PyTorch being transcribed as “pie towards.”The example suggests metadata containing PyTorch could help the system recognize the intended word.Source: §1 · p. 1
• Acoustically similar names such as Sean and Shaun can be difficult to distinguish when they appear with similar frequencies and contexts in training.Related text metadata may favor the intended name.Source: §1 · p. 1
• The approach biases ASR with accompanying video text metadata, which may be unstructured or irrelevant and therefore must be selectively used or ignored.The evaluation focuses on entity names because the paper says they carry high semantic meaning relative to other transcript words.Source: §1 · p. 1

2. Prior Work

• Prior work used contextual words through on-the-fly rescoring or as additional DNN input, called shallow fusion and deep contextualization, respectively.This work follows the deep contextualization approach.Source: §2 · p. 1
• CLAS used unpaired context words for closed-domain tasks such as virtual-assistant contact names; later work added phonetic representations and.This paper differs by applying deep contextualization to RNN-T in open-domain video ASR with noisy metadata.Source: §2 · p. 1
• RNN-T deep contextualization was also studied for keyword spotting using one-hot phoneme sequences [13], while another video-metadata approach rescored hybrid-ASR lattices.The current work instead uses metadata as context in an RNN-T model.Source: §2 · p. 2

3. RNN Tranducer

• The RNN-T has three components: an audio encoder, a text predictor, and a joiner.In this work, the audio encoder is a stack of bidirectional LSTM layers, while the text predictor is a stack of LSTM layers.Source: §3 · p. 2
• The audio encoder maps audio frames to embeddings, and the text predictor maps the last non-blank target unit to a text embedding.The text predictor uses sentence pieces as target units.Source: §3 · p. 2
• The joiner combines the audio and text-predictor outputs into an embedding, then a linear transformation and softmax produce probabilities over target units and a blank symbol.The projections align audio and text embeddings in dimension before their combination.Source: §3 · p. 2
• Blank outputs advance time in the audio sequence, while non-blank outputs advance the text history, creating multiple alignment paths whose probability sum gives the output-sequence probability.The paths span a lattice of size T ∗ U.Source: §3 · p. 2

4. Contextual RNN-T

• The contextual RNN-T adds an Embedding Extractor, Attention Module, and Biasing Module to the base model.The contextual information is incorporated into the Text Predictor and Joiner.Source: §4 · p. 2
• Context words are encoded as sentence-piece sequences, with a BLSTM's last state serving as each word's embedding.For example, “Jarred” may be represented as [Ja, r, re, d].Source: §4 · p. 2
• The Attention Module uses predictor output and context-word embeddings to compute attention, incorporating prior-step weights for location-aware alignment.The weighted word embeddings form a context vector that is concatenated with the Text Predictor output for the Joiner.Source: §4 · pp. 2–3
• The Biasing Module identifies context words matching the unfinished transcript prefix and uses attention-weighted sentence-piece indicators to bias the Joiner toward them.The word-level biases are summed, linearly projected, and passed through optional Dropout before reaching the Joiner.Source: §4 · p. 3
• A biasing-only variant sets each attention weight to 1, while Equations 4a and 4b remain unchanged.This variant modifies the Joiner equation to use the bias vector without computing attention weights.Source: §4 · p. 3

5.1. Dataset

• The de-identified public Facebook-video data includes about 8k training hours and 170 evaluation hours, segmented into 10-second samples with metadata from video titles and descriptions.Evaluation is divided into CommonNonZero and CommonZero according to whether reference words overlap the video's metadata words.Source: §5.1 · p. 3

5.2. Model

• The Contextual RNN-T adds an Embedding Extractor, Attention Module, and optional Biasing Module to the baseline, with both models trained from scratch.The model uses a 4-layer BLSTM audio encoder and a 2-layer LSTM text predictor; the baseline omits the added modules.Source: §5.2 · p. 3

5.3. Impact on WER-NE and WER

• On CommonNonZero, AttModule improves WER-NE about 13% relative to baseline; BiasingModule adds another 3%, while Biasing-only yields about 14%.Combining the modules further improves WER-NE.Source: §5.3 · pp. 3–4, Table 2 · p. 4
• On CommonZero, Contextual RNN-T models do not significantly impact WER or WER-NE when metadata words do not intersect the reference.Source: Table 3 · p. 4, §5.3 · p. 4
• Across both test sets, the best contextual model improves context-word recall by 8.3% and degrades precision by 1.3% relative to baseline.Source: Table 4 · p. 4, §5.3 · p. 4

• In the example, the baseline struggles with Android and PyTorch, while metadata attention accompanies the Contextual RNN-T's correct transcription of Android.Both models recognize the common entity Africa.Source: §6 · p. 5

• Noisy contextual metadata can improve named-entity recognition for open-domain social-media-video ASR within an E2E RNN-T; future work may explore image or semantic metadata embeddings.Source: §7 · p. 5