PaperlayerView exact revision on arXiv

Source-linked AI summary

A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition

Erik McDermott, Hasim Sak, Ehsan Variani

arXiv:2002.11268v3eess.AScs.CLcs.SD

TL;DR

End-to-end ASR lacks an independent language-model component, despite the availability of substantially more text-only data and the value of adapting language models to specific contexts. The paper applies Bayes’ rule to combine a source-trained RNN-T with source- and target-domain LMs, and reports that Density Ratio consistently outperforms Shallow Fusion in the examined settings.

  • Problem

    End-to-end ASR models do not include independently trained language-model components, although text-only data is typically orders of magnitude more abundant than paired audio-transcript data.

    Source: §1 · p. 1

  • Method

    The Density Ratio approach applies Bayes’ rule to end-to-end models and separate language models to estimate a target-domain posterior.

    Source: §8 · p. 6

  • Results

    Across examined cross-domain scenarios, Density Ratio consistently outperformed Shallow Fusion; without fine-tuning, it achieved up to 28% relative WER reduction (17.5% →12.5%) versus up to 17% (17.5% →14.5%).

    Source: §7 · p. 6

  • Takeaways & Limitations

    Language-model fusion significantly boosts Voice Search performance when only limited target-domain fine-tuning data is used.

    Source: §7 · p. 6

    The derivation assumes acoustic consistency between domains: pψ(X|W) is roughly equal to pτ(X|W).

    Source: §3.1 · p. 2

Abstract

from arXiv · show

This article describes a density ratio approach to integrating external Language Models (LMs) into end-to-end models for Automatic Speech Recognition (ASR). Applied to a Recurrent Neural Network Transducer (RNN-T) ASR model trained on a given domain, a matched in-domain RNN-LM, and a target domain RNN-LM, the proposed method uses Bayes' Rule to define RNN-T posteriors for the target domain, in a manner directly analogous to the classic hybrid model for ASR based on Deep Neural Networks (DNNs) or LSTMs in the Hidden Markov Model (HMM) framework (Bourlard & Morgan, 1994). The proposed approach is evaluated in cross-domain and limited-data scenarios, for which a significant amount of target domain text data is used for LM training, but only limited (or no) {audio, transcript} training data pairs are used to train the RNN-T. Specifically, an RNN-T model trained on paired audio & transcript data from YouTube is evaluated for its ability to generalize to Voice Search data. The Density Ratio method was found to consistently outperform the dominant approach to LM and end-to-end ASR integration, Shallow Fusion.

1. INTRODUCTION

End-to-end ASR models lack separately trained language-model components, even though text-only data is often far more abundant than paired speech data. The article introduces a Bayes-grounded Density Ratio approach and reports that it consistently outperforms Shallow Fusion in the examined scenarios.

1. INTRODUCTION

  • End-to-end models directly estimate P(W|X), but lack independently trained acoustic-only or symbols-only components that give separate language models a clear role.The paper frames separate LMs as useful because text-only data can greatly exceed paired audio-transcript data and can support context-specific vocabulary adaptation.Source: §1 · p. 1
  • Shallow Fusion interpolates end-to-end and separate LM scores at decoding time, while Density Ratio extends its simplicity with a Bayes’ rule grounding.Deep, Cold, and Component Fusion jointly train with a pretrained LM, but had not displaced Shallow Fusion as the community’s go-to approach.Source: §1 · p. 1
  • The Density Ratio method consistently outperformed Shallow Fusion in cross-domain and limited Voice Search fine-tuning scenarios.The experiments used a YouTube-trained RNN-T, Voice Search speech, and Voice Search transcripts for language-model training.Source: §1 · p. 2

2. A BRIEF HISTORY OF LANGUAGE MODEL INCORPORATION IN ASR

The Noisy Channel Model uses Bayes’ rule to combine acoustic and language models, providing a framework for ASR decoding. It separates the acoustic likelihood from the language-model prior.

2. A BRIEF HISTORY OF LANGUAGE MODEL INCORPORATION IN ASR

  • The Noisy Channel Model combines acoustic likelihood p(X|W) and symbol-sequence prior P(W) with Bayes’ rule to obtain P(W|X).The posterior is the quantity used for ASR decoding.Source: §2 · p. 2
  • The model represents audio as feature sequence X and the recognized word or subword sequence as W.Possible alignments and a prior over alignments can be used in the formulation.Source: §2 · p. 2
  • The framework separates the acoustic model p(X|W) from the language model P(W), although the passage notes its limited discriminative power.Source: §2 · p. 2

Hybrid model for DNNs/LSTMs within original ASR

DNNs and LSTMs produce state posteriors rather than acoustic likelihoods, prompting a hybrid approach that rescales posteriors using state priors. End-to-end LM fusion instead commonly uses Shallow Fusion, a score interpolation without a direct probability-theoretic interpretation.

Hybrid model for DNNs/LSTMs within original ASR

  • The hybrid model uses scaled DNN/LSTM posteriors—posteriors divided by separately estimated state priors—to obtain likelihood-like scores.These neural models produce phoneme- or state-level posteriors rather than acoustic likelihoods.Source: Hybrid model for DNNs/LSTMs within… · p. 2
  • The resulting pseudo-generative score can be combined with an arbitrary language-model prior for ASR decoding.The passage describes this hybrid approach as state of the art for much of the ASR community.Source: Hybrid model for DNNs/LSTMs within… · p. 2
  • Shallow Fusion linearly interpolates end-to-end and separately trained LM scores, often adding a sequence-length reward, but lacks a direct probability-theoretic interpretation.Source: Hybrid model for DNNs/LSTMs within… · p. 2

3. LANGUAGE MODEL INCORPORATION INTO END-TO-END ASR, USING BAYES’ RULE

The paper derives a target-domain posterior for end-to-end ASR by combining a source-domain model with source- and target-domain language models. It applies this Density Ratio formulation to RNN-T decoding and contrasts it with Shallow Fusion.

3.1. A Sequence-level Hybrid Pseudo-Generative Model

  • The Density Ratio derivation estimates the target posterior using source and target domain models, assuming the domains are acoustically consistent.The method is presented as a sequence-level extension of the classic hybrid model and as Bayes’ rule applied to end-to-end models.Source: §3.1 · p. 2

3.2. Top-down fundamentals of RNN-T

  • RNN-T defines P(W|X) by summing posterior probabilities over possible alignments of output symbols to acoustic frames.Its symbol predictions are produced by acoustic and label encoders combined in a joint network, and decoding can use Viterbi-based beam search.Source: §3.2 · p. 3

3.3. Application of Shallow Fusion to RNN-T

  • RNN-T Shallow Fusion adds the LM score only for non-blank symbol extensions, leaving blank-extension scores unchanged.Source: §3.3 · p. 3

3.4. Application of the Density Ratio Method to RNN-T

  • The RNN-T Density Ratio method estimates a pseudo-posterior for non-blank symbols using source and target LM scores; the estimate is not normalized over symbol outputs.Scaling factors for the LM scores and a non-blank reward are used in the final decoding score.Source: §3.4 · p. 3

3.5. Implementation

  • Implementation trains a source-domain end-to-end model and matching LM, then modifies RNN-T output at target-domain decoding with the target-to-source LM ratio.It is a decode-time method requiring LM scaling-factor tuning, for which a held-out set can be used.Source: §3.5 · p. 3

4. TRAINING, DEVELOPMENT AND EVALUATION DATA

The study uses YouTube audio/transcript pairs and text from YouTube and Voice Search to train its RNN-T and language models. Separate YouTube and Voice Search sets support development and evaluation.

4.1. Training data

  • The source-domain RNN-T uses about 120M YouTube utterances (190,000 hours), while its normalizing RNN-LM uses transcripts from that same set, totaling about 3B sub-word tokens.Source: §4.1 · p. 3
  • The target-domain RNN-LM is trained on 21M Voice Search transcripts, about 275M sub-word tokens.Source: §4.1 · p. 3
  • Voice Search RNN-T fine-tuning uses sets of 10K, 100K, 1M, or 21M utterance-level audio/transcript pairs.These sets correspond to approximately 10, 100, 1,000, and 21,000 hours of audio.Source: §4.1 · p. 3
  • Table 1 reports morph-level RNN-LM training-set size and test perplexity by training and testing domain.Source: Table 1 · p. 4

4.2. Dev and Eval Sets

  • The YouTube evaluation set comprises 25 hours of audio and 250,000 word tokens from 296 videos across 13 categories.Source: §4.2 · p. 4
  • Voice Search development and evaluation sets each contain about 7,500 utterances, 33,000 words, and 8 hours of audio, distinct from fine-tuning data.Source: §4.2 · p. 4

5. CROSS-DOMAIN EVALUATION: YOUTUBE-TRAINED RNN-T →VOICE SEARCH

The cross-domain experiments evaluate a YouTube-trained RNN-T on Voice Search with external language-model fusion. They tune language-model and sequence-length scaling factors on development data before reporting final evaluation results.

5. CROSS-DOMAIN EVALUATION: YOUTUBE-TRAINED RNN-T →VOICE SEARCH

  • The cross-domain setup tests a YouTube-trained RNN-T on Voice Search utterances with and without fusion to an external language model.Source: §5 · p. 4

5.1. RNN-T and RNN-LM model settings

  • The RNN-T has a 6-layer bidirectional LSTM acoustic encoder and a 1-layer LSTM label encoder, with about 340M parameters.Its acoustic features are 768-dimensional, derived from stacked log-mel vectors and subsampled to a 60-millisecond step size.Source: §5.1 · p. 4
  • The source and target RNN-LMs match the RNN-T decoder structure, and Voice Search training improves Voice Search test perplexity.The RNN-LMs were independently trained; the source LM uses YouTube transcripts and the target LM uses Voice Search transcripts.Source: §5.1 · p. 4

5.2. Experiments and results

  • With λψ = λτ, development experiments examine how WER responds to language-model scaling and sequence-length scaling for Shallow Fusion and Density Ratio.Source: §5.2 · pp. 4–5, Figure 2 · p. 4, Figure 3 · p. 5
  • A second development experiment fixes β at −0.1 and evaluates combinations of λτ and λψ without the equality constraint.Source: §5.2 · p. 5
  • Final evaluation reports WER and deletion, insertion, and substitution rates for selected scaling factors in Tables 2 and 3.The supplied passages describe the reported measures and tables but do not provide their values.Source: §5.2 · p. 5

6. FINE-TUNING A YOUTUBE-TRAINED RNN-T USING LIMITED VOICE SEARCH AUDIO DATA

The experiments fine-tune a YouTube-trained RNN-T on limited Voice Search audio/transcript data, then evaluate fusion with a Voice Search language model. The supplied passages describe the setup but not the final table values.

6. FINE-TUNING A YOUTUBE-TRAINED RNN-T USING LIMITED VOICE SEARCH AUDIO DATA

  • Fine-tuning warm-starts from the YouTube-trained RNN-T and can reach decent task performance within a few thousand steps, but further training can over-train.A held-out set can gauge over-training and guide when to stop, across differing amounts of fine-tuning data.Source: §6 · p. 5
  • The fine-tuning experiments use 10, 100, and 1,000 hours of Voice Search data, then compare Shallow Fusion and Density Ratio with a Voice Search RNN-LM trained on 21M transcripts.The source-domain RNN-LM is not fine-tuned; development data selects the scaling factors and β before final evaluation.Source: §6 · pp. 5–6
  • Table 3 reports Voice Search test results across fine-tuning-data amounts, with and without language-model fusion; its no-fine-tuning baseline carries over from Table 2.Source: Table 3 · p. 6

7. DISCUSSION

Across the examined YouTube-to-Voice-Search settings, Density Ratio consistently outperformed Shallow Fusion, including with and without target-domain fine-tuning. Fusion gains persisted with fine-tuning, though they diminished as more target-domain audio was used.

7. DISCUSSION

  • Using 100% of available Voice Search text, the experiments varied Voice Search audio from zero to 100%; fusion significantly boosted performance with limited fine-tuning data.Voice Search-specific fine-tuning and LM fusion improved the YouTube-trained RNN-T’s Voice Search performance.Source: §7 · p. 6
  • Up to 28% relative WER reduction (17.5% →12.5%) with Density Ratio exceeded Shallow Fusion’s up to 17% relative reduction (17.5% →14.5%) in the no-fine-tuning comparison.Density Ratio consistently outperformed Shallow Fusion across the examined cross-domain scenarios, with and without target-domain audio fine-tuning.Source: §7 · p. 6
  • Density Ratio had a significantly larger effective tuning “sweet spot” than Shallow Fusion, allowing larger absolute scaling-factor values.The sweet spot concerns combinations of LM scaling factor and sequence-length scaling factor.Source: §7 · p. 6
  • A full sweep of λψ and λτ improved slightly over the constrained setting λψ = λτ; optimal settings followed a roughly linear off-diagonal band.Source: §7 · p. 6
  • With 10 hours of fine-tuning, Density Ratio gave a 20% relative WER gain versus 12% for Shallow Fusion; with 1000 hours, gains were 10.5% versus 7%.At 21,000 hours, Density Ratio still improved WER from 7.8% to 7.4%, a 5% relative improvement; gains diminish as fine-tuning data increases.Source: §7 · p. 6
  • A stated weakness is the apparent need for LM-output scaling factors; the authors suggest this may reflect a more limited implicit RNN-T LM than the external RNN-LMs.Source: §7 · p. 6

8. SUMMARY

The paper proposes Density Ratio as a Bayes’ rule-based alternative to Shallow Fusion for incorporating an external LM into cross-domain end-to-end RNN-T ASR. In YouTube-to-Voice-Search evaluation, it consistently outperformed Shallow Fusion, with gains that persisted but diminished as target-domain fine-tuning data increased.

8. SUMMARY

  • Density Ratio was proposed and experimentally evaluated as an alternative to Shallow Fusion for incorporating an external LM into an end-to-end RNN-T for a different target domain.Source: §8 · p. 6
  • The method is conceptually simple, easy to implement, grounded in Bayes’ rule, and extends the classic hybrid ASR model to end-to-end models.Source: §8 · p. 6
  • In YouTube-to-Voice-Search evaluation, Density Ratio yielded up to 28% relative word-error gains over the non-fused baseline and consistently outperformed Shallow Fusion by a significant margin.Source: §8 · p. 6
  • Density Ratio continued to give gains after paired target-domain fine-tuning, but those gains diminished as more fine-tuning data was used.Source: §8 · p. 6
  • Evaluation across a variety of cross-domain scenarios is needed to establish the method’s general effectiveness.Source: §8 · p. 6

Related papers

Loading 2002.11268v3…