Source-linked AI summary
Hybrid Autoregressive Transducer (hat)
Ehsan Variani, David Rybach, Cyril Allauzen, Michael Riley
TL;DR
Conventional ASR modularity supports independent acoustic and language modeling but contrasts with jointly optimized encoder-decoder systems. The paper proposes HAT, which combines Seq2Seq modeling with modular inference and measures internal language-model quality. On a large-scale voice-search task, HAT improves WER over RNNT and gains further from external language-model use, while the authors note unresolved behavior requiring deeper analysis.
Problem
Encoder-decoder ASR can weaken the modular lexicon and external-language-model integration of conventional systems, motivating a model that combines joint modeling with modular inference.
Method
HAT is a time-synchronous encoder-decoder that preserves external lexicon and language-model integration while providing a measurable internal language-model quality.
Results
HAT outperforms baseline RNNT by 1.4% absolute WER, a 17.5% relative gain, and a second-pass language model provides an additional 10% gain.
Takeaways & Limitations
The internal language-model measure helps assess whether external-language-model decoding is beneficial, while HAT preserves speech-recognition modularity.
Takeaways & Limitations
The authors state that further in-depth analysis is needed to confirm the source of the decoder’s observed behavior and construct models whose prior and posterior behave as expected.
Abstract
from arXiv · showhide
This paper proposes and evaluates the hybrid autoregressive transducer (HAT) model, a time-synchronous encoderdecoder model that preserves the modularity of conventional automatic speech recognition systems. The HAT model provides a way to measure the quality of the internal language model that can be used to decide whether inference with an external language model is beneficial or not. This article also presents a finite context version of the HAT model that addresses the exposure bias problem and significantly simplifies the overall training and inference. We evaluate our proposed model on a large-scale voice search task. Our experiments show significant improvements in WER compared to the state-of-the-art approaches.
1. INTRODUCTION
Conventional ASR gains modularity by factorizing acoustic and language modeling, while newer encoder-decoder models optimize mappings jointly but can weaken external lexicon and language-model integration. HAT is proposed to combine Seq2Seq modeling with modular inference and to assess when an external language model helps.
- Motivation: ASR is formulated as maximum a posteriori decoding over acoustic features and candidate word sequences.The posterior combines an acoustic likelihood with a language-model prior.
- Motivation: Conventional factorization supports independent training of acoustic and language models on different data sets.Language models can use text-only data, while acoustic models can use comparatively small amounts of transcribed audio.
- Motivation: Modular ASR also enables vocabulary augmentation, language-model adaptation, and contextual biasing through established component-model modifications.
- Motivation: Separate objectives limit joint optimization, whereas Seq2Seq models optimize a direct feature-to-label mapping at the sequence level.The newer models use recurrent architectures and character-based labels such as graphemes or wordpieces.
- Contribution: HAT combines Seq2Seq probabilistic modeling with time-synchronous inference that preserves external lexicon and language-model integration.It also provides an internal language-model quality measure for deciding whether an external language model is beneficial.
2. TIME-SYNCHRONOUS ESTIMATION OF POSTERIOR: PREVIOUS WORK
Time-synchronous ASR models define permitted alignment paths in a lattice and compute sequence posteriors by assigning probabilities to those paths and marginalizing them. Prior models differ in lattice structure, independence assumptions, and how they support external-language-model search.
- Alignment lattices: An alignment path links acoustic frames and label positions through nondecreasing time and label coordinates, with many possible paths for each input-label pair.Permitted paths form a model-specific path lattice mapped to the corresponding label sequence.
- Posterior estimation: Time-synchronous models calculate P(Y|X) by marginalizing posterior probabilities over permitted alignment paths.
- Previous models: CE uses one forced-alignment path and conditionally independent frame-level label posteriors.
- Previous models: CTC permits blank-augmented paths whose collapsed labels match the target, then marginalizes their alignment posteriors under conditional independence.
- Previous models: RNNT uses paths of length T + U − 1 and permits multiple labels in one time frame, with local posteriors conditioned on encoder and decoder representations.External-language-model inference is formulated as a search using modified posteriors, language-model weighting, and a coverage penalty.
3. HYBRID AUTOREGRESSIVE TRANSDUCER
HAT is a time-synchronous encoder-decoder model that separates blank and label posteriors, enabling internal language-model scoring and mathematically justified external-LM integration. Its alignment posterior is built from history-dependent edge probabilities and marginalized over RNNT-style paths.
- HAT formulation: HAT distinguishes blank transitions from label emissions, using a Bernoulli blank posterior and a label posterior over the vocabulary.The blank probability depends on the full history of labels and features, while label emissions are conditioned on the input and prior labels.
- HAT formulation: The encoder maps acoustic features to vectors, while the decoder maps the label sequence to label-context representations used by the joint network.The joint score combines encoder and decoder representations before normalization across labels.
- HAT formulation: HAT computes alignment-path posteriors by chaining local edge posteriors and obtains the total posterior by marginalizing over paths.The model uses the RNNT lattice definition for comparison, while alternative lattice functions remain possible.
- Internal Language Model Score: Separating blank and label posteriors produces a local and sequence-level internal language-model score based on label history rather than acoustic features.The local score removes encoder activations and is normalized into a prior distribution for the next label.
- Internal Language Model Score: The internal language-model definition is approximate in general because its exact expectation over acoustic features is difficult to compute.The paper notes that the definition is justified for special cases and otherwise may require approximation methods.
- External LM integration: The HAT inference objective subtracts the internal language-model score from the path posterior before combining it with an external language model through an FST decoder.The decoder graph can encode phone context-dependency, a pronunciation dictionary, and an external n-gram language model.
4. EXPERIMENTS
Experiments on a large-scale anonymized voice-search task evaluate HAT against time-synchronous baselines, its internal language-model behavior, and limited-context variants. HAT improves WER over RNNT, while a finite context of two matches the baseline HAT performance and supports decoder simplification.
- Experimental setup: The evaluation uses 40 M training utterances, an 8 k-utterance development set, and a 25-hour anonymized test set representative of Google traffic queries.Inputs use 256-dimensional log-Mel features, with training examples noisified using 25 noise styles.
- HAT performance: At λ2 = 0.95, significantly down-weighting HAT’s internal language model and relying more on the external language model yields the best decoding quality.The best region is observed empirically for λ1 ∈ (2.0, 3.0) and λ2 ≈ 1.0.
- HAT performance: 1.4% absolute WER improvement over baseline RNNT corresponds to a 17.5% relative gain for HAT.A second-pass language model provides an additional 10% gain; the HAT error pattern contains twice as many insertions as deletions.
- Internal language model: The internal language-model prior cost decreases early in training but later increases for both training and test curves, suggesting the decoder deviates from language-model behavior.The authors note that this behavior may reflect posterior optimization favoring parameters that do not also maximize the prior.
- Limited context: HAT models with context sizes 2 and 4 match baseline HAT performance, while context size 1 shows 12% relative WER degradation and context size 0 matches CE and CTC baselines.Because context size 2 is sufficient, the decoder can replace recurrent kernels with a |V|^2 embedding vector, trading computation for memory.
5. CONCLUSION
The HAT model aims to improve acoustic modeling while preserving speech-recognition modularity. It introduces an internal language-model quantity for analyzing encoder-decoder behavior and deciding whether an external language model is beneficial.
- HAT preserves speech-recognition modularity while advancing acoustic modeling.
- Its internal language-model quantity supports decisions about whether to equip the system with an external language model.
- The analysis finds that the decoder behaves more like a finite-context model than a language model.
- The paper identifies possible explanations but says further analysis is needed to confirm the cause and construct fully end-to-end models.
A. INTERNAL LANGUAGE MODEL SCORE
The section derives the internal language-model score by relating the transducer score to a joint probability and then applying Bayes’ rule and marginalization.
- The score is related to the joint probability of labels, acoustic features, and label history up to an additive constant.
- Bayes’ rule decomposes the joint probability into an acoustic conditional distribution and a label-history prior.
- Exponentiation and marginalization over acoustic features yield a proportionality relation because the dropped constant is independent of the label sequence.
- The derivation uses the identity P(y|y0:u) = P(y, y0:u) / P(y0:u) to connect the joint prior with the conditional language-model probability.