Source-linked AI summary
Cold Fusion: Training Seq2Seq Models Together with Language Models
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, Adam Coates
TL;DR
Seq2Seq systems need to exploit abundant unlabeled language data without forcing their decoders to relearn redundant, domain-biased language models. Cold Fusion trains a Seq2Seq model together with a fixed pre-trained language model, yielding faster convergence, improved generalization, and near-complete speech-recognition domain transfer with less than 10% labeled data.
Problem
Independently trained fusion makes the Seq2Seq decoder learn redundant language knowledge from limited, domain-specific labels, wasting capacity and hindering adaptation.
Method
Cold Fusion trains the Seq2Seq model from scratch together with a fixed pre-trained language model so the decoder learns to use external language information during training.
Results
Cold Fusion reduces word error rates by up to 18% versus Deep Fusion and nearly fully transfers to a new domain using only 10% of labeled data.
Takeaways & Limitations
Cold Fusion enables better generalization and easier domain transfer while allowing the decoder to focus on task-relevant information.
Takeaways & Limitations
Deep Fusion cannot swap language models because of the state discrepancy issue, so its models use the same language model for training and evaluation.
Abstract
from arXiv · showhide
Sequence-to-sequence (Seq2Seq) models with attention have excelled at tasks which involve generating natural language sentences such as machine translation, image captioning and speech recognition. Performance has further been improved by leveraging unlabeled data, often in the form of a language model. In this work, we present the Cold Fusion method, which leverages a pre-trained language model during training, and show its effectiveness on the speech recognition task. We show that Seq2Seq models with Cold Fusion are able to better utilize language information enjoying i) faster convergence and better generalization, and ii) almost complete transfer to a new domain while using less than 10% of the labeled training data.
1. Introduction
Seq2Seq models can benefit from abundant unlabeled text through language models, but independently trained fusion methods leave the decoder learning redundant and domain-biased language knowledge. Cold Fusion trains the Seq2Seq decoder with a fixed pre-trained language model, improving training speed and new-domain adaptation.
- Motivation: Language models trained on abundant unsupervised text can provide rich linguistic information that improves Seq2Seq performance.Such corpora can contain as many as one billion tokens.
- Related approaches: Shallow Fusion combines task-specific and language-model scores only during inference, while Deep Fusion combines their hidden states after independent training.Deep Fusion uses a learned gating mechanism to control the contribution of each model.
- Limitations: Independent training makes the Seq2Seq decoder learn a redundant language model from limited labels, consuming decoder capacity and biasing it toward the parallel corpus domain.This bias complicates adaptation when deployment and training domains differ.
- Cold Fusion: Cold Fusion trains the Seq2Seq decoder to use an external language model during training rather than learning language knowledge independently.The paper presents this as a way to leverage unsupervised text while focusing decoder learning on task-relevant information.
- Reported benefits: Cold Fusion is reported to train faster and almost completely transfer speech recognition to a new domain with 10 times less labeled data.The experiments focus on speech recognition and domain adaptation.
2. Background and Related work
Seq2Seq models map input sequences to output sequences, and language models can be integrated during decoding through shallow or deep fusion. These approaches improve language modeling but differ in whether fusion occurs only at inference or after independently training the component models.
- Seq2Seq background: A Seq2Seq encoder maps an input sequence to an intermediate representation, while an attention-equipped decoder generates a variable-length output sequence.For speech recognition, the model maps spectrogram features to characters.
- Inference: Inference seeks the most likely output sequence, approximated in practice with left-to-right beam search that retains the top K partial hypotheses.Decoding stops when the end symbol is emitted.
- Language-model integration: Shallow Fusion incorporates a language model by combining its sequence probability with the task-specific Seq2Seq probability during inference.The language model is not used to train the Seq2Seq model in this approach.
- Language-model integration: Deep Fusion combines the hidden states of independently trained Seq2Seq and language models using a parametric gate trained on a small amount of data.The gate determines how much each model contributes at each time step.
- Limitations: Deep Fusion wastes decoder capacity because the task-specific model must learn a language model from comparatively parsimonious training labels before fusion.The fused output layer must then overcome this bias to incorporate new language information.
- Other approaches: Unlabeled-data methods include backtranslation and unsupervised pre-training, but backtranslation transfers poorly to tasks such as speech recognition and pre-training may erase language-model knowledge during parallel training.Both methods avoid architectural changes.
3. Cold Fusion
Cold Fusion trains a Seq2Seq decoder from scratch alongside a fixed pretrained language model, encouraging it to use external language information during training. The method combines task and language-model signals through a gated fusion layer and speeds convergence.
- Cold Fusion trains the Seq2Seq model from scratch together with a fixed pretrained language model.
- The method encourages the decoder to use the language model for language-specific information while learning source-to-target mappings.This can increase effective model capacity and supports strong performance with a very small decoder.
- Cold Fusion improves the fusion mechanism with task and language-model states, fine-grained gating, and language-model probabilities as inputs.The task state helps determine reliance on the language model under input uncertainty, while fine-grained gates select individual hidden-node contributions.
- Using language-model probabilities instead of hidden states supports integration of different language models and permits training with or swapping in n-gram models during inference.
- The fused state is transformed by a deep neural network before producing the output distribution.The experiments used a single affine layer with ReLU activation before softmax.
- Training with a language model speeds convergence considerably compared with the baseline.Figure 1 plots dev-set cross-entropy loss against training iteration for the baseline and proposed models.
4. Experiments
Experiments on speech recognition show that Cold Fusion improves in-domain performance, generalization, domain adaptation, decoder efficiency, and training speed. It also approaches the target-domain performance of models trained directly on target data using very little labeled data.
- Setup: The experiments evaluate character error rate and word error rate on source-domain and target-domain held-out speech-recognition sets.The source domain uses search-query speech, while the target domain uses movie-script speech.
- Training and evaluation: Cold Fusion consistently outperforms the baselines on both metrics across source and target domains.On in-domain prediction, the baseline WER is 14.68%, while the best Cold Fusion model improves relatively by more than 21% and by 15% over Deep Fusion.
- Improved Generalization: 38.17% is the remaining domain gap fraction for Cold Fusion, compared with 76.57% for Deep Fusion.Applying the same architectural changes to Deep Fusion yields little improvement, while probability projection substantially helps Cold Fusion generalization.
- Decoder Efficiency: Cold Fusion still outperforms the full attention decoder with 4× fewer parameters as decoder capacity decreases.Its performance degrades gradually with smaller decoder cells, unlike attention models whose performance deteriorates abruptly beyond a point.
- Training: Training is accelerated by a factor of 3 with Cold Fusion.The pretrained fixed language model provides lower-level language features, allowing error signals to propagate more directly into the attention context.
- Fine-tuning for Domain Adaptation: With 0.6% of labeled target data, the domain gap decreases from 38.2% to 21.3%, and with less than 10% it falls to only 8%.During fine-tuning, the Seq2Seq parameters remain fixed, so improvements come from combining acoustic and language-model information through the fusion mechanism.
- Fine-tuning for Domain Adaptation: Cold Fusion produces better out-of-domain grammatical completion than vanilla attention and Deep Fusion on movie-script speech.The source training data consists of read search queries, whose sentence fragments make weak grammar knowledge a prominent error mode.
5. Conclusion
Cold Fusion is a Seq2Seq architecture that trains a decoder together with a pre-trained language model. It reduces WER relative to Deep Fusion, transfers more easily to new domains, and nearly fully transfers with only 10% of labeled data.
- 5. Conclusion: Cold Fusion trains a Seq2Seq decoder together with a pre-trained language model.The method includes architectural changes intended to help the model fully leverage language-model information.
- 5. Conclusion: Cold Fusion reduces word error rates by up to 18% compared to Deep Fusion.The paper also reports easier transfer to new domains, with nearly full transfer using only 10% of labeled data.