Source-linked AI summary
Prompt-tuning in ASR systems for efficient domain-adaptation
Saket Dingliwal, Ashish Shenoy, Sravan Bodapati, Ankur Gandhe, Ravi Teja Gadde, Katrin Kirchhoff
TL;DR
Maintaining domain-specific transformer language models for ASR is costly, while a generic model performs worse in-domain. This paper tests prompt-tuning as a small-parameter alternative and reports improved perplexity and fastfood ASR CWER close to fine-tuning.
Problem
Domain-specific LMs perform better than generic models in-domain, but maintaining multiple parameter-heavy Transformer LMs for ASR rescoring is practically infeasible.
Source: §1 · p. 1Method
The approach learns domain-specific prompt token embeddings that prime a pretrained language model, rather than fine-tuning all of its parameters.
Source: §1 · p. 1Results
With 5 prompt-tokens, perplexity improved from 130 in the out-of-the-box model to 37; in fastfood ASR, 10 prompt tokens achieved 5.8% relative CWER improvement over noadaptation and 400 almost exactly matched fine-tuning.
Source: §2 · p. 2Takeaways & Limitations
A single base model can be retained while saving domain-specific prompt token embeddings instead of maintaining separate language-model copies for each domain.
Source: §2 · p. 2The experiments used complete training data, and the authors plan further WER evaluation across multiple domains and datasets.
Source: §3 · p. 2
Abstract
from arXiv · showhide
Automatic Speech Recognition (ASR) systems have found their use in numerous industrial applications in very diverse domains. Since domain-specific systems perform better than their generic counterparts on in-domain evaluation, the need for memory and compute-efficient domain adaptation is obvious. Particularly, adapting parameter-heavy transformer-based language models used for rescoring ASR hypothesis is challenging. In this work, we overcome the problem using prompt-tuning, a methodology that trains a small number of domain token embedding parameters to prime a transformer-based LM to a particular domain. With just a handful of extra parameters per domain, we achieve much better perplexity scores over the baseline of using an unadapted LM. Despite being parameter-efficient, these improvements are comparable to those of fully-fine-tuned models with hundreds of millions of parameters. We replicate our findings in perplexity numbers to Word Error Rate in a domain-specific ASR system for one such domain.
1 Introduction
Maintaining domain-specific language models for ASR is costly, while a generic model performs worse in-domain. The paper applies prompt-tuning to adapt a language model to different domains using learned prompt embeddings.
1 Introduction
- Domain-specific LMs can outperform generic models in-domain, but maintaining multiple parameter-heavy Transformer LMs is practically infeasible.The section frames the need for a middle ground between performance and costs.Source: §1 · p. 1
- Prompt-tuning learns prompt token embeddings from labeled examples to prime a language model for a task or domain.The prompt is appended to the input rather than manually supplied or learned by fine-tuning the Transformer itself.Source: §1 · p. 1
- The authors present this as the first application of prompt-tuning to ASR domain adaptation and report that prompt-tuned models can approach fine-tuned performance with fewer than 0.1% of the parameters.They evaluate prompt-tuned and fine-tuned language models on sentences from different domains of a dialog dataset and ablate prompt size.Source: §1 · p. 1
2 Experiments
The experiments assess prompt-tuning for language-model domain adaptation and ASR rescoring. Prompt-tuned models improve perplexity over an unadapted model and, in fastfood ASR, approach fine-tuned CWER with 400 prompt tokens.
2 Experiments
- The proof-of-concept experiments fine-tuned and prompt-tuned pretrained GPT-2 on each MultiDoGo domain, using train, dev, and test splits and comparing against unadapted GPT-2.They evaluated perplexity and performance, memory, and computation time; the authors report that fastfood results were consistent across domains.Source: §2 · pp. 1–2
- With 5 prompt-tokens, perplexity improved from 130 in the out-of-the-box model to 37; with 400 prompt-tokens, it was very similar to the fully fine-tuned model.Figure 1 compares performance, computation, and memory costs; the prose reports that larger prompts also increase memory and computation costs.Source: Figure 1 · p. 2, §2 · p. 2
- Increasing prompt size improved performance but also increased memory and computation costs, making prompt size a control for the trade-off.Source: §2 · p. 2
- 7.3% relative CWER improvement over noadaptation came from fine-tuning, while 10 prompt tokens achieved 5.8% and 400 prompt tokens almost exactly matched fine-tuning.The CWER comparison uses an internal fastfood audio dataset and a hybrid acoustic model.Source: §2 · p. 2
3 Future Work
The authors describe the experiments as an ongoing proof of concept and identify broader evaluation and initialization choices as future work. Their experiments used the complete training data, leaving limited-domain-data performance unresolved.
3 Future Work
- The authors plan further WER experiments across multiple domains and datasets to examine prompt-tuning's benefits in ASR.They characterize the current work as ongoing and preliminary.Source: §3 · p. 2
- The experiments used the complete training data, so whether prompt-tuned models outperform fine-tuned models with limited domain data remains a hypothesis.The authors suggest fewer learnable parameters might help in that setting, particularly for new ASR domains where high-quality data is difficult to obtain.Source: §3 · p. 2
- Future work includes testing prompt initialization choices, such as embeddings for a domain's most frequent words, and comparing against several other adaptation approaches.The approaches named include AdapterHub, meta-learning, semantic embeddings, and freezing all but the Transformer's last layer.Source: §3 · p. 2