Source-linked AI summary
Scalable language model adaptation for spoken dialogue systems
Ankur Gandhe, Ariya Rastrow, Bjorn Hoffmeister
TL;DR
Past-user training data may not represent interactions for newly released intents, and adaptation must avoid degrading existing applications. The paper extracts n-gram counts from intent grammars and optimizes interpolation weights under a past-performance constraint. With expected WER as the constraint, new-application WER improvements exceed 10% relative for Flights Info and Recipes while past-data WER degrades only marginally.
Problem
Past-user data can be insufficient to model interactions expected with new application intents, leaving the challenge of adapting language models without degrading existing applications.
Source: §1 · p. 1Method
The paper extracts n-gram counts directly from intent grammars and uses constrained optimization to adapt language-model interpolation weights while protecting performance on past usage.
Source: §1 · pp. 1–2, §3 · p. 3Results
More than 10% relative WER improvements occur for Flights Info and Recipes with expected WER as the constraint, while past-data WER degrades only marginally.
Source: §4.2 · p. 5Takeaways & Limitations
Using WER as a constraint leads to better adaptation for new applications and removes the need for adaptation data.
Source: §5 · p. 5Evaluation used only a subset of production data, and its in-house expert test set was not reflective of actual live application usage.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Language models (LM) for interactive speech recognition systems are trained on large amounts of data and the model parameters are optimized on past user data. New application intents and interaction types are released for these systems over time, imposing challenges to adapt the LMs since the existing training data is no longer sufficient to model the future user interactions. It is unclear how to adapt LMs to new application intents without degrading the performance on existing applications. In this paper, we propose a solution to (a) estimate n-gram counts directly from the hand-written grammar for training LMs and (b) use constrained optimization to optimize the system parameters for future use cases, while not degrading the performance on past usage. We evaluated our approach on new applications intents for a personal assistant system and find that the adaptation improves the word error rate by up to 15% on new applications even when there is no adaptation data available for an application.
1. INTRODUCTION
Past-user data can poorly represent interactions for newly released intents, while adapting language models without harming existing applications remains a challenge. The paper proposes extracting n-gram counts from intent grammars and using constrained optimization to address these adaptation pressures.
1. INTRODUCTION
- Past-user training and tuning data may not represent interactions expected after a new intent launches, often reducing recognition quality on those applications.The paper describes this challenge for interactive speech recognition systems using transcribed user utterances as substantial training data.Source: §1 · p. 1
- For a new intent, a probabilistic context-free grammar can encode expected interactions; its non-terminals expand from entity catalogs, and weights assign probabilities to generated sentences.The introduction gives a cooking-recipe intent as an example.Source: §1 · p. 1
- Grammar sampling can require many generated examples when non-terminals encode exponentially many paths, while grammar–n-gram FST unions can enlarge models and increase latency as applications accumulate.The proposed direct extraction of n-gram counts from grammars avoids generating sampled data.Source: §1 · p. 1
- Prior work adapted recurrent neural language models to a single domain or used domain vectors for multi-domain models.The introduction identifies optimization across domains while maintaining past-use performance as an important missing aspect.Source: §1 · pp. 1–2
- Constrained optimization minimizes loss on a new intent while preventing degradation on applications supported by past usage.The paper also proposes extending the same constrained loss to neural language-model training.Source: §1 · p. 2
2. ESTIMATING N-GRAM MODELS FROM PCFG
The paper extracts fractional n-gram counts directly from a probabilistic grammar and uses them to estimate language models. Its method computes counts over an FST without enumerating every generated sentence, then applies smoothing to build the n-gram model.
2. ESTIMATING N-GRAM MODELS FROM PCFG
- For commonly used smoothing techniques, n-gram counts suffice to estimate an n-gram language model; this paper extracts fractional counts from a grammar for that purpose.The paper uses smoothing techniques that accept expected counts.Source: §2 · p. 2
2.1. Expected n-gram counts
- The method represents a grammar as a weighted FST whose generated sequences carry probabilities, and sums sequence-level n-gram counts weighted by those probabilities.The paper notes that direct enumeration is non-trivial and says dynamic programming can compute the counts and normalized probabilities efficiently.Source: §2.1 · p. 2
- An n-gram’s expected count is the sum of probabilities of FST paths containing its corresponding arc sequence, with forward-backward quantities and a normalization factor computing that path mass.The prose states that the forward-backward algorithm computes the state values and normalization factor efficiently.Source: §2.1 · p. 2
- A forward topological traversal extends incoming n-grams over each outgoing word arc, computes their expected counts, and propagates them with accumulated probabilities.This iterative procedure computes counts across the FST without explicitly enumerating every sentence.Source: §2.1 · p. 2
2.2. Estimating Language Models
- The paper builds its final n-gram model using Katz smoothing on scaled fractional counts, while noting that count-of-count methods require extracting count-of-counts.The extracted counts can also support feed-forward neural-network training or interpolation, but those options are not used for the paper’s final n-gram model.Source: §2.2 · p. 2
3. OPTIMIZING INTERPOLATION WEIGHTS
The paper adapts an existing LM by interpolating it with an application-intent LM and optimizing weights to fit the new intent while constraining loss on past usage. It offers loss choices based on available adaptation data, extends the objective to multiple applications, and penalizes degradation on existing applications.
3. OPTIMIZING INTERPOLATION WEIGHTS
- Traversing the FST can be expensive for adjacent non-terminals or cyclic grammars; a threshold can avoid propagating low-count n-grams.The passage presents this as a possible heuristic to mitigate the cost.Source: §3 · p. 2
- The constrained objective fits representative new-application data while limiting loss on a development set based on past usage.The interpolation weights combine the pre-existing LM and application-intent LM.Source: §3 · p. 3
3.1. Loss functions
- The method offers three loss functions, with the choice depending on the adaptation data available for a new intent.The options address no adaptation data, sample text, or transcribed audio.Source: §3.1 · p. 3, §3.1.1 · p. 3, §3.1.2 · p. 3
3.1.1. Negative squared minimization
- With no adaptation data, negative squared minimization increases the application-intent interpolation weight until the past-data constraint is violated.The method assigns the feature LM the maximum possible weight subject to that constraint.Source: §3.1.1 · p. 3
3.1.2. Perplexity minimization
- When sample text is available, the method minimizes the interpolated model's perplexity on that data.The loss uses the model's probability for each word and the total word count.Source: §3.1.2 · p. 3
3.1.3. Expected WER minimization
- With transcribed audio, the method can minimize expected recognition error directly because perplexity may not correlate with WER.Expected WER sums hypothesis recognition errors weighted by posterior probability; the supplied passage says the sum is approximated using an n-best list but does not complete its description.Source: §3.1.3 · p. 3
3.2. Constraint optimization
- A penalty term constrains loss on past usage relative to the pre-existing LM's baseline loss.The paper uses a static penalty coefficient of 1000; the loss can be perplexity or expected WER.Source: §3.2 · p. 3
3.3. Scaling to multiple applications
- The objective scales to multiple application grammars, with each application's loss and constraint chosen independently.This permits choosing each intent's loss according to its data availability.Source: §3.3 · p. 3
4. EXPERIMENTS
Experiments compare grammar-derived n-gram counts with sampling and test constrained interpolation across new intents. Expected-WER constraints yield larger improvements for new intents, while WER increases only nonsignificantly through 12 added applications.
4. EXPERIMENTS
- The baseline combined user transcripts with in-house sources; three low-coverage intents were tested on intent-targeted held-out sets of about 500–700 utterances.The test sets were collected in-house by language experts and were not reflective of actual live usage.Source: §4 · pp. 3–4
4.1. n-gram count extraction
- For Flight Info, direct grammar n-gram counts produced a large perplexity and WER drop compared with sampling, where many non-terminal pairs create exponentially many n-grams.Across intents, the performance difference varied; for Recipes and Stock Price, the perplexity difference was small, and Stock Price WER degraded with larger sample sizes.Source: §4.1 · pp. 4–5, Figure 2 · p. 4
4.2. Comparing Loss function and constraint
- More than 10% relative WER improvements were reported for Flights Info and Recipes with an expected-WER constraint.Table 1 compares loss functions and constraints; both constraints keep past-data WER degradation marginal.Source: Table 1 · p. 4, §4.2 · p. 5
4.3. Scalability
- WER increased with each added application, but the increase was not significant even with 12 new applications.The figure tracks WER for three application intents and past data as applications are added.Source: §4.3 · p. 5, Figure 3 · p. 5
5. CONCLUSION
The paper proposes constrained optimization for adapting spoken-dialogue language models to multiple application intents while protecting existing applications. It reports better new-application adaptation with an expected-WER constraint, including when adaptation data is unavailable.
5. CONCLUSION
- The proposed adaptation method is designed to scale to multiple application intents.Source: §5 · p. 5
- Constrained optimization is intended to prevent system accuracy from degrading as new applications are added.Source: §5 · p. 5
- Using WER as the constraint improves adaptation for new applications and removes the need for adaptation data.Source: §5 · p. 5
- Extending the framework to neural-network language models is left for future work.Source: §5 · p. 5