A Discriminative Entity-Aware Language Model for Virtual Assistants

1. Introduction

• ASR struggles with named entities that are rare or newly popular because they appear rarely or not at all in training data.The example contrasts the requested song name “Canyon Moon” with the acoustically similar top-choice hypothesis “Can you Moon.”Source: §1 · p. 1
• Many named-entity recognition errors conflict with real-world knowledge, which the authors use to improve ASR hypotheses.For example, knowing which songs Harry Styles performs can help identify “Canyon Moon.”Source: §1 · p. 1
• DEAL extends discriminative n-gram language modeling with Knowledge Graph-consistent features for entity types and relationships between entities.The prior approach uses a log-linear model with a base-model score and word n-gram indicators, trained discriminatively on base-system n-best lists or lattices.Source: §1 · p. 1
• The model uses efficient lattice rescoring and achieves large improvements on targeted test sets for unpopular entities with minimal degradation on a uniformly sampled VA test set.The paper also analyzes feature sets and presents an efficient lattice-rescoring algorithm.Source: §1 · p. 1

2. Approach

• A log-linear model scores each hypothesis using a base-system score and weighted n-gram features, with weights learned to minimize a discriminative loss on training n-best lists.At test time, the learned model rescores n-best lists or lattices from the base system.Source: §2 · p. 1

2.1. Features

• The Knowledge Graph represents real-world entities through names, semantic types, and relationships to other entities, with word counts for names and popularity for relationships.Relationship popularity indicates how frequently an entity and its related entity appear together.Source: §2.1 · p. 2, Figure 3 · p. 2
• Feature n-grams combine words with entity-type non-terminals that match valid entity names, optionally conditioned on popularity, name length, or entities seen earlier in a hypothesis.Popularity and word-count entity lists intentionally overlap to reduce features with few training examples.Source: §2.1 · p. 2
• The feature set includes 3-grams containing a non-terminal and 4-grams bounded by non-terminals, then factors them into variants with conditional non-terminals.Figure 2 illustrates feature n-grams, learned weights, and the resulting Rtop grammar.Source: §2.1 · p. 2, Figure 2 · p. 2

2.2. Lattice Rescoring

• Lattice rescoring can cover more hypotheses efficiently than n-best rescoring and avoid redundant computation across overlapping hypotheses.Expanding every Knowledge Graph entity into a static rescoring automaton would create prohibitively large automata and require rebuilding when the graph changes.Source: §2.2 · p. 2
• The method builds Rtop offline, constructs Rkg on-demand from Rtop and the Knowledge Graph, then determinizes Rkg on-demand to form Rdet.Non-terminal/realization pairs are retained in dynamic states to support features conditioned on entities previously seen in a hypothesis.Source: §2.2 · pp. 2–3

3. Related Work

• Knowledge Graph language models in and use neural approaches, while this work is a non-neural counterpart with a discriminative optimization criterion.Source: §3 · p. 3
• Other VA-focused systems,,, and use log-linear n-gram features and WFST composition, but determine weights heuristically rather than through discriminative training.Systems,, and apply their models earlier in recognition rather than during lattice rescoring.Source: §3 · p. 3
• System is especially similar because its n-gram features can include non-terminals, but it semantically tags lattices and does not model entity-entity relationships.Source: §3 · p. 3

4.1. Data

• The knowledge graph covers five US English entity types, and synthesized training and test data target entities and relations across popularity strata.Training used 7,000 general VA utterances and 2,000 synthesized utterances per entity type or pair and stratum; training and test synthesis used different voices.Source: §4.1 · p. 3

4.2. Baseline Systems

• Baseline-1 uses VA-derived 4-gram components, while baseline-2 adds separate 4-gram components trained on synthetic text for each entity type or pair.The synthetic text data were generated at much larger scale than the synthesized training data.Source: §4.2 · pp. 3–4

4.3. Results

• DEAL1 improves over baseline-1 on synthetic datasets with minimal degradation on general VA queries, and DEAL2 reduces SER over baseline-2.The results table reports SER; its caption defines bold values as the lowest SER for each baseline.Source: Table 1 · p. 4, §4.3 · p. 4
• SER rises from head toward torso and tail entities across models, consistent with less popular entities being less represented in baseline training data.Oracle-2 generally has lower SER than Oracle-1 after targeted synthetic data is included in the baseline LM.Source: §4.3 · p. 4
• Entity-entity features add gains mainly on related Music Title/Artist and City/State test sets; adding popularity and word-count features further lowers SER.DEAL1-rpc has lower SER than either popularity or word-count features alone.Source: §4.3 · p. 4
• DEAL1-rpc outperforms baseline-2 across several datasets despite lacking its additional lattice hypotheses.DEAL2 also reduces SER over baseline-2, with further improvements from entity-entity features and combined word-count and popularity features.Source: §4.3 · p. 4

• The approach achieves relative SER reductions above 25% on some synthetic tests covering less common entities, with minimal degradation on a general VA test set.Entity-entity features yield large reductions on related-pair tests; popularity and word-count features produce a modest reduction across synthesized entity tests.Source: §5 · p. 4