Source-linked AI summary
Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities
Christophe Van Gysel
TL;DR
Spoken virtual-assistant queries are difficult to transcribe, especially when the language model has limited support for ambiguous or entity-rich queries. The paper surveys IR-informed approaches and ASR challenges, concluding that domain classification, external knowledge, and personalization offer opportunities to improve recognition.
Problem
Accurately transcribing spoken queries for retrieval is challenging, particularly when an ambiguous query receives a low language-model likelihood.
Source: §1 · p. 1Method
The paper reviews language models in ASR and surveys IR-informed recognition approaches involving query domains, knowledge graphs and external data, and personalization.
Source: §1 · p. 1, §3.2 · p. 2, §3.3 · p. 3Results
The paper identifies opportunities for IR methods to improve VA ASR through domain classification, knowledge graphs, external data, and personalization, and reviews ASR challenges.
Source: §4 · p. 3Takeaways & Limitations
The paper concludes that IR research can contribute to more accurate recognition of spoken information queries for virtual assistants.
Source: §4 · p. 3Integration of external language models and personalization into end-to-end ASR remains an active research area.
Source: §3.4 · p. 3
Abstract
from arXiv · showhide
Virtual assistants are becoming increasingly important speech-driven Information Retrieval platforms that assist users with various tasks. We discuss open problems and challenges with respect to modeling spoken information queries for virtual assistants, and list opportunities where Information Retrieval methods and research can be applied to improve the quality of virtual assistant speech recognition. We discuss how query domain classification, knowledge graphs and user interaction data, and query personalization can be helpful to improve the accurate recognition of spoken information domain queries. Finally, we also provide a brief overview of current problems and challenges in speech recognition.
1 INTRODUCTION
Recognizing spoken virtual-assistant queries is challenging, and IR application knowledge can help ASR transcribe them accurately. The paper outlines how domain classification, knowledge graphs, and personalization may improve recognition, alongside ASR constraints.
1 INTRODUCTION
- Spoken-query transcription is challenging, but knowledge of the IR application can help recognition before retrieval processing.Source: §1 · p. 1
- Language models provide a text-trained query prior that matters especially for ambiguous speech, where a rare entity such as “borrelia” may be misrecognized as “gorilla.”Source: §1 · p. 1
- Real-time response, on-device size, model updating, and reasonable training time constrain VA speech recognition.Source: §1 · p. 1
- The paper surveys ways to use IR knowledge to improve ASR for entity-heavy virtual-assistant queries, including domain classification, knowledge graphs, and personalization.Source: §1 · p. 1
2 ASR PRIMER
The primer explains how hybrid and end-to-end ASR produce word sequences from speech, and how language models contribute text-based probabilities. It also describes the chain-rule basis of language modeling and sentence boundary markers.
2 ASR PRIMER
- ASR translates a speech signal into a word string; hybrid systems combine acoustic and language models, while end-to-end systems predict text sequences directly from acoustic representations.Source: §2 · p. 1
2.1 Hybrid ASR systems
- Hybrid ASR uses Bayes’ rule to combine an acoustic-model likelihood with a language-model prior, with the models trained independently.Source: §2.1 · p. 1
2.2 End-to-end ASR systems
- End-to-end ASR models output-string probabilities from speech, and an external text-trained language model may be interpolated with them.Source: §2.2 · p. 2
2.3 Language models
- Language models use the chain rule to assign a string probability from conditional word probabilities, with start and end markers delimiting a sentence.Source: §2.3 · p. 2
3.1 Use of query domain classifications
Query-domain classifications can inform ASR decoding or language-model training. The section discusses contextual expert-model weighting at recognition time and domain-informed allocation of language-model capacity during training.
3.1 Use of query domain classifications
- VA queries span domains such as media playback and encyclopedic information, and domain classifications can guide decoding or language-model training.Source: §3.1 · p. 2
3.1.1 Improving the ASR decoding process.
- At decoding time, partial hypotheses or location can weight domain-specific language models, but effective integration of contextual signals into end-to-end ASR remains open.Source: §3.1.1 · p. 2
3.1.2 Building beter LMs by leveraging query domain classifica-
- Training-query domain labels can allocate more model capacity to domains with many tail entities; spoken-query errors mean typed-query classification methods may not transfer directly.Source: §3.1.2 · p. 2
3.2 KGs and other external data
The paper identifies query-log error feedback as a challenge and discusses external data, entity popularity, and knowledge-graph relations as opportunities to improve recognition of entity-rich VA queries.
3.2 KGs and other external data
- Recognition errors in query logs can reinforce themselves when those logs train language models; external data offers an alternative for entity-rich queries.Filtering may relieve the feedback loop but can introduce undesirable training-data biases.Source: §3.2 · p. 2
3.2.1 Qery templatization and entity popularity.
- Gandhe et al. [9] estimate n-gram LMs from entity-rich templates for new intents, but adapting external entity popularity to VA users remains open.Popularity can differ between the source application and VA users, for example because of demographic differences.Source: §3.2.1 · p. 2
3.2.2 Using KG relations during ASR decoding.
- Saebi et al. [28] and others [15] [22] use entity types and relations during decoding to boost hypotheses consistent with knowledge-graph relations.For example, a song–artist relation can favor “Canyon Moon” over “can you moon” when recognizing a query naming Harry Styles.Source: §3.2.2 · pp. 2–3
3.3 Personalization
The section presents personalization as a way to use user-related signals to improve on-device VA ASR, including location-specific query language models.
3.3 Personalization
- Location-bucketed query LMs significantly improved recognition of spoken point-of-interest queries, while user models based on query behavior or other signals may support further on-device personalization.The described location-based approach personalizes the ASR query model during decoding.Source: §3.3 · p. 3
3.4 Beyond IR
The section surveys ASR-side developments and unresolved practical challenges, including integrating language models with end-to-end systems and deploying neural language models.
3.4 Beyond IR
- End-to-end ASR uses fewer components and reduces maintenance costs, but integrating external LMs and personalizing these systems remain active research areas.Source: §3.4 · p. 3
- Neural LMs can be more storage-economical than n-gram LMs, yet heterogeneous-corpus training, inference latency, and federated learning remain challenges.Large pretrained Transformer LMs have also been used for ASR and synthetic data generation, although a domain gap may exist.Source: §3.4 · p. 3
4 CONCLUSIONS
The conclusion recaps opportunities for IR research to improve VA speech recognition through query-domain classification, knowledge graphs, external data, and personalization. It also points to cross-disciplinary research directions and briefly reviews ASR challenges.
4 CONCLUSIONS
- The paper identifies query-domain classification, knowledge graphs, interaction-based external data, and personalization as IR opportunities for improving VA ASR.Source: §4 · p. 3
- The conclusion also briefly reviews open problems and challenges within ASR.Source: §4 · p. 3
- The authors hope the discussion inspires cross-disciplinary research directions and new application domains for existing methods.Source: §4 · p. 3
SPEAKER BIOGRAPHY
Christophe Van Gysel works on the boundary between ASR and Search at Apple. His doctoral work focused on neural ranking and entity-centered representation learning.
SPEAKER BIOGRAPHY
- Van Gysel is a Staff Research Scientist on Apple’s Siri Speech language modeling team, working at the boundary between ASR and Search.Source: SPEAKER BIOGRAPHY · p. 3
- He received a Computer Science PhD from the University of Amsterdam in 2017.Source: SPEAKER BIOGRAPHY · p. 3
- During his PhD, he studied neural ranking with representation learning focused on entities and published across venues including WWW, SIGIR, CIKM, WSDM, and TOIS.Source: SPEAKER BIOGRAPHY · p. 3
COMPANY PROFILE
The company profile presents Apple’s products, software platforms, and services, alongside its scale of more than 100,000 employees.
COMPANY PROFILE
- Apple’s products include iPhone, iPad, Mac, Apple Watch, and Apple TV.Source: COMPANY PROFILE · p. 3
- Apple lists five software platforms: iOS, iPadOS, macOS, watchOS, and tvOS.Source: COMPANY PROFILE · p. 3
- Apple describes services including the App Store, Apple Music, Apple Pay, and iCloud, and reports more than 100,000 employees.Source: COMPANY PROFILE · p. 3