Source-linked AI summary
From Audio to Semantics: Approaches to end-to-end spoken language understanding
Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro Moreno, Rohit Prabhavalkar, Zhongdi Qu, Austin Waters
TL;DR
Conventional SLU independently chains ASR and NLU, leaving recognition errors to affect semantic prediction. This paper formulates audio-to-semantics understanding as sequence-to-sequence learning and compares jointly trained encoder-decoder approaches. The results show that intermediate text representation is important and that joint optimization improves performance, including an 18% relative reduction in argument WER over an independently trained baseline.
Problem
Conventional SLU independently optimizes ASR transcription and NLU semantic prediction, while ASR errors can worsen NLU performance.
Method
The paper compares end-to-end encoder-decoder sequence-to-sequence models that jointly optimize ASR and NLU from acoustic features, optionally producing transcripts, semantics, or both.
Results
An intermediate text representation is important for semantic quality, and the best jointly trained model improves argument WER by 18% relative to the independently trained baseline while achieving competitive domain and intent prediction.
Takeaways & Limitations
Joint optimization can improve audio-based SLU, particularly argument prediction, while direct semantic prediction that ignores the ground-truth transcript performs less well.
Takeaways & Limitations
The multistage approach still depends on future work to exploit larger ASR corpora, alternative losses such as MBR, and incorporation of new grammars with limited training data.
Abstract
from arXiv · showhide
Conventional spoken language understanding systems consist of two main components: an automatic speech recognition module that converts audio to a transcript, and a natural language understanding module that transforms the resulting text (or top N hypotheses) into a set of domains, intents, and arguments. These modules are typically optimized independently. In this paper, we formulate audio to semantic understanding as a sequence-to-sequence problem [1]. We propose and compare various encoder-decoder based approaches that optimize both modules jointly, in an end-to-end manner. Evaluations on a real-world task show that 1) having an intermediate text representation is crucial for the quality of the predicted semantics, especially the intent arguments and 2) jointly optimizing the full system improves overall accuracy of prediction. Compared to independently trained models, our best jointly trained model achieves similar domain and intent prediction F1 scores, but improves argument word error rate by 18% relative.
1. INTRODUCTION
Spoken language understanding traditionally separates speech recognition from semantic interpretation, but recognition errors can harm the downstream prediction of domains, intents, and arguments. This work explores jointly trained audio-to-semantics models as an end-to-end alternative.
- Conventional SLU pipeline: Traditional SLU first transcribes audio with ASR, then uses NLU to predict domains, intents, and argument values from the transcript.NLU includes domain classification, intent detection, and slot filling, with slots and arguments used interchangeably.
- Motivation for joint optimization: Recent work jointly optimizes domain, intent, and argument prediction to improve the overall quality of the NLU component.The paper uses semantics to refer collectively to domains, intents, and arguments.
- Motivation for joint optimization: ASR errors can worsen NLU performance, although errors affecting irrelevant transcript content may matter less than errors affecting intents and arguments.The paper motivates joint training partly because SLU semantics often consist of an action with relevant arguments rather than the entire transcript.
- Related approaches: Prior approaches have predicted domain and intent directly from audio or incorporated multiple ASR hypotheses, confusion networks, recognition lattices, or ASR correction modules.The cited approaches address recognition uncertainty, but independent ASR and NLU optimization can remain sub-optimal.
- Paper contribution: This work compares end-to-end encoder-decoder approaches that jointly predict semantics from audio and can simplify SLU architecture.The study reports competitive audio-to-semantics prediction and claims that domain, intent, and arguments can all be predicted from audio with competitive results.
2. SYSTEM ARCHITECTURE
The paper compares encoder-decoder architectures for predicting semantics from acoustic features, including direct, joint, multitask, and multistage formulations. These models vary in whether they generate transcripts, semantics, or both, and in how transcript and semantic prediction are coupled.
- Notation: The system takes acoustic features X as input and represents transcripts W and serialized semantics S as output sequences.S contains domain, intent, argument labels, and graphemes representing argument values.
- Direct model: The direct model predicts semantic sequence S directly from acoustic features X without fully transcribing the input.Its decoder generates only the semantic sequence, including argument values when relevant.
- Joint model: The joint model generates the transcript followed by semantics as one concatenated output sequence [W : S].This introduces an intermediate transcript representation while retaining a single encoder-decoder structure.
- Multitask model: The multitask model shares an encoder while using separate decoders for transcript and semantic prediction.The shared encoder is trained to produce representations supporting both tasks.
- Multistage model: The multistage model treats semantics as conditionally independent of acoustics given the transcript and can train its two stages independently.Weak coupling can instead pass predicted transcripts between stages, allowing gradients from the second stage to affect the first.
- Multistage model: Multistage training permits separate corpora for speech recognition and language understanding, but stronger coupling and alternative losses remain future work.The authors specifically identify using additional ASR data and minimum Bayes risk optimization as future directions.
3. EXPERIMENTAL SETUP
The experiments serialize domain, intent, and arguments as output sequences and compare encoder-decoder architectures under a common setup. Models use acoustic inputs, LSTM encoders and decoders, attention, and either independent or weakly coupled ASR–NLU stages.
- Data: 24M anonymized English utterances form the training set, while 16K hand-transcribed utterances form the test set.Both sets represent a slice of Google Home traffic, with semantic labels generated from ground-truth transcripts using context-free grammars.
- Representation: Semantics are serialized by placing domain, intent, and optional argument labels and values into a tagged sequence.A parser later de-serializes the predicted sequence by tokenizing domain tags, intent tags, and argument names.
- Representation: The vocabulary contains 110 symbols, including semantic tags, names, English graphemes, digits, punctuation, and special symbols.The representation supports both transcript content and argument values.
- Acoustic features: All models use 80-dimensional log-Mel filterbanks, stack three contiguous frames, and downsample inputs to a 30ms frame rate.The resulting stacked acoustic input is 240-dimensional before downsampling.
- Architectures: The architectures use LSTM encoders and decoders, with bidirectional second encoders in multistage models and 4-headed additive attention in every decoder.Models are kept within 15% of one another in parameter count to support fair comparison.
- Training and baselines: The baseline trains ASR and NLU stages independently, while weakly coupled variants pass either argmax transcripts or sampled softmax outputs between stages.All neural networks are trained with cross-entropy, and inference uses beam search with beam size 8.
- Metrics: Evaluation reports transcript WER, domain and intent multi-class F1, and argument WER for semantic prediction.Argument WER is used because transcript and output arguments may not match directly.
4. RESULTS
Joint and multitask models perform strongly on domain and intent prediction, while intermediate text and acoustic features are important for intent. Argument WER varies more across models, and jointly optimized systems outperform the independently trained baseline.
- Domain prediction: Joint achieves the best domain F1 score at 96.8%, with Direct only 0.6% lower in absolute terms.All models perform relatively similarly on domain classification.
- Intent prediction: Multitask and Joint achieve the best intent F1 scores at 95.8% and 95.7%, respectively.Both models use acoustic features and predict transcripts, unlike the Direct model.
- Intent prediction: Acoustic features and an intermediate text representation are important for intent prediction.Weakly coupled multistage models are generally worse than Joint and Multitask models on domain and intent prediction.
- Argument prediction: Direct has the worst argument WER at 18.2, while including transcription loss improves argument prediction in end-to-end models.Multistage ArgMax and SampledSoftmax outperform Joint and Multitask on argument WER, despite weaker domain and intent results.
- Argument prediction: Multistage SampledSoftmax improves argument WER over the independently trained baseline by 18% relative.All jointly optimized models outperform the independently trained baseline on argument prediction.
- Pipeline evaluation: Pipelining predicted transcripts through the same CFG parsers produces similar domain F1, better intent F1, and significantly improved argument WER.These results come from a second experiment using parser-derived semantics from predicted transcripts.
5. CONCLUDING REMARKS
The paper evaluates jointly optimized end-to-end SLU approaches, finding that intermediate text representations and ASR–NLU coupling improve semantic prediction. It also identifies challenges in extending such systems to new grammars and domains, despite their simpler deployment and scalability.
- Results: Jointly optimized end-to-end approaches improve SLU performance, including when CFG-parsers process transcripts generated by the jointly trained model.Table 5 measures transcription WER, domain and intent F1, and argument WER on the model’s top recognized transcript.
- Results: Intermediate text representations are important for learning end-to-end SLU systems, while direct semantic prediction from audio performs less well.Joint optimization also helps models focus on errors that matter more for SLU, reflected in lower argument WER.
- Limitations and future work: End-to-end optimization makes introducing new domains more challenging because new grammars require data for the model to learn them.CFG-parsing approaches can incorporate additional grammars without the same dependence on end-to-end training data.
- Implications: Sequence-to-sequence end-to-end SLU simplifies system complexity, scales to tasks with multiple intents, and avoids additional inference resources such as lexicons, language models, and parsers.This makes the approach suitable for devices with limited compute and memory.