Source-linked AI summary
Towards end-to-end spoken language understanding
Dmitriy Serdyuk, Yongqiang Wang, Christian Fuegen, Anuj Kumar, Baiyang Liu, Yoshua Bengio
TL;DR
Traditional SLU pipelines independently recognize speech and interpret the resulting text, motivating a unified alternative. This paper trains an encoder-decoder directly on audio features for speech-to-domain and speech-to-intent classification, showing reasonable performance and audio-aligned semantic responses, while noting that the approach is not yet superior to the pipeline.
Problem
Traditional SLU independently optimizes ASR and NLU, so ASR errors can propagate into NLU even though NLU is trained on clean transcripts.
Method
The paper uses an encoder-decoder with a multilayer bidirectional GRU encoder, temporal subsampling, max pooling, and log-Mel filterbank inputs for speech-to-domain and speech-to-intent classification.
Results
The model achieves reasonable end-to-end SLU performance, with saliency responses corresponding to meaningful utterance regions and domain classification reported as close to perfect on the dataset.
Takeaways & Limitations
Audio features can support language-understanding models that capture semantic attention without an intermediate text representation.
Takeaways & Limitations
The end-to-end approach does not show superior performance and reaches 10% relatively worse accuracy with significantly fewer parameters.
Abstract
from arXiv · showhide
Spoken language understanding system is traditionally designed as a pipeline of a number of components. First, the audio signal is processed by an automatic speech recognizer for transcription or n-best hypotheses. With the recognition results, a natural language understanding system classifies the text to structured data as domain, intent and slots for down-streaming consumers, such as dialog system, hands-free applications. These components are usually developed and optimized independently. In this paper, we present our study on an end-to-end learning system for spoken language understanding. With this unified approach, we can infer the semantic meaning directly from audio features without the intermediate text representation. This study showed that the trained model can achieve reasonable good result and demonstrated that the model can capture the semantic attention directly from the audio features.
1. INTRODUCTION
Traditional SLU converts speech to text with ASR and then predicts domain, intent, and slots with NLU. The paper explores jointly optimizing SLU from audio features to avoid intermediate text and study audio-based semantic modeling.
- Standard SLU uses an ASR-to-NLU pipeline for domain classification, intent detection, and slot filling.The ASR output may be normalized before NLU processing.
- Independent module objectives can misalign because ASR typically minimizes WER although words differ in their impact on intent and slot accuracy.NLU is trained on clean transcripts but evaluated on recognized text, allowing ASR errors to propagate.
- The paper extends end-to-end ASR learning to include NLU and focuses initially on single-turn intent classification from log-Mel filterbank features.The authors also investigate degradation under artificially added noise.
- The study demonstrates the possibility of training language understanding directly from audio features and reports degradation of standard SLU when evaluated on ASR output.
2. STANDARD ASR AND NLU SYSTEMS
A conventional SLU system separates acoustic recognition from language understanding. ASR combines acoustic and language models to produce words, while NLU encodes those words for domain, intent, and slot prediction.
- Standard ASR: The ASR acoustic model commonly uses recurrent networks to predict HMM states before language-model-guided decoding.
- Standard ASR: Traditional ASR uses an acoustic model and language model, with Viterbi decoding and a dictionary to search for the most likely word sequence.The acoustic model predicts context-dependent HMM-state likelihoods for each frame.
- Standard NLU: NLU receives the ASR word sequence, embeds the words, and uses an LSTM-based utterance classifier for domain and intent classification.The described classifier uses a two-layer bidirectional LSTM encoder.
- Pipeline behavior: In the pipeline, ASR and NLU are trained independently: NLU uses human transcripts during training but ASR output during evaluation.
3. END-TO-END SPOKEN LANGUAGE UNDERSTANDING
The proposed end-to-end SLU system treats speech-to-domain and speech-to-intent as sequence classification from log-Mel filterbank features. A bidirectional GRU encoder subsamples and pools the acoustic sequence into a class prediction.
- Task formulation: The initial end-to-end tasks are speech-to-domain and speech-to-intent classification using log-Mel filterbank feature vectors as input.
- Encoder: The encoder is a multilayer bidirectional GRU network that processes acoustic features directly instead of an intermediate text representation.
- Temporal reduction: Subsampling hidden activations at every bidirectional GRU layer shortens the sequence and yields a roughly syllable-level representation.The authors report reduced training and prediction computation, enabling bidirectional GRU use for real-time classification.
- Pooling: A max-pooling layer over encoder outputs compresses the sequence into a fixed representation for classification.
4. EXPERIMENTS
The experiments evaluate audio-based domain and intent classification on a large in-house VR-command dataset, including clean and noise-corrupted conditions. The end-to-end models are compared with transcript- and recognition-based NLU baselines.
- Data: The dataset contains about 320 hours of near-field annotated VR commands from more than 1,000 de-identified speakers.It includes scripted and free-speech scenarios.
- Model: The end-to-end model uses log-Mel features, a four-layer bidirectional GRU encoder, stride-2 subsampling, max pooling, and a feed-forward classifier.
- Baselines: The baseline NLU is evaluated with transcript text and recognized text, while the ASR uses the same 320-hour training data and achieves 3.5% WER on evaluation data.
- Noise conditions: Noise experiments corrupt training and evaluation audio with room reverberation and background noise across specified SNR ranges.Training utterances are distorted twice using different acoustic conditions.
- Qualitative analysis: Saliency maps visualize filter-bank inputs and show responses aligned with weather-related and music-related words in selected utterances.
5. RESULTS AND DISCUSSION
The end-to-end model achieves near-perfect domain classification on this dataset and reasonable intent-classification performance, while remaining substantially smaller than the standard SLU system. Its performance degrades on noise-corrupted data, and the authors report that it does not show superior overall performance.
- Domain classification: Domain-classification accuracy is close to perfect on this dataset.Evaluation on clean transcripts represents the maximum performance achievable with this model.
- Intent classification: Max-pooling activations and batch normalization improve end-to-end intent-classification accuracy.The authors associate max pooling with long speech sequences and batch normalization with optimization difficulty.
- Model size and efficiency: The much smaller end-to-end model achieves reasonable intent-classification performance while reducing parameter count.Table 2 reports parameter counts to demonstrate the compactness of the end-to-end model.
- Robustness: Noise-corrupted data causes both intent-classification models to degrade significantly.
- Model size and efficiency: 0.002 real-time factor enables intent or domain prediction in real time with a bi-GRU model.
- Overall discussion: The end-to-end approach does not show superior performance, although the authors report 10% relatively worse accuracy with significantly fewer parameters.