Source-linked AI summary
Speech Model Pre-training for End-to-End Spoken Language Understanding
Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, Yoshua Bengio
TL;DR
End-to-end SLU must learn speech representations directly from limited labeled data, motivating approaches that reduce its data requirements. The paper pre-trains models on word and phoneme ASR targets, introduces Fluent Speech Commands, and reports improved performance on full and small training sets, while identifying failures on unseen synonyms.
Problem
End-to-end SLU seeks to map speech directly to intent, but learning hierarchical representations from variable speech is difficult with limited training data.
Method
The paper pre-trains lower model layers to predict words and phonemes, then discards the ASR classifiers and trains the resulting model for SLU.
Results
The pre-training techniques improve end-to-end SLU performance on both full and small training sets, with pre-trained models reaching 97% validation accuracy on a new wording experiment.
Takeaways & Limitations
Fluent Speech Commands enables low-resource SLU experiments, and ASR pre-training provides useful features for end-to-end SLU across training-set sizes.
Takeaways & Limitations
The model does not always generalize correctly to unseen synonyms, such as “bathroom” when trained on “bedroom” and “washroom.”
Abstract
from arXiv · showhide
Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map speech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.
1. Introduction
End-to-end SLU maps speech directly to intent, avoiding an intermediate text-recognition stage, but learning useful speech representations is difficult with limited data. The paper proposes ASR-based pre-training and introduces a dataset to study low-resource SLU.
- End-to-end SLU directly optimizes intent recognition accuracy instead of estimating an intermediate text representation.
- Compared with conventional pipelines, end-to-end models avoid an error-prone intermediate search process and can use intent-relevant cues absent from transcripts, such as prosody.
- Learning hierarchical speech representations from high-dimensional, variable signals is difficult without substantial training data.
- The proposed methodology pre-trains end-to-end SLU models on related ASR tasks to improve performance, especially with very little SLU training data.
- The paper introduces a dataset for realistic SLU experiments and uses it to demonstrate speech-model pre-training techniques for low-resource SLU.
- The authors make their code and data publicly available for replication and further research.
2. Related work
Earlier end-to-end SLU studies used varied initialization and multi-task strategies, often in high-resource or restricted-data settings. The paper addresses reproducibility by introducing an open dataset suited to testing data requirements and generalization.
- Prior end-to-end SLU work used no pre-training, auto-encoder initialization, grapheme pre-training, or shared ASR-SLU encoders.
- The proposed model removes the softmax bottleneck from related grapheme-pretraining work and uses alternative training targets.
- Much previous SLU research used very high-resource datasets, while other limited-data work did not examine pre-training on additional speech data.
- Closed-source or too-small datasets made it difficult to test data requirements and conduct high-quality, reproducible end-to-end SLU research.
- Fluent Speech Commands was created as an open-source SLU dataset to fill this experimental gap.
3. Dataset
Fluent Speech Commands is a free audio dataset of multiword commands labeled with action, object, and location slots. Its varied wordings and speaker-independent splits support experiments on SLU generalization and data requirements.
- Fluent Speech Commands contains 16 kHz single-channel recordings of spoken English commands for smart-home and virtual-assistant scenarios.
- Each utterance has action, object, and location labels, whose combination defines one of 31 intents.
- The paper uses simpler multi-label classification of the three slots rather than modeling their dependencies autoregressively.
- Each intent has multiple possible wordings, with 248 phrases created before data collection from suggestions by native and non-native English speakers.
- Crowdsourced recordings were collected twice per wording for each speaker and validated by separate crowdsourcers.
- Speaker-independent train, validation, and test splits retain all wordings while supporting experiments that withhold selected wordings during training.
- Unlike related datasets, Fluent Speech Commands is free, audio-based, reasonably large, and includes multiple recordings for multiple-word commands.
- The model architecture pre-trains lower layers on word and phoneme ASR targets, discards their classifiers, and feeds the resulting features into an SLU-trained module.
4. Model and Pre-training Strategy
The model uses a stack of modules pre-trained to predict phonemes and words before end-to-end SLU training. The pre-trained representations are then mapped to intent, with gradual layer unfreezing used during SLU fine-tuning.
- The model pre-trains initial modules on phoneme and word prediction, discards their classifiers, and trains the full network end-to-end for SLU.The word and phoneme classifiers are removed before supervised intent training.
- Whole-word pre-training uses the 10,000 most common words, while Montreal Forced Aligner alignments support phoneme- and word-level supervision on LibriSpeech.The alignments also enable short random crops, reducing pre-training computation and memory requirements.
- The phoneme module processes raw waveforms with SincNet, convolutional and recurrent layers, pooling, and dropout before producing phoneme representations.
- The word module receives phoneme representations rather than phoneme logits and produces word representations for the subsequent intent module.Forwarding hidden representations preserves flexibility in layer size and avoids retaining an approximately 2.5 million-parameter word-classifier matrix.
- The intent module maps word representations to fixed three-slot intent predictions using a recurrent layer followed by max-pooling.
- Gradual unfreezing fine-tunes the pre-trained layers by releasing one layer per epoch until a predetermined stopping layer.The authors report that gradual unfreezing works better than unfreezing all layers simultaneously.
5. Experiments
Experiments on Fluent Speech Commands evaluate pre-training with full and 10% training data, including generalization to unseen wordings. Pre-trained models outperform random initialization, especially with limited data, but do not generalize correctly in all cases.
- Experimental setup: The experiments compare models trained on the full dataset, a 10% subset, and restricted wordings to evaluate performance and generalization.Validation accuracy is shown over time, while test accuracy is reported for full and 10% training conditions.
- 5.1. Full dataset: Pre-trained models outperform randomly initialized models on the full dataset, with gradual unfreezing of word layers producing the best validation results.The fully unfrozen model may forget general phonetic knowledge acquired during pre-training.
- 5.2. Partial dataset: With 10% of the training data, unfreezing word layers remains best, while pre-trained models lose only slightly in final test accuracy and widen their advantage over random initialization.Validation accuracy is evaluated on the entire validation set rather than a subset.
- 5.3. Generalizing to new wordings: Pre-trained models generalize to the unseen phrase “switch off the lights” with 97% validation accuracy, whereas the randomly initialized model cannot fit the tiny training set.Training uses 273 utterances covering three phrases; testing includes a fourth phrase not seen during training.
- 5.3. Generalizing to new wordings: Generalization remains unreliable for synonyms such as “bathroom” when training includes only acoustically different alternatives such as “bedroom” and “washroom”.The paper suggests embedding-like word representations as a possible way to recognize meanings of unseen synonyms.
6. Conclusion
The conclusion presents pre-training and Fluent Speech Commands as a way to improve end-to-end SLU performance with both large and small training sets. It identifies unseen wordings and synonyms as continuing areas for investigation.
- 6. Conclusion: The paper proposes a pre-training methodology, introduces Fluent Speech Commands, and reports improved end-to-end SLU performance with large and small training sets.Future work will examine limitations involving new wordings and unobserved synonyms.