Source-linked AI summary
SLURP: A Spoken Language Understanding Resource Package
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, Verena Rieser
TL;DR
SLU resources have been limited in size and diversity, constraining development of systems that infer semantics directly from audio. The paper releases SLURP, a challenging multi-domain dataset, introduces the SLU-F1 metric, and evaluates strong ASR/NLU baselines. Modular systems provide a strong baseline that current end-to-end systems have yet to meet on this challenging data.
Problem
Publicly available SLU datasets lack sufficient size and diversity across lexical richness, vocalizations, domains, and semantic contexts.
Method
The paper releases SLURP and introduces SLU-F1 while benchmarking state-of-the-art NLU systems and ASR engines with detailed error analysis.
Results
SOTA modular approaches provide a strong baseline for SLURP that has yet to be met by SOTA end-to-end systems.
Takeaways & Limitations
SLURP supports development and analysis of both end-to-end and traditional pipeline SLU systems through challenging data and transparent entity evaluation.
Takeaways & Limitations
Existing end-to-end models developed for easier single-domain tasks did not converge or produce meaningful outputs on SLURP, leaving suitable architecture development for future work.
Abstract
from arXiv · showhide
Spoken Language Understanding infers semantic meaning directly from audio data, and thus promises to reduce error propagation and misunderstandings in end-user applications. However, publicly available SLU resources are limited. In this paper, we release SLURP, a new SLU package containing the following: (1) A new challenging dataset in English spanning 18 domains, which is substantially bigger and linguistically more diverse than existing datasets; (2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A new transparent metric for entity labelling which enables a detailed error analysis for identifying potential areas of improvement. SLURP is available at https: //github.com/pswietojanski/slurp.
1 Introduction
SLURP addresses the lack of large, diverse audio-semantic resources that limits end-to-end spoken language understanding. It contributes a challenging multi-domain dataset, the SLU-F1 entity metric, and baseline evaluations with error analysis.
- Motivation: Existing SLU datasets are limited in lexical and semantic richness, vocalization count, domain coverage, and semantic contexts.These limitations create a bottleneck for building end-to-end systems from paired audio and semantic structures.
- Dataset: SLURP provides ~72k single-turn home-assistant recordings across over 18 scenarios, with annotations for Scenario, Action, and Entities.The dataset defines 46 actions and 55 entity types.
- Metric: SLU-F1 evaluates entity prediction by producing an interpretable distribution rather than only a single accuracy or F1 score.The distribution can yield confidence scores and reflect how ASR errors affect NLU.
- Evaluation: The paper evaluates four competitive pipeline baselines: two state-of-the-art NLU systems and two ASR engines.The evaluation includes error analysis of ASR error propagation and semantic disambiguation, which are potential advantages of end-to-end modeling.
2 Related Work
Earlier SLU resources paired audio with semantic annotations, while recent work introduced end-to-end methods and focused on addressing data sparsity through transfer learning and pre-training.
- Related Work: Early audio-semantic corpora included ATIS and Switchboard-DAMSL, but end-to-end SLU approaches emerged only recently.Recent research has examined transfer learning and pre-training to address data sparsity.
3 SLURP data
SLURP combines carefully collected audio with layered semantic annotations and quality-controlled processing. Compared with existing smart-home SLU datasets, it offers substantially broader lexical, syntactic, and semantic coverage.
- Data collection: SLURP began with prompted textual commands, manually annotated at scenario, action, and entity levels, then used as gold transcriptions for audio.Prompts were designed to avoid lexical priming and increase linguistic variability.
- Data collection: Audio was recorded from 100+ participants in typical home or office conditions using distant microphone arrays and, for some users, close-talking headsets.Recording sessions generally lasted one hour and were divided into four parts with changing microphone-array positions.
- Audio processing: Quality control combined ASR-based prompt matching, endpoint checks, entity-filler WER, and cross-microphone validation before filtering the recordings.The ASR systems used for quality control were not estimated from SLURP acoustic data, avoiding reinforcement of potential errors.
- Audio processing: ~58 hours of acoustic material remained after filtering; nearly 60% had WER=0 from at least one ASR system, rising to ~73% when EntityWER=0 was included.Higher WER does not necessarily indicate missing content, because challenging recognition and semantic sufficiency are distinct concerns.
- Linguistic analysis: SLURP has ~100× more lexical units than FSC and ~10× more than Snips, with higher lexical sophistication and richness except for average sentence length.The comparison uses lexicalised and delexicalised datasets; delexicalisation affects FSC and Snips more because entity names contribute substantially to their lexical diversity.
- Linguistic analysis: SLURP contains more varied syntax and broader semantics: only 74% of sentences are Snips-like Level 0–1, while ~5% reach D-Level 7, and its coverage is 9 times wider in scenarios and ~6.5 times wider in actions.SLURP uses three semantic layers—scenarios, actions, and entities—and includes more mixed syntactic complexity than FSC or Snips.
4 SLURP Metrics
The paper identifies shortcomings in standard entity accuracy and introduces SLU-F1, which combines span-based matching with text distance to separate tagging and transcription effects.
- Motivation: Standard accuracy treats an utterance as incorrect when any slot differs, but does not distinguish semantic mislabelling from textual misalignment.It also fails to differentiate entity labels from lexical fillers.
- SLU-F1: SLU-F1 relaxes exact entity matching when labels agree, then accounts for filler mismatch through distance-based false positives and false negatives.This reduces penalties for ASR-induced misalignments while retaining entity-tagging information.
- Distance functions: Word-F1 uses word error rate, whereas Char-F1 uses normalized character-level Levenshtein distance to capture different transcription variations.Word-F1 is stricter for token changes; Char-F1 is less sensitive to small local string variations.
- Algorithm: The dist-F1 algorithm matches predicted and gold entities by label and selects the candidate minimizing the chosen text distance.Matched entities increment true positives, while distance contributes to both false positives and false negatives.
- Combined metric: SLU-F1 combines Word-F1 and Char-F1 by evaluating the summed confusion matrices from both metrics.The resulting score integrates word- and character-level entity comparison.
5 Experiments
Experiments evaluate ASR, NLU, and combined SLU pipelines on SLURP using competitive systems. Multi-ASR adapted with SLURP data performs best acoustically, while HerMiT achieves the strongest reported semantic and combined-system results.
- Experimental setup: The experiments assess two ASR baselines, state-of-the-art NLU systems, and several ASR–NLU pipelines on SLURP.The evaluation covers acoustic, semantic, and combined SLU performance.
- 5.1 Acoustic evaluation: Multi-ASR uses 24,000 hours of augmented out-of-domain audio and an interpolated language model, providing a competitive baseline against Google-ASR.The system is built with Kaldi and includes environmental-robustness augmentation.
- 5.1 Acoustic evaluation: 1.6% improvement follows from adding synthetic data to SLURP-ASR, while synthetic data alone produces 98% WER.These results concern acoustic models trained with SLURP data and synthetic data.
- 5.1 Acoustic evaluation: Multi-ASR adapted to SLURP achieves the best acoustic performance, but Multi-ASR generally outperforms systems trained only on in-domain SLURP data.The authors attribute the remaining challenge to limited acoustic data relative to SLURP’s lexical richness and noise conditions.
- 5.2 Semantic evaluation: HerMiT and SF-ID provide attention-based NLU baselines with different propagation directions: top-down for HerMiT and entity-first for SF-ID.HerMiT predicts Scenario, Action, and Entities hierarchically; SF-ID executes entity detection before intent detection.
- 5.2 Semantic evaluation: HerMiT with Multi-ASR achieves top performance across Scenario, Action, and their combination, remaining approximately 5% below gold-transcription performance despite 16% WER.The strongest configuration is HerMiT with Multi-ASR adapted using SLURP data.
- 5.2 Semantic evaluation: HerMiT is the stronger NLU system on gold-transcribed data and outperforms other combinations for entity prediction with Multi-ASR.Table 8 reports system performance on entity prediction.
6 Error Analysis
The error analysis uses SLU-F1 distributions to separate ASR and NLU effects in entity prediction. HerMiT has lower overall error rates than SF-ID except for ASR errors and appears more robust to noise propagation.
- Error propagation: HerMiT has lower error rates than SF-ID for all categories except ASR errors and can recover correct entities from mismatched transcriptions.The comparison is performed on noisy ASR data.
- Metric behavior: Word-F1 and Char-F1 produce the same system ordering but are weakly correlated, with ρ = 0.2 and p ≪0.0001.The authors interpret this as evidence that the metrics capture different aspects of performance.
- Distribution analysis: For HerMiT/Multi-ASR, entity-level distance distributions are examined against sentence-level WER to characterize system behavior.The analysis uses distributions rather than only an overall score.
- Distribution analysis: With sentence-WER=0, remaining errors are NLU span misalignments; with sentence-WER > 0, entities commonly fall into distance ranges (0, 0.5] or (0.5, 1].Examples include shortened or extended spans and mis-transcribed entities.
- Distribution analysis: Character-level distance identifies negligible ASR variations such as morphological changes, compound splitting or merging, and small spelling differences.Examples include singular/plural changes and Sara versus Sarah.
7 Discussion
SLURP is deliberately a substantially more challenging benchmark than prior datasets, and existing end-to-end systems were unable to produce meaningful outputs on it. State-of-the-art modular systems nevertheless provide a strong baseline that remains ahead of current end-to-end systems.
- SLURP is a magnitude more challenging than previous datasets and targets limitations in scalability and data efficiency of current end-to-end approaches.The release is intended as a use-case-inspired challenge rather than merely another benchmark dataset.
- Figure 6 relates sentence-level WER to entity-level WER and normalized character-based Levenshtein values for error analysis.Sentence-level WER is grouped in intervals of 0.5; entity-level WER uses intervals of 0.5 and normalized Levenshtein values use intervals of 0.1.
- Existing state-of-the-art end-to-end models that perform well on the FSC corpus did not converge or produce meaningful outputs after retraining on SLURP.Those models were developed for easier tasks, such as single-domain settings.
- State-of-the-art modular approaches provide a strong baseline on SLURP that current state-of-the-art end-to-end systems have not yet matched.The paper presents this modular baseline as closer to how real-world applications commonly build SLU systems.
8 Conclusion
The paper introduces SLURP as a larger SLU resource, adds SLU-F1 for transparent entity-prediction analysis, and compares modular processing choices. Sequential decoding performs better for entity tagging, while extending the dataset to spontaneous speech remains future work.
- SLURP is a substantially larger public SLU dataset whose greater linguistic challenge is demonstrated through linguistic analysis and reduced ASR and NLU performance.
- SLU-F1 evaluates entity prediction through an inspectable distribution that helps developers identify error types and system weaknesses.
- Sequential decoding from scenario and action to entities produces better entity-tagging results than bottom-up decoding from entities.
- The error analysis suggests that sequential decoding handles noise better by priming entity tagging, which is more difficult than scenario or action recognition.
- Future work will extend SLURP with spontaneous speech to increase complexity and move the resource closer to real-life applications.