Source-linked AI summary
Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, Joseph Dureau
TL;DR
Small IoT devices need accurate spoken-language understanding without the large models, computational resources, and cloud dependence associated with modern speech recognition. The paper presents the offline Snips Voice Platform, combining compact ASR and NLU methods with privacy-preserving data generation, and reports real-time embedded operation with strong performance. Its scope remains bounded by limitations in data-generation approaches and by future work still needed for privacy-preserving analytics and federated learning.
Problem
Large deep-learning ASR models require model sizes and computational resources that make real-time deployment on small devices impractical, encouraging cloud-dependent speech recognition.
Method
The paper designs an offline embedded SLU platform using compact ASR and NLU models, mutually consistent language modeling, confidence scoring, and privacy-preserving synthetic data generation.
Results
The platform runs faster than real time on Raspberry Pi 3 and NXP imx7D, with language-model improvements preserving decoding accuracy while drastically reducing model size.
Takeaways & Limitations
Snips demonstrates that high-performance spoken-language understanding can run offline on small devices without collecting or storing user queries.
Takeaways & Limitations
Grammar-based data generation produces highly correlated queries with limited diversity and is unfit for natural-language training data.
Abstract
from arXiv · showhide
This paper presents the machine learning architecture of the Snips Voice Platform, a software solution to perform Spoken Language Understanding on microprocessors typical of IoT devices. The embedded inference is fast and accurate while enforcing privacy by design, as no personal user data is ever collected. Focusing on Automatic Speech Recognition and Natural Language Understanding, we detail our approach to training high-performance Machine Learning models that are small enough to run in real-time on small devices. Additionally, we describe a data generation procedure that provides sufficient, high-quality training data without compromising user privacy.
1 Introduction
The Snips Voice Platform is designed as a private-by-design, cloud-independent SLU system for embedded IoT devices. It addresses the privacy and resource constraints that make large cloud-dependent speech-recognition models unsuitable for small hardware.
- Privacy-by-design is motivated by the sensitivity of voice data and the risks associated with voice-based speaker identification.The paper notes that European data-protection rules require explicit consent before collecting user data.
- The platform runs voice-assistant inference offline on typical IoT microprocessors, so user data is never collected or stored.Its design separates server-side training from on-device inference.
- The Snips ecosystem combines a web console with ASR and NLU components for building assistants, skills, intents, and entity slots.Dialog management uses extracted information to provide feedback or trigger actions on connected devices.
- Large deep-learning ASR models achieve strong accuracy but require model sizes and computational resources that prevent real-time deployment on small devices.Such systems therefore commonly rely on cloud-based speech recognition.
- The paper develops compact, efficient SLU models and data-generation methods to support reliable private inference under tight size and computational constraints.The approach includes small acoustic models, consistency between ASR language modeling and NLU, and automatically generated training data instead of user data.
2 Acoustic model
The acoustic model is a hybrid NN/HMM component designed to provide accurate, real-time ASR on small devices while respecting tight memory and compute constraints. The approach combines privacy-preserving data augmentation, configurable model sizes, and language-model adaptation to balance accuracy, speed, and footprint.
- Pipeline role: The acoustic model converts audio into phone-state probabilities that a language model decodes into the user’s words.It is the first SLU stage, so decoding errors can affect downstream intent and slot extraction.
- Training data: Training uses realigned public or commercial speech data, split into training, testing, and development sets.Transcript realignment matches speech to timestamps and helps remove transcription errors.
- Data augmentation: Thousands of simulated rooms augment mostly close-field speech with noisy, far-field, and reverberant conditions without collecting user utterances.Random room sizes and microphone and speaker locations are used to simulate rerecordings.
- Model architecture: The models use a configurable TDNN-LSTM neural architecture within a hybrid NN/HMM acoustic-modeling recipe.Different layer and neuron counts support deployment targets with different accuracy and computational requirements.
- Accuracy evaluation: Adding data has little effect on clean LibriSpeech test sets but lowers WER on datasets marked as other, while augmentation improves noisy and reverberant performance.The reported experiments vary training-data quantity and include augmented clean data.
- Accuracy–efficiency tradeoff: Larger networks generalize better, but their memory and computational costs can prohibit small-device deployment.The selected model balances accuracy with real-time factor and storage and RAM requirements; nnet-256 has 2.6M parameters versus 15.4M for nnet-768 and runs 6 to 10 times faster.
3 Language Modeling
The language-modeling component combines an LM that maps acoustic-model predictions to likely sentences with an NLU model that extracts intents and slots. Domain specialization reduces model size and decoding cost while preserving speech-to-meaning performance within supported use cases.
- The language-modeling component contains an LM for likely sentence prediction and an NLU model for extracting intents and slots from ASR output.The LM uses word co-occurrence probabilities, while the NLU processes the ASR engine’s decoded query.
- Large-vocabulary LMs can reach terabyte scale and expand the ASR search space, increasing memory and computational demands.
- Domain-specialized LMs restrict vocabulary and query variety to sentences supported by the NLU model.This focuses modeling capacity on the assistant’s intended domain rather than all-purpose spoken language.
- Specialization greatly reduces model size and increases decoding speed while providing robust in-domain ASR under hardware constraints.The paper evaluates the SLU system through end-to-end speech-to-meaning accuracy rather than ASR word error rate alone.
3.1 Data
The shared LM and NLU dataset consists of written, entity-dependent queries annotated with intents and slots. Preprocessing verbalizes and normalizes these examples consistently for both training components so ASR outputs match NLU inputs.
- The same dataset is used to train both the language model and the NLU model.
- Training examples are written queries that exemplify intents depending on entities.
- Entities encode possible values for intent attributes, while slots identify the particular entity values appearing in queries.Entities may be shared across intents, such as room values used by both SwitchLightOn and SwitchLightOff.
- Custom entities use exhaustively listed values, whereas built-in entities represent common value types that users cannot easily enumerate.Examples include numbers, dates, times, durations, and amounts with units.
- Queries are associated with intents and annotated by labeling text chunks with their corresponding entities.For example, “kitchen” and “65” can be labeled as room and brightness slots in a lighting query.
- The dataset is tokenized, normalized, and verbalized into canonical lexicon-matching forms before training.The same preprocessing is applied to NLU training to ensure ASR words match those seen by the NLU.
3.2 Language model
The Snips language model uses class-based n-gram modeling and weighted finite-state transducers to represent domain patterns and interchangeable entity values. Delayed composition and modular entity models reduce embedded decoding costs while supporting personalization and confidence-aware recognition.
- Language-model construction: ASR decoding maps acoustic-model outputs to word sequences through Viterbi search over a composed wFST graph H∗C∗L∗G.H, C, L, and G represent context-dependent HMMs, context dependency, the lexicon, and the language model.
- Language-model construction: Statistical n-gram and class-based modeling let the ASR generalize to unseen in-domain queries and interchangeable entity values.
- Language-model construction: Entity replacement abstracts training queries into patterns, trains Gp on those patterns, and represents each entity’s possible values with Gei.Custom entity acceptors can encode exhaustively listed values or use entity-specific n-gram models.
- Language-model construction: G = Replace(Gp, {Gei, ∀i ∈[1, n]}) expands entity symbols in the pattern model into entity transducers, enabling unseen formulations and value swapping.In the “play ARTIST” example, ARTIST arcs are expanded with the artist entity model.
- Dynamic decoding: The static decoding graph requires determinization and minimization, but its memory footprint can consume much of a Raspberry Pi 3’s 1GB RAM or exceed smaller devices.
- Dynamic decoding: Delayed composition builds transducer states and transitions at runtime, splitting the graph into smaller pieces whose combined size is typically several times smaller than static HCLG.The pieces include HCL and G, with further delayed replacement separating the pattern and entity language models.
- Dynamic decoding: The resulting improvements achieve real-time Raspberry Pi 3 decoding with small overhead, preserved accuracy, drastically reduced disk size, and simpler on-device entity updates.
- Personalization: Entity injection adds user-specific vocabulary, such as contacts, to an embedded entity model without sending sensitive data off-device.
3.3 Natural Language Understanding
Snips NLU extracts intents, slots, and normalized built-in entities through deterministic and probabilistic parsing. Its feature-rich models and embedded implementation support competitive accuracy with low memory and runtime requirements.
- Pipeline: Snips NLU successively performs intent classification, slot filling, and entity resolution to extract structured data from natural-language queries.Entity resolution converts values such as dates, temperatures, and numbers into normalized forms.
- Models: The NLU Engine first applies a deterministic intent parser, then invokes a probabilistic parser only when nothing is extracted.The deterministic parser uses regular-expression patterns generated from training examples, while the probabilistic parser generalizes beyond them.
- Models: Additional entity, word-cluster, gazetteer, and dropout features improve slot filling and reduce generalization error.Word clusters reduce the effective vocabulary from typically 50K words to a few hundred clusters.
- Evaluation: Snips NLU is as accurate or better than competing cloud-based solutions in slot filling across the evaluated training-set sizes.The benchmark compares precision, recall, and F1-score on identical datasets against five cloud-based providers.
- Deployment: Rust-based inference keeps memory usage from a few hundred kilobytes of RAM to a few megabytes and supports deployment on Raspberry Pi or mobile devices.The Rust pipeline also provides low inference runtimes, with measurements reported in Table 8.
4 End-to-end Evaluation
The evaluation measures end-to-end speech-to-meaning performance on SmartLights and Weather assistants. The resulting language models run faster than real time on small hardware while supporting generalization to unseen queries.
- Evaluation setup: The end-to-end evaluation applies ASR followed by NLU to crowdsourced speech and compares predicted intents and slots with dataset ground truth.It evaluates two real-world assistants of different sizes: SmartLights and Weather.
- Generalization: Five-fold cross-validation tests whether the SLU components generalize from four-fifths of each domain dataset to an unseen fifth.The training procedure is otherwise identical to that described earlier in the paper.
- Generalization: Confidence scoring correlates sentence-level confidence with word error rate and can reject uncertain words to favor precision over recall.This strategy is especially relevant when unseen test queries contain out-of-vocabulary words.
- Runtime: The SmartLights and Weather SLU components each total 1.5MB including the acoustic model and run faster than real time on a Raspberry Pi 3 and NXP imx7D.The models are trained in under thirty seconds through the Snips web console.
5 Training models without user data
Because private-by-design assistants cannot collect user queries, Snips combines crowdsourcing and machine learning to generate and validate training data. More training queries substantially improve slot-filling performance, especially for complex intents.
- Generation methods: Grammar-based generation provides exact intent and slot supervision but produces correlated, linguistically limited queries and requires time-consuming pattern enumeration.The paper therefore describes this approach as unfit for generating natural-language queries.
- Generation methods: Crowdsourcing increases formulation diversity but can introduce intent and slot-labeling errors that affect end-to-end SLU performance.The pipeline fixes entity values during generation and uses validation tasks to limit annotation errors.
- Evaluation: 32% absolute average F1 improvement occurs with 500 rather than 10 training queries, ranging from 22% for RateBook to 44% for GetWeather.RateBook rises from 0.76 to 0.98, whereas GetWeather rises from 0.44 to 0.88; gains depend strongly on intent complexity.
- Motivation: Private-by-design assistants need generated training queries because they do not gather user queries, and sufficient data supports both model training and pre-deployment validation.The paper identifies this need as particularly important for assistant-specific language models and NLU engines.
6 Conclusion
The paper presents Snips as an offline, embedded SLU platform designed to preserve privacy on small IoT devices. It combines compact high-performing ASR/NLU models with privacy-preserving training-data generation.
- Contributions: Snips Voice Platform runs entirely offline on small devices, and assistants never send user queries to the cloud.The conclusion frames this design as compliant with the privacy-by-design principle.
- Contributions: The paper describes compact acoustic models, consistent ASR and NLU language modeling, real-world SLU evaluation, and crowdsourcing-plus-machine-learning data generation.These components address performance, deployment constraints, and training without compromising user privacy.
- Future work: Future research directions include privacy-preserving analytics and federated learning as a complement to data generation.These directions are presented as part of a broader effort toward more private and ubiquitous artificial intelligence.
Appendix: NLU benchmark on an in-house dataset
The appendix presents an in-house slot-filling dataset summary and reports aggregate precision, recall, and F1-score results for all slots.
- The in-house dataset summary is organized by intent name, slots, samples, and number of utterances.
- Aggregate slot-filling performance is evaluated using precision, recall, and F1-score averaged across all slots.The evaluation is reported for an in-house dataset run in June 2017.
- A second table also reports precision, recall, and F1-score averaged across all slots in the June 2017 in-house evaluation.