Source-linked AI summary
Personalized Speech recognition on mobile devices
Ian McGraw, Rohit Prabhavalkar, Raziel Alvarez, Montse Gonzalez Arenas, Kanishka Rao, David Rybach, Ouais Alsharif, Hasim Sak, Alexander Gruenstein, Francoise Beaufays, Carolina Parada
TL;DR
Mobile speech recognition must balance accuracy, latency, memory, and computation when network access is unreliable. This paper combines compact LSTM acoustic modeling with shared and personalized language-model techniques, producing an on-device system that runs 7× faster than real-time with a 20.3 MB footprint.
Problem
Speech recognition run on remote servers can have high latency or fail when network connections are unreliable, motivating accurate local systems with small resource requirements.
Method
The system uses a quantized CTC-trained LSTM acoustic model with context-independent phone outputs, SVD compression, Bayesian interpolation across dictation and voice commands, and vocabulary injection with on-the-fly language-model biasing.
Results
7× faster than real-time with a total system footprint of 20.3 MB on a Nexus 5.
Takeaways & Limitations
The combined techniques yield a compact large-vocabulary speech recognition system that runs efficiently on mobile devices with low latency.
Takeaways & Limitations
Language-model experiments are evaluated using the quantized LSTM context-independent CTC acoustic model, isolating language-model effects.
Abstract
from arXiv · showhide
We describe a large vocabulary speech recognition system that is accurate, has low latency, and yet has a small enough memory and computational footprint to run faster than real-time on a Nexus 5 Android smartphone. We employ a quantized Long Short-Term Memory (LSTM) acoustic model trained with connectionist temporal classification (CTC) to directly predict phoneme targets, and further reduce its memory footprint using an SVD-based compression scheme. Additionally, we minimize our memory footprint by using a single language model for both dictation and voice command domains, constructed using Bayesian interpolation. Finally, in order to properly handle device-specific information, such as proper names and other context-dependent information, we inject vocabulary items into the decoder graph and bias the language model on-the-fly. Our system achieves 13.5% word error rate on an open-ended dictation task, running with a median speed that is seven times faster than real-time.
1. INTRODUCTION
The paper targets accurate, low-latency speech recognition running locally on resource-constrained mobile devices. It combines compact acoustic and language models with personalization techniques for dictation and voice commands.
- Local mobile recognition can improve reliability and latency but must remain accurate while using limited memory and computation.
- Quantized LSTM models trained with CTC and sMBR improve WER and latency under similar size and computation constraints.The models use 8-bit parameters, context-independent phone outputs, and SVD compression.
- A single language model supports both dictation and voice commands through Bayesian interpolation, which outperforms simple linear interpolation.
- Vocabulary injection and on-the-fly language-model biasing improve accuracy for device-specific information such as personal contact names.
2. BASELINE SYSTEM
The baseline follows an embedded speech recognizer but replaces its feedforward DNN with a deep LSTM acoustic model. It uses delayed cross-entropy training and a compact first-pass language-model decoder with on-the-fly rescoring.
- The baseline uses three hidden LSTM layers with 850 cells each and recurrent projection layers of size 450.
- The acoustic model predicts 2,000 context-dependent states and is trained with cross-entropy using labels delayed by 5 frames.
- 40-dimensional log mel-filterbank features are computed every 10ms from 25ms windows, with one frame supplied per input step.
- The decoder restricts the vocabulary to 64K, uses a 70K-n-gram first-pass graph, and rescoring uses a roughly 1.5M-n-gram compact language model.The rescoring language model uses LOUDS compression.
3. ON-DEVICE ACOUSTIC MODELING
The on-device acoustic model uses a compact CTC LSTM that predicts context-independent phonemes, with training, compression, and quantization designed for mobile deployment. Experiments evaluate accuracy and runtime on open-ended dictation.
- ON-DEVICE ACOUSTIC MODELING: The CTC acoustic model has five hidden layers with 500 LSTM cells each and predicts 41 context-independent phonemes plus a blank target.
- ON-DEVICE ACOUSTIC MODELING: 8-frame stacking with 7 frames of right context and presenting every third stacked frame stabilizes CTC training and reduces computation.
- ON-DEVICE ACOUSTIC MODELING: The models use 3M anonymized hand-transcribed utterances, approximately 2,000 hours, with 20 synthetically distorted versions per utterance for multi-style training.
- AM Experiments: 13.3K anonymized utterances form the open-ended dictation test set, while runtime is benchmarked on 100 utterances using a Nexus 5.
- AM Experiments: The CTC-trained context-independent-phone LSTM outperforms the cross-entropy LSTM predicting 2,000 context-dependent states and runs about 4× faster with comparable parameter counts.
- AM Experiments: SVD compression jointly factorizes recurrent and inter-layer matrices to determine shared projection layers rather than randomly initializing high-rank-reduction projections.
- AM Experiments: 12.8% relative improvement over the SVD-compressed models comes from adapting the acoustic model with 1M anonymized dictation utterances.
- AM Experiments: 8-bit quantization reduces the acoustic model from 11.9 MB floating point to a final 3 MB footprint.
4. ON-DEVICE LANGUAGE MODELLING
The language-modeling system shares one compact model across dictation and voice commands, then compares interpolation and decoding strategies. Bayesian interpolation improves both domains, while on-the-fly rescoring reduces decoder storage requirements.
- ON-DEVICE LANGUAGE MODELLING: A single language model serves dictation and voice commands, using a 64K vocabulary and logs from approximately 100M dictation and 2M command utterances.
- ON-DEVICE LANGUAGE MODELLING: The voice-command test set covers Device, Planning, and Communication commands totaling approximately 19K utterances.
- ON-DEVICE LANGUAGE MODELLING: All language-modeling results use the quantized LSTM context-independent CTC acoustic model, isolating language-model effects.
- Decoder: The decoder graph includes context-independent phoneme states, a generic blank state, and blank-label self-loops for the CTC topology.
- ON-DEVICE LANGUAGE MODELLING: Linear-interpolation weights are estimated on held-out development data, while Bayesian task priors are swept to minimize word error rates.
- ON-DEVICE LANGUAGE MODELLING: Bayesian interpolation outperforms linear interpolation for both dictation and voice-command domains.
- ON-DEVICE LANGUAGE MODELLING: On-the-fly rescoring decomposes an unacceptable 29 MB fully composed decoder into an 8.3 MB graph and a 6.8 MB LOUDS-compressed rescoring language model.
5. DECODER
The decoder uses an FST-based search graph adapted to CTC’s single-phoneme-state topology, reducing explicit graph and search-state overhead for efficient decoding.
- CTC decoder topology: CTC decoding represents each phoneme with one acoustic-model output state plus a generic blank state, unlike conventional 3-state HMMs.The decoder graph is built by composing lexicon and language-model transducers without a context-dependency transducer.
- Decoder optimizations: Blank self-loop transitions are applied implicitly rather than stored as explicit decoder-graph arcs.This is one of the CTC-specific optimizations used to reduce computation time and memory usage.
- Decoder optimizations: Removing dynamic HMM-state-sequence expansion produces a more compact in-memory search space and simpler hypothesis expansion.The optimization targets both decoder memory usage and search procedure complexity.
- Personalized decoding evaluation: Table 3 reports the impact of contact injection and biasing on WER and latency.The supplied table caption identifies the two evaluation dimensions but does not provide their numerical values.
6. PERSONALIZATION
Personalization addresses device-specific contact names by injecting candidate vocabulary into the decoder and biasing recognition toward those names, with pronunciations supplied by an LSTM-based G2P model.
- Evaluation setup: Contact-name personalization is evaluated on message-context utterances and utterances containing only contact names.The experiments use an 8K-utterance Communication set and a separate 1.2K-utterance contact-only set.
- Vocabulary injection: A dynamic CONTACTS class lets the decoder inject a user’s personal contacts at run time to cover long-tail names beyond the language model vocabulary.Training replaces contact names with a CONTACTS placeholder, while run-time decoding inserts a personal-contact FST at those locations.
- Pronunciation generation: An LSTM-based grapheme-to-phoneme model generates contact pronunciations from transcribed word-pronunciation pairs.The G2P model has four LSTM layers with 64 cells each and is trained with the CTC objective.
- Evaluation setup: For each utterance, N candidate contacts are injected into the decoder, and contact-only utterances are additionally tested with on-the-fly biasing.The evaluation includes cases where one injected candidate is the correct contact.
- Personalized recognition results: Biasing toward CONTACTS can produce dramatic improvements when a single contact name is expected, especially after narrowing candidates to two.The text attributes the gains to many test terms being out of vocabulary and reports that injecting 50 pronunciations on the fly does not significantly affect median real-time factors.
7. SYSTEM FOOTPRINT
Combining model compression, compact first-pass decoding, and on-the-fly rescoring yields a small system footprint without compromising the reported accuracy or latency.
- Overall footprint: 20.3 MB is the reported size of the overall speech-recognition system.The footprint combines SVD-based compression, quantization, compact first-pass decoding, and on-the-fly rescoring with a larger language model.
8. CONCLUSION
The paper concludes that compact mobile speech recognition combines a compressed CTC-based LSTM acoustic model with shared-language-model support and personalized decoding.
- Acoustic model: A CTC-based LSTM acoustic model predicts context-independent phones and is compressed to one-tenth its original size using SVD-based compression and quantization.The conclusion presents this compressed acoustic model as the core of the compact system.
- Language model and personalization: Bayesian interpolation supports open-ended dictation and voice commands with one shared language model.Personalization additionally combines vocabulary injection with on-the-fly language-model biasing.
- System result: The combined system runs 7× faster than real time on a Nexus 5 with a 20.3 MB total footprint.The result is attributed to on-the-fly rescoring and additional CTC decoder optimizations that reduce computation and memory usage.