Source-linked AI summary
Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spotting
Ming Sun, Anirudh Raju, George Tucker, Sankaran Panchapagesan, Gengshen Fu, Arindam Mandal, Spyros Matsoukas, Nikko Strom, Shiv Vitaladevuni
TL;DR
Keyword spotting through large-vocabulary speech recognition can require substantial computation and introduce latency, while on-device use calls for compact, low-latency systems. This paper trains unidirectional LSTMs with max-pooling loss, optionally initialized from cross-entropy training, and reports a 67.6% AUC reduction for its best system in the low miss-rate range.
Problem
LVCSR-based keyword spotting requires relatively high decoding resources and introduces latency, while on-device systems need low CPU and memory use and low latency.
Source: §1 · p. 1Method
The paper trains unidirectional LSTMs for keyword spotting with max-pooling loss, optionally initialized from a cross-entropy-trained network, and evaluates smoothed posterior thresholding.
Source: §1 · pp. 1–2, §2.2.2 · p. 3, §4 · p. 6Results
The best LSTM system, trained with max-pooling loss and cross-entropy pre-training, reduces AUC by 67.6% in the low miss-rate range.
Source: §4 · p. 6Takeaways & Limitations
The reported results favor max-pooling loss with cross-entropy pre-training among the evaluated LSTM systems for keyword spotting.
Source: §4 · p. 6, §3.2 · p. 6Evaluation counts a firing within an Nlat-frame window after a keyword segment as aligned with that keyword.
Source: §2.3 · p. 4
Abstract
from arXiv · showhide
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothing based evaluation approach is employed to measure keyword spotting performance. Our experimental results show that LSTM models trained using cross-entropy loss or max-pooling loss outperform a cross-entropy loss trained baseline feed-forward Deep Neural Network (DNN). In addition, max-pooling loss trained LSTM with randomly initialized network performs better compared to cross-entropy loss trained LSTM. Finally, the max-pooling loss trained LSTM initialized with a cross-entropy pre-trained network shows the best performance, which yields $67.6\%$ relative reduction compared to baseline feed-forward DNN in Area Under the Curve (AUC) measure.
1. INTRODUCTION
Large-vocabulary speech recognition can make keyword spotting computationally demanding and introduce latency, while on-device systems need low CPU and memory use. This paper’s main contribution is applying max-pooling loss to unidirectional LSTMs for small-footprint keyword spotting.
1. INTRODUCTION
- LVCSR-based keyword spotting requires relatively high computational resources for decoding and introduces latency.It detects keywords by searching the resulting lattices or confusion networks.Source: §1 · p. 1
- On-device keyword spotting needs low CPU and memory use, high recall, low false accepts, and low latency.The systems stream audio to the cloud for LVCSR only after detecting a keyword on device.Source: §1 · p. 1
- The paper’s main contribution is applying max-pooling loss to LSTM training for keyword spotting.The authors focus on unidirectional LSTMs because the low-latency constraint makes bidirectional LSTMs an unsuitable fit in principle.Source: §1 · pp. 1–2
- The system triggers when a sliding-window-averaged keyword posterior exceeds a threshold, and locks out after each detection.The lockout is designed to avoid unnecessary false accepts and reduce decoding computational cost.Source: §1 · p. 2
2. SYSTEM OVERVIEW
The system processes acoustic features with a unidirectional LSTM and evaluates keyword detections using posterior smoothing. Its max-pooling loss trains on the highest-posterior frame in each keyword segment while retaining background-frame losses.
2. SYSTEM OVERVIEW
- The system uses 20-dimensional Log Mel Filter-Bank Energies over 25 ms frames with a 10 ms shift; the LSTM outputs keyword and non-keyword targets for evaluation.The evaluation module receives the keyword-spotting system output for decision making.Source: §2 · p. 2
2.1. LSTM
- LSTM memory blocks use input, output, and forget gates to control information flow; a projection layer can reduce model complexity.The projection linearly maps the LSTM output to a lower-dimensional recurrent signal, also used to compute the network output.Source: §2.1 · p. 2
2.2. Loss functions
- The training considers cross-entropy and max-pooling loss; cross-entropy for a sequence is formed from frame-level losses.The paper formulates the frame loss using the aligned target class and then gives the loss for the full frame sequence.Source: §2.2 · p. 3, §2.2.1 · p. 3
2.2.2. Max-pooling
- For each keyword segment, max-pooling back-propagates only from its highest-posterior frame, while cross-entropy is calculated for all background frames.The authors describe this as a transition from frame-level to segment-level loss and suggest it may mitigate inaccurate keyword-boundary alignment.Source: §2.2.2 · p. 3
- A cross-entropy-trained LSTM can initialize max-pooling training, providing basic keyword knowledge as a starting point.The paper contrasts this with random initialization, which may learn poorly in early epochs when keyword firing is random.Source: §2.2.2 · pp. 3–4
2.3. Evaluation Method
- Evaluation smooths keyword posteriors over a sliding context window, triggers above a threshold, and applies a lockout after each firing.The first firing in a keyword segment plus its latency window is a valid detection; additional firings there or firings outside those windows count as false accepts.Source: §2.3 · p. 4
- A short latency window after each keyword segment allows a firing to count as aligned with that keyword.The evaluation uses miss rate and normalized false accept rate as performance metrics.Source: §2.3 · p. 4
3. EXPERIMENTAL RESULTS
Experiments use an in-house far-field corpus for spotting “Alexa,” with an order of magnitude more keyword and background utterances than the largest previous studies. In the low-miss-rate range, max-pooling LSTM with cross-entropy pre-training achieves the largest AUC reduction against the baseline DNN.
3. EXPERIMENTAL RESULTS
- The in-house far-field corpus targets “Alexa” and contains an order of magnitude more keyword and background utterances for training and testing than the largest previous studies.The authors describe far-field data as a more challenging task by nature.Source: §3 · p. 4
3.1. Model Training
- Training halves the learning rate and repeats an epoch when development-set loss degrades, stopping at the minimum learning rate or 20 epochs.The initial learning rate and batch size are tuned on the development set.Source: §3.1 · p. 5
- The baseline DNN has four 128-node hidden layers and ~129K parameters; the unidirectional LSTM uses 64 memory blocks, a 32-dimensional projection, and ~118K parameters.The LSTM also uses 10 frames of left and right context.Source: §3.1 · p. 5
3.2. System Performance
- Figure 5 compares DET curves over a miss-rate range of ≤20%, with false accept rate on the x-axis and miss rate on the y-axis; lower values indicate better performance.False accept counts are obscured, and rates are plotted up to a multiplicative constant.Source: §3.2 · pp. 5–6
- 67.6% AUC reduction comes from the max-pooling LSTM initialized with cross-entropy pre-training; random-initialized max-pooling and cross-entropy LSTMs yield 48.2% and 34.4% reductions, respectively, versus baseline DNN.AUC is computed on DET curves, where lower is better.Source: Table 1 · p. 6, §3.2 · p. 6
4. CONCLUSION AND FUTURE WORK
The paper trains small-footprint LSTMs to spot “Alexa” in far-field conditions and reports the best result with max-pooling loss and cross-entropy pre-training. Future work would vary the loss weighting and model structure, and benchmark against other segmental losses.
4. CONCLUSION AND FUTURE WORK
- 67.6% AUC reduction is reported for the best LSTM, trained with max-pooling loss and cross-entropy pre-training, in the low-miss-rate range.The comparison is against the baseline feed-forward DNN.Source: §4 · p. 6
- The evaluation uses smoothed-posterior thresholding and measures keyword spotting with miss rate and false accept rate.The target is “Alexa” in far-field conditions.Source: §4 · p. 6
- Planned comparisons include weighted max-pooling loss, additional feed-forward layers, and segmental losses such as geometric mean of framewise posteriors and CTC.The proposed weighting would scale back-propagated loss for selected keyword frames.Source: §4 · p. 6