Source-linked AI summary
PromptKWS: A Novel Prompt-Guided Open-Vocabulary Keyword Spotting Framework
Gaopeng Xu, Chengfei Li, Xianliang Wang, Lin Zhu, Juan Wei, Wenpeng Li, Jianwei Niu, Jie Gao
TL;DR
Open-vocabulary KWS must support customizable keywords despite dataset and adaptation challenges. PromptKWS introduces a lightweight prompt-embedding network and injects its representations into a KWS encoder through Prompt-acoustic MHCA. It improves accuracy and wake-up performance across general, rapid-speech, noisy, and few-shot settings.
Problem
Customizable open-vocabulary KWS remains challenging because existing systems require large target-keyword datasets and inflexible keyword configurations.
Method
PromptKWS uses a lightweight PPN to encode keyword prompts and injects the embeddings into a prompt-guided KWS encoder through Prompt-acoustic MHCA.
Results
PromptKWS significantly outperforms the baseline across general, rapid-speech, noisy, and few-shot test settings.
Takeaways & Limitations
PromptKWS provides a compact prompt-guided framework for adapting open-vocabulary KWS to challenging speech conditions and limited-shot scenarios.
Abstract
from arXiv · showhide
In this paper, we present PromptKWS, a novel Prompt-guided keyword spotting (KWS) framework to improve the accuracy of open vocabulary KWS systems. In specific terms, we introduce the Prompt Phrases Prediction Network (PPN), an encoder-decoder architecture designed to effectively extract keyword prompts embeddings. we employ the PPN encoder to encode the keyword prompts and infuse the prompt embedding into the Prompt-guided KWS encoder by utilizing a Prompt-acoustic Multi-head Cross-attention (MHCA). Experiments show that PromptKWS improves the wakeup rate by over 10% compared to baseline system. Notably, another strength of PromptKWS is its ability to effectively leverage keyword prompts for adapting to complex real-world environments involving noise and pronunciation variations. In comparison to purely acoustic models, which often struggle in such situations, PromptKWS demonstrates remarkable performance, with an average accuracy improvement of over 15% in test sets.
1. Introduction
Open-vocabulary KWS supports customizable keyword detection but remains difficult to adapt to user-defined keywords in continuous speech. PromptKWS addresses this challenge by using lightweight keyword-prompt embeddings to improve performance without the large, inflexible keyword datasets required by existing approaches.
- Customizable KWS remains challenging because existing systems require large target-keyword datasets and inflexible keyword configurations.
- Open-vocabulary KWS commonly separates acoustic modeling from keyword search, enabling keyword extension through ASR-style search.
- Prior work has optimized acoustic-model architectures and added post-processing modules to improve KWS training, inference, and performance.
- PromptKWS uses keyword prompts to improve open-vocabulary KWS while keeping its prompt encoder lightweight for edge-device deployment.Its PPN extracts keyword-prompt embeddings with a smaller architecture than the BERT-based prompting used by PromptASR.
2. PromptKWS
PromptKWS combines prompt encoding, prompt-acoustic cross-attention, and CTC keyword decoding to condition speech recognition on keyword and scene prompts. Its PPN adds a prompt-prediction objective during training while remaining lightweight during inference.
- 2.1. Model Architecture: PromptKWS comprises a PPN module, a Prompt-guided KWS encoder, and a KWS decoder.The encoder receives acoustic features and PPN-derived keyword-prompt embeddings through cross-attention.
- 2.1. Model Architecture: The architecture is illustrated as a PromptKWS pipeline connecting the PPN, Prompt-guided KWS encoder, and KWS decoder.
- 2.4. KWS Decoder: The KWS decoder uses a linear layer with softmax activation to transform encoder outputs into probability distributions.
- 2.2. Prompt Phrases Prediction Network: The PPN encodes keyword prompts into fixed-length vectors and predicts prompt tokens with a decoder during training.Its cross-entropy loss encourages prompt embeddings to remain discriminative, while only the encoder is used during inference.
- 2.3. Prompt-guided KWS Encoder: The Prompt-guided encoder uses acoustic hidden states as queries and prompt embeddings as key-value pairs in Prompt-acoustic MHCA.
- 2.4. KWS Decoder: The KWS decoder is implemented with Connectionist Temporal Classification to generate token sequences for detected keywords.
- 2.5. Details of Prompts: PromptKWS supports keyword prompts containing wake-word information and scene prompts describing the keyword context.A special <no prompt> identifier handles operation without a specific keyword prompt.
- 2.6. Training Criterion: The joint training objective combines CTC loss for keyword labels with PPN cross-entropy loss for prompt phrase labels.λ weights the CTC and PPN loss terms.
3. Experiments
Experiments evaluate PromptKWS on Mandarin datasets using accuracy and ROC metrics, including general, rapid, noisy, and few-shot scenarios. PromptKWS consistently outperforms the baseline, with especially large gains in rapid and noisy speech.
- 92.3% accuracy on Dgeneral improves 11.4% over the baseline, while 87.9% on Drapid gains 13.8%.
- 15.9% accuracy improvement on Dnoisy raises PromptKWS performance to 85.4%.
- PromptKWS consistently outperforms the baseline across few-shot scenarios in Dfew.The supplied few-shot passage reports this overall comparison without specifying the figure’s axes beyond sample count.
- 61.0% accuracy in the 0-shot setting is 22.9% higher than the baseline, and 90.0% in the 15-shot setting remains 11.0% higher.
- 10.5% and 12.7% accuracy increases occur in noisy and rapid speech after integrating keyword prompt embeddings through Prompt-acoustic MHCA.This comparison isolates the Prompt-guided model from the baseline model.
- 3.1% average accuracy improvement across test sets follows adding the PPN loss to the guided model.The paper attributes this result to more discriminative prompt embeddings.
- ROC analysis covers general, rapid, and noisy speech, with noisy-scenario FRR of 0.146 for PromptKWS versus 0.309 for the baseline at zero false alarms.The noisy comparison represents a 16% improvement according to the reported analysis.
4. Conclusions
PromptKWS is presented as an open-vocabulary KWS approach combining a PPN module with a Prompt-guided KWS encoder to adapt to keyword prompts. Experiments report performance gains over the baseline across challenging real-world scenarios, including low-sample, rapid-speech, and high-noise conditions.
- PromptKWS combines a PPN module with a Prompt-guided KWS encoder based on Prompt-acoustic MHCA.The encoder dynamically adapts to the keyword according to its prompt.
- PromptKWS is compact at 2.7M parameters, supporting its suitability for edge-device deployment.
- PromptKWS experiments cover low-sample, rapid-speech, and high-noise environments.
- PromptKWS significantly outperforms the baseline across a wide range of scenarios.The conclusion highlights its superiority and adaptability in real-world applications.