Source-linked AI summary
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Pete Warden
TL;DR
Keyword spotting requires specialized data because on-device models continuously process mostly non-speech under tight compute and energy constraints. The paper introduces Speech Commands and reproducible evaluation protocols for this setting, with version 2 improving results on equivalent test data over the original dataset.
Problem
Keyword spotting differs from full-sentence recognition because on-device systems continuously process silence, noise, and unrelated speech under limited compute and energy budgets.
Method
The paper builds a specialized Speech Commands dataset and defines shared training, validation, testing, and accuracy procedures for reproducible keyword-spotting evaluation.
Results
Version 2 shows improved results on equivalent test data compared with the original dataset.
Takeaways & Limitations
Speech Commands provides a common basis for comparing keyword-spotting models and reproducing baseline models trained on identical data.
Takeaways & Limitations
Top-One evaluation uses utterances aligned to their beginnings and equally weights categories, which does not capture continuous-stream conditions or typical trigger and silence distributions.
Abstract
from arXiv · showhide
Describes an audio dataset of spoken words designed to help train and evaluate keyword spotting systems. Discusses why this task is an interesting challenge, and why it requires a specialized dataset that is different from conventional datasets used for automatic speech recognition of full sentences. Suggests a methodology for reproducible and comparable accuracy metrics for this task. Describes how the data was collected and verified, what it contains, previous versions and properties. Concludes by reporting baseline results of models trained on this dataset.
2 Introduction
Speech Commands addresses the limited availability of accessible speech datasets by providing a standard resource for training and evaluating small keyword-spotting models. Its open license and defined task are intended to support reproducible comparisons and broader participation.
- Broader dataset access can encourage collaboration and apples-for-apples comparisons between approaches.
- Speech Commands targets models that detect one spoken word from ten or fewer targets while minimizing false positives from noise or unrelated speech.
- The Creative Commons BY 4.0 license allows the dataset to be incorporated into tutorials and scripts without registration or permission requests.
3 Related Work
Existing speech datasets differ in licensing, recording conditions, file formats, and annotation granularity, limiting their suitability for keyword spotting. In particular, sentence-level alignment in larger speech corpora is better suited to full-sentence recognition.
- The collection process used volunteers reading requested phrases through a web application.
- LibriSpeech contains 1,000 hours of read English speech but has sentence-level rather than word-level labels, making it more suitable for full automatic speech recognition.
- TIDIGITS contains 25,000 digit sequences from 300 speakers, but uses a commercial license and the difficult-to-decode NIST SPHERE format.
- CHiME-5 provides 50 hours of home-recorded speech in 16 KHz WAV files under a restricted license, with sentence-level alignment.
4 Motivations
Keyword spotting serves as an on-device trigger for voice interfaces, where continuous local processing must operate under compute, energy, and false-positive constraints. Speech Commands is intended to make such systems comparable through a shared dataset, metrics, and reproducible baselines.
- Keyword spotting detects trigger phrases locally before audio is sent to a cloud speech-recognition service.
- On-device models must use fewer calculations and energy than cloud models because mobile hardware has limited compute, battery, and thermal capacity.
- Keyword-spotting systems must minimize false positives from silence, background noise, and unrelated speech.
- The task recognizes a single word or short phrase rather than an entire sentence, making general speech datasets difficult to adapt directly.
- The dataset supports comparable architecture metrics and reproducible baseline training on identical data.
- A publicly available task can help hardware manufacturers compare accuracy and energy usage against product requirements.
5 Collection
The dataset was collected to reflect natural, noisy on-device trigger-phrase conditions while keeping recordings short, speaker-independent, privacy-conscious, and easy to gather. An open web application captured and uploaded one-word utterances under controlled vocabulary and consent procedures.
- 5.1 Requirements: Utterances were recorded through phone or laptop microphones in users’ ordinary environments rather than studios, to reflect noise and lower-quality equipment.
- 5.1 Requirements: Background conversations were excluded so contributors could record alone with the door closed for privacy reasons.
- 5.1 Requirements: The collection focused on English for pragmatic scope and quality-control reasons while seeking varied accents and enabling future multilingual collection.
- 5.1 Requirements: Speaker independence motivated a quick, easy recording process intended to maximize participation.
- 5.1 Requirements: The dataset avoided personally identifiable contributor information and required agreement to a data-usage policy.
- 5.1 Requirements: Recordings were restricted to one-second isolated words, simplifying training, evaluation, and labeling while excluding longer words.
- 5.1 Requirements: The core vocabulary contained twenty common words, including digits zero through nine and command words for IoT or robotics applications.
- 5.3 Implementation: An open-source web application used WebAudioAPI to record utterances on supported desktop browsers and Android devices.
5.4 Quality Control
The dataset used layered quality-control procedures to reject unintelligible, incorrect, extremely short, or quiet recordings, then standardized audio into 16 KHz WAV files.
- 5.4 Quality Control: Human-listener criteria rejected clips when the spoken word was unintelligible or sounded incorrect.This guideline was applied through several review layers.
- 5.4 Quality Control: Files smaller than 5 KB were treated as unlikely to contain valid recordings and targeted for removal.The heuristic specifically addressed extremely short or quiet OGG clips.
- 5.4 Quality Control: OGG recordings were converted to uncompressed PCM WAV files at 16 KHz for easier further processing.The conversion used a command-line ffmpeg pipeline.
- 5.4 Quality Control: Audio from other sources was also resampled to 16 KHz WAV format.
5.5 Extract Loudest Section
The Extract Loudest Section tool addressed residual silence and inconsistent word placement by locating the loudest portion of each recording and extracting a one-second clip.
- 5.5 Extract Loudest Section: Manual inspection found many recordings that were too quiet or silent, while word alignment within the 1.5-second files varied with response speed.
- 5.5 Extract Loudest Section: The tool examined overall clip volume by summing the absolute differences of audio samples.
- 5.5 Extract Loudest Section: It extracted the one-second segment with the highest overall volume to approximate alignment and center the spoken word.This assumes the utterance is the loudest part of the recording.
- 5.5 Extract Loudest Section: The processing workflow was run from a dedicated extraction directory and wrote results to a trimmed_wavs directory.
5.6 Manual Review
Manual review supplemented automated filtering by using crowdsourced transcription checks to identify recordings whose words were wrong or unintelligible, while anonymized speaker identifiers preserved stable release-level linkage.
- 5.6 Manual Review: Crowdsourcing workers typed the word they heard from each clip using the expected-word list as examples.
- 5.6 Manual Review: Clips whose worker responses did not match their expected labels were filtered out.Each clip was evaluated by a single worker.
- 5.6 Manual Review: Speaker identifiers from the web application and paid crowdsourcing sources were hashed into a common eight-digit hexadecimal format.The stated purpose was removing links to worker IDs and personally identifiable information.
- 5.6 Manual Review: The stable hash function allowed speaker identifiers to remain consistent across future releases.
5.8 Background Noise
Background-noise data was added to train and test speech-versus-no-speech detection, combining real recordings from noisy environments with mathematically generated noise and a dedicated archive folder.
- 5.8 Background Noise: The dataset added several minute-long 16 KHz WAV files containing background noise to support speech-versus-no-speech detection.
- 5.8 Background Noise: Some noise recordings came directly from environments such as running water or machinery.
- 5.8 Background Noise: Other background-noise files were generated mathematically with Python commands.The supplied commands include separate white- and pink-noise generation steps.
- 5.8 Background Noise: Noise files were placed in a specially named _background_noise_ folder at the archive root.
6 Properties
The final dataset contains 105,829 utterances spanning 35 words, recorded from 2,618 speakers and stored as compact one-second-or-less audio files.
- 105,829 utterances cover 35 words in the final dataset.
- 2,618 speakers are represented, each identified by a unique eight-digit hexadecimal identifier.
- Each utterance is a one-second-or-less WAVE file using 16-bit mono PCM sampled at 16 KHz.
- The uncompressed dataset occupies approximately 3.8 GB, or 2.7 GB as a gzip-compressed tar archive.
7 Evaluation
The evaluation framework defines reproducible Top-One and streaming metrics, using fixed data splits, category weighting, and shared test audio. Baselines include 88.2% Top-One accuracy and streaming scores that separate matched, wrong, and false-positive detections.
- Evaluation protocol: The dataset specifies validation and testing file lists so models can be trained and evaluated on reproducible, non-overlapping utterance sets.The accompanying code implements the set-assignment procedure.
- Top-One Error: Top-One evaluation uses ten target words plus Unknown Word and Silence, with equal numbers of examples from all twelve categories.Each category contributes approximately 8.3% of the test set.
- Top-One Error: Top-One evaluation is limited because aligned utterances and equal category weighting do not represent continuous real-world audio streams.Product models must detect word boundaries and handle the differing prevalence of triggers and silence.
- Streaming Error Metrics: Streaming evaluation scores continuous audio with matched, wrong, and false-positive percentages, rather than relying on a single accuracy value.Matched detections are time-tolerant, wrong detections identify speech with an incorrect class, and false positives occur during nonspeech.
- Streaming Error Metrics: 49.0% matched and 46.0% correctly are reported for the V2 baseline streaming evaluation.The supplied result passage also begins a third percentage but does not provide its complete label or value.
- Streaming Error Metrics: A one-hour mixture of utterances and noise, paired with timestamped ground-truth labels, supports reproducible comparisons for streaming applications.The shared audio was generated using the TensorFlow tutorial script.
- Historical Evaluations: Historical evaluation reports 85.4% Top-One for the V1-trained model, 88.2% for V2 training on V2 data, and 89.7% for V2 training on the V1 test set.The paper states that V2 training data is responsible for a substantial improvement over V1.
8 Conclusion
The Speech Commands dataset supports training and evaluation across varied models, while version two improves results on equivalent test data relative to the original.
- Version two shows improved results compared with the original dataset on equivalent test data.