Source-linked AI summary
The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines
Jon Barker, Shinji Watanabe, Emmanuel Vincent, Jan Trmal
TL;DR
The paper addresses the challenge of distant, noisy, overlapping conversational ASR in real home environments. It introduces the CHiME-5 dataset and challenge structure, including multi-array recordings, separate tracks, and baseline systems. The baselines reveal substantial array-versus-binaural performance differences, while end-to-end ASR remains data-limited but promising with binaural testing.
Problem
Distant-microphone recognition of noisy, overlapping conversational speech requires evaluation in realistic home environments beyond earlier laboratory-focused corpora.
Method
The paper constructs a 20-home dinner-party corpus with Kinect and binaural recordings, defines single- and multiple-array tracks with distinct rankings, and presents synchronization, enhancement, conventional ASR, and end-to-end baselines.
Results
The baseline results show a 33.4% absolute performance difference between array and binaural microphone results for LF-MMI TDNN, while end-to-end ASR performs poorly with limited training data.
Takeaways & Limitations
The publicly available dataset, baselines, and ranking instructions support comparable research across the recognition pipeline for conversational ASR in real homes.
Takeaways & Limitations
The end-to-end ASR baseline performs poorly because the available training data are insufficient, although binaural testing gives better results than the Kaldi GMM system.
Abstract
from arXiv · showhide
The CHiME challenge series aims to advance robust automatic speech recognition (ASR) technology by promoting research at the interface of speech and language processing, signal processing , and machine learning. This paper introduces the 5th CHiME Challenge, which considers the task of distant multi-microphone conversational ASR in real home environments. Speech material was elicited using a dinner party scenario with efforts taken to capture data that is representative of natural conversational speech and recorded by 6 Kinect microphone arrays and 4 binaural microphone pairs. The challenge features a single-array track and a multiple-array track and, for each track, distinct rankings will be produced for systems focusing on robustness with respect to distant-microphone capture vs. systems attempting to address all aspects of the task including conversational language modeling. We discuss the rationale for the challenge and provide a detailed description of the data collection procedure, the task, and the baseline systems for array synchronization, speech enhancement, and conventional and end-to-end ASR.
1. Introduction
The CHiME-5 Challenge targets distant, noisy, overlapping conversational ASR in real homes, addressing limitations of earlier laboratory-focused corpora. It introduces a large-scale multi-home dataset and separate tracks for acoustic robustness and broader system design.
- Distant-microphone recognition of noisy, overlapping conversational speech is presented as the next frontier for robust ASR.
- Earlier meeting and interaction corpora used specialized microphone arrays that could be installed in only a few laboratory rooms.
- CHiME-5 provides a large-scale corpus of real multi-speaker conversational speech recorded with commercially available hardware in multiple homes.
- The corpus uses a four-person dinner-party scenario recorded by six distant Kinect arrays and four binaural microphone pairs in 20 homes.
- Single-array and multiple-array tracks receive separate rankings for acoustic robustness versus systems addressing all aspects of the task.
2. Dataset
CHiME-5 records natural dinner-party conversations in real homes using distributed Kinect arrays and participant-worn binaural microphones. The dataset includes aligned, fully transcribed recordings with privacy and conversational-noise annotations.
- The dataset contains 20 dinner parties in real homes, each with four friends acting as two hosts and two guests.
- Each party lasts at least two hours and covers kitchen, dining, and living phases while participants converse naturally on self-selected topics.
- Participants may move between locations naturally, with each phase lasting at least 30 minutes.
- Six Kinect devices per party provide distributed four-microphone arrays, cameras, raw signals, video, and separate recordings from each laptop.
- Participant-worn binaural microphones recorded audio through adapters and Tascam stereo recorders to facilitate transcription.
- Reference transcriptions manually provide utterance times and words, while timings for other devices compensate for inter-device asynchrony.
- Transcriptions annotate speaker noise, unclear speech, laughter, and privacy-redacted signal segments.
3. Task
The task separates data by speaker-disjoint training, development, and evaluation sets and supports single-array or multiple-array recognition. Two ranking categories distinguish acoustic-model robustness from systems that can modify broader recognition components.
- 3.1. Training, development, and evaluation sets: The 20 parties are divided into disjoint training, development, and evaluation sets with no speaker overlap.
- 3.1. Training, development, and evaluation sets: Development and evaluation transcriptions specify each utterance’s location and a reference array situated in the same area.
- 3.2. Tracks and ranking: The single-array track restricts recognition to the utterance’s reference array, whereas the multiple-array track permits use of all arrays.
- 3.2. Tracks and ranking: Ranking A evaluates conventional acoustic models with the official language model and fixed lexicon, while Ranking B includes other system types and modifications.
- 3.2. Tracks and ranking: Ranking A focuses on acoustic robustness, whereas Ranking B addresses all aspects of the task.
- 3.3. Instructions: Systems may use session recordings and supplied utterance timing, speaker, and location information under the challenge instructions.
- 3.3. Instructions: Parameters must be tuned on training or development data, and transformations designed to fit evaluation data are forbidden.
- 3.3. Instructions: Participants may replace baseline components or the entire system, with architecture determining Ranking A or B eligibility.
4.1. Array synchronization
Because devices are not precisely synchronized across arrays, post-recording synchronization uses an approximate tone-based alignment, while clock drift and frame dropping can create later misalignment.
- Signals within one device are sample-synchronous, but synchronization across devices cannot be guaranteed.
- A synchronization tone approximately aligns signal start times after recording, while clock drift and frame dropping can cause devices to lose synchrony.
4.2. Speech enhancement
CHiME-5 uses BeamformIt’s weighted delay-and-sum beamformer as the default multichannel speech-enhancement approach, operating on four signals from a reference array.
- BeamformIt performs weighted delay-and-sum beamforming for multichannel speech enhancement.The approach follows the CHiME-4 recipe.
- The beamformer uses four microphone signals attached to the reference array.Organizers provide the reference-array information in the JSON transcription file.
4.3. Conventional ASR
The conventional baseline uses Kaldi for HMM/GMM and LF-MMI TDNN recognition, adding pronunciation handling, data cleaning, and neural acoustic modeling for spontaneous CHiME-5 speech.
- The conventional ASR baseline is distributed through the Kaldi repository.
- Grapheme-to-phoneme conversion supplies pronunciations for out-of-vocabulary words absent from the CMU dictionary.Phonetisaurus G2P is used for this conversion.
- The baseline prepares 250k training utterances from binaural channels and a subset of Kinect recordings.The data includes 150k binaural and 100k Kinect utterances.
- Training and recognition use an HMM/GMM system with triphone acoustic models and LDA, MLLT, fMLLR, and speaker-adaptive training.
- Removing 15% of training utterances during cleaning yields consistent improvement in subsequent LF-MMI TDNN training.
- The LF-MMI TDNN baseline requires multiple GPUs, many CPUs, and substantial storage for training and augmentation.TDNN training takes 18 hours with 2–4 GPUs.
- Compared with the CHiME-4 baseline, CHiME-5 adds grapheme-to-phoneme conversion, data cleaning, and lattice-free MMI training.Together, these techniques provide a reasonable baseline for the challenging task.
4.4. End-to-end ASR
The end-to-end baseline uses ESPnet to train a hybrid CTC/attention encoder-decoder with character-based language modeling, beamforming, and Kaldi-derived input features.
- CHiME-5 provides an ESPnet end-to-end ASR baseline using Chainer or PyTorch.
- The end-to-end pipeline uses BeamformIt beamforming and does not require lexicon generation or FST preparation.
- Training data preparation combines 350k utterances from binaural channels and a Kinect subset.The set contains 150k binaural and 200k Kinect utterances.
- The pipeline converts Kaldi data into JSON and creates a character table containing 45 transcription characters.
- A character-based LSTM language model is integrated with the decoder network.
- The hybrid CTC/attention encoder-decoder trains for 12 hours on one TitanX GPU with PyTorch.This is less computationally demanding than Kaldi LF-MMI TDNN training, which takes 18 hours with 2–4 GPUs.
- Recognition combines the trained LSTM language model and end-to-end ASR model using multiple CPUs.
4.5. Baseline results
Baseline WERs are high in CHiME-5’s difficult environments, with a large gap between oracle binaural and reference Kinect-array results; kitchen conditions are the poorest.
- Tables 2 and 3 report development-set WERs for oracle binaural microphones and the reference Kinect array challenge baseline.
- 33.4% absolute separates LF-MMI TDNN performance for binaural and array microphones.This gap indicates that source and microphone distance is a major difficulty alongside spontaneous and overlapping speech.
- CHiME-5 baseline WERs are high across methods because the environments are highly challenging.
- Speech enhancement is a major challenge because it must raise reference-array performance toward binaural-microphone performance.
- The current end-to-end baseline performs poorly because of insufficient training data.With binaural microphones, it outperforms the Kaldi GMM system and is described as promising for this environment.
- Performance is poorest in the kitchen condition, probably because of kitchen background noise and greater speaker movement.
5. Conclusion
The fifth CHiME challenge targets conversational ASR in real-world conditions using a multi-array dinner-party dataset. It releases the dataset and software baselines while designing instructions to support meaningful system comparisons.
- The fifth CHiME edition evaluates conversational speech recognition in an informal dinner-party scenario recorded with multiple microphone arrays.
- The full dataset and state-of-the-art software baselines have been made publicly available.
- Challenge instructions are designed to enable meaningful comparisons between systems and maximise scientific outcomes.