Source-linked AI summary
CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings
Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj, David Snyder, Aswin Shanmugam Subramanian, Jan Trmal, Bar Ben Yair, Christoph Boeddeker, Zhaoheng Ni, Yusuke Fujita, Shota Horiguchi, Naoyuki Kanda, Takuya Yoshioka, Neville Ryant
TL;DR
CHiME-6 addresses distant-microphone multispeaker conversational recognition and diarization in natural home environments. It introduces two tracks with reproducible open-source enhancement, diarization, and recognition baselines. The baselines expose substantial difficulty in unsegmented recognition, especially from diarization.
Problem
Distant-microphone conversational speech in homes involves multispeaker, noisy, overlapping, and unsegmented recordings that require both recognition and diarization.
Method
The challenge uses natural dinner-party recordings and provides open-source Kaldi baselines combining speech enhancement, speaker diarization, and speech recognition.
Results
Diarization baselines exceed 60% error rates for both DER and JER, while diarization accounts for around 15% absolute degradation between Track 1 and Track 2 baselines.
Takeaways & Limitations
Track 2 remains substantially harder than segmented recognition because unsegmented multispeaker recognition requires reliable speaker diarization.
Takeaways & Limitations
The diarization baseline operates only on the U06 array, and synchronized device files may require truncation because their durations differ.
Abstract
from arXiv · showhide
Following the success of the 1st, 2nd, 3rd, 4th and 5th CHiME challenges we organize the 6th CHiME Speech Separation and Recognition Challenge (CHiME-6). The new challenge revisits the previous CHiME-5 challenge and further considers the problem of distant multi-microphone conversational speech diarization and recognition in everyday home environments. Speech material is the same as the previous CHiME-5 recordings except for accurate array synchronization. The material was elicited using a dinner party scenario with efforts taken to capture data that is representative of natural conversational speech. This paper provides a baseline description of the CHiME-6 challenge for both segmented multispeaker speech recognition (Track 1) and unsegmented multispeaker speech recognition (Track 2). Of note, Track 2 is the first challenge activity in the community to tackle an unsegmented multispeaker speech recognition scenario with a complete set of reproducible open source baselines providing speech enhancement, speaker diarization, and speech recognition modules.
1. Introduction
CHiME-6 targets distant-microphone conversational speech recognition in everyday homes using natural dinner-party recordings. It extends robust ASR evaluation to multispeaker, noisy, overlapping speech with reproducible baselines.
- Distant-microphone recognition of noisy, overlapping conversational speech is identified as the next frontier for robust ASR.
- The challenge uses speech from twenty real dinner parties recorded in real homes with multiple commercially available 4-channel microphone arrays.
- The paper describes two challenge tracks covering segmented and unsegmented multispeaker speech recognition.
- Recordings capture relaxed unscripted conversation across varied room acoustics and domestic background noises in twenty homes.
2. Scenario and tracks
CHiME-6 records natural four-person dinner parties across multiple home locations and introduces separate ASR-only and diarization-plus-ASR tracks. The challenge also standardizes synchronization, baselines, and evaluation of diarization impact.
- Scenario: Twenty dinner parties feature four participants, natural interaction, and movement among kitchen, dining, and living areas.Each party lasts at least two hours, with each phase lasting at least 30 minutes.
- Recording setup: Six Kinect devices record each party using multiple synchronized 4-channel microphone arrays, with at least two devices covering every location.
- Tracks: Track 1 performs ASR using ground-truth diarization, whereas Track 2 performs both diarization and ASR.
- Tracks: Both tracks can use microphones from all arrays, while Track 2 measures how diarization errors affect recognition errors.
- Baselines and evaluation: The challenge provides accurate array synchronization and upgraded open-source Kaldi baselines for diarization, enhancement, and recognition.
- Evaluation: Two ASR rankings distinguish conventional acoustic modeling with official language modeling from other systems, including end-to-end ASR.
3. Track 1
The Track 1 recipe provides synchronized-data preparation, enhancement, and recognition baselines integrated into Kaldi. Its enhancement options combine dereverberation with either multiarray GSS or reference-array BeamformIt.
- Baseline pipeline: The Track 1 baseline integrates array synchronization, speech enhancement, and speech recognition in the Kaldi CHiME-6 recipe.
- Baseline pipeline: The training pipeline performs data preparation, augmentation, feature extraction, GMM-HMM training, data cleaning, and chain model training.
- Decoding: Decoding applies speech enhancement and recognition to the test set, with a two-stage decoding procedure that refines i-vector extraction from first-pass results.
- Array synchronization: The synchronization baseline compensates for Kinect frame drops by inserting zeros and corrects clock drift using cross-correlation and linear fitting.
- Array synchronization: After synchronization, device WAV files may have different durations and can be safely truncated to the shortest signal, although the tool does not perform that truncation.
- Speech enhancement: The enhancement front ends combine WPE dereverberation with either GSS and beamforming across arrays or BeamformIt on a reference array.
4. Track 2
Track 2 addresses unsegmented multispeaker recognition by integrating synchronization, enhancement, speech activity detection, diarization, and ASR in an open baseline pipeline. Its recipe trains SAD and diarization models, processes raw recordings, and scores multispeaker output with cpWER.
- Overview: Track 2 provides integrated baselines for array synchronization, speech enhancement, SAD, speaker diarization, and speech recognition.The systems are integrated in the Kaldi CHiME-6 recipe.
- Training: The training pipeline adds SAD training on CHiME-6 data and diarization training on VoxCeleb data to the Track 1 stages.The Track 2 recipe retains earlier processing and model-training stages before adding these components.
- Inference: At inference, raw recordings pass through speech enhancement, SAD, speaker diarization, and ASR decoding and scoring.Track 2 receives no segment or speaker information, so decoding performs the complete pipeline.
- Scope and constraints: The baseline performs SAD, diarization, and ASR only for the U06 array, while multi-array fusion remains an integral challenge direction.Track 2 also provides only BeamformIt enhancement because estimated diarization can degrade GSS performance.
- Speaker diarization: The diarization baseline uses SAD segments, VoxCeleb-trained x-vectors, CHiME-6-trained PLDA, and agglomerative hierarchical clustering with four-speaker prior information.The baseline computes DER and JER from RTTM diarization outputs.
- Scoring: cpWER concatenates each speaker’s utterances, evaluates all 24 hypothesis-speaker permutations, and selects the lowest WER.The metric is directly affected by speaker diarization results.
5. Baseline results
The baselines show strong segmented recognition, difficult SAD evaluation, high diarization error, and a substantial Track 1–Track 2 recognition gap. The reported degradation is attributed mainly to speaker diarization when comparing BeamformIt-based systems.
- Track 1: The CHiME-6 Track 1 baseline is significantly better than the CHiME-5 baseline and close to its best multiple-device system.This result is achieved without system combination, despite the comparison system using complex multi-path enhancement and combination.
- SAD: The evaluation set is more difficult than the development set for SAD, while old and new RTTMs differ significantly in development but only marginally in evaluation.The comparison uses human-annotated and force-aligned RTTMs.
- Diarization: Both DER and JER exceed 60% for the state-of-the-art diarization baseline, making diarization a main Track 2 challenge.The result is reported for the CHiME-6 baseline speaker diarization system.
- Track comparison: The Track 1–Track 2 gap is around 15% absolute with BeamformIt, and the main degradation comes from speaker diarization.Both tracks use the same acoustic and language models; their other differences include enhancement and segmentation sources.
6. Summary
CHiME-6 introduces synchronized audio and open Kaldi recipes for segmented and unsegmented multispeaker recognition. Track 2 is harder because it requires speaker diarization, and the challenge targets a realistic unsolved problem.
- Summary: The challenge provides complete open-source Kaldi recipes combining speech enhancement, speaker diarization, and speech recognition for both tracks.The recipes are intended to help participants address Track 2’s difficulties.
- Summary: The authors describe this as the community’s first trial providing open-source recipes for unsegmented multispeaker ASR.They also identify future analysis of techniques, evaluation metrics, and relationships between diarization and recognition errors.