Source-linked AI summary
The Second DIHARD Diarization Challenge: Dataset, task, and baselines
Neville Ryant, Kenneth Church, Christopher Cieri, Alejandrina Cristia, Jun Du, Sriram Ganapathy, Mark Liberman
TL;DR
Diarization research lacked a common evaluation framework robust to diverse recording and conversational conditions. DIHARD II addresses this gap with four standardized tracks, diverse single- and multichannel datasets, and baseline systems; DIHARD I results showed substantial difficulty, while DIHARD II attracted 48 teams from 17 countries.
Problem
Diarization evaluation was fragmented across domains and could overestimate performance on difficult phenomena such as overlap and backchannels.
Method
DIHARD II standardizes four tracks spanning single- or multichannel audio and reference- or system-generated speech segmentation, using diverse datasets and reporting DER and JER.
Results
DIHARD I top-system DER ranged from 23.73% with reference SAD to 35.51% from scratch, while DIHARD II attracted 48 registered teams from 17 countries.
Takeaways & Limitations
The challenge provides a common basis for comparing diarization systems and focuses research attention on robustness across domains and recording conditions.
Abstract
from arXiv · showhide
This paper introduces the second DIHARD challenge, the second in a series of speaker diarization challenges intended to improve the robustness of diarization systems to variation in recording equipment, noise conditions, and conversational domain. The challenge comprises four tracks evaluating diarization performance under two input conditions (single channel vs. multi-channel) and two segmentation conditions (diarization from a reference speech segmentation vs. diarization from scratch). In order to prevent participants from overtuning to a particular combination of recording conditions and conversational domain, recordings are drawn from a variety of sources ranging from read audiobooks to meeting speech, to child language acquisition recordings, to dinner parties, to web video. We describe the task and metrics, challenge design, datasets, and baseline systems for speech enhancement, speech activity detection, and diarization.
1. Introduction
DIHARD II addresses fragmented and potentially over-optimistic diarization evaluation by introducing a standardized challenge focused on robust performance across difficult conversational conditions. It evaluates systems across varied domains, recording conditions, and segmentation settings.
- Motivation: Diarization performance became difficult to compare because research groups focused on different datasets and domains after the NIST RT evaluations ended.This fragmentation may also have encouraged overfitting to individual domains.
- Motivation: Overlaps, backchannels, and short turns exposed weaknesses that could produce over-optimistic performance assessments.Real-user experience with an IBM diarization API likewise found performance lacking for these phenomena.
- Challenge design: DIHARD introduces standardized data, tasks, and metrics to facilitate system comparison and promote diarization robust to difficult speech and recording variation.The challenge targets interactive and overlapping speech across domains, equipment, environments, noise levels, speaker counts, and demographics.
- DIHARD I context: DIHARD I top-system evaluation DER ranged from 23.73% with reference SAD to 35.51% from scratch, exceeding contemporary CALL-HOME state of the art.For some domains, best-system error rates exceeded 49% with reference SAD and 75% from scratch.
- DIHARD II design: DIHARD II adds multichannel evaluation to the two SAD conditions and spans 11 demanding single-channel domains, including audiobooks, restaurants, and child language recordings.The multichannel condition uses farfield microphone arrays and dinner-party speech from CHiME-5.
2. Tracks
DIHARD II crosses two audio input conditions with two speech-activity conditions, producing four evaluation tracks. The design separates single- versus multichannel processing and supplied versus system-generated speech segmentation.
- Audio input conditions: The challenge evaluates both single-channel and multichannel audio input conditions.Multichannel sessions contain one or more distant microphone arrays, and participants produce one output per array using any selection of channels.
- SAD conditions: Reference SAD supplies speech segmentation, whereas system SAD requires participants to generate segmentation from raw audio.The two SAD conditions reflect the strong relationship between system performance and SAD quality.
- Evaluation tracks: The four tracks combine single- or multichannel audio with reference or system SAD.Tracks 1 and 2 use single-channel audio; tracks 3 and 4 use multichannel audio.
- Participation: All teams must register for at least one of track 1 or track 3.
3. Performance Metrics
DIHARD II ranks systems primarily with DER and reports JER as a secondary metric. The metrics are designed to score speaker attribution and segmentation without forgiveness collars or excluding overlap.
- Primary metric: DER sums missed speech, false-alarm speech, and speaker-misclassification error rates.For tracks with reference SAD, DER exclusively measures speaker-misclassification error and ranks leaderboard systems.
- Secondary metric: JER is a secondary DIHARD II metric based on one minus the average optimal speaker-pair Jaccard similarity, expressed as a percentage.It compares intersections and unions of reference and system speaker segments and ranges from 0% for identical pairwise segmentation to 100% for no overlap.
- Scoring protocol: All metrics are computed with dscore version 1.0.1, without forgiveness collars and with overlapped speech scored.
4. Datasets
DIHARD II uses diverse single-channel and multichannel datasets spanning recording environments, equipment, noise, speaker populations, and conversational domains. Its multichannel data come from CHiME-5 dinner parties recorded in real homes with desynchronized microphone arrays.
- Dataset scope: The datasets vary widely in recording equipment, environment, ambient noise, speaker count, and speaker demographics.The single-channel collection adds six hours to ensure every domain appears in both development and evaluation sets.
- Dataset protocol: Participants may use the development set for development or training, while no dedicated training set is provided.They may train systems on any public or proprietary data.
- Single-channel data: Single-channel tracks contain 5–10-minute samples from 11 conversational domains, with approximately two hours per domain.Audio is distributed as 16 kHz monochannel FLAC files.
- Single-channel data: The single-channel domains include audiobooks, broadcast interviews, child language, and clinical recordings.These examples illustrate the range of sources represented in the collection.
- Multichannel data: Multichannel tracks use CHiME-5 dinner-party speech recorded in 18 homes for development and two homes for evaluation.Each session used six Microsoft Kinect devices with four-channel linear arrays distributed throughout the homes.
- Multichannel data: Kinect clock drift and frame dropping caused severe desynchronization, so each Kinect was treated as a separate recording.Binaural-to-Kinect lag was estimated with normalized crosscorrelation and used to correct annotated speech boundaries.
5. Baseline system
The baseline combines enhancement, speech activity detection, and x-vector diarization components across the challenge tracks. Its evaluation shows substantially higher error rates in the multichannel tracks, especially under more difficult conditions.
- Speech enhancement: The speech-enhancement baseline uses a densely connected LSTM to predict ideal ratio masks from log-power spectra.It is trained with progressive multi-target learning on corrupted WSJ0 and Chinese speech data.
- Multichannel processing: Multichannel processing applies weighted delay-and-sum beamforming independently to each Kinect using all four channels.The implementation follows the CHiME-5 recipe through BeamformIt.
- Speech activity detection: The SAD baselines for tracks 2 and 4 use WebRTC’s speech activity detector with distributed scripts for reproducing the settings.
- Diarization: The diarization baseline extracts x-vectors from overlapping segments, scores them with PLDA, and clusters them using agglomerative hierarchical clustering.Unlike the original JHU system, it omits variational Bayes resegmentation.
- Diarization: The baseline uses MFCC-based 512-dimensional embeddings, DIHARD II domain adaptation, Gaussian PLDA, and AHC thresholding optimized for DER.MFCCs are extracted every 10 ms with a 25 ms window and normalized using a 3-second sliding window.
- Baseline results: 50.85% and 77.34% DER are reported for tracks 3 and 4, while track 2 reaches 15% without enhancement.The authors attribute the track-2 difference from DIHARD I partly to SAD quality and more careful segmentation, and relate tracks 3–4 to difficult restaurant and child-language domains.
6. Conclusion
DIHARD II reflects rapid growth in interest in robust speaker diarization after earlier evaluations focused on easier data and forgiving metrics. Its broader participation underscores continued work toward robust systems.
- 6. Conclusion: Earlier diarization research often used relatively easy datasets and forgiving metrics, making system comparison difficult.
- 6. Conclusion: 48 teams from 17 countries registered for DIHARD II, more than double the registrations for DIHARD I.
- 6. Conclusion: The authors hope DIHARD II contributions produce marked progress toward truly robust diarization.