Source-linked AI summary
The Third DIHARD Diarization Challenge
Neville Ryant, Prachi Singh, Venkat Krishnamohan, Rajat Varma, Kenneth Church, Christopher Cieri, Jun Du, Sriram Ganapathy, Mark Liberman
TL;DR
DIHARD III addresses the need for robust, comparable speaker-diarization evaluation across diverse domains and recording conditions. It evaluates reference-SAD and from-scratch systems using common metrics and balanced or full partitions. Performance improved markedly over earlier DIHARD challenges, but many domains remain far from solved, especially without reference speech activity.
Problem
Diarization lacked a common benchmark across diverse domains, making comparisons difficult and potentially encouraging systems that did not generalize.
Method
DIHARD III evaluates diarization under reference-SAD and from-scratch conditions across 11 diverse domains using DER and JER on core and full evaluation sets.
Results
Median DER fell from over 30% in DIHARD I to under 20% in DIHARD III for track 1 and from 40% to under 25% for track 2.
Takeaways & Limitations
Diarization has improved substantially since DIHARD I, particularly for two-party interactions, but performance remains poor for many domains, especially without reference SAD.
Takeaways & Limitations
Manual annotation required highly skilled annotators and was extremely slow, expensive, and laborious, so DIHARD III used forced alignment for some data.
Abstract
from arXiv · showhide
DIHARD III was the third in a series of speaker diarization challenges intended to improve the robustness of diarization systems to variability in recording equipment, noise conditions, and conversational domain. Speaker diarization was evaluated under two speech activity conditions (diarization from a reference speech activity vs. diarization from scratch) and 11 diverse domains. The domains span a range of recording conditions and interaction types, including read audio-books, meeting speech, clinical interviews, web videos, and, for the first time, conversational telephone speech. A total of 30 organizations (forming 21teams) from industry and academia submitted 499 valid system outputs. The evaluation results indicate that speaker diarization has improved markedly since DIHARD I, particularly for two-party interactions, but that for many domains (e.g., web video) the problem remains far from solved.
1. Introduction
DIHARD III addresses the lack of a common, diverse benchmark by evaluating robust speaker diarization across varied recording conditions, domains, and speech-activity settings. It builds on earlier DIHARD challenges to measure progress while exposing domains where diarization remains difficult.
- Earlier diarization research lacked a common benchmark, making comparisons difficult and potentially encouraging over-fitting to individual domains or datasets.
- DIHARD created renewed interest in a common task for systematic benchmarking after a period without major diarization evaluations.
- DIHARD III follows DIHARD I and II, which evaluated diverse single-channel recordings and attracted substantial participation from international teams.
- DIHARD III targets robust diarization across variation in domain, equipment, environment, reverberation, noise, speaker count, and demographics.
- The challenge evaluates reference-SAD and from-scratch diarization across 11 diverse domains, while excluding multi-channel audio.
2. Task
The task is to detect and label every speaker segment in each recording, including the number of speakers and each speaker’s activity regions. Two tracks separate diarization with supplied speech activity from diarization that must also produce speech segmentation.
- The challenge requires systems to determine how many speakers are present and identify all corresponding speech segments for each speaker.
- Track 1 provides reference speech segmentation generated by merging speaker turns in the reference diarization.
- Track 2 supplies only raw audio, requiring systems to produce their own speech segmentation before diarization.
3. Performance Metrics
DIHARD III ranks systems primarily with diarization error rate and also reports Jaccard error rate. DER aggregates missed speech, false alarms, and speaker misclassification, while JER measures disagreement between mapped reference and system segmentations.
- DER is the primary metric and sums missed speech, false alarm speech, and speaker misclassification error rates.
- For reference-SAD track 1, DER measures only speaker misclassification because speech segmentation is supplied.
- JER is a secondary metric defined as 1 minus the average Jaccard similarity between optimally mapped reference and system speaker segmentations.
- Metrics are computed with dscore version 1.0.1, without forgiveness collars and with overlapped speech scored.
4. Datasets
DIHARD III uses short samples from 11 domains spanning recording conditions, noise, speaker counts, demographics, languages, and bandwidths. It scores balanced core and larger full evaluation partitions, while combining manual annotations where available with forced-alignment boundaries elsewhere.
- 4. Datasets: Development and evaluation samples last 5–10 minutes and cover 11 domains with wide variation in equipment, environment, noise, speaker count, and demographics.
- 4. Datasets: The domains range from clean single-speaker read audiobooks to noisy restaurant conversations involving up to six diners, with adult, child, English, and Chinese speech.
- 4.2. Scoring partitions: The core evaluation set approximately balances domain durations, whereas the full set includes all available selections and additional clinical-interview and CTS material.
- 4.3. Annotation: Reference diarization segments turns using pauses over 200 ms and seeks boundaries within 10 ms of true boundaries without truncating word edges.
- 4.3. Annotation: Because fully manual segmentation was slow, expensive, and laborious, DIHARD III used existing manual annotations where available and forced alignment for other data.
5. Baseline system
The baseline combines x-vector extraction, PLDA scoring, agglomerative clustering, and VB-HMM resegmentation for two diarization tracks. VB-HMM resegmentation improves both DER and JER, especially for track 2.
- Baseline SAD: Track 2 uses a TDNN speech activity detector trained to classify speech and non-speech from DIHARD III development data.The model uses 40-D MFCCs and five TDNN layers followed by two statistics-pooling layers.
- Baseline diarization recipe: The diarization baseline extracts x-vectors, scores them with PLDA, clusters them using AHC, and refines the output with VB-HMM resegmentation.The baseline models and recipes are provided for both tracks.
- Evaluation partitions: The evaluation reports baseline SAD and diarization results on core and full DEV/EVAL partitions, including conditions with and without VB-HMM resegmentation.The core and full partitions are explicitly distinguished in the baseline tables.
- Baseline results: VB-HMM resegmentation reliably improves DER and JER for both tracks, with larger gains for track 2.The authors suggest domain-adapted UBM-GMM and variability-matrix training as a possible enhancement but do not evaluate it.
6. Results and Discussion
DIHARD III systems substantially outperform the baseline overall, but performance remains highly domain-dependent and is markedly worse when speech activity must be detected from scratch. Improvements over earlier DIHARD challenges are broad, while several difficult domains remain unsolved.
- Overall results: Nearly 4% absolute DER improvement for track 1 and 5% for track 2 was achieved by the mean submission relative to the baseline.The winning submission improved over the baseline by 7% DER absolute on track 1 and 8% on track 2 on the core EVAL set.
- Track comparison: Track 2 was substantially more difficult than track 1, by approximately 5% for both DER and JER.The authors associate this pattern with continuing difficulty in speech activity detection for some domains.
- Domain variability: Track 1 median DER is below 10% in 6 domains, while meeting speech, web videos, and restaurant recordings have median DER between 35% and 45%.Clinical interviews and conversational telephone speech generally range from 10% to 20% for track 1 and 15% to 25% for track 2.
- Progress across challenges: Median DER fell from over 30% in DIHARD I to under 20% in DIHARD III for track 1, and from 40% to under 25% for track 2.For the best single system, DER decreased by 43% for track 1 and 46% for track 2.
- Sources of progress: Reported sources of progress include improved overlap handling, VB-HMM, neural speech enhancement, TS-VAD, system combination, and strong end-to-end systems.These explanations come from the authors’ survey of submitted system descriptions.
7. Summary
DIHARD III shows substantial progress in robust speaker diarization, but even the best systems perform poorly in half of the evaluated domains, especially without reference speech activity. The results indicate that further work is needed for robustness across real-world interaction types and recording conditions.
- Summary: DIHARD III demonstrates substantial progress since DIHARD I, but even the best system performs poorly in half of the evaluated domains.Performance is particularly poor when reference speech activity is not supplied.
- Summary: Further work is needed to achieve robust diarization across the interaction types and recording conditions found in real-world data.The conclusion specifically emphasizes graceful handling of wide variability.