Source-linked AI summary
Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario
Ivan Medennikov, Maxim Korenevsky, Tatiana Prisyach, Yuri Khokhlov, Mariya Korenevskaya, Ivan Sorokin, Tatiana Timofeeva, Anton Mitrofanov, Andrei Andrusenko, Ivan Podluzhny, Aleksandr Laptev, Aleksei Romanenko
TL;DR
Real-life diarization is difficult because conventional clustering systems struggle with highly overlapping speech. The paper proposes TS-VAD, which predicts each speaker’s frame-level activity from speech features and speaker i-vectors, including iterative i-vector estimation and multi-microphone processing. On CHiME-6, it achieves state-of-the-art results, reducing DER to 33% and 36% on the development and evaluation sets.
Problem
Method
TS-VAD predicts each speaker’s frame-level activity from MFCC-like speech features and speaker i-vectors, with iterative i-vector estimation and multi-channel extensions.
Source: §1 · p. 1, §3.3 · p. 3, §4 · p. 4Results
Takeaways & Limitations
TS-VAD shows potential for diarization in CHiME-6-like complex, multi-speaker scenarios.
Source: §1 · p. 1, §6 · p. 4Direct comparison with DIHARD II is not possible because its reference RTTM segmentation differs substantially from CHiME-6’s.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Speaker diarization for real-life scenarios is an extremely challenging problem. Widely used clustering-based diarization approaches perform rather poorly in such conditions, mainly due to the limited ability to handle overlapping speech. We propose a novel Target-Speaker Voice Activity Detection (TS-VAD) approach, which directly predicts an activity of each speaker on each time frame. TS-VAD model takes conventional speech features (e.g., MFCC) along with i-vectors for each speaker as inputs. A set of binary classification output layers produces activities of each speaker. I-vectors can be estimated iteratively, starting with a strong clustering-based diarization. We also extend the TS-VAD approach to the multi-microphone case using a simple attention mechanism on top of hidden representations extracted from the single-channel TS-VAD model. Moreover, post-processing strategies for the predicted speaker activity probabilities are investigated. Experiments on the CHiME-6 unsegmented data show that TS-VAD achieves state-of-the-art results outperforming the baseline x-vector-based system by more than 30% Diarization Error Rate (DER) abs.
1. Introduction
Real-world diarization remains difficult because conventional pipelines handle highly overlapping speech poorly. The paper introduces TS-VAD, combining target-speaker conditioning with frame-level multi-speaker activity prediction for CHiME-6.
- Complex acoustic environments remain an unsolved diarization problem despite high accuracy on benchmarks such as CallHome.Source: §1 · p. 1
- Prior promising directions include EEND and target-speaker methods such as Target-Speaker ASR, Speaker Beam, Voice Filter, and Personal VAD.Source: §1 · p. 1
- TS-VAD combines these ideas to predict independent activity streams for conversation participants from speaker features and speech features.Source: §1 · p. 1
- 33% and 36% DER were obtained on the CHiME-6 development and evaluation sets, respectively.The paper notes that direct comparison with DIHARD II is not fully correct.Source: §1 · p. 1
2. CHiME-6 Challenge
CHiME-6 evaluates diarization in noisy, reverberant, multi-channel dinner-party conversations with overlapping speakers. Its reference segmentation and scoring conventions also make direct comparison with DIHARD II inappropriate.
- CHiME-6 uses multi-channel recordings from six four-microphone Kinect arrays across three rooms and four-person dinner-party conversations.Source: §2 · p. 2
- CHiME-6 and DIHARD II use substantially different reference RTTM segmentations, producing about a 30–40% DER difference and preventing direct comparison.Source: §2 · p. 2
- The data are noisy, reverberant, and contain substantial overlapping speech, complicating accurate clustering-based diarization.Source: §2 · p. 2
- 25.61% and 21.76% DER are the miss-only lower bounds after removing multiple speakers from overlapping segments on development and evaluation data.Source: §2 · p. 2
- Table 1 distributes audio according to the number of simultaneously speaking persons.Source: Table 1 · p. 2
3. Single-channel TS-VAD
The single-channel TS-VAD development progressed from target-speaker detection toward simultaneous prediction for all session speakers. Improvements came from multi-speaker modeling, stronger initialization, and iterative soft-weighted i-vector estimation, while several alternatives and caveats were evaluated.
3.1. Single-Speaker model
- Personal VAD inspired a target-speaker formulation that detects a target’s speech in overlapping conditions using an i-vector.Source: §3.1 · p. 2
- The initial single-speaker TS-VAD used a three-layer BLSTM with projections and classified silence, target speech, and non-target speech.Source: §3.1 · p. 2
- 66.81% DER was achieved on development data when i-vectors were computed from manually segmented non-overlapping speech.Source: §3.1 · p. 2
- 46.12% DER followed after post-processing suppressed speaker probabilities dominated by the framewise maximum.Source: §3.1 · pp. 2–3
3.2. Multi-Speaker model
- The multi-speaker model jointly predicts four speaker probabilities using shared speaker-detection parameters and a binary cross-entropy objective.Source: §3.2 · p. 3
3.3. Estimation of i-vectors
- Iterative soft-weighted i-vector re-estimation improved initialization, with the second iteration helping and the third providing no improvement.Source: §3.3 · p. 3
4. Multi-channel processing
Multi-channel TS-VAD combines information across microphone channels to improve diarization, using both channel-level processing and a jointly trained attention-based model.
- Multi-channel WPE dereverberation improves TS-VAD by about 1% DER absolute.Source: §4 · p. 4
- Averaging per-channel TS-VAD probabilities provides up to 2% absolute DER reduction.Source: §4 · p. 4
- TS-VAD-MC jointly processes TS-VAD-1C hidden SD-block outputs from a set of 10 Kinect recordings.Source: §4 · p. 4
- The multi-channel model randomly selects Kinect channels during training and uses CH1 and CH4 at test time.Source: §4 · p. 4
- System fusion combines three single-channel and three multi-channel TS-VAD models using a weighted average of probability streams.Source: §4 · p. 4
5. Post-processing
TS-VAD probabilities are converted into diarization segments through filtering, thresholding, duration-based cleanup, or alternative Viterbi decoding.
- The standard post-processing applies 51-tap median filtering and binarizes probabilities with a threshold of 0.4.Source: §5 · p. 4
- Speech segments separated by pauses shorter than 0.3s are combined, while segments shorter than 0.2s are deleted.Source: §5 · p. 4
- Viterbi decoding uses an 11-state HMM representing silence, individual-speaker speech, and pairwise overlapping speech.Source: §5 · p. 4
- Table 3 reports DER changes associated with the investigated post-processing techniques.Source: §5 · p. 4, Table 3 · p. 4
6. Conclusions
The paper presents TS-VAD as a state-of-the-art approach for multi-speaker diarization in a complex multi-channel dinner-party scenario, while identifying important scope and embedding limitations.
- TS-VAD selects each participant’s speech using that speaker’s i-vector together with MFCC features.Source: §6 · p. 4
- The approach achieved state-of-the-art diarization results in a complex multi-channel dinner-party scenario.Source: §6 · p. 4
- Replacing i-vectors with x-vectors produced much worse results in the authors’ experiments.Source: §6 · p. 4
- The final solution is task-dependent because it uses multichannel input and a fixed number of speakers.Source: §6 · p. 4