Source-linked AI summary
Powerset multi-class cross entropy loss for neural speaker diarization
Alexis Plaquet, Hervé Bredin
TL;DR
Speaker diarization systems using multi-label EEND and hybrid clustering still leave the formulation of overlapping speech and threshold tuning unresolved. The paper applies powerset multi-class classification to short diarization chunks and evaluates it across in-domain and out-of-domain benchmarks. It reports better diarization error rates, especially for overlapping speech, stronger robustness to domain mismatch, and no detection-threshold hyperparameter.
Problem
Existing EEND and hybrid diarization approaches retain multi-label classification, while EEND also faces speaker-count, data, and long-conversation scalability limitations.
Method
The paper replaces multi-label segmentation with powerset multi-class classification for short audio chunks, using Kmax = 3 and domain adaptation within a hybrid diarization pipeline.
Results
Across 8 in-domain datasets and 11 DIHARD domains, powerset encoding improves diarization error rate, mainly through better overlapping-speech predictions, and improves robustness to domain mismatch.
Takeaways & Limitations
Powerset training provides state-of-the-art performance on AISHELL-4, AliMeeting, AMI, Ego4D, MSDWild, and REPERE while eliminating detection-threshold tuning.
Takeaways & Limitations
EEND approaches do not scale well to long conversations because of their internal self-attention mechanism.
Abstract
from arXiv · showhide
Since its introduction in 2019, the whole end-to-end neural diarization (EEND) line of work has been addressing speaker diarization as a frame-wise multi-label classification problem with permutation-invariant training. Despite EEND showing great promise, a few recent works took a step back and studied the possible combination of (local) supervised EEND diarization with (global) unsupervised clustering. Yet, these hybrid contributions did not question the original multi-label formulation. We propose to switch from multi-label (where any two speakers can be active at the same time) to powerset multi-class classification (where dedicated classes are assigned to pairs of overlapping speakers). Through extensive experiments on 9 different benchmarks, we show that this formulation leads to significantly better performance (mostly on overlapping speech) and robustness to domain mismatch, while eliminating the detection threshold hyperparameter, critical for the multi-label formulation.
1. Introduction
Traditional diarization pipelines propagate errors across cascaded steps and require extra handling for overlapping speech. EEND addresses these issues with a single neural network, while hybrid methods combine local EEND processing with global clustering; this paper proposes powerset classification for short chunks.
- Cascaded diarization pipelines can escalate errors across voice activity detection, speaker embedding, and clustering stages.
- EEND directly maps audio recordings to diarization with a single neural network and uses permutation-invariant training for speaker assignment.
- Hybrid BoBW systems split long conversations, diarize short chunks, and stitch them using speaker embeddings and unsupervised clustering.
- Short 5s chunks improve scalability and training efficiency while making Kmax = 3 a practical local speaker limit across the benchmarks.
- The proposed powerset formulation removes the detection-threshold hyperparameter and yields significant overall performance improvement.
2. From multi-label classification...
The reference segmentation is represented as binary speaker-activity frames, but speaker-dimension permutations are equally valid. Training therefore uses permutation-invariant binary cross entropy, while sigmoid outputs require thresholding at test time.
- Reference segmentations encode each frame as binary activity indicators across Kmax speaker dimensions.
- Speaker dimensions may be arbitrarily permuted, so the loss evaluates all valid permutations of the reference segmentation.
- The optimal permutation is found efficiently with pairwise binary cross entropy losses and the Hungarian algorithm.
- The segmentation model uses SincNet, four bidirectional LSTMs, fully connected layers, and sigmoid speaker-activity outputs.
- Test-time diarization compares each sigmoid output with a tuned detection threshold θ.
3. ... to powerset multi-class classification
Powerset encoding replaces simultaneous multi-label speaker outputs with mutually exclusive classes representing non-speech, single-speaker, and overlapping-speaker frames. The change modifies the output layer, activation, and loss, while permutation handling converts through multi-label space.
- With Kmax = 3, powerset encoding uses Kpowerset = 7 mutually exclusive classes for non-speech, single-speaker, and two-speaker-overlap frames.
- Switching to powerset encoding changes the classification layer from 3 to 7 outputs, sigmoid to softmax, and binary to regular cross-entropy.
- Powerset speaker permutations are resolved by converting targets and binarized predictions to multi-label encoding before returning to powerset space.
- The approach keeps Kmax = 3 for short audio chunks, while later unsupervised clustering estimates the conversation-specific speaker count.
4. Experiments
The experiments evaluate a reproducible three-stage diarization pipeline across nine datasets, using compound multi-domain training and separate domain-adaptation experiments. The protocol compares powerset and multi-label speaker segmentation under in-domain and domain-mismatch conditions.
- Experimental protocol: The pipeline applies 5-second sliding-window segmentation, extracts embeddings for active speakers, and clusters them to reconstruct diarization.Segmentation uses a 5-second window with a 500 ms step and binarization by detection threshold θ.
- Datasets: Performance is evaluated on nine datasets, including AISHELL-4, AliMeeting, two AMI variants, DIHARD III, Ego4D, MSDWild, REPERE, and VoxConverse.AMI uses headset mix and first-channel far-field array variants; DIHARD domains are tested individually.
- Datasets: Ego4D results use its development subset because test labels were unavailable when the paper was written.These results should therefore be interpreted cautiously.
- Datasets: The compound training set concatenates training data from all listed datasets except DIHARD, which is held out to measure robustness to domain mismatch.The compound development set is balanced using either random 5-second chunks or randomly selected files from each dataset.
- Experimental protocol: Models are trained for up to one hundred hours with Adam, an initial learning rate of 10^-3, and learning-rate halving after 30 unimproved epochs.Model selection uses local diarization error rate on the balanced dev-duration set.
- Experimental protocol: Multi-label models tune both detection threshold θ and clustering threshold δ, whereas powerset models tune only δ because argmax replaces θ.Domain-adaptation experiments fine-tune segmentation models on each dataset or DIHARD domain and tune pipeline hyperparameters on the corresponding development data.
5. Results and discussions
Powerset compound training outperforms the multi-label counterpart on in-domain and unseen-domain evaluations, with gains concentrated in overlapping-speech detection. Domain adaptation further improves the balance between false alarms and missed detections without materially changing speaker confusion.
- In-domain results: 8% average relative improvement reduces in-domain diarization error rate from 25.6% to 23.5% with powerset compound training versus multi-label training.Further dataset-specific adaptation adds another 8% relative improvement and reaches state-of-the-art performance for most datasets.
- Benchmark comparison: Table 1 reports diarization error rates on eight datasets, using official test sets except for Ego4D, whose test labels were unavailable.Most values use no forgiveness collar; italicized grey-background values use a 250 ms collar for literature comparison.
- Domain mismatch: 11% relative improvement reduces DIHARD III diarization error rate from 33.8% to 29.9% with powerset training on unseen domains.The authors associate this robustness with removing the sensitive detection threshold θ.
- Error analysis: Missed overlapping-speech detection is the main source of improvement across approaches.The error analysis decomposes diarization error rate into false alarm, missed detection, and speaker confusion, marking overlap-related errors with hatching.
- Error analysis: 13.1% to 9.9% average missed-detection rate reduction is observed on in-domain data.The authors link this pattern to explicit powerset modeling of overlapping-speaker classes.
- Domain adaptation: Domain adaptation tends to improve the false-alarm/missed-detection compromise on DIHARD with little to no impact on speaker confusion.The authors suggest that this step also functions like the detection-threshold adjustment required by multi-label models.
6. Conclusion
The paper finds that powerset multi-class encoding improves diarization error rates, especially for overlapping speech, and increases robustness to domain mismatch across in-domain datasets and DIHARD domains. It also reports state-of-the-art performance on six benchmarks and releases code, models, and outputs.
- Conclusion: Powerset multi-class encoding improves diarization error rates over classic multi-label encoding, mainly through better overlapping-speech predictions.The evaluation covers eight in-domain datasets and eleven DIHARD domains used as out-of-domain data.
- Conclusion: The approach achieves state-of-the-art performance on AISHELL-4, AliMeeting, AMI, Ego4D, MSDWild, and REPERE.Powerset segmentation code is available in pyannote.audio, alongside compound-trained models and precomputed dataset outputs.