Source-linked AI summary
End-to-End Neural Speaker Diarization with Permutation-Free Objectives
Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Kenji Nagamatsu, Shinji Watanabe
TL;DR
Speaker diarization needs to assign speech activity to speakers despite overlapping speech, while conventional clustering-based systems have limitations in this setting and cannot directly optimize diarization errors. The paper proposes EEND, a single neural network using multi-label classification and a permutation-free objective, and reports improved performance over conventional clustering methods with further DER reduction after CALLHOME domain adaptation.
Problem
Clustering-based diarization assumes one speaker per segment and cannot directly optimize diarization errors, while source-separation training requires clean, non-overlapping reference speech.
Method
EEND directly estimates joint frame-level speech activities for multiple speakers using an end-to-end neural network and a permutation-free objective for multi-label diarization.
Results
The proposed method outperformed conventional clustering-based methods on simulated speech mixtures, and domain adaptation with real speech data achieved a significant DER reduction on CALLHOME.
Takeaways & Limitations
EEND combines diarization functions in one model, explicitly handles overlapping speech, and can use real overlapping conversations for domain adaptation.
Takeaways & Limitations
On simulated mixtures, EEND degraded as overlap decreased, suggesting overfitting to the 27.3% training overlap ratio.
Abstract
from arXiv · showhide
In this paper, we propose a novel end-to-end neural-network-based speaker diarization method. Unlike most existing methods, our proposed method does not have separate modules for extraction and clustering of speaker representations. Instead, our model has a single neural network that directly outputs speaker diarization results. To realize such a model, we formulate the speaker diarization problem as a multi-label classification problem, and introduces a permutation-free objective function to directly minimize diarization errors without being suffered from the speaker-label permutation problem. Besides its end-to-end simplicity, the proposed method also benefits from being able to explicitly handle overlapping speech during training and inference. Because of the benefit, our model can be easily trained/adapted with real-recorded multi-speaker conversations just by feeding the corresponding multi-speaker segment labels. We evaluated the proposed method on simulated speech mixtures. The proposed method achieved diarization error rate of 12.28%, while a conventional clustering-based system produced diarization error rate of 28.77%. Furthermore, the domain adaptation with real-recorded speech provided 25.6% relative improvement on the CALLHOME dataset. Our source code is available online at https://github.com/hitachi-speech/EEND.
1. Introduction
Conventional speaker diarization relies on extracting and clustering speaker representations, but this creates difficulties with overlapping speech and direct diarization-error optimization. EEND addresses these issues with a single end-to-end neural network using multi-label classification and a permutation-free objective.
- Speaker diarization partitions speech segments by speaker identity and supports information retrieval and multi-speaker automatic speech recognition.Typical systems extract i-vectors, d-vectors, or x-vectors before clustering short-segment representations.
- Clustering-based systems implicitly assume one speaker per segment, making overlapping speech difficult to handle.
- Unsupervised clustering prevents direct optimization for minimizing diarization errors.
- Source-separation approaches can handle overlap but use objectives that do not necessarily minimize diarization errors and require clean, non-overlapping reference speech for training.This prevents training with real multi-speaker recordings in the described setup.
- EEND estimates joint speech activities for all speakers frame-by-frame as a multi-label classification problem and resolves ambiguous speaker labels with a permutation-free training objective.The model is trained end-to-end to provide minimal diarization errors.
- EEND explicitly handles overlap, integrates speech activity detection, speaker identification, source separation, and clustering into one neural network, and supports adaptation with real overlapping conversations.It does not require separate modules or clean, non-overlapping training speech for synthetic conversational mixtures.
2. Proposed Method
The proposed EEND models speaker diarization as multi-label sequence classification and directly estimates joint speaker activity with a recurrent neural network. Permutation-free PIT and DPCL objectives address label ambiguity while combining diarization accuracy with speaker-discriminative representations.
- Neural probabilistic model: Speaker diarization is formulated as multi-label classification, with each frame label representing the joint activity of multiple speakers.Multiple active speaker labels explicitly represent overlapping speech.
- Neural probabilistic model: A stacked BLSTM maps the input sequence to frame-wise speaker posteriors through a fully connected layer and sigmoid function.The network uses P-layer BLSTMs, with 2H-dimensional hidden activations before producing C speaker outputs.
- Permutation-free training: Speaker-label permutations create label ambiguity because exchanging speaker orders preserves a correct label sequence but conflicts with standard binary cross entropy training.The ambiguity is inherent to speaker identities that are not fixed across output dimensions.
- Permutation-free training: The PIT loss considers all ground-truth speaker-label permutations and applies binary cross entropy to the permutation with the appropriate sequence alignment.The criterion is applied at the utterance level to speaker-label sequences.
- Deep Clustering loss: The DPCL loss shapes middle-layer embeddings into speaker-dependent, overlapping, and non-speech clusters, while multi-objective training mixes it with PIT using α.Embeddings from a middle BLSTM layer are normalized before DPCL encourages same-cluster proximity and different-cluster separation.
3. Experiments
Experiments used controlled, diarization-style simulated telephone-speech mixtures with varied noise, room responses, and speaker utterance counts. The evaluation data comprised held-out speakers and 500 test mixtures.
- 3. Experiments: The experiments simulated diarization-style mixtures containing dozens of utterances per speaker with controlled silence intervals.This differs from source-separation mixture simulation by modeling conversational utterance structure.
- 3.1. Data: The speech data comprised 6,381 telephone-speech speakers, split into 5,743 training speakers and 638 test speakers.The corpora included Switchboard-2, Switchboard Cellular, and NIST Speaker Recognition Evaluation datasets.
- 3.1. Data: Each two-speaker mixture used 20–40 utterances per speaker, with 500 mixtures generated for testing.Training and test sets used differing numbers of mixtures.
- 3.1. Data: Background noise came from 37 MUSAN recordings, room responses from 10,000 simulated RIRs, and SNR values of 10, 15, or 20 dB.The simulation sampled these environmental conditions when generating mixtures.
15 Determine a mixing scale p from r, y, and n
The experiments evaluate implementation choices, loss functions, training-set size, simulated overlap conditions, and CALLHOME adaptation using diarization error rate and its components.
- Evaluation: Evaluation DER includes miss, false-alarm, and overlapping-speech errors rather than relying on oracle speech/non-speech labels.This evaluation reflects the proposed model’s integrated speech-activity and overlapping-speech detection functionality.
- Loss functions: PIT loss is essential for training the neural network, while DPCL loss further improves performance.The loss-function comparison used 10,000 simulated mixtures generated with β = 2.
- Training data: Increasing the number of training mixtures improves performance and permits use of larger speech corpora to improve robustness.The model can be trained with speech mixtures paired with corresponding time annotations.
- Simulated results: On simulated mixtures, baseline DER is dominated by miss errors, whereas the proposed method achieves a significantly lower miss rate.The proposed method detects overlapping, single-speaker, and silence segments; its confusion error rate is better than i-vector but worse than x-vector.
- Overlap robustness: The proposed method degrades when evaluated on less overlapping speech, suggesting overfitting to the 27.3% training overlap ratio.The authors identify varied overlap-ratio training as future work.
- CALLHOME: Domain adaptation on CALLHOME reduced DER by 25.6% relative to the non-adapted system.The training and test overlap ratios differed: 5.8% versus 11.8%.
4. Conclusion
The paper presents an end-to-end neural speaker diarization method optimized directly for diarization errors, outperforming clustering-based methods on simulated mixtures and benefiting from CALLHOME domain adaptation.
- The proposed end-to-end method is directly optimized with a diarization-error-oriented objective and outperforms conventional clustering-based methods on simulated mixtures.