Source-linked AI summary
End-to-End Neural Speaker Diarization with Self-attention
Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Yawen Xue, Kenji Nagamatsu, Shinji Watanabe
TL;DR
Speaker diarization needs to partition recordings by speaker while handling overlaps and minimizing diarization errors directly. This paper replaces BLSTM blocks in EEND with self-attention, which conditions on all input frames and directly predicts joint speaker activities. Across simulated and real recordings, the self-attention system reduced DERs, generalized better than BLSTM-based diarization, and outperformed x-vector clustering systems.
Problem
Clustering-based diarization is not directly optimized for diarization errors and implicitly assumes one speaker per segment, limiting overlap handling.
Method
SA-EEND uses a self-attention-based neural network to directly output joint speech activities for multiple speakers with end-to-end, permutation-free training.
Results
Self-attention significantly reduced DERs, generalized better than BLSTM-based diarization, and outperformed x-vector clustering systems on simulated mixtures and two real datasets.
Takeaways & Limitations
Self-attention captured global speaker characteristics in addition to local speech activity dynamics, supporting its role in speaker diarization.
Takeaways & Limitations
On real test sets, performance was limited by the small number of mixtures and low overlap ratio in the real training set.
Abstract
from arXiv · showhide
Speaker diarization has been mainly developed based on the clustering of speaker embeddings. However, the clustering-based approach has two major problems; i.e., (i) it is not optimized to minimize diarization errors directly, and (ii) it cannot handle speaker overlaps correctly. To solve these problems, the End-to-End Neural Diarization (EEND), in which a bidirectional long short-term memory (BLSTM) network directly outputs speaker diarization results given a multi-talker recording, was recently proposed. In this study, we enhance EEND by introducing self-attention blocks instead of BLSTM blocks. In contrast to BLSTM, which is conditioned only on its previous and next hidden states, self-attention is directly conditioned on all the other frames, making it much suitable for dealing with the speaker diarization problem. We evaluated our proposed method on simulated mixtures, real telephone calls, and real dialogue recordings. The experimental results revealed that the self-attention was the key to achieving good performance and that our proposed method performed significantly better than the conventional BLSTM-based method. Our method was even better than that of the state-of-the-art x-vector clustering-based method. Finally, by visualizing the latent representation, we show that the self-attention can capture global speaker characteristics in addition to local speech activity dynamics. Our source code is available online at https://github.com/hitachi-speech/EEND.
1. INTRODUCTION
Clustering-based diarization struggles with direct error optimization and overlapping speakers. SA-EEND addresses these limitations by directly predicting all speakers’ activities with a self-attention network and outperforms BLSTM-based EEND and x-vector clustering methods.
- Clustering-based diarization uses short-segment speaker embeddings partitioned into speaker clusters with unsupervised algorithms.
- These methods cannot directly minimize diarization errors because clustering is unsupervised and implicitly assumes one speaker per segment.They also struggle to adapt embeddings trained on non-overlapping speech to recordings containing overlaps.
- SA-EEND directly outputs joint speech activities for all speakers at each frame without clustering.It uses a self-attention-based neural network given a multi-speaker recording.
- Self-attention improves performance over BLSTM-based EEND and state-of-the-art x-vector diarization across multiple datasets.The paper identifies self-attention as key to good speaker-diarization performance.
- Self-attention can capture global speaker characteristics alongside local speech activity dynamics.The authors support this conclusion by visualizing learned representations and attention heads.
2. RELATED WORK
Prior diarization systems commonly combine several independently prepared models with clustering. EEND replaces this pipeline with one end-to-end neural network, extending it here with self-attention to model multiple speakers and overlaps directly.
- 2.1. Clustering-based methods: x-vector clustering systems require separate speech activity detection, x-vector extraction, and PLDA models.None of these models is trained to directly minimize diarization errors.
- 2.1. Clustering-based methods: Joint modeling reduces dependencies among some components, but the clustering process remains unsupervised.
- 2.1. Clustering-based methods: EEND uses one neural network that directly computes final diarization outputs and can be optimized against reference diarization results.
- 2.2. Direct optimization minimizing diarization errors: Earlier fully supervised approaches jointly trained modules but still required separate SAD and speaker-embedding training and assumed one speaker per segment.
- 2.2. Direct optimization minimizing diarization errors: EEND accepts audio features and outputs joint activities for multiple speakers, including non-speech and overlapping speech in training.Its objective is oriented toward diarization errors.
- 2.3. Self-attention mechanism: Self-attention had already shown strong performance in text, vision, sound-event detection, ASR, and speaker recognition before its use in this diarization model.
3. PROPOSED METHOD: SELF-ATTENTIVE END-TO-END NEURAL DIARIZATION
SA-EEND formulates diarization as multi-label sequence estimation and replaces BLSTM encoding with self-attention blocks. It directly predicts all speakers’ frame-level activities, supports overlaps, and resolves label ambiguity through permutation-free training.
- End-to-end neural diarization: Speaker diarization estimates a multi-speaker label sequence from frame-level acoustic observations, with multiple active labels representing overlap.Each frame contains a binary activity label for every speaker.
- Self-attention-based neural network: SA-EEND replaces BLSTM encoders with self-attention-based encoding blocks for estimating frame-wise speaker posteriors.Input features are projected into D-dimensional vectors and passed through P encoder blocks before the output layer.
- Self-attention-based neural network: Each encoder block contains multi-head self-attention followed by a position-wise feed-forward layer, with residual connections and layer normalization.The self-attention layer computes pairwise frame similarities, forms attention-weighted context vectors, and projects concatenated head outputs.
- Self-attention-based neural network: The output layer applies layer normalization, a fully connected projection, and element-wise sigmoid activation to produce frame-wise posteriors.The resulting posterior vector contains one score per speaker at each time frame.
- Permutation-free training: Permutation-free training minimizes binary cross entropy over all permutations of reference speaker labels, addressing label ambiguity during neural-network training.Changing the order of speakers in a correct label sequence is treated as equivalent.
4. EXPERIMENTAL SETUP
The experiments evaluate SA-EEND across simulated and real datasets designed to vary speaker-overlap conditions. Training data differ from x-vector systems by using multi-speaker mixtures, including mixtures with overlaps.
- Datasets: The evaluation uses two training sets and five test sets spanning simulated and real datasets with varying overlap situations.Overlap ratio measures audio time with two or more active speakers divided by audio time with at least one active speaker.
- Training data: The x-vector baseline trains on single-speaker segments, whereas EEND trains on multi-speaker audio mixtures.EEND mixtures can be simulated from single-speaker segments and can also use real overlapping mixtures.
- Mixture simulation: Mixtures contain dozens of utterances per speaker with controlled silence intervals, and larger β values generate speech with less overlap.The simulation follows a diarization-style mixture rather than a source-separation mixture.
- Simulated corpus: The simulated corpus combines Switchboard-2, Switchboard Cellular, and NIST Speaker Recognition Evaluation telephone recordings containing 6,381 speakers.The speakers are split into 5,743 training speakers and 638 test speakers.
15 Determine a mixing scale p from r, y, and n
The evaluation compares EEND variants with clustering-based systems across simulated and real speech datasets, using DER that includes speech-activity and overlap errors. Self-attention improves generalization and, with domain adaptation, achieves better CALLHOME results than x-vector clustering.
- Results: Clustering-based systems performed poorly on heavily overlapped simulated mixtures because they did not model speaker overlaps.Higher overlap ratios produced more misses for these systems.
- Results: SA-EEND trained on simulated data improved substantially on CALLHOME and CSJ, indicating strong generalization of the self-attention blocks.On CSJ, SA-EEND outperformed x-vector clustering even without domain adaptation.
- Implementation: The self-attentive system was trained on non-overlapping 50-second segments but used entire recordings during inference.This segmentation was required because the system used more memory than the BLSTM-based network.
- Evaluation: Table 2 reports DER (%) across simulated mixtures, CALLHOME, and CSJ, with CALLHOME EEND results obtained using domain adaptation.The compared systems include i-vector and x-vector clustering baselines alongside EEND systems.
- Results: The EEND models overfit the training overlap ratio, while domain adaptation significantly reduced CALLHOME DER.After adaptation, the proposed system achieved better results than the x-vector-based system.
- Evaluation: The evaluation uses DER, including misses, false alarms, and errors on overlapping speech segments.This differs from prior reports that used oracle speech/non-speech labels or excluded overlap.
5. RESULTS
SA-EEND consistently improved diarization over BLSTM-EEND across test conditions, including real datasets, while self-attention captured both global speaker characteristics and local speech activity dynamics.
- Evaluation across test sets: Clustering-based systems performed poorly on heavily overlapped simulated mixtures because they did not consider speaker overlaps.Higher overlap produced more misses.
- Evaluation across test sets: SA-EEND trained on simulated data had significantly fewer DERs than BLSTM-EEND on every test set and was more robust to variable overlap ratios.Its DER degradation on less-overlapping conditions was smaller than BLSTM-EEND’s.
- Evaluation on real test sets: SA-EEND showed remarkable improvements on CALLHOME and CSJ, outperforming x-vector clustering on CSJ even without domain adaptation.These results indicate strong generalization of the self-attention blocks.
- Evaluation on real test sets: SA-EEND trained with real data performed best on real test sets but had poor DERs on simulated mixtures.The authors attributed this to the real training set’s small size and low overlap ratio.
- Effect of domain adaptation: Domain adaptation significantly reduced CALLHOME DER and enabled SA-EEND to outperform the x-vector-based system.The EEND models trained on simulated data were overfitted to the training overlap ratio.
- Visualization of self-attention: Self-attention heads captured global speaker characteristics from distant frames while other heads supported speech/non-speech detection through position-independent transforms.The authors connect this combination of global and local representations to reduced DER.
6. CONCLUSION
The paper replaces BLSTM blocks in end-to-end neural diarization with self-attention and evaluates the resulting system on simulated and real datasets. Self-attention significantly reduces DER, generalizes better than BLSTM-based diarization, and outperforms x-vector clustering systems.
- Self-attention was incorporated into an end-to-end neural diarization model and evaluated on simulated mixtures and two real datasets.
- Self-attention significantly reduced DERs and showed higher generalization quality than a BLSTM-based neural diarization system.
- The self-attention-based systems outperformed x-vector clustering-based systems and captured global speaker characteristics in latent representations.