Source-linked AI summary
End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors
Shota Horiguchi, Yusuke Fujita, Shinji Watanabe, Yawen Xue, Kenji Nagamatsu
TL;DR
The paper addresses the limited flexibility of end-to-end diarization when the number of speakers is unknown. It introduces encoder-decoder based attractor calculation within SA-EEND to generate a flexible number of speaker outputs, achieving state-of-the-art diarization error rates across fixed- and flexible-speaker conditions. Its evaluation is bounded in some settings by sparse high-speaker CALLHOME data and a seven-speaker output limit on DIHARD.
Problem
End-to-end diarization can outperform clustering-based methods but its network architecture fixes the maximum number of speakers.
Method
EDA generates a flexible number of attractors from an embedding sequence and uses them within SA-EEND to calculate speaker activities.
Results
EDA achieved state-of-the-art DERs across conditions with fixed and flexible speaker counts, including 15.29 % DER on CALLHOME.
Takeaways & Limitations
EDA enables end-to-end speaker diarization for mixtures with a flexible number of speakers without prior knowledge of the number of clusters.
Takeaways & Limitations
CALLHOME contains only ten recordings with more than four speakers, and the DIHARD system outputs only the seven most dominant speakers when mixtures contain more than seven.
Abstract
from arXiv · showhide
End-to-end speaker diarization for an unknown number of speakers is addressed in this paper. Recently proposed end-to-end speaker diarization outperformed conventional clustering-based speaker diarization, but it has one drawback: it is less flexible in terms of the number of speakers. This paper proposes a method for encoder-decoder based attractor calculation (EDA), which first generates a flexible number of attractors from a speech embedding sequence. Then, the generated multiple attractors are multiplied by the speech embedding sequence to produce the same number of speaker activities. The speech embedding sequence is extracted using the conventional self-attentive end-to-end neural speaker diarization (SA-EEND) network. In a two-speaker condition, our method achieved a 2.69 % diarization error rate (DER) on simulated mixtures and a 8.07 % DER on the two-speaker subset of CALLHOME, while vanilla SA-EEND attained 4.56 % and 9.54 %, respectively. In unknown numbers of speakers conditions, our method attained a 15.29 % DER on CALLHOME, while the x-vector-based clustering method achieved a 19.43 % DER.
1. Introduction
Speaker diarization estimates who spoke when and supports multi-talker ASR applications. EEND improves diarization through end-to-end training but is less flexible than clustering when speaker counts vary; EDA addresses this limitation.
- Motivation: Speaker diarization estimates “who spoke when” and supports ASR in telephone conversations, meetings, conferences, lectures, television, and movies.Accurate diarization can improve ASR by constraining a speech mask for beamforming in speech separation.
- Prior approaches: Clustering-based diarization sequentially performs speech activity detection, segmentation, feature extraction, and clustering.Speaker embeddings such as x-vectors and d-vectors have enabled accurate clustering-based diarization.
- Prior approaches: Most clustering-based approaches assign each time slot to one speaker, preventing them from handling speaker overlap.
- End-to-end diarization: EEND calculates diarization results for every speaker using permutation invariant training and, especially in SA-EEND, outperforms conventional clustering-based methods.
- Proposed method: EDA generates a flexible, theoretically infinite number of attractors from speech embeddings and applies them within SA-EEND for flexible-speaker diarization.Evaluations reported better results than x-vector clustering and conventional SA-EEND with fixed and unknown speaker counts.
2. Related work
Prior work sought flexible-speaker processing through iterative extraction, attractor methods, set transformations, and end-to-end clustering. EDA differs by producing a flexible number of attractors without prior knowledge of the number of clusters.
- Flexible-speaker separation: One-vs-rest speech-separation methods iteratively extract speakers, but computation time increases linearly with the number of speakers.
- Flexible-speaker separation: Attractor-based speech-separation methods do not limit the number of speakers during inference, although the passage does not state their remaining constraint.
- End-to-end representative estimation: Set Transformer performs set-to-set transformation, but its number of outputs must be defined beforehand.
- End-to-end representative estimation: End-to-end clustering estimates distributions for possible cluster counts K ∈{1, . . . , Kmax}, limiting the maximum through network architecture.
- EDA distinction: EDA determines a flexible number of attractors from an embedding sequence without prior knowledge of the number of clusters.Unlike encoder-decoder clustering, EDA is designed for speaker overlap rather than assigning each time slot to one cluster.
3. End-to-end neural diarization: Review
The EEND framework maps acoustic features to speaker activity posteriors using neural embeddings and a fixed output dimension. Permutation invariant training compares these outputs with ground-truth speaker labels.
- Framework: EEND accepts a T-length sequence of log-scaled Mel-filterbank features and uses BLSTM or Transformer encoders to produce an embedding e_t ∈R^D at each time slot.
- Framework: A linear transformation f:R^D→R^S with element-wise sigmoid produces posteriors for S speakers at each time slot.
- Training: During training, EEND uses permutation invariant training to compare predicted posteriors with ground-truth labels.
- Training: The loss uses the permutation of speaker labels that minimizes the discrepancy between predictions and ground truth.The framework defines the permuted labels over all possible speaker permutations.
- Training: The binary cross entropy H(y_t, ŷ_t) evaluates predicted speaker activities against the corresponding labels.
4. Proposed method
EDA uses an LSTM encoder-decoder to generate a flexible number of attractors from SA-EEND embeddings, then derives diarization outputs from attractor–embedding interactions.
- Encoder-decoder based attractor calculation: EDA determines a flexible—and theoretically infinite—number of attractors from a speech embedding sequence.
- Encoder-decoder based attractor calculation: The SA-EEND backbone extracts a D-dimensional embedding at each time slot for EDA processing.
- Encoder-decoder based attractor calculation: An LSTM encoder maps the embedding sequence to final hidden and cell states that initialize the decoder.
- Encoder-decoder based attractor calculation: The decoder uses zero-vector inputs and a sigmoid probability to determine whether attractors continue to exist, theoretically permitting unlimited attractor generation.
- Speaker diarization using EDA: For unknown speaker counts, EDA estimates the count using threshold τ and retains the first estimated number of attractors.
- Speaker diarization using EDA: Diarization posteriors use inner products between embeddings and attractors, with output size determined by the number of attractors.
5. Experiments
Experiments evaluate EDA with SA-EEND on simulated and real speech under fixed and flexible speaker-count conditions. Results examine training setup, input-order sensitivity, visualization, and performance across datasets.
- Experimental setup: The experiments use simulated mixtures from multiple speech corpora, MUSAN noise, and simulated room impulse responses, with non-overlapping training and test speaker sets.Datasets include 1-, 2-, 3-, and 4-speaker mixtures with similar overlap ratios.
- Experimental setup: The system uses four Transformer encoders in SA-EEND and feeds 256-dimensional embeddings from its final layer normalization into EDA.SA-EEND provides both the baseline and the backbone for the proposed method.
- Fixed number of speakers: EDA achieved its best fixed-speaker DERs with shuffled training embeddings, while chronological training slightly degraded performance.The method was evaluated on both 2-speaker and 3-speaker conditions against conventional x-vector clustering and vanilla SA-EEND.
- Fixed number of speakers: Training EDA on shuffled embeddings reduced sensitivity to input order and subsampling, whereas chronological training showed severe degradation for subsampled sequences.Using the last 1/N of chronologically ordered sequences partly suppressed degradation, indicating sensitivity to speech-length tendencies.
- Fixed number of speakers: PCA visualizations separated speaker embeddings from silence and placed overlap embeddings between speaker clusters, while EDA calculated one attractor for each speaker.The visualization uses 2-speaker mixtures from Sim2spk with ρ = 34.4 %.
- Flexible number of speakers: 4.33 % and 8.94 % DERs were obtained on simulated two- and three-speaker mixtures with flexible speaker counts, respectively.These results were 1.64 and 0.56 points worse than corresponding speaker-specific models, and performance improved when the actual speaker count was provided.
- Flexible number of speakers: 15.29 % DER was achieved on CALLHOME with a flexible number of speakers, outperforming the clustering-based method.Performance was weaker above four speakers because CALLHOME contains only ten recordings with more than four speakers.
- Flexible number of speakers: 32.59 % DER was achieved on DIHARD, outperforming the baseline and the best pre-is2019-deadline system but not the best post-is2019-deadline system.The system output only the seven most dominant speakers, used 8 kHz audio, and left higher-resolution evaluation with additional data for future work.
6. Conclusions
The paper proposes EDA with SA-EEND for end-to-end diarization of mixtures containing a flexible number of speakers, achieving state-of-the-art DERs under fixed and flexible-speaker conditions.
- EDA calculates attractors from embedding sequences and applies them with SA-EEND for diarization with a flexible number of speakers.
- The method achieved state-of-the-art DERs under conditions with both fixed and flexible numbers of speakers.
- The evaluation included DERs and JERs on DIHARD eval.