Source-linked AI summary
Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds
Keisuke Kinoshita, Marc Delcroix, Naohiro Tawara
TL;DR
Clustering-based diarization cannot handle overlapped speech, while EEND faces memory, generalization, and label-permutation challenges on long recordings. EEND-vector clustering adds global speaker embeddings for clustering across blocks; it is reported to outperform original EEND especially on long recordings while matching its performance on short ones.
Problem
Clustering-based diarization cannot handle overlapped speech, while EEND has difficulty with long recordings because of memory and generalization constraints and chunking creates label-permutation ambiguity.
Source: §1 · p. 1Method
EEND-vector clustering modifies EEND to output diarization results and global speaker embeddings in each chunk, then clusters embeddings across blocks to resolve label ambiguity.
Source: §1 · p. 1Results
EEND-vector clustering significantly outperforms original EEND especially on recordings longer than 5 minutes, while maintaining the same performance on short recordings.
Source: §1 · p. 1Takeaways & Limitations
The framework is reported to work with overlapped speech and long recordings containing an arbitrary number of speakers.
Source: §1 · p. 1The method assumes a chosen maximum number of active speakers per chunk, SLocal, that is no greater than the recording's total speakers and at least the chunk's maximum simultaneous speakers.
Source: §2.3.1 · p. 3
Abstract
from arXiv · showhide
Recent diarization technologies can be categorized into two approaches, i.e., clustering and end-to-end neural approaches, which have different pros and cons. The clustering-based approaches assign speaker labels to speech regions by clustering speaker embeddings such as x-vectors. While it can be seen as a current state-of-the-art approach that works for various challenging data with reasonable robustness and accuracy, it has a critical disadvantage that it cannot handle overlapped speech that is inevitable in natural conversational data. In contrast, the end-to-end neural diarization (EEND), which directly predicts diarization labels using a neural network, was devised to handle the overlapped speech. While the EEND, which can easily incorporate emerging deep-learning technologies, has started outperforming the x-vector clustering approach in some realistic database, it is difficult to make it work for `long' recordings (e.g., recordings longer than 10 minutes) because of, e.g., its huge memory consumption. Block-wise independent processing is also difficult because it poses an inter-block label permutation problem, i.e., an ambiguity of the speaker label assignments between blocks. In this paper, we propose a simple but effective hybrid diarization framework that works with overlapped speech and for long recordings containing an arbitrary number of speakers. It modifies the conventional EEND framework to simultaneously output global speaker embeddings so that speaker clustering can be performed across blocks to solve the permutation problem. With experiments based on simulated noisy reverberant 2-speaker meeting-like data, we show that the proposed framework works significantly better than the original EEND especially when the input data is long.
1. INTRODUCTION
Clustering-based diarization supports challenging data and arbitrary speaker counts but misses overlap, while EEND handles overlap yet struggles with long recordings and chunk-wise label consistency. EEND-vector clustering adds global speaker embeddings to EEND, and is reported to outperform original EEND especially on long recordings while matching it on short ones.
1. INTRODUCTION
- Clustering-based systems handle challenging scenarios and arbitrary speaker counts, but their embedding extraction assumes one active speaker and cannot handle overlap.Overlapped speech is reported at about 5–10% in professional meetings and can exceed 20% in informal get-togethers.Source: §1 · p. 1
- EEND directly predicts frame-level speaker activity and handles overlap, but long recordings strain memory and generalization, while independent chunks create label-permutation ambiguity.The cited block-online extension [16] typically falls short of original EEND performance and cannot handle an arbitrary number of speakers.Source: §1 · p. 1
- EEND-vector clustering significantly outperforms original EEND especially on recordings longer than 5 minutes, while maintaining the same performance on short recordings.The framework is reported to work with overlapped speech and long recordings containing an arbitrary number of speakers.Source: §1 · p. 1
- The modified EEND outputs global speaker embeddings alongside diarization results in each chunk, enabling clustering across blocks to resolve label-permutation ambiguity.This combines EEND's overlap handling with clustering across blocks.Source: §1 · p. 1
2. PROPOSED DIARIZATION FRAMEWORK: EEND-VECTOR CLUSTERING
EEND-vector clustering processes recordings in chunks, estimates diarization and speaker embeddings for each chunk, then clusters embeddings to associate speaker labels across chunks. Its training combines diarization and speaker-embedding objectives, while the diarization loss permits speaker-label permutations between chunks.
2.1. Overall framework
- The recording is divided into chunks with frame-feature sequences, and each chunk is modeled with a fixed maximum of two active speakers, generalizable to more or unknown numbers.The chunk size for embedding estimation can exceed that of homogeneous x-vector blocks because a chunk may contain multiple speakers.Source: §2.1 · p. 2
- Chunk-level speaker embeddings are clustered to associate diarization outputs across chunks, allowing the framework to handle meetings with an arbitrary total number of speakers.The clustering can use cannot-link constraints so embeddings from the same chunk are not assigned to one speaker cluster.Source: §2.1 · p. 2
2.2. Neural diarization with speaker embedding estimation
- A shared encoder and speaker-specific output layers estimate diarization labels and speaker embeddings for each local speaker in a chunk.Speaker embeddings are formed as normalized weighted sums of frame-level embeddings, with diarization outputs providing the weights.Source: §2.2 · pp. 2–3
2.3. Training objectives
- Training combines diarization and speaker-embedding losses with a weight λ, reflecting the network’s simultaneous prediction of both outputs.The speaker-embedding loss encourages small within-speaker and large between-speaker distances; training uses speaker identities, but test speakers may differ.Source: §2.3 · p. 3, §2.3.2 · p. 3
2.3.1. Diarization loss
- Permutation-invariant diarization training selects the speaker-label permutation minimizing binary cross-entropy, but allows labels to permute between chunks.The local speaker limit must cover the maximum number speaking in a chunk and cannot exceed the recording’s total speaker count; silent virtual labels fill unused slots.Source: §2.3.1 · p. 3
2.3.2. Speaker embedding loss
- The speaker-embedding objective compares estimated embeddings with a learnable dictionary of global training-speaker embeddings using rescaled squared Euclidean distances.The resulting loss is a log softmax over those distances.Source: §2.3.2 · p. 3
3. EXPERIMENTS
In simulated noisy, reverberant two-speaker mixtures, EEND-vector clustering outperformed conventional EEND, with its advantage growing on longer recordings. The experiments also examined overlap conditions, speaker embeddings, and the effects of chunking and speaker loss.
3. EXPERIMENTS
- The evaluation compared the proposed method with conventional EEND on long recordings containing significant overlap; x-vector clustering was omitted based on earlier results.The authors cite prior evidence that conventional EEND outperforms x-vector clustering when data contains overlapped speech.Source: §3 · p. 3
3.1. Data
- Test data comprised simulated noisy, reverberant two-speaker mixtures with 10–90% overlap and average durations of 3, 5, 10, and 20 minutes.The test speakers were 26 held-out speakers from Librispeech.Source: §3.1 · pp. 3–4
3.3. Results
- EEND-vector clustering outperformed conventional EEND, with an increasing advantage as duration grew and stable diarization on 10- and 20-minute data.Conventional EEND struggled on those longer recordings because of poor generalization and CPU memory constraints.Source: Table 1 · p. 4, §3.3 · p. 4
3.4.1. Evaluation in terms of overlapping ratio
- Across overlap-ratio conditions in 10-minute test mixtures, the proposed method largely outperformed conventional EEND.The comparison grouped mixtures by their amount of overlapped speech.Source: Table 2 · p. 4, §3.4.1 · p. 5
3.2. NN training and hyper-parameters
- Training used 50-second chunks and assumed at most two active speakers per chunk; DER standard deviation across ten random initializations was below 0.2%.The proposed method’s test inference used non-overlapping 50-second chunks.Source: §3.2 · p. 4
3.4.2. Speaker embedding estimation accuracy
- A t-SNE visualization showed distinct clusters for all 26 test speakers using embeddings from 10-minute data, despite significant overlapped speech.The authors interpret this as evidence that global speaker embeddings were estimated accurately under overlap.Source: Figure 2 · p. 4, §3.4.2 · p. 5
3.3. Results
- Chunking without clustering degraded performance because of inter-block label permutation, while adding speaker loss alone left diarization performance almost equal to conventional EEND.The authors report that constrained clustering effectively stitches chunk-level diarization results.Source: §3.3 · p. 4
4. CONCLUSIONS
EEND-vector clustering estimates diarization results and speaker embeddings, using the embeddings to solve inter-block label permutation. It performed significantly better than original EEND especially on long inputs, while handling arbitrary speaker counts within a chunk remains future work.
4. CONCLUSIONS
- EEND-vector clustering estimates diarization results and speaker embeddings, using the embeddings to solve inter-block label permutation.Source: §4 · p. 5
- The proposed framework worked significantly better than original EEND, especially when the input data was long.Source: §4 · p. 5
- Future work includes more challenging conditions and extending the method to handle arbitrary numbers of speakers within a chunk.Source: §4 · p. 5