Source-linked AI summary
Fully Supervised Speaker Diarization
Aonan Zhang, Quan Wang, Zhenyao Zhu, John Paisley, Chong Wang
TL;DR
Speaker diarization needs a trainable alternative to the unsupervised clustering that assigns segment labels and determines speaker counts. UIS-RNN replaces clustering with a supervised online generative model combining parameter-sharing speaker RNNs and a ddCRP, and outperforms spectral offline clustering on CALLHOME using the same embeddings.
Problem
Most diarization systems retain an unsupervised clustering module even when high-quality time-stamped speaker labels are available for training.
Method
UIS-RNN replaces clustering with an online generative process using parameter-sharing speaker RNNs, interleaved speaker states, and a ddCRP for an unbounded number of speakers.
Results
Using exactly the same speaker embeddings, UIS-RNN outperforms the state-of-the-art spectral offline clustering algorithm on NIST SRE 2000 CALLHOME.
Takeaways & Limitations
The approach provides a fully supervised speaker diarization system that operates online when high-quality time-stamped training labels are available.
Takeaways & Limitations
The system assumes speaker embeddings as observations, evaluates single-channel audio without overlapped speech, and tolerates boundary errors below 250ms.
Abstract
from arXiv · showhide
In this paper, we propose a fully supervised speaker diarization approach, named unbounded interleaved-state recurrent neural networks (UIS-RNN). Given extracted speaker-discriminative embeddings (a.k.a. d-vectors) from input utterances, each individual speaker is modeled by a parameter-sharing RNN, while the RNN states for different speakers interleave in the time domain. This RNN is naturally integrated with a distance-dependent Chinese restaurant process (ddCRP) to accommodate an unknown number of speakers. Our system is fully supervised and is able to learn from examples where time-stamped speaker labels are annotated. We achieved a 7.6% diarization error rate on NIST SRE 2000 CALLHOME, which is better than the state-of-the-art method using spectral clustering. Moreover, our method decodes in an online fashion while most state-of-the-art systems rely on offline clustering.
1. INTRODUCTION
The paper targets the unsupervised clustering bottleneck in speaker diarization by replacing it with a trainable, fully supervised UIS-RNN approach. The method is designed to learn from time-stamped speaker labels while supporting online decoding and an unknown number of speakers.
- Speaker diarization systems typically combine segmentation, embedding extraction, and clustering to determine “who spoke when.”The clustering module assigns speaker identities to segments and determines the number of speakers.
- Neural-network d-vectors improve diarization robustness across speaker accents and acoustic conditions by leveraging large training datasets.
- Most modern diarization systems still use unsupervised clustering, including Gaussian mixture models, mean shift, hierarchical clustering, k-means, Links, and spectral clustering.
- Because clustering determines speaker counts and segment labels, its unsupervised nature prevents learning from available time-stamped ground truth, despite the availability of high-quality annotated data in many applications.
- UIS-RNN replaces unsupervised clustering with an online generative process whose parameter-sharing RNN instances can grow without a fixed speaker limit and interleave over time.Within the supervised framework, it learns speaker counts through a Bayesian non-parametric process and carries information through time with the RNN.
- The paper contributes a trainable temporal-segmentation and clustering model, a fully supervised diarization framework, state-of-the-art CALLHOME performance, and online decoding with offline quality.
2. BASELINE SYSTEM USING CLUSTERING
The baseline system extracts segment-level d-vectors from speech and then applies a clustering algorithm to produce diarization results. Its architecture separates segmentation, embedding extraction, and clustering components.
- The proposed system retains the baseline segmentation and embedding modules while replacing its clustering module with an unbounded interleaved-state RNN.
- The baseline uses a text-independent speaker-recognition network to extract embeddings from 240ms sliding windows with 50% overlap.
- A two-Gaussian voice activity detector removes non-speech and partitions speech into nonoverlapping segments of at most 400ms.
- Window-level embeddings are averaged into segment-level d-vectors, which are passed to clustering for final diarization results.
- The embedding network contains three LSTM layers and one linear layer and is trained with generalized end-to-end loss.
3.1. Overview of approach
UIS-RNN models an utterance as an online generative process over embedding observations and speaker labels. Its factorization separates sequence generation, speaker assignment, and speaker-change modeling.
- The input is an embedding sequence X=(x_1,x_2,...,x_T), where each x_t is a d-dimensional d-vector for an original audio segment.
- The label sequence Y records each segment’s speaker identity, with labels represented as positive integers ordered by speaker appearance.
- UIS-RNN generates the entire utterance pair (X,Y) online.
- A binary sequence Z marks speaker changes, with z_t=1 when the current speaker differs from the previous segment and z_t=0 otherwise.Z is uniquely determined by Y, but Z alone does not identify which speaker follows a change.
- The joint process factorizes each step into sequence generation, speaker assignment, and speaker-change components.
- The first label is fixed as y_1=1, so speaker assignment and speaker-change probabilities are not modeled for the first sequence entry.
3.2. Details on model components
The model combines a speaker-change process, a ddCRP-based speaker-assignment process, and parameter-sharing GRUs for sequence generation. Together, these components represent existing or new speakers while conditioning observations on speaker-specific recurrent states.
- 3.2.1. Speaker change: Speaker changes are represented by binary variables z_t, with a parameterized function g_λ governing their probabilities.
- 3.2.1. Speaker change: For simplicity, the paper sets g_λ to a constant p_0, making speaker-change indicators independent binary variables.
- 3.2.2. Speaker assignment process: The ddCRP assigns a changed segment either to an existing speaker proportionally to prior speech blocks or to a new speaker proportionally to α.
- 3.2.2. Speaker assignment process: A speaker’s prior count N_{k,t−1} measures the number of continuous speech blocks previously associated with that speaker.This makes returning to a previously appeared speaker more likely when that speaker has more prior speech blocks.
- 3.2.3. Sequence generation: The model uses multiple GRU instances, one per speaker, that share parameters θ and memorize long-term dependencies.
- 3.2.3. Sequence generation: At each step, the probability of a new observation depends on both the previous label sequence and previous observation sequence, while y_7 has four existing-or-new speaker options in the illustrated state.
3.3. MLE Estimation
MLE estimation maximizes the labeled-data log joint likelihood over the model’s hyperparameters, using closed-form or stochastic-gradient updates where available.
- The training objective maximizes the log joint likelihood of N labeled utterances over θ, α, σ2, and λ.The likelihood terms can be factorized exactly according to the model formulation.
- λ has a closed-form solution when gλ(z[t−1]) is fixed to p0.The estimation depends on the definition of gλ(·).
- θ and σ2 are updated with stochastic gradient ascent using randomly selected minibatches of b utterances.The same update form applies to σ2 after replacing θ with σ2.
- Training repeats updates with a constant step size ρ(τ) = ρ until convergence.
3.4. MAP Decoding
MAP decoding seeks the most probable speaker-label sequence, but exhaustive search is impractical, so UIS-RNN uses sequential online greedy decoding.
- The ideal decoding goal is to find the speaker-label sequence with maximum posterior probability for an unlabeled test utterance.The test input is an embedding sequence Xtest = (x1, x2, . . . , xT).
- Exhaustive search has complexity O(T!), whereas online greedy decoding reduces complexity to O(T^2).When the number of speakers per utterance is bounded by a constant C, complexity further reduces to O(T).
- The online decoder sequentially assigns labels and can use beam search with adjustable look-ahead entries.
4. EXPERIMENTS
Experiments evaluate speaker-recognition variants and UIS-RNN diarization on CALLHOME under in-domain and off-domain training settings. Results show gains from variable-length-window speaker recognition, UIS-RNN over spectral clustering, and additional training data.
- 4.1. Speaker recognition model: d-vector V2 adds non-US English, far-field, and public-dataset speech to the speaker-recognition training data.The non-public portion contains 34M utterances from 138K speakers.
- 4.1. Speaker recognition model: d-vector V3 trains with uniformly sampled window lengths from 240ms to 1600ms, matching the diarization inference range better than fixed 1600ms windows.
- 4.1. Speaker recognition model: Variable-length-window training slightly further improves speaker-verification EER after larger training data already improves performance.
- 4.2. Evaluation setup: CALLHOME contains 500 utterances across six languages, with 2 to 7 speakers per utterance.Evaluation excludes overlapped speech, uses single-channel audio, tolerates boundaries under 250ms, and reports confusion error as DER.
- 4.5. Results: UIS-RNN is compared with k-means and spectral offline clustering across in-domain, off-domain, and combined training settings.The in-domain setting uses five-fold cross-validation.
- 4.5. Results: The largest DER improvement comes from upgrading d-vector V2 to V3 because V3 aligns training and inference window sizes.
- 4.5. Results: UIS-RNN performs noticeably better than spectral offline clustering with the same speaker-recognition model while producing labels online.
- 4.5. Results: Additional training data improves DER, and adding in-domain data to large-scale off-domain training further improves performance.
5. CONCLUSIONS
The paper concludes that UIS-RNN replaces conventional clustering with a trainable, online approach for speaker diarization. It outperforms spectral offline clustering on CALLHOME and may generalize to other sequential clustering tasks.
- UIS-RNN replaces the commonly used clustering module with a trainable unbounded interleaved-state recurrent neural network.
- On NIST SRE 2000 CALLHOME, UIS-RNN outperforms the state-of-the-art spectral offline clustering algorithm using the same speaker embeddings.
- UIS-RNN is presented as a generic solution to sequential clustering, with potential applications including face clustering in videos.
- Using acoustic features instead of pre-trained embeddings is proposed as future work toward an end-to-end speaker diarization model.