Source-linked AI summary
Speaker Diarization with LSTM
Quan Wang, Carlton Downey, Li Wan, Philip Andrew Mansfield, Ignacio Lopez Moreno
TL;DR
Speaker diarization needs embeddings that work on arbitrary speech, while prior neural embedding applications often targeted text-dependent verification. The paper combines LSTM-based text-independent d-vectors with non-parametric spectral clustering and reports significantly lower DER than i-vector systems across three public datasets. Its evaluation is bounded by inconsistent protocols and by data and modeling choices that leave room for improvement.
Problem
Speaker diarization requires text-independent embeddings for arbitrary speech, whereas many prior neural embedding applications focused on text-dependent verification.
Method
The system combines LSTM-based text-independent d-vector audio embeddings with non-parametric spectral clustering and affinity-matrix refinement.
Results
The experiments found that d-vector systems achieve significantly lower diarization error rates than i-vector systems across three standard public datasets.
Takeaways & Limitations
LSTM-based d-vector diarization offers a state-of-the-art approach that performs strongly even when the model is trained on out-of-domain English-only data.
Takeaways & Limitations
Reported comparisons are constrained by differing evaluation protocols, including inconsistent treatment of false alarm, miss, confusion, and overlapped speech.
Abstract
from arXiv · showhide
For many years, i-vector based audio embedding techniques were the dominant approach for speaker verification and speaker diarization applications. However, mirroring the rise of deep learning in various domains, neural network based audio embeddings, also known as d-vectors, have consistently demonstrated superior speaker verification performance. In this paper, we build on the success of d-vector based speaker verification systems to develop a new d-vector based approach to speaker diarization. Specifically, we combine LSTM-based d-vector audio embeddings with recent work in non-parametric clustering to obtain a state-of-the-art speaker diarization system. Our system is evaluated on three standard public datasets, suggesting that d-vector based diarization systems offer significant advantages over traditional i-vector based systems. We achieved a 12.0% diarization error rate on NIST SRE 2000 CALLHOME, while our model is trained with out-of-domain data from voice search logs.
1. INTRODUCTION
Speaker diarization partitions audio by speaker, but text-independent embeddings are needed for arbitrary speech. This paper combines LSTM-based d-vectors with non-parametric spectral clustering to build a diarization system.
- Motivation: Speaker diarization partitions an audio stream into homogeneous speaker segments and answers who spoke when.Its boundaries can also improve acoustic speech recognition accuracy.
- Motivation: A typical diarization pipeline performs speech segmentation, audio-embedding extraction, clustering, and optional resegmentation.The clustering stage determines the number of speakers and assigns embeddings to them.
- Motivation: Neural d-vector embeddings often outperform i-vectors in speaker verification, but diarization requires text-independent embeddings for arbitrary speech.The paper targets this mismatch between prior verification applications and diarization requirements.
- Contribution: The proposed system trains an LSTM-based text-independent speaker-verification model and combines it with non-parametric spectral clustering.The approach adapts d-vector verification technology to diarization.
- Contribution: Unlike prior diarization work focused largely on feed-forward DNN embeddings, this approach uses LSTMs and generalized end-to-end training that simulates enrollment and verification.The LSTM choice is motivated by the sequential nature of audio signals.
- Contribution: The paper combines LSTM-based d-vectors with spectral clustering and introduces affinity-matrix refinements intended to denoise affinities.The authors describe this combination as novel and the refinement sequence as crucial to system success.
2. DIARIZATION WITH D-VECTORS
The diarization system converts sliding audio windows into fixed-length LSTM d-vectors, aggregates them into segment embeddings, and clusters those embeddings to identify speakers.
- D-vector extraction: The paper adapts an LSTM-based text-independent speaker-verification network, originally trained on fixed-length arbitrary-speech segments, for diarization feature extraction.The prior model produced d-vectors that outperformed i-vectors in a two-stage enrollment-verification application.
- D-vector extraction: An LSTM processes fixed-length sliding windows over log-mel-filterbank frame features, using its last-frame output as each window’s d-vector.Input frames have 25ms width, 10ms step, and 40-dimensional log-mel-filterbank energies.
- D-vector extraction: The system uses VAD-defined speech segments, divides them into nonoverlapping bounded segments, and averages their L2-normalized d-vectors.A maximal segment length such as 400ms determines diarization temporal resolution.
- Clustering: The resulting sequence of fixed-length embeddings is clustered to estimate the number of speakers and assign each audio part to a speaker.This converts arbitrary-length audio into representations suitable for clustering.
3. CLUSTERING
The paper compares online and offline clustering for speaker diarization, culminating in a modified spectral method that refines affinity matrices before clustering. It motivates this approach by showing why conventional parametric clustering can struggle with speech data.
- Online and offline clustering: Offline clustering uses all segment embeddings and typically outperforms online clustering, but online methods suit latency-sensitive applications.Offline processing also permits a final resegmentation step.
- Online and offline clustering: Online threshold clustering assigns each new segment to the most similar existing centroid or creates a new cluster when all similarities fall below threshold.The method uses cosine similarity between embeddings.
- Spectral offline clustering: The spectral algorithm constructs an affinity matrix from pairwise cosine similarities, sets diagonal entries to each row’s maximum, and applies refinement operations.The operations include Gaussian blur, row-wise thresholding, symmetrization, diffusion, and row-wise max normalization.
- Spectral offline clustering: Affinity refinement smooths and denoises similarity data by reducing outlier effects and suppressing affinities between embeddings from different speakers.The refinements exploit temporal locality because contiguous speech segments should have similar embeddings.
- Spectral offline clustering: After refinement, eigen-decomposition and the maximal eigen-gap determine the cluster count, while leading eigen-vectors provide embeddings for K-Means speaker labeling.The eigen-vectors corresponding to the largest selected eigen-values replace the original segment embeddings.
- Discussion: K-Means can perform poorly on speech because distributions are non-Gaussian, speaker activity is imbalanced, and speakers have hierarchical similarities such as gender or age.Non-parametric connection-based clustering such as spectral clustering is presented as a way to mitigate these problems.
4. EXPERIMENTS
The experiments compare i-vector and d-vector systems across four clustering algorithms and three public datasets under dataset-specific evaluation protocols. Results favor d-vector diarization, while cross-paper comparisons remain difficult because pipeline components and evaluation procedures differ.
- Experimental design: Both embedding models are trained on an anonymized voice-search collection containing around 36M utterances and 18K speakers.
- Evaluation protocol: Parameters are tuned on development sets, while DER is reported on held-out evaluation sets.Tuned parameters include the VAD threshold, LSTM window size and step, and clustering parameters.
- Results: D-vector diarization systems significantly outperform i-vector systems, with an optimal d-vector sliding window of 240ms and a 120ms step.Offline diarization also performs significantly better than online diarization, which more often predicts the wrong number of speakers.
- Results: The d-vector plus spectral-clustering system achieves state-of-the-art performance on multilingual NIST SRE 2000 CALLHOME despite English-only, out-of-domain training.The authors identify in-domain training data and final resegmentation as possible improvement directions.
- Evaluation caveats: DER comparisons across papers are difficult to align because systems use different VAD marks, training data, development sets, protocols, software, and treatment of error components.The paper notes that excluding false alarm and miss errors can make clustering appear easier when VAD misses difficult speech.
5. CONCLUSIONS
The paper develops a d-vector approach to speaker diarization and evaluates it across clustering algorithms and three public datasets. Across these evaluations, d-vector systems achieve significantly lower DER than i-vector systems.
- D-vector systems achieve significantly lower DER than i-vector systems across the reported evaluations.
- The system combines LSTM-based d-vector audio embeddings with non-parametric clustering for speaker diarization.
- The experiments compare four clustering algorithms combined with both i-vectors and d-vectors.
- The evaluation uses CALLHOME American English, NIST RT-03 English CTS, and NIST SRE 2000.