A sticky HDP-HMM with application to speaker diarization

Emily B. Fox; Erik B. Sudderth; Michael I. Jordan; Alan S. Willsky

Source-linked AI summary

A sticky HDP-HMM with application to speaker diarization

Emily B. Fox, Erik B. Sudderth, Michael I. Jordan, Alan S. Willsky

arXiv:0905.2592v4May 15, 2009stat.MEstat.APstat.ML

TL;DR

Speaker diarization requires segmenting meetings by speaker without knowing how many participants are present. This paper develops a sticky HDP-HMM with nonparametric emissions and blocked inference, achieving a lower overall diarization error rate than the original HDP-HMM on a NIST benchmark.

Problem
Speaker diarization must segment meetings without assuming the number of speakers, while the HDP-HMM can over-segment through redundant states and rapid switching.
Method
The paper adds a prior-modeled self-transition bias to the HDP-HMM, permits nonparametric emissions, and uses truncated blocked Gibbs sampling with forward–backward recursions.
Results
17.84% overall DER was obtained with the sticky HDP-HMM versus 23.91% with the original HDP-HMM on the NIST meetings.
Takeaways & Limitations
The sticky HDP-HMM supports Bayesian nonparametric modeling of multimodal emissions while improving speaker diarization performance on the benchmark.
Takeaways & Limitations
HMM observations are assumed conditionally i.i.d. given the state sequence, which may not capture complex temporal dependencies in real-world data.

Abstract

from arXiv · show

We consider the problem of speaker diarization, the problem of segmenting an audio recording of a meeting into temporal segments corresponding to individual speakers. The problem is rendered particularly difficult by the fact that we are not allowed to assume knowledge of the number of people participating in the meeting. To address this problem, we take a Bayesian nonparametric approach to speaker diarization that builds on the hierarchical Dirichlet process hidden Markov model (HDP-HMM) of Teh et al. [J. Amer. Statist. Assoc. 101 (2006) 1566--1581]. Although the basic HDP-HMM tends to over-segment the audio data---creating redundant states and rapidly switching among them---we describe an augmented HDP-HMM that provides effective control over the switching rate. We also show that this augmentation makes it possible to treat emission distributions nonparametrically. To scale the resulting architecture to realistic diarization problems, we develop a sampling algorithm that employs a truncated approximation of the Dirichlet process to jointly resample the full state sequence, greatly improving mixing rates. Working with a benchmark NIST data set, we show that our Bayesian nonparametric architecture yields state-of-the-art speaker diarization results.

1. Introduction.

Speaker diarization must segment meetings by speaker without knowing the number or speech patterns of participants. The paper addresses HDP-HMM over-segmentation by adding self-transition persistence, nonparametric emissions, and blocked inference.

Speaker diarization segments meeting recordings into intervals associated with individual speakers without assuming their number or speech patterns.
Classical HMMs require the number of states to be fixed a priori, making them difficult to adapt when the number of speakers is unknown.
The HDP-HMM supports posterior inference over countably many states but can assign high probability to redundant states and unrealistically rapid switching.
The paper augments the HDP-HMM with a separately prior-modeled self-transition bias to provide a general solution for state persistence.
The augmented model enables nonparametric emission distributions and uses truncated blocked Gibbs sampling with forward–backward recursions to jointly resample states and emissions.

2. The speaker diarization task.

Speaker diarization methods must jointly handle temporal segmentation, recurring speakers, unknown speaker identities, and non-Gaussian acoustic emissions. The paper evaluates a Bayesian nonparametric approach on a 21-meeting NIST benchmark after feature preprocessing.

Classical diarization often segments audio first and then clusters segments, but segmentation errors can degrade subsequent clustering.
Joint segmentation and clustering with HMMs captures repeated returns of speakers while avoiding some two-stage pipeline weaknesses.
The experiments use 21 NIST meetings with varying speaker sets and 19 MFCC features computed every 10 ms after nonspeech removal.
Observations are averaged over nonoverlapping 250 ms blocks to match speaker dynamics at a coarser temporal granularity.
Speaker-specific emissions are not well approximated by a single Gaussian, motivating mixture-based or otherwise non-Gaussian emission models.

3. Dirichlet processes.

Dirichlet processes provide priors over probability measures and mixture models with a random number of components. Their hierarchical extension shares atoms across group-specific measures, supporting clustered structure across groups.

A Dirichlet process is defined by a base measure H and concentration parameter γ and places a distribution over probability measures.
The stick-breaking representation expresses DP weights as proportions of a remaining unit-length stick, with β ∼ GEM(γ).
The concentration parameter γ controls relative mixture-weight magnitudes and therefore model complexity in terms of significant components.
DP mixture models use random component parameters and indicator variables to partition or cluster observations.

4. Hierarchical Dirichlet processes.

The hierarchical Dirichlet process ties group-specific Dirichlet processes through a shared, atomic global measure, allowing atoms to be shared across groups. Its Chinese restaurant franchise representation provides marginal characterizations of this sharing structure.

The HDP uses a collection of group-specific Dirichlet process mixture models for observations organized into groups.
A global Dirichlet process base measure ties the group-specific mixtures so different groups can reuse shared atoms.An absolutely continuous base measure would otherwise make atoms distinct across groups with probability one.
Because the global base measure is atomic, only its atoms can appear in each group-specific measure, enabling sharing across the collection.
Integrating out the random measures yields marginal probabilities represented by a Chinese restaurant franchise with restaurant-specific tables and globally shared dishes.
Multiple tables may serve the same dish, allowing each group-specific distribution to be expressed over unique shared dishes rather than tables.

5. The sticky HDP-HMM.

The sticky HDP-HMM extends the infinite-state HDP-HMM with a self-transition bias and supports nonparametric emissions. A blocked forward–backward sampler jointly resamples state sequences, addressing mixing problems in sequential inference.

An HMM models a Markovian discrete state sequence with conditionally independent observations generated from state-specific emission distributions.
The HDP-HMM treats each state as a speaker and removes an upper bound on the number of speakers through an infinite state space.
Because the original HDP-HMM does not distinguish self-transitions from other transitions, it can assign high posterior probability to unrealistically rapid switching and redundant states.
A nonparametric emission extension uses state-specific Dirichlet-process mixtures whose components generate observations.
The sticky HDP-HMM adds κ > 0 to the self-transition component, increasing the prior probability of remaining in the same state; κ = 0 recovers the original model.
The sticky model reduces uncertainty from fast-switching explanations, but its self-transition bias can split temporally separated observations from one state and impede sequential regrouping.
A truncated blocked Gibbs sampler uses forward–backward recursions to jointly resample the full state sequence and emission assignments.

6. Experiments with synthetic data.

Synthetic experiments compare sticky and original HDP-HMMs on segmentation, predictive likelihood, and sampler mixing. The experiments include both persistent-state and rapidly switching regimes under weakly informative priors.

Synthetic experiments evaluate segmentation accuracy, held-out predictive likelihood, and mixing-rate improvements from blocked sampling.
The experiments compare the sticky HDP-HMM with the original model, defined by κ = 0.
Fast dynamics: The study also generates data with a high probability of leaving the current state to test whether the sticky HDP-HMM can capture fast dynamics.
Simulated-data experiments use weakly informative hyperpriors, including Beta(10,1) for the self-transition proportion parameter ρ.
High persistence: A high-persistence data set is generated from a three-state HMM with 0.98 self-transition probability and equal transition probability to the other two states.

State persistence.

State-persistence experiments show that blocked sampling with the sticky HDP-HMM mixes faster than the compared sampler–model combinations. Sequential direct assignment can become trapped in redundant state assignments, while beam sampling mixes more slowly than blocked sampling.

Hamming distance between true and estimated state sequences is tracked across Gibbs iterations using quantiles and state-index alignment.The Munkres algorithm maps estimated state indices to maximize overlap with the true sequence.
The sticky HDP-HMM blocked sampler has a significantly faster burn-in rate than every other sampler–model combination shown.
The sequential direct assignment sampler can remain stuck in high-error state-sequence assignments because coordinate-by-coordinate updates impede global changes.
The nonsticky HDP-HMM beam sampler mixes substantially more slowly than the blocked sampler in the comparison.
The beam sampler adapts its effective truncation level, whereas the truncated blocked sampler uses a fixed level L = 20.
The paper presents subsequent results only from blocked sampling because of its observed advantages over sequential direct assignment sampling.

Fast state-switching.

The experiments examine whether the sticky HDP-HMM can represent fast-switching dynamics and whether self-transition bias improves inference with multinomial emissions.

Fast-switching dynamics: The sticky HDP-HMM infers a small self-transition probability close to the true value and captures fast dynamics as quickly as the original HDP-HMM.The sticky sampler has higher variability in this fast-switching setting.
Fast-switching dynamics: The fast-switching experiment uses a four-state HMM and evaluates Hamming distance, inferred self-transition probability, and sampler behavior.The blocked sampler is assessed over 1,000 samples from 200 chains and 30,000 samples from three chains.
Multinomial emissions: For multinomial observations, the nonsticky model retains substantial posterior probability on fast-switching state sequences even after burn-in.These sequences produce sweeps through regions of larger Hamming distance error.
Multinomial emissions: Those fast-switching sequences have negligible posterior probability under the sticky HDP-HMM formulation.The comparison concerns a five-state HMM with multinomial emissions and 0.98 self-transition probability.
Predictive performance: Fragmentation into redundant states can degrade predictive performance, so the sticky parameter matters for Bayesian nonparametric model averaging as well as segmentation.The predictive comparison evaluates test-sequence likelihoods using samples from Gibbs iterations 10,000–30,000.
Sparse transition structure: The sparse Dirichlet comparison studies transition modeling with and without the sticky parameter on a nine-state HMM containing one main state and eight sub-states.The sub-states have significant self-transition, preferential transitions back to the main state, and small probabilities of transitioning among sub-states.

7. Multimodal emission densities.

The paper extends the sticky HDP-HMM with nonparametric emission distributions, representing each state by a Dirichlet-process mixture of Gaussians and sampling the augmented sequence jointly.

Model construction: Each HDP-HMM state receives a countably infinite Gaussian mixture emission distribution with its own stick-breaking mixture weights.The augmented state includes z_t for the HMM state and s_t for its mixture component.
Inference: The augmented state (z_t,s_t) is sampled by first sampling the HMM state and then sampling its mixture-component index conditionally.The mixture-component conditional depends on observations currently assigned to that component.
Inference: Blocked resampling uses weak-limit truncations at levels L and L′ for the HDP-HMM and DP emissions, respectively.The posterior for the emission mixture weights ψ_k is derived under this approximation.
Inference: The blocked procedure for sampling the full augmented state sequence is derived as part of the resulting Gibbs sampler.The implementation tracks how many observations are assigned to each state’s mixture component.
Evaluation: The multimodal-emission experiment uses a five-state HMM whose Gaussian emission mixtures contain between 1 and 10 randomly selected components.Each mixture component is equally weighted, and the self-transition probability is 0.98.
Evaluation: With DP emissions, the nonsticky model retains rapidly dynamic state paths after 30,000 Gibbs samples and has slightly reduced predictive performance on test sequences.The comparison is between sticky and original HDP-HMMs with DP emissions.

8. Speaker diarization results.

On the 21-meeting NIST diarization benchmark, the sticky HDP-HMM with DP emissions substantially outperformed the original HDP-HMM and achieved performance comparable to ICSI. The results also show that posterior uncertainty, emission modeling, and sampling convergence materially affect diarization performance.

Overall performance: The sticky model’s temporal smoothing substantially improved diarization performance despite the original model often attaining higher likelihoods.The original model’s higher likelihood is attributed to over-segmentation and overfitting, making likelihood selection less robust.
Overall performance: 17.84% overall DER was achieved by the sticky HDP-HMM with DP emissions, versus 23.91% for the original HDP-HMM.Both figures use the specified Gibbs-iteration settings and minimum expected Hamming distance selection.
Emission models: 34.97% and 36.89% overall DERs were obtained by the sticky and original HDP-HMMs, respectively, when emissions were restricted to single Gaussians.The comparison indicates the reported DP-emission results were substantially better than the single-Gaussian configuration.
Sampling behavior: Meeting 16 improved after extending the chains to 50,000 Gibbs iterations because its chains had not mixed after 10,000 iterations.The meeting contained especially subtle speaker differences, making parameter separation difficult for the sampler.
Posterior uncertainty: 14.61% overall DER was obtained by selecting the best per-meeting DER among the five most likely samples.The Bayesian formulation supports multiple posterior segmentations, although NIST evaluations require a single segmentation.
Benchmark comparison: The sticky HDP-HMM with DP emissions performed comparably to ICSI, while the original HDP-HMM with DP emissions performed significantly worse.This comparison is reported for the 21-meeting benchmark under the evaluation conditions shown in Figure 14.

9. Discussion.

The sticky HDP-HMM combines nonparametric multimodal emissions with improved sampling, but the authors identify slow mixing and broader HMM temporal-dependence limits as important discussion points.

The sticky HDP-HMM enables a fully Bayesian nonparametric treatment of multimodal emissions, which is essential for speaker diarization.The authors also suggest multimodal emissions may matter in other speech-technology applications.
The proposed samplers improve mixing rates by using a truncated HDP approximation and block-sampling the full state sequence with a forward–backward variant.The authors report substantial improvement over sequential direct-assignment samplers while noting remaining mixing difficulties.
The authors suggest split-merge algorithms as one possible way to address difficulties in adding new states and improving mixing.They compare this possibility to split-merge methods developed for DP mixture models.
10,000 Gibbs iterations were used for meeting 5, while meeting 16 required 100,000 iterations to demonstrate especially slow mixing.Trace plots show log-likelihood, Hamming distance, and speaker counts for these chains.
HMM observations are assumed conditionally i.i.d. given the state sequence, which may inadequately capture complex temporal dependencies in real-world data.Future work includes Bayesian nonparametric switching linear dynamical systems and switching VAR processes.

SUPPLEMENTARY MATERIAL

The supplementary material provides derivations, notation, analogies, and pseudocode supporting the paper’s Gibbs samplers.

The supplement derives conditional distributions and pseudocode for direct-assignment and blocked Gibbs samplers using HDP restaurant analogies.It also explains the notational conventions and Chinese restaurant franchise constructions used in those derivations.