PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks

Federico Landini, Ján Profant, Mireia Diez, Lukáš Burget

arXiv:2012.14952v1eess.AScs.SD

TL;DR

AMI evaluations often use differing partitions, references, and audio setups, complicating comparisons. The paper presents VBx derivations and update formulae, reports state-of-the-art results on CALLHOME, AMI, and DIHARDII, and proposes an AMI evaluation protocol.

  • Problem

    AMI studies often choose their own partitions, references, and audio setups, making comparisons between works difficult.

  • Method

    The paper presents derivations and update formulae for clustering x-vectors with a Bayesian hidden Markov model, known as VBx.

  • Results

    VBx achieves state-of-the-art results on CALLHOME, AMI, and DIHARDII.

  • Takeaways & Limitations

    The proposed AMI evaluation protocol addresses a major contribution concerning standardized evaluation.

  • Takeaways & Limitations

    The authors advocate more challenging evaluation setups that also consider overlapped speech.

Abstract

from arXiv · show

The recently proposed VBx diarization method uses a Bayesian hidden Markov model to find speaker clusters in a sequence of x-vectors. In this work we perform an extensive comparison of performance of the VBx diarization with other approaches in the literature and we show that VBx achieves superior performance on three of the most popular datasets for evaluating diarization: CALLHOME, AMI and DIHARDII datasets. Further, we present for the first time the derivation and update formulae for the VBx model, focusing on the efficiency and simplicity of this model as compared to the previous and more complex BHMM model working on frame-by-frame standard Cepstral features. Together with this publication, we release the recipe for training the x-vector extractors used in our experiments on both wide and narrowband data, and the VBx recipes that attain state-of-the-art performance on all three datasets. Besides, we point out the lack of a standardized evaluation protocol for AMI dataset and we propose a new protocol for both Beamformed and Mix-Headset audios based on the official AMI partitions and transcriptions.

1. Introduction

The paper presents VBx, a Bayesian HMM approach for clustering x-vectors, and evaluates it against prior diarization systems on CALLHOME, AMI, and DIHARDII. It also derives simplified update formulae, analyzes AMI evaluation inconsistencies, and releases reproducible training and diarization recipes.

  • VBx clusters x-vectors with a Bayesian hidden Markov model and a ResNet101 x-vector extractor.
  • VBx achieves superior results on the CALLHOME, AMI, and DIHARDII datasets.
  • The paper compares VBx with strong literature systems and establishes new baselines for representative diarization datasets.
  • AMI comparisons are difficult because published studies use different partitions, references, and audio setups.
  • The proposed AMI protocol specifies train, development, and evaluation lists, references, and audio based on the official corpus partition.
  • The paper derives simplified VBx inference updates and explains how its design improves computational efficiency relative to the more complex BHMM.
  • The authors publicly release x-vector extractor recipes, trained extractors, and a VBx diarization pipeline with AHC initialization.

2. VBx Diarization model

This section introduces the paper’s VBx model as a simplified, self-contained treatment of an earlier BHMM formulation. It reuses the earlier structure and symbols to make the derivation easier to follow and comparison more direct.

  • VBx is introduced together with derivations of its inference formulae for the simple model used throughout the paper.
  • The presentation rewrites and simplifies the corresponding section of the earlier BHMM work to address only the model considered here.
  • The authors intentionally retain the earlier text, structure, and symbols to make the treatment self-contained and facilitate model comparison.

2.1. Model overview

The VBx model represents consecutive speech segments as x-vector observations generated by speaker-specific HMM states. Speaker distributions are derived from pretrained PLDA, and the most likely latent-state sequence provides diarization.

  • VBx takes a sequence of x-vectors extracted from consecutive short speech segments as its observations.
  • An HMM with speaker-specific state distributions models the x-vector sequence and clusters observations according to speaker identity.
  • A pretrained PLDA model derives the speaker-specific distributions from speaker-labeled x-vectors.
  • The model assumes each speech frame is generated by one speaker state and therefore does not model overlapped speech.
  • The latent variables Z define a hard alignment in which z_t = s means speaker state s generates observation x_t.
  • Diarization jointly estimates speaker distributions and latent variables, then selects the most likely Z sequence as the speaker alignment.

2.2. HMM topology

VBx uses an ergodic HMM whose states correspond one-to-one with speakers, with transitions shaped to reflect speaker turns. Bayesian relevance determination can remove redundant speaker states and estimate the conversation’s speaker count.

  • Each HMM state corresponds to one speaker, while transition probabilities model speaker-turn duration and discourage overly frequent changes.
  • The HMM is ergodic, so transitions between all speaker states are possible.
  • The model assigns probability Ploop to remaining with the same speaker and probability 1−Ploop to changing speaker through the non-emitting node.
  • From the non-emitting node, the model enters speaker state s with probability π_s, which also controls the initial state.
  • Automatic relevance determination learns zero π_s values for redundant speakers, allowing the model to drop them and estimate the speaker count.
  • Figure 1 illustrates the topology for three speakers with one state per speaker and a dummy non-emitting node.

2.3. Speaker-specific distributions

VBx derives speaker-specific x-vector distributions from a simplified PLDA model, using a transformed space with diagonal between-speaker covariance and identity within-speaker covariance. Speaker vectors are latent Gaussian variables, enabling Bayesian HMM states to model speaker variability efficiently.

  • Speaker-specific x-vectors use a Gaussian distribution with shared within-speaker covariance and speaker-dependent means.Speaker means are themselves modeled by a Gaussian distribution around the global mean with between-speaker covariance.
  • The PLDA model is transformed so the within-speaker covariance is identity and the between-speaker covariance is diagonal.This transformation simplifies matrix operations in subsequent VB inference.
  • Selecting the R largest eigenvalues and corresponding eigenvectors performs LDA dimensionality reduction to an R-dimensional space.R is used as a VBx hyper-parameter, and the global mean is subtracted to zero-center the transformed x-vectors.
  • The speaker vector y_s is a latent variable with a standard normal prior, while the shared matrix V defines speaker-specific distributions.This incorporates PLDA modeling of between- and across-speaker variability into the Bayesian HMM.
  • The resulting model can discriminate between speakers similarly to PLDA used for speaker verification.The paper describes this as a two-covariance model.

2.4. Bayesian HMM

The complete diarization model is a Bayesian HMM whose states represent speakers and whose observations are generated from speaker-specific x-vector distributions. It combines latent speaker vectors with HMM transition probabilities, without imposing a prior on transitions.

  • VBx models each conversation’s x-vector sequence through a generative process with speaker-specific HMM state distributions.The output probabilities are defined by the state-specific distributions, while transitions are specified separately.
  • Unlike other fully Bayesian HMM implementations, VBx does not impose a prior on transition probabilities.This is an explicit modeling simplification in the proposed system.
  • The model represents speaker-specific latent variables as the set Y = {y1, y2, ..., yS}.These variables describe the speaker states used in the generative model.

2.5. Diarization inference

VBx performs variational Bayesian inference by alternating updates of the speaker-vector posterior and the state-sequence posterior while maximizing an ELBO objective. The updates use Gaussian speaker posteriors and efficient forward-backward recursions over soft state responsibilities.

  • Diarization seeks the assignment of frames to speakers, represented by the latent state sequence Z.The exact posterior over Z requires integrating over latent speaker variables, which is intractable.
  • Variational Bayes approximates p(Z, Y|X) with q(Z, Y) using a mean-field factorization.Inference then alternates updates of q(Y) and q(Z) while maximizing the ELBO.
  • The ELBO combines expected log-likelihood with KL-divergence terms for q(Y) and q(Z), with optional scaling factors FA and FB.FA = FB = 1 gives the theoretically correct original ELBO, while other values provide inference control.
  • 2.5.1. Updating q(Y): Updating q(Y) yields Gaussian speaker posteriors whose responsibilities γ_ts softly align x-vector x_t with speaker s.γ_ts is the marginal approximate posterior q(z_t = s) derived from q(Z).
  • 2.5.1. Updating q(Y): Diagonal covariance makes the required matrix inversions and applications trivial, improving computational efficiency.The expected log-likelihood also accounts for uncertainty in the speaker vector posterior.
  • 2.5.2. Updating q(Z): Updating q(Z) requires only the marginal responsibilities γ_ts, which are computed efficiently with a forward-backward algorithm.The method avoids inferring the complete distribution over all possible alignments.
  • 2.5.4. Evaluating the ELBO: Speaker priors π_s are updated by maximizing the ELBO and can reach zero for redundant speakers, selecting the conversation’s speaker count.ELBO evaluation can be monitored for debugging, but the supplied expression evaluates improvement only immediately after q(Z) updates.

3. Experimental setup

The experiments train separate wideband and narrowband x-vector extractors and PLDA models, then extract normalized x-vectors, initialize clusters with AHC, and refine them using VBx. Evaluation covers CALLHOME, AMI, and DIHARD-related recordings, while highlighting protocol and split inconsistencies.

  • x-vector extractor architecture: Both 8 kHz and 16 kHz extractors use the same deep neural network with 64 log Mel filter bank features.The systems differ in Mel-filter frequency ranges: 20-7700 Hz for 16 kHz and 20-3700 Hz for 8 kHz.
  • x-vector extractor architecture: The extractors use statistical pooling and a linear projection to produce 256-dimensional x-vectors.Training uses 4 s segments, corresponding to T = 400 frames.
  • PLDA training: VBx uses PLDA models trained on recording-level x-vectors, then applies them to x-vectors from shorter 1.5 s segments.The authors report that this duration mismatch does not seem to negatively affect diarization performance.
  • VBx diarization pipeline: Input recordings are split into speech segments, and normalized x-vectors are extracted every 0.25 s from overlapping 1.5 s subsegments.The vectors are centered, whitened, and length normalized before clustering.
  • VBx diarization pipeline: AHC supplies initial speaker labels, after which VBx further clusters the x-vectors using 128-dimensional representations.AHC is tuned to under-cluster so VBx can search for a larger optimal speaker count.
  • Evaluation: VBx parameters are tuned on development data for DIHARDII, AMI, and CALLHOME, while AHC-only systems provide baselines.CALLHOME lacks a development-evaluation split, so the dataset is divided into two halves for cross-validation.
  • Evaluation: AMI lacks a standard evaluation protocol, motivating the paper’s proposed protocol based on official partitions and transcriptions.The proposed protocol covers both Beamformed and Mix-Headset audio.

4. AMI evaluation protocols

AMI results are difficult to compare because published studies use inconsistent partitions, references, audio types, and scoring setups. The paper proposes a consistent protocol using official partitions and transcriptions across Beamformed and Mix-Headset audio.

  • Motivation: AMI studies use different partitions, references, audio types, collars, and overlap criteria, making cross-paper comparisons practically impossible.Evaluation speech ranges from around 13000 to 52000 seconds on the eval set, depending on the setup.
  • Proposed protocol: The proposed protocol uses the official Full-corpus-ASR partition and derives references directly from AMI manual annotations.Adjacent same-speaker word segments are merged, while segments separated by pauses remain separate.
  • Reference construction: All words are included as speech, whereas vocal sounds are discarded because some are ambiguous or lack time annotations.This conservative choice is described as consistent with speaker-attributed ASR.
  • Scope and adoption: The same partition and references are applied to AMI Mix-Headset and beamformed microphone-array audio, and the protocol is shared for future use.The protocol is also being adopted by the latest Kaldi and pyannote recipes.

5. Results

The experiments evaluate VBx against baselines on CALLHOME, AMI, and DIHARDII under multiple protocols. VBx achieves strong results on CALLHOME and AMI, while DIHARDII performance is competitive but not the best reported without adaptation and additional processing.

  • CALLHOME: 4.42% DER on CALLHOME is achieved by VBx under the forgiving evaluation setup, outperforming previously published systems.With the fair setup, including overlapped speech that VBx does not handle, performance is 14.21% DER.
  • AMI: 4.08% DER on AMI dev and 3.8% DER on eval are obtained by VBx, versus 7.7% and 5.2% DER for a non-overlap baseline.The comparison uses the corresponding dev and eval results reported for the baseline system.
  • AMI: 3.9% DER is achieved on AMI Beamformed eval audio under forgiving evaluation, while full evaluation reaches 20.84% DER and 26.92% JER.For Mix-Headset audio, VBx obtains 2.10%, 12.53%, and 18.99% DER under forgiving, fair, and full setups.
  • DIHARDII: Without development-set adaptation, VBx outperforms both compared systems on DIHARDII.The paper aims to use a generic approach without development-set training or adaptation.

6. Conclusion

VBx is a Bayesian HMM-based x-vector clustering system that achieves state-of-the-art results on CALLHOME, AMI, and DIHARDII without dataset-specific model adaptation. The paper also proposes a fair AMI evaluation protocol and highlights both the method’s compatibility with newer embeddings and the need for more challenging evaluation setups.

  • The paper presents a Bayesian HMM model for clustering x-vectors, referred to as VBx.
  • VBx achieves state-of-the-art results on CALLHOME, AMI, and DIHARDII without performing dataset-specific model adaptation.
  • VBx clustering can be combined with most newly proposed embedding or x-vector approaches, indicating further potential for the method.
  • The proposed AMI evaluation protocol is intended to provide a fully fair comparison and may become a new standard.
  • CALLHOME and AMI systems reach very low diarization error rates under the standard 0.25 s collar without considering overlapped speech regions.
  • The paper argues for more challenging evaluation setups that include overlapped speech, while future work will replace oracle labels with real VAD.
Loading 2012.14952v1…