Source-linked AI summary
Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model
Sicheng Mao, Baihan Li, Mathieu Fontaine, Anthony Larcher, Roland Badeau
TL;DR
Distant speaker diarization is challenging in difficult acoustic environments with overlapping speech and varying speaker counts. The paper extends neural FCASA’s joint separation and diarization model with heavy-tailed source models. Across evaluated corpora, selected settings consistently improve on the baseline, with improvements reaching up to 20% absolute (35% relative) on CHiME-6.
Problem
Distant diarization must handle difficult acoustic conditions, varying numbers of speakers, and overlapping speech in multichannel recordings.
Source: §I · p. 1Method
The paper extends neural FCASA’s joint source-separation and diarization model by replacing Gaussian source modeling with Leptokurtic Generalized Gaussian and Student’s t distributions.
Source: §I · p. 1, A. Generative model of multichannel… · p. 2Results
Selected heavy-tailed settings consistently improve on the baseline across AMI, AliMeeting, and CHiME-6; CHiME-6 gains reach up to 20% absolute (35% relative).
Source: C. Results and discussions · pp. 4–5Takeaways & Limitations
Experiments identify multiple heavy-tailed parameter settings that outperform the baseline by a large margin.
Source: §V · p. 5Some Leptokurtic Generalized Gaussian settings degrade diarization performance relative to the baseline, indicating sensitivity to hyperparameter tuning.
Source: D. Limitations · p. 5
Abstract
from arXiv · showhide
Distant speaker diarization remains challenging due to difficult acoustic environments, varying numbers of speakers and overlapping speech. Model-driven methods are proposed to exploit the speech source features in multi-channel recordings that help diarization. This paper generalizes a neural model that jointly learns to perform blind source separation and diarization over speech mixtures (neural FCASA) with heavy-tailed models. The popular Gaussian distribution has been applied for variance modeling in the original source separation model, which we replace with two families of heavy-tailed models (Leptokurtic Generalized Gaussian distribution and Student's t distribution) to better capture the heavy-tailedness in speech signals. Thanks to the Gaussian scale mixture model, we are able to unify the proposed method and the original one under the same form of learning objective. Our experiments show consistent large improvements in Diarization Error Rate (DER) and Jaccard Error Rate (JER) compared to the baseline on various corpora.
I. INTRODUCTION
Distant diarization is difficult in noisy, reverberant meeting recordings with variable speaker counts and overlapping speech. This section motivates extending neural FCASA’s joint source-separation and diarization model with heavy-tailed source models.
I. INTRODUCTION
- Distant meeting diarization must handle noisy, reverberant multichannel audio, varying speaker counts, and overlapping speech.Data-driven systems use augmentation and speaker-feature extractors, while model-driven approaches exploit multichannel cues such as direction of arrival and inter-channel time delay.Source: §I · p. 1
- The proposed heavy-tailed neural FCASA extends joint source separation and diarization, and outperforms the baseline across AMI, AliMeeting, and CHiME-6.The authors replace the Gaussian model with Leptokurtic Generalized Gaussian and Student’s t distributions.Source: §I · p. 1
A. Generative model of multichannel mixture signal
- Neural FCASA models a multichannel mixture as source images and uses a binary source-activity mask to jointly train separation and diarization.Its generative model represents source power spectra with a neural decoder from latent source features, under a Gaussian source assumption.Source: A. Generative model of multichannel… · p. 2
B. Inference model
The inference model estimates source features and separation parameters from the mixture, then uses the inferred quantities for source separation and diarization.
B. Inference model
- A hybrid inference model combines a neural encoder for latent source features and activity posteriors with an ISS diagonalizer for projection matrices and diagonal coefficients.The encoder estimates qϕ(Z|X) and qϕ(U|X).Source: B. Inference model · p. 2
- The inferred latent features have Gaussian variational posteriors, while the source-activity mask has a Bernoulli posterior.The Gaussian posterior is parameterized by a mean and variance, and the Bernoulli posterior by logits.Source: B. Inference model · p. 2
- The model estimates isolated image signals with a multichannel Wiener filter and obtains diarization decisions by thresholding activity logits to 0 or 1.Source: B. Inference model · p. 2
III. HEAVY-TAILED NEURAL BLIND SOURCE SEPARATION AND DIARIZATION
The method replaces Gaussian source power-spectrum modeling with heavy-tailed alternatives to better estimate isolated-signal power spectra. It expresses this extension as a Gaussian scale mixture.
A. Heavy-tailed modeling for power spectrum density
- Meeting recordings’ wide overlap-ratio range gives their signals a stronger heavy-tailed property than full-overlap source-separation signals.The paper also notes that heavy-tailed models have shown better robustness to impulsive noise or uncommon scenarios.Source: A. Heavy-tailed modeling for power… · p. 2
- The heavy-tailed extension adds a random positive impulse variable to the Gaussian variance, forming a Gaussian scale mixture.Marginalizing the impulse variable yields specific heavy-tailed models; a deterministic variable recovers the Gaussian model.Source: A. Heavy-tailed modeling for power… · p. 2
- The model uses Leptokurtic Generalized Gaussian and Student’s t families, which approach Gaussian distributions as β approaches 2 and ν approaches infinity, respectively.The shape parameters are β ∈ (0, 2) and ν ∈ R+.Source: A. Heavy-tailed modeling for power… · p. 2, Figure 2 · p. 3
B. Training and inference
Training maximizes the mixture-and-label log-likelihood through an ELBO, combining source reconstruction, latent-variable regularization, and speaker-activity supervision. The method uses PIT for speaker-permutation ambiguity and substitutes the heavy-tailed objective while retaining the baseline training and inference procedure.
B. Training and inference
- The model maximizes the log-likelihood of mixture observations and speaker labels using amortized variational inference to optimize its ELBO.The inference family qϕ(Z|X,U) is approximated by qϕ(Z|X).Source: B. Training and inference · p. 3
- The conditional likelihood uses a one-point Monte Carlo estimate, sampling Z from q(Z|X) and taking U directly from the ground-truth label.The conditional likelihood is specified separately for the Leptokurtic Generalized Gaussian and Student’s t distributions.Source: B. Training and inference · p. 3
- The speaker-activity term directly maximizes the surrogate posterior log qϕ(U|X), equivalent to binary cross entropy between ground-truth masks and logits.The model uses ground-truth activity labels as teacher-forcing for the first-term estimate.Source: B. Training and inference · p. 3
- Permutation invariant training addresses speaker-permutation ambiguity before the final maximization objective is formed.The objective includes a log p(U) term, while the method does not assume a prior model for U.Source: B. Training and inference · p. 3
- The heavy-tailed extension replaces the original objective with a modified one while inheriting the neural FCASA training and inference procedure.The objective uses scaling hyperparameters δ and γ.Source: B. Training and inference · p. 3
IV. EXPERIMENTS
The experiments use the neural FCASA baseline’s settings to support a fair comparison.
IV. EXPERIMENTS
- The proposed method follows the neural FCASA baseline’s experimental settings for a fair comparison.The passage cites for the baseline settings.Source: §IV · p. 3
A. Dataset description
The experiments cover three public multi-channel far-field corpora and evaluate diarization under several scoring setups. They tune model settings on AMI, then test generalization on AliMeeting and CHiME-6, including cross-corpus tests.
A. Dataset description
- The evaluation covers AMI, AliMeeting, and CHiME-6 multi-channel far-field recordings, spanning meetings and dinner-party conversations.AMI and AliMeeting use eight-microphone circular arrays; CHiME-6 recordings from two devices are merged into eight channels.Source: A. Dataset description · pp. 3–4
- AMI contains 100 hours of English meetings, AliMeeting 118.75 hours of Mandarin meetings, and CHiME-6 50 hours of English dinner-party conversations.The datasets have 3–5, 2–4, and 4 participants, respectively.Source: A. Dataset description · pp. 3–4
B. Configurations
- The models use six sources—five speaker channels and one noise channel—with latent dimensions 64 for speakers and 10 for noise.Signals are dereverberated with WPE; training uses randomly sampled 10-second crops from 20-second segments.Source: B. Configurations · p. 4
- DER, its miss, false-alarm, and speaker-confusion components, and JER are reported under Forgiving, Fair, Full, and Overlap setups.Overlap evaluates only overlapping speech without a collar; source-separation metrics are omitted because the corpora lack isolated ground truth.Source: B. Configurations · p. 4
- The study sweeps five generalized-Gaussian shape parameters and four Student’s t degrees-of-freedom settings, alongside a retrained Gaussian neural FCASA baseline.The best checkpoints are selected using validation diarization loss L3.Source: B. Configurations · p. 4
- After selecting settings on AMI, the models are trained and evaluated on AliMeeting and CHiME-6, with additional train-on-one, test-on-another cross-tests.The AMI results table reports percentage metrics and notes that its 10-second chunk-wise evaluation is not directly comparable to Pyannote 3.1.Source: B. Configurations · p. 4, Table I · p. 4
C. Results and discussions
Heavy-tailed models improved on the baseline in reported in-domain evaluations, while cross-test gains were less consistent and remained below in-domain performance.
C. Results and discussions
- On AMI, Student’s t models with ν = {0.1, 1} beat baseline error rates by 2-3% absolute and 10-15% relative, while stricter overlap setups raised error rates.Leptokurtic Generalized Gaussian models with β = 1.0 and 1.6 also performed better than baseline; shape parameter choice substantially affected performance.Source: C. Results and discussions · pp. 4–5
- AliMeeting gains were 1%-2% absolute (10%-20% relative), and CHiME-6 gains reached 20% absolute (35% relative) versus baseline.The reported evaluations used Student’s t settings selected on AMI; the CHiME-6 comparison reflects a difficult corpus and relatively poor baseline performance.Source: Table II · p. 5, Table III · p. 5, C. Results and discussions · p. 5
- AMI–AliMeeting cross-tests showed minor but consistent baseline improvements, whereas CHiME-6 cross-tests were less consistent and all cross-test results remained worse than in-domain results.The authors relate the mismatch to differences in language, scenario, and microphone-array configuration.Source: Table IV · p. 5, C. Results and discussions · p. 5
D. Limitations
Some Leptokurtic Generalized Gaussian parameter settings degraded diarization performance relative to baseline, indicating sensitivity to hyperparameter tuning.
D. Limitations
- Leptokurtic Generalized Gaussian settings β ∈ {0.4, 0.8, 1.2} degraded diarization performance versus baseline.The authors identify hyperparameter sensitivity and leave investigation, including direct parameter estimation, for future work.Source: D. Limitations · p. 5
- The reported degradation indicates that performance depends on the selected shape parameter.The authors describe this as potential sensitivity to hyperparameter tuning.Source: D. Limitations · p. 5
- Direct estimation of the parameters is identified as a possible future direction for addressing tuning.Source: D. Limitations · p. 5
V. CONCLUSION AND FUTURE WORK
The paper concludes that a heavy-tailed extension of neural FCASA jointly performs speaker separation and diarization, with future work spanning model families, parameter tuning, and additional modeling.
V. CONCLUSION AND FUTURE WORK
- The proposed heavy-tailed neural FCASA jointly performs speaker separation and diarization and is formulated as a Gaussian scale mixture with a random impulse variable on variance.This yields a simple objective modification of the original baseline; experiments found various parameter settings that outperform it by a large margin.Source: §V · p. 5
- Future work includes exploring more heavy-tailed model families and their performance on speaker separation.Source: §V · p. 5
- Other proposed directions are improved shape-parameter tuning, speaker-activity modeling, and incorporating microphone-array geometry.Source: §V · pp. 5–6