Source-linked AI summary
Neural Multichannel Distant Speaker Diarization and Source Separation with Beta Speaker Activity Prior
Sicheng Mao, Mathieu Fontaine, Anthony Larcher, Roland Badeau
TL;DR
Distant speaker diarization is challenged by adverse acoustics, changing speaker activity, and overlapping speech. The paper adds a beta speaker-activity prior to neural FCASA and replaces its diarization cross-entropy objective with a variational ELBO. On AMI, diarization improved by at least 3% in DER and 4% in JER relative to baseline.
Problem
Distant speaker diarization must operate under noise, reverberation, variable active speakers, and frequent speech overlaps.
Method
The method adds a beta prior over speaker activity and trains the diarization and separation components jointly with variational inference.
Results
At least 3% DER improvement and 4% JER improvement were achieved relative to the baseline on the AMI dataset.
Takeaways & Limitations
The results support incorporating an explicit speaker-activity prior into the diarization model.
Takeaways & Limitations
Source-separation objective metrics are not evaluated because AMI lacks isolated ground-truth references.
Abstract
from arXiv · showhide
Distant speaker diarization remains challenging due to adverse acoustic conditions, varying numbers of speakers and overlapping speech. While data-driven approaches have shown strong performance, model-driven methods offer a compelling alternative by leveraging spatial information from multichannel recordings. This paper is motivated to propose a Bayesian diarization model for a model-driven method called neural FCASA to enhance its robustness. Specifically, we propose a beta prior over speaker activity and hence a variational lower bound objective that can be seen as a regularized continuous speaker activity score in place of the original cross-entropy loss to train the diarization model. Our experiments show significant improvements in terms of Diarization Error Rate by at least 3% (16% relatively) and Jaccard Error Rate by at least 4% (20% relatively) on the AMI dataset compared to the baseline.
1. Introduction
Distant speaker diarization is difficult in noisy, reverberant multichannel settings with variable speaker activity and overlapping speech. The paper extends neural FCASA with a Bayesian speaker-activity model using a beta prior and variational training.
- Distant diarization must handle noise, reverberation, variable numbers of active speakers, and frequent speech overlaps.
- Data-driven systems improve robustness through augmentation, learned speaker representations, end-to-end diarization, and increasingly large language models.
- Model-driven approaches use multichannel spatial cues such as direction of arrival and inter-channel time delay to characterize speech sources.
- Neural FCASA jointly performs multichannel separation and diarization, but its separation is Bayesian while its diarization is learned non-Bayesianly.
- The proposed model adds a beta prior over speaker activity, enabling Bayesian learning of both diarization and separation through variational inference.
2. Background
Neural FCASA models multichannel mixtures with latent source features, activity masks, and spatial covariance structure, then infers separation and diarization outputs. The paper introduces the beta distribution as a speaker-activity prior alongside the model’s VAE-style and signal-processing components.
- The baseline neural FCASA is structured as a variational autoencoder with neural inference and generative models.
- Its mixture model combines latent spectral features, source power spectral densities, binary activity masks, and spatial covariance matrices.
- The inference model estimates latent source features and spatial covariance matrices using a neural encoder and an ISS diagonalizer.
- The baseline trains diarization with binary cross-entropy while training separation in a Bayesian manner.
- Separation uses a multichannel Wiener filter, while diarization thresholds speaker-activity logits into binary outputs.
- The beta distribution has support x ∈ (0, 1), with its shape controlled by positive hyperparameters α and β.
3. The Beta Neural FCASA
The beta neural FCASA introduces a Bayesian speaker activity model built around a beta prior and PERT parameterization. Its variational training replaces cross-entropy diarization supervision with a continuous activity objective regularized by beta-distribution KL divergence.
- 3.1. Generative modeling with beta speaker activity prior: Speaker activity tendency ηϕ,nt is modeled as a continuous probability in (0, 1), distinct from the binary observed activity mask unt.The tendency represents an inner-state inclination to speak, while the mask records whether speech is active.
- 3.1. Generative modeling with beta speaker activity prior: The beta prior is chosen because speaker activity tendency parameterizes a Bernoulli distribution, making beta its conjugate prior.The motivation also connects the prior to speaker character and conversational ambiance.
- 3.2. The PERT parametrization: The model uses PERT parameters m and λ to control the beta distribution’s mode and concentration, respectively.The PERT parameterization provides a differentiable bijection for α, β ≥1 and separates control over distribution shape.
- 3.2. The PERT parametrization: The model restricts beta-distribution parameters to α, β ≥1, yielding a unimodal activity tendency rather than a U-shaped distribution.The paper argues that typical speakers’ inclination to speak or remain silent should be concentrated rather than simultaneously strong in both directions.
- 3.4. Training: Training maximizes an evidence lower bound using amortized variational inference, with the separation loss retained from neural FCASA.The diarization loss is computed in closed form as the KL divergence between beta distributions, rather than by Monte Carlo approximation.
- 3.4. Training: The diarization objective replaces neural FCASA’s cross-entropy approximation with an ELBO that supervises a continuous tendency variable and applies KL-divergence regularization.The full objective is a weighted sum of separation and diarization loss terms.
4. Experiments
The experiments evaluate beta neural FCASA on the AMI meeting corpus under defined preprocessing, training, and diarization protocols. Compared with the baseline, the proposed model consistently reduces diarization error rates across evaluation setups, while source-separation metrics are not evaluated because isolated references are unavailable.
- Dataset: AMI contains about 100 hours of 16 kHz English meetings with 3–5 participants recorded using an 8-mic circular array.The official split includes 80.7 hours for training, 9.7 hours for development, and 9.1 hours for evaluation.
- Model configuration: The beta neural FCASA follows the neural FCASA architecture, adding only 257 parameters to the original 24 million.The modification therefore has a negligible reported impact on training and inference speed.
- Evaluation: The evaluation reports DER, Miss, false alarm, speaker confusion, and JER under Forgiving, Fair, Full, and Overlap protocols.The protocols differ in collar treatment and whether overlapping regions are included.
- Training: The experiments train six beta-prior settings and a retrained Gaussian baseline, retaining final and development-selected checkpoints before reporting the best metrics.The beta settings vary m between 0.3, 0.5, and 0.7 and λ between 4 and 10.
- Evaluation scope: Source-separation metrics are not evaluated because AMI conversations lack isolated ground-truth references.The reported evaluation therefore focuses on diarization metrics.
- Results: 3% to 4% DER and 4% to 6% JER improvements are obtained across setups, corresponding to relative gains of 16% to 30% and 20% to 27%.The setting m = 0.3, λ = 4 performs best in general, and stricter overlap evaluation produces larger error rates for every model.
5. Conclusion
The paper concludes that adding a beta-distribution speaker-activity prior improves neural FCASA through a fully Bayesian variational formulation. On AMI, this yields at least 3% DER and 4% JER improvements over the baseline, while several extensions remain future work.
- Contribution: The method adds an explicit beta-distribution prior over speaker activity and trains the joint system using variational Bayes.The diarization ELBO is computed in closed form, with PERT parametrization used for learning.
- Results: At least 3% DER and 4% JER improvements are achieved relative to the baseline.The reported relative improvements are at least 16% for DER and 20% for JER.
7. Generative AI Use Disclosure
The authors disclose using Claude to polish selected sections of the paper, after which the generated text was further modified to reflect their own writing.
- Disclosure: Claude was used to rephrase the abstract, parts of the introduction, and selected sections in an academic manner.The outputs were subsequently modified by the authors.