PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion

Israel D. Gebru, Silèye Ba, Xiaofei Li, Radu Horaud

arXiv:1603.09725v2cs.CVcs.SD

TL;DR

Diarization is challenging in informal multi-party dialogue, where speech may overlap and speakers may move or turn away from cameras. This paper combines visual tracking and audio-visual evidence in a temporal diarization model; its method reports strong benchmark performance, including with simultaneous speakers.

  • Problem

    Assigning speech segments to speakers is difficult in multi-party dialogue with noisy, overlapping speech and participants who move, turn away, or become occluded.

    Source: §I · p. 1

  • Method

    The method combines multiple-person tracking and audio-visual fusion with a latent temporal model that estimates speaker identities and speech turns over time.

    Source: §I · p. 2, §IV · p. 5

  • Results

    With TREM, the proposed method achieves the best results on almost all tested scenarios, performs reasonably with simultaneous speakers, and outperforms other methods on AVDIAR.

    Source: C. Results and Discussion · p. 11

  • Takeaways & Limitations

    For a limited number of speakers, of the order of ten, the diarization formulation is efficiently solved via exact inference.

    Source: §IX · p. 13

Abstract

from arXiv · show

Speaker diarization consists of assigning speech signals to people engaged in a dialogue. An audio-visual spatiotemporal diarization model is proposed. The model is well suited for challenging scenarios that consist of several participants engaged in multi-party interaction while they move around and turn their heads towards the other participants rather than facing the cameras and the microphones. Multiple-person visual tracking is combined with multiple speech-source localization in order to tackle the speech-to-person association problem. The latter is solved within a novel audio-visual fusion method on the following grounds: binaural spectral features are first extracted from a microphone pair, then a supervised audio-visual alignment technique maps these features onto an image, and finally a semi-supervised clustering method assigns binaural spectral features to visible persons. The main advantage of this method over previous work is that it processes in a principled way speech signals uttered simultaneously by multiple persons. The diarization itself is cast into a latent-variable temporal graphical model that infers speaker identities and speech turns, based on the output of an audio-visual association process, executed at each time slice, and on the dynamics of the diarization variable itself. The proposed formulation yields an efficient exact inference procedure. A novel dataset, that contains audio-visual training data as well as a number of scenarios involving several participants engaged in formal and informal dialogue, is introduced. The proposed method is thoroughly tested and benchmarked with respect to several state-of-the art diarization algorithms.

I. INTRODUCTION

Diarization in informal multi-party dialogue is difficult when speech overlaps and participants move, turn away, or become occluded. The paper combines visual tracking, speech-source localization, and temporal inference to address these conditions and the limits of prior approaches.

II. RELATED WORK

  • Prior temporal-association methods work best with clean, single-speaker segments, long turns, and participants who continuously face cameras.Methods relying on long audiovisual sequences are limited to offline use.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • Informal dialogue brings noisy, reverberant audio, interruptions and overlap, while participants may move, turn away, become occluded, or enter and leave view.Audio-only signals are ambiguous, and visual speaker detection is challenging when people are not facing cameras.Source: §I · p. 1
  • The proposed approach combines multiple-person visual tracking, multiple speech-source localization, and temporal inference to associate speakers with people, including during simultaneous speech.It also maps acoustic features onto the image plane and uses semi-supervised clustering to assign them to visible persons.Source: §I · p. 2, §II · p. 3

II. RELATED WORK

  • Earlier source-localization methods do not necessarily align sources with visible people or track them over time, while tracking methods do not solve diarization.The cited tracking methods assume people speak continuously; some also require reliable audio or distributed cameras that preserve frontal views.Source: §II · p. 3
  • Unlike methods requiring long past, present, and future frame sequences, the proposed model is causal and can be used online.It addresses mixed speech using spectral sparsity and frequency-wise semi-supervised complex-Gaussian mixture models.Source: §II · p. 3

III. PROPOSED MODEL

The model represents synchronized audio and video as observations over time. It uses tracked person locations and visibility alongside binaural audio features and speech-activity masks.

III. PROPOSED MODEL

  • The synchronized audio-visual stream is indexed by time-step t, with uppercase letters denoting random variables and lowercase letters their realizations.The image sequence is synchronized with two microphone signals.Source: §III · p. 3
  • Visual observations include image-plane locations for up to N people and binary visibility masks indicating which tracked people are present at each time-step.The person locations and visibility masks can be estimated with a multiple-person tracker.Source: §III · pp. 3–4
  • Audio observations are binaural spectrograms formed from left- and right-microphone short-time Fourier transform coefficients across frequencies and frames.The spectrogram coefficients are complex-valued.Source: §III · p. 4
  • The method exploits sparse speech spectrograms and uses binary activity masks to identify which spectrogram entries contain speech amid background noise and other non-speech sounds.Binaural spectrograms and their speech-activity masks characterize the audio observations.Source: §III · p. 4

A. Speaker Diarization Model

The diarization model estimates a binary speaking state for each person from audio-visual observations and visibility controls. Its filtering update combines current evidence with temporal state predictions, enabling exact inference for small groups.

A. Speaker Diarization Model

  • Each person’s binary latent state indicates whether that person is speaking at the current time-step, allowing multiple people to speak simultaneously.The model’s graphical structure distinguishes observed variables, latent variables, and visibility masks treated as control variables.Source: A. Speaker Diarization Model · p. 4, Figure 1 · p. 5
  • The filtering distribution uses observed person locations, binaural audio, and speech-activity masks, while visibility variables serve as control inputs.The model assumes observations are conditionally independent given the speaking state and control input, and the current state depends on the previous state and controls.Source: A. Speaker Diarization Model · p. 4
  • The filtering update combines an observation likelihood with a predictive distribution built from state transitions and the previous filtering distribution.The normalization term sums over possible current states.Source: A. Speaker Diarization Model · p. 4
  • For 2 to 6 people, solving the MAP problem is computationally efficient; the update evaluates the likelihood, transition probabilities, prior filtering distribution, and normalization term.The number of possible state configurations is 2^N, where N is the maximum number of people.Source: A. Speaker Diarization Model · p. 4

B. State Transition Model

The model combines visibility-conditioned speaking-state transitions with visual tracking and binaural audio observations. It also limits diarization to people who are both seen and heard.

B. State Transition Model

  • Speaking-state transitions assume each person's dynamics are independent and vary with whether that person is visible across adjacent time slices.For continuous visibility, a self-transition prior q models remaining in the same state; a newly visible person receives a uniform prior, and an invisible person is assigned a non-speaking prior.Source: B. State Transition Model · p. 5
  • Diarization considers only people who are both seen and heard, while the audio-visual association model addresses sources outside the camera view.The stated boundary applies to the tracking and diarization model; the passage points to the association model as addressing out-of-view sources.Source: B. State Transition Model · p. 5

IV. VISUAL OBSERVATIONS

  • A multiple-person tracker estimates person locations and identities while handling variable track counts, fragmented tracks, appearance changes, and similar-looking people.Its online discriminative learning distinguishes similar appearances, and its handling of head turns supports tracking when people turn away from the camera.Source: §IV · p. 5

V. AUDIO OBSERVATIONS / A. Single Audio Source

  • For a static source, binaural features derived from cross-PSD and auto-PSD are frame-invariant and carry source-location information.The cross-PSD estimate assumes signal-noise cross terms can be neglected; if noise is spatially uncorrelated, noise-noise cross terms can also be neglected.Source: A. Single Audio Source · p. 6

B. Multiple Speech Sources

For simultaneous speech, the method uses spectrogram sparsity to separate speech-bearing frequency-frame points from noise. The resulting binaural coefficients encode source location rather than source content.

B. Multiple Speech Sources

  • Speech sparsity motivates assuming that only one speech source is active at each frequency-frame point, so each point contains one speaker's speech or noise.The method uses this assumption to estimate the binaural spectrogram and its associated binary mask.Source: B. Multiple Speech Sources · p. 6
  • A binary mask is estimated at each frequency-frame point using an adaptive threshold based on noise statistics.The threshold is then used in computing the binaural spectrogram coefficients.Source: B. Multiple Speech Sources · p. 6
  • The resulting binaural coefficients depend on sound-source location, not source content, allowing different sound types for localization training and speech-source prediction.The passage identifies source-independence and location-dependence as the property supporting this use.Source: B. Multiple Speech Sources · pp. 6–7

VI. AUDIO-VISUAL FUSION

The fusion method aligns binaural features with image locations using paired audio-visual training data, then models feature assignments to visible persons. It estimates person speaking probabilities from the resulting association.

VI. AUDIO-VISUAL FUSION

  • Training pairs associate binaural features from loudspeaker recordings with pixel locations corresponding to their sound directions.White noise is used for the training signals, and a visual marker links each loudspeaker location to a pixel location.Source: §VI · p. 7
  • A nearest-neighbor search links a visible person's location to a training audio-source location and its associated binaural feature vector.The method then evaluates the speaking probability of a visible person from the association results.Source: §VI · p. 7
  • A complex-Gaussian mixture models observed binaural features as either person-generated inliers or unknown-source outliers.The outlier component is zero-centered with a very large variance; the mixture parameters are estimated with a simplified EM algorithm.Source: §VI · p. 7

VII. AUDIO-VISUAL DATASETS

The paper introduces AVDIAR, a dataset with audio-visual training data and scenarios spanning varied participant positions and interactions. It also describes the recording setup and several comparison datasets used in evaluation.

Dataset scope

  • AVDIAR was gathered for varied multiple-speaker scenarios, including participants facing the camera, moving, or looking at one another.The dataset includes static and moving participant scenarios and is intended to cover formal and informal interaction settings.Source: §VII · pp. 7–9

Recording setup

  • The camera is attached to a binaural dummy head, and synchronized audio and video are segmented into overlapping 0.4-second slices.Audio is recorded at 44,100 Hz and downsampled to 16,000 Hz; video is captured at 25 FPS.Source: §VII · p. 8

Training data

  • Training data pair white-noise loudspeaker recordings with image locations across 1,600 positions on two 20×40 grids.The grids cover the camera field of view at distances of 1.5 m and 2.5 m.Source: §VII · p. 8, Figure 2 · p. 8

Test scenarios

  • Test scenarios include static participants facing the camera and participants who move around or interrupt one another while speaking.The participants are recorded at distances from 1.0 m to 3.5 m.Source: §VII · pp. 8–10

Other datasets

  • The comparison datasets cover different conditions: MVAD uses camera-facing participants, AVASM includes simultaneous speech, and AV16P3 features moving speakers with overlapping turns.AV16P3 was designed for audiovisual tracking rather than diarization.Source: §VII · pp. 8–9

A. Diarization Performance Measure

Diarization performance is measured with diarization error rate (DER), where lower values indicate better performance. DER counts errors in speech detection and speaker assignment.

Diarization Performance Measure

B. Diarization Algorithms and Setup

The evaluation compares the proposed approach with four diarization methods and two sound-source localization methods. The compared diarization methods vary in their use of audio-visual cues and assumptions about speaker views or overlap.

Diarization Algorithms and Setup

C. Results and Discussion

Across four datasets, the proposed method performs well in several challenging settings, including overlapping speech, and outperforms other methods on AVDIAR. Results also show limits as speaker counts rise and for methods relying on long, clean speech segments or frontal facial views.

Cross-dataset comparison

  • The onset-coincidence method performs least well overall, and DiarTK is second least-performing because it relies on long speech segments with almost no overlap between consecutive segments.With 19-dimensional MFCC vectors from 20 ms frames and a minimum of 50 samples for GMM parameter estimation, segments of at least 19 s are needed.Source: C. Results and Discussion · pp. 10–11

MVAD

  • On MVAD, the proposed method with TREM achieves the best results on almost all tested scenarios, but performance drops as the number of people increases.The authors attribute the drop mainly to sound-source localization resolution of about 10° horizontally, which makes nearby speaking and silent people difficult to disambiguate.Source: Table II · p. 11, C. Results and Discussion · p. 11

AVASM

  • On AVASM, the proposed method performs reasonably well when people speak simultaneously; only Moving-Speaker-01 lacks simultaneous speech.The authors report no meaningful DiarTK results for this dataset.Source: Table III · p. 11, C. Results and Discussion · p. 11

AV16P3

  • On AV16P3, the proposed method matches with one active speaker, while degrades rapidly with two or three people speaking almost simultaneously.The dataset was recorded to benchmark audio-visual tracking, not diarization; DiarTK did not yield meaningful results.Source: Table IV · p. 11, C. Results and Discussion · p. 11

AVDIAR

  • On AVDIAR, the proposed method outperforms all other methods, and the full method performs better than either the TREM or GCC-PHAT variant.The authors attribute the full method's advantage to its robust semi-supervised audio-visual association.Source: Table V · p. 11, C. Results and Discussion · p. 11

IX. CONCLUSIONS

The paper presents a spatiotemporal Bayesian diarization method combining visual tracking, speech-source localization, and audio-visual clustering. It introduces AVDIAR for evaluating mobile, non-frontal dialogue scenarios, while noting planned extensions to richer features and larger groups.

IX. CONCLUSIONS

  • The diarization model combines visual tracking and speech-source localization in a latent-variable dynamic graphical model, with exact inference efficient for a limited number of speakers, around ten.The paper also describes multiple-source localization and weakly supervised audio-visual clustering methods.Source: §IX · p. 13
  • AVDIAR provides carefully annotated scenarios with free movement and turned heads, enabling evaluation in conditions not available in existing datasets.The authors could not compare their method with and because those methods require long speech segments and their software was unavailable.Source: §IX · p. 13
  • Planned extensions include richer visual and audio features and distributed or wearable sensors for scenarios with tens of participants.The proposed visual additions include head-pose estimation and tracking; audio additions include source separation followed by speech recognition.Source: §IX · p. 13

Related papers

Loading 1603.09725v2…