PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Speaker Diarization: Recent Advances with Deep Learning

Tae Jin Park, Naoyuki Kanda, Dimitrios Dimitriadis, Kyu J. Han, Shinji Watanabe, Shrikanth Narayanan

arXiv:2101.09624v4eess.AScs.CLcs.SD

TL;DR

Speaker diarization must identify speaker turns in increasingly complex multispeaker recordings, while prior surveys did not consolidate recent neural developments. This paper reviews historical and neural approaches, their integration with speech recognition, and the field’s movement toward joint modeling. It concludes that deep learning has progressed from replacing individual modules to fully end-to-end diarization, while overlap remains a central challenge.

  • Problem

    Earlier surveys covered historical, broadcast, telephone, or meeting-focused developments, but recent deep-learning advances required a consolidated review.

  • Method

    The paper surveys modular, neural, joint-optimization, and speech-recognition-integrated speaker diarization approaches across the field’s development.

  • Results

    The review finds a progression from replacing individual diarization modules with deep-learning methods to fully end-to-end diarization and tighter integration with speech recognition.

  • Takeaways & Limitations

    Neural methods consolidate recent diarization developments and support continued progress toward more efficient systems, including approaches that handle overlap and integrate recognition.

  • Takeaways & Limitations

    Speaker overlap remains a scope challenge because many traditional systems focus on non-overlapping regions and may exclude overlap from evaluation.

Abstract

from arXiv · show

Speaker diarization is a task to label audio or video recordings with classes that correspond to speaker identity, or in short, a task to identify "who spoke when". In the early years, speaker diarization algorithms were developed for speech recognition on multispeaker audio recordings to enable speaker adaptive processing. These algorithms also gained their own value as a standalone application over time to provide speaker-specific metainformation for downstream tasks such as audio retrieval. More recently, with the emergence of deep learning technology, which has driven revolutionary changes in research and practices across speech application domains, rapid advancements have been made for speaker diarization. In this paper, we review not only the historical development of speaker diarization technology but also the recent advancements in neural speaker diarization approaches. Furthermore, we discuss how speaker diarization systems have been integrated with speech recognition applications and how the recent surge of deep learning is leading the way of jointly modeling these two components to be complementary to each other. By considering such exciting technical trends, we believe that this paper is a valuable contribution to the community to provide a survey work by consolidating the recent developments with neural methods and thus facilitating further progress toward a more efficient speaker diarization.

1. Introduction

Speaker diarization identifies who spoke when by segmenting and clustering multispeaker audio, evolving from modular systems toward neural and jointly modeled approaches. This survey consolidates historical developments, recent neural methods, evaluation measures, and connections with speech recognition.

  • Speaker diarization segments audio into speech groups sharing speaker identities and detects events such as speech transitions and speaker turns.
  • Traditional systems use independent modules for front-end processing, speech activity detection, feature extraction, clustering, and post-processing.
  • Early diarization supported speaker-adaptive automatic speech recognition, while later systems became standalone sources of speaker-specific information for downstream tasks.
  • i-vectors improved diarization representations over MFCCs and speaker factors, while neural d-vectors and x-vectors later replaced them with learned speaker embeddings.
  • The survey organizes methods by diarization-oriented objectives and optimization structure, while explicitly avoiding claims that taxonomy categories are superior or inferior.
  • DER combines false alarm, missed speech, and speaker-confusion errors, whereas JER averages per-speaker errors and WDER targets lexical transcription-side discrepancies.

2. Modular Speaker Diarization Systems

This section surveys non-diarization-objective methods by following the modules of the traditional speaker diarization system and summarizing recent techniques within each module.

  • The section reviews non-diarization-objective techniques according to the traditional system’s individual modules.
  • Each subsection explains one traditional module and summarizes recent techniques used within it.
  • The modular organization covers systems whose components are not jointly optimized for a diarization objective.

2.1. Front-end Processing

Front-end processing estimates cleaner or separated speaker signals before downstream diarization. Recent systems use enhancement, dereverberation, and multichannel separation, with reported gains in meeting and challenge tasks.

  • Front-end processing estimates each speaker’s original signal from multichannel observations for downstream diarization.
  • Deep-learning speech enhancement suppresses noise, and LSTM-based enhancement was used in the DIHARD II baseline.
  • 18.3% to 13.9% DER was achieved in the LibriCSS meeting task using mask-based MVDR beamforming.
  • Dereverberation: WPE dereverberation estimates filters that preserve early reflections while suppressing late reverberation through maximum-likelihood estimation.
  • Dereverberation: WPE is widely used in DIHARD and CHiME baselines and top-performing systems, providing solid improvement across almost all tasks without signal distortions.
  • Multichannel processing improves enhancement, dereverberation, and separation effectiveness, while GSS-based extraction achieved a top CHiME-6 challenge result.

2.2. Speech Activity Detection

Speech activity detection separates speech from non-speech and strongly affects diarization because its errors propagate through subsequent processing. Deep neural SAD systems have surpassed traditional statistical approaches.

  • Speech activity detection distinguishes speech from non-speech and serves as preprocessing for diarization, speaker recognition, and speech recognition.
  • SAD combines acoustic feature extraction with a classifier that predicts whether each input frame contains speech.
  • Traditional spectrum, GMM, and HMM systems were followed by MLP-, CNN-, and LSTM-based systems with superior performance.
  • SAD errors can create false positive events or miss speech segments, so evaluations distinguish oracle SAD from actual system SAD output.

2.3. Segmentation

Speech segmentation divides audio into speaker-uniform segments, using either detected speaker-change points or fixed windows. Modern systems largely favor uniform segmentation because variable segment lengths can impair speaker representations, despite a segment-length trade-off.

  • Speaker segmentation produces speaker-uniform output units through either speaker-change point detection or uniform segmentation.Change-point methods test whether adjacent windows come from the same or different speakers; uniform methods use fixed window and overlap lengths.
  • Earlier change-point systems compared same-speaker and different-speaker hypotheses using metric-based criteria including KL distance, GLR, and BIC.BIC became especially widely used, with covariance-based scoring and a penalty term.
  • After i-vector and DNN-based embeddings emerged, uniform segmentation mostly replaced change-point segmentation because variable lengths added variability and reduced speaker-representation fidelity.
  • Uniform segmentation keeps diarization output duration constant by applying fixed window and overlap lengths.
  • Uniform segmentation faces a segment-length trade-off: segments must be short enough to avoid multiple speakers but long enough to support reliable representation.

2.4. Speaker Representations and Similarity Measure

Speaker diarization relies on representations and similarity measures that evolved from metric-based comparisons and GMM-derived models to i-vectors and neural embeddings. These developments improved speaker characterization and addressed variability across speakers, channels, and sessions.

  • Speaker representations and similarity measures determine how diarization systems compare and cluster speech segments.
  • Metric-based approaches such as KL distance, GLR, and BIC were widely used before learned speaker representations became prevalent.BIC was extensively used because of its effectiveness and ease of implementation.
  • GMM-UBM systems modeled speaker identity but suffered intersession variability because MAP adaptation captured channel noise and other acoustic-environment nuisances.
  • JFA separated inter-speaker and channel or session variability, while the i-vector combined these spaces through a total variability matrix.The i-vector is a latent variable weighting the columns of that matrix and serves as a speaker representation.
  • i-vector representations were widely adopted for diarization and showed superior performance to BIC, GLR, and KL metric-based methods.Extraction uses MAP estimation with Baum–Welch statistics, a UBM supervector, and a trained total variability matrix.
  • PLDA uses a log-likelihood ratio to choose between same-speaker and different-speaker hypotheses and to measure cluster similarity in diarization.
  • Deep learning introduced d-vectors and x-vectors, with x-vectors using time-delay architecture and statistics pooling to aggregate frame-level outputs.Neural embeddings are typically derived from bottleneck-layer outputs of DNNs trained for speaker recognition.

2.5. Clustering

Speaker diarization clustering groups speech segments using speaker representations and similarity measures. AHC iteratively merges similar clusters, while spectral clustering constructs graph-based embeddings and can outperform AHC with PLDA depending on the dataset.

  • Clustering groups speech segments according to speaker representations and similarity measures.
  • Agglomerative Hierarchical Clustering: AHC starts with singleton clusters and repeatedly merges the most similar pair until a stopping criterion is reached.Stopping can use a similarity threshold or a target speaker count, with thresholds adjusted using development data.
  • Spectral Clustering: Spectral clustering builds an affinity matrix, computes a graph Laplacian, performs eigen decomposition, optionally renormalizes rows, estimates speaker count from the maximum eigengap, and clusters spectral embeddings.The k smallest eigenvalues and corresponding eigenvectors form embeddings that are commonly clustered with k-means.
  • Spectral Clustering: NJW spectral clustering is often used with varied affinity kernels and typically cosine distance for speaker diarization.
  • Spectral Clustering: Depending on the dataset, spectral clustering with cosine distance can outperform AHC with PLDA when both use the same speaker representation.
  • K-means is simple and widely used, but generally underperforms spectral clustering and AHC in speaker diarization.

2.6. Post-processing

Post-processing refines diarization boundaries and combines multiple hypotheses to improve robustness, while newer fusion methods address overlapping speech.

  • Viterbi resegmentation alternates speaker-GMM estimation with boundary refinement using the Baum-Welch algorithm.
  • VB-HMM-based diarization jointly optimizes segmentation and clustering and was shown to be superior to Viterbi resegmentation.
  • DOVER aligns speaker labels across hypotheses, then selects labels for regions by weighted voting.
  • DOVER assumes that at most one speaker is active at each time index, limiting its direct use for overlapping speech.
  • Modified DOVER and DOVER-Lap showed DER improvement on diarization results containing speaker overlaps.

2.7. Joint Optimization of Segmentation and Clustering

VB-HMM provides a joint framework for segmentation and clustering, while VBx adapts this approach to x-vector representations and models speaker turns and duration.

  • VB-HMM models speech features with speaker-specific HMM states and infers the speaker trajectory jointly with latent speaker variables.
  • Variational Bayes estimates tractable approximations to the posterior because direct inference of the joint distribution is intractable.
  • VBx calculates emissions from x-vectors using PLDA and operates at x-vector rather than frame-level granularity.
  • VBx can be viewed as clustering that jointly models speaker turns and speaker duration.
  • In common use, AHC first under-clusters x-vectors before VBx refines the resulting speaker clusters.

3. Recent Advances in Speaker Diarization Using Deep Learning

Recent deep-learning diarization work enhances individual modules and increasingly unifies segmentation, clustering, speaker representation, and overlap handling in neural systems.

  • Deep-learning diarization research covers both neural enhancement of individual components and unified networks spanning multiple diarization stages.
  • 3.1.1. Speaker clustering Enhanced by Deep Learning: A GNN purifies the affinity matrix used by spectral clustering by learning from speaker-embedding sequences.
  • 3.1.1. Speaker clustering Enhanced by Deep Learning: The GNN method significantly outperformed conventional clustering methods on CALLHOME and an in-house meeting dataset.
  • 3.1.1. Speaker clustering Enhanced by Deep Learning: Deep embedded clustering methods iteratively refine soft speaker-cluster assignments, with IDEC adding reconstruction and other loss terms.
  • Speaker-embedding duration creates a trade-off between time resolution and representation robustness, while embeddings are not explicitly optimized for diarization.
  • 3.1.3. Post Processing Based on Deep Learning: TS-VAD uses MFCC sequences and speaker i-vectors to estimate each available speaker’s frame-level speech activity probability.
  • 3.1.3. Post Processing Based on Deep Learning: TS-VAD achieved significantly better DER than conventional clustering in the CHiME-6 winning system, but its output dimension limits the maximum speakers handled.
  • 3.2.2. Joint Segmentation, Embedding Extraction, and Resegmentation: RPN-based diarization handles overlapping speech with possibly any number of speakers and achieved significantly better DER than conventional clustering across multiple datasets.

4. Speaker Diarization in the Context of ASR

The paper treats diarization as an ASR-related component whose errors can affect recognition, then reviews approaches that use lexical information and jointly model both tasks. Recent joint systems include speaker tags, iterative joint decoding, and end-to-end speaker-attributed ASR.

  • Motivation: Traditional diarization processes speech sequentially without considering ASR performance, and tight segment boundaries can cause word truncation or deletion errors.The section frames diarization and ASR as mutually relevant rather than independent pipeline stages.
  • Lexical information: ASR lexical information has been used for segmentation, SAD refinement, and clustering to improve speaker diarization.Sequence-to-sequence speaker-turn tokens exploit word boundaries from ASR output.
  • Joint modeling: Joint modeling leverages ASR cues to improve diarization and diarization outputs to improve ASR, including speaker adaptation.The reviewed approaches aim to exploit the inter-dependency between the two tasks.
  • Speaker-tag transcription: Speaker tags inserted into end-to-end ASR reduced WDER with marginal WER degradation in doctor-patient conversations.The approach inserts speaker role tags such as doctor and patient into RNN-T output.
  • Joint decoding: Iterative joint decoding combines target-speaker ASR with overlap-aware speaker-embedding estimation, achieving speaker-attributed WER similar to oracle-embedding target-speaker ASR.The method uses Viterbi-based decomposition into iterative problems and is challenging to apply online.
  • End-to-end SA-ASR: End-to-end SA-ASR jointly performs speaker counting, multi-talker ASR, and speaker identification, using speaker profiles or dummy profiles with internal speaker clustering.On LibriCSS, it achieved significantly better cpWER than combining multitalker ASR with speaker diarization.

5. Diarization Evaluation Series and Datasets

The review surveys evaluation series and datasets used to assess speaker diarization across broadcast, telephone, meeting, conversational, and challenging real-world conditions. These resources differ in speakers, recording setups, languages, overlap, and environmental difficulty.

  • Telephone speech: CALLHOME contains 500 multilingual telephone sessions with two to seven speakers and two dominant speakers per conversation.It is described as the most widely used dataset for speaker diarization in recent papers.
  • Meeting corpora: AMI provides 100 hours of meeting recordings across 171 sessions using lapel microphones and omnidirectional microphone arrays.Its multiple recording sources support evaluation of diarization integrated with ASR.
  • Meeting corpora: The ICSI Meeting Corpus contains 75 meetings with three to ten participants and includes word timings, transcripts, speaker labels, and multichannel recordings.It combines close-talking individual microphones with six tabletop microphones.
  • Noisy conversational data: CHiME-5 contains 50 hours of multiparty real conversations recorded in everyday home environments with six four-channel microphone arrays.The dataset includes speaker labels, segmentation, and corresponding transcriptions.
  • Overlapped speech: LibriCSS contains 10 hours of seven-channel multichannel recordings designed for speech separation, recognition, and diarization research.It consists of ten sessions involving eight speakers and varying overlap conditions.
  • Evaluation series: Diarization evaluations cover broadcast news, conversational telephone speech, meetings, clinical interviews, web videos, speech in the wild, and daily conversations.The RT series focuses on ASR and diarization with descriptive speaker metadata, while DIHARD targets challenging domains.

6. Applications

Speaker diarization supports meeting transcription, behavioral and medical conversation analysis, audio indexing, and conversational AI. Applications must often balance richer multimodal information and speaker attribution against low latency, overlap, noise, and complex recording setups.

  • Meeting transcription: Meeting transcription aims to generate speaker-attributed transcripts for tasks such as summarization, topic extraction, and healthcare applications.The same transcription system can support multiple downstream domains.
  • Meeting transcription: Recent advances in recognition, far-field processing, speaker identification, and diarization improved speaker-attributed transcription accuracy and enabled commercialization.Combining cameras with microphone arrays further improved overall performance.
  • Meeting transcription: Meeting transcription systems may need low-latency output, known-geometry microphone arrays, multimodal inputs, and support for changing numbers of audio channels.Flexible modular architectures are desirable for these varying settings.
  • Meeting transcription: Overlapping speech remains a major meeting-transcription challenge because existing separation methods have had limited success with reverberation and background noise.The passage specifically notes limitations of ICA, spatial clustering, PIT, and deep clustering approaches.
  • Meeting transcription: Natural-meeting speaker-attributed ASR requires streaming recognition, dereverberation, accurate diarization, and speaker identification, which are usually optimized separately.The resulting multi-stage pipeline is described as frequently inefficient.
  • Conversation analysis: Diarization enables quantitative turn-taking analysis and behavioral signal processing using speaker-specific speech segments without explicit lexical transcription.Behavioral applications include analysis of affective, social, relational, and health-related patterns.
  • Conversation analysis: In medical conversations, diarization combined with ASR-derived speech and language features supports prediction of a patient’s memory-disorder type.The cited system analyzes neurologist-patient interactions.
  • Audio indexing: Diarization augments ASR transcripts with speaker identity and timing, enabling per-speaker summaries and keyword lists for content-based audio retrieval.Aggregating utterances by speaker provides additional query values for indexing audio databases.

7. Challenges and the Future of Speaker Diarization

Speaker diarization has advanced substantially through deep learning, integration with ASR, and multimodal modeling, but important practical challenges remain. Future work must address latency, domain mismatch, speaker overlap, unresolved ASR integration choices, and audiovisual modeling.

  • Online processing of speaker diarization: Online diarization remains difficult because many applications require very short latency, although both clustering-based and neural systems have been attempted.Most methods assume the entire recording is available before diarization.
  • Domain mismatch: Models trained in one domain can perform poorly in another, and EEND models may overfit to the speaker-overlap distribution of their training data.The paper identifies domain mismatch as a general issue for training-based diarization methods.
  • Speaker overlap: 12% to 15% of speaker overlap occurs in meeting recordings, yet many traditional systems focus on non-overlapping regions or exclude overlap from evaluation.Recent approaches include speech separation, post-processing, and joint speech separation–diarization modeling.
  • Integration with ASR: The best architecture for combining speaker diarization and ASR remains open because modular systems are strong on specific tasks and joint frameworks are not yet fully validated against tuned modular systems.Joint modeling may leverage inter-dependency between the two tasks, but comparative evidence remains incomplete.
  • Audiovisual modeling: Audiovisual modeling has improved diarization accuracy in meeting transcription and YouTube-video studies, but the research direction remains unfinished.Visual information provides a strong clue for speaker identification.
Loading 2101.09624v4…