Source-linked AI summary
FASTDIAR: Frame-level speaker encoder for Streaming Diarization
Nikita Torgashov, Okan Köpüklü
TL;DR
Speaker diarization must provide labels as conversations unfold, but common systems repeatedly encode overlapping chunks and may not be optimized to localize speaker changes. FASTDIAR uses a causal, bounded-past frame encoder and confidence-gated online clustering. It performs best on low-overlap benchmarks at sub-second latency, is more robust than cache-based systems as speaker counts grow, and runs five times faster than real time on one CPU thread.
Problem
Streaming diarization needs low-latency speaker labels for conversational systems, while common chunk-based encoders repeatedly process overlapping audio and are not trained to localize speaker changes.
Source: §1 · p. 1Method
FASTDIAR pairs a causal frame-level encoder with a two-second bounded past window and online clustering that gates centroid updates using embedding self-similarity.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
Takeaways & Limitations
Its frame-level encoder is reusable as a drop-in source of speaker identity for other real-time models.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Real-time conversational agents require speaker diarization that streams and runs on a CPU. Most systems apply an utterance-level speaker encoder to short, heavily overlapping chunks, which wastes computation and leaves the model optimized for the wrong task. We instead turn a state-of-the-art speaker recognition architecture into a causal frame-level encoder that reads the stream once and emits one embedding every 80~ms from a bounded two-second window of past audio, and pair it with online clustering that gates every update on the self-similarity of the stream. Trained only by distillation from an utterance-level teacher on simulated and out-of-domain mixtures, and evaluated with one fixed set of hyperparameters, the system is the most accurate streaming diarizer on low-overlap benchmarks at sub-second latency, degrades far less than cache-based systems as the number of speakers grows, and runs five times faster than real time on a single CPU thread.
1. INTRODUCTION
Streaming diarization must identify speakers as a conversation unfolds, often on CPU, but common encoder-based approaches reprocess overlapping audio and are not trained to localize speaker changes. FASTDIAR combines a causal frame-level encoder with confidence-gated online clustering to address these constraints.
1. INTRODUCTION
- Low-latency streaming matters because conversational systems need speaker labels while a conversation unfolds, often without an available GPU.Personalized ASR, VAD, and speech enhancement depend on knowing which speaker is active.Source: §1 · p. 1
- Off-the-shelf speaker encoders applied to heavily overlapping chunks repeatedly re-encode samples and are not trained to forget speakers or localize changes.Their verification training assumes one speaker per utterance.Source: §1 · p. 1
- Offline cascaded and end-to-end diarizers require the whole recording before emitting labels, so they cannot stream [7].Existing streaming alternatives trade accuracy, speaker capacity, CPU speed, or performance beyond two speakers.Source: §1 · p. 1
- Prior frame-rate approaches lack a bounded-past streaming encoder: one uses future context, while another carries the whole recording in recurrent state.Earlier streaming encoders also commonly use segment-level windows or turn detection.Source: §1 · p. 1
- FASTDIAR emits causal frame-level embeddings from a bounded two-second past window without chunking, pooling, or recomputation, and pairs them with confidence-gated online clustering.The authors report sub-second latency, stronger low-overlap benchmark accuracy, greater robustness as speaker count grows, and five-times-real-time CPU speed.Source: §1 · p. 1
2. METHOD
FASTDIAR adapts a speaker-recognition network into a causal frame-level encoder with bounded past context, then uses embedding self-similarity to decide which frames may update online speaker clusters. A fixed correction window bounds emission delay while allowing labels to be refined.
2.1. Streaming speaker encoder
- Causal convolutions, frequency-and-channel-only fully connected blocks, and removal of temporal pooling yield 192-dimensional frame embeddings with a two-second past receptive field.The encoder emits an ℓ2-normalized embedding every 80 ms, reads the stream once, and does not recompute samples; this trades verification accuracy for prompt speaker-change handling.Source: §2.1 · p. 2
2.2. Online clustering
- The confidence gate admits only frames whose embedding is sufficiently self-similar to one D frames earlier into cluster updates; all other frames are still labeled.Low similarity can indicate a speaker change, silence, or noise, while excluding such frames avoids dragging cluster centroids together.Source: §2.2 · p. 2
- Confident frames join the closest cluster above a similarity threshold or open a new one, and each cluster is represented by the ℓ2-normalized median of its confident-frame embeddings.The median is used because it resists outliers better than the mean.Source: §2.2 · p. 2
- A 960 ms correction window delays labels so later confident frames and online merges can refine them, while unconfident frames are assigned without updating a centroid.The fixed window prevents unbounded waits during low-confidence stretches; non-speech frames are emitted as silence, and clustering thresholds are tuned on development data then fixed.Source: §2.2 · p. 2
3. EXPERIMENTS
The experiments train a frame-level student by distilling an utterance-level teacher on simulated and out-of-domain mixtures, then compare it with streaming systems and a chunked encoder baseline. Evaluation spans diarization, verification, latency, and CPU processing speed under specified protocols.
3.1. Datasets
- Benchmarks span six diarization corpora with 2–21 speakers per recording, alongside VoxCeleb1-E speaker-recognition evaluation.Training mixtures draw mainly on VoxCeleb2 crops and also include two- or three-speaker LibriHeavyMix examples, noise, and simulated room responses.Source: §3.1 · p. 3
3.2. Model and training
- The frame-level student is distilled from a ReDimNet2-B6 utterance-level teacher by maximizing cosine similarity on augmented mixture frames where exactly one speaker is active.The teacher embeds a clean crop for the active speaker; overlap frames are excluded from the loss.Source: §3.2 · p. 3
3.3. Baseline models
- Comparisons include reproduced Streaming Sortformer, diart, and LS-EEND systems, plus an unmodified ReDimNet2-B6 applied to two-second windows every 80 ms with the same VAD and clustering.LS-EEND uses its DIHARD3 checkpoint by default and its AMI checkpoint for that test.Source: §3.3 · p. 3
3.4. Evaluation protocol
- Diarization is measured by DER with a zero collar, with and without overlap and split by speaker count; latency is fixed at 960 ms for every frame.RTF measures processing time over input duration on one CPU thread, while verification uses EER; Table 2 describes VoxCeleb1-E context and shift in milliseconds.Source: §3.4 · pp. 3–4, Table 2 · p. 4
4. RESULTS AND DISCUSSION
FASTDIAR leads on most low-overlap benchmarks, but its relative performance changes when overlap is counted and it is last on DIHARD3. It degrades less than cache-based systems as speaker count grows, trading verification accuracy for a faster frame-level encoder.
Diarization accuracy
- Without overlap, FASTDIAR leads on NOTSOFAR, VoxConverse, and RAMC, trailing LS-EEND only on AMI; counting overlap puts Sortformer ahead on four corpora.With overlap counted, FASTDIAR leads on VoxConverse and RAMC.Source: §4 · p. 4
Speaker count
- Across the four-speaker boundary, DER grows 1.2 to 1.5 for FASTDIAR versus 2.3 to 4.4 for Sortformer; FASTDIAR leads the ≥5-speaker column on two corpora.FASTDIAR is not tied to a fixed speaker count, unlike cache-based systems.Source: §4 · p. 4
Generalization
- FASTDIAR is last on DIHARD3 under both protocols; its simulated mixtures cover only a fraction of the scenarios, while every system ahead has seen DIHARD3.VoxConverse and NOTSOFAR were not fully unseen because checkpoint and threshold tuning used their development sets.Source: §4 · p. 4
Ablation
- Table 3 compares model scaling and offline post-processing, excluding overlap and fixing latency at 960 ms.ReDimNet2-B6-S is the better verification model, yet Medium and Large perform better on diarization; accuracy also improves monotonically from Small to Large.Source: Table 3 · p. 4, §4 · p. 4
Encoder trade-off
- The proposed encoder is more than 25 times faster than the chunked model, but its EER is around 3 times worse because it cannot pool over an utterance.Embedding one trial side with the offline teacher cuts X-EER from 1.51 to 0.86 for the proposed encoder and from 0.44 to 0.39 for the chunked model.Source: §4 · p. 4
Limitations
- FASTDIAR assigns one label per frame, so overlapped regions count as misses; post-processing can merge similar clusters after the stream ends.The overlap limitation comes from decoding, not the encoder; future work proposes multi-label frame prediction without changing encoder inference cost.Source: §4 · p. 4
5. CONCLUSION
FASTDIAR uses a causal frame-level speaker encoder that reads the stream once and emits embeddings every 80 ms from a two-second past window. With confidence-gated online clustering, it achieves strong low-overlap streaming diarization results and the encoder can be reused by other real-time models needing frame-level speaker identity.
Conclusion
- FASTDIAR reads the stream once and emits an embedding every 80 ms from a causal two-second past window.The encoder needs no chunking, temporal pooling, or recomputation.Source: §5 · p. 4
- With confidence-gated online clustering at a fixed 960 ms delay, FASTDIAR is the most accurate streaming diarizer on low-overlap benchmarks and degrades less than cache-based systems beyond four speakers.It runs five times faster than real time on one CPU thread and needs no diarization corpus in training.Source: §5 · p. 4
- The frame-level encoder is reusable by any real-time model that needs speaker identity.The diarizer described here is one consumer of the encoder.Source: §5 · p. 4