Source-linked AI summary
Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering
Ivan Medennikov, Taejin Park, Weiqing Wang, He Huang, Kunal Dhawan, Jinhan Wang, Jagadeesh Balam, Boris Ginsburg
TL;DR
Streaming speaker diarization is relatively underexplored, while offline Sortformer is limited by its reliance on full-length self-attention. This paper adds an arrival-ordered speaker cache that stores frame-level embeddings and updates them using model predictions. The system achieves state-of-the-art results among published streaming systems and remains competitive at 0.32-second latency.
Problem
Streaming speaker diarization is relatively underexplored, and offline Sortformer’s full-length self-attention makes it unsuitable for streaming and limits long-recording processing.
Source: §1 · p. 1Method
The Arrival-Order Speaker Cache stores pre-encoder frame embeddings by speaker arrival order and updates the cache by selecting high-scoring frames from model predictions.
Source: §3.1 · p. 2, §3.2 · p. 3Results
The system achieves state-of-the-art results among previously published streaming systems and remains highly competitive at 0.32-second latency.
Source: §4.3 · p. 4Takeaways & Limitations
The authors plan to extend the system to eight speakers and integrate it into multi-speaker ASR, speech translation, and summarization.
Source: §5 · p. 4CH109 recordings overlap slightly with CALLHOME part1 training data, though the authors expect this to have no significant effect on results.
Source: §4.3 · p. 3
Abstract
from arXiv · showhide
This paper presents a streaming extension for the Sortformer speaker diarization framework, whose key property is the arrival-time ordering of output speakers. The proposed approach employs an Arrival-Order Speaker Cache (AOSC) to store frame-level acoustic embeddings of previously observed speakers. Unlike conventional speaker-tracing buffers, AOSC orders embeddings by speaker index corresponding to their arrival time order, and is dynamically updated by selecting frames with the highest scores based on the model's past predictions. Notably, the number of stored embeddings per speaker is determined dynamically by the update mechanism, ensuring efficient cache utilization and precise speaker tracking. Experiments on benchmark datasets confirm the effectiveness and flexibility of our approach, even in low-latency setups. These results establish Streaming Sortformer as a robust solution for real-time multi-speaker tracking and a foundation for streaming multi-talker speech processing.
1. Introduction
Streaming speaker diarization remains underexplored, and offline Sortformer’s full-length self-attention limits its use on streams and long recordings. The paper extends Sortformer with an arrival-ordered speaker cache for streaming predictions.
1. Introduction
- Streaming diarization is less explored than offline diarization, despite demand for real-time speaker labels in applications such as live captioning and virtual meetings.The introduction also notes a wider performance gap between offline and streaming diarization than between offline and online ASR.Source: §1 · p. 1
- Offline Sortformer’s full-length self-attention makes it unsuitable for streaming and constrains processing of long recordings.Sortformer’s arrival-time speaker ordering is a useful basis for a streaming extension.Source: §1 · p. 1
- The proposed streaming extension combines the speaker-tracing-buffer concept with Sortformer’s arrival-time ordering in an Arrival-Order Speaker Cache (AOSC).Unlike conventional speaker-tracing buffers, AOSC stores embeddings by speaker index, which corresponds to speakers’ arrival order.Source: §1 · pp. 1–2
- Each streaming step concatenates the speaker cache, preceding audio chunks in a FIFO queue, and the input buffer to form predictions.The input buffer contains the current chunk and right context.Source: Figure 1 · p. 1, §1 · p. 2
- The approach resolves between-chunk speaker permutations without attractors or permutation-resolution operations for speaker-tracing buffers.The paper contrasts this with prior approaches using self-attention attractors, local or global attractors, or buffer permutation resolution.Source: §1 · p. 2
2. Related Work
Offline Sortformer sorts output speakers by arrival time, while chunk-wise online processing must maintain consistent speaker identities across chunks. Speaker-tracing buffers address this consistency problem through stored frames and predictions.
2.1. Foundation: Offline Sortformer
- Sortformer uses a self-supervised NEST encoder and Transformer layers, producing four speaker-activation outputs with an effective prediction step of 80 ms.Its input features have a 10 ms frame step, and the NEST pre-encoder downsamples by 8x.Source: §2.1 · p. 2
- Sortformer’s Sort Loss computes binary cross-entropy over arrival-time-sorted target labels alongside conventional permutation-invariant loss.This sorting gives the model arrival-time-ordered speaker outputs.Source: §2.1 · p. 2
2.2. Chunk-Wise Processing with a Speaker-Tracing Buffer
- Chunk-wise processing predicts speakers for each input chunk, but must resolve permutations across chunk predictions to produce a consistent output sequence.The chunks are indexed n, and each chunk prediction covers s speakers.Source: §2.2 · p. 2
- Speaker-tracing buffers address cross-chunk consistency by storing prior frames and predictions, then using a speaker permutation that maximizes correlation with buffer predictions.The buffer is updated at each step, and the resulting predictions are consistently permuted.Source: §2.2 · p. 2
3. Proposed Method
The method adapts Sortformer for streaming with an Arrival-Order Speaker Cache (AOSC) that stores embeddings in speaker arrival order, alongside a FIFO queue for context. The cache update selects high-scoring frames while preserving speaker representation and dynamically allocating embeddings per speaker.
3.1. Arrival-Order Speaker Cache
- AOSC orders frame-level embeddings by speaker index, matching arrival order and automatically resolving between-chunk permutations.The cache stores embeddings from the pre-encode NEST module.Source: §3.1 · p. 2
3.2. Speaker Cache Update Mechanism
- For sequences longer than M, AOSC selects top-scoring frames, favors recent frames, enforces K-frame speaker representation, and appends average-silence embeddings.The number of embeddings per speaker is determined dynamically by their scores, while at least K frames are retained when their scores exceed −∞.Source: §3.2 · pp. 2–3
3.3. Streaming Inference with a FIFO Queue
- A FIFO queue alongside AOSC provides additional context and permits cache updates less often than after every chunk, improving robustness and efficiency.Frames pushed out of the queue are processed by the speaker-cache update mechanism.Source: §3.3 · p. 3
4. Experimental Results
Streaming Sortformer achieves state-of-the-art results among published streaming systems, with competitive performance at 0.32-second latency. It also outperforms offline Sortformer on two evaluations and tracks dominant speakers in subsets with more than four speakers despite its four-speaker design limit.
4.1. Datasets
- Training uses 5,150 hours of simulated mixtures and 2,030 hours of real multi-talker speech; evaluation covers DIHARD III Eval, CALLHOME Part2, and two-speaker CH1094.The data includes multiple meeting and conversational speech datasets, with segments cut to 90 seconds and up to four speakers.Source: §4.1 · p. 3
4.2. Training Setup
- The 117M-parameter offline Sortformer checkpoint is fine-tuned with AOSC using sequential 15-second windows and a 15-second speaker cache.Training also randomizes cache speaker order and limits right context to 7 frames with 50% probability per batch.Source: §4.2 · p. 3
4.3. Evaluation and Comparative Analysis
- Table 1 reports DER with overlapping speech included and dataset-specific collar tolerances; Table 2 gives latency setups, while RTF measures processing time divided by recording length.The stated latency values refer to input-buffer delay and exclude computational latency.Source: Table 1 · p. 4, Table 2 · p. 4, §4.3 · p. 4
- One model is evaluated across all datasets without dataset-specific fine-tuning, while timestamp post-processing parameters are tuned on development or training splits.The post-processing sets are tuned on DIHARD III Dev and CALLHOME Part1 recordings with at most four speakers.Source: §4.3 · p. 4
- AOSC can be applied to offline Sortformer without fine-tuning, but DER significantly increases; the authors identify AOSC fine-tuning as essential for optimal results.The evaluation compares offline Sortformer, inference-only Offline Sortformer-AOSC, and fine-tuned Streaming Sortformer-AOSC.Source: §4.3 · pp. 3–4
- On 5+ speaker subsets, the four-speaker system shows solid performance, which the authors attribute to robustly tracking the four most dominant speakers.The evaluation also examines benchmarks beyond the system's primary design scope of up to four speakers.Source: §4.3 · pp. 3–4
- On DIHARD III and CALLHOME 4+ speaker evaluations, streaming Sortformer outperforms offline Sortformer, which the authors say likely struggles with long recordings after training on 90-second samples.The streaming system avoids this issue with a fixed inference window.Source: §4.3 · p. 4
- Streaming Sortformer achieves state-of-the-art results among published streaming systems, with competitive performance even at 0.32-second latency.Performance degrades as latency is reduced, but the paper describes the degradation as not severe.Source: §4.3 · p. 4
5. Conclusion
The paper proposes streaming Sortformer with AOSC, which dynamically adjusts each speaker's cache size to prioritize valuable speech frames. It reports state-of-the-art performance on DIHARD III and CALLHOME for up to four speakers, with future work aimed at expanding speaker capacity and integrating the diarizer into other speech-processing tasks.
5. Conclusion
- The proposed streaming Sortformer incorporates AOSC, which dynamically adjusts each speaker's cache size and focuses on valuable speech frames.The authors plan to extend the system to handle up to eight speakers.Source: §5 · p. 4
- The framework achieves state-of-the-art performance on DIHARD III and CALLHOME scenarios with up to four speakers.Source: §5 · p. 4
- Future work aims to integrate the diarizer into ASR, speech translation, and summarization, and to broaden its speaker capacity to eight.The paper names broadcasting and meeting transcription as potential real-world applications.Source: §5 · p. 4