Source-linked AI summary
S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0
Jincheng Wang, Li Li, Weixin Zhu, Yannan Wang, Tao Yu, Juan Liu, Ming Li
TL;DR
Many diarization methods are primarily designed for offline or online use, while S2SND-v2 seeks to improve acoustic representations and their use in a unified framework. It combines a speaker-adapted w2v-BERT 2.0 front end with overlap-aware masking and late-speaker training. Across eight test sets, the pruned system achieves average DERs of 14.52% online and 12.03% offline.
Problem
Many existing diarization methods are designed primarily for either offline or online use, and S2SND’s acoustic representations and back-end use of speaker information remain improvement targets.
Source: §1 · p. 1Method
Results
14.52% online and 12.03% offline average DER across eight test sets; the pruned system outperforms S2SND in both modes and has the lowest reported offline average DER among listed systems.
Source: §4 · p. 4Takeaways & Limitations
Structured pruning reduces w2v-BERT encoder parameters by approximately 85% while preserving most of the full-model gains.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper presents S2SND-v2, a unified online and offline speaker diarization framework built on S2SND that improves acoustic feature extraction and the use of the resulting representations. For acoustic feature extraction, we introduce a speaker-adapted w2v-BERT 2.0 front end and compare full and pruned construction routes. To better use these features, overlap-aware activity-query masking improves speaker embedding extraction under overlap, while late-speaker training aligns unknown-speaker training with online inference conditions. Combining these improvements, the pruned system achieves competitive average diarization error rates of 14.52% online and 12.03% offline across eight test sets.
1. INTRODUCTION
Existing diarization methods are often designed primarily for offline or online use, motivating a unified framework. S2SND-v2 strengthens S2SND’s acoustic representations and improves how its back end extracts and uses speaker information.
1. INTRODUCTION
- Existing diarization methods are often designed primarily for either offline or online use, motivating a unified framework that supports both modes.S2SND couples detection and representation decoders, but its performance depends on acoustic representations and the back end’s use of speaker information.Source: §1 · p. 1
- Overlapping frames can remain in activity queries even when accepted embeddings are selected by non-overlapping speech duration, potentially degrading embedding extraction.This motivates overlap-aware masking of the queries used by the representation decoder.Source: §1 · p. 1
- S2SND-v2 is evaluated on eight test sets.Source: §1 · p. 1
- Speaker-adapted w2v-BERT 2.0 is integrated into S2SND, with full and pruned front-end construction routes compared.The front end is motivated by the need for stronger speaker representations that preserve frame-level temporal resolution for diarization.Source: §1 · p. 1
- Late-speaker training combines late-onset mixture generation with onset-aware masked-speaker selection to increase exposure to unknown speakers with late activity onsets.Source: §1 · p. 1
2. METHODS
S2SND-v2 retains S2SND’s coupled-decoder framework and refines its front end, query construction, and training for unified online and offline diarization.
2. METHODS
- S2SND-v2 retains S2SND’s coupled decoders and unified online and offline inference while adding a speaker-adapted w2v-BERT 2.0 front end and back-end refinements.Source: §2 · p. 2
2.1. Speaker-adapted w2v-BERT 2.0 Front End
- The w2v-BERT 2.0 front end uses layer adapters and multi-scale feature aggregation to produce frame-level features, with full and pruned construction routes feeding the same S2SND back end.The prune-after-diarization route distills from a diarization-adapted teacher; structured pruning reduces the encoder from 580M to 91M parameters, excluding adapters and S2SND modules.Source: §2.1 · p. 2
2.2. Overlap-aware Activity-query Masking
- Activity-query values are zeroed at frames with two or more active speakers before embedding extraction, using reference activities in training and predicted activities in inference.Embeddings with less than 1.28s of cumulative non-overlapping target-speaker speech in a block are assigned to reserved class 0 during training and not accepted into the inference speaker matrix.Source: §2.2 · pp. 2–3
2.3. Late-speaker Training
- Late-speaker training selects speakers first active in the latter half of a window with probability 0.7 and uses late-onset mixtures to better match online unknown-speaker conditions.For simulated mixtures with at least two speakers, one speaker’s onset is placed uniformly between 50% and 80% of mixture duration with probability 0.7; inference is unchanged.Source: §2.3 · p. 3
3. EXPERIMENTAL SETUP AND RESULTS
S2SND-v2 combines a speaker-adapted w2v-BERT front end with back-end refinements and is evaluated across eight diarization test sets. The pruned system improves on the retrained S2SND baseline in both inference modes.
3.1. Datasets
- The eight-set evaluation uses DER without oracle VAD and with a 0s collar; the reported average is an unweighted mean.Real-data adaptation and evaluation cover AMI-SDM, AISHELL-4, AliMeeting, NOTSOFAR-1, MSDWild, DIHARD III, MagicData-RAMC, and VoxConverse.Source: §3.1 · p. 3
3.2. Experimental Setup
- Training uses simulated mixtures followed by two mixed-data stages at a 4:1 simulated-to-real ratio, with 8s windows and specified online and offline decoding settings.The online setup uses an 8s sliding window, a 0.64s step, and 0.16s look-ahead; offline inference re-decodes using accumulated speaker embeddings.Source: §3.2 · p. 3
3.3. Overall Results
- Pruning w2v-BERT from 580M to 91M parameters reduces per-window computation from 475.37G to 75.63G FLOPs, with prune-before and prune-after DERs of 15.26%/12.70% and 15.30%/12.61%.The full model reaches 14.74% online and 12.15% offline average DER; neither pruning order consistently outperforms the other across datasets or modes.Source: §3.3 · p. 3, Table 1 · p. 4
3.4. Comparison with Existing Systems
- 14.52% online average DER and 12.03% offline average DER improve on the retrained S2SND baseline; offline DER is 12.03% versus DiariZen-Large-s80-v2’s 12.67%.S2SND-v2 improves on all eight test sets in both modes; it has the lowest DER on six datasets, while DSE-CBM performs best on VoxConverse.Source: Table 2 · p. 4, §3.4 · p. 4
3.3. Overall Results
- Adding overlap-aware activity-query masking and duration-conditioned identity supervision reduces average DER from 15.26% to 14.75% online and from 12.70% to 12.23% offline.Late-speaker training further reduces average DER to 14.52% online and 12.03% offline, improving five of eight test sets in each mode; the first refinement slightly degrades a few sets.Source: §3.3 · p. 4, Table 1 · p. 4
4. CONCLUSION
S2SND-v2 combines a speaker-adapted w2v-BERT 2.0 front end with query-construction and training refinements. Its pruned system achieves competitive online and lowest reported offline average DER among the listed systems.
4. CONCLUSION
- 14.52% online and 12.03% offline average DER across eight test sets are achieved by the pruned system, which outperforms S2SND in both modes.The paper reports competitive online performance and the lowest reported offline average DER among the listed systems.Source: §4 · p. 4
- Structured pruning reduces w2v-BERT encoder parameters by approximately 85% while preserving most of the full-model gains.Source: §4 · p. 4
- The system combines a speaker-adapted w2v-BERT 2.0 front end with overlap-aware activity-query masking and late-speaker training.Source: §4 · p. 4
5. GENERATIVE AI USE DISCLOSURE
The disclosure limits LLM use to language editing and states that authors developed and verified the scientific content.
5. GENERATIVE AI USE DISCLOSURE
- LLMs were used exclusively for language editing, including rephrasing and grammatical refinement.Source: §5 · p. 5
- LLMs were not involved in idea development, methodology design, experiments, data analysis, or interpretation of results.Source: §5 · p. 5
- The authors state that they developed and verified all scientific content.Source: §5 · p. 5