Source-linked AI summary
Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization
Yehoshua Dissen, Joseph Keshet, Eduard Golshtein
TL;DR
Speaker-embedding clustering and neural diarization offer complementary information, but their intermediate speaker representations are not directly compatible. TFAF transfers neural speaker structure as continuous affinity evidence into the embedding graph before global clustering, and improves over both constituent systems on AMI and CALLHOME.
Problem
Embedding-based and neural diarization provide complementary speaker information but use incompatible representations, limiting direct fusion of their affinities and speaker structure.
Source: §I · p. 1Method
TFAF conditions local neural representations on the neural speaker partition, builds a continuous affinity, and combines it with embedding-based acoustic affinity before one global clustering step.
Source: §I · p. 1Results
TFAF improves over both constituent systems on AMI and CALLHOME; on CALLHOME, its DER is 2.74 compared with 3.85 for Equal-w-MS-Clus.
Source: §V · p. 3Takeaways & Limitations
Ablations identify the neural speaker partition as the main source of transferable information, while preserving continuous acoustic affinities improves on hard partition fusion.
Source: §V · pp. 3–4The conclusion identifies extending TFAF to overlapping speech as future work.
Source: §VI · p. 4
Abstract
from arXiv · showhide
Speaker diarization systems based on speaker embeddings and neural diarization exploit complementary forms of speaker information, but their intermediate representations are not directly compatible. We introduce Training-Free Affinity Fusion (TFAF), which integrates the speaker structure inferred by a neural diarizer into an embedding-based diarization system. The neural speaker partition is used to condition local speaker representations, from which we construct a continuous affinity matrix and combine it with the embedding-based acoustic affinity before a single global clustering step. The method requires no additional training, shared embedding space, speaker-label alignment, or hard transfer of the neural diarizer's speaker count. Experiments on AMI and CALLHOME show consistent DER improvements over both constituent systems; on AMI, fusion also improves speaker-attributed transcription. Ablations show that the neural speaker partition accounts for most of the gain, while retaining the continuous embedding-based affinities provides additional benefit over hard partition fusion.
I. INTRODUCTION
Embedding-based clustering and neural diarization offer complementary speaker information but use incompatible representations. TFAF transfers neural speaker structure into the embedding affinity graph without training, label alignment, or hard adoption of the neural system’s speaker count.
I. INTRODUCTION
- Embedding clustering preserves recording-level speaker identity but trades temporal resolution against speaker-discriminative embeddings, while neural diarization uses sequence context for short turns, transitions, and overlap.Longer embedding windows strengthen speaker representations but blur brief changes; shorter windows localize turns better but weaken representations.Source: §I · p. 1
- Neural diarization contributes detailed local speaker information, whereas embedding clustering contributes recording-level discrimination and global consistency.Source: §I · p. 1
- The systems cannot generally fuse speaker labels or representations directly because embedding methods use continuous pairwise similarities while neural diarizers produce recording-internal assignments in a separate representation space.The missing interface is one that transfers neural speaker structure into an embedding affinity graph while retaining continuous acoustic information.Source: §I · p. 1
- TFAF constructs a continuous partition-conditioned affinity from neural local representations and adds it to the embedding affinity before final clustering.It requires no additional training, shared embedding space, speaker-label alignment, or hard transfer of the neural system’s speaker count or final partition.Source: §I · p. 1
II. RELATED WORK
Prior diarization work combines information at different representations and pipeline stages, but TFAF preserves the embedding system’s continuous acoustic affinities while adding neural speaker structure before final clustering.
II. RELATED WORK
- Embedding-based diarization spans x-vector representations, TitaNet encoders, multiscale methods,, and self-supervised representations learned from unlabeled speech,.Source: §II · pp. 1–2
- Neural diarization models temporal context for short turns, transitions, and overlap; hybrid systems combine neural local diarization with clustering or tracking for longer recordings and variable speaker counts,.Source: §II · p. 2
- Earlier combinations include lexical-acoustic affinity fusion, turn-constrained embedding diarization, aligned output fusion,, hard-partition co-association,, and calibrated neural probability fusion [21].Source: §II · p. 2
- Unlike output alignment or hard-partition fusion, TFAF combines continuous embedding affinities with neural speaker structure before clustering, without label alignment or hard transfer of the neural speaker count.Source: §II · p. 2
III. METHOD
TFAF combines multiscale embedding affinities with partition-conditioned neural representations on a shared temporal grid, then applies the embedding system’s clustering procedure once to the fused graph.
III. METHOD
- The embedding system computes per-scale affinities on base segments, and its embedding affinity is the sum of those affinities with equal weights.The number of acoustic scales is K.Source: §III · p. 2
- The neural diarizer supplies local representations, recording-level speaker assignments, and frame activity, without requiring its local representations to define a globally consistent speaker metric.Its speaker labels do not correspond to the embedding system’s clusters.Source: §III · p. 2
- For each local neural speaker, TFAF blends the normalized local representation with its normalized speaker centroid; α = 0.5 gives them equal vector weight.Source: §III · p. 2
- TFAF maps blended representations onto a 20 ms frame grid, averaging and normalizing contributions from single-speaker frames; uncovered cells contribute no neural evidence.Segments crossing a neural speaker boundary can contain contributions from both speakers rather than receiving a hard label.Source: §III · p. 2
- The system forms a continuous neural affinity from cosine similarities between covered segments, adds it to the embedding affinity, and runs the embedding clustering procedure once to determine the final partition and speaker count.The partition-only ablation replaces the continuous neural affinity with same-speaker evidence derived from frame assignments; the full system uses α = 0.5 and λ = 1 unless stated otherwise.Source: §III · p. 2
A. Systems
The evaluation combines the Equal-w-MS-Clus embedding system with a frozen DiariZen neural diarizer, mapping neural evidence onto the embedding system’s temporal grid without learned alignment.
A. Systems
- The embedding baseline is NeMo Equal-w-MS-Clus, using TitaNet-L speaker embeddings and NME-SC clustering.AMI uses six window durations from 3.0 to 0.5 s; CALLHOME uses five telephony durations from 1.5 to 0.5 s.Source: A. Systems · p. 3
- The neural system is the frozen public DiariZen wavlm-large-s80-md checkpoint with its default pipeline, processing recordings in 16 s chunks at 1.6 s steps and producing 20 ms activities for up to four chunk-local speakers.Its speaker-embedding stage extracts 256-dimensional local representations, which DiariZen clustering assigns to recording-level speakers.Source: A. Systems · p. 3
- Neural evidence is mapped onto a 0.5 s window, 0.25 s shift grid; cells average and normalize single-speaker-frame contributions, while frames with multiple active speakers are excluded.Source: A. Systems · p. 3
- The neural grid acts as an additional affinity scale, with each base segment mapped to its nearest cell; segments spanning a neural speaker change receive a normalized mixture rather than a hard label.No additional training or learned alignment is introduced.Source: A. Systems · p. 3
B. Datasets and scoring
The evaluation uses matched oracle speech coverage on AMI and CALLHOME, with speaker-attributed transcription and tcpWER scored alongside diarization. The scoring rules distinguish wrong-speaker words from errors that also include recognition.
B. Datasets and scoring
- AMI uses the official 16-meeting MixHeadset test split and only-words references; CALLHOME uses SRE-2000 disc 8 with the matched telephony configuration.Both main comparisons use oracle speech activity; AMI ignores overlap and applies a 0.25 s DER collar.Source: B. Datasets and scoring · p. 3
- Word err. counts reference words assigned to the wrong speaker after optimal speaker-label mapping, excluding overlap-ambiguous words.The recognized transcript is held fixed across diarization systems.Source: B. Datasets and scoring · p. 3
- tcpWER uses a 5 s collar and includes both recognition and speaker-attribution errors, unlike Word err.For oracle-SAD experiments, systems are adapted to the same speech coverage before scoring.Source: B. Datasets and scoring · p. 3
V. RESULTS
TFAF improves on both constituent systems on AMI and CALLHOME. Ablations indicate that the neural speaker partition supplies most of the gain, while comparisons and error analyses probe the value and limits of the transferred evidence.
V. RESULTS
- TFAF improves over both constituent systems on AMI and CALLHOME.The comparison is made under identical oracle speech coverage.Source: §V · p. 3
- On AMI, partition-only fusion yields 521 Word err. errors, while adding speaker-centroid information lowers this to 465; hard count transfer raises errors from 739 to 3,598.The ablation changes only the information transferred from the same DiariZen inference.Source: §V · p. 3
- On CALLHOME, the binary partition reduces DER from 3.85 to 2.90 and TFAF reaches 2.74; imposing DiariZen’s estimated count raises DER to 4.85.DiariZen estimates the exact speaker count on 83.6% of files, compared with 71.7% for Equal-w-MS-Clus.Source: §V · p. 3, Table III · p. 4
- On CALLHOME, λ values 0.5, 1, and 2 yield DERs of 3.04, 2.74, and 2.69, respectively; λ = 1 is within 0.05 DER points of the best tested value.Source: §V · p. 4
- At acoustic affinities of 0.20–0.25, same-speaker DiariZen assignments correspond to the same reference speaker with probability 0.902, versus 0.017 for different-speaker assignments.In the opposite direction, acoustic affinity distinguishes false splits from true different-speaker pairs with AUC 0.858.Source: §V · p. 4
- Across 432 synthetic splits, attribution-error increases correlate with rcut (Spearman ρ = 0.80); coherent balanced splits are more damaging than small fragments.DiariZen’s actual errors have pooled rcut 0.018 and mostly consist of small incoherent pieces, a regime found less damaging in the corruption experiment.Source: §V · p. 4
- Affinity-level fusion preserves the acoustic graph and outperforms hard co-association; three-system DOVER-Lap improves on Equal-w-MS-Clus on both corpora, with mixed AMI comparisons against TFAF.On AMI, DOVER-Lap has slightly lower collared DER, while TFAF has fewer wrong-speaker words and lower tcpWER; on CALLHOME, TFAF has lower DER.Source: Table IV · p. 4, §V · p. 4
- Approximately two thirds of attribution errors occur within 250 ms of reference speaker boundaries, which the standard DER collar excludes.Small changes in collared DER therefore need not reflect changes in speaker-attributed transcription quality.Source: §V · p. 4
VI. DISCUSSION AND CONCLUSION
TFAF combines neural and embedding-based diarization despite their incompatible speaker labels and representation spaces. It improves over both systems on AMI and CALLHOME, with ablations attributing most transferable information to the neural speaker partition.
VI. DISCUSSION AND CONCLUSION
- TFAF transfers recording-level neural speaker structure as soft affinity evidence while preserving the continuous acoustic graph for global clustering.The method provides a training-free interface between systems with otherwise incompatible speaker labels and representation spaces.Source: §VI · p. 4
- TFAF improves over both constituent systems on AMI and CALLHOME.Source: §VI · p. 4
- Ablations show that most transferable information comes from the neural speaker partition; future work will extend TFAF to overlapping speech.Source: §VI · p. 4