ECAPA-TDNN Embeddings for Speaker Diarization

• Modern diarization systems use neural speaker embeddings with clustering, and x-vectors have become a fundamental component of state-of-the-art systems.Source: §1 · p. 1
• Speaker diarization remains one of speech processing’s most challenging tasks, despite recent progress and active research.Source: §1 · p. 1
• The paper is the first to use ECAPA-TDNN for diarization and trains it with contaminated versions of each signal concatenated in the same batch.The authors also describe this as an on-the-fly augmentation scheme combining different contamination techniques.Source: §1 · p. 1
• On the AMI meeting dataset, the proposed system reports highly competitive performance and overtakes recent speaker diarization approaches.Source: §1 · p. 1

2.1. Speaker embeddings

• ECAPA-TDNN adds channel- and context-dependent attention, Squeeze-Excitation and hierarchical Res2 blocks, and Multi-layer Feature Aggregation before statistics pooling.MFA concatenates the final frame-level feature map with intermediate maps from preceding layers.Source: §2.1 · p. 2
• AAM-softmax trains the network on speaker identities by directly optimizing cosine distance between speaker embeddings.The paper states this can avoid complex scoring backends such as PLDA.Source: §2.1 · p. 2

2.2. Data augmentation

• The augmentation scheme applies contamination on the fly and concatenates the original signal with its contaminated versions so each batch contains six views of the same sentence.The authors describe these views as providing a regularization effect that makes the model robust against signal variations.Source: §2.2 · p. 2

2.3. Spectral clustering

• Spectral clustering builds a cosine-similarity affinity matrix, prunes smaller values, estimates speaker count by the maximum eigengap, then clusters spectral embeddings with k-means.Source: §2.3 · p. 2

3.1. Datasets

• VoxCeleb1 and VoxCeleb2 train the ECAPA-TDNN model, while AMI meetings provide diarization data with the official Full ASR split and manual annotations as ground truth.The Dev and Eval sets exclude TNO meetings, and microphone-array channels are beamformed with BeamformIt.Source: §3.1 · pp. 2–3

3.2. Speaker embeddings

• The model is trained on 3-second random crops and achieves 0.69% EER and 0.0826 minDCF on the original cleaned VoxCeleb1 verification set.Source: §3.2 · p. 3

3.3. Diarization setup

• Diarization embeddings use 3-second sliding windows shifted by 1.5 seconds, with a maximum of 10 speakers; scoring uses oracle speech labels and no realignment post-processing.The evaluation also uses a 0.25-second forgiveness collar and ignores overlap regions.Source: §3.3 · p. 3

4.1. Baseline systems

• The comparison includes ClusterGAN and MCGAN systems using fused x-vector embeddings with spectral clustering, plus VBx using ResNet101 x-vectors and a Bayesian HMM.These approaches are described as competitive or state-of-the-art on AMI, making them strong baselines.Source: §4.1 · p. 3

4.2. Comparison with baseline systems

• 37.4% and 38.8% relative DER improvements over x-vector+ClusterGAN with SC on Eval, for oracle and estimated speaker counts, respectively.The system also improves DER by 34.3% (from 4.58% to 3.01%) over VBx with an unknown number of speakers.Source: Table 1 · p. 3, §4.2 · p. 3
• With k-means, ECAPA-TDNN improves Eval DER over x-vector+ClusterGAN by 42.9% (from 6.46% to 3.69%) and 40.9% (from 8.63% to 5.10%) for oracle and estimated speaker counts.Against x-vector+MCGAN, the reported improvements are 32.7% (from 5.48% to 3.69%) and 41.8% (from 8.76% to 5.10%), respectively.Source: §4.2 · p. 3

4.3. Comparison with x-vector embedding

• ECAPA-TDNN embeddings substantially outperform standard x-vector embeddings with SC on beamformed data, with the same trend on individual distant microphones, HeadsetMix, and LapelMix.The x-vector embeddings were trained using the same augmentation scheme described for the proposed system.Source: Table 2 · p. 4, §4.3 · p. 4

4.4. Ablation study on augmentation

• The proposed augmentation performs better than standard augmentation across audio streams, while no augmentation yields worse DER; the authors attribute improved robustness to its batch construction.Standard augmentation applies contamination to each training sentence; the proposed method uses a distinct batch-construction technique.Source: Table 3 · p. 4, §4.4 · p. 4

4.5. Distant and close talking microphones

• HeadsetMix and LapelMix results estimate the lowest achievable DERs because their signals are relatively clean; distant-microphone performance is comparable to beamformed audio, with limited degradation in close-talking conditions.Best reported DERs include 2.02% and 1.78% on HeadsetMix for Dev and Eval with oracle speaker counts, and 2.34% and 2.57% on LapelMix for Dev and Eval with unknown speaker counts.Source: Table 4 · p. 4, §4.5 · p. 4

5. Conclusion and Future Work

• The AMI system combines ECAPA-TDNN speaker embeddings with augmentation that gathers multiple augmented views of a signal in the same batch, and surpasses ClusterGAN, MCGAN, and VBx.Its diarization performance remains consistent across distant- and close-talking microphones.Source: §5 · p. 4
• Further work could explore resegmentation, VBx Bayesian HMM, iterative clustering with longer-segment embeddings in a second pass, and automated speaker-count estimation such as NME-SC.Source: §5 · p. 4