Source-linked AI summary
SNAP: Speaker Nulling for Artifact Projection in Speech Deepfake Detection
Kyudan Jung, Jihwan Kim, Minwoo Lee, Soyoon Kim, Jeonghoon Kim, Jaegul Choo, Cheonbok Park
TL;DR
Deepfake detectors can over-rely on speaker identity, limiting generalization across speakers. SNAP removes speaker-dependent components through orthogonal projection, isolating synthesis artifacts and achieving state-of-the-art performance with robust generalization to unseen speakers and TTS models.
Problem
WavLM features cluster primarily by speaker identity rather than real-synthetic status, limiting synthetic-speech detection across unseen speakers.
Method
SNAP defines a speaker-dependent subspace and removes it from acoustic embeddings via orthogonal projection, producing residual features focused on synthesis artifacts.
Results
SNAP achieves state-of-the-art performance while generalizing well to unseen speakers and TTS models.
Takeaways & Limitations
Speaker nulling enables detection to focus on synthetic artifacts rather than speaker identities.
Takeaways & Limitations
SNAP assumes the feature space can be decomposed into speaker-dependent, artifact, and residual context subspaces.
Abstract
from arXiv · showhide
Recent advancements in text-to-speech technologies enable generating high-fidelity synthetic speech nearly indistinguishable from real human voices. While recent studies show the efficacy of self-supervised learning-based speech encoders for deepfake detection, these models struggle to generalize across unseen speakers. Our quantitative analysis suggests these encoder representations are substantially influenced by speaker information, causing detectors to exploit speaker-specific correlations rather than artifact-related cues. We call this phenomenon speaker entanglement. To mitigate this reliance, we introduce SNAP, a speaker-nulling framework. We estimate a speaker subspace and apply orthogonal projection to suppress speaker-dependent components, isolating synthesis artifacts within the residual features. By reducing speaker entanglement, SNAP encourages detectors to focus on artifact-related patterns, leading to state-of-the-art performance.
1 Introduction
High-fidelity speech generation increases deepfake misuse risks, while SSL-based detectors can overfit speaker identity instead of synthesis artifacts. SNAP addresses this speaker entanglement by nullifying speaker information through orthogonal projection, enabling state-of-the-art detection with a simple classifier and robust generalization.
- High-fidelity speech generation is virtually indistinguishable from authentic recordings, increasing the potential for misinformation through misuse.
- SSL speech encoders excel at deepfake detection but are strongly influenced by speaker identity, encouraging speaker-specific overfitting.The introduction highlights WavLM’s exceptional speaker identification capabilities and speaker entanglement in SSL representations.
- SNAP decomposes the feature space into speaker-dependent, context, and synthesis-artifact subspaces, then nullifies speaker information via orthogonal projection.The resulting residual features are designed to isolate synthesis artifacts without relying on complex deep learning detectors.
- A linear classifier trained on speaker-nulled residual features achieves state-of-the-art detection performance and robust generalization to unseen speakers and TTS models.The method uses logistic regression with only 2,049 parameters.
2 Speaker Entanglement Phenomenon
This section identifies speaker entanglement in WavLM features as a limitation for synthetic-speech detection and hypothesizes a decomposition into speaker-dependent, artifact, and residual context subspaces. It motivates nulling the speaker subspace to make artifact features more prominent.
- Speaker Entanglement: WavLM explicitly preserves speaker characteristics, but its strong embedding of speaker identity can hinder content or artifact analysis.WavLM achieves state-of-the-art performance in speaker verification, while the investigation of WavLM-Large for synthetic-versus-real speech detection reveals a critical limitation.
- Subspace Decomposition: The high-dimensional WavLM feature space H is hypothesized to decompose into speaker-dependent S, speaker-independent artifact A, and residual context C subspaces.This decomposition formalizes the observed speaker entanglement phenomenon.
- Speaker Nulling: The proposed method nulls information in the speaker-dependent subspace S to make artifact features A more prominent for deepfake detection.The method is designed to enhance deepfake detection performance by suppressing speaker-related information.
3 Methodology
SNAP builds speaker-agnostic speech representations by extracting multi-level WavLM-Large features, estimating a speaker subspace, and projecting embeddings onto its orthogonal complement. The resulting residual features are classified with logistic regression to emphasize synthesis artifacts while reducing speaker information.
- SNAP framework: SNAP removes speaker-dependent information from acoustic embeddings through orthogonal projection, directing classification toward synthesis artifacts.The framework targets speaker entanglement by suppressing speaker-dependent components while retaining artifact-related information.
- Feature extraction: WavLM-Large provides hidden states from the 8th and 22nd transformer layers, which are concatenated and mean-pooled into an utterance-level feature.The two layers capture complementary information ranging from low-level acoustic details to high-level semantic content.
- Speaker subspace estimation: SNAP estimates speaker variation from normalized speaker centroids, applies PCA, and uses the top-K eigenvectors to define the speaker subspace.The hyperparameter K controls the dimensionality of the estimated speaker subspace.
- Speaker nulling: The projection onto the orthogonal complement produces residual features that retain minimal speaker identity information while preserving synthesis artifacts and linguistic content.The residual is obtained by subtracting the speaker-subspace component using the orthogonal projection matrix.
- Classification and inference: The projected features are classified with logistic regression optimized using binary cross-entropy, and inference reuses the learned projection matrix and classifier parameters.The paper presents the simple classifier as evidence that the projected features capture discriminative artifact information independently of speaker characteristics.
4 Experiments and Results
Experiments evaluate SNAP with WavLM-Large features projected onto a speaker-nullified subspace before logistic-regression classification. Results show improved detection, reduced speaker clustering, stronger real-versus-spoof separation, and robust generalization across unseen speakers, attacks, channels, and TTS domains.
- Main detection results: 2,049 parameters suffice for the classifier after speaker information is neutralized, highlighting SNAP’s efficiency.The passage attributes the performance boost to feature-level speaker neutralization followed by a simple classifier.
- Speaker entanglement analysis: Speaker-identity silhouette scores decrease from 0.026 to −0.002, while real-versus-spoof silhouette scores rise from 0.118 to 0.181 with SNAP.These changes indicate reduced speaker clustering and enhanced discriminability between bona fide and spoof speech.
- Out-of-domain evaluation: SNAP achieves flawless detection for in-domain evaluation with unseen speakers and consistently low EER across both CosyVoice2-to-F5-TTS and F5-TTS-to-CosyVoice2 directions.The supplied passages state that SNAP significantly outperforms all baselines in these cross-domain evaluations, but do not provide the individual EER values.
- Out-of-domain evaluation: Zero errors in the mixed-domain environment show that SNAP mitigates cross-domain vulnerabilities when exposed to diverse synthesis artifacts.Figure 3 additionally compares SNAP with a WavLM baseline as the number of training speakers increases; the baseline exhibits EER fluctuations or degradation.
5 Conclusion
The paper addresses speaker entanglement in SSL-based deepfake detection with SNAP, which uses orthogonal projection to isolate synthetic artifacts from speaker identities. The approach achieves state-of-the-art performance while generalizing to unseen speakers and TTS models, with stable convergence as training speaker populations increase.
- Method: SNAP addresses speaker entanglement in SSL-based deepfake detection through speaker nulling via orthogonal projection.The method isolates synthetic artifacts from speaker identities by projecting away speaker-dependent components.
- Generalization: SNAP achieves state-of-the-art performance while generalizing well to unseen speakers.
- Generalization: SNAP also demonstrates robust generalization to unseen TTS models.
- Training behavior: Training shows stable convergence as the speaker population increases.