Source-linked AI summary
CRAF: Cross-View Residual-Aware Fusion for Deepfake Speech Detection
Minh-Xuan Phan, Khalid Zaman, Candy Olivia Mawalim, Masashi Unoki
TL;DR
Deepfake detectors face persistent difficulty generalizing to unseen spoofing attacks, and direct SSL–ALLM fusion does not distinguish shared from complementary information. CRAF uses ALLM-guided learning and an ALLM-conditioned reference to refine and integrate residual SSL information; on ASVspoof 5, its Kimi-Audio configuration achieves an EER of 5.96% and a minDCF of 0.1192.
Problem
Deepfake detectors struggle to generalize to unseen synthesis methods, recording conditions, and datasets, while direct SSL–ALLM fusion does not separate shared from complementary information.
Source: §1 · p. 1Method
CRAF enriches SSL representations with ALLM guidance, models the remaining complementary SSL information as a residual, and selectively integrates it through SSL-primary fusion.
Source: §1 · pp. 1–2, §2 · p. 2Results
On ASVspoof 5, CRAF with Kimi-Audio achieves an evaluation EER of 5.96% and a minDCF of 0.1192, improving over the Kimi-Audio and XLS-R baselines.
Source: §3.3 · p. 3Takeaways & Limitations
Experiments show improved performance over individual encoders and direct fusion for robust detection of diverse and unseen spoofing attacks.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Recent advances in speech synthesis and voice conversion have made deepfake speech increasingly realistic, making generalization to unseen spoofing attacks a critical challenge. Pretrained speech and audio models offer a promising direction for improving robustness to such unseen attacks. Self-supervised learning (SSL) models capture fine-grained, low-level acoustic characteristics, whereas Auditory Large Language Models (ALLMs) provide higher-level contextual representations. These complementary views can provide useful cues for improving generalization to unseen attacks. However, direct fusion does not explicitly disentangle information shared across the two views from view-specific complementary information, limiting effective cross-view integration. To address this, we propose CRAF, a cross-view residual-aware fusion framework that uses ALLM-guided cross-view attention to enrich SSL representations and adopts ALLM as a high-level reference to separate ALLM-explainable information from complementary SSL residual information. The residual is selectively refined through adaptive gating and integrated through SSL-primary fusion. Experiments on ASVspoof 5 show that CRAF with Kimi-Audio achieves an EER of 5.96% and a minDCF of 0.1192, demonstrating robustness to unseen spoofing attacks.
1. INTRODUCTION
Deepfake detectors still struggle to generalize to unseen synthesis methods, recording conditions, and datasets. CRAF addresses this by using ALLM guidance to enrich SSL representations while separating and selectively integrating complementary SSL information.
1. INTRODUCTION
- Generalization to unseen synthesis methods, recording conditions, and datasets remains challenging for deepfake detectors.This motivates representations that capture transferable spoofing characteristics rather than attack- or dataset-specific patterns.Source: §1 · p. 1
- SSL captures fine-grained acoustic characteristics, whereas ALLM provides higher-level semantic and contextual representations.The two views therefore offer complementary cues for spoof detection.Source: §1 · p. 1
- Directly fusing SSL and ALLM treats the views uniformly, without separating shared information from complementary SSL-specific characteristics.This can introduce redundancy and reduce emphasis on fine-grained SSL information weakly represented by the ALLM.Source: §1 · p. 1
- CRAF uses ALLM-guided learning to enrich SSL, estimates the SSL information recoverable under ALLM guidance, and treats the remainder as a selectively refined cross-view residual.SSL remains the primary representation, and the framework selectively integrates guided and residual information while retaining the original SSL stream.Source: §1 · p. 1
- CRAF's contributions are an asymmetric SSL–ALLM formulation, guided cross-view residual learning, and SSL-primary adaptive fusion.The formulation retains SSL as the primary fine-grained representation while using ALLM for high-level guidance.Source: §1 · pp. 1–2
2. PROPOSED METHOD
CRAF keeps SSL as the primary stream, adds ALLM context through cross-view attention, and separates ALLM-explainable content from complementary SSL residuals. It selectively refines and fuses these representations while preserving the original SSL path.
- Frozen SSL and ALLM encoders produce complementary representations, projected by trainable layers into a shared 256-dimensional space.Source: §2 · p. 2
- ALLM-guided cross-view attention uses SSL as queries and ALLM as keys and values, injecting context while retaining SSL temporal resolution.Source: §2 · p. 2
- An independently parameterized ALLM-conditioned reference estimates SSL information explained by ALLM; the resulting residual isolates complementary SSL information.Source: §2 · p. 2
- A feature-wise gate conditioned on the original SSL representation, reference, and residual selectively refines residual information.Source: §2 · p. 2
- SSL-primary adaptive fusion combines original S, guided Sg, and refined R while preserving S and injecting contextual and complementary information.Source: §2 · pp. 2–3
- The AASIST-based classifier predicts bonafide or deepfake posteriors; weighted cross-entropy with label smoothing trains the modules while pretrained encoders remain frozen.Source: §2 · p. 3
3. EXPERIMENTS
Experiments on ASVspoof 5 evaluate CRAF across ALLM encoders, component ablations, and spoofing attacks. CRAF (Kimi-Audio) achieves the best reported evaluation results, while attack-wise findings show both broader gains over the SSL-only baseline and attack-dependent component behavior.
3.1. Dataset and evaluation metics
- ASVspoof 5 is the evaluation dataset, and the study reports EER and minDCF.Source: §3.1 · p. 3
3.2. Experimental Setup
- The setup uses frozen XLS-R 300M and Step-Audio-2, Qwen2-Audio-7B, or Kimi-Audio-7B encoders with an AASIST-style backend.Audio is resampled to 16 kHz; models use weighted cross-entropy, and the best development checkpoint is evaluated.Source: §3.2 · p. 3, §3.3 · p. 3
3.3. Results
- 5.96% EER and 0.1192 minDCF: CRAF (Kimi-Audio) achieves the best performance among evaluated configurations, improving over both Kimi-Audio and XLS-R baselines.CRAF also achieves the best evaluation EER among the recent single-system methods considered, without model- or score-level ensembling.Source: Table 2 · p. 3, §3.3 · p. 3
3.5. Attack-wise Analysis
- Attack-wise gains are broader over XLS-R+AASIST than Kimi-Audio+AASIST, while component rankings vary across attacks.CRAF reduces EER on 13 of 16 attacks versus XLS-R+AASIST, and on 7 of 16 versus Kimi-Audio+AASIST; A28 EERs range from 21.11% to 38.90%.Source: Figure 2 · p. 4, §3.5 · p. 4
3.4. Ablation Studies
- Component ablations show that cross-view attention, residual learning, and SSL-primary fusion contribute, with full Kimi-Audio CRAF outperforming individual component variants.Removing residual gating raises EER to 6.69% and minDCF to 0.1335.Source: Table 3 · p. 4, §3.4 · p. 4
4. CONCLUSION
The conclusion presents CRAF as a framework that jointly uses complementary SSL and ALLM representations for deepfake speech detection. ASVspoof 5 experiments show improved performance over individual encoders and direct fusion, while broader dataset generalization remains future work.
4. CONCLUSION
- CRAF jointly exploits complementary SSL and ALLM representations for deepfake speech detection.Source: §4 · p. 4
- ASVspoof 5 experiments show improved performance over individual encoders and direct fusion for diverse and unseen spoofing attacks.Source: §4 · p. 4
- Future work will investigate improved cross-view interaction and generalization across different deepfake datasets.Source: §4 · p. 4