Source-linked AI summary
Beyond Speech: Dual-Domain SSL Fusion for Unified All-Type Audio Deepfake Detection
Cunhang Fan, Junqin Cao, Tian Gao, Zhipeng Xie, Jun Xue, Zhao Lv, Xin Fang
TL;DR
All-type audio deepfake detection requires one real/fake decision across heterogeneous audio when its type is unknown at inference. This paper fuses complementary EAT-large and XLS-R-300M representations through layer- and token-level aggregation, then uses attentive statistics pooling and binary classification. With conservative speech refinement, the submitted system achieves 95.58% Macro-F1 on AT-ADD Track 2 and ranks second in the challenge.
Problem
All-type detection must classify speech, environmental sound, singing voice, and music with one binary decision despite unknown audio type at inference.
Method
The method fuses complementary EAT-large and XLS-R-300M SSL features across layers and token streams into a shared authenticity space, then pools and classifies them.
Results
95.58% Macro-F1 on the AT-ADD Track 2 evaluation set ranked the submitted system second in the challenge.
Takeaways & Limitations
The results support dual-domain fusion with conservative refinement across heterogeneous audio while retaining a shared detector rather than type-dependent inference routes.
Takeaways & Limitations
The available XLS-R-only systems use speech-related subsets rather than all four audio types, so EAT-large’s per-type marginal contribution is not isolated.
Abstract
from arXiv · showhide
Unified all-type audio deepfake detection aims to determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice, or music. Existing speech-centric or type-dependent solutions are insufficient for this setting because the test-time audio type is unknown, while the required output is still a single binary decision. To address these issues, this paper proposes a dual-domain SSL fusion method that maps heterogeneous audio into a shared binary authenticity space. EAT-large and wav2vec 2.0 XLS-R-300M are used as complementary SSL feature sources, providing broad acoustic and event-level representations as well as waveform-level, vocal, and speech-sensitive representations. Layer-wise weighted fusion integrates multi-level artifacts from different transformer depths, while token-level fusion forms a unified feature pool without enforcing frame-level alignment between the two SSL streams. The fused tokens are summarized by multi-head attentive statistics pooling and classified with a binary MLP head. With conservative speech refinement applied on top of this unified core detector, the submitted system achieves 95.58% Macro-F1 on the AT-ADD Track 2 evaluation set and ranks second in the challenge.
1 Introduction
All-type audio deepfake detection must make one reliable real/fake decision across speech, environmental sound, singing voice, and music when the audio type is unknown. The paper addresses this setting with complementary SSL fusion and reports a challenge-leading result.
- Unknown audio type requires a single reliable binary real/fake decision across speech, environmental sound, singing voice, and music.
- All-type detection needs non-speech representations because its artifacts can involve acoustic events, background scenes, vocal generation, songs, and synthetic music.
- The proposed detector combines EAT-large and XLS-R-300M as complementary SSL sources in one shared binary authenticity space.EAT-large supplies broad acoustic and event-level cues, while XLS-R-300M supplies waveform-level, vocal, and speech-sensitive cues.
- Layer-wise weighted fusion and token-level fusion create a unified feature pool without requiring frame-level alignment between the two SSL streams.
- 95.58% Macro-F1 on the AT-ADD Track 2 evaluation set ranked the submitted system second in the challenge.
2 Method
The method builds a type-agnostic detector by fusing complementary EAT-large and XLS-R-300M representations across transformer layers and token streams. Attentive statistics pooling then summarizes the unified tokens for binary classification.
- Dual-domain front-ends: EAT-large processes time-frequency features for broad acoustic and event-level cues, while XLS-R-300M processes raw waveforms for vocal and speech-sensitive cues.
- Layer-wise fusion: Learnable layer-wise weighting combines hidden states from multiple transformer depths to retain local acoustic, phonetic or event-level, and semantic cues.
- Token-level fusion: The two streams’ layer-fused tokens are concatenated along the token dimension, avoiding assumptions of frame-level alignment.Their shared hidden dimension permits direct concatenation without stream-specific projection, cross-stream normalization, temporal resampling, or frontend-identity embedding.
- Projection and pooling: A shared SwiGLU projection processes the concatenated EAT and XLS-R token pool before attention aggregation.The gated nonlinear layer enhances useful responses and suppresses less informative activations.
- Projection and pooling: Multi-head attentive statistics pooling assigns temporal importance scores within feature heads and computes weighted means and standard deviations.Different heads can emphasize distinct sources, regions, or artifact patterns while contributing to one type-agnostic decision.
- Classification: The pooled representation is concatenated across heads and passed to an MLP that produces binary real/fake logits.
3 Experiments
The experiments evaluate unified all-type detection across four unknown-at-inference audio types using controlled splits, model variants, progress-set comparisons, and evaluation-set fusion analysis. Results support complementary SSL fusion, conservative speech refinement, and a shared authenticity representation.
- Experimental setup: AT-ADD Track 2 averages binary Macro-F1 across speech, environmental sound, singing voice, and music while requiring one type-agnostic real/fake decision.Audio type is unavailable during inference and is not itself a prediction target.
- Experimental setup: 237,850 officially released samples were merged into five stratified train/validation configurations, preserving audio type, label, and generator identity distributions.Each configuration uses 47,570 validation samples and 190,280 training samples, with seed 42 fixed.
- System comparisons: The model suite compares unified EAT and EAT–XLS-R detectors, auxiliary type supervision, and hard type-based routing under shared progress-set evaluation.Hard routing selects specialized binary models using an intermediate audio-type prediction, making its decision dependent on routing correctness.
- Evaluation results: 1.14 points was the final fusion gain over S2 on the evaluation set, improving speech, sound, singing, and music by 1.31, 1.09, 1.33, and 0.87 points, respectively.The improvement was distributed across all four diagnostic categories rather than concentrated in speech.
- Analysis: The unified EAT–XLS-R systems improved over the EAT-only baseline, while S5 and S2 achieved 94.45% and 94.55% Macro-F1, respectively, on the progress set.S5’s auxiliary type head supplied cues and complementary logits for final fusion despite not improving standalone performance.
4 Conclusion
The final detector preserves a unified, type-agnostic path while adding conditional speech refinement. It ranks second on AT-ADD Track 2, achieving 95.58% Macro-F1 and outperforming the baseline by 20.30 points.
- 4 Conclusion: The final submission combines EAT-large and XLS-R-300M through layer-wise and token-level fusion without frame alignment, retaining one all-type detection path.A conditional speech refiner is added on top, using joint auxiliary-type and frozen-ASR cues while the core systems remain active for every sample.
- 4 Conclusion: 95.58% Macro-F1 places the system second on the AT-ADD Track 2 evaluation set, 20.30 points above the baseline.The results support dual-domain fusion with conservative refinement across heterogeneous audio.