Source-linked AI summary
Task-Aware Joint Pruning and Distillation for Efficient Audio Deepfake Detection
Miao He, Peng Cheng, Zhongjie Ba, Qing Wen, Li Lu, Xin Yang, Kui Ren
TL;DR
Deepfake detectors based on SSL are computationally demanding, while existing compression approaches are not tailored to deepfake detection. The paper combines cross-domain distillation with movement-guided structured pruning; at 31.9M parameters and 6.3× FLOPs reduction, it reports a 1.30% average performance drop across datasets versus the uncompressed baseline.
Problem
SSL-based deepfake detectors achieve state-of-the-art performance but their computational demands hinder deployment on resource-constrained devices, while existing compression methods are poorly suited to the task.
Source: §1 · p. 1Method
The framework combines cross-domain knowledge distillation from unlabeled in-domain and out-of-domain samples with movement-guided structured pruning to preserve forgery-discriminative structures.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2Results
1.30% average performance drop across multiple datasets, with 31.9M parameters and 6.3× FLOPs reduction, is reported versus the uncompressed baseline.
Source: §1 · p. 1Takeaways & Limitations
Abstract
from arXiv · showhide
Advances in speech synthesis have made deepfake speeches increasingly convincing, posing growing threats to security. While self-supervised learning (SSL) based detectors achieve state-of-the-art performance, their computational demands (typically 300M+ parameters) prevent deployment on resource-constrained devices. Existing compression methods, designed mainly for content-centric tasks, struggle to maintain competitive performance when directly adapted to deepfake detection. We propose a Task-Aware Joint Pruning and Distillation framework that combines cross-domain knowledge distillation with movement-guided structured pruning to transfer forgery-discriminative knowledge and preserve critical structures under aggressive compression. Our framework reduces the model to 31.9M parameters with 6.3$\times$ FLOPs reduction, with an average performance drop of only 1.30\% across multiple datasets compared to the uncompressed baseline, demonstrating strong potential for on-device deployment.
1. Introduction
Convincing synthesized speech creates security threats, while resource-intensive SSL detectors and task-mismatched compression methods limit reliable on-device deepfake detection. The paper proposes joint pruning and distillation to address these constraints while retaining competitive performance.
1. Introduction
- Deepfake speech can deceive people and Automatic Speaker Verification systems, and reported voice-cloning incidents include telecommunications fraud and identity theft.The passage describes high-fidelity synthesized speech as perceptually indistinguishable from bona fide audio.Source: §1 · p. 1
- SSL-based detectors typically have 300M+ parameters; their immense computational complexity precludes deployment on resource-constrained devices and necessitates cloud-based inference, introducing network-dependency and privacy bottlenecks.Cloud inference requires stable connectivity, while uploading user audio to cloud servers raises privacy concerns.Source: §1 · p. 1
- Existing compression methods mainly target content-centric tasks, and task-specific loss or in-domain distillation alone may inadequately transfer discriminative knowledge and preserve critical structures.The authors report severe degradation above 75% sparsity, especially out of domain.Source: §1 · p. 1
- 1.30% average performance drop across multiple datasets, with 10× parameter reduction and 6.3× FLOPs reduction, is reported versus the uncompressed baseline.The compressed model has 31.9M parameters.Source: §1 · p. 1
2. Background and Related Work
The background describes SSL as effective for deepfake detection but computationally demanding, and reviews distillation and structured pruning as compression approaches with limitations for this task.
2. Background and Related Work
- Speech SSL frontends use large-scale pretraining to learn rich representations and are effective for deepfake detection, but typically require 300M+ parameters and 130+ GFLOPs.The passage identifies CNNs and Transformers as typical frontend components.Source: §2 · p. 1
- Figure 1 presents the framework as task-specific preparation, joint pruning and distillation, and backend optimization.The compressed model is described as enabling efficient on-device deepfake detection.Source: Figure 1 · p. 2
- Knowledge distillation transfers knowledge to student models via layer-wise distillation, but requires manual architecture design.Prior deepfake-detection distillation works optimize accuracy without compressing for on-device deployment.Source: §2 · p. 2
- Structured pruning removes redundant units through task-agnostic or task-specific approaches [21], but can degrade performance severely at high compression ratios.The framework is designed to address these limitations.Source: §2 · p. 2
3. Methodology
The framework combines movement-guided structured pruning with cross-domain distillation to preserve forgery-discriminative structures and representations. It jointly optimizes these objectives toward a target compression ratio.
3. Methodology
- The three-stage framework prepares a task-specific model and movement scores, jointly prunes and distills, then fine-tunes the pruned model with its backend classifier.Movement scores cover attention heads, FFN intermediates, and CNN channels.Source: §3 · p. 2
3.1. Cross-Domain Knowledge Distillation
- Cross-domain distillation transfers teacher representations across unlabeled in-domain and out-of-domain samples to support generalization under compression.The method uses codec variations and environmental distortions, with ASVspoof2021 LA, 21 DF, and In-the-Wild as out-of-domain datasets; CKA guides representative-layer selection, such as Lset = {5, 14, 24}.Source: §3.1 · p. 2
3.2. Movement-Guided Structured Pruning
- Movement-guided pruning uses fine-tuning movement scores as importance weights, encouraging removal of low-movement redundant units while retaining structures essential for deepfake detection.The method assigns inverse penalty weights to structural units based on their movement scores.Source: §3.2 · p. 2
3.3. Joint Optimization Objective
- Joint optimization combines distillation and a movement penalty while constraining the retained structure to a target compression ratio.Hard Concrete reparameterizes the binary mask to make the L0 norm differentiable with respect to learnable parameters; an Augmented Lagrangian enforces target sparsity.Source: §3.3 · p. 2
- Table 1 compares baseline EER (%) across 60%, 75%, and 90% sparsity, while Table 2 reports EER (%) for an ablation study at 75% sparsity.The tables identify the comparison scope but do not provide cell values in these passages.Source: Table 1 · p. 3, Table 2 · p. 3
- Across ASVspoof2019 LA Eval, ASVspoof2021 DF, and ASVspoof5, the method consistently outperforms Finetune-Pruning as sparsity increases.Figure 2 encodes the method as a red line and Finetune-Pruning as a blue dashed line.Source: Figure 2 · p. 3
4. Experiments
Experiments compare compressed XLS-R against pruning baselines and examine results, component ablations, and pruning patterns. At 90% sparsity, the model retains competitive performance, while ablations show different contributions from cross-domain distillation and movement-guided pruning.
4.1. Experimental Setup
- Evaluation uses ASVspoof2019 LA training for fine-tuning, unlabeled datasets for distillation, and in-domain, cross-domain, and unseen benchmarks; three pruning baselines are reimplemented under a unified framework.The reported detection metric is EER, alongside model parameters and FLOPs.Source: §4.1 · p. 3
4.2. Main Results
- At 90% sparsity, average EER increase is only 1.30% across multiple datasets, with 31.9M parameters and 6.3× FLOPs reduction versus the 317M/146.3G baseline.The comparison also reports 10× parameter reduction.Source: §4.2 · pp. 3–4
4.3. Ablation Studies
- Removing cross-domain distillation raises EER on In-the-Wild from 8.43% to 25.84%, ASVspoof5 from 17.75% to 27.48%, and ASVspoof2021 LA from 3.03% to 10.14%.The authors report only a minimal in-domain 19LA Eval change, from 0.22% to 0.39%.Source: §4.3 · p. 4
- Removing movement-guided pruning changes ASVspoof5 EER from 17.75% to 19.95% and FoR from 10.51% to 13.56%, but changes 21DF from 2.76% to 3.13%.The authors attribute its value in harder scenarios to preserving discriminative structures under compression.Source: §4.3 · p. 4
4.4. Further Analysis
- At 90% sparsity, most CNN channels remain, attention heads are retained in layers 1–8 and 20–24, and deeper FFN dimensions shrink to near zero.The reported original dimensions are 512 CNN channels, 16 attention heads, and 4096 FFN intermediates.Source: §4.4 · p. 4, Figure 4 · p. 4
- CKA similarity analysis reveals block-like structures; additional experiments show layer choices within the same block (e.g., 5 vs 8, 10 vs 14) yield comparable performance, suggesting representative sampling matters more than exact indices.Source: §4.4 · p. 4
5. Conclusion
The paper proposes joint pruning and distillation for compressing SSL-based deepfake detectors. The resulting model reduces computation while maintaining competitive performance across diverse scenarios.
5. Conclusion
- The framework combines cross-domain knowledge distillation and movement-guided structured pruning to preserve forgery-discriminative structure and generalization under compression.The model reaches 31.9M parameters and 6.3× FLOPs reduction while maintaining competitive performance.Source: §5 · p. 4