Hybrid Pruning: In-Situ Compression of Self-Supervised Speech Models for Speaker Verification and Anti-Spoofing

1. INTRODUCTION

• 94-million-parameter WavLM-Base and over-300-million-parameter WavLM-Large models pose deployment challenges in resource-constrained environments.The introduction names mobile devices and edge systems as examples.Source: §1 · p. 1
• Pruning during pre-training or after fine-tuning is disconnected from the final application, limiting guidance from end-task requirements.This makes it difficult to find a compressed architecture tailored to the downstream task.Source: §1 · p. 1
• Speaker verification and anti-spoofing provide a contrasting testbed: they rely on stable, long-term speaker characteristics and subtle, transient synthesis artifacts, respectively.The paper asks whether task, dataset size, and layer-wise representation changes yield distinct pruning patterns.Source: §1 · p. 1
• Hybrid Pruning integrates structured pruning into downstream fine-tuning, jointly optimizing task-specific classification loss and sparsity without separate distillation or multi-stage pipelines.The paper reports up to 70% sparsity with minimal performance loss on large datasets and improved low-resource results through mitigating overfitting.Source: §1 · p. 1, Figure 1 · p. 2

2. RELATED WORK

• Task-agnostic pruning integrates pruning into SSL pre-training, while task-specific approaches prune after task fine-tuning, often using multi-stage recovery and distillation.Hybrid Pruning instead integrates structured pruning directly into fine-tuning without a frozen teacher model.Source: §2 · p. 2

3. THE HYBRID PRUNING FRAMEWORK

• Hybrid Pruning jointly optimizes model architecture and weights in one stage, fine-tuning for a downstream task while pruning toward a target sparsity.It uses gradient-based L0 regularization adapted from.Source: §3 · p. 2

3.1. Dynamic Architectural Adaptation via Stochastic Gates

• Learnable stochastic gates control CNN kernels, multi-headed self-attention heads, and feed-forward neurons in an unfrozen pre-trained SSL model.The Hard Concrete distribution permits gates to equal zero with non-zero probability, allowing components to be turned off through gradient descent.Source: §3.1 · p. 2

3.2. Joint Optimization for Performance and Sparsity

• The joint objective minimizes loss over model and gate parameters while maximizing over sparsity-controlling parameters, trading task accuracy against model complexity.This optimization occurs within a single training loop.Source: §3.2 · pp. 2–3
• The task loss is AAM Softmax for speaker verification and binary cross-entropy for anti-spoofing, while an expected L0-based augmented-Lagrangian regularizer steers sparsity toward a predefined target.Learnable Lagrange multipliers are updated by gradient ascent to enforce the sparsity constraint.Source: §3.2 · p. 3
• After training, gates become deterministic binary values and zero-gated components are removed from the computational graph.The resulting smaller, faster model requires no special sparse-computation libraries for inference.Source: §3.2 · p. 3

4.1. Setup

• Experiments train SV models on VoxCeleb2 and evaluate them on Vox1-O/E/H and CNCeleb; anti-spoofing models are trained and tested on ASVspoof5 and SpoofCeleb.The evaluation uses Base and Large pretrained SSL models.Source: §4.1 · p. 3

4.2. Main Results on SV and Anti-Spoofing Benchmarks

• At 10% sparsity, HP alone beats the unpruned ASVspoof5 baseline, reaching 3.70% EER; it also outperforms DP-HUBERT and Structured Pruning at every sparsity.FLOPs are calculated for a 4-second audio input.Source: Table 1 · p. 3, §4.2 · p. 3
• At 60% sparsity, WavLM Base+ reduces parameters from 95.6M to 38.9M and yields Vox1-O/E/H EERs of 0.70%/0.78%/1.50%, with 2.2× CPU and 2.0× GPU speedups.The reported EERs are comparable to the dense model’s 0.70%/0.69%/1.40%.Source: Table 2 · p. 3, §4.2 · p. 3

4.3. Task- and Domain-Adaptive Architectures after Pruning

• At 50% sparsity, HP learns non-uniform patterns: anti-spoofing preserves lower MHSA, SV upper-middle layers, and CNCeleb-trained SV prunes upper layers more than VoxCeleb-trained SV.The paper reports that the learned sub-architectures adapt to both downstream task and data domain.Source: Figure 3 · p. 4, §4.3 · p. 4

4.4. Generalization Gains from Moderate Pruning

• On CNCeleb and SpoofCeleb, moderate sparsity outperforms the unpruned baseline in U-shaped EER curves, consistent with pruning acting as a regularizer.The authors suggest standard fine-tuning may lead to overfitting on the downstream set.Source: Figure 4 · p. 4, §4.4 · p. 4

4.5. Pruning Effectiveness Across Model Scales

• Pruned Base consistently outperforms pruned Large on Vox1-H; HP prunes Base by over 70% with negligible degradation versus unpruned Base.The paper argues that choosing a suitable foundation model and applying fine-grained pruning is more effective than simply compressing the largest available model.Source: Figure 5 · p. 4, §4.5 · p. 4

• Up to 70% parameter reduction came with negligible performance degradation on speaker verification and anti-spoofing benchmarks.Source: §5 · p. 4
• In low-resource scenarios, HP improved generalization by mitigating overfitting.Source: §5 · p. 4
• HP learned task-specific architectures, retaining different layers for speaker verification versus anti-spoofing.Source: §5 · p. 4
• Pruning a smaller base model can be more effective than compressing a larger one.Source: §5 · p. 4