Source-linked AI summary
What Survives the Codec Shift: Pooled No-Vocals Residuals for Speech Deepfake Detection
Jiajun Xu, Menglu Li, Xiao-Ping Zhang
TL;DR
Speech deepfake detectors face a difficult generalization setting as synthesis shifts from vocoders to neural codecs, and it remains unclear which acoustic representations retain discriminative information. The paper compares representations and proposes MN-P, combining token-level XLS-R with an utterance-level pooled no-vocals residual through adaptive gating. MN-P reduces EER against the best retrained system, with larger relative reduction on codec-unseen samples and gains across detector backends.
Problem
It remains unclear which acoustic representations retain discriminative information when speech synthesis shifts from vocoders to neural codecs, especially for detectors relying on speech-oriented evidence.
Source: §1 · p. 1Method
The paper compares 12 acoustic representations with a shared low-capacity linear classifier, then proposes MN-P to fuse token-level XLS-R features and an utterance-level pooled no-vocals residual through adaptive gating.
Source: §1 · p. 1Results
MN-P-AASIST reduces the strongest retrained system's EER by 54.2% and CU EER by 60.9%, with gains retained across detector backends.
Source: §6.1 · p. 4, §7 · p. 4Takeaways & Limitations
Pooled no-vocals residual statistics provide complementary evidence to speech representations and can substantially improve generalization under changes in the generation mechanism.
Source: §7 · p. 4
Abstract
from arXiv · showhide
The transition from vocoder-based to neural-codec speech synthesis makes generalization more difficult for speech deepfake detectors, particularly those relying on speech-oriented representations. It remains unclear which acoustic representations retain discriminative information when the generation mechanism changes. We therefore compare 12 acoustic representations using a shared low-capacity linear classifier to identify effective evidence under codec shift. The analysis shows that hierarchical XLS-R leads on the pooled test set, while pooled no-vocals residual statistics perform best on the unseen-codec condition, revealing complementary behavior across generation conditions. Building on this finding, we propose MN-P, a dual-view detector that integrates an utterance-level pooled no-vocals representation with token-level XLS-R features through adaptive gating. The proposed MN-P reduces EER by 54.2% overall and by 60.9% on the codec-unseen condition relative to the best-performing retrained state-of-the-art system, with consistent gains across different detector backends. These results indicate that pooled no-vocals residual statistics provide effective complementary evidence for cross-generation speech deepfake detection.
1. INTRODUCTION
Neural-codec synthesis creates a difficult generalization setting for detectors that rely on speech-oriented evidence, and it is unclear which representations remain discriminative across generation shifts. The paper compares acoustic representations and introduces MN-P, pairing token-level XLS-R features with an utterance-level pooled no-vocals residual through adaptive gating.
1. INTRODUCTION
- Neural-codec synthesis produces high-fidelity outputs through a substantially different mechanism, making unseen codec-based attacks difficult to detect reliably.Speech deepfake detection initially focused mainly on vocoder-based text-to-speech and voice-conversion systems.Source: §1 · p. 1
- As codec-based synthesis narrows the perceptual gap to real speech, evidence from speech-bearing signals becomes harder to separate, motivating complementary evidence beyond dominant speech content.Existing evidence includes handcrafted spectral and phase cues and pretrained self-supervised representations.Source: §1 · p. 1
- Hierarchical XLS-R provides the strongest overall discrimination, while pooled no-vocals residuals are most effective on unseen codecs and carry less linearly decodable source identity.The complementary behavior motivates a dual-view detector rather than further strengthening the speech representation alone.Source: §1 · p. 1
- MN-P combines a MoEF-based XLS-R speech branch with an utterance-level pooled no-vocals view through adaptive gating.The residual vector complements token-level speech features without requiring temporal alignment between the views.Source: §1 · p. 1
2. RELATED WORK
Prior speech-centric detectors leave open whether speech-oriented evidence remains sufficient when synthesis mechanisms change, while complementary non-verbal cues have not been explicitly assessed for the vocoder-to-neural-codec shift. The paper identifies pooled no-vocals residuals as complementary to hierarchical XLS-R and builds MN-P around their integration.
2. RELATED WORK
- Most systems derive evidence primarily from speech-oriented representations, leaving their sufficiency under substantial synthesis shifts unresolved.Earlier approaches include handcrafted spectral and phase cues, raw-waveform, graph-based, and self-supervised detectors, and intermediate self-supervised layers.Source: §2 · p. 2
- Silence, breathing, background, and environmental cues have been explored, but their robustness under codec shift remains unclear.These components are examples of evidence outside dominant verbal content.Source: §2 · p. 2
- Codec-aware methods address the vocoder-to-neural-codec shift mainly through new training data and learning objectives, leaving the representation question less explored.Examples include CodecFake and subsequent codec-aware methods.Source: §2 · p. 2
- Pooled no-vocals residual statistics are most effective on unseen codecs while complementing hierarchical XLS-R in MN-P.The residual is aggregated over the utterance and integrated with token-level XLS-R features through adaptive gating.Source: §2 · p. 2
3. REPRESENTATION ANALYSIS
A controlled comparison of 12 representations finds that hierarchical XLS-R leads on overall test EER, whereas no-vocals statistics lead on codec-unseen EER. The no-vocals view also has lower linearly decodable source identity than XLS-R, supporting its use as complementary evidence.
3. REPRESENTATION ANALYSIS
- The analysis evaluates 12 representations with a shared low-capacity linear classifier and reports both overall detection and codec-unseen performance.The candidates span spectral, codec-residual, non-vocal, and hierarchical self-supervised feature families.Source: §3 · p. 2
- The test protocol includes unseen vocoders, seen codecs, and a held-out DAC codec F07 for the codec-unseen condition.Training and development include A01–A06 vocoders and F01–F06 codecs; F07 is absent from both.Source: §3 · p. 2
- B6 achieves the lowest overall EER at 4.843%, while B2-NV achieves the lowest CU EER at 8.619%, compared with 9.291% for B6.B2-NV also yields the lowest Source F1 at 0.539, against 0.726 for B6.Source: §3 · p. 2, Table 1 · p. 2
- Replacing B2's vocals view with its no-vocals view reduces EER from 25.999% to 13.859% and CU EER from 41.201% to 8.619%.B2-XV has a comparable overall EER of 13.448% but a higher CU EER of 21.935%, making the codec-unseen advantage specific to B2-NV.Source: §3 · p. 2
4. PROPOSED METHOD
MN-P combines a temporal XLS-R speech representation with a pooled utterance-level no-vocals residual. The residual is formed from separated non-vocal audio, summarized as a 127-dimensional vector, and fused with the speech sequence through an utterance-conditioned gate.
4. PROPOSED METHOD
- MN-P combines a MoEF-fused hierarchical XLS-R representation with a pooled no-vocals representation as an utterance-level auxiliary vector.Pooling aggregates residual statistics over the utterance without requiring frame-level alignment with the XLS-R sequence.Source: §4 · p. 2, Figure 2 · p. 3
4.1. Pooled No-Vocals Residual
- The no-vocals residual retains background content, separation artifacts, and residual speech energy that may preserve traces of the synthesis pipeline.HTDemucs separates the signal, after which the vocals are discarded and the remaining sources are summed, downmixed, and resampled.Source: §4.1 · pp. 2–3
- Utterance-level spectrotemporal statistics are concatenated into a single 127-D residual vector that serves as the auxiliary view fused with XLS-R.The statistics summarize frequency-band trajectories and cross-band, temporal, local spectral-residual, and spectral-shape features.Source: §4.1 · p. 3
4.2. Dual-View Integration
- CrossViewGate estimates the relative contributions of the two representations for each utterance, then broadcasts the residual vector across the XLS-R token grid for fusion.The dense two-view gate allows both representations to contribute, and the fused sequence is classified by an AASIST backend.Source: §4.2 · p. 3
5. EXPERIMENTAL SETUP
The evaluation combines vocoder-based ASVspoof 2019 LA fakes and neural-codec Codecfake fakes, with a held-out DAC codec defining the codec-unseen subset. Experiments use a shared evaluation protocol and three-seed averages.
5.1. Datasets and Metrics
- The corpora combine ASVspoof 2019 LA vocoder fakes and Codecfake neural-codec fakes, retaining mono 16-kHz, 64,000-sample waveforms that meet RMS and peak-amplitude thresholds.RMS must be at least 10^-2 and peak amplitude at least 3 × 10^-2.Source: §5.1 · p. 3
- The 39,996-utterance test set spans bona fide speech, seen and unseen vocoders, seen codecs, held-out DAC F07, and unseen ALM-based fakes; F07 defines the codec-unseen subset.F07 is excluded from training and development; the codec-unseen subset contains 4,444 fake and 4,444 bona fide utterances.Source: §5.1 · p. 3
- AUC and EER use test scores, while fake miss uses the EER threshold selected on development data.Source: §5.1 · p. 3
5.2. Implementation Details
- Main-model and ablation results are averaged over three seeds; XLS-R is frozen, HTDemucs residual vectors are precomputed, and comparison systems share data, preprocessing, and protocol.Trainable modules use AdamW, and evaluation uses the checkpoint with minimum development loss.Source: §5.2 · p. 4
6. RESULTS AND DISCUSSION
MN-P improves on retrained state-of-the-art systems, with its largest reported advantage on codec-unseen speech. The ablation indicates that utterance-level pooling of the no-vocals residual is more effective than retaining it as a frame-level sequence.
6.1. Main Comparison
- MN-P reduces AASIST EER from 7.010% to 2.744% and CU EER from 18.639% to 6.136%; with Nes2Net, CU EER falls from 15.632% to 6.706%.MN-P-Nes2Net ranks second across all reported metrics, showing gains across detector backends.Source: §6.1 · p. 4
- 54.2% and 60.9% relative reductions in EER and CU EER, respectively, are achieved by MN-P-AASIST against W2V2-AASIST with CSAM, without codec-specific training.The comparison baseline reports 5.986% EER and 15.692% CU EER; MN-P also outperforms the retrained systems across reported metrics.Source: §6.1 · p. 4
6.2. Ablation on Model Structure
- The ablation compares M0 without the residual view, MN-S with a frame-level residual sequence, and MN-P with an utterance-level residual vector.The ablation uses an independent split with 2,000 training, 1,000 development, and 2,000 test samples over three seeds.Source: §6.2 · p. 4, Table 3 · p. 4
- CU EER falls from 34.830% for M0 to 18.168% with a frame-level residual sequence, then to 8.709% with utterance-level pooling.CU miss also falls from 57.808% to 14.865% with pooling, which avoids explicit alignment with the XLS-R sequence.Source: §6.2 · p. 4
7. CONCLUSION
The paper finds that pooled no-vocals residual statistics complement XLS-R under codec shift and proposes MN-P to combine them. MN-P more than halves codec-unseen EER against the best retrained system, while future work will examine residual components and broader conditions.
7. CONCLUSION
- Pooled no-vocals residual statistics outperform hierarchical XLS-R in the unseen-codec condition, although the strongest overall representation is not necessarily strongest under that condition.Source: §7 · p. 4
- MN-P combines token-level XLS-R features with an utterance-level pooled no-vocals representation through adaptive gating.Source: §7 · p. 4
- MN-P more than halves codec-unseen EER relative to the best-performing retrained state-of-the-art system, with gains retained across detector backends.Source: §7 · p. 4
- Future work will disentangle background content from separation artifacts in the residual and extend evaluation to more diverse conditions.Source: §7 · p. 4