Source-linked AI summary
Neural Audio Codec for Robust Audio Deepfake Detection
Jungwoo Kim, Joonyong Park, Junyoung Koh, Jong-Seok Lee
TL;DR
Audio deepfake detectors often encounter compressed audio, while perceptual-quality-focused neural codecs may alter forensic cues. This paper uses mixed-pair evaluation to distinguish codec-specific class-side failures and adapts a pretrained codec with forensic supervision. At 0.5 kbps, FP-NAC lowers EER across three detectors while maintaining closely matched reconstruction quality and transferring gains to held-out detectors.
Problem
Neural audio codecs are optimized for perceptual quality, but it is unclear how coding changes forensic evidence or which class-side cues are affected.
Source: §1 · p. 1Method
The paper uses mixed-pair evaluation to isolate codec-induced bona fide and spoof effects, then adapts a pretrained codec with detector-guided forensic supervision.
Source: §1 · p. 1, §3 · p. 2, §4.2 · p. 3Results
At 0.5 kbps, EER decreases from 63.47% to 13.66% on AASIST, from 45.93% to 14.78% on AASIST-L, and from 25.74% to 13.20% on RawNet2, while reconstruction metrics remain closely matched across most operating points.
Source: §5.2 · p. 4Takeaways & Limitations
Low-rate gains transfer from AASIST supervision to held-out detectors, supporting forensic preservation as a codec-design objective alongside reconstruction quality.
Source: §5.2 · p. 4
Abstract
from arXiv · showhide
Audio deepfake detectors are typically evaluated on uncompressed audio, although real-world audio often undergoes low-bitrate coding. In this work, we investigate how audio coding affects deepfake detection across codecs, bitrates, and detectors, finding higher errors at lower rates. A mixed-pair protocol isolates codec-induced changes in bona fide and spoof audio, revealing asymmetric, codec-dependent failures: low-rate DAC and EnCodec mainly degrade bona fide detection, whereas X-Codec shows a stronger spoof-side limitation. Motivated by these, we propose a forensic-preserving neural audio codec (FP-NAC), which fine-tunes a pretrained codec using a detector-guided objective while preserving its native hard quantization path and bitrate. On ASVspoof 2019 LA, FP-NAC reduces EER by up to 49.8~pp compared with the original DAC at 0.5~kbps while maintaining comparable reconstruction quality. Although supervised by only one detector, FP-NAC improves performance across multiple detectors, highlighting forensic transparency as a codec design objective alongside perceptual quality. Our codes are available at https://github.com/kjungwoo03/FP-NAC.
1. INTRODUCTION
Audio deepfake detectors often receive compressed audio, but neural audio codecs optimized for perceptual quality may not preserve forensic cues. The paper introduces mixed-pair evaluation and FP-NAC to diagnose codec-specific failures and reduce detection error while maintaining reconstruction quality.
1. INTRODUCTION
- Audio detectors often process compressed audio, while neural audio codecs target low-bitrate perceptual quality that may not preserve spoof-detection cues.Coding artifacts can appear in both bona fide and synthetic audio, and codec token spaces can also contain information useful for identifying spoofed audio.Source: §1 · p. 1
- FP-NAC adapts a pretrained codec with detector-guided forensic supervision, reducing low-bitrate detection error while maintaining comparable reconstruction quality and transferring gains to held-out detectors.The framework retains the pretrained codec backbone and makes forensic preservation an explicit part of neural audio coding.Source: §1 · p. 1
- A mixed-pair protocol separately probes codec-induced changes on bona fide and spoof trials, revealing asymmetric, codec-dependent class-side failures.Acoustic codecs fail predominantly on bona fide trials, while X-Codec has comparatively stronger spoof-side failure.Source: §1 · p. 1
- Prior studies largely use aggregate detection performance, leaving unclear whether coding shifts bona fide speech toward spoof-like representations, suppresses spoof evidence, or affects both.Existing mitigation is predominantly detector-centric, requiring coding-artifact robustness to be learned separately for each detector.Source: §1 · p. 1
2. RELATED WORK
Neural audio codecs use learned encoder–quantizer–decoder architectures, but their perceptual-quality objectives do not ensure preservation of forensic evidence. The supplied findings describe detection degradation at lower bitrates and distinct codec behavior.
2. RELATED WORK
- SoundStream and EnCodec use residual vector quantization for scalable low-rate coding, DAC emphasizes high-fidelity reconstruction, and X-Codec incorporates semantic information.These codecs are optimized primarily for perceptual quality rather than task-specific forensic evidence.Source: §2 · pp. 1–2
- EER generally increases as bitrate decreases, while FP-NAC substantially reduces EER compared with other neural audio codecs on ASVspoof 2019 LA.The figure reports full EER across three detectors and measured bitrate, with uncompressed-audio EER marked for comparison.Source: Figure 1 · p. 2
- Under AASIST, DAC and EnCodec show large H1 that shrinks with bitrate, whereas X-Codec shows persistently elevated H2.The figure caption characterizes these as codec-dependent asymmetric failure patterns.Source: Figure 2 · p. 2
- AASIST and RawNet2 use complementary representations, yet remain vulnerable to unseen attacks and domain shifts; codec artifacts and spoof-relevant information can also degrade detection.The cited studies report that neural codecs can cause severe detection drops even when perceptual quality remains high.Source: §2 · p. 2
3. MIXED-PAIR EVALUATION OF CODEC-INDUCED FORENSIC FAILURES
The section evaluates codec-induced detection changes using mixed-pair trials that separate bona fide-side and spoof-side effects. Its findings show asymmetric failures that vary by codec and are directionally structured in detector space.
3. MIXED-PAIR EVALUATION OF CODEC-INDUCED FORENSIC FAILURES
- Detection performance degrades as bitrate decreases across multiple detectors, but full EER does not identify which class-side evidence coding affects.Full EER measures separability between coded bona fide and coded spoof samples.Source: §3 · p. 2
- H1 pairs coded bona fide with clean spoof audio to probe spoofward shifts, while H2 pairs clean bona fide with coded spoof audio to probe suppression of spoof cues.These independently thresholded diagnostic EERs are not additive components of full EER.Source: §3 · p. 2
- DAC and EnCodec exhibit high H1 but low H2, whereas X-Codec shows low H1 but high H2, indicating codec-dependent class-side failures.For DAC and EnCodec, codec-specific artifacts can make coded bona fide audio more spoof-like; X-Codec instead shows stronger suppression of spoof-discriminative evidence.Source: §3 · p. 2
- At 0.5 kbps, DAC shifts AASIST bona fide embeddings toward the clean spoof region, while coded spoof embeddings move little toward bona fide along the centroid axis.The centroid-defined axis represents the clean class-separation direction, and the coded bona fide and clean spoof representations consequently overlap strongly.Source: §3 · p. 2, Figure 3 · p. 2
- The findings characterize codec-induced degradation as class-asymmetric, rate-dependent, and directionally structured in detector space, motivating codec-side evidence preservation across rates.The proposed intervention is intended to preserve detector-relevant evidence for both bona fide and spoofed audio.Source: §3 · p. 2
4. PROPOSED METHOD
FP-NAC adapts a frozen DAC codec with RVQ-depth-conditioned residual adapters to preserve detector-relevant evidence across rates. It jointly optimizes reconstruction quality and forensic preservation while retaining hard quantization, and trains across the full operating-rate range with greater emphasis on lower rates.
4. PROPOSED METHOD
- Frozen DAC backbones receive trainable residual adapters conditioned on active RVQ depth K, enabling rate-specific corrections while retaining the original coding path.Encoder-side adapters modify representations before RVQ; decoder-side adapters modify reconstruction after RVQ and decoder blocks.Source: §4 · p. 3, Figure 4 · p. 3, §4.1 · p. 3
4.1. Rate-Conditioned Residual Adaptation
- Each adapter uses a bottleneck, depthwise temporal processing, and K-conditioned FiLM modulation within a residual update.The rate-specific residual gate is initialized to zero, so each adapter initially acts as the identity.Source: §4.1 · p. 3
4.2. Forensic-Preserving Optimization
- Hard code assignments remain in the forward pass, and removing the soft-backward surrogate at inference preserves DAC's bitstream format and nominal bitrate.The soft-assignment surrogate is used only to backpropagate gradients to encoder-side adapters.Source: §4.2 · p. 3
- FP-NAC combines reconstruction quality and forensic preservation in a joint objective, using a frozen pretrained detector as a forensic teacher.The objective also preserves detector representations and spoof-trial spectral envelopes; forensic supervision is gradually introduced during early training.Source: §4.2 · p. 3
- FP-NAC trains one checkpoint over K = 1 to 12, emphasizing smaller K while retaining coverage of the full rate range.The sampling distribution reflects stronger forensic degradation at lower rates.Source: §4.2 · p. 3, Figure 5 · p. 3
5. EXPERIMENTS
On ASVspoof 2019 LA, FP-NAC reduces low-rate detection errors across the evaluated detectors while maintaining comparable reconstruction quality. Ablations show that joint encoder–decoder adaptation performs best on average, while rate sampling and conditioning shape performance across bitrate regimes.
5.1. Experimental Setup
- FP-NAC is trained and evaluated on ASVspoof 2019 LA, with AASIST as the frozen teacher and AASIST-L and RawNet2 as held-out detectors.Detection uses EER; waveform inputs are standardized to 64,600 samples.Source: §5.1 · pp. 3–4
5.2. Main Results
- At 0.5 kbps, FP-NAC lowers EER from 63.47% to 13.66% on AASIST, 45.93% to 14.78% on AASIST-L, and 25.74% to 13.20% on RawNet2.Forensic supervision uses AASIST, with gains transferring to both held-out detectors.Source: §5.2 · p. 4
- FP-NAC and reconstruction-only adaptation improve low-to-mid-rate PESQ-WB and STOI over Plain DAC, with closely matched quality across most operating points.At higher rates, differences diminish as all variants approach the original DAC performance.Source: §5.2 · p. 4, Figure 6 · p. 4
- FP-NAC moves coded bona fide representations away from the spoof region while coded spoof representations remain on the spoof side.This restores codec-domain class separation collapsed by Plain DAC and aligns with reduced H1-dominant failure at low bitrates.Source: §5.2 · p. 4
5.3. Ablation Studies
- Joint encoder–decoder adaptation achieves the best average performance across three detectors; FP-NAC uses 1.30M trainable parameters, or 1.75% of original DAC.Encoder-only and decoder-only variants use 0.72M (0.97%) and 0.58M (0.78%), respectively.Source: §5.3 · p. 4, Table 1 · p. 4
- Uniform rate sampling reduces higher-rate average EER from 3.24% to 2.91%, but raises K = 1 EER by 5.62 pp and increases bona fide collapse.The proposed distribution prioritizes low rates, where codec-induced detection degradation is more severe.Source: §5.3 · p. 4
- Removing K-conditioning leaves low-rate performance nearly unchanged but raises higher-rate average EER from 3.24% to 3.87% and lowers PESQ-WB from 3.65 to 2.93.Conditioning enables rate-specific corrections that limit cross-rate interference.Source: §5.3 · p. 4
6. CONCLUSION
Mixed-pair evaluation reveals asymmetric, codec-dependent low-bitrate failures in deepfake detection. FP-NAC adapts DAC with detector-guided forensic supervision, reducing EER by up to 49.8 pp at 0.5 kbps on ASVspoof 2019 LA while transferring to held-out detectors and maintaining comparable reconstruction quality.
6. CONCLUSION
- On ASVspoof 2019 LA, FP-NAC reduces EER by up to 49.8 pp at 0.5 kbps, transfers consistently to held-out detectors, and maintains comparable objective reconstruction quality.The method adapts original DAC using detector-guided forensic supervision.Source: §6 · p. 4