PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory Dynamics in Self-Supervised Learning Latent Space for Audio Deepfake Detection

Tomás Andrade Weber

arXiv:2608.13817v1eess.AScs.LGcs.SD

TL;DR

Generalisation of SSL-based audio deepfake detectors to diverse real-world synthesis remains an open problem. This paper models trajectory dynamics in AntiDeepfake SSL space with a bonafide-only causal LSTM predictor and finds increasing gains over static pooling as benchmark difficulty grows, especially on hard cross-corpus tests.

  • Problem

    SSL-based detectors are strong on matched benchmarks, but their generalisation to diverse real-world synthesis remains limited on hard cross-corpus evaluations.

  • Method

    The paper trains a causal LSTM next-frame predictor on bonafide AntiDeepfake SSL features and compares it with static pooling, with a supervised MLP stage using frozen LSTM states.

  • Results

    Trajectory dynamics are near parity with static pooling on near-domain benchmarks but provide increasingly large gains on harder cross-corpus benchmarks, reaching 22.2% on DE2024 over the static baseline.

  • Takeaways & Limitations

    Trajectory dynamics provide a reliable one-class detection signal beyond static pooling, with their advantage scaling with benchmark difficulty.

Abstract

from arXiv · show

Human speech production is constrained by physiology, giving rise to characteristic temporal structure on acoustic signals. We hypothesise that these constraints manifest as structured trajectory dynamics in the latent space of Self-Supervised Learning (SSL) models, and that synthetic speech violates them detectably. To test this hypothesis, we train a causal Long Short-Term Memory (LSTM) next-frame predictor on bonafide speech only (Stage 1), using the deepfake-specialised SSL backbone Wav2Vec2-Large-AntiDeepfake, and compare against a static global-average-pooling baseline using identical features, thus isolating the contribution of temporal modelling. A supervised Stage 2, which trains a Multi-Layer Perceptron on the frozen LSTM internal states using labelled data, is included to characterise the role of spoof supervision. Our system achieves competitive or state-of-the-art performance across six benchmarks: ASVspoof 2019/2021, Codecfake, In-the-Wild, MLAAD-EN, and Deepfake-Eval-2024, including best published EER on ASVspoof 2021 (0.75\%) and, notably, Stage 1 trained on bonafide speech only surpasses the published supervised baseline from the same backbone on DE2024 (30.35\%). On near-domain benchmarks, static and dynamic approaches perform comparably. On harder cross-corpus benchmarks with diverse synthesis methods, trajectory dynamics provide substantial gains, confirming that temporal physiological constraints carry detection signal beyond utterance-level statistics.

I. INTRODUCTION

Synthetic speech detection remains difficult because SSL-based systems generalise poorly beyond matched benchmarks. This work therefore models physiological trajectory dynamics in SSL latent space, finding increasing gains over static representations as benchmark difficulty rises.

  • Motivation: Realistic TTS and VC systems create societal risk, motivating benchmarks from ASVspoof to increasingly challenging real-world evaluations.
  • Challenge: SSL-based detectors perform strongly on matched benchmarks but degrade severely on hard cross-corpus evaluations such as MLAAD and DE2024.
  • Research gap: Most SSL-based systems discard temporal order through global average pooling or attention-weighted frame summarisation.
  • Hypothesis and approach: The authors hypothesise that physiological production constraints create structured SSL trajectory dynamics that synthetic speech violates detectably.
  • Key result: Trajectory dynamics increasingly outperform the static baseline as benchmark difficulty grows, from near parity on ASVspoof to large gains on MLAAD-EN and DE2024.The system also outperforms SLIM, QAMO, and BreathNet on all shared benchmarks without backbone fine-tuning.

II. RELATED WORK · A. SSL-based Deepfake Detection

SSL speech representations are established as strong frontends for audio deepfake detection because they provide rich phonetic, speaker, and forensic information. AntiDeepfake extends this line of work through post-trained SSL models developed using large-scale genuine and synthetic speech data.

  • A. SSL-based Deepfake Detection: SSL speech representations have become the dominant frontend for audio deepfake detection.
  • A. SSL-based Deepfake Detection: Frozen wav2vec 2.0, HuBERT, and XLS-R embeddings support strong detection with simple classifiers.
  • A. SSL-based Deepfake Detection: These SSL models provide rich phonetic and speaker embeddings without task-specific supervision.
  • A. SSL-based Deepfake Detection: Strong performance from simple classifiers confirms that SSL representations encode rich forensic information.
  • A. SSL-based Deepfake Detection: AntiDeepfake comprises post-trained SSL models designed specifically for audio deepfake detection.
  • A. SSL-based Deepfake Detection: The AntiDeepfake models were developed using large-scale genuine and synthetic speech data.

B. One-Class and Anomaly-Based Detection · C. Temporal Modelling

Prior one-class methods detect synthetic speech through static utterance-level deviations from bonafide centroids, whereas temporal approaches model localized artifacts or physiological cues. The paper’s approach shares the one-class philosophy but differs fundamentally from static anomaly scoring by exploiting temporal structure.

  • B. One-Class and Anomaly-Based Detection: One-class deepfake detection learns compact bonafide representations and flags deviations as synthetic.
  • B. One-Class and Anomaly-Based Detection: OC-Softmax applies one-class learning with an angular margin loss.
  • B. One-Class and Anomaly-Based Detection: SAMO extends one-class detection with speaker-attractor multi-centre objectives.
  • B. One-Class and Anomaly-Based Detection: QAMO introduces quality-aware multi-centroid learning by splitting bonafide speech into high- and low-quality subsets using MOS scores.
  • B. One-Class and Anomaly-Based Detection: These methods define anomaly scores from static utterance-level distances to learned centroids, while the proposed Stage 1 shares their one-class philosophy but differs fundamentally.
  • C. Temporal Modelling: Temporal modelling complements spectral features by targeting locally suspicious frames or physiological cues.
  • C. Temporal Modelling: FGFM uses multi-head voting for fine-grained frame selection to identify spatially localised artifacts.
  • C. Temporal Modelling: BreathNet detects breath frames and uses their locations to modulate XLS-R features during training.The backbone implicitly encodes breath-related physiological cues even at inference without the breath mask.

D. Cross-Corpus Generalisation · III. METHOD · A. AntiDeepfake Features and Static Baseline

The method uses Wav2Vec2-Large-AntiDeepfake frame-level features and a matched static centroid-distance baseline to isolate the contribution of trajectory dynamics. This design addresses out-of-domain generalisation by retaining or discarding temporal order while keeping the backbone fixed.

  • A. AntiDeepfake Features and Static Baseline: Wav2Vec2-Large-AntiDeepfake supplies the speech representation for all experiments.The backbone provides 1024-dimensional frame-level features at 50 frames per second.
  • A. AntiDeepfake Features and Static Baseline: 1024-dimensional representations are extracted at 50 frames per second.These are frame-level features used throughout the experiments.
  • A. AntiDeepfake Features and Static Baseline: The features are post-trained to discriminate bonafide from synthetic speech.This provides a strong representational foundation before temporal modelling.
  • A. AntiDeepfake Features and Static Baseline: The static baseline uses the same features to isolate trajectory dynamics from backbone quality.Using identical features makes the comparison target the contribution of temporal modelling.
  • A. AntiDeepfake Features and Static Baseline: For each dataset, evaluation utterances are scored by L2 distance from the bonafide-training centroid.The centroid is computed from all bonafide training utterance vectors, while each evaluation vector is globally average pooled.
  • D. Cross-Corpus Generalisation: Global average pooling discards temporal order, mirroring the original AntiDeepfake evaluation readout.The baseline therefore contrasts utterance-level pooled representations with temporally structured modelling.

B. Stage 1: Causal LSTM Trajectory Predictor

Stage 1 models bonafide speech trajectories with a causal two-layer LSTM that predicts each next frame from preceding SSL embeddings. It learns dynamics exclusively from ASVspoof 2019 LA bonafide training utterances, with higher anomaly scores indicating greater deviation from those dynamics.

  • Architecture: A 2-layer LSTM with hidden dimension 512 causally predicts each next frame from its predecessors in an ordered sequence of T frame embeddings h1, . . . , hT ∈R1024.The embeddings are extracted from a single utterance.
  • Training: The predictor is trained exclusively on 2,580 bonafide utterances from the ASVspoof 2019 LA training set using mean squared error between predicted and true next frames.Training uses bonafide speech only.
  • Anomaly scoring: Higher anomaly scores indicate trajectories that deviate more from the bonafide dynamics learned during training.The causal prediction at step t is represented as ˆht = fθ(h1, . . . , ht).

C. Stage 2: Supervised MLP · IV. EXPERIMENTS

Stage 2 uses the frozen Stage 1 LSTM as an utterance-level feature extractor and trains a supervised MLP to predict spoof probability. The model is trained on 25,380 ASVspoof 2019 LA training utterances represented by 512-dimensional mean-pooled hidden states.

  • C. Stage 2: Supervised MLP: Stage 2 freezes the Stage 1 LSTM and uses it as a feature extractor.This stage introduces spoof supervision, unlike the preceding unsupervised training setup.
  • C. Stage 2: Supervised MLP: Mean-pooling hidden states across all frames produces a fixed-length 512-dimensional utterance representation.The representation is derived from the LSTM’s internal states.
  • C. Stage 2: Supervised MLP: The Stage 2 training set contains 25,380 ASVspoof 2019 LA utterances.The set comprises 2,580 bonafide and 22,800 spoof utterances.
  • C. Stage 2: Supervised MLP: The training data include 2,580 bonafide utterances and 22,800 spoof utterances.These counts sum to the full ASVspoof 2019 LA training set used for Stage 2.
  • C. Stage 2: Supervised MLP: A three-layer MLP maps the 512-dimensional representation through dimensions [512 →256 →128 →1].The network is trained to output a spoof probability.
  • C. Stage 2: Supervised MLP: The MLP uses dropout p = 0.3 and outputs a spoof probability.Its supervised objective distinguishes bonafide from spoof utterances.

A. Datasets · B. Implementation Details

The study trains and evaluates on bonafide-only Stage 1 data and labelled Stage 2 data across six benchmarks, using controlled, in-the-wild, codec-synthesis, and cross-domain conditions. Implementation standardises optimisation, evaluation, and variance estimation across repeated runs.

  • A. Datasets: Stage 1 uses 2,580 bonafide ASVspoof 2019 LA utterances, while Stage 2 additionally uses 22,800 spoof utterances.Evaluation covers ASVspoof 2019 LA, ASVspoof 2021 DF, In-the-Wild, Codecfake, MLAAD-EN, and DE2024.
  • A. Datasets: The six benchmarks span controlled studio-quality TTS and VC attacks, spontaneous celebrity speech with unknown synthesis methods, and LLM-based neural codec synthesis.The supplied passage identifies ASVspoof 2019/2021 as controlled evaluations, In-the-Wild as spontaneous celebrity speech, and Codecfake as codec-synthesis evaluation.
  • A. Datasets: Only In-the-Wild and DE2024 are fully out-of-distribution for the backbone, whereas ASVspoof 2019/2021, Codecfake, and MLAAD spoof audio appear in its post-training data.For MLAAD, the backbone saw synthetic audio but not the bonafide M-AILABS reference speech, making the bonafide trajectory model genuinely zero-shot.
  • B. Implementation Details: Stage 1 training runs for 500 epochs with AdamW, cosine annealing and warm restarts, learning rate 10^-3, dropout p = 0.1, and sequences capped at 500 frames.Warm-restart settings are T0 = 50 and Tmult = 2.
  • B. Implementation Details: Stage 2 trains the MLP for 100 epochs with dropout p = 0.3, while all experiments use a single H100 GPU and report Equal Error Rate (EER %).The reporting convention follows the ASVspoof challenge protocol.
  • B. Implementation Details: Variance estimation uses five Stage 1 seeds and five Stage 2 classifiers per seed, yielding 25 runs per benchmark.Reported means and standard deviations capture both LSTM initialisation variance and MLP training variance; the static baseline is deterministic.

V. RESULTS

Across six benchmarks, Stage 2 is strongest on near-domain datasets, while Stage 1 provides the largest gains on out-of-domain benchmarks. On DE2024, Stage 1 surpasses the NII-GAP baseline without spoof supervision, whereas Stage 2 matches other supervised systems.

  • Benchmark comparison: On near-domain benchmarks, Stage 2 performs comparably to the static baseline, significantly outperforms Stage 1, and exceeds all published competitors.These benchmarks are ASVspoof 2019, ASVspoof 2021, and Codecfake.
  • Benchmark comparison: On out-of-domain benchmarks, Stage 1 outperforms the static baseline by a large margin and exceeds Stage 2.The out-of-domain benchmarks are MLAAD-EN and DE2024.
  • DE2024: On DE2024, Stage 1 surpasses the NII-GAP baseline without spoof supervision, while Stage 2 converges to performance similar to other supervised systems trained on the same data.This contrasts the benefits of bonafide-only training with supervised performance on the same benchmark.

A. The Difficulty Gradient · B. One-Class vs Supervised Generalisation

Performance differences widen with benchmark difficulty: static and supervised methods are close on near-domain datasets, while Stage 1 gains the strongest results on harder out-of-domain benchmarks. This one-class advantage is linked to reduced mismatch with unseen synthesis conditions.

  • A. The Difficulty Gradient: On ASVspoof 2019/2021 and Codecfake, the static baseline and Stage 2 differ by less than 0.5% EER.The small gap indicates that backbone representations largely suffice to separate bonafide from synthetic speech.
  • A. The Difficulty Gradient: On In-the-Wild, Stage 2 retains a −0.75% EER advantage over Stage 1.This is where the gap between static and supervised approaches begins to open.
  • A. The Difficulty Gradient: Stage 1 provides the best result on the hardest out-of-domain benchmarks, MLAAD-EN and DE2024.Its advantage over the static baseline grows from 17.2% on MLAAD-EN to 22.2% on DE2024.
  • A. The Difficulty Gradient: UMAP shows tight, well-separated bonafide clusters for ASVspoof 2019/2021 and Codecfake, consistent with their low static EER.The projections used approximately 175,000 randomly sampled frames per dataset.
  • A. The Difficulty Gradient: In-the-Wild has a more diffuse bonafide distribution with partial overlap, while MLAAD-EN shows heavy overlap and DE2024 near-complete mixing.These visual patterns track the increasing difficulty across datasets.
  • B. One-Class vs Supervised Generalisation: Stage 2 supervision is based on ASVspoof 2019 attack types A01–A06, which are not representative of MLAAD-EN’s 140+ TTS systems or DE2024 real-world deepfakes.The passage identifies this mismatch as relevant to harder-benchmark generalisation.
  • B. One-Class vs Supervised Generalisation: Trained only on bonafide speech, one-class Stage 1 is unaffected by this mismatch and generalises more robustly to unseen synthesis conditions.On both MLAAD-EN and DE2024, Stage 2 converges to performance comparable to published results.

VI. CONCLUSION

Trajectory dynamics in AntiDeepfake SSL space provide a reliable one-class detection signal beyond static pooling, with increasing benefits as benchmark difficulty rises. Bonafide-only modelling generalises better than supervised spoof classification to unseen synthesis methods, while several extensions remain future work.

  • Main conclusion: Trajectory dynamics provide a reliable one-class detection signal that static pooling cannot capture, with the advantage scaling with benchmark difficulty.The conclusion attributes the growing advantage to increasingly difficult benchmarks.
  • Main conclusion: Bonafide/spoof overlap increases from tight separation to near-complete mixing across benchmarks, explaining static degradation and growing trajectory-dynamics gains.The progression spans ASVspoof 2019/2021 and Codecfake, through In-the-Wild and MLAAD-EN, to DE2024.
  • Generalisation: One-class bonafide modelling generalises better than supervised spoof classification to unseen synthesis methods.This conclusion is associated with the reversal from Stage 1 > Stage 2 on hard out-of-domain benchmarks.
  • Future work: Future work includes comparing bidirectional LSTM and Transformer sequence models, fine-tuning with DE2024 in-domain training, and evaluating multilingual generalisation.These directions are explicitly identified as unresolved extensions.
Loading 2608.13817v1…