Source-linked AI summary
Audio Deepfake Detection Using Temporal Coherence Analysis
Justin D. Norman, Sarah Barrington
TL;DR
AI-generated audio raises information-integrity concerns, and audio deepfake detection remains less explored than image and video detection. This paper analyzes temporal patterns in CLAP segment embeddings and reports competitive performance alongside feature-direction reversals across deployment settings and audio domains.
Problem
Audio deepfake detection remains underexplored, despite synthetic speech and music posing information-integrity concerns.
Source: §I · p. 1Method
The framework computes pairwise cosine similarities among segment-level CLAP embeddings, extracts statistical features, and uses lightweight classifiers for audio deepfake detection.
Source: §I · p. 1, §II · p. 2Results
The approach achieves competitive performance with end-to-end neural methods, while 21 of 29 features reverse direction between training and in-the-wild evaluation and entropy discriminates oppositely for speech and music.
Source: §I · p. 1, §VI · p. 5Takeaways & Limitations
Temporal coherence analysis provides interpretable feature-based explanations across speech and music benchmarks.
Source: §VI · p. 5The multi-expert ensemble requires training data from multiple source datasets, which may not always be available.
Source: §V · p. 5
Abstract
from arXiv · showhide
The proliferation of AI-generated audio (so-called "deepfake" audio) poses significant threats to information integrity, from voice cloning fraud to synthetic music copyright disputes. We present a temporal coherence analysis framework built upon Contrastive Language-Audio Pretraining (CLAP) embeddings that spans speech, instrumental music, and music with vocals. By computing pairwise cosine similarities between audio segment embeddings and extracting statistical features from the resulting distributions, we train lightweight ensemble classifiers that reliably distinguish authentic from synthetic audio. Our work provides an interpretable, computationally efficient alternative to common deep learning methods while still achieving competitive performance across speech and music domains. Further, we reveal two notable empirical findings about audio deepfakes: (1) a feature-label inversion phenomenon in which 21 of 29 statistical features reverse their discriminative direction between training and in-the-wild deployment, and (2) a speech--music direction reversal in which entropy discriminates in opposite directions for speech and music deepfakes.
I. INTRODUCTION
AI-generated speech and music raise information-integrity concerns, while audio deepfake detection remains less developed than image and video detection. The paper proposes a domain-neutral CLAP-based temporal-coherence approach and reports feature-direction reversals across deployment settings and audio domains.
I. INTRODUCTION
- Voice cloning and synthetic music can produce convincing audio, enabling fraud, impersonation, and disinformation at scale.The passage associates these risks with voice cloning using minimal training data and increasingly indistinguishable synthetic compositions.Source: §I · p. 1
- Audio deepfake detection remains underexplored compared with image and video methods, while audio systems often focus on single domains or lack interpretable temporal analysis.The introduction notes that image and video methods often report greater than 95% accuracy for videos.Source: §I · p. 1
- The proposed pipeline analyzes how CLAP embeddings vary across temporal audio segments rather than relying on biometric identity consistency.CLAP embeddings are non-speech-specific and trained on varied audio, including human activities, natural sounds, and audio effects [10].Source: §I · p. 1
- 21 of 29 statistical features reverse their discriminative direction between controlled training data and in-the-wild evaluation.The paper also reports that entropy discriminates in opposite directions for speech and music deepfakes.Source: §I · p. 1
II. METHODS
The method converts recordings into segment-level CLAP embeddings, summarizes their pairwise similarities with statistical features, and classifies audio using speech or music strategies.
II. METHODS
- Audio is converted to 48 kHz mono with peak normalization, and overlapping 2-second windows are extracted at 1-second hops.Files with fewer than three valid segments are excluded.Source: §II · p. 1
- Pairwise cosine similarities among segment embeddings form a distribution intended to characterize each recording’s internal temporal coherence.The analysis computes similarities for unique segment pairs.Source: §II · p. 2
- The pipeline extracts 29 statistical features, ranks them by training-set AUC-ROC, and trains the final XGBoost classifier on eight selected features.The eight-feature subset is supported by an ablation study; classifiers are trained separately by dataset using five-fold stratified cross-validation and robust scaling.Source: §II · p. 2
- For speech, five expert classifiers trained on different datasets are combined by a weighted average of their predicted deepfake probabilities.The weights and decision threshold are optimized on a held-out validation set.Source: §II · p. 2
- For music, samples below the test batch’s median mean-similarity value are classified as deepfake, without requiring target-domain labels.The classifier uses the 50th percentile of mean similarity as its decision boundary.Source: §II · p. 2
A. Feature Analysis
Feature analysis finds that entropy’s discriminative direction differs between speech and music, while 21 of 29 statistical features reverse direction between training and in-the-wild evaluation. These shifts qualify the cross-domain value of features that appear strong in training.
a) Direction Reversal
- Entropy points in opposite directions by audio type: authentic single-voice speech has higher entropy, while deepfake music has higher entropy.The reported effect sizes are d = +1.12 for single-voice speech and d = −1.59 for music.Source: A. Feature Analysis · p. 3
b) Universal Features
- Eleven features exceed individual AUC 0.60 across all three audio types, and entropy has the highest mean AUC at 0.871.Percentile and central-tendency features can outperform entropy on held-out evaluation despite lower training AUC.Source: A. Feature Analysis · p. 3, Table III · p. 3
c) Feature-Label Inversion Across Domains
- 21 of 29 features reverse their discriminative direction between training and in-the-wild evaluation, with deepfakes shifting from lower to higher entropy.The authors suggest this inversion may help explain why distribution-alignment methods fail.Source: A. Feature Analysis · p. 3, Table IV · p. 3
B. Classification Performance
The five-expert speech ensemble performs strongly on a controlled benchmark, while speech and music results expose cross-domain challenges. A percentile-based music classifier improves cross-domain performance and exceeds a published music benchmark.
a) Speech Deepfake Detection
- On ASVspoof5, the five-expert ensemble reaches EER = 17.7% and AUC = 0.879, versus AASIST’s EER = 29.1% and RawNet2’s EER = 36.0%.On In-the-Wild, it reaches F1 = 0.679 and AUC = 0.718, but ensemble weights and threshold were tuned on held-out data from that dataset.Source: B. Classification Performance · p. 3, Table VI · p. 4
b) Music Deepfake Detection
- Music detection reaches F1=0.995 and AUC=1.000 in-distribution, while the original model reaches F1=0.667 cross-domain and the adaptive classifier reaches F1=0.938.The adaptive classifier does not require target-domain labels.Source: B. Classification Performance · pp. 3–4
c) Comparison to State-of-the-Art
- For cross-domain music evaluation, the adaptive approach reaches F1=0.938 versus CLAM’s F1=0.925, without target-domain training.On speech, the ensemble has slightly worse EER than RawNet2 on In-the-Wild, while using statistical features and XGBoost rather than deep neural networks.Source: B. Classification Performance · p. 4
C. Ablation Studies
The ablations show gains from adding complementary speech experts and calibrating the classification threshold, alongside cross-domain costs from standard adaptation methods. Simple top-N feature selection outperforms greedy forward selection.
a) Ensemble Size
- Ensemble F1 rises from 0.654 with one model to 0.679 with five experts, with diminishing returns after three.Even an expert with F1=0.301 contributes through complementary detection biases.Source: C. Ablation Studies · p. 4
b) Threshold Calibration
- Lowering the classification threshold from 0.50 to 0.30 increases F1 by 1.3 percentage points, trading precision for recall.The passage attributes this calibration issue to domain shift between training and evaluation data.Source: C. Ablation Studies · p. 4
c) Domain Adaptation Methods
- CORAL and target normalization degrade performance; the analysis attributes this to feature-label relationships reversing across domains.The adaptation methods align feature distributions while preserving learned feature-label relationships.Source: C. Ablation Studies · p. 4
d) Feature Selection Method
- Simple top-N selection by individual AUC outperforms greedy forward selection, which tends to overfit training-distribution artifacts.The final classifier uses eight jointly selected features.Source: C. Ablation Studies · p. 4
IV. DISCUSSION
The discussion reports a temporal approach outperforming a static whole-clip baseline on ASVspoof5 and examines music domain shift and computational costs.
Music Domain Shift
- On SONICS/FMA, mean-similarity evaluation AUC was 0.976, while entropy AUC collapsed to 0.496 despite a training AUC of 0.923.The mean feature had training AUC = 0.922; the authors attribute the selected features' failure to training-generator distributional artifacts.Source: §IV · p. 4
Temporal vs. Static Ablation
- EER fell by 8.2 percentage points with temporal analysis versus the best static model on ASVspoof5, under matched acoustic conditions.The temporal mixture-of-experts achieved EER = 17.7% (AUC = 0.879), compared with EER = 25.9% (AUC = 0.73) for the best static model.Source: §IV · pp. 4–5
Computational Efficiency
- A typical 10-second clip requires 36 pairwise similarity computations, while embedding extraction is the computational bottleneck at 1.1–1.3 files per second.The frozen encoder and up-to-29-feature XGBoost classifier can be used without expensive end-to-end training, and classifier training takes seconds on CPU.Source: §IV · p. 5
V. LIMITATIONS AND FUTURE WORK
The authors identify data, distribution-shift, and corpus-composition constraints, and point to recalibration and stronger authentic-music coverage as future needs.
Ensemble constraints
- The multi-expert ensemble requires training data from multiple source datasets, though adding diverse experts improves in-the-wild performance.The authors identify learned per-sample expert routing as a future direction.Source: §V · p. 5, Table XI · p. 5
Music adaptation
- The music adaptive classifier requires reference statistics from the target domain, although it does not require target labels.Source: §V · p. 5
Dataset limitations
- Music data are imbalanced: 144 real tracks comprising 1,008 stems versus 5,521 deepfake samples, making more authentic samples important for robustness and generalization.Class weighting is applied to mitigate the imbalance.Source: §V · p. 5
Distribution shift
- Feature-label inversion means models trained on current generators may require recalibration as synthesis technology evolves.The authors propose unsupervised detection of inversion at deployment as a route to automatic recalibration for novel distributions.Source: §V · p. 5
Corpus differences
- Residual spectral or mastering differences could partly explain the reported in-distribution AUC = 1.000, rather than synthesis artifacts alone.Source: §V · p. 5
VI. CONCLUSIONS
The conclusion presents temporal statistical analysis of CLAP embeddings as a lightweight, interpretable approach with competitive performance across speech and music benchmarks.
Conclusions
- 21 of 29 statistical features reverse discriminative direction between training and in-the-wild deployment, while entropy discriminates in opposite directions for speech and music deepfakes.Source: §VI · p. 5