Source-linked AI summary
Cross-modal Translation via Conditional Latent Denoising for Video Deepfake Detection
Xinzhe Li, Youzhi Tu, Kong Aik Lee
TL;DR
Video deepfake detectors often overlook audio–visual correspondences, despite their relevance to detection. CTCLD factorizes the joint distribution and uses conditional latent denoising to translate between modalities; it reports competitive results on FakeAVCeleb and DFDC.
Problem
Existing approaches often overlook audio–visual correspondences, while independent-stream modeling and simple mappings can leave cues underused or degrade information transfer.
Source: §1 · p. 1Method
CTCLD decomposes the audio–visual joint distribution and uses conditional latent denoising for bidirectional cross-modal translation.
Source: §1 · p. 1, §2.1 · p. 2, §3.2 · p. 4Results
99.4% Acc and 99.5% AUC on FakeAVCeleb, and 98.5% Acc and 99.9% AUC on DFDC.
Source: §4 · p. 4Takeaways & Limitations
The authors envision generative translation as a paradigm that may inspire broader applications in multimodal signal processing.
Source: §4 · p. 4
Abstract
from arXiv · showhide
The growing threat of video deepfakes necessitates multimodal detection. Beyond serving as independent indicators of authenticity, audio and visual signals have intrinsic dependencies that also provide an essential criterion for detection. Previous methods often overlook the cross-modal correspondences, hindering information transfer between domains and leaving crucial detection cues unexplored. To address this challenge, we propose a framework called Cross-modal Translation via Conditional Latent Denoising (CTCLD) for video deepfake detection. It connects the distinct distributions of heterogeneous modalities in latent spaces, enabling smooth cross-domain information transfer to improve detection performance. We first establish a Bayesian foundation by decomposing the audio-visual joint distribution. Subsequently, CTCLD translates both modalities via bidirectional latent denoising conditioned on each other, effectively capturing subtle inconsistencies in the manipulated signals. Experimental results demonstrate that the proposed CTCLD enables comprehensive domain alignment, resulting in a robust video deepfake detection approach with competitive performance.
1. INTRODUCTION
Video deepfake detection can use audio–visual correspondences, but existing approaches often model modalities independently or use simple mappings that degrade information. CTCLD uses Bayesian factorization and bidirectional latent denoising to transfer information across modalities, with competitive detection performance across datasets.
1. INTRODUCTION
- Prior methods either model audio and visual streams independently or use simple cross-domain mappings that can degrade information.These approaches leave cross-modal correspondences underused as detection cues.Source: §1 · p. 1
- CTCLD treats cross-modal correspondences as an essential detection criterion and decomposes the audio–visual joint distribution into marginal and conditional distributions.The factorization provides the basis for bridging heterogeneous modalities.Source: §1 · p. 1
- CTCLD trains modality-specific encoders on authentic videos, then models conditional distributions from their priors through bidirectional latent denoising.The framework uses these translated representations to retrieve detection criteria for isolating subtle manipulation artifacts.Source: §1 · p. 1
- The authors report that CTCLD better bridges heterogeneous modalities and achieves competitive detection performance across datasets.The stated goal is to exploit cross-modal correspondences as video deepfake detection criteria.Source: §1 · p. 1
2. METHODOLOGY
CTCLD decomposes audio-visual correspondence into modality-specific marginals and bidirectional conditional distributions. It uses those cross-modal priors to guide latent denoising in both directions, then integrates the reconstructed representations for forgery detection.
2.1. Formulation of Cross-Modal Correspondences
- The joint audio-visual latent distribution is factorized into two modality-specific marginals and two bidirectional conditional distributions.The marginals come from modality-specific encoders, while the conditionals represent cross-modal distributions.Source: §2.1 · p. 2
2.2. Cross-Modal Translation
- Audio and visual encoders produce latent embeddings, and each modality guides a translation and denoising process that reconstructs the other.The audio-conditioned visual prior and visual-conditioned audio prior condition their respective latent reconstructions.Source: §2.2 · p. 2
- A Gaussian corruption chain adds noise over T timesteps, and the reverse process reconstructs visual latents from noise conditioned on the audio-derived prior.The visual-to-audio direction is also optimized with a corresponding denoising objective.Source: §2.2 · pp. 2–3
- The denoising network predicts injected noise; its adaptive blocks prioritize global cross-modal alignment at high noise and localized correspondences at low noise.The visual-side objective reduces estimation of the reverse-process mean to noise prediction.Source: §2.2 · p. 3
- CTCLD concatenates reconstructed modality representations for a detection head, with an objective combining forgery classification and cross-modal reconstruction.The reconstruction term is balanced against binary cross-entropy by hyperparameter λ.Source: §2.2 · p. 3
3. EXPERIMENTS
CTCLD achieves the best reported performance on FakeAVCeleb and competitive performance on DFDC, with experiments also examining bidirectional translation, learned representations, and sensitivity to hyperparameters.
3.1. Implementation Details
- Encoders were pretrained on VoxCeleb2, then CTCLD trained on FakeAVCeleb and DFDC using 16 sampled frames and 16-kHz Mel-spectrogram audio.The sampled frames come from 3.2-second clips; the Mel-spectrograms use 128 mel-frequency bins.Source: §3.1 · p. 3
3.2. Overall Comparison and Ablation Study
- CTCLD achieves best performance across both datasets: FakeAVCeleb 99.4% Acc and 99.5% AUC, and DFDC 98.5% Acc and 99.9% AUC.On FakeAVCeleb, it outperforms the second-best method by 0.8% in Acc and 0.4% in AUC.Source: Table 1 · p. 4, §3.2 · p. 4
- Removing either audio- or visual-conditioned latent denoising lowers Acc and AUC, supporting the use of bidirectional translation to capture complementary inconsistencies.The ablation is conducted on FakeAVCeleb.Source: §3.2 · p. 4, Table 2 · p. 4
3.3. Visualization and Sensitivity Analysis
- The concatenated CTCLD representations show clear separation between real and fake videos in the t-SNE visualization.The visualization is used to qualitatively assess the learned representations' discriminative capability.Source: Figure 2 · p. 4, §3.3 · p. 4
- Competitive results persist across reasonable λ values, while excessive weighting degrades performance through an inappropriate shift in optimization focus.The tested λ values are 10^-4, 10^-3, 10^-2, 10^-1, and 1.Source: Figure 3 · p. 4, §3.3 · p. 4
- Detection performance decreases as sampling steps increase, while coarse-grained sampling preserves subtle manipulation artifacts that denser sampling tends to reconstruct.The tested sampling-step counts are 5, 10, 20, 40, and 80.Source: Figure 3 · p. 4, §3.3 · p. 4
4. CONCLUSION
CTCLD bridges heterogeneous audio and visual modalities in latent space to capture subtle multimodal inconsistencies in video deepfakes. It achieves the reported Acc and AUC scores on FakeAVCeleb and DFDC, and the authors envision broader multimodal signal-processing applications.
- CTCLD bridges heterogeneous modalities in latent space to capture subtle multimodal inconsistencies in video deepfakes.Source: §4 · p. 4
- On FakeAVCeleb, CTCLD achieves 99.4% Acc and 99.5% AUC; on DFDC, it achieves 98.5% Acc and 99.9% AUC.Source: §4 · p. 4
- The authors envision the generative translation paradigm inspiring broader applications in multimodal signal processing.Source: §4 · p. 4