Source-linked AI summary
Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization
Komal Chugh, Parul Gupta, Abhinav Dhall, Ramanathan Subramanian
TL;DR
Deepfake detection must address increasingly realistic manipulations affecting visual or audio content and their synchronization. The paper introduces a Modality Dissonance Score from chunk-wise audio-visual dissimilarity, supported by unimodal cross-entropy and inter-modal contrastive losses. Experiments on DFDC and DeepFake-TIMIT report up to 7% improvement over the state of the art and temporal forgery localization.
Problem
Deepfake detection is challenging because AI-generated videos can be nearly indistinguishable from genuine videos and may manipulate either visual or audio content.
Method
The method computes a Modality Dissonance Score by aggregating audio-visual dissimilarities over 1-second chunks, while learning unimodal discriminative features with cross-entropy and cross-modal similarity with contrastive loss.
Results
The framework achieves state-of-the-art performance on DFDC, improving AUC by up to 7%, and identifies manipulated video segments temporally.
Takeaways & Limitations
Segment-level audio-visual dissonance supports both deepfake classification and temporal forgery localization, including cases involving audio or visual manipulation.
Takeaways & Limitations
Future work is needed for transient forgery detection, real-time detection, and integration of human assessments such as EEG and eye-gaze sensing.
Abstract
from arXiv · showhide
We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, eg, loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as an aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.
1 INTRODUCTION
Deepfake detection is difficult because increasingly realistic AI-generated media can manipulate either visual or audio content. The paper detects such fakes through audio-visual dissonance, using segment-level comparison and complementary unimodal and contrastive learning.
- Motivation: AI-generated deepfakes increasingly produce naturalistic media that can evade conventional detection methods.Existing methods have primarily examined frame-level visual inconsistencies, while audio manipulation introduces an additional corruption channel.
- Motivation: Visual artifacts such as lip-sync failures and unnatural facial or lip movements characterize many deepfakes.These inconsistencies create dissonance between the audio and visual channels.
- Approach: The Modality Dissonance Score aggregates audio-visual dissimilarity across 1-second video chunks to label videos as real or fake.Chunk-wise analysis also supports fine-grained examination of temporal inconsistencies.
- Approach: Contrastive loss brings audio and visual features closer for real videos and farther apart for fake videos.Separate audio and visual subnetworks also learn discriminative real/fake features using cross-entropy loss.
- Contributions: The framework achieves state-of-the-art DFDC performance, improving AUC by up to 7%.The paper also proposes temporal forgery localization over manipulated video segments.
2 LITERATURE REVIEW
Prior deepfake detectors largely rely on spatial or temporal visual cues, whereas fewer approaches incorporate audio or cross-modal information. The paper positions its method as a segment-level audio-visual approach that detects dissonance and localizes short manipulated portions.
- Visual methods: Visual deepfake methods use cues from face modeling, rendering, synthesis, warping, reflections, illumination, landmarks, and other spatial inconsistencies.These approaches primarily identify artifacts within manipulated images or frames.
- Temporal methods: Temporal detectors exploit inconsistent blinking, facial-action-unit correlations, or recurrent models over frame-level CNN features.Temporal modeling addresses manipulations that may appear realistic in individual frames.
- Multimodal methods: Some prior work combines audio and video by comparing multimedia, emotion, facial-movement, speech, or lip-sync differences.Such multimodal approaches remain less common than visual-only detection methods.
- Positioning: The paper uses 1-second audio-visual segments to model temporal characteristics and enable localization when only a small video portion is forged.The authors state that prior works focused on assigning real/fake video labels rather than temporal localization.
- Positioning: The method applies contrastive loss to deepfake detection, extending its prior use for enforcing similarity between lip-synced audio-visual counterparts.This addresses cases where either the audio or video stream may be corrupted.
3 MDS-BASED FAKE VIDEO DETECTION
The method detects deepfakes by measuring audio-visual dissonance across short synchronized segments, combining modality-specific classification with contrastive learning. Segment dissimilarities are aggregated into MDS for video-level labeling, with a training-derived threshold for inference.
- 3 MDS-BASED FAKE VIDEO DETECTION: MDS aggregates segment-level audio-visual dissimilarities over the video length to provide a figure of merit for real/fake labeling.The dissimilarity is computed on 1-second chunks for fine-grained analysis.
- 3 MDS-BASED FAKE VIDEO DETECTION: The pipeline splits each input video into corresponding audio and visual segments and processes them through separate audio and video streams.The visual modality is face-tracked before stream processing.
- 3 MDS-BASED FAKE VIDEO DETECTION: The network learns modality-specific discriminative features with cross-entropy losses while contrastive loss brings real audio-visual features closer and pushes fake ones farther apart.This produces low expected MDS for real videos and high MDS for fake videos.
- 3 MDS-BASED FAKE VIDEO DETECTION: Visual features come from a 3D-ResNet-inspired stream, while MFCC audio features are encoded by a convolutional audio stream for cross-modal comparison.The visual and audio fc8 representations are used for contrastive learning.
- 3 MDS-BASED FAKE VIDEO DETECTION: Test videos are labeled by comparing MDS with a threshold determined from training-set averages for real and fake videos.The threshold is the midpoint between the average training MDS values of the two classes.
4 EXPERIMENTS
Experiments evaluate MDS on DFDC and DeepFake-TIMIT, including loss-function ablations, unimodal streams, temporal segments, comparative benchmarks, and forgery localization. Combined audiovisual training improves separability, while segment-level scores support localization of manipulated portions.
- Dataset Description: The experiments use public DFDC and DeepFake-TIMIT audio-visual deepfake datasets.
- Ablation Studies: 89.7% visual-stream AUC substantially exceeds 50.0% audio-stream AUC on DFDC.The audio stream performs poorly because the dataset contains minimal audio manipulation.
- Ablation Studies: A 1-second segment duration is empirically most optimal among the tested 1-, 2-, and 4-second durations.
- Evaluation on DFDC Dataset: On DFDC, MDS achieves 91.54% AUC overall and 93.50% without multiple-subject test videos, outperforming other compared approaches.The reported frame-wise AUC is 91.60%.
- Evaluation on DFTIMIT Dataset: On DeepFake-TIMIT, MDS achieves comparable AUCs of 97.92% for LQ and 96.87% for HQ, while misclassifying two LQ and three HQ samples.The test set contains 96 videos, making each misclassification consequential to overall AUC.
- Temporal Forgery Localization: Forgery localization labels segments above a dissimilarity threshold as fake and those below it as real.Figure 5 plots original segment labels against segment-level dissimilarity scores, with detected fake and real regions shown in red and blue.
5 CONCLUSION AND FUTURE WORK
The paper concludes that its MDS-based bimodal framework achieves state-of-the-art DFDC detection, with unimodal cross-entropy losses enhancing contrastive learning. It also demonstrates interpretability and forgery localization while identifying real-time detection and transient forgery handling as future priorities.
- MDS-based deepfake detection achieves state-of-the-art performance on the DFDC dataset.
- Unimodal cross-entropy losses provide additional benefit beyond contrastive loss, enhancing fake-detection performance.
- Audio-visual distance distributions, Grad-CAM attention regions, and forgery localization results demonstrate explainability and interpretability.
- Future work includes adding EEG and eye-gaze human assessments, exploring multiple instance learning for transient forgeries, and achieving real-time detection.The proposed processing of 1-second audio-visual segments is described as promising for real-time detection.