Source-linked AI summary
Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization
Komal Chugh, Parul Gupta, Abhinav Dhall, Ramanathan Subramanian
TL;DR
Deepfake detection remains challenging because methods have primarily focused on visual inconsistencies despite audio-based manipulation. This paper scores audio-visual dissonance across video segments and reports up to 7% AUC improvement on DFDC.
Problem
Realistic deepfakes are difficult to distinguish, while detection methods have primarily examined frame-level visual inconsistencies despite audio-based manipulation.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The approach demonstrates temporal forgery localization by identifying manipulated video segments.
Source: §1 · p. 2The authors identify real-time forgery detection as future work.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, eg, loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as an aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.
1 INTRODUCTION
The paper proposes detecting deepfakes through audio-visual dissonance: manipulated audio or video is expected to disrupt cross-modal harmony. It measures that dissonance over short video segments to detect and localize forgeries.
1 INTRODUCTION
- Deepfake detection is challenging because increasingly naturalistic AI-generated media can evade detectors, while manipulation may affect either the visual or audio channel.The introduction motivates audio-visual dissonance as a detection cue; prior work found that out-of-sync audio and video increased cognitive load.Source: §1 · p. 1
- The Modality Dissonance Score (MDS) aggregates audio-visual dissimilarity across 1-second chunks to label videos as real or fake.Chunk-wise analysis also enables localization of transient forgeries, where only some frames in a sequence are corrupted.Source: §1 · pp. 1–2
- The model combines contrastive loss, which brings audio and video features closer for real videos and farther apart for fake ones, with modality-specific cross-entropy losses.The paper reports that unimodal losses improve real-fake discriminability over contrastive loss alone and allow operation when only the audio or visual channel is available.Source: §1 · p. 2
- On DFDC, the method improves state-of-the-art AUC by up to 7%.Experiments on DFDC and DF-TIMIT also report up to 7% improvement over the state of the art.Source: §1 · p. 2
2 LITERATURE REVIEW
Prior work detects deepfakes using spatial and temporal visual cues, audio-visual comparisons, and combinations of visual features. This work is positioned around chunk-wise audio-visual dissonance to detect and localize transient manipulations.
2.1 Image-based
- Visual fakes include face swapping and reenactment (DeepFakes1, FaceSwap2, Face2Face); image-based detectors use facial artifacts, landmark-pose, and warped-region cues, [28], [13].Other designs include capsule networks for spoof detection [19], joint detection and segmentation [18], noise-residual and camera-characteristic streams [30], and MesoNet for fake-face classification [1].Source: §2.1 · p. 2, §2 · p. 2
2.2 Video-based
- Video-based detectors exploit temporal inconsistencies, including eye-blinking patterns, facial-action-unit correlations, and frame-level CNN features modeled with recurrent networks.These methods address cases where individual frames look realistic but manipulation warping is inconsistent over time.Source: §2.2 · p. 2
2.3 Audio-visual features based
- Audio-visual approaches compare multimedia and emotional facial-speech differences [16] or learn audio-visual closeness for in-sync versus out-of-sync clips.The lip-sync method is not specifically designed for deepfake detection, though lip-sync issues may occur in deepfakes.Source: §2.3 · p. 2
2.4 Bimodal Approaches
- Bimodal approaches can combine visual cues: recurrent networks model intraframe and inter-frame consistency, while other methods learn warped-face differences [13] and hierarchical frame relations [19].The paper notes that multiple visual cues can help address sophisticated visual manipulations.Source: §2.4 · pp. 2–3
2.5 Analysis of Related Work
- One-second audio-visual segments enable fine-grained temporal modeling and localization of manipulations, beyond prior methods that assign only video-level real/fake labels.The authors argue temporally consistent warping is difficult even over a few seconds, and small chunks can expose forgeries affecting only part of a video.Source: §2.5 · p. 3
- The paper applies contrastive loss, previously used for lip-sync detection, to deepfake detection and argues chunk-wise dissonance aggregation is more beneficial than video-level feature learning.The contrastive loss draws genuine audio-visual pairs closer; the proposed approach aggregates dissonance measures across small chunks over the video.Source: §2.5 · p. 3
- Unlike [16], the approach does not require emotional coherence or real–fake video pairs, and compares aggregated chunk-level features to localize transient forgeries.The authors note forged videos need not be emotional, and their training follows the traditional protocol rather than requiring paired videos.Source: §2.5 · p. 3
- Because many datasets lack audio, the architecture learns unimodal audio and visual features with cross-entropy alongside contrastive loss; the authors report state-of-the-art DFDC performance.They report that adding cross-entropy is more beneficial than using contrastive loss alone.Source: §2.5 · p. 3
3 MDS-BASED FAKE VIDEO DETECTION
The method detects deepfakes by aggregating audio–visual dissimilarity across 1-second chunks into a Modality Dissonance Score (MDS), expected to be higher for fake than real videos. It trains modality-specific features with cross-entropy and uses contrastive learning to separate real and manipulated audio–visual segments.
3.1 Overview
- The pipeline extracts audio with ffmpeg, divides audio and video into matching segments, tracks faces with S3FD, and processes paired segments through audio and visual streams.The streams are trained with contrastive and modality-specific cross-entropy losses; the unimodal losses still support feature learning when either modality is missing.Source: §3.1 · p. 3
3.2 Visual Stream
- The visual stream uses a 3D-ResNet-inspired architecture [10], with its fc8 feature representation used for contrastive learning and a two-neuron classification layer producing a visual prediction.Source: §3.2 · p. 3
3.3 Audio Stream
- The audio stream converts each segment into MFCC heat maps with 13 mel-frequency bands per time step, then uses a CNN-based stream whose fc8 features support contrastive learning and whose classifier supplies audio cross-entropy.MFCC features are used in speaker and speech recognition.Source: §3.3 · p. 4
3.4 Loss functions
- Contrastive loss pushes audio–visual feature distances higher for manipulated videos and lower for real videos, while weighted unimodal cross-entropy losses train the visual and audio classifiers.The combined objective is a weighted sum of the contrastive and two cross-entropy losses.Source: §3.4 · p. 4
3.5 Test Inference
- At test time, the method aggregates segment dissimilarities into each video’s MDS and labels it by comparing that score with a threshold set midway between training-set real and fake mean MDS values.The threshold is determined from the training set.Source: §3.5 · p. 5
4 EXPERIMENTS
Experiments evaluate the MDS network on DFDC and DeepFake-TIMIT, testing its loss components and reporting benchmark results and temporal forgery localization. The combined-loss model achieves 91.54% AUC on DFDC, while localization assigns segment labels using a dissimilarity threshold.
4.1 Dataset Description
- The experiments use two public audio-visual datasets; DeepFake-TIMIT contains 16 similar-looking pairs selected from VIDTIMIT and manipulated with a GAN-based approach.Its 32 subjects each have 10 videos, with low-quality 64 × 64 and high-quality 128 × 128 versions.Source: §4.1 · p. 5
4.2 Training Parameters
- Both datasets use 1-second segments and a 0.99 margin; evaluation uses video-wise AUC.DFDC training used 100 epochs and batch size 8, while DeepFake-TIMIT used 50 epochs and batch size 16; both used Adam at learning rate 0.001.Source: §4.2 · p. 6
4.3 Ablation Studies
- 92.7% AUC with combined losses exceeds 86.1% with contrastive loss alone on the DFDC ablation set.The real/fake MDS distributions overlap less with all three losses, as the gap between their average MDS scores widens.Source: Figure 2 · p. 5, §4.3 · p. 6, Table 3 · p. 7
- 89.7% for the visual-only stream exceeds the audio-only stream’s 50.0% on DFDC.Audio inter-pair L2 distances cluster near zero, while visual distances are spread across the dataset; the paper attributes weaker audio performance to limited audio manipulation.Source: §4.3 · p. 6, Figure 3 · p. 6
- D=1 second is the most effective tested segment duration among 1, 2, and 4 seconds on DFDC.The paper attributes this to more segments per video, allowing finer-grained comparison of audio and visual data.Source: §4.3 · pp. 6–7
4.4 Evaluation on DFDC Dataset
- 91.54% AUC on DFDC outperforms the compared visual-only and audio-visual methods, with a reported ~8% relative gain over the other audio-visual approach [16].The result uses 18,000 videos and a 3,281-video test set; excluding videos with two subjects gives 93.50%, and frame-wise AUC is 91.60%.Source: §4.4 · p. 7, Table 2 · p. 7
4.5 Evaluation on DFTIMIT Dataset
- 97.92% LQ and 96.87% HQ AUC on DeepFake-TIMIT are comparable to [13], whose scores are 99.9% LQ and 99.7% HQ.The test set contains 96 videos; the method misclassified 2 LQ and 3 HQ samples. On DFDC, [13] achieves 72.7% AUC, while this method reports an 18% gain; frame-wise AUC is 98.3% LQ and 94.7% HQ.Source: §4.5 · p. 7, Table 2 · p. 7
4.6 Temporal Forgery Localization
- Segment-level dissimilarity enables temporal localization: segments above a threshold are labeled fake, and those below it are labeled real.The demonstrations mix real and fake segments from the same subject at random locations in two DFDC videos; the authors present localization as making the framework more interpretable.Source: §4.6 · p. 8, Figure 5 · p. 8
5 CONCLUSION AND FUTURE WORK
The conclusion presents an MDS-based bimodal detector that combines cross-modal contrastive learning with unimodal discrimination, reports state-of-the-art DFDC performance, and outlines human-informed, transient, and real-time detection as future directions.
5 CONCLUSION AND FUTURE WORK
- The MDS-based framework achieves state-of-the-art DFDC performance; contrastive segment-level audiovisual features separate genuine from fake streams, with unimodal cross-entropy adding benefit.The contrastive loss constrains genuine audio-visual streams to be closer than fake counterparts, while cross-entropy trains each modality to learn discriminative features.Source: §5 · p. 8
- Interpretability is demonstrated through real-versus-fake audio-visual distance distributions, Grad-CAM attention regions, and forgery localization results.Source: §5 · p. 8
- Future work proposes adding EEG and eye-gaze human assessments, exploring multiple-instance learning for transient forgery detection, and pursuing real-time detection.The real-time direction is motivated by the promise of processing audio-visual information over 1-second segments.Source: §5 · p. 8