Source-linked AI summary
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, Dinesh Manocha
TL;DR
Deepfake detection has largely focused on visual evidence, motivating methods that use relationships between audio and video within the same clip. This paper compares cross-modal similarity and perceived-emotion cues using a Siamese-style network with triplet loss, reporting 84.4% per-video AUC on DFDC and 96.6% on DF-TIMIT. The authors also identify limitations involving emotion mismatches in real videos and multiple people per video.
Problem
Prior deepfake detection methods largely focus on visual cues, while using multiple modalities and their perceived-emotion relationships remains less explored.
Method
The method compares audio-visual similarity and perceived-emotion cues within videos using a Siamese network-based architecture trained with triplet loss.
Results
84.4% per-video AUC on DFDC improves about 9% over SOTA, while the method reports 96.6% per-video AUC on DF-TIMIT.
Takeaways & Limitations
Perceived-emotion information can help detect deepfake content by revealing mismatches between audio and facial modalities.
Takeaways & Limitations
The approach can misclassify real videos when their modalities express perceived emotions differently and may require extension for videos containing multiple people.
Abstract
from arXiv · showhide
We present a learning-based method for detecting real and fake deepfake multimedia content. To maximize information for learning, we extract and analyze the similarity between the two audio and visual modalities from within the same video. Additionally, we extract and compare affective cues corresponding to perceived emotion from the two modalities within a video to infer whether the input video is "real" or "fake". We propose a deep learning network, inspired by the Siamese network architecture and the triplet loss. To validate our model, we report the AUC metric on two large-scale deepfake detection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach with several SOTA deepfake detection methods and report per-video AUC of 84.4% on the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of our knowledge, ours is the first approach that simultaneously exploits audio and video modalities and also perceived emotions from the two modalities for deepfake detection.
1 INTRODUCTION
The paper addresses deepfake detection by comparing audio and visual modalities within the same video, including their perceived-emotion cues. It introduces a Siamese-style learning approach and evaluates it against benchmark datasets and prior methods.
- Motivation: The method detects falsification by exploiting relationships between facial and speech modalities extracted from the same video.Prior work identified correlations between audio-visual modalities and compatible affective cues in a shared space.
- Approach: The approach simultaneously models audio, video, and perceived-emotion features from both modalities.It obtains modality and emotion embedding vectors for real and fake videos containing the same subject.
- Approach: A Siamese network-based architecture with triplet loss maximizes cross-modal similarity for real videos and minimizes it for fake videos.Training compares a real video with its corresponding deepfake.
- Novelty: The proposed deep learning approach models similarity or dissimilarity between facial and speech modalities for deepfake detection.Perceived emotion information is additionally used to detect modality-signal similarity or dissimilarity.
- Evaluation: 84.4% per-video AUC on DFDC improves about 9% over SOTA, while performance is at-par with prior methods on DF-TIMIT.The model is evaluated on the DeepFake-TIMIT Dataset and DFDC.
2 RELATED WORK
Prior deepfake detectors largely focus on visual artifacts, while multimodal deepfake detection and affective-cue analysis provide complementary directions. The paper positions its audio-visual affective approach within this related work and evaluates it on datasets containing both modalities.
- Positioning: The proposed method is presented as complementary to other deep learning-based approaches for media forensics and deception detection.Its focus is exploiting mismatch between two modalities.
- Unimodal detection: Deepfake detection has commonly relied on frame-level facial artifacts, temporal features, head poses, and inconsistencies in eyes, teeth, or facial contours.These approaches primarily analyze visual evidence from manipulated videos.
- Multimodal detection: Few prior methods exploit multiple modalities from the same video, although existing work studies fake talking heads, lip-sync inconsistencies, and multimodal deception detection.The paper distinguishes its modality-mismatch approach from these methods.
- Datasets: Deepfake datasets differ in modalities and manipulation types, with DFDC containing manipulated faces, audio, or both.Only DFDC and DF-TIMIT among the discussed datasets contain both audio and video for the proposed analysis.
- Affective computing: Affective-computing research links perceived emotion across modalities and motivates comparing visual and auditory affective features.The cited literature argues that features representing the same emotion can remain correlated despite differing appearances.
3 OUR APPROACH
The approach detects deepfakes by comparing face and speech embeddings, including perceived-emotion embeddings, using shared networks, similarity scores, and triplet-based training. At test time, distances between the modalities are combined and thresholded to classify a single video as real or fake.
- 3.1 Problem Statement and Overview: The model uses shared modality and perceived-emotion networks on paired real and fake videos from the same subject.The architecture operates on face and speech features through F1, F2, S1, and S2, reusing weights across the two inputs.
- 3.4 Training Routine: The training objective combines two triplet-style similarity losses for modality relationships and affective-cue relationships.The losses compare distances across real and fake face-speech embeddings and propagate their cumulative value through the network.
- 3.5 Testing Routine: At testing, the model computes modality and emotion distances, adds them, and labels the video fake when the sum exceeds the learned threshold τ.The threshold is empirically derived from distance values on real and fake training videos.
- 3.2 F1 and S1: Video/Audio Modality Embeddings: F1 and S1 produce 250-dimensional unit-normalized embeddings for face and speech modalities.They use convolutional, pooling, fully connected, ReLU, and final unit-normalization layers.
- 3.3 F2 and S2: Video/Audio Perceived Emotion Embedding: F2 and S2 produce 250-dimensional unit-normalized affect embeddings using single-view MFNs pretrained on CMU-MOSEI.The perceived-emotion space includes six discrete emotions plus neutral, with face and speech treated as separate views.
4 IMPLEMENTATION AND EVALUATION
The evaluation uses DF-TIMIT and DFDC, the two datasets identified as containing both face and speech modalities, with train-test splits and dataset-specific training settings. Facial and speech inputs are extracted using OpenFace and pyAudioAnalysis before model evaluation.
- Datasets: The experiments use the DF-TIMIT and DFDC datasets, split into 85% training and 15% testing sets.The full DF-TIMIT dataset was used, while 18,000 DFDC videos were randomly sampled because of computational overhead.
- Training Settings: DFDC training uses batch size 128 for 500 epochs, whereas DF-TIMIT uses batch size 32 for 100 epochs.Both settings use Adam with a learning rate of 0.01.
- Feature Extraction: OpenFace extracts 430-dimensional facial features, while pyAudioAnalysis extracts 13 Mel Frequency Cepstral Coefficients for speech.The facial features include landmarks, head pose orientation, and gaze features.
5 RESULTS AND ANALYSIS
The evaluation compares the model with prior methods on DFDC and DF-TIMIT, then examines qualitative behavior, learned correlations, ablations, and failure cases.
- Comparison with SOTA Methods: DF-TIMIT contains face-centered videos, whereas DFDC includes full-body poses with faces occupying less than 50% of each frame.The authors connect this dataset difference to the relative usefulness of face-warping artifact detectors.
- Qualitative Results: Real-video examples show matching perceived-emotion labels across speech and face, while fake examples show mismatches between the modalities.In one DFDC example, the real video is neutral in both modalities, whereas the fake video is surprise in the face and neutral in speech.
- Interpreting the Correlations: Real-video embeddings commonly have smaller face-speech separation around 0.2 than fake-video embeddings around 0.5.The embedding-distance visualization uses unit-normalized face and speech embeddings from 1,000 randomly chosen DFDC training points.
- Ablation Experiments: Ablation experiments report that modality embedding similarity achieves better AUC scores than perceived emotion embedding similarity alone.The study removes one correlation at a time to assess each similarity signal.
- Failure Cases: The model can misclassify fake videos as real when incorrect perceived-emotion embeddings fail to reveal a cross-modal mismatch.The reported failure cases include one sample from each dataset.
6 CONCLUSION, LIMITATIONS AND FUTURE WORK
The paper concludes that combining audio-visual similarity with affective-cue similarity supports deepfake detection, while noting misclassification and multi-person-video limitations. Future work considers additional modalities, context, and visual artifact cues.
- Conclusion: The method uses audio-visual modality similarity and affective-cue similarity to infer whether videos are real or fake.The approach is evaluated on DFDC and DF-TIMIT.
- Limitations: Real videos can be falsely classified as fake when people express perceived emotions differently across modalities.The authors also report that fake videos lacking such emotional mismatches may be classified as real.
- Limitations: Many deepfake datasets contain multiple people per video, motivating extensions that model the perceived emotional state of multiple persons.The authors identify this as a limitation and propose a corresponding detection scheme.
- Future Work: Future work proposes incorporating more modalities and contextual information, alongside visual artifact cues and improved audio-cue methods.Suggested visual cues include lip-speech synchronization, head-pose orientation, and artifacts in teeth, nose, and eyes across frames.