AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

• The method targets audio-visual cues such as correspondence between mouth visemes and speech phonemes, which deepfakes can struggle to replicate faithfully.Source: §1 · p. 1
• Prior supervised correspondence and labeled-dataset methods may overlook subtle audio-visual cues that could help detect unseen deepfakes.The introduction contrasts supervised contrastive methods with single-stage classification trained on labeled deepfake datasets.Source: §1 · p. 2
• AVFF first learns representations self-supervised on real face videos, then trains a supervised classifier to distinguish real and fake videos.The representation stage combines contrastive learning and autoencoding, with complementary masking and cross-modal feature fusion.Source: §1 · p. 2
• 98.6% accuracy and 99.1% AUC on FakeAVCeleb, surpassing the existing audio-visual state-of-the-art by 14.9% and 9.9%, respectively.Source: §1 · p. 2

2.1. Multi-Modal Representation Learning

• AVFF builds on multimodal encoders and masked autoencoders but differs through post-encoding complementary masking and replacing masked tokens with predictions from the other modality.Prior work includes SyncNet, CLIP [51], AudioCLIP [20], AV-MAE [18], and CAV-MAE [19].Source: §2.1 · p. 2

2.2. Deepfake Detection

• Visual-only deepfake detectors use lip movements, inconsistent head pose, and eye blinking as artifacts; FTCN combines a CNN and transformer to exploit short-time and long-time temporal incoherence.LipForensics [21] relies on lip movements; XceptionNet [52] is trained end-to-end as a CNN.Source: §2.2 · pp. 2–3
• Audio-visual detectors compare affective cues, modality dissonance, voice-face matching, or phoneme-viseme mismatch, while newer methods fuse features through cross-attention or embedding-space fusion.Examples include Emotions Don’t Lie, MDS, VFD [7], AV-DFD, AVFakeNet, and AVoiD-DF.Source: §2.2 · p. 3

3. Method

• AVFF uses self-supervised representation learning on real face videos before supervised classification of real and fake videos.The first stage serves as pretraining for the downstream deepfake-detection task.Source: §3 · p. 4

3.1. Preprocessing

• Visual frames are sampled at 5 fps, facial regions are cropped to remove background, and audio waveforms are sampled at 16 kHz and converted to log-mel spectrograms.The paper motivates removing background because it typically has minimal correspondence with speech audio.Source: §3.1 · p. 4

3.2. Representation Learning Stage

• Audio and video inputs are patch-tokenized and divided into eight temporal slices aligned to the same time intervals.Audio uses 16 × 16 patches and video uses 2 × 16 × 16 spatiotemporal patches.Source: §3.2 · p. 4
• Representation learning combines a bidirectional audio-visual contrastive loss with autoencoding losses for reconstruction and adversarial training.The contrastive objective enforces similarity between audio and visual embeddings from the same sample.Source: §3.2 · p. 5

3.3. Deepfake Classification Stage

• The supervised classifier concatenates each modality’s unimodal and cross-modal features, reduces patches, and predicts whether the video is real or fake.It is trained with cross-entropy on labeled real and fake videos.Source: Figure 3 · p. 5, §3.3 · pp. 5–6

3.2. Representation Learning Stage

• Complementary masks hide corresponding slices in opposite modalities, and cross-modal networks replace each masked slice with a feature from the other modality for reconstruction.The masked-slice autoencoding objective makes the decoder learn from the other modality, explicitly enforcing audio-visual correspondence.Source: §3.2 · pp. 4–5

3.3. Deepfake Classification Stage

• At inference, videos are split into overlapping blocks with a step size of one temporal slice, and their output logits are averaged for the classification decision.The block length matches the sample length used in training.Source: §3.3 · p. 6

4.1. Implementation

• Stage 1 uses only real LRS3 videos for representation learning; Stage 2 trains a supervised classifier on real and fake FakeAVCeleb videos.FakeAVCeleb includes videos with visual, audio, or both modalities synthesized using several deepfake algorithms.Source: §4.1 · p. 6

4.2. Evaluation and Discussion

• Evaluation covers intra-dataset performance, cross-manipulation generalization, and cross-dataset generalization, reporting accuracy, average precision, and AUC across multiple runs.For audio-visual methods, a video is fake if either or both modalities are manipulated; for visual-only methods, only visual manipulation counts as fake.Source: §4.2 · p. 6
• 14.9% accuracy (+9.9% AUC) over AVoiD-DF and 8.7% accuracy (+4.5% AUC) over RealForensics on FakeAVCeleb’s 70%-30% split.The evaluation uses a 70% training and 30% unseen test split.Source: Table 1 · p. 6, §4.2 · p. 6
• AUC > 92+% and AP > 93+% across held-out FakeAVCeleb categories, with the best performance in almost all cases or at par with the rest.The method is evaluated by leaving one category out for testing while training on the others.Source: §4.2 · p. 7, Table 2 · p. 7
• AVFF performs at par with RealForensics when trained on FakeAVCeleb and tested on a subset of KoDF.Table 3 evaluates cross-dataset generalization on KoDF.Source: §4.2 · p. 7, Table 3 · p. 7
• The learned embeddings separate real and fake videos and distinguish deepfake categories, including categories generated by algorithms not encountered during representation learning.Stage 1 uses only real videos, while the t-SNE samples come from FakeAVCeleb; adjacent clusters can share a generation algorithm.Source: §4.2 · p. 7, Figure 4 · p. 8

5. Ablation Study

• Removing the autoencoding objective reduces downstream performance, with complementary masking, cross-modality fusion, and decoding also deactivated.The contrastive-only variant uses encoder-output embeddings for downstream training.Source: §5 · p. 8
• Replacing A2V/V2A masked-token prediction with shared learnable masked tokens diminishes performance, especially AP.The authors describe cross-modal fusion as supplementing each modality's representation with information from the other.Source: §5 · p. 8
• Replacing complementary masking with random masking produces a notable drop in AP and AUC.The authors attribute the drop to difficulty learning audio-visual correspondences.Source: §5 · p. 8
• Using the combined embeddings improves performance over using either the unimodal or cross-modal embeddings in isolation.The classification stage concatenates the feature embeddings with the cross-modal embeddings.Source: §5 · p. 8
• The proposed AVFF pipeline performs best among the considered ablations.Table 4 presents the ablation evaluation; the supplied caption does not give numerical values.Source: Table 4 · p. 8
• Replacing the unimodal patch-reduction networks with mean pooling slightly dents performance.The authors suggest linear averaging may discard subtle cues compared with an MLP's weighted nonlinear averaging.Source: §5 · p. 8

6. Conclusion, Limitations, and Future Work

• AVFF uses self-supervised learning to capture audio-visual correspondence, then supervised learning to classify deepfakes.The representation stage combines contrastive learning with an autoencoding objective using complementary masking and cross-modal fusion.Source: §6 · p. 8
• The method relies on audio-visual coherence and requires both modalities, making asynchronous or mismatched audio, multiple speakers, and occlusions challenging.Videos containing only audio or only visual content are unsupported.Source: §6 · p. 8
• Future work includes preserving unimodal cues, adapting the approach to emotion recognition, and generalizing to non-humanoid audio-driven videos.The authors particularly mention diffusion-based approaches for the latter direction.Source: §6 · p. 8

A. Overview

• Section B covers implementation details.Source: A. Overview · p. 9
• Section C covers dataset details, while Section D presents additional results.Source: A. Overview · p. 9
• Section E presents visual examples of challenging scenarios.Source: A. Overview · p. 9

B.1. Representation Learning Stage

• Stage 1 samples real LRS3 clips of 3.2 seconds, with 16 visual frames and 768 audio frames, and pairs clips from the same video at different times for contrastive learning.Audio is represented as a 128-bin Mel-spectrogram; frames are resized to 224 × 224.Source: §B.1 · p. 9
• Stage 1 uses VideoMAE-based ViT-B encoders and decoders, with A2V/V2A networks comprising a token-matching linear layer and one transformer block.Source: §B.1 · p. 9
• Stage 1 initializes audio and visual components from AudioMAE and MARLIN checkpoints, then trains end-to-end for 500 epochs with AdamW.Training uses a 1.5e-4 learning rate, 40 warmup epochs, batch size 32, and four RTX A6000 GPUs.Source: §B.1 · p. 9

B.2. Deepfake Classification Stage

• Stage 2 samples FakeAVCeleb videos with manipulated audio, visuals, or both, using weighted sampling to mitigate class imbalance.Unlike Stage 1, it does not draw another clip from the same video for contrastive learning.Source: §B.2 · p. 9
• Stage 2 uses three-layer MLPs for unimodal patch reduction and a four-layer MLP classifier, retaining the representation-learning architecture.Source: §B.2 · p. 9
• Stage 2 initializes from the Stage 1 checkpoint and trains end-to-end for 50 epochs with AdamW.Training uses a maximum learning rate of 1.0e-4, batch size 32, and four RTX A6000 GPUs; Table 5 tests a FakeAVCeleb-trained model on DF-TIMIT and a DFDC subset.Source: §B.2 · p. 9, Table 5 · p. 9

C. Dataset Details

• LRS3 contains 5,594 real videos spanning over 400 hours of TED and TED-X talks, with synchronized audio and face-containing frames.The videos are in English.Source: C. Dataset Details · p. 9
• FakeAVCeleb contains 20,000 clips: 500 real videos and 19,500 deepfakes spanning real/fake audio and visual combinations.Its manipulations include FaceSwap, FaceSwapGAN, and Wav2Lip variants.Source: C. Dataset Details · p. 9
• KoDF includes 62K+ real and 175K+ synthetic videos, with a subset used to evaluate cross-dataset generalization.The fakes use six algorithms, including FaceSwap, DeepFaceLab, and Wav2Lip.Source: C. Dataset Details · pp. 9–10
• Figure 5 compares visual-perturbation AUC on FakeAVCeleb’s test set and reports greater robustness than RealForensics.The figure shows results across perturbation intensity levels.Source: Figure 5 · p. 10
• DF-TIMIT’s 320 higher-quality videos and a 3,215-video DFDC subset are used for cross-dataset generalization evaluation.DF-TIMIT uses FaceSwapGAN manipulations; DFDC includes samples with fake audio.Source: C. Dataset Details · p. 10

D. Additional Results

• The supplementary experiments extend the main-paper results to provide a more comprehensive evaluation and strengthen confidence in the approach’s efficacy.The supplied passage does not specify the experiments or their outcomes.Source: D. Additional Results · p. 10

D.1. Cross-Dataset Generalization

• The model achieves the best cross-dataset generalization performance on both DF-TIMIT and DFDC among the compared baselines.Comparison was limited to open-source baselines because other models or results could not be obtained.Source: §D.1 · p. 10

D.2. Robustness to Unseen Perturbations

• Across five intensity levels, visual tests cover saturation, contrast, block-wise distortion, Gaussian noise, blur, JPEG, and video compression; the model exceeds RealForensics in most cases, especially three listed distortions.The model was trained without augmentations; the comparison is on FakeAVCeleb’s test set.Source: §D.2 · pp. 10–11
• Audio tests find overall robustness, with slight performance declines under stronger Gaussian noise and pitch shift, minimal reverberance fluctuations, and lower AP under high-intensity compression.The tests vary Gaussian noise, pitch, reverberance, and compression across five intensity levels.Source: §D.2 · p. 11, Figure 6 · p. 11

D.3. Classification Performance on the Learned Representation

• With encoders and A2V/V2A networks frozen, both the downstream MLP and RBF-kernel SVM achieve reasonably high accuracy and AUC on the learned representation.This is presented as evidence that the Stage 1 representation is discriminative for real-versus-fake classification.Source: §D.3 · p. 11, Table 7 · p. 11

E. Visual Examples of Challenging Scenarios

• Near-profile poses and face occlusions with masks or hands can make it challenging to establish the audio-visual correspondence the method relies on.The paper identifies scenarios where correspondence cannot be established as challenging for detection.Source: E. Visual Examples of Challenging… · p. 11, Figure 7 · p. 11