PaperlayerView exact revision on arXiv

Source-linked AI summary

Voice-Face Homogeneity Tells Deepfake

Harry Cheng, Yangyang Guo, Tianyi Wang, Qi Li, Xiaojun Chang, Liqiang Nie

arXiv:2203.02195v3cs.CVcs.AIcs.MM

TL;DR

Artifact-based deepfake detectors face poor generalization across datasets with differing manipulation algorithms. This paper proposes voice-face matching, using generic-data pretraining followed by deepfake-data fine-tuning; it reports state-of-the-art results across tested datasets and competitive performance with limited fine-tuning data.

  • Problem

    Artifact-based detectors can generalize poorly across datasets because manipulation algorithms and salient facial regions differ.

    Source: §1 · p. 1

  • Method

    VFD learns voice-face matching on generic audio-visual data, then fine-tunes on deepfake data with an upgraded real-fake contrastive loss.

    Source: §1 · p. 2

  • Results

    VFD reports state-of-the-art performance across DFDC, FakeAVCeleb, and DeepfakeTIMIT, including AUCs of 85.13% on DFDC and 86.11% on FakeAVCeleb.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    The reported results support voice-face matching as a deepfake-detection approach that uses paired voices and faces and depends less on auxiliary deepfake data.

    Source: §1 · p. 2

    When both voices and faces are maliciously edited, VFD may suffer matching collapse; the paper notes that this setting lacks studied high-quality datasets and approaches.

    Source: §5 · p. 11

Abstract

from arXiv · show

Detecting forgery videos is highly desirable due to the abuse of deepfake. Existing detection approaches contribute to exploring the specific artifacts in deepfake videos and fit well on certain data. However, the growing technique on these artifacts keeps challenging the robustness of traditional deepfake detectors. As a result, the development of generalizability of these approaches has reached a blockage. To address this issue, given the empirical results that the identities behind voices and faces are often mismatched in deepfake videos, and the voices and faces have homogeneity to some extent, in this paper, we propose to perform the deepfake detection from an unexplored voice-face matching view. To this end, a voice-face matching method is devised to measure the matching degree of these two. Nevertheless, training on specific deepfake datasets makes the model overfit certain traits of deepfake algorithms. We instead, advocate a method that quickly adapts to untapped forgery, with a pre-training then fine-tuning paradigm. Specifically, we first pre-train the model on a generic audio-visual dataset, followed by the fine-tuning on downstream deepfake data. We conduct extensive experiments over three widely exploited deepfake datasets - DFDC, FakeAVCeleb, and DeepfakeTIMIT. Our method obtains significant performance gains as compared to other state-of-the-art competitors. It is also worth noting that our method already achieves competitive results when fine-tuned on limited deepfake data.

1 INTRODUCTION

The paper proposes deepfake detection through voice-face matching, motivated by dataset-specific artifacts and mismatches between voices and faces in forged videos. VFD is pretrained on generic audio-visual data and fine-tuned on deepfake data, achieving strong benchmark results and competitive performance with limited fine-tuning data.

1 INTRODUCTION

  • Prior artifact-based detectors generalize poorly because manipulation algorithms and salient facial regions differ across datasets.Models trained on FF++ attend to eyes but can fail on DFDC, where salient regions include the forehead and nose.Source: §1 · p. 1, Figure 1 · p. 1
  • The paper proposes voice-face matching as a detection signal because deepfake editing can mismatch a face and voice, despite their association with identity.The introduction gives an edited Obama face matched with Reagan’s voice as an example.Source: §1 · p. 2
  • VFD first learns voice-face matching on generic audio-visual data with InfoNCE, then fine-tunes on deepfake data using an upgraded real-fake contrastive loss.The authors say the pretraining data supports adaptation with few fine-tuning steps and reduces reliance on expensive auxiliary deepfake data.Source: §1 · p. 2
  • AUCs of 85.13% on DFDC and 86.11% on FakeAVCeleb accompany VFD’s reported state-of-the-art results across three datasets and competitive results with limited fine-tuning data.The experiments also include DeepfakeTIMIT.Source: §1 · pp. 2–3
  • The approach uses facial-audio correlation without additional modalities or visual features and aligns the contrastive objective between generic and deepfake datasets.The authors describe the contribution as a multimodal matching framework for distinguishing real and fake videos.Source: §1 · p. 2

2 RELATED WORK

The related work covers face-manipulation methods, multimodal and cross-dataset deepfake detection, and cross-modal biometric matching. The paper positions voice-face matching as a proxy for distinguishing real and fake videos.

2.1 Deepfake

  • Deepfake generation includes image-based face replacement and 3D methods, while existing approaches focus on facial regions and can struggle to maintain voice-face consistency.Examples include AAD-based generation, Wav2Lip, 3DMM, and HifiFace.Source: §2.1 · p. 3

2.2 Deepfake Detection

  • Deepfake detectors use visual artifacts or multimodal cues, but unknown-data transfer remains difficult and cross-dataset approaches can require auxiliary data and added training.The paper proposes voice-face matching as a proxy for distinguishing real and fake videos, requiring paired voices and faces for training.Source: §2.2 · p. 3

2.3 Cross-modal Biometric Matching

  • Cross-modal biometric matching methods retrieve corresponding audio and video or align their features, supporting the premise that one person’s voice and face are correlated.Prior methods include cross-entropy or cosine-loss matching, local-global alignment, adversarial matching, and speech-to-face reconstruction.Source: §2.3 · p. 3

3 METHODOLOGY

The methodology section presents evidence for the method’s intuition, introduces a matching model pretrained on generic audio-visual data, and then describes fine-tuning and detection on downstream deepfake datasets.

3 METHODOLOGY

  • The section begins by presenting evidence for the method’s intuition.It then introduces the multi-modal matching model and its pretraining on generic audio-visual data.Source: §3 · p. 3
  • The method description covers a multi-modal matching model pretrained on generic audio-visual data.The section ends with fine-tuning and detection on downstream deepfake datasets.Source: §3 · p. 3
  • The section concludes with fine-tuning and detection on downstream deepfake datasets.This follows the presentation of the method’s intuition and pretrained matching model.Source: §3 · p. 3

3.1 Method Intuition

The method’s intuition rests on voice-face homogeneity and identity-related cues. Probing tests examine voice-face similarity in deepfakes and whether voice features discriminate identities.

3.1 Method Intuition

  • Voices and faces are closely associated with identity, motivating voice-face matching as a possible proxy for deepfake detection.The section frames its probing tests around voice-face matching in deepfake videos and voice-based identity discrimination.Source: §3.1 · pp. 3–4

3.1.1 Voice-Face Mismatching in Deepfake Data (Q1)

  • Real videos have much higher voice-face similarity than fake videos, with a split around 0.0, though some real videos overlap because of ambient recording noise.The analysis samples 1,800 real and 4,000 corresponding fake DFDC videos and compares cosine similarity of voice and face features.Source: §3.1.1 · p. 4

3.1.2 Voice Discrimination over Identities (Q2)

  • Voice features from the same identity cluster with apparent boundaries, and the learned features also reflect group characteristics such as gender and age.The visualization uses 5,120 voice clips from 163 Voxceleb2 identities.Source: §3.1.2 · p. 4, Figure 4 · p. 4

3.2 VFD for Pre-training

VFD learns voice-face matching by processing each modality separately and comparing identity-related features. It pre-trains on matched and unmatched identities before adapting the model to deepfake data.

3.2 VFD for Pre-training

  • VFD uses separate voice and face streams, with modulators guiding extractors toward identity-related features before a matching function measures their degree of match.InfoNCE is given as an example of the matching function.Source: §3.2 · p. 4

3.2.1 Data Preprocessing

  • Pre-training uses a three-second, 16 kHz voice clip and one randomly selected face frame, with same-identity faces as positives and other-identity faces as negatives.The voice is the anchor; hard-negative mining is left for future work.Source: §3.2.1 · p. 5

3.2.2 Feature Extraction

  • The feature extractor combines convolutional face maps with self-attention for nonlocal identity features, while the voice and face branches use separate parameters.The voice branch is designed similarly to the face branch, and both produce representative feature vectors.Source: §3.2.2 · pp. 5–6

3.2.3 Pre-training Protocol

  • Pre-training applies cross-entropy losses to the voice and face modulators for identity information and InfoNCE to learn voice-face homogeneity.The InfoNCE setup uses matched and negative face instances, with cosine similarity in the joint space.Source: §3.2.3 · p. 6

3.3 VFD for Fine-tuning

VFD transfers its generic audio-visual pre-training to deepfake data for fine-tuning. Its real-fake contrastive loss uses labeled real and fake pairs to adapt the matching objective.

3.3 VFD for Fine-tuning

  • Fine-tuning transfers the pre-trained VFD model to deepfake videos, where real voice-face pairs are matched and fake pairs may combine mismatched identities.The authors say pre-training makes the model sensitive to mismatches, allowing adaptation with part of a deepfake dataset and reduced computational overhead.Source: §3.3 · p. 6
  • Because edited voices and faces make it difficult to confirm individual mismatches, traditional InfoNCE may be confused during fine-tuning.The method therefore uses real and fake videos as positive and negative pairs, respectively.Source: §3.3 · p. 6
  • RFC uses real-fake labels in a cross-entropy objective to increase positive-pair similarity relative to negative-pair similarity, without manually selecting negative examples.The authors describe RFC as an upgrade of vanilla contrastive methods for deepfake data.Source: §3.3 · p. 6

3.4 VFD for Deepfake Detection

For detection, VFD compares a video’s face image and voice clip in its joint latent space. A validation-set threshold converts their similarity into a matched or mismatched decision.

3.4 VFD for Deepfake Detection

  • VFD takes the face image and voice clip from a video and computes their similarity in the joint latent space.The method uses the output to determine voice-face matching.Source: §3.4 · p. 6
  • The threshold λ is selected from the validation set to classify the voice-face pair as matched or mismatched.The passage specifies no threshold value here.Source: §3.4 · p. 6
  • A True matching result denotes a real video, whereas a False result corresponds to a video edited by deepfake algorithms.Source: §3.4 · p. 6

4 EXPERIMENT

The experiments use Voxceleb2 for pre-training and three deepfake datasets for fine-tuning and evaluation. The paper compares VFD with single-modality, multimodal, auxiliary-data, and fine-tuning baselines using ACC and AUC.

4.1 Dataset

  • Voxceleb2 supplies generic audio-visual pre-training data, followed by fine-tuning and evaluation on DeepfakeTIMIT, DFDC, and FakeAVCeleb.The described dataset splits avoid duplicated identities across pre-training, fine-tuning, and testing.Source: §4.1 · pp. 6–7

4.2 Implement Details

  • The implementation uses AdamW, a batch size of 128, and four Tesla V100 GPUs, with separate learning rates for pre-training and fine-tuning.The learning rates are 1 × 10−4 and 5 × 10−6, respectively.Source: §4.2 · p. 7

4.3 Compared Baselines and Evaluation Metrics

  • Baselines span single-modality detectors, visual-auditory models, auxiliary-data methods, and fine-tuning approaches; evaluation uses ACC and AUC.The comparison includes fine-tuned unimodal models, including ViT F with the same pre-training and fine-tuning strategy as VFD.Source: §4.3 · p. 7

4.4 Performance Comparison

VFD outperforms the reported baselines across the evaluated datasets, while single-modality and some fine-tuned methods are less reliable. Auxiliary landmarks are competitive with traditional unimodal methods, but VFD’s voice-face matching performs best among the compared baselines.

4.4.1 Comparison on modality

  • Some single-modality models perform poorly, while audio- or emotion-based multimodal methods can outperform them; simple ensembling has mixed effects.The authors attribute unstable ensemble performance to fusing classifier predictions without modeling relationships among modalities.Source: §4.4.1 · p. 7

4.4 Performance Comparison

  • VFD achieves state-of-the-art performance across DFDC, FakeAVCeleb, and DeepfakeTIMIT, with around 85% AUC on the two more challenging datasets.Baselines on DFDC and FakeAVCeleb can fall below 50% accuracy, while most baselines exceed 90% AUC on DeepfakeTIMIT.Source: §4.4 · p. 7, Table 1 · p. 8

4.4.2 Comparison on auxiliary data

  • DSP-FWA gains AUC over 75% on DFDC and 99% on DeepfakeTIMIT, on par with ViT; Headpose only outperforms Meso-4.The authors suggest that powerful visual models may capture facial-movement features represented by landmarks, achieving proximate performance.Source: §4.4.2 · p. 8

4.4.3 Comparison on fine-tuning strategy

  • VFD performs best among the baselines, with around 12% improvement over LipForensics, despite using only one-quarter of the dataset in comparison with Xception F.Fine-tuning traditional approaches can reduce performance; the authors note those methods use one-quarter of the data for faster training and lower overhead.Source: §4.4.3 · pp. 8–9

4.5 Qualitative Results

VFD’s heatmaps emphasize whole-face identity information, and its voice-face similarity separates real and fake videos around a consistent cutoff. The experiments also identify image conditions that undermine detection and show how fine-tuning data and negative samples affect performance.

4.5.1 Heatmap

  • VFD heatmaps focus on the whole face rather than localized artifacts, and the authors report stable attention regions across datasets.Figure 6 compares Xception and F3-Net heatmaps with VFD heatmaps and original face images.Source: §4.5.1 · p. 9

4.5.2 Similarity comparison

  • Real and fake videos separate around a voice-face similarity cutoff of -0.1, including when fine-tuning and testing are swapped between DeepfakeTIMIT’s LQ and HQ subsets.The authors report that image quality does not significantly affect VFD’s effectiveness in this subset-swap experiment.Source: §4.5.2 · p. 9, Figure 9 · p. 9

4.5.3 Detection failure cases

  • Detection failures arise with inadequate illumination, extreme facial angles, and edited attributes such as glasses or beards that leave identity unchanged.The paper explains that dim lighting mixes faces with backgrounds, while extreme angles make facial features difficult to extract.Source: §4.5.3 · p. 9, Figure 10 · p. 10, Figure 10c · p. 10

4.6 Hyperparameter Study

  • With 1,000 or 2,000 real DFDC samples for fine-tuning, VFD reaches 77% AUC; with 100 samples, it reaches 65.87% AUC.Performance is best at 5,000 fine-tuning instances; the authors compare this with more than 20,000 videos used by traditional approaches.Source: §4.6 · p. 10, Figure 11 · p. 10
  • Adding more negative samples significantly boosts performance, with gains saturating when their number reaches 10.The setting with zero negatives uses no fake pairs for fine-tuning.Source: §4.6 · p. 10, Figure 11 · p. 10

4.7 Ablation Study

Ablations find the full VFD configuration performs best, with pre-training, fine-tuning, RFC, and transformer-based extractors each supported by the reported comparisons. The study also identifies voice recognition as a bottleneck and shows matching accuracy close to 97% on the generic dataset.

4.7.1 Module evaluation

  • Combining all devised modules achieves the best result across the evaluated variants.The reported comparisons include DFDC and FakeAVCeleb.Source: §4.7.1 · p. 10, Table 2 · p. 11
  • Visual-auditory and Modulator underperform Visual-only in most cases, while Modulator improves on Visual-auditory and RFC is more robust than InfoNCE on DFDC.InfoNCE performs comparably with RFC on FakeAVCeleb but fails severely on DFDC; the authors attribute this difference to dataset annotation detail.Source: §4.7.1 · p. 10, Table 2 · p. 11

4.7.2 Strategy evaluation

  • Removing fine-tuning drops DFDC AUC by 30%, indicating fine-tuning is essential to narrow the gap between generic and deepfake datasets; pre-training contributes more than 10% and 5% AUC improvements.Source: §4.7.2 · pp. 10–11

4.7.3 Feature extractor evaluation

  • Transformer-based extractors outperform VGG-based extractors across metrics; face recognition accuracy exceeds 98% and 94%, respectively.The authors attribute the transformer advantage to modeling global features more effectively for identity-based matching.Source: §4.7.3 · p. 11, Table 4 · p. 11
  • VFD achieves matching accuracy close to 97% on the generic dataset, while voice identification remains the performance bottleneck.The authors suggest multimodal contrastive learning compensates to some extent for voice-feature errors.Source: §4.7.3 · p. 11

5 CONCLUSION AND DISCUSSION

VFD addresses deepfake detection by modeling voice-face homogeneity through pre-training, fine-tuning, and an RFC loss. The authors report effectiveness against existing competitors, while noting a limitation when both voices and faces are maliciously edited.

5 CONCLUSION AND DISCUSSION

  • VFD learns to match voices and faces from a single identity through a pre-training then fine-tuning pipeline and a novel RFC loss.The conclusion describes this as a homogeneity-modeling perspective on deepfake detection.Source: §5 · p. 11
  • Experimental and visualization results demonstrate VFD's effectiveness against existing elaborately designed competitors.Source: §5 · p. 11
  • When both voices and faces are maliciously edited, matching may collapse; the authors note no high-quality datasets or associated approaches have been studied for this setting.They describe this dual-edit case as an unusual limitation and an area needing future exploration.Source: §5 · p. 11
Loading 2203.02195v3…