PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Prior Forgery Knowledge: Mining Critical Clues for General Face Forgery Detection

Anwei Luo, Chenqi Kong, Jiwu Huang, Yongjian Hu, Xiangui Kang, Alex C. Kot

arXiv:2304.12489v1cs.CVcs.CR

TL;DR

Face forgery detectors can overfit to training domains or rely on specific clues that are vulnerable to attacks and distortions. CFM combines prior knowledge-agnostic augmentation, fine-grained relation learning, and progressive feature control; the authors report improved robustness and generalization over state-of-the-art methods across six public datasets.

  • Problem

    Learning-based detectors can overfit training data, while specific prior clues can be targeted or disrupted by image distortions, limiting generalization to unforeseen domains.

    Source: §I · p. 1

  • Method

    CFM combines prior knowledge-agnostic augmentation and fine-grained triplet relation learning with instance and local similarity-aware losses and progressive learning control.

    Source: §I · pp. 1–2, 1) Fine-grained Triplet Construction · p. 3

  • Results

    Across six public datasets, CFM outperforms state-of-the-art methods in robustness and generalization capability.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that learning global critical features alongside subtle local artifacts contributes to better final detection performance.

    Source: §V · p. 11

Abstract

from arXiv · show

Face forgery detection is essential in combating malicious digital face attacks. Previous methods mainly rely on prior expert knowledge to capture specific forgery clues, such as noise patterns, blending boundaries, and frequency artifacts. However, these methods tend to get trapped in local optima, resulting in limited robustness and generalization capability. To address these issues, we propose a novel Critical Forgery Mining (CFM) framework, which can be flexibly assembled with various backbones to boost their generalization and robustness performance. Specifically, we first build a fine-grained triplet and suppress specific forgery traces through prior knowledge-agnostic data augmentation. Subsequently, we propose a fine-grained relation learning prototype to mine critical information in forgeries through instance and local similarity-aware losses. Moreover, we design a novel progressive learning controller to guide the model to focus on principal feature components, enabling it to learn critical forgery features in a coarse-to-fine manner. The proposed method achieves state-of-the-art forgery detection performance under various challenging evaluation settings.

I. INTRODUCTION

Face forgery detectors face a generalization challenge: data-driven models can overfit, while detectors built around specific forgery clues can be targeted and fail under image distortions. CFM aims to mine more critical clues through prior knowledge-agnostic augmentation, fine-grained relation learning, and progressive control.

I. INTRODUCTION

  • Prior learning-based detectors can overfit training data and generalize unsatisfactorily to unforeseen domains; specific noise, boundary, and frequency clues can also be targeted or disrupted.Selecting a suitable clue type for a particular detection task is also challenging.Source: §I · p. 1
  • CFM is designed to detect manipulated faces from unseen datasets and manipulation techniques, as well as forged faces affected by common image degradations.These goals address overfitting to specific clues and the need to capture more critical ones.Source: §I · p. 1
  • CFM combines prior knowledge-agnostic augmentation, fine-grained triplet relation learning, principal-component regularization, and instance- and local-similarity losses.The losses target global critical features and local subtle artifacts, while the progressive controller guides coarse-to-fine learning.Source: §I · p. 1
  • With EfficientNet-B4 held constant, Grad-CAM visualizations show CFM focusing on critical face regions across image degradation types, unlike the vulnerable vanilla and noise-clue models.The comparison includes downsampling and Gaussian noise.Source: §I · pp. 1–2, Figure 1 · p. 2
  • Across six public datasets, CFM outperforms state-of-the-art methods in robustness and generalization capability.The authors also report visualization results that verify the method’s effectiveness.Source: §I · p. 2

II. RELATED WORK

Prior face-forgery detectors use expert-selected clues or representation-learning strategies, but specific clues are vulnerable and fine-grained general features remain underexplored. CFM uses a fine-grained triplet to relate matched and mismatched face regions while suppressing specific prior clues.

III. PROPOSED METHOD

  • CFM constructs fine-grained triplets, suppresses specific clues through prior knowledge-agnostic augmentation, learns global and local relations with ISL and LSL, and uses PLC to steer fine-tuning toward principal information.The PLC is described as avoiding early collapse to non-critical features.Source: Figure 2 · p. 3

A. Prior Knowledge-based Face Forgery Detection

  • Prior-knowledge methods exploit noise, boundary, and frequency clues, but overemphasizing a particular clue can leave detectors unreliable on WildDeepfake and sensitive to perturbations.CFM instead suppresses these prior clues to mine more general feature representations.Source: A. Prior Knowledge-based Face Forgery… · pp. 2–3

B. Face Forgery Detection via Representation Learning

  • Representation-learning work includes domain-general features, localization invariance, local-region relations, and contrastive learning, but fine-grained features for general face-forgery detection remain underexplored.CFM addresses this gap with a prototype task that learns relations in a fine-grained triplet.Source: B. Face Forgery Detection via… · p. 3

1) Fine-grained Triplet Construction:

  • The anchor and positive are different frames from the same real or fake video, while the time-aligned negative comes from the corresponding video with the opposite authenticity.The anchor and negative are matched for pose, expression, and background; the anchor-positive interval exceeds one second.Source: 1) Fine-grained Triplet Construction · pp. 3–4

2) Prior Knowledge-agnostic Data Augmentation:

The framework suppresses specific forgery traces with prior knowledge-agnostic augmentation, learns relations from fine-grained triplets, and progressively drops less important feature channels. The controller is designed to support coarse-to-fine learning and avoid local optima.

2) Prior Knowledge-agnostic Data Augmentation:

  • Prior knowledge-agnostic augmentation suppresses high-frequency clues, color mismatch, noise artifacts, and identity information while using the same augmentation sampling for anchor and negative inputs.The strategy uses blur, downscaling, compression, ColorJitter, Gaussian noise, and Random Grid Shuffle.Source: 2) Prior Knowledge-agnostic Data… · p. 4

B. Fine-grained Triplet Relation Learning

  • The fine-grained triplet task uses an EMA-updated teacher to produce positive and negative representations for student learning.The anchor is encoded by the student, while positive and negative inputs are encoded by the teacher; temporal ensembling is described as yielding more reliable intermediate representations.Source: B. Fine-grained Triplet Relation… · p. 4

1) Progressive Learning Controller:

  • PLC measures each channel’s importance from anchor-positive and anchor-negative feature-response differences, stabilizes the importance vector with EMA, sorts channels ascending, and drops the last floor(C × ρ) feature maps as unimportant.The modified feature maps are scaled by 1/(1 − ρ) to make the feature unbiased.Source: 1) Progressive Learning Controller · pp. 4–5
  • PLC is designed to guide coarse-to-fine learning, stabilize training, and prevent local optima to boost generalization.It initially drops less important channels so gradient contributions come from principal features, then gradually adds dropped channels back.Source: 1) Progressive Learning Controller · pp. 4–5

C. Objective Functions

The objective functions combine instance- and local-similarity-aware losses to supervise training. The instance loss promotes compactness within classes and separability between classes in fine-grained triplets.

C. Objective Functions

1) Instance Similarity-aware Loss:

  • ISL promotes critical-information mining in fine-grained triplets by constraining intra-class compactness and inter-class separability.It encourages anchor-positive similarity to exceed anchor-negative similarity by a margin, supporting invariant features across different transformations.Source: 1) Instance Similarity-aware Loss · p. 5

2) Local Similarity-aware Loss:

The local similarity-aware loss learns patch-level relationships that complement instance-level triplet learning. It encourages each anchor patch to match same-label positive patches and differ from negative patches, with hard examples emphasized.

2) Local Similarity-aware Loss:

  • The loss complements instance-level semantic triplet learning by modeling local relationships between patches to capture subtle informative features.The method assigns local real/fake labels using pixel-level difference masks and maps features into a low-dimensional embedding space.Source: 2) Local Similarity-aware Loss · p. 5
  • For each anchor patch, same-label patches in the positive image are positives, while different-label patches in the negative image are negatives.The loss encourages similarity to positive patches and penalizes similarity to negative patches.Source: 2) Local Similarity-aware Loss · p. 5
  • The loss filters easy examples and emphasizes hard examples so the model can learn more critical local information.The figure marks manipulation patches in the negative image as negative patches.Source: 2) Local Similarity-aware Loss · p. 5, Figure 4 · p. 5
  • Lloc requires positive-patch similarity to exceed spos and negative-patch similarity to remain below sneg, separating real and fake patches by a threshold margin.The illustrated rule applies to each anchor patch; the caption notes that augmentation is omitted for visualization.Source: Figure 4 · p. 5, 2) Local Similarity-aware Loss · p. 6

3) Total Loss:

The framework uses cross-entropy supervision and combines three components in its overall objective; at test time, the classifier output supplies the final prediction.

3) Total Loss:

  • The framework is trained end to end using cross-entropy between its prediction and the ground-truth label.The label is 0 for real faces and 1 for fake faces.Source: 3) Total Loss · p. 6
  • The overall objective function consists of three components.Source: 3) Total Loss · p. 6
  • At test time, the classifier output is used as the final prediction.Source: 3) Total Loss · p. 6

IV. EXPERIMENTS

Experiments use an EfficientNet-B4 backbone, report image-level metrics by default, and evaluate across six public datasets with specified dataset splits.

1) Implementation Details:

  • Implementation uses MTCNN-cropped faces resized to 320 × 320, 20 sampled training frames per video, and ImageNet-pretrained EfficientNet-B4.Training runs for 30 epochs with Adam, batch size 32, and learning-rate decay by 0.5 every five epochs.Source: 1) Implementation Details · p. 6

2) Evaluation Metrics:

  • Evaluation reports ACC, AUC, and EER, using image-level results by default and averaging frame scores for video-level detection.Source: 2) Evaluation Metrics · p. 6

3) Dataset:

  • The six-dataset evaluation includes FF++ at C23 and C40 quality levels, Celeb-DF-v2, WildDeepfake, and DFDC.FF++ contains four manipulation types; Celeb-DF-v2 is described as more visually realistic, while WildDeepfake uses internet-collected sequences from diverse scenes and methods.Source: 3) Dataset · p. 6
  • The evaluation also includes Google’s DeepFakeDetection and DeeperForenics-1.0; all DFD videos are used, while the other five datasets follow official meta-file splits.DFD contains more than 3,000 manipulated videos; DeeperForenics-1.0 applies perturbations to FF++ real videos.Source: 3) Dataset · p. 6

B. Intra-dataset Evaluation

CFM improves cross-dataset results over the EN-B4 backbone and reports strong transfer across manipulation types. On FF++ C40 it also improves AUC over RECCE, while additional tables cover perturbation robustness, backbones, and loss thresholds.

B. Intra-dataset Evaluation

  • On FF++ C40, CFM raises AUC from 94.57% with RECCE to 96.97%; it is second-best on C23 and superior in video-level detection on both quality levels.The authors attribute the C40 result to learning local differences and fine-grained clues in highly compressed data.Source: B. Intra-dataset Evaluation · p. 7

1) Cross-dataset Evaluation:

  • 8.06% average AUC improvement and 5.92% average EER improvement over EN-B4 are reported across five unseen datasets.CFM is reported to outperform prior methods on all unseen datasets and to achieve better cross-dataset detection than FTCN.Source: 1) Cross-dataset Evaluation · p. 7, Table II · p. 7

2) Cross-manipulation Evaluation:

  • When trained on DF, CFM improves Cross Avg. AUC over EN-B4 by nearly 10% and outperforms RECCE across all four manipulation evaluation settings.RECCE’s strong intra-manipulation result when trained on NT drops on the other three types; CFM is reported to generalize to unseen forgery types.Source: 2) Cross-manipulation Evaluation · pp. 7–8, Table III · p. 7

C. Generalization Evaluation

D. Robustness Evaluation

CFM improves robustness under common image perturbations and transfers across backbones, while ablations identify useful settings for its fine-grained relation-learning losses.

D. Robustness Evaluation

  • 7.90% average improvement over EN-B4, from -11.61% to -3.71%, accompanies CFM’s best robustness performance across the evaluated perturbations.The reported values are average AUC score drops compared with pristine data.Source: D. Robustness Evaluation · p. 8

E. Flexibility Evaluation

  • CFM improves Xception, EfficientNet-B0, and MobileNet-V2 performance, with especially promising cross-dataset results and state-of-the-art generalization for lightweight backbones.Models were trained on FF++ (C23) and evaluated in intra- and cross-dataset settings.Source: E. Flexibility Evaluation · p. 8

F. Ablation Experiments

  • Setting spos=0.8 and sneg=−0.5 gives the best reported performance; increasing dins improves intra-dataset results but lowers cross-dataset results, so dins is set to 1.2.The authors report that spos=1 and sneg=−1 leave no easy examples, reducing the contribution of useful hard-example information.Source: 1) Impacts of dins, spos and sneg · pp. 8–9
  • The LSL weighting strategy improves both intra- and cross-dataset performance, and the authors identify hard examples as important for learning discriminative features.Table VII reports image-level AUC scores.Source: Table VII · p. 8, 1) Impacts of dins, spos and sneg · p. 9

2) Effectiveness of Different Proposed Components:

Ablations find contributions from the proposed components, prior knowledge-agnostic augmentation, progressive channel masking, and matched augmentation of anchor-negative pairs.

2) Effectiveness of Different Proposed Components:

  • Removing augmentation lowers CDF generalization AUC from 71.57% to 65.24%, while removing PLC lowers AUC from 82.78% to 78.84%.Removing ISL or LSL also reduces performance in both intra- and cross-dataset evaluations.Source: 2) Effectiveness of Different Proposed… · p. 9

3) Impacts of Different Masking Strategies for PLC:

  • PLC’s masked channels stabilize below a 5% change ratio after a few iterations, versus around 50% for random masking, and PLC—not random dropout—improves generalization.The generalization comparison is reported in Table IX.Source: 3) Impacts of Different Masking… · p. 9, Table IX · p. 9

4) Impacts of the Prior Knowledge-agnostic Augmentation:

  • Removing any augmentation type degrades generalization; removing high-frequency relative augmentation drops AUC by 12.65% on CDF and 4.77% on WDF.Table X covers high-frequency, color-mismatch, noise-artifact, and identity-inconsistency relative augmentations.Source: 4) Impacts of the Prior… · p. 9, Table X · p. 9

5) Effectiveness of Pairwise Constraint for Anchor-Negative data Pair:

  • Applying the same prior knowledge-agnostic augmentation to anchor and negative images improves generalization on CDF and WDF by steering attention toward inherent forgery information.The pairwise-constraint ablation is summarized in Table XI.Source: Table XI · p. 9, 5) Effectiveness of Pairwise… · p. 10

G. Visualization

The visualization section compares detector attention and feature distributions; its Grad-CAM and t-SNE results illustrate CFM’s behavior across datasets, backbones, and perturbations.

1) Visualization for Detector’s Attention:

  • On FF++, CFM’s Grad-CAM maps highlight manipulated regions: the mouth for NT and F2F, and facial features for DF and FS.These locations correspond to the regions modified by the respective manipulations.Source: 1) Visualization for Detector’s… · p. 10
  • Across CDF, DFDC, and WDF, baseline maps locate similar regions in real and fake images and can misclassify real images, while CFM maps distinguish them more effectively.For the DFDC real-fake pair, CFM identifies prominent differences between the faces.Source: 1) Visualization for Detector’s… · pp. 10–11
  • Across backbones, CFM maps focus on facial feature regions and similar areas for the same fake image; under perturbations, maps still identify prominent manipulated regions, including after patch shuffling.The authors interpret the patch-shuffling visualization as evidence that CFM mines local fine-grained forgery clues.Source: 1) Visualization for Detector’s… · p. 11

2) T-SNE Feature Embedding Visualization:

V. CONCLUSION AND FUTURE WORK

  • The conclusion attributes CFM’s robustness and generalization to prior knowledge-agnostic augmentation, fine-grained triplet learning, progressive coarse-to-fine feature mining, and instance/local similarity-aware objectives.The paper identifies temporal information as under-explored and points to temporal features and fake-video temporal inconsistency as future work.Source: §V · p. 11

Related papers

Loading 2304.12489v1…