PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Relation Learning for Face Forgery Detection

Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, Jilin Li, Rongrong Ji

arXiv:2105.02577v1cs.CV

TL;DR

Face forgery detection needs more generalized features than global classification or mask supervision provides, because realistic manipulations raise security concerns and can lead to overfitting. The paper learns local-region relations with MPSM and fuses RGB and frequency information with RFAM. Extensive experiments report consistent gains over compared methods on widely used benchmarks, including unseen manipulation and dataset settings.

  • Problem

    Existing global classification and manipulation-mask supervisions do not model local-region correlations sufficiently for generalized face forgery detection and are prone to overfitting.

  • Method

    The method uses MPSM to measure local-region similarity and RFAM to fuse RGB and frequency information for comprehensive local representations.

  • Results

    The method consistently outperforms compared methods across quality settings, manipulation methods, and unseen datasets.

  • Takeaways & Limitations

    Local similarity patterns provide a robust and generalized representation for face forgery detection across varied manipulation and data conditions.

Abstract

from arXiv · show

With the rapid development of facial manipulation techniques, face forgery detection has received considerable attention in digital media forensics due to security concerns. Most existing methods formulate face forgery detection as a classification problem and utilize binary labels or manipulated region masks as supervision. However, without considering the correlation between local regions, these global supervisions are insufficient to learn a generalized feature and prone to overfitting. To address this issue, we propose a novel perspective of face forgery detection via local relation learning. Specifically, we propose a Multi-scale Patch Similarity Module (MPSM), which measures the similarity between features of local regions and forms a robust and generalized similarity pattern. Moreover, we propose an RGB-Frequency Attention Module (RFAM) to fuse information in both RGB and frequency domains for more comprehensive local feature representation, which further improves the reliability of the similarity pattern. Extensive experiments show that the proposed method consistently outperforms the state-of-the-arts on widely-used benchmarks. Furthermore, detailed visualization shows the robustness and interpretability of our method.

1 Introduction

Face forgery detection is challenged by overfitting, limited interpretability, disturbances, and poor generalization to unseen manipulation methods. The paper addresses these issues by learning relations among local regions and combining RGB and frequency information.

  • Face forgery detection is important because realistic manipulations can evade human judgment and create security and trust concerns.
  • Binary classification and manipulation-mask supervision can overfit, lack interpretability, remain vulnerable to compression or noise, and generalize poorly to unseen manipulations.
  • Local relation learning models a generalized descriptor that captures forged traces through relationships among local regions.
  • The Multi-scale Patch Similarity Module measures pair-wise cosine similarity among local regions, distinguishing similar real or forged regions from dissimilar real-forged pairs.
  • The RGB-Frequency Attention Module fuses RGB and frequency information to learn comprehensive local representations and improve similarity-pattern reliability.

2 Related Works

Related work mainly approaches face forgery detection through binary classification, manipulated-region localization, or frequency-domain cues. These approaches provide useful detection signals but remain vulnerable to overfitting, compression, noise, or limited generalization.

  • Early methods use neural networks to extract discriminative features for binary real-versus-forgery classification.
  • Attention and self-supervised methods highlight informative regions or forged boundaries, but can be affected by compression and noise.
  • Frequency-based methods extract or amplify frequency cues using DFT, Laplacian-of-Gaussian, or frequency-aware image components.

3 Approach

The approach combines RGB and frequency cues with multi-scale local-region similarity learning to detect forged faces. It uses attention-based feature fusion, patch-level relations, and supervised losses for classification and localization.

  • Frequency-aware Cue: Frequency cues expose artifacts caused by anomalous frequency statistics in forged faces.The method transforms RGB images with DCT, suppresses low frequencies, and reconstructs a frequency-domain representation emphasizing subtle high-frequency artifacts.
  • RGB-Frequency Attention Module: RFAM fuses RGB and frequency streams at multiple semantic layers to learn comprehensive local-region features.RGB features help locate anomalous textures, while frequency features amplify subtle manipulated artifacts; attention highlights important spatial regions.
  • Multi-scale Patch Similarity Module: MPSM measures cosine similarity between multi-scale local patches instead of relying on global feature-map pooling.Low-level features support localization, high-level features support identification, and concatenated multi-scale features are partitioned into patches for pairwise comparison.
  • Experiments: The framework is evaluated on FaceForensics++ across quality settings, manipulation methods, unseen datasets, and component ablations.The listed experiments cover ACC and AUC on compressed and raw videos, manipulation-specific performance, cross-dataset testing, and model-component influence.
  • Multi-scale Patch Similarity Module: Lower patch similarity indicates a larger discrepancy between regions, suggesting that one patch is real and the other forged.The learned similarity matrix is supervised using patch-level forged probabilities derived from manipulated-region masks.
  • Loss Functions: The similarity pattern is flattened and passed through fully connected and Sigmoid layers for face-level prediction, while a decoder predicts forged-region masks.Cross-entropy, segmentation, and similarity losses jointly supervise classification, localization, and local relations.

4 Experiments

The experiments evaluate the proposed algorithm against state-of-the-art methods, test robustness under unseen manipulation methods, analyze component contributions, and visualize interpretability.

  • The evaluation compares the proposed algorithm with state-of-the-art techniques and investigates robustness under unseen manipulation methods.
  • An ablation study examines the influence of the proposed components.
  • Visualization analysis demonstrates the interpretability of the approach.

4.1 Experimental Setup

The experiments train on FaceForensics++ and evaluate robustness on several additional face-manipulation datasets, using specified implementation and optimization settings.

  • Datasets: FaceForensics++ contains 1,000 original videos, four manipulation methods, and raw, high-quality, and low-quality outputs; 720, 140, and 140 videos are assigned to training, validation, and testing.
  • Datasets: Robustness is evaluated on DFDC, Celeb-DF, and DFD in addition to FaceForensics++.
  • Implementation: The framework is implemented in PyTorch with α=0.33, k=5, λ1=10, λ2=1, 299 × 299 inputs, and 50 training epochs.
  • Implementation: Training uses Adam with learning rate 2e-4, batch size 32, betas 0.9 and 0.999, weight decay 1e-5, and halving the learning rate every 10 epochs.

4.2 Results and Discussions

The proposed method consistently outperforms comparison methods across quality settings, manipulation methods, and unseen datasets, while visualizations show local similarity patterns aligned with forged regions.

  • Quality settings: The method exceeds F3 Net in AUC by 0.12%, 1.36%, and 1.91% under Raw, HQ, and LQ settings, respectively.
  • Quality settings: On LQ data, the method achieves 33.61% higher AUC than Face X-ray.
  • Manipulation methods: On NeuralTextures, the most challenging manipulation method, the model exceeds F3 Net by 5.99% in ACC.
  • Visualization: Visualizations show predicted similarity patterns and masks matching ground truth while distinguishing similar real regions from manipulated-region discrepancies.
  • Cross-dataset generalization: Training on FF++ and testing on Celeb-DF, DFDC, and DFD yields significant improvements over Xception on all unseen datasets.

4.3 Ablation Study

The ablation study finds complementary benefits from frequency information, RFAM, and MPSM, with the complete model achieving the best reported ACC and AUC.

  • Component contributions: Adding frequency information consistently improves ACC and AUC over the RGB-only baseline.
  • Component contributions: Fusing RGB and frequency streams through RFAM further improves performance.
  • Component contributions: Using only MPSM increases ACC and AUC to 91.06% and 94.96%, respectively.
  • Complete model: Combining all proposed modules achieves 91.47% ACC and 95.21% AUC.

4.4 Visualization

Visualizations show that local similarity patterns distinguish real from forged faces, while RGB-frequency features focus on manipulated regions and remain robust under noise.

  • Local similarity patterns: Real faces exhibit consistent local similarity across regions, whereas forged faces produce manipulation-specific patterns tied to altered areas.These patterns capture forged regions across different manipulation algorithms and serve as a basis for detection.
  • Predictive regions: For real faces, predictive attention is evenly distributed; forgeries concentrate attention in manipulated regions such as the nose or mouth.The visualization uses the most predictive patch selected from each image according to final feature-map activation.
  • Feature-map interpretation: Compared with the baseline, our method produces nearly uniform activations on real faces and localized activations on forged regions.The method constrains local features to learn content-independent representations focused on artifacts rather than central facial content.
  • Feature-map interpretation: RGB and frequency feature maps jointly locate forged-face information, providing complementary cues for local representation.Figure 5 identifies separate RGB and frequency feature maps among the visualized columns.
  • Robustness to noise: Under blur, Gaussian noise, compression, and random patches, our local similarity pattern remains consistent while the baseline makes substantial mask errors.The method remains robust even when random patches disrupt face structure.

5 Conclusion

The paper introduces face forgery detection through modeling relations among local regions. Its RGB-Frequency Attention Module and Multi-scale Patch Similarity supervision learn comprehensive local relations, with experiments and visualizations demonstrating robustness and generalizability.

  • The proposed approach models relations among local regions for face forgery detection.
  • An RGB-Frequency Attention Module and Multi-scale Patch Similarity supervision jointly learn comprehensive local relations using RGB and frequency information.
  • Experiments and visualizations demonstrate the method’s robustness and generalizability on widely used face forgery detection datasets.
Loading 2105.02577v1…