PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalizing Face Forgery Detection with High-frequency Features

Yuchen Luo, Yong Zhang, Junchi Yan, Wei Liu

arXiv:2103.12376v1cs.CV

TL;DR

Cross-database face forgery detection remains difficult because CNN detectors overfit to manipulation-specific color textures. The paper uses multi-scale high-frequency noise features with residual-guided and cross-modality attention, reporting superior generalization on several benchmarks.

  • Problem

    Cross-database detection suffers severe performance drops because manipulation techniques generate diversified data distributions and CNN detectors overfit to method-specific textures.

  • Method

    The model combines RGB and high-frequency modalities using multi-scale noise extraction, residual-guided spatial attention, and dual cross-modality attention.

  • Results

    Comprehensive evaluations on several benchmark databases demonstrate the proposed model’s superior generalization ability.

  • Takeaways & Limitations

    High-frequency noise features provide a texture-irrelevant modality that complements RGB features for face forgery detection.

  • Takeaways & Limitations

    The generalization rationale assumes that blending violates original data distributions and that authentic and tampered regions have discoverable characteristic discrepancies.

Abstract

from arXiv · show

Current face forgery detection methods achieve high accuracy under the within-database scenario where training and testing forgeries are synthesized by the same algorithm. However, few of them gain satisfying performance under the cross-database scenario where training and testing forgeries are synthesized by different algorithms. In this paper, we find that current CNN-based detectors tend to overfit to method-specific color textures and thus fail to generalize. Observing that image noises remove color textures and expose discrepancies between authentic and tampered regions, we propose to utilize the high-frequency noises for face forgery detection. We carefully devise three functional modules to take full advantage of the high-frequency features. The first is the multi-scale high-frequency feature extraction module that extracts high-frequency noises at multiple scales and composes a novel modality. The second is the residual-guided spatial attention module that guides the low-level RGB feature extractor to concentrate more on forgery traces from a new perspective. The last is the cross-modality attention module that leverages the correlation between the two complementary modalities to promote feature learning for each other. Comprehensive evaluations on several benchmark databases corroborate the superior generalization performance of our proposed method.

1. Introduction

The paper targets cross-database face forgery detection, where changing manipulation algorithms causes severe generalization problems. It attributes this failure to CNN texture bias and proposes a high-frequency-feature model with three complementary modules.

  • Motivation: Cross-database detection is more challenging because training and testing forgeries are synthesized by different algorithms.Within-database detection uses the same synthesis algorithm for training and testing, whereas cross-database detection changes the algorithm.
  • Motivation: Methods with strong within-database accuracy can suffer severe performance drops on forgeries generated by unseen manipulation techniques.Diversified data distributions across manipulation techniques limit broader applications.
  • Motivation: The analysis finds that CNN-based detectors overfit to method-specific color textures, motivating texture-irrelevant forgery cues.High-frequency noises can suppress image textures and expose statistical discrepancies between authentic and tampered regions.
  • Method: The proposed model combines RGB and high-frequency modalities through multi-scale extraction, residual-guided spatial attention, and dual cross-modality attention.The modalities interact to provide complementary information and mutually promote representation learning.
  • Evaluation: Comprehensive evaluations on several benchmarks demonstrate the proposed model’s superior generalization ability.

2. Related Work

Prior face forgery detectors emphasize conventional editing artifacts, specific forgery cues, auxiliary information, or frequency-domain representations. These approaches often remain limited by changing artifacts, extra data requirements, or incomplete treatment of generalization.

  • Conventional image forgery detection: Conventional image forgery detectors target copy-move or copy-paste artifacts that differ from those produced by GAN-based face forgeries.Face forgeries are smaller, lower quality, and may contain few visible artifacts.
  • Specific artifacts or novel architectures: Artifact-specific methods target cues such as abnormal eye-blinking frequency or head-pose inconsistency, but improved forgeries may lack these artifacts.
  • Specific artifacts or novel architectures: Alternative network designs improve representation or computational efficiency without explicitly addressing generalization.Examples include compact mesoscopic-feature networks and capsule networks.
  • Auxiliary tasks or synthetic data: Auxiliary-task and synthetic-data methods address generalization using additional annotations or extra samples.Examples include jointly predicting manipulated-region attention and generating samples resembling target-domain distributions.
  • High-frequency features: Frequency-domain methods exploit Fourier, DFT, steganalysis, or DCT information to mine forgery clues and compression errors.

3. Analysis of Generalizable Forgery Detector

The analysis links poor cross-database performance to CNN overfitting on method-specific textures and examines manipulation-induced discrepancies that remain across methods. It motivates high-frequency noise features, which improve generalization over RGB textures in the reported comparison.

  • Why current methods fail to generalize?: CNN detectors learn method-specific texture patterns, making generalization difficult across forgery algorithms with different architectures and processing streams.
  • Why current methods fail to generalize?: Grad-CAM shows an Xception detector trained on Face2Face focuses on mouth-region textures and mistakes unseen NT, DF, and FS forgeries as real.The responses on unseen forgeries resemble those for authentic faces rather than the training-set activation pattern.
  • Manipulation pipeline: The manipulation pipeline creates method-specific texture artifacts during face creation and characteristics discrepancies during face blending.
  • What is common in forged face images?: Because the background remains source-consistent while the face is manipulated, blending can violate original data distributions and create authentic–tampered discrepancies.The proposed analysis seeks inconsistency between authentic and tampered regions rather than relying only on blending-boundary patterns.
  • Use SRM to extract high-frequency noise features: High-frequency noises suppress color content while exposing intrinsic image characteristics and forgery traces that are difficult to recognize in RGB space.SRM examples show continuous authentic-face noise distributions versus smoother or sharper tampered-face patterns.
  • Use SRM to extract high-frequency noise features: SRM-noise Xception generalizes better than RGB Xception, especially on FaceSwap and NeuralTextures methods.The comparison trains both models on Face2Face forgeries and evaluates them across four methods.

4. The Proposed Method

The proposed method uses two complementary streams for RGB images and high-frequency noises, combining multi-scale extraction, residual-guided spatial attention, and dual cross-modality attention. These components enrich high-frequency representations, focus RGB features on forgery traces, and model interactions between modalities before attention-based classification.

  • Multi-scale High-frequency Feature Extraction: SRM high-pass filters are applied to the input image and low-level feature maps at multiple scales to compose richer high-frequency features.Compared with input residuals alone, the resulting representation contains high-frequency signals from both the image and low-level features.
  • Overall Architecture: The model uses a two-stream architecture that processes RGB images and high-frequency noises, then fuses their features for final classification.The entry flow extracts multi-scale high-frequency and residual-guided spatial features; the middle flow applies DCMA modules before attention-based fusion.
  • Residual Guided Spatial Attention: Residual-guided spatial attention uses SRM residuals to predict attention maps that guide low-level RGB feature learning toward manipulation traces.Because spatial correspondence between residuals and low-level RGB features is maintained, the attention modules are placed in the entry part.
  • Residual Guided Spatial Attention: Attention visualizations show high responses around abnormal facial boundaries in manipulated faces, whereas responses distribute uniformly in real faces.This pattern indicates that residual-guided spatial attention helps the feature extractor focus on forgery traces.
  • Dual Cross-modality Attention: The dual cross-modality attention module models correlations between RGB and high-frequency features and re-weights each modality using information from the other.It computes cross-modal correlations from key components, produces attention maps for both streams, and can be applied at multiple scales.
  • Feature Fusion and Loss Function: High-level features from both modalities are concatenated, refined with channel-wise attention, and used for prediction under the AM-Softmax objective.The stated objective is to reduce intra-class variations and enlarge inter-class differences relative to regular cross-entropy loss.

5. Experiments

Experiments evaluate the model through ablations and cross-database comparisons, showing that its high-frequency and complementary-modality design improves generalization across manipulation methods and datasets.

  • Experimental Setup: Five benchmark databases are used to evaluate generalization, with FF++ evaluated under raw, lightly compressed, and heavily compressed conditions.The FF++ split uses 740 training, 140 validation, and 140 testing videos.
  • Ablation Study: The ablation study trains on F2F and tests across four datasets, comparing RGB, SRM, two-stream Fusion, RSA, DCMA, and Multi-scale variants.Table 3 reports AUC, with gray results indicating within-dataset performance.
  • Ablation Study: The two-stream model outperforms both RGB and SRM on all datasets, while progressively adding modules improves performance step-by-step.The results indicate complementary information between RGB textures and high-frequency noise features.
  • Cross-Database Generalization: When trained on one FF++ manipulation method and tested on all four, the proposed model exceeds competitors in most cases.The model combines textures and noises and captures blending effects in noise space, unlike texture-dependent Xception and boundary-focused Face X-ray.
  • Cross-Database Generalization: Training on FF++ and testing on DFD, DFDC, CelebDF, and DF1.0 yields apparent improvements over Xception and Face X-ray, especially on DFD, DFDC, and CelebDF.The authors attribute this to more robust representations than Face X-ray’s boundary patterns, which are sensitive to database-specific post-processing.
  • Comparison with Recent Works: On heavily compressed FF++, the method achieves comparable robustness to F3Net and better performance than SRMNet, Bayar Conv, and SSTNet.Video-level predictions average frame-level predictions from 25 sampled images per video.
  • Comparison with Recent Works: The method surpasses multi-task competitors in within-database and cross-database evaluations and outperforms FWA and FFD by more than 15% in AUC on CelebDF.ForensicTrans requires fine-tuning on a few target-domain samples, whereas the comparison uses the proposed model trained on F2F.

6. Conclusion

The paper concludes that CNN detectors overfit to method-specific textures and proposes high-frequency noise features alongside RGB textures for more robust face forgery detection. Three attention and feature-extraction modules support complementary-modality learning, and comprehensive experiments demonstrate effectiveness and superior generalization.

  • Conclusion: CNN-based forgery detectors overfit to method-specific texture patterns, motivating high-frequency noise features that remove color textures and reveal forgery traces.The proposed detector combines color textures with high-frequency noises.
  • Conclusion: The detector uses multi-scale high-frequency extraction, residual guided spatial attention, and dual cross-modality attention to learn complementary representations.The modules extract informative features and model correlation and interaction between the two modalities.
  • Conclusion: Comprehensive experiments demonstrate the effectiveness of each module and corroborate the model’s superior generalization ability against competing methods.
Loading 2103.12376v1…