PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Detection of Digital Face Manipulation

Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, Anil Jain

arXiv:1910.01717v5cs.CV

TL;DR

Digital face manipulation creates a need to detect fake facial images and localize altered regions. This paper introduces an attention-based detector, a diverse facial-forgery dataset, and an attention-map metric, showing improved detection and localization with state-of-the-art performance across seen and unseen manipulation methods.

  • Problem

    Increasingly realistic and widespread facial forgeries create a need for effective detection and localization methods to address security, privacy, and social-media concerns.

  • Method

    The paper combines an attention-based feature-processing layer, a diverse facial-forgery dataset, and the IINC metric for evaluating manipulated-region maps.

  • Results

    The attention mechanism improves forgery detection and manipulated-region localization, achieving state-of-the-art performance across seen and unseen manipulation methods.

  • Takeaways & Limitations

    A unified attention-based approach can detect diverse facial manipulation attacks while highlighting informative and manipulated facial regions.

Abstract

from arXiv · show

Detecting manipulated facial images and videos is an increasingly important topic in digital media forensics. As advanced face synthesis and manipulation methods are made available, new types of fake face representations are being created which have raised significant concerns for their use in social media. Hence, it is crucial to detect manipulated face images and localize manipulated regions. Instead of simply using multi-task learning to simultaneously detect manipulated images and predict the manipulated mask (regions), we propose to utilize an attention mechanism to process and improve the feature maps for the classification task. The learned attention maps highlight the informative regions to further improve the binary classification (genuine face v. fake face), and also visualize the manipulated regions. To enable our study of manipulated face detection and localization, we collect a large-scale database that contains numerous types of facial forgeries. With this dataset, we perform a thorough analysis of data-driven fake face detection. We show that the use of an attention mechanism improves facial forgery detection and manipulated region localization.

1. Introduction

Digital facial manipulation creates security and privacy concerns across physical spoofing, adversarial, and digital manipulation attacks. This work addresses limited evaluation data by introducing a large-scale dataset and an attention-based approach that detects forgeries, localizes manipulated regions, and evaluates attention maps.

  • Motivation: Facial forgery attacks comprise physical spoofing, adversarial attacks, and digital manipulation attacks.Examples include printed or replayed faces and 3D masks, imperceptible adversarial images, and digitally manipulated facial content.
  • Background: Digital facial manipulation methods include expression swap, identity swap, attribute manipulation, and entire face synthesis.These methods support applications ranging from real-time expression transfer and face swapping to modifying attributes such as gender, age, skin color, hair, and glasses.
  • Dataset: 2.6 million images comprise the Diverse Fake Face Dataset, addressing prior reliance on small datasets with limited manipulation types.The dataset contains 0.8M real and 1.8M fake faces generated by diverse face modification methods, with an accompanying evaluation protocol.
  • Method: The proposed attention-based layer improves classification performance while producing attention maps that indicate manipulated facial regions.The method outputs a genuine-versus-manipulated decision and uses spatial information to focus on important regions for detection.
  • Evaluation: The Inverse Intersection Non-Containment metric evaluates attention maps with more coherent evaluation than existing metrics.The work also reports state-of-the-art digital facial forgery detection for seen and unseen manipulation methods.

2. Related Work

Prior work uses graphics-based 3D reconstruction and geometry warping for facial identity or expression transfer, while existing datasets were limited to two fake types. The paper addresses this limitation by collecting a dataset spanning identity and expression swaps, face-attribute manipulations, and fully synthetic faces.

  • Digital Face Manipulation Methods: Graphics-based methods reconstruct 3D source and target faces and use corresponding geometry to warp between them for identity or expression transfer.
  • Digital Face Manipulation Methods: Existing face-manipulation datasets were limited to identity and expression swaps, including Deepfake and FaceSwap manipulations.
  • Digital Face Manipulation Methods: The collected dataset includes identity and expression swaps, FaceAPP-based attribute manipulations, and complete synthetic faces generated with StyleGAN and PGGAN.

3. Proposed Method

The proposed method treats manipulated-face detection as CNN-based binary classification augmented by an attention mechanism that highlights manipulated regions and refines classification features. The attention module is designed for explainability, usefulness, and modularity, with supervised, weakly supervised, and unsupervised training options.

  • Attention mechanism: An attention map processes classifier feature maps to identify regions influencing the CNN decision and guide discovery of discriminative features.Each attention-map pixel represents the probability that its receptive field corresponds to a manipulated region.
  • Design objectives: The attention map predicts modified pixels, masks irrelevant high-dimensional features, and provides an auxiliary explanation of the spatial evidence underlying classification.The design targets images that may be entirely or partially manipulated.
  • Design objectives: Feeding the attention map back into the network suppresses non-activated regions so partial fake images are classified using features from manipulated regions.The module is intended to enhance final binary classification performance.
  • Implementation: The modular attention layer can plug into existing or pre-trained backbone networks by adding a convolutional layer, associated losses, and feature masking.The layer can be applied to any classification feature map and supports direct convolutional regression or manipulation appearance modeling.
  • Training: Attention-map learning supports supervised masks, weak supervision using a maximum fake-map response of 0.75, and unsupervised training.For supervised learning, real faces use zero-maps, entirely synthesized faces use one-maps, and partial manipulations derive masks from source-image differences thresholded at 0.1.

4. Diverse Fake Face Dataset

The Diverse Fake Face Dataset (DFFD) was constructed to support face manipulation detection and localization through greater diversity than previous datasets. It covers identity and expression swaps, attribute manipulation, entire face synthesis, and varied real-face samples.

  • Dataset construction: DFFD combines four facial forgery categories: identity swap, expression swap, attribute manipulation, and entire synthesized faces.The dataset’s greater diversity is described as crucial for detecting and localizing face manipulations.
  • Real faces: Real samples come from FFHQ, CelebA, and FaceForensics++, covering variation in race, age, gender, pose, illumination, expression, resolution, and camera quality.Both real and fake samples include low- and high-quality images to reduce bias in gender, age, and face-size distributions.
  • Identity and expression swap: FaceForensics++ contributes 1,000 real videos and 3,000 manipulated versions spanning identity swaps from FaceSwap and Deepfake and expression swaps from Face2Face.Additional identity-swap videos were collected from a public website and generated by Deep Face Lab.
  • Attributes manipulation: Attribute manipulations are generated with FaceAPP and StarGAN using 4,000 FFHQ faces and 2,000 CelebA faces as input real images.FaceAPP provides 28 filters for modifying attributes such as gender, age, hair, beard, and glasses.
  • Entire face synthesis: PGGAN and StyleGAN generate 200k and 100k high-quality entire fake images, respectively.These pretrained models produce the entire synthesized-face category of DFFD.
  • Protocols: The collection contains 781,727 real samples and 1,872,007 fake samples, with a selected subset of 58,703 real and 240,336 fake images for dataset manageability and balance.Ground-truth manipulation masks are generated for fake images, and source-derived fakes use the source real face’s landmarks for cropping consistency.

5. Experimental Results

Experiments show that attention-based processing improves facial forgery detection and localization across backbone networks, with performance depending on supervision and map-estimation strategy. The study also introduces IINC to evaluate attention maps more meaningfully than existing metrics.

  • Supervised learning outperforms weakly supervised and unsupervised learning for both detection and localization accuracy.
  • Regression-based attention maps perform best with supervision, whereas MAM-based maps are superior in weakly supervised and unsupervised settings.MAM provides a strong constraint for map estimation in settings with limited supervision.
  • Effect on Backbone Networks: The attention mechanism improves detection on both XceptionNet and VGG16 backbones.With XceptionNet, directly produced attention maps better predict manipulated regions; with VGG16, MAM priors reduce parameter-space contention.
  • Benefit of Attention map: At low FDR, direct regression produces the best-performing attention network, and the attention layer substantially outperforms conventional XceptionNet.The improvement is especially pronounced for facial identity and expression swaps.
  • Manipulation Localization: The proposed IINC metric evaluates non-overlap ratios for both predicted and ground-truth maps, addressing limitations of IoU, Cosine Similarity, and PBCA.IINC remains discriminative when existing metrics assign identical scores to maps with substantially different properties.

6. Conclusion

The paper addresses manipulated-face detection and localization with an attention-based method that improves detection features while highlighting manipulated regions. It also introduces a facial forgery dataset containing diverse fake-face types.

  • The method uses attention to process detection-model feature maps, highlighting informative and manipulated facial regions.These attention maps support both detection improvement and visualization of manipulated regions.
  • The authors collect a facial forgery dataset containing diverse types of fake faces.

A. Details

This section details the Diverse Fake Face Dataset and the proposed attention map, then analyzes additional experiments.

  • The section further details the Diverse Fake Face Dataset.
  • It further details the proposed attention map.
  • It analyzes additional experiments.

A.1. DFFD Dataset Details

The DFFD was built from widely used facial-recognition datasets, using FFHQ and CelebA as sources of real images and manipulated-image generation. It spans substantial variation in face size and human age across both real and manipulated images.

  • Dataset construction: The DFFD was constructed from large, commonly used facial-recognition datasets.The paper specifically identifies FFHQ and CelebA as the real-image sources.
  • Dataset construction: FFHQ and CelebA were selected as real-image sources for generating manipulated images.Their widespread use in facial recognition supports this dataset construction choice.
  • Dataset diversity: The DFFD covers wide variation in face size and human age for both real and manipulated images.Figure 10 illustrates this variance across the two image categories.

A.2. Network Architecture Details · B. Additional Experimental Results · B.1. Human Study

The attention layer is modular within Xception, while a human study evaluates real-versus-manipulated image classification and region marking. Human judgments emphasize semantic artifacts, whereas the proposed solution detects subtle image fingerprints.

  • A.2. Network Architecture Details: The attention layer can be added modularly to the Xception network and placed at different locations.Its placement is illustrated in Fig. 11.
  • B.1. Human Study: The human study included 10 participants evaluating 110 randomly selected DFFD images.The sample contained 10 images from each row in Table 6.
  • B.1. Human Study: Participants classified each image as Real, Entire Fake, or Partial Fake.The task covered distinctions among real, entirely manipulated, and partially manipulated images.
  • B.1. Human Study: For Partial Fakes, participants additionally provided polygon-based regions of interest as attention maps.These regions represented the areas humans identified as relevant to the manipulation.
  • B.1. Human Study: Humans largely focused on image quality, large artifacts, and unusual lighting or color when judging real and fake images.These judgments emphasize semantic concepts rather than subtle forensic cues.
  • B.1. Human Study: The proposed solution detected subtle differences in the image “fingerprint” that humans did not detect.Figure 12 compares manipulation maps produced by the proposed solution with maps produced by humans.

B.2. Additional Performance Evaluation

The additional evaluation analyzes the best-performing Xception Regression Map with supervision through failure cases and fake-type-specific accuracy. Completely synthesized images are reported as easiest to detect, while high-quality manipulated or synthetic images can cause failures.

  • Evaluation Scope: The best-performing model, Xception Regression Map with supervision, is evaluated through worst-case test samples and accuracy across fake types.The analysis covers real, identity-manipulated, expression-manipulated, completely generated, and attribute-modified faces.
  • Failure Analysis: Worst real-face cases include heavy makeup and low image quality, while manipulated or entirely synthetic failures are high quality and artifact-free.The real samples have the lowest Softmax probability of being classified as real.
  • Fake-Type Accuracy: Completely synthesized images are the easiest fake type to detect according to the fake-type-specific accuracy evaluation.The passage attributes this observation to an explanation that is truncated in the supplied text.
  • Evaluation Materials: The evaluation materials include DFFD composition statistics, XceptionNet architecture and attention placement, attention-map comparisons with humans, and human-versus-model detection and localization results.The XceptionNet middle flow contains eight blocks, and the attention layer can be added after any block.

B.3. Additional Ablation Study

The ablation study shows that attention-layer placement trades off classification performance against attention-map refinement: earlier placement improves AUC and EER, while later placement improves PBCA. A threshold of 0.1 provides strong, semantically reasonable performance when binarizing attention maps.

  • Attention-layer placement: Earlier attention-layer placement lowers AUC and EER, whereas later placement increases PBCA in the Xception middle flow.Later placement enables more finely tuned attention maps through greater computational flexibility and depth.
  • Attention-layer placement: Earlier attention gives the network more time to inspect selected features and distinguish real from manipulated images semantically.This explains why earlier attention can benefit classification despite producing less finely tuned maps.
  • Map binarization: A threshold of 0.1 yields strong performance when converting continuous predicted or ground-truth attention maps into binary values.The other map’s threshold was fixed at 0.1 in each graph.
  • Map binarization: A 0.1 map-value modification corresponds to magnitude 25 in the typical RGB range of [0,255].Modifications below 10 are almost undetectable by humans, while those above 25 are sign...
Loading 1910.01717v5…