PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery Detection And Segmentation In-The-Wild

Trung-Nghia Le, Huy H. Nguyen, Junichi Yamagishi, Isao Echizen

arXiv:2107.14480v1cs.CV

TL;DR

Prior forgery-recognition methods handle one cropped face at a time, leaving multi-face detection and segmentation in unrestricted scenes unaddressed. The paper introduces those tasks, builds the face-wise annotated OpenForensics dataset, and evaluates detection and segmentation methods across scenarios.

  • Problem

    Prior face-forgery methods process one cropped face at a time, depend on independent face detection, and have been evaluated in simple laboratory settings; the paper also identifies a lack of large-scale data for the new tasks.

    Source: §1 · pp. 1–2, §2.3 · p. 3

  • Method

    The authors construct OpenForensics using a forgery workflow and establish benchmarks for multi-face forgery detection and segmentation.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    OpenForensics contains 115K unrestricted images and 334K human faces with face-wise annotations, and the paper reports extensive benchmarking of state-of-the-art instance segmentation methods across settings.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The dataset supports multi-face forgery detection and segmentation as well as conventional tasks involving general human faces.

    Source: §1 · p. 2

Abstract

from arXiv · show

The proliferation of deepfake media is raising concerns among the public and relevant authorities. It has become essential to develop countermeasures against forged faces in social media. This paper presents a comprehensive study on two new countermeasure tasks: multi-face forgery detection and segmentation in-the-wild. Localizing forged faces among multiple human faces in unrestricted natural scenes is far more challenging than the traditional deepfake recognition task. To promote these new tasks, we have created the first large-scale dataset posing a high level of challenges that is designed with face-wise rich annotations explicitly for face forgery detection and segmentation, namely OpenForensics. With its rich annotations, our OpenForensics dataset has great potentials for research in both deepfake prevention and general human face detection. We have also developed a suite of benchmarks for these tasks by conducting an extensive evaluation of state-of-the-art instance detection and segmentation methods on our newly constructed dataset in various scenarios. The dataset, benchmark results, codes, and supplementary materials will be publicly available on our project page: https://sites.google.com/view/ltnghia/research/openforensics

1. Introduction

The paper frames multi-face forgery localization in unrestricted scenes as an unmet countermeasure need, then introduces new tasks, a synthesis workflow, OpenForensics, and benchmarks.

1. Introduction

  • Multi-face forgery detection and segmentation in-the-wild address a gap left by single-face methods that rely on separate face detection and were evaluated in simple laboratory scenes.The paper attributes the lack of such methods partly to the absence of a large-scale dataset.Source: §1 · pp. 1–2
  • The proposed GAN workflow generates an unlimited number of fake identities for non-target face-swapping without repeatedly training a deepfake AE, reducing fake-data synthesis costs.Source: §1 · p. 2
  • OpenForensics is a large-scale image dataset for these tasks, with face-wise labels for forgery category, bounding boxes, segmentation masks, forgery boundaries, and facial landmarks.The dataset also supports conventional tasks involving general human faces.Source: §1 · p. 2
  • The paper presents a benchmark suite and evaluates state-of-the-art instance detection and segmentation models in various scenarios.Source: §1 · p. 2

2. Related Work

Existing forensic datasets largely support single-face classification and can contain repeated backgrounds, while OpenForensics is designed for multi-face detection and segmentation in diverse scenes. The paper also describes a face-generation approach that avoids repeatedly training autoencoders and reviews classification methods’ lack of localization ability.

2.1. Existing Forensic Datasets

  • Existing video-based forensic datasets can repeat backgrounds and were developed for single-face classification, whereas OpenForensics uses diverse backgrounds and targets multi-face detection and segmentation.The paper also notes that these tasks require more annotation than classification.Source: §2.1 · pp. 2–3

2.2. Face Manipulation and Generation

  • The proposed synthesis approach transforms GAN-generated high-quality faces into original poses, producing many fake identities without repeatedly training autoencoders.Existing deepfake methods require face pairs and image sequences, and their generated faces are usually low-resolution.Source: §2.2 · p. 3

2.3. Face Forgery Classification

  • Face forgery classification methods process cropped faces and lack localization, while this work addresses multi-face detection and segmentation in-the-wild.Current data-driven classification methods train on real and fake images or videos and do not rely on specific artifacts.Source: §2.3 · p. 3

3. Large-Scale OpenForensics Dataset

OpenForensics was built to support multi-face forgery detection and segmentation in-the-wild, combining synthesized forged faces, challenging image scenarios, and face-wise annotations. Its user study and dataset analyses characterize the dataset’s diversity and the difficulty of recognizing its forgeries.

3.1.1 Real Human Image Collection

  • 45,473 real-person images were collected from Google Open Images after removing images without people and unreal or human-like faces.These images were used as pristine data for dataset construction.Source: §3.1.1 · p. 4

3.1.2 Forged Face Image Synthesis

  • The synthesis workflow generates new identities and accepts swaps only when they spoof a simple classifier, enabling unlimited fake identities at low cost without paired training.It produces 512×512 faces and uses Poisson blending, color adaptation, and 68 facial landmarks for smoother masks and finer boundaries than rectangular or rough convex-hull masks.Source: §3.1.2 · p. 4

3.1.3 Challenging Scenario Augmentation

  • 45,000 test-challenge images were produced using random perturbations at easy, medium, and hard intensity levels.The operators include color and edge manipulation, block-wise distortion, image corruption, convolution-mask transformations, and external effects.Source: §3.1.3 · pp. 4–5

3.2. Dataset Description

  • OpenForensics is one of the largest detection and segmentation datasets, large enough to train and evaluate deep networks, and provides face-wise labels for forgery category, bounding box, segmentation mask, forgery boundary, and facial landmarks.Source: §3.2 · p. 5
  • OpenForensics has an order of magnitude more scenarios than existing datasets, varied scenes and face sizes, and 2.9 faces per image on average.The dataset also includes high-resolution images and diverse face ages and genders; faces tend to lie near image centers.Source: §3.2 · p. 5, Figure 6 · p. 5

3.3. User Study

  • OpenForensics forged faces received a mean opinion score of 4.0, versus 3.2 for the second-best dataset, Celeb-DF.Participants rated 200 cropped forged heads on a five-point realism scale.Source: §3.3 · p. 6
  • 59.7% overall accuracy on cropped-head classification was the lowest among the five datasets, indicating participants had the most trouble distinguishing OpenForensics real and fake images.The other reported accuracies were 68.7% for Celeb-DF, 72.0% for DFDC, 82.0% for FaceForensics++, and 82.9% for DeeperForensics.Source: §3.3 · p. 6
  • 86% accuracy when images had no forged faces fell to 0% when the number of forged faces exceeded seven in the multi-face study.Accuracy tended to decrease as the number of forged faces increased.Source: §3.3 · p. 7

4. Benchmark Suite

The benchmark compares instance detection and segmentation methods using AP and oLRP across standard and challenge images. BlendMask led on standard images, while all methods weakened on unseen images and YOLACT++ and BlendMask were most robust.

4.1. Baseline Methods

  • The benchmark evaluated two-stage and single-stage instance detection and segmentation methods, including methods that directly output masks without bounding boxes.All methods used the same FPN-ResNet50 backbone and were trained for 12 epochs with ImageNet initialization.Source: §4.1 · p. 7

4.2. Evaluation Metrics

  • Evaluation used COCO-style mean AP and scale-specific AP, alongside mean optimal LRP error and its localization, false-positive, and false-negative components.Higher AP and lower oLRP error indicate better performance.Source: §4.2 · p. 7, Figure 11 · p. 7

4.3. Overall Evaluation

  • BlendMask had the highest AP and lowest oLRP error for detection and segmentation on standard images; YOLACT++ and BlendMask were most robust on unseen images.The two-stage methods had medium performance, while YOLACT methods had the worst performance on both standard-image tasks.Source: Figure 11 · p. 7, §4.3 · p. 7

4.4. Multi-Face Forgery Detection Benchmark

  • For detection, BlendMask achieved the highest AP (87.0) and lowest oLRP error (19.5), and led AP at every object scale.Modern single-stage methods had minor location errors and false positive rates, while MaskRCNN and MSRCNN had low false negative rates.Source: §4.4 · p. 7, Table 4 · p. 8

4.5. Multi-Face Forgery Segmentation Benchmark

  • For segmentation, BlendMask led with AP of almost 90 and oLRP error of approximately 18 on test-dev; performance decreased with face size.All baseline methods performed highly only for medium and large faces, with a marginal difference between small and medium/large faces; single-stage methods had fewer false positives and two-stage methods fewer false negatives.Source: §4.5 · pp. 7–8, Table 4 · p. 8

4.6. Robustness Evaluation

  • All methods showed a substantial performance drop on unseen images, with top AP below 60 on the test-challenge set.The authors conclude that in-the-wild multi-face forgery detection and segmentation remain far from solved.Source: §4.6 · p. 8

5. Conclusion and Outlook

The paper presents OpenForensics and benchmarks for multi-face forgery detection and segmentation in-the-wild. Its rich annotations are intended to support further forgery and general face research.

5. Conclusion and Outlook

  • The study created a large-scale dataset and evaluated state-of-the-art instance segmentation methods across experimental settings.The authors expect OpenForensics to boost deepfake-prevention research and intend to enlarge the dataset as deepfake technology develops.Source: §5 · p. 8
  • The authors identify weakly, semi-, and self-supervised methods, multi-task networks, robustness, boundary detection, ranking, anonymization, and general face tasks as research directions.The listed general face tasks include face detection and segmentation and facial landmark prediction.Source: §5 · p. 8
  • OpenForensics’ rich annotations are described as a basis for both forgery studies and general face research.The authors intend to continue enlarging the dataset to accompany future developments in deepfake technology.Source: §5 · p. 8

Related papers

Loading 2107.14480v1…