Source-linked AI summary
Robust Attentive Deep Neural Network for Exposing GAN-generated Faces
Hui Guo, Shu Hu, Xin Wang, Ming-Ching Chang, Siwei Lyu
TL;DR
GAN-face detection methods can perform well on limited public datasets, but those datasets may not reflect real-world views and imbalanced distributions. This paper detects faces by learning eye-iris inconsistencies and jointly using BCE and AUC-related loss; on FFHQ-GAN, it reports its highest performance across metrics against ResNet-50 and Xception.
Problem
Existing GAN-face detection methods can perform well on limited public datasets, but those datasets do not adequately represent real-world view variations and imbalanced face distributions.
Source: §I · pp. 1–2Method
A Residual Attention Network learns to localize inconsistencies between the two iris regions, trained with jointly combined BCE and WMW-AUC losses.
Source: §I · p. 2Results
The proposed method achieves the highest performance in all metrics compared with ResNet-50 and Xception on the balanced and imbalanced FFHQ-GAN datasets.
Source: C. EVALUATION ON THE FFHQ-GAN DATASET · p. 7Takeaways & Limitations
The attention maps highlight whole iris regions for real images and highlight parts for fake images, offering people an intuitive cue for examination.
Source: E. QUALITATIVE RESULTS · p. 7
Abstract
from arXiv · showhide
GAN-based techniques that generate and synthesize realistic faces have caused severe social concerns and security problems. Existing methods for detecting GAN-generated faces can perform well on limited public datasets. However, images from existing public datasets do not represent real-world scenarios well enough in terms of view variations and data distributions (where real faces largely outnumber synthetic faces). The state-of-the-art methods do not generalize well in real-world problems and lack the interpretability of detection results. Performance of existing GAN-face detection models degrades significantly when facing imbalanced data distributions. To address these shortcomings, we propose a robust, attentive, end-to-end network that can spot GAN-generated faces by analyzing their eye inconsistencies. Specifically, our model learns to identify inconsistent eye components by localizing and comparing the iris artifacts between the two eyes automatically. Our deep network addresses the imbalance learning issues by considering the AUC loss and the traditional cross-entropy loss jointly. Comprehensive evaluations of the FFHQ dataset in terms of both balanced and imbalanced scenarios demonstrate the superiority of the proposed method.
I. INTRODUCTION
GAN-face detectors can struggle to generalize beyond limited public data, while real-world face collections are strongly imbalanced. This paper proposes end-to-end detection from inconsistencies between the eyes, using a joint loss for imbalance and evaluating on FFHQ-GAN under balanced and imbalanced conditions.
I. INTRODUCTION
- The method learns to localize inconsistencies between the eyes’ corneal highlights and other artifacts using a Residual Attention Network, rather than relying on low-level highlight segmentation.The authors describe it as data-driven and end-to-end.Source: §I · p. 2
- Real faces usually outnumber GAN-generated faces in real-world examinations, while existing detectors can generalize poorly and lack interpretability.Prior physiology-based detection also assumes frontal faces and distant light reflections from both eyes.Source: §I · pp. 1–2
- Jointly combining binary cross-entropy and WMW-AUC loss yields the best end-to-end result, addressing imbalance through an approximation of direct AUC optimization.The introduction contrasts this approach with sampling and data augmentation.Source: §I · p. 2
- The proposed eye cues are described as interpretable and usable by people examining faces.The paper frames the comparison of the two eyes as a visual examination cue.Source: §I · p. 2
- The authors create FFHQ-GAN from FFHQ real faces and StyleGAN2 images, and report plausible performance especially on imbalanced data.They evaluate balanced and imbalanced conditions and report ablations of the attention module and loss design.Source: §I · p. 2
II. RELATED WORKS
The related-work section reviews data-driven and physical or physiological approaches to GAN-face detection. The detection subsection describes learned feature-based classifiers alongside methods that inspect artifacts and inconsistencies.
A. GAN-GENERATED FACE DETECTION
- Data-driven methods – train deep networks to distinguish real from GAN-generated faces and learn representations in high-dimensional feature spaces.The section contrasts these with physical and physiological methods that inspect signal traces, artifacts, or inconsistencies.Source: A. GAN-GENERATED FACE DETECTION · pp. 2–3
- Physical and physiological approaches include cues such as color differences [33], [34], noise fingerprints,, upsampling artifacts –, facial landmarks, head poses, and eye-highlight inconsistencies.The section notes that simple color cues are no longer effective as GAN methods advance.Source: A. GAN-GENERATED FACE DETECTION · p. 3
B. ATTENTION MECHANISM
The section distinguishes attention used to visualize predictions from attention integrated into a network to focus on relevant features during training. It also reviews sampling and augmentation as approaches to imbalanced learning, noting limits of each.
B. ATTENTION MECHANISM
- CAM and Grad-CAM visualize significant image portions, while integrated attention can guide a network toward relevant regions during training.The section presents these as distinct uses of attention.Source: B. ATTENTION MECHANISM · p. 3
- Residual Attention Network stacks residual attention blocks, combining residual units with attention to improve performance and reduce model complexity.Channel and spatial attention are also combined jointly in,.Source: B. ATTENTION MECHANISM · p. 3
C. IMBALANCED DATA LEARNING
- Sampling can discard important majority-class samples or overfit minority-class examples, while augmentation only partly alleviates imbalance and depends on a diverse original dataset.Examples include undersampling, oversampling, and image mirroring or rotation.Source: C. IMBALANCED DATA LEARNING · p. 3
III. METHOD
The method extracts both iris regions from a face and classifies the pair with a residual-attention network. It targets ROC-AUC performance using a WMW relaxation because the discrete AUC values do not support naive gradient-based end-to-end optimization.
III. METHOD
- DLib localizes facial landmarks, Mask R-CNN segments the two iris regions, and a residual-attention network classifies the iris pair as real or fake.The network is trained to maximize ROC-AUC, with WMW statistics used to approximate that objective because direct gradient-based optimization does not work end-to-end.Source: §III · pp. 3–4
SEGMENTATION
- Mask R-CNN uses a Region Proposal Network to generate candidate boxes, then extracts proposal features with RoIAlign before classification, box regression, and parallel mask prediction.The model is trained using eye-region images from datasets,.Source: A. FACIAL LANDMARK LOCALIZATION AND… · p. 4
B. RESIDUAL ATTENTION NETWORK
The Residual Attention Network processes extracted iris pairs and uses attention to focus on inconsistencies between the eyes. Its attention modules combine trunk features with learned masks, and the network outputs a score indicating whether an input is GAN-generated.
B. RESIDUAL ATTENTION NETWORK
- Each Attention Module combines a residual-block trunk branch with a U-net soft-mask branch to weight the input feature map.The attended feature map is formed by element-wise multiplication of the trunk output and soft mask.Source: B. RESIDUAL ATTENTION NETWORK · p. 4
- Mixed attention weights each channel and spatial location, while adding the attention map to the original feature map supports stacking modules like a ResNet.The paper says mixed attention yields the best performance and uses a sigmoid function for its weights.Source: B. RESIDUAL ATTENTION NETWORK · pp. 4–5
- The RAN receives extracted iris pairs and outputs a score indicating the likelihood that an input is GAN-generated.The extracted pairs can expose inconsistent corneal specular highlights in GAN-generated faces.Source: Figure 4 · p. 4, Figure 5 · p. 5, B. RESIDUAL ATTENTION NETWORK · p. 5
C. AUC OF ROC FOR CLASSIFICATION EVALUATION
The section motivates AUC-based evaluation and optimization as a way to address imbalanced classification, where common losses can yield biased predictions. It defines ROC through threshold-dependent false- and true-positive rates, with AUC measuring the area under that curve.
C. AUC OF ROC FOR CLASSIFICATION EVALUATION
- Common classification losses, including cross-entropy, can produce accurate but biased predictions on imbalanced data, motivating loss designs that address imbalance directly.The section proposes directly maximizing AUC because it is described as robust for balanced and imbalanced data.Source: C. AUC OF ROC FOR CLASSIFICATION… · p. 5
- For labeled samples, the classifier assigns prediction scores, and a threshold λ determines which examples count as false positives and true positives.The passage distinguishes positive and negative instances by their labels and defines the classifier's prediction score for each sample.Source: C. AUC OF ROC FOR CLASSIFICATION… · p. 5
- The ROC plots false-positive rate against true-positive rate across thresholds, and AUC is the area enclosed by the curve.Source: C. AUC OF ROC FOR CLASSIFICATION… · p. 5
D. WMW AUC RELAXATION FOR LOSS DESIGN
The paper relaxes the discrete AUC objective into a differentiable loss and combines it with binary cross-entropy. The AUC component measures incorrect ranking of positive-negative prediction pairs independently of a classification threshold.
D. WMW AUC RELAXATION FOR LOSS DESIGN
- The AUC loss is the fraction of positive-negative score pairs ranked incorrectly, and it is independent of the classification threshold.A perfect ranking gives an AUC loss of 0.Source: D. WMW AUC RELAXATION FOR LOSS DESIGN · p. 5
- The discrete AUC objective is nondifferentiable, so the method uses a differentiable approximation inspired by the Wilcoxon-Mann-Whitney statistic.The approximation uses hyperparameters γ and p.Source: D. WMW AUC RELAXATION FOR LOSS DESIGN · pp. 5–6
- Training combines binary cross-entropy and AUC loss in a weighted sum, with α balancing their contributions.Source: D. WMW AUC RELAXATION FOR LOSS DESIGN · p. 6
IV. EXPERIMENT
The experiment evaluates the proposed method against state-of-the-art methods on the newly constructed FFHQ-GAN dataset. It covers balanced and imbalanced subsets, implementation details, quantitative evaluation, ablations, and qualitative results.
IV. EXPERIMENT
- The experimental section covers dataset construction, implementation, balanced and imbalanced performance evaluation, ablations, and qualitative results.Source: §IV · p. 6
A. THE NEW FFHQ-GAN DATASET
The paper constructs FFHQ-GAN from FFHQ real faces and StyleGAN2-generated faces, then evaluates the approach on balanced and imbalanced subsets. Implementation uses iris pairs as input to a Residual Attention Network.
A. THE NEW FFHQ-GAN DATASET
- After discarding images without detected iris pairs, FFHQ-GAN contains 3,739 real faces and 3,748 fake faces, split 8:2 for training and testing.The source images are sampled from FFHQ and StyleGAN2-generated faces.Source: A. THE NEW FFHQ-GAN DATASET · p. 6
- The dataset is sampled to form an imbalanced subset alongside the balanced subset for evaluating both data scenarios.Source: A. THE NEW FFHQ-GAN DATASET · p. 6, Table 2 · p. 6
B. IMPLEMENTATION DETAILS
- The pipeline localizes both iris regions and resizes iris pairs to 96 × 96 pixels before feeding them to the Residual Attention Network.The network uses three stacked Attention Modules and is trained with Adam at learning rate 0.001 and batch size 128.Source: B. IMPLEMENTATION DETAILS · p. 6
- For imbalanced data, α is set to 0.4; for balanced data, it is 0.2, with p = 2 in both settings.The selected hyperparameters are reported as yielding the best performance.Source: B. IMPLEMENTATION DETAILS · p. 6
C. EVALUATION ON THE FFHQ-GAN DATASET
On the FFHQ-GAN balanced and imbalanced datasets, the proposed method achieves the highest performance across all reported metrics. Ablations report gains from the joint BCE+AUC loss and identify α = 0.4 as best for imbalanced data.
C. EVALUATION ON THE FFHQ-GAN DATASET
- The proposed method achieves the highest performance across all reported metrics on both balanced and imbalanced FFHQ-GAN data.ResNet-50 and Xception obtain low recall on the imbalanced data.Source: C. EVALUATION ON THE FFHQ-GAN DATASET · p. 7
D. ABLATION STUDIES
- The joint BCE+AUC loss outperforms BCE-only training for the same RAN model across all evaluation metrics.The comparison uses results on the balanced and imbalanced dataset.Source: D. ABLATION STUDIES · p. 7
- For imbalanced-data detection, α = 0.4 yields the best detection performance among tested α values ranging from 0 to 1.Figure 8 reports obtained AUC scores versus α.Source: D. ABLATION STUDIES · p. 7
E. QUALITATIVE RESULTS
The qualitative maps distinguish real and GAN-generated faces by where the network attends, while the conclusion reports superior detection performance on balanced and imbalanced datasets.
E. QUALITATIVE RESULTS
- Figure 9 shows attention spread across real irises but focused on highlight artifacts in GAN-generated irises, offering an intuitive cue for human comparison.Source: E. QUALITATIVE RESULTS · p. 7, Figure 9 · p. 8