PaperlayerView exact revision on arXiv

Source-linked AI summary

Fusing Global and Local Features for Generalized AI-Synthesized Image Detection

Yan Ju, Shan Jia, Lipeng Ke, Hongfei Xue, Koki Nagano, Siwei Lyu

arXiv:2203.13964v2cs.CV

TL;DR

AI-synthesized image detectors perform well when training and testing use similar data but tend to generalize poorly to open-world images from unknown models or source data. This paper combines global image features with informative local patches and reports high accuracy and good generalization on a diverse test set spanning 19 models.

  • Problem

    Detectors perform well in closed settings but tend to lose performance on open-world images generated by unknown models or from unseen source data.

    Source: §1 · p. 1

  • Method

    A two-branch model combines whole-image global features with local features from automatically selected informative patches, then fuses them using attention.

    Source: §1 · p. 2

  • Results

    Experiments on a diverse test set spanning 19 generation models demonstrate high accuracy and good generalization for AI-synthesized image detection.

    Source: §1 · p. 2, §4.1 · p. 3, §5 · p. 4

  • Takeaways & Limitations

    The authors conclude that their model shows good generalization for detecting AI-synthesized images across a diverse set of generation models.

    Source: §1 · p. 2, §5 · p. 4

Abstract

from arXiv · show

With the development of the Generative Adversarial Networks (GANs) and DeepFakes, AI-synthesized images are now of such high quality that humans can hardly distinguish them from real images. It is imperative for media forensics to develop detectors to expose them accurately. Existing detection methods have shown high performance in generated images detection, but they tend to generalize poorly in the real-world scenarios, where the synthetic images are usually generated with unseen models using unknown source data. In this work, we emphasize the importance of combining information from the whole image and informative patches in improving the generalization ability of AI-synthesized image detection. Specifically, we design a two-branch model to combine global spatial information from the whole image and local informative features from multiple patches selected by a novel patch selection module. Multi-head attention mechanism is further utilized to fuse the global and local features. We collect a highly diverse dataset synthesized by 19 models with various objects and resolutions to evaluate our model. Experimental results demonstrate the high accuracy and good generalization ability of our method in detecting generated images. Our code is available at https://github.com/littlejuyan/FusingGlobalandLocal.

1. INTRODUCTION

AI-synthesized image detectors perform well in closed settings but can falter on images from unseen models or data. This work combines global image features with informative local patches to improve detection generalization.

1. INTRODUCTION

  • Detectors perform well when training and testing images share models and similar source data, but their performance can decline in open-world settings with unseen models or data.The cited evaluation considers cross-model, cross-data, and post-processing conditions and finds current detectors insufficiently robust for real applications.Source: §1 · p. 1
  • Real/fake differences can be subtle and local, while manipulations range from whole-image synthesis to local edits, challenging detectors that rely only on global features.Examples include refined global image quality with subtle local artifacts and partial manipulations such as face-region swaps.Source: §1 · p. 1, Figure 2 · p. 2
  • The proposed two-branch model extracts overall structural features from the whole image and fine-grained artifacts from automatically selected informative patches, then fuses them for binary classification.Its Patch Selection Module avoids random or manually predefined patches, and its Attention-based Feature Fusion Module combines the two branches.Source: §1 · p. 2
  • The authors collect a diverse dataset spanning resolutions, manipulation types, and synthesis models, and report that their method has outstanding generalization ability.The framework combines whole-image global features with local features from informative patches to enhance detection generalization.Source: §1 · p. 2

2. RELATED WORK

Prior work spans GAN- and autoencoder-based image synthesis and detection methods that emphasize either whole-image features or informative local patches, each with noted limitations.

Related work

  • Unconditional GANs synthesize images from random vectors, conditional GANs such as CycleGAN generate specified image classes or styles, and autoencoders support partial manipulations.StyleGAN-V1/2/3 are cited as unconditional models that generate realistic high-resolution images; FaceForensic++ [20] and Celeb-DF are large-scale datasets for manipulated media.Source: §2 · p. 2
  • Whole-image binary classifiers [9] [21] extract global features but often miss subtle local artifacts.The cited detector studies use deep neural networks to distinguish real from synthetic or manipulated images.Source: §2 · p. 2
  • Patch-based detection emphasizes informative local features, but focusing too heavily on local artifacts may limit generalization to whole and partial forgeries.Study uses limited receptive fields and reports excellent generalization across face-forgery datasets; evaluates individual facial parts.Source: §2 · p. 2

3. METHOD

The method combines global spatial features with informative local patches selected from the input image, then uses multi-head attention to fuse their embeddings for classification.

3.1. Overall Architecture

  • The two-branch model extracts global features from the image, selects informative patches for local feature extraction, and fuses both branches’ embeddings for classification.The overall architecture uses a CNN backbone such as ResNet50 for the global branch, a Patch Selection Module (PSM) for patch coordinates, and an Attention-based Feature Fusion Module (AFFM).Source: §3.1 · p. 2

3.2. Global Branch

  • The global branch applies a CNN to obtain final-convolution feature maps, then average pooling and fully connected layers produce embeddings containing spatial and structural information.The feature maps are denoted F ∈ RC×H×W, with C channels and spatial size H × W.Source: §3.2 · p. 2

3.3. Local Branch with Patch Selection Module

  • The PSM scores candidate patches by aggregating global feature maps across channels and averaging activation within sliding windows, then selects high-scoring, nonredundant patches for local CNN embeddings.It maps selected window coordinates back to the input image and crops the patches; non-maximum suppression reduces redundancy.Source: §3.3 · pp. 2–3

3.4. Attention-based Feature Fusion Module

  • AFFM uses multi-head attention over concatenated global and patch embeddings to learn their correlations and weights, producing a fused feature for the final classifier.The concatenated embeddings serve as query, key, and value; the concatenated attention heads are linearly transformed into the fused feature.Source: §3.4 · p. 3

4. EXPERIMENTS

The experiments evaluate Our PSM on a diverse test set spanning 19 generation models. Results report the best overall detection performance, robustness to post-processing, and gains over alternative patch-selection methods.

4.1. Datasets

  • Testing covered 128,424 images from 19 generation models across six families, with diverse objects and resolutions and both whole-image synthesis and face swaps.Training used 362K real LSUN images and 362K ProGAN-generated images from 20 object classes.Source: §4.1 · p. 3

4.2. Implementation Details

  • Implementation uses a ResNet-50 global/local pipeline with PSM-selected patches, fused by three four-head attention layers.The model was trained with Adam and BCE; AP was used for evaluation, with mAP averaged across families and baselines trained on the same training set.Source: §4.2 · pp. 3–4

4.3. Evaluation for AI-synthesized Image Detection

  • Our PSM achieves the best Total mAP and Global AP, and the highest mAP for most unseen model families, while trailing No down on Low-level Vision and DeepFakes.Table 2 reports results across seen and unseen models, including family mAP, Total mAP, and Global AP.Source: §4.3 · p. 4, Table 2 · p. 4
  • All methods achieve 100% accuracy on seen ProGAN except Patch forensics, which is limited to face detection and performs poorly on diverse test objects.Source: §4.3 · p. 4
  • Our PSM outperforms random cropping and resize-to-448 patch selection across all families except Perceptual Loss, and performs best overall.The authors attribute this to selecting the most distinguishable patches while balancing image information and training efficiency.Source: §4.3 · p. 4, Table 2 · p. 4
  • Our method shows good robustness to Gaussian blur and JPEG compression on diverse unseen images; the authors note training augmentation may help.Source: §4.3 · p. 4, Figure 3 · p. 4

5. CONCLUSION

The proposed two-branch model fuses whole-image structure with subtle features from informative local patches for generalized AI-synthesized image detection. Experiments on a diverse test dataset spanning 19 models show high accuracy and good generalization.

5. CONCLUSION

  • Experiments on a highly diverse test dataset containing 19 models show that the method achieves high accuracy and good generalization ability.Source: §5 · p. 4
  • The two-branch model combines whole-image structure with subtle local features selected by PSM, then fuses them with multi-head attention for binary classification.Source: §5 · p. 4
  • Using global high-level features, the model unsupervisedly focuses on the most discriminative local patches; future work will explore differentiable patch selection.Source: §5 · p. 4

Related papers

Loading 2203.13964v2…