Source-linked AI summary
Fusing Global and Local Features for Generalized AI-Synthesized Image Detection
Yan Ju, Shan Jia, Lipeng Ke, Hongfei Xue, Koki Nagano, Siwei Lyu
TL;DR
Existing detectors often generalize poorly when synthetic images come from unseen models, domains, or source data. The paper combines whole-image and informative local features through patch selection and multi-head attention, and reports high accuracy and good generalization across a diverse 19-model evaluation dataset.
Problem
Existing detectors perform well in closed settings but degrade considerably for images from different domains, unknown models, or unseen source data.
Method
A two-branch model combines global whole-image structure with local features from automatically selected informative patches, then fuses them using multi-head attention.
Results
The proposed method achieves high accuracy and good generalization on a highly diverse dataset containing images from 19 synthesis models.
Takeaways & Limitations
Combining global and local information provides the paper’s supported approach for improving generalized AI-synthesized image detection.
Abstract
from arXiv · showhide
With the development of the Generative Adversarial Networks (GANs) and DeepFakes, AI-synthesized images are now of such high quality that humans can hardly distinguish them from real images. It is imperative for media forensics to develop detectors to expose them accurately. Existing detection methods have shown high performance in generated images detection, but they tend to generalize poorly in the real-world scenarios, where the synthetic images are usually generated with unseen models using unknown source data. In this work, we emphasize the importance of combining information from the whole image and informative patches in improving the generalization ability of AI-synthesized image detection. Specifically, we design a two-branch model to combine global spatial information from the whole image and local informative features from multiple patches selected by a novel patch selection module. Multi-head attention mechanism is further utilized to fuse the global and local features. We collect a highly diverse dataset synthesized by 19 models with various objects and resolutions to evaluate our model. Experimental results demonstrate the high accuracy and good generalization ability of our method in detecting generated images. Our code is available at https://github.com/littlejuyan/FusingGlobalandLocal.
‡NVIDIA
The paper identifies AI-synthesized image detection and image forensics as its central focus, with feature fusion and attention as key themes.
- The work focuses on AI-synthesized image detection, image forensics, feature fusion, and attention mechanisms.
1. INTRODUCTION
The paper addresses the sharp drop in detector performance when synthesized images come from unseen models, domains, or source data. It proposes combining whole-image structure with automatically selected local artifacts and attention-based fusion, evaluated on diverse manipulations and models.
- Current detectors perform well in closed settings but degrade considerably on images from different domains, unknown models, or unseen source data.
- Advanced generation makes real–fake differences increasingly subtle and local, motivating deeper study of detector generalization.
- The proposed model combines global structural features with fine-grained features from multiple automatically selected patches.
- An Attention-based Feature Fusion Module combines the two branches before binary classification.
- The study evaluates generalization using a large, diverse dataset spanning different resolutions, manipulation types, and synthesis models.
2. RELATED WORK
The related work covers image synthesis with GANs and auto-encoder-based partial manipulation, alongside datasets and methods for distinguishing real from synthetic images.
- GANs synthesize entire images, while auto-encoder models commonly support partial manipulations such as face swaps, puppetry, and attribute editing.
- StyleGAN variants generate realistic high-resolution images, and conditional GANs can alter object classes or styles from input images.
- Large-scale datasets such as FaceForensic++ and Celeb-DF support research on partially manipulated images and videos.
3. METHOD
The method uses a global branch for whole-image structure and a local branch that selects informative patches from feature-map activations. Multi-head attention fuses the resulting embeddings for classification.
- The global branch extracts feature maps and a global embedding from the input image using a backbone network.
- The Patch Selection Module predicts informative patch coordinates from global feature maps for processing by the local branch.
- Patch informativeness is estimated by aggregating feature-map channels into an activation map and averaging scores over sliding windows.
- Higher-scoring windows are selected after non-maximum suppression, mapped to the input image, cropped, and encoded as local embeddings.
- The Attention-based Feature Fusion Module applies multi-head attention to global and patch embeddings, producing a fused feature for final prediction.
4. EXPERIMENTS
Experiments use a diverse 128,424-image test set spanning 19 generation models and evaluate detection with family-wise mAP and mixed-dataset Global AP. Our PSM generally achieves the strongest overall generalization, outperforms alternative patch-selection methods, and remains robust to Gaussian blur and JPEG compression.
- Datasets: The test set contains 128,424 images from 19 generation models, six model families, varied objects, manipulation types, and resolutions.Training uses 362K real LSUN images and 362K ProGAN images at 256 × 256.
- Evaluation protocol: Average precision is reported as family-wise mAP, Total mAP across models, and Global AP on the mixed testing dataset.The mixed dataset measures performance when image sources and generation models are unknown.
- Detection results: Our PSM achieves the best Total mAP and Global AP overall, while performing slightly worse than No down on Low-level Vision and DeepFakes families.All methods reach 100% accuracy on seen ProGAN except Patch forensics, whose coverage is limited for diverse objects.
- Patch-selection ablation: Our PSM outperforms random cropping and corner-center resizing in all testing families.These ablations retain the global branch and attention-based fusion while replacing the proposed patch selection module.
- Post-processing robustness: The model shows good robustness to Gaussian blur and JPEG compression on the diverse unseen dataset.The authors associate this robustness with data augmentation during training.
5. CONCLUSION
The paper concludes that combining global image structure with informative local features improves generalized AI-synthesized image detection. Its two-branch model uses patch selection and multi-head attention, while future work targets differentiable patch selection.
- Conclusion: The proposed two-branch model fuses whole-image structure with informative local features for generalized AI-synthesized image detection.The global branch captures structure, while the local branch extracts subtle features from selected regions.
- Conclusion: Patch selection and multi-head attention combine discriminative local patches with global high-level features for binary classification.The model selects patches unsupervised and uses attention-based fusion to combine complementary information.
- Conclusion: Experiments on a 19-model diverse testing dataset demonstrate high accuracy and good generalization ability.
- Future work: Future work will investigate a differentiable patch selection module for more efficient patch learning.