PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Fine-Grained Image Forgery Detection and Localization

Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Iacopo Masi, Xiaoming Liu

arXiv:2303.17111v1cs.CV

TL;DR

Unified IFDL is difficult because CNN-synthesized and image-editing forgeries have substantially different attributes. The paper introduces hierarchical fine-grained learning with HiFi-Net and a dedicated dataset, reporting state-of-the-art IFDL performance across 7 benchmarks and competitive attribute classification. Its authors note limited generalization to diffusion-based inpainting and the potential value of larger datasets.

  • Problem

    CNN-synthesized and image-editing forgeries differ substantially in attributes, making unified image forgery detection and localization challenging.

  • Method

    HiFi-Net predicts multi-level forgery attributes through hierarchical dependencies and combines feature extraction, localization, and classification modules.

  • Results

    Across 7 benchmarks, the method outperforms the state of the art on IFDL and achieves competitive forgery-attribute classification performance.

  • Takeaways & Limitations

    The hierarchical formulation supports comprehensive IFDL representations across CNN-synthesized and image-editing domains.

  • Takeaways & Limitations

    The model performs well on conventional image editing but can generalize poorly to diffusion-based inpainting; larger forgery datasets may improve learning.

Abstract

from arXiv · show

Differences in forgery attributes of images generated in CNN-synthesized and image-editing domains are large, and such differences make a unified image forgery detection and localization (IFDL) challenging. To this end, we present a hierarchical fine-grained formulation for IFDL representation learning. Specifically, we first represent forgery attributes of a manipulated image with multiple labels at different levels. Then we perform fine-grained classification at these levels using the hierarchical dependency between them. As a result, the algorithm is encouraged to learn both comprehensive features and inherent hierarchical nature of different forgery attributes, thereby improving the IFDL representation. Our proposed IFDL framework contains three components: multi-branch feature extractor, localization and classification modules. Each branch of the feature extractor learns to classify forgery attributes at one level, while localization and classification modules segment the pixel-level forgery region and detect image-level forgery, respectively. Lastly, we construct a hierarchical fine-grained dataset to facilitate our study. We demonstrate the effectiveness of our method on $7$ different benchmarks, for both tasks of IFDL and forgery attribute classification. Our source code and dataset can be found: \href{https://github.com/CHELSEA234/HiFi_IFDL}{github.com/CHELSEA234/HiFi-IFDL}.

1. Introduction

The paper targets unified image forgery detection and localization across CNN-synthesized and image-editing domains, where forgery attributes differ substantially. It proposes hierarchical fine-grained representation learning, HiFi-Net, and the HiFi-IFDL dataset.

  • Motivation: CNN-synthesized and image-editing forgeries differ in attributes and region distributions, complicating unified IFDL algorithms.Forgery attributes include whether an image is fully synthesized or partially manipulated, and which method generated it.
  • Approach: The method represents each manipulated image with multiple forgery-attribute labels organized across hierarchical levels.The hierarchy progresses from fully synthesized versus partially manipulated to specific methods such as DDPM and DDIM.
  • Approach: Hierarchical path prediction conditions each node’s classification probability on preceding attributes rather than treating attributes as mutually exclusive.For example, DDPM prediction follows the path Forgery→Fully Synthesis→Diffusion→Unconditional→DDPM.
  • Approach: HiFi-Net combines a multi-branch feature extractor, localization module, and classification module for attribute learning, mask generation, and image-level detection.The localization module uses a deep-metric objective, while classification processes masked images with partial convolution.
  • Dataset: The HiFi-IFDL dataset contains 13 forgery methods, hierarchical forgery categories, and high-resolution ground-truth masks.The dataset supports both forgery-attribute classification and pixel-level localization.
  • Results: Across 7 benchmarks, HiFi-Net outperforms the state of the art on IFDL and achieves competitive forgery-attribute classification performance.The evaluation covers both image forgery detection and localization and forgery-attribute classification.

2. Related Work

Prior work includes image-level forgery detection, including GAN and facial forgery detection, but the paper compares its approach with existing methods for detection and localization.

  • Image Forgery Detection: Prior studies report difficulty generalizing image-level classifiers across different GANs, while another baseline generalizes when trained on ProGAN images.These findings motivate comparison across forgery-generation settings.
  • Comparison: Table 1 compares previous works using detection and localization metrics across synthesis and forgery settings.The table abbreviations include Det. for detection, Loc. for localization, and Syn. for synthesis.
  • Image Forgery Detection: Related work also includes facial forgery detection and biometric applications, with these methods specializing in image-level detection.The cited research does not encompass the full IFDL setting described here.

3. HiFi-Net

HiFi-Net combines hierarchical fine-grained classification with pixel-level localization and image-level forgery detection. Its branches model different attribute levels, while mask-guided features and self-attention support localization and classification.

  • Multi-Branch Feature Extractor: Four branches classify hierarchical forgery attributes, using feature maps at corresponding resolutions; the finest branch identifies individual forgery methods.High-resolution features preserve spatial information needed to model contents across image locations.
  • Architecture: HiFi-Net comprises a multi-branch feature extractor, localization module, and detection/classification modules for hierarchical IFDL.The extractor performs fine-grained classification at different attribute levels, while the other modules support localization and detection.
  • Localization Module: The localization module maps highest-resolution features to a forgery mask using self-attention and a metric-learning objective.Self-attention models long-range pixel interactions, while the objective separates real and forged pixel features.
  • Localization Module: The localization loss compacts real-pixel features and separates forged-pixel features by a margin τ, using one margin for pixel-level learning.The authors contrast this with prior objectives using different separation designs or two image-level margins.
  • Classification Module: The classification module masks the input image and applies partial convolution so forged regions guide fine-grained attribute classification.The masked region is intended to provide a prior for distinguishing methods such as STGAN and FaceShifter.
  • Training and Inference: Branch probabilities are conditioned on preceding coarse-level probabilities, enforcing the hierarchical dependency among forgery attributes.At inference, level-4 probabilities determine fine-grained attributes and forged-versus-real classification.

4. Hierarchical Fine-grained IFDL dataset

HiFi-IFDL is a hierarchical benchmark spanning current CNN-based and image-editing forgery methods. It provides large-scale images, hierarchical labels, and high-resolution masks for localization.

  • Dataset Composition: HiFi-IFDL contains 13 forgery methods spanning CNN-based manipulations and image editing.The dataset focuses on updated and representative forgery types rather than attempting to include every generative category.
  • Dataset Scale: Each forgery method contributes 100,000 images, with real images selected from six datasets.Training, validation, and test sets contain 1,710K, 15K, and 174K images, respectively.
  • Hierarchical Taxonomy: The hierarchy progresses from fully-synthesized versus partial-manipulated images to methodology, conditionality, and specific forgery method.The final level includes real images and 13 specific forgery methods.
  • Localization Annotations: Manipulated images are paired with high-resolution ground-truth forgery masks for pixel-level localization.The masks are part of the dataset’s support for IFDL evaluation.

5. Experiments

Experiments evaluate HiFi-Net across forgery detection, localization, and attribute classification, showing strong average performance across heterogeneous datasets and sensitivity to its hierarchical components.

  • Evaluation setup: HiFi-Net is evaluated for IFDL on 7 datasets and for forgery attribute classification on HiFi-IFDL.Image-level detection and pixel-level localization use AUC and F1 on HiFi-IFDL.
  • HiFi-IFDL results: 2.6% AUC and 2.0% F1 improvements over PSCC are reported for forgery localization.The authors associate this result with more comprehensive localization features learned through hierarchical fine-grained formulation.
  • Image editing datasets: HiFi-Net achieves the best average performance in image-editing localization after fine-tuning, while falling behind only on NIST16.Its image-level detection results on CASIA are comparable to ObjectFormer, and qualitative masks capture semantically meaningful object shapes.
  • DFFD: HiFi-Net achieves competitive detection and the best localization performance on partially manipulated images in DFFD.The evaluation uses IoU and PBCA for localization, and AUC and PBCA for detection.
  • Ablation study: Removing localization or classification loss causes 24.1% F1 detection and 29.3% AUC localization drops, respectively.Removing classification loss also reduces localization by 1.9% AUC and F1, supporting classification features as a localization prior.
  • Forgery attribute classification: Fourth-level forgery attribute classification is the most challenging, with failures involving real-image artifacts, small inpainting regions, and similar StyleGAN outputs.The reported failure scenarios include watermarks, extreme lighting, distortion, small forged regions, and highly similar generated images.

6. Conclusion

The paper proposes HiFi-Net and the HiFi-IFDL dataset for unified forgery detection and localization across CNN-synthesized and image-editing domains. It also identifies poor generalization from conventional image editing to diffusion-based inpainting as a limitation.

  • Conclusion: HiFi-Net formulates IFDL as hierarchical fine-grained classification across CNN-synthesized and image-editing forgeries.The model predicts individual forgery methods through complete hierarchical paths.
  • Limitation: Models performing well on conventional image editing may generalize poorly to diffusion-based inpainting.The authors also suggest that larger forgery datasets could improve IFDL learning.

7. Supplementary

The supplementary material details the HiFi-IFDL dataset, evaluates generalization and robustness, and reports additional localization and hierarchical classification results. It also describes implementation choices for training and localization.

  • Dataset Collection Details: The HiFi-IFDL dataset contains 13 forgery methods spanning CNN-synthesized and representative image-editing methods, with hierarchical categories and high-resolution ground-truth masks.Its real images combine LSUN, CelebaHQ, FFHQ, AFHQ, MSCOCO, and real face images from FaceForensics.
  • Generalization Performance: HiFi-Net generalizes imperfectly to unseen data: DSGAN and PNDM can fool the detector, while real images from a different domain may also cause failure.The reported unseen-domain inference uses 1,000 images per generative method and unseen real-image domain with a fixed 0.5 threshold.
  • Generalization Performance: Diffusion-based partial manipulation remains a generalization challenge because its edited regions vary rather than following rigid copy-move or removal patterns.The authors identify differences from conventional editing and variation in diffusion inpainting regions as contributing factors.
  • Post-processing Robustness: The proposed method is more robust than prior work across tested post-processing operations, except resizing to 0.78 times the image and JPEG compression at 50% quality.These comparisons are reported on NIST16.
  • DFFD Localization: On DFFD, HiFi-Net achieves more accurate localization than Attention Xception for partially manipulated and fully synthesized images, while remaining comparable on real images.Table 8 reports IoU and PBCA as higher-is-better metrics and IINC and Cosine Similarity as lower-is-better metrics.
  • Forgery Attribute Classification: Hierarchical classification can recover fine-level prediction accuracy even when a coarser-level classification is weaker.Examples 5 and 10 show lower third-level prediction probability than fourth-level probability.
  • Implementation Details: The architecture uses four feature-map resolutions—256, 128, 64, and 32 pixels—and 2-, 4-, 6-, and 14-way classification at levels one through four.The third level contains six categories covering diffusion, GAN, CNN-based partial manipulation, and image editing.
  • Implementation Details: Localization uses pixel features from the initialized network, averages real-image features into c, computes pixelwise ℓ2 distances, and sets τ to 2.5 · Dmax.The architecture is trained end-to-end with separate classification and localization losses, applying all branches to manipulated inputs and only the last branch to real inputs.

Image GT PSCC

Additional qualitative results are shown for the CASIA, NIST16, and Coverage datasets.

  • Figure 15 presents additional qualitative results on CASIA, NIST16, and Coverage.
Loading 2303.17111v1…