Hierarchical Fine-Grained Image Forgery Detection and Localization

Motivation

• CNN-synthesized and image-editing forgeries differ in attributes and region distributions, complicating unified IFDL algorithms.Forgery attributes include whether an image is fully synthesized or partially manipulated, and which method generated it.Source: §1 · p. 1, Figure 1a · p. 1, Figure 1 · p. 1

Approach

• The method represents each manipulated image with multiple forgery-attribute labels organized across hierarchical levels.The hierarchy progresses from fully synthesized versus partially manipulated to specific methods such as DDPM and DDIM.Source: §1 · p. 2, Figure 2 · p. 2
• Hierarchical path prediction conditions each node’s classification probability on preceding attributes rather than treating attributes as mutually exclusive.For example, DDPM prediction follows the path Forgery→Fully Synthesis→Diffusion→Unconditional→DDPM.Source: §1 · p. 2
• HiFi-Net combines a multi-branch feature extractor, localization module, and classification module for attribute learning, mask generation, and image-level detection.The localization module uses a deep-metric objective, while classification processes masked images with partial convolution.Source: §1 · p. 2

Dataset

• The HiFi-IFDL dataset contains 13 forgery methods, hierarchical forgery categories, and high-resolution ground-truth masks.The dataset supports both forgery-attribute classification and pixel-level localization.Source: §1 · p. 2

Results

• Across 7 benchmarks, HiFi-Net outperforms the state of the art on IFDL and achieves competitive forgery-attribute classification performance.The evaluation covers both image forgery detection and localization and forgery-attribute classification.Source: §1 · p. 2

Image Forgery Detection

• Prior studies report difficulty generalizing image-level classifiers across different GANs, while another baseline generalizes when trained on ProGAN images.These findings motivate comparison across forgery-generation settings.Source: §2 · pp. 2–3

Comparison

• Table 1 compares previous works using detection and localization metrics across synthesis and forgery settings.The table abbreviations include Det. for detection, Loc. for localization, and Syn. for synthesis.Source: Table 1 · p. 3

Image Forgery Detection

• Related work also includes facial forgery detection and biometric applications, with these methods specializing in image-level detection.The cited research does not encompass the full IFDL setting described here.Source: §2 · p. 3

Multi-Branch Feature Extractor

• Four branches classify hierarchical forgery attributes, using feature maps at corresponding resolutions; the finest branch identifies individual forgery methods.High-resolution features preserve spatial information needed to model contents across image locations.Source: §3 · p. 3, §3.1 · p. 3

Architecture

• HiFi-Net comprises a multi-branch feature extractor, localization module, and detection/classification modules for hierarchical IFDL.The extractor performs fine-grained classification at different attribute levels, while the other modules support localization and detection.Source: §3 · p. 3

Localization Module

• The localization module maps highest-resolution features to a forgery mask using self-attention and a metric-learning objective.Self-attention models long-range pixel interactions, while the objective separates real and forged pixel features.Source: §3.2 · p. 4, Figure 4 · p. 5
• The localization loss compacts real-pixel features and separates forged-pixel features by a margin τ, using one margin for pixel-level learning.The authors contrast this with prior objectives using different separation designs or two image-level margins.Source: §3.2 · p. 4

Classification Module

• The classification module masks the input image and applies partial convolution so forged regions guide fine-grained attribute classification.The masked region is intended to provide a prior for distinguishing methods such as STGAN and FaceShifter.Source: §3.3 · pp. 4–5, Figure 3 · p. 4

Training and Inference

• Branch probabilities are conditioned on preceding coarse-level probabilities, enforcing the hierarchical dependency among forgery attributes.At inference, level-4 probabilities determine fine-grained attributes and forged-versus-real classification.Source: §3.3 · p. 5, Figure 5 · p. 5, §3.4 · p. 5

Dataset Composition

• HiFi-IFDL contains 13 forgery methods spanning CNN-based manipulations and image editing.The dataset focuses on updated and representative forgery types rather than attempting to include every generative category.Source: §4 · p. 5

Dataset Scale

• Each forgery method contributes 100,000 images, with real images selected from six datasets.Training, validation, and test sets contain 1,710K, 15K, and 174K images, respectively.Source: §4 · p. 5

Hierarchical Taxonomy

• The hierarchy progresses from fully-synthesized versus partial-manipulated images to methodology, conditionality, and specific forgery method.The final level includes real images and 13 specific forgery methods.Source: Figure 6 · p. 6, §4 · p. 6

Localization Annotations

• Manipulated images are paired with high-resolution ground-truth forgery masks for pixel-level localization.The masks are part of the dataset’s support for IFDL evaluation.Source: Figure 6 · p. 6

Evaluation setup

• HiFi-Net is evaluated for IFDL on 7 datasets and for forgery attribute classification on HiFi-IFDL.Image-level detection and pixel-level localization use AUC and F1 on HiFi-IFDL.Source: §5 · p. 6, §5.1.1 · p. 6

HiFi-IFDL results

• 2.6% AUC and 2.0% F1 improvements over PSCC are reported for forgery localization.The authors associate this result with more comprehensive localization features learned through hierarchical fine-grained formulation.Source: §5.1.1 · p. 7

Image editing datasets

• HiFi-Net achieves the best average performance in image-editing localization after fine-tuning, while falling behind only on NIST16.Its image-level detection results on CASIA are comparable to ObjectFormer, and qualitative masks capture semantically meaningful object shapes.Source: §5.1.2 · p. 7

DFFD

• HiFi-Net achieves competitive detection and the best localization performance on partially manipulated images in DFFD.The evaluation uses IoU and PBCA for localization, and AUC and PBCA for detection.Source: §5.1.3 · pp. 7–8

Ablation study

• Removing localization or classification loss causes 24.1% F1 detection and 29.3% AUC localization drops, respectively.Removing classification loss also reduces localization by 1.9% AUC and F1, supporting classification features as a localization prior.Source: §5.2 · p. 8

Forgery attribute classification

• Fourth-level forgery attribute classification is the most challenging, with failures involving real-image artifacts, small inpainting regions, and similar StyleGAN outputs.The reported failure scenarios include watermarks, extreme lighting, distortion, small forged regions, and highly similar generated images.Source: §5.3 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

Conclusion

• HiFi-Net formulates IFDL as hierarchical fine-grained classification across CNN-synthesized and image-editing forgeries.The model predicts individual forgery methods through complete hierarchical paths.Source: §6 · p. 8

Limitation

• Models performing well on conventional image editing may generalize poorly to diffusion-based inpainting.The authors also suggest that larger forgery datasets could improve IFDL learning.Source: §6 · p. 8

Dataset Collection Details

• The HiFi-IFDL dataset contains 13 forgery methods spanning CNN-synthesized and representative image-editing methods, with hierarchical categories and high-resolution ground-truth masks.Its real images combine LSUN, CelebaHQ, FFHQ, AFHQ, MSCOCO, and real face images from FaceForensics.Source: §7.1 · p. 9

Generalization Performance

• HiFi-Net generalizes imperfectly to unseen data: DSGAN and PNDM can fool the detector, while real images from a different domain may also cause failure.The reported unseen-domain inference uses 1,000 images per generative method and unseen real-image domain with a fixed 0.5 threshold.Source: §7.2 · p. 9, Figure 10 · p. 9
• Diffusion-based partial manipulation remains a generalization challenge because its edited regions vary rather than following rigid copy-move or removal patterns.The authors identify differences from conventional editing and variation in diffusion inpainting regions as contributing factors.Source: §7.2 · pp. 9–10

Post-processing Robustness

• The proposed method is more robust than prior work across tested post-processing operations, except resizing to 0.78 times the image and JPEG compression at 50% quality.These comparisons are reported on NIST16.Source: §7.3 · p. 10, Table 7 · p. 10

DFFD Localization

• On DFFD, HiFi-Net achieves more accurate localization than Attention Xception for partially manipulated and fully synthesized images, while remaining comparable on real images.Table 8 reports IoU and PBCA as higher-is-better metrics and IINC and Cosine Similarity as lower-is-better metrics.Source: §7.4 · p. 10, Table 8 · p. 10

Forgery Attribute Classification

• Hierarchical classification can recover fine-level prediction accuracy even when a coarser-level classification is weaker.Examples 5 and 10 show lower third-level prediction probability than fourth-level probability.Source: §7.5 · p. 10, Figure 12 · p. 11

Implementation Details

• The architecture uses four feature-map resolutions—256, 128, 64, and 32 pixels—and 2-, 4-, 6-, and 14-way classification at levels one through four.The third level contains six categories covering diffusion, GAN, CNN-based partial manipulation, and image editing.Source: §7.6 · p. 10
• Localization uses pixel features from the initialized network, averages real-image features into c, computes pixelwise ℓ2 distances, and sets τ to 2.5 · Dmax.The architecture is trained end-to-end with separate classification and localization losses, applying all branches to manipulated inputs and only the last branch to real inputs.Source: §7.6 · p. 10

• Figure 15 presents additional qualitative results on CASIA, NIST16, and Coverage.Source: Figure 15 · p. 14