Source-linked AI summary
From Detection to Localization: A Unified Forensics Framework for Fully Synthetic and Tampered Images
Annalisa Gallina, Marco Fiorucci, Marco Brigo, Federica Battisti, Lamberto Ballan
TL;DR
Highly realistic generative models and binary forensic formulations make it difficult to distinguish and localize diverse image manipulations. This paper extends RINE with a unified three-class detector and conditional segmentation branch using shared DINOv2 representations. On So-Fake validation, it reports 92.4% detection accuracy and 77.8% localization IoU, while acknowledging that universal generalization remains challenging.
Problem
Existing forensic systems are predominantly binary and often lack a unified formulation for real, fully synthetic, and locally tampered images.
Method
The framework uses shared frozen DINOv2 features for three-class image classification and conditional pixel-level tampering localization.
Results
92.4% detection accuracy and 77.8% localization IoU are reported on the So-Fake validation set, with 72.95 IoU in the cross-dataset setting.
Takeaways & Limitations
The framework provides a unified and computationally efficient approach that improves dense forensic localization across diverse image content.
Takeaways & Limitations
Universal generalization across the evolving landscape of generative models remains a primary challenge.
Abstract
from arXiv · showhide
The rapid advancement of generative models has significantly worsened the problem of manipulated image detection, as these methods are capable of producing highly realistic forgeries, reinforcing the importance of multimedia forensics. Conventional approaches typically frame image manipulation detection as a binary classification task (real vs. generated), which limits the capability to distinguish and localize different forms of manipulation. To address these constraints, this work extends an existing detector by introducing a unified multiclass framework (real vs. fully generated vs. tampered). In addition to classifying image authenticity, the framework incorporates a segmentation branch to enable pixel-level localization of tampered regions. The proposed approach outperforms selected recent benchmarks, offering an efficient solution with improved classification accuracy and higher IoU scores for the localization task. Find the code at https://github.com/anngal01/From-Detection-to-Localization-A-Unified-Forensics-Framework-for-Fully-Synthetic-and-Tampered-Images.
1 INTRODUCTION
Photorealistic generative models challenge visual-content reliability, while conventional forensic detectors remain largely binary and globally focused. The proposed framework extends detection to three-way classification and conditional pixel-level localization using shared representations.
- Diffusion-based models now produce photorealistic images at scale, challenging the reliability of visual content.
- Early forensic methods targeted GAN-specific architectural footprints and statistical inconsistencies, but diffusion synthesis has weakened those legacy cues.
- Conventional detectors primarily distinguish entirely real from fully synthetic content, limiting their handling of partially manipulated images.
- The framework classifies images as real, fully synthetic, or partially tampered, then activates a lightweight segmentation branch for pixel-level localization.
- Shared features from a frozen DINOv2 backbone support image-level classification and pixel-level tampering-mask prediction through two task-specific branches.
2 RELATED WORKS
Existing deepfake forensics commonly separate synthetic-image detection from local forgery localization, while generator-dependent artifacts limit transferability. The paper addresses this gap with a unified, efficient framework for real, synthetic, and tampered images.
- Spatial and Frequency Artifacts: Modern deepfake cues are often subtle spatial or frequency artifacts that can be attenuated by compression, rescaling, or enhancement.
- Spatial and Frequency Artifacts: Frequency-only forensic signatures remain architecture-dependent, limiting transferability from GANs to diffusion-based generators.
- Image Deepfake Detection and Localization: Most existing approaches use binary formulations, and localization models commonly focus only on whether manipulation is present.
- Image Deepfake Detection and Localization: SID-Set and So-Fake Set provide larger, more diverse benchmarks for unified evaluation across real, fully synthetic, and locally tampered images.
- Image Deepfake Detection and Localization: The proposed framework jointly classifies real, synthetic, and tampered images while localizing manipulated regions at lower computational cost than heavy multimodal models.
- Large Vision-Language Models: Large vision-language models offer nuanced visual reasoning for detection and localization but rely on prompting or supervised fine-tuning for adaptation.
3 PROPOSED METHOD
The proposed method unifies three-way image authenticity classification with conditional pixel-level tampering localization using shared frozen DINOv2 representations. It combines global classification objectives with a lightweight segmentation pipeline that reuses multiscale patch features.
- Architecture: A two-branch architecture jointly performs image-level detection and pixel-level localization using a shared DINOv2 backbone.The shared frozen representations support computational efficiency while capturing global semantic inconsistencies and fine-grained artifacts.
- Detection Branch: The detection branch classifies images as Real, Tampered, or Fake rather than using a binary authenticity formulation.Its lightweight head maps extracted features to three class probabilities.
- Feature Extraction: DINOv2 is selected because its spatially consistent attention and inter-patch correlations better support localization than CLIP-based representations.CLIP-based models may focus on proxy tokens and overlook subtle local inconsistencies in dense tampering tasks.
- Detection Branch: The classification objective combines categorical cross-entropy with supervised contrastive loss to optimize class prediction and latent feature structure.Cross-entropy measures predicted-versus-ground-truth class discrepancies, while contrastive learning pulls same-class features together and separates different-class features.
4 EXPERIMENTAL RESULTS
Experiments evaluate the framework on So-Fake and SID-Set, including detection, localization, qualitative behavior, backbone choice, and computational efficiency. The method achieves strong validation and cross-dataset results, while remaining substantially more efficient than LVLM-based approaches and showing identifiable localization failure modes.
- Experimental Setup: The system is trained and evaluated on So-Fake using the original training and test splits, with classification trained on all images and segmentation only on tampered images.Segmentation is restricted to tampered samples because localization is not meaningful for pristine or fully synthetic images.
- Experimental Setup: Detection uses image-level accuracy and F1, while segmentation uses IoU and F1; baselines include specialized detectors, localization models, and LVLMs.Most baseline results are taken from prior work, with separate evaluation exceptions for FakeShield and HIFI-Net.
- Quantitative Results: 92.4% accuracy on So-Fake validation closely matches So-Fake-R1 and exceeds the other existing baselines.Localization is evaluated on all tampered validation images, independently of whether the detection branch correctly identifies them as tampered.
- Quantitative Results: 77.8% IoU on So-Fake validation surpasses previous methods by a large margin for manipulated-region localization.The reported evaluation decouples predicted-mask quality from detection-branch correctness.
- Cross-Dataset Evaluation: 72.95 IoU is achieved when the So-Fake-trained model localizes tampering on SID-Set without target-dataset fine-tuning.Cross-dataset detection reports only Real and Fake classes for consistency with the prior protocol, despite the model’s three-class output.
- Qualitative Results: Qualitative masks closely match ground truth across FLUX.1-dev, Latent Diffusion, SD-XL, and Imagic image sources.Probability maps show high activation in tampered regions and low activation in authentic regions.
- Ablation Studies: DINOv2 improves detection over CLIP, supporting its use for stable and informative local feature extraction.The comparison uses different preprocessing strategies because the backbones require different input resolutions.
- Model Complexity and Computational Efficiency: 80× fewer parameters, 40× smaller model size, and approximately 16× faster inference distinguish the method from LVLM-based baselines.The method uses 96.9M parameters versus 7.71B and reduces average per-image processing time from 262.61ms to 16.40ms while supporting detection and localization.
5 CONCLUSIONS
The framework advances image forensics through unified multiclass detection and pixel-level localization, with evaluations reporting improved localization performance and computational efficiency. Universal generalization across evolving generative models remains an open challenge.
- The framework distinguishes real, synthetic, and tampered images before activating a lightweight branch for pixel-level localization.
- Evaluations on SID-Set and So-Fake-Set report that the framework outperforms the state of the art in localization.
- The architecture is presented as maintaining computational efficiency while identifying tampered regions with high precision.
- Universal generalization across an evolving landscape of generative models remains a primary challenge.Future work targets deeper ablations and invariant frequency-domain synthetic fingerprints.