Source-linked AI summary
GAP-SAM: A Global Artifact Prior for Generalizable AI-Generated Image Manipulation Localization
Haozhen Yan, Siyuan Shan, Zijian Yu, Youqi Wang, Yan Hong, Jun Lan, Jianfu Zhang
TL;DR
Image manipulation localization must generalize beyond training-data distributions, but semantic-boundary shortcuts and mismatched artifact alignment remain challenges. The paper constructs COCO-ControlNet and proposes GAP-SAM, using globally transferable reconstruction evidence to guide dense decoding; COCO-ControlNet improves mean OOD Pixel-F1 across seven retrained localizers.
Problem
Image manipulation localization often degrades in real-world scenarios, while training masks and semantic contours can encourage models to associate manipulation boundaries with dataset-specific geometry.
Method
The paper constructs COCO-ControlNet with source-image Canny edges and depth maps, then proposes GAP-SAM, which fuses an image with its frozen VAE reconstruction into a global artifact prior for SAM3 decoding.
Results
All seven localizers retrained on COCO-ControlNet achieve higher mean OOD Pixel-F1 than their official checkpoints.
Takeaways & Limitations
A global artifact prior can guide localization toward forensically supported boundaries without supplying a spatial shortcut, bridging robust image-level evidence and precise localization.
Takeaways & Limitations
Mask-VAE transfers poorly because global VAE reconstruction traces do not match the spatial support and artifact formation of local diffusion inpainting.
Abstract
from arXiv · showhide
AI-generated image manipulation localization identifies edited pixels, but its OOD performance lags behind image-level detection partly because pixel supervision entangles forensic evidence with dataset-specific mask geometry and semantic boundaries. Extending image-level distribution alignment to localization, we construct COCO-ControlNet with source-image Canny edges and depth maps to align semantics and geometry, improving OOD performance across multiple localizers. Yet tighter Mask-VAE Reconstruction Alignment (Mask-VAE) underperforms COCO-ControlNet, showing that VAE reconstruction artifacts transfer poorly to local diffusion-inpainting artifacts. We also identify \emph{boundary adhesion}, where fine-tuned segmentation models snap predictions to semantic object contours rather than true manipulation boundaries. These findings motivate GAP-SAM, which encodes an image and its frozen VAE reconstruction into a global artifact token and injects it into SAM3's feature pyramid via zero-gated FiLM before pixel decoding. Without prescribing a spatial region, this token modulates dense decoding to preserve localization while suppressing semantic-boundary shortcuts. Across six datasets, GAP-SAM averages 79.8 Pixel-F1, outperforming the strongest prior method by 12.6 points. It also performs best at every tested severity of JPEG compression, Gaussian blur, and resizing.
1 Introduction
AI-generated image manipulation localization struggles with OOD generalization because pixel supervision can encode dataset-specific geometry and semantic boundaries. The paper addresses this through distribution-aligned data construction and a global artifact prior for SAM3-based decoding.
- OOD generalization is a central challenge because localization methods degrade substantially in real-world scenarios.
- Localization suffers a larger cross-domain performance drop than image-level prediction, supporting greater sensitivity to non-causal pixel-level factors.
- COCO-ControlNet conditions generation on source-image Canny edges and depth maps to improve semantic and geometric alignment.
- All seven localizers retrained on COCO-ControlNet achieve higher mean OOD Pixel-F1 than their official checkpoints.
- Mask-VAE underperforms COCO-ControlNet because VAE reconstruction artifacts mismatch the spatial support and formation process of local diffusion-inpainting artifacts.
- Boundary adhesion occurs when segmentation models snap predictions to semantic object contours instead of true manipulation boundaries.
- GAP-SAM fuses an image with its frozen-VAE reconstruction into a global artifact prior injected into SAM3 before pixel decoding.
2 Related Work
Related work shows that manipulation localizers remain vulnerable to cross-model and distribution shifts. Existing evidence points to mask geometry, semantic content, and cross-space interfaces as potential shortcuts that limit OOD localization.
- Conventional localizers model manipulation traces, spatial dependencies, compression artifacts, noise, boundary cues, and spatial-channel correlations.
- Benchmarks reveal substantial cross-model degradation, while COCO-Inpaint shows sensitivity to mask shape and area.
- Foundation-model approaches adapt SAM with forensic cues or multimodal representations, but cross-space interfaces may lose forensic information and favor semantic objects.
- Image-level detection research links closer real-synthetic distribution alignment with reduced shortcut reliance and stronger OOD robustness.
3 Motivation and Analysis
Localization generalizes worse across domains than classification, motivating distribution-aligned training. COCO-ControlNet improves OOD localization across localizers, whereas tighter Mask-VAE alignment transfers poorly and segmentation models exhibit boundary adhesion.
- Why Localization Generalizes Worse: Localization suffers a larger cross-domain F1 drop than classification, reflecting greater sensitivity to domain shifts.Pixel-level supervision associates forensic evidence with precise locations and introduces additional non-causal factors, including dataset-specific mask properties.
- COCO-ControlNet for Alignment: COCO-ControlNet combines masks, optional prompts, and source-image Canny or depth conditioning to generate manipulated regions or fully generated images.The construction is designed to align generated content with source-image semantics and geometry.
- COCO-ControlNet for Alignment: All seven baselines retrained on COCO-ControlNet achieve higher mean OOD Pixel-F1 than their official checkpoints.The consistent improvement supports distribution-aligned training as an effective strategy for OOD localization.
- Is There a Better Alignment Strategy?: COCO-ControlNet training outperforms Mask-VAE training in every reported F1/IoU column for both CoDE and MaskCLIP.This result shows that maximizing pixel similarity does not necessarily yield optimal generalization.
- Is There a Better Alignment Strategy?: Mask-VAE artifacts transfer poorly to local diffusion-inpainting artifacts because their spatial support and artifact formation differ.VAE reconstruction creates global decoder traces, while local diffusion inpainting depends on mask boundaries, context, conditioning, and denoising.
- Boundary Adhesion in Segmentation Models: Boundary adhesion occurs when fine-tuned segmentation models expand or shrink predictions along semantic object contours instead of true manipulation boundaries.GAP-SAM addresses this shortcut by using a global forensic prior to guide evidence-supported boundaries without directly specifying a spatial mask.
4 Method
GAP-SAM forms a global artifact prior from an input image and its frozen VAE reconstruction, then injects that prior into SAM3’s feature pyramid before native mask decoding. Its design combines paired artifact encoding, zero-gated FiLM conditioning, artifact classification, and retained SAM3 localization supervision.
- Global Artifact Prior: GAP-SAM encodes the input image and frozen VAE reconstruction with adaptive and frozen SAM3 encoders, producing paired FPN feature pyramids.The observed-image pyramid is retained for localization, while the reconstructed branch remains frozen to separate artifact-prior extraction from localization-oriented feature learning.
- FPN Mask Decoding: Figure 5 routes the token through zero-gated FiLM to modulate the input-image feature pyramid before SAM3’s native localization pipeline predicts the manipulation mask.Zero initialization starts modulation as an identity transformation, while the shared gate learns how strongly the prior modifies multiscale features.
- Global Artifact Prior: The Paired Artifact Encoder pools paired final-layer features, fuses them through linear layers, and produces a 256-dimensional Global Artifact Prior Token.Global pooling removes spatial coordinates so the token represents reconstruction-related artifact evidence rather than a localization proposal.
- FPN Mask Decoding: Broadcast FiLM conditioning supplies image-specific forensic context without inserting a reconstruction-derived spatial region or boundary into dense decoding.Ground-truth masks continue to supervise where manipulation lies, but no longer solely determine which evidence the spatial representation emphasizes.
- Learning Objective: The artifact classifier constrains pooled original and reconstruction descriptors along a real-versus-reconstructed generation axis without adding mask, boundary, semantic, or type-classification losses.The localization objective retains SAM3’s official fine-tuning configuration, while artifact supervision acts only on globally pooled descriptors.
5 Experiments
Experiments evaluate OOD localization across six test sets, compare baselines and ablations, and test robustness to common image distortions. GAP-SAM achieves the strongest overall localization and robustness results, while ablations identify effective conditioning choices and module contributions.
- Evaluation setup: Evaluation uses an in-domain COCO-ControlNet split, five OOD benchmarks, Pixel-F1 and IoU at threshold 0.5, and unweighted OOD averages.Baselines include conventional, Transformer-based, vision-model fusion, process-aware, and large-multimodal-model localizers.
- Comparison with prior methods: Across all six test sets, GAP-SAM achieves the highest Pixel-F1 and IoU, improving six-set averages over the strongest prior by 12.6 F1 and 16.2 IoU points.Qualitative comparisons show reduced boundary adhesion, recovery of low-texture edits, and localization across multiple objects.
- Incremental module additions: OOD performance improves consistently as proposed modules are added to SAM3 LoRA, with VAE reconstruction evidence providing the key gain and Artifact Loss refining artifact representations.The ablation separates the effect of introducing the prior branch from the additional contribution of reconstruction evidence.
- Conditioning operator: FiLM achieves the highest COCO-CN, OOD, and six-set averages among tested conditioning operators and is selected for the reference design.The comparison concerns operators used to apply the global token.
- Prior injection site: Injecting the prior into the FPN before the Pixel Decoder provides the best OOD generalization and is used by default.The native query-mask output is held fixed in this injection-site comparison.
- Encoder and prior representation: The frozen encoder and global pooling provide the strongest or comparable OOD generalization while keeping the design simple and reference features stable.These choices are therefore used by default.
- Robustness: GAP-SAM remains robust across JPEG compression, Gaussian blur, and resizing, outperforming all baselines.Robustness is evaluated on COCO-ControlNet.
6 Conclusion
The conclusion links OOD localization failures to shortcut cues and mismatched artifact support, then presents GAP-SAM as a global-prior bridge between robust detection and precise localization. Its design uses structural conditioning, frozen-VAE evidence, zero-initialized FiLM, and artifact classification without supplying a spatial shortcut.
- Conclusion: Mask-VAE suggests that alignment alone is insufficient when artifact formation and spatial support differ from the localization target.The conclusion frames this mismatch as part of why OOD generalization deteriorates for pixel masks.
- Conclusion: COCO-ControlNet uses structural conditioning to reduce semantic shortcuts while preserving diffusion-based inpainting artifacts.This connects aligned training conditions with the artifact characteristics targeted by localization.
- Conclusion: GAP-SAM repurposes globally transferable frozen-VAE evidence as a compact prior for SAM3’s feature pyramid.Zero-initialized FiLM and artifact classification steer decoding away from semantic boundary adhesion without supplying a spatial shortcut.
- Conclusion: The resulting framework bridges robust whole-image detection and precise manipulation localization.The conclusion presents this as the framework’s supported scope.