Source-linked AI summary
ObjectFormer for Image Manipulation Detection and Localization
Junke Wang, Zuxuan Wu, Jingjing Chen, Xintong Han, Abhinav Shrivastava, Ser-Nam Lim, Yu-Gang Jiang
TL;DR
Widely accessible image-editing tools raise the need to detect manipulated images and localize altered regions. ObjectFormer combines RGB and frequency features with learnable object prototypes that model object- and patch-level consistencies. Extensive experiments show state-of-the-art detection and localization performance, while a reported limitation is lower localization AUC than PSCCNet on Columbia under pretraining.
Problem
Accessible image-editing techniques create a need for methods that determine whether images were modified and identify the modified regions.
Method
ObjectFormer combines RGB and frequency features into multimodal patch embeddings, then uses learnable object prototypes to model object-level consistency and refine patch embeddings.
Results
ObjectFormer achieves state-of-the-art image manipulation detection and localization performance across multiple benchmarks.
Takeaways & Limitations
Modeling frequency information and object- and patch-level visual consistencies supports image tampering detection and localization.
Takeaways & Limitations
On Columbia, pretrained ObjectFormer has a 2.7% lower localization AUC than PSCCNet, possibly because PSCCNet's pretraining data more closely match Columbia's distribution.
Abstract
from arXiv · showhide
Recent advances in image editing techniques have posed serious challenges to the trustworthiness of multimedia data, which drives the research of image tampering detection. In this paper, we propose ObjectFormer to detect and localize image manipulations. To capture subtle manipulation traces that are no longer visible in the RGB domain, we extract high-frequency features of the images and combine them with RGB features as multimodal patch embeddings. Additionally, we use a set of learnable object prototypes as mid-level representations to model the object-level consistencies among different regions, which are further used to refine patch embeddings to capture the patch-level consistencies. We conduct extensive experiments on various datasets and the results verify the effectiveness of the proposed method, outperforming state-of-the-art tampering detection and localization methods.
1. Introduction
Accessible generative editing tools have increased concerns about malicious image manipulation, motivating methods that detect tampering and localize modified regions. ObjectFormer addresses this challenge by combining RGB and frequency features with object-level prototypes, achieving state-of-the-art detection and localization performance.
- Motivation: Image editing tools make photorealistic manipulation accessible, increasing the need to detect modified images and identify altered regions.
- Motivation: Splicing, copy-move, and removal methods often manipulate semantically meaningful objects to create convincing images.
- Motivation: Existing CNN-based methods typically map images directly to authentic/manipulated labels without explicitly modeling object-level representations.
- ObjectFormer: ObjectFormer combines RGB and frequency features into multimodal patch embeddings to identify tampering artifacts.
- ObjectFormer: Learnable object prototypes model object-level consistencies and refine patch embeddings to capture patch-level consistencies.
- Evaluation: ObjectFormer achieves state-of-the-art performance for image manipulation detection and localization across multiple benchmarks.
2. Related Work
Related work covers manipulation-specific and general image tampering detection, including RGB-noise streams and multi-scale pixel-relation modeling. It also situates ObjectFormer within vision-transformer research focused on long-range dependencies and patch-based representations.
- Image Manipulation Detection / Localization: Early manipulation-detection studies target specific operations such as splicing, copy-move, or removal.
- Image Manipulation Detection / Localization: Because real-world manipulation types are unknown, later work addresses general manipulation detection.
- Image Manipulation Detection / Localization: RGB-N combines RGB features for visual artifacts with noise features for inconsistencies between tampered and untouched regions.
- Visual Transformer: Vision transformers are used in computer vision because self-attention can model long-range dependencies.
- Visual Transformer: ViT represents images as flattened patch sequences processed by transformer encoders for image classification.
3. Method
ObjectFormer combines RGB and high-frequency image features with learnable object prototypes to model visual consistencies at object and patch levels. Its refined representations support both binary manipulation detection and pixel-level localization.
- High-frequency Feature Extraction: RGB and frequency feature maps produce equal-sized patches whose flattened sequences are concatenated into multimodal patch embeddings with positional information.The resulting multimodal patch vector has dimensionality R2L×C.
- High-frequency Feature Extraction: ObjectFormer transforms RGB images into high-frequency representations with DCT and a high-pass filter, then extracts frequency features alongside RGB features.The filtered frequency representation is transformed back to the RGB domain before convolutional feature extraction.
- Object Encoder: Learnable object prototypes attend to patch embeddings, aggregate contextual information, and interact through a learned projection to model object-level visual inconsistencies.The prototype set contains N representations, with N empirically set to 16.
- Patch Decoder: Updated object representations cross-attend to patch embeddings, refining patch features so they incorporate object-level consistency information.The patch refinement uses query, key, and value projections followed by an MLP residual transformation.
- Patch Decoder: A boundary-sensitive contextual incoherence module compares local feature similarities, while stacked encoder-decoder iterations alternately update object and patch representations.The framework uses 8 sequential iterations by default and produces representations containing object- and patch-level consistency information.
- Detection and Localization: Global average pooling and a fully connected layer produce binary detection, while progressive upsampling produces a localization mask trained jointly with classification.The objective combines binary cross-entropy classification and segmentation losses, with λseg set to 1 by default.
4. Experiments
Experiments evaluate ObjectFormer on manipulation localization and detection across multiple benchmarks, including fine-tuning, robustness, ablation, and qualitative analyses. The results report strong performance, with component and prototype studies supporting the multimodal and consistency-modeling design.
- Image Manipulation Localization: ObjectFormer achieves 82.1% localization AUC on IMD20, outperforming PSCCNet by 1.9%.On Columbia, it surpasses SPAN and ManTraNet but trails PSCCNet by 2.7%.
- Image Manipulation Localization: Fine-tuning produces significant performance gains on specific datasets.The authors attribute these gains to multimodal features and object-level and patch-level consistency modeling.
- Image Manipulation Detection: 99.70% AUC and 97.34% F1 are reported for tampering detection on CASIA-D.These are described as state-of-the-art detection results.
- Robustness Evaluation: ObjectFormer is 1.1% higher than PSCCNet at JPEG quality 100 and 1.0% higher at quality 50 on NIST16 localization.The robustness evaluation also considers scaling, Gaussian blur, and Gaussian noise.
- Ablation Analysis: Removing HFE decreases AUC by 14.6% on CASIA and 11.0% on NIST16, while removing BCIM decreases it by 6.2% and 2.4%, respectively.Replacing object representations with vanilla self-attention reduces NIST16 AUC by 5% and F1 by 12.5%.
- Ablation Analysis: Localization performance generally increases with the number of object prototypes, with the best Columbia and CASIA results at N = 16.The visualizations also indicate that iterative updates associate object representations with meaningful image regions.
5. Conclusion
ObjectFormer is an end-to-end multimodal framework for detecting and localizing image tampering. It combines frequency-domain and RGB features with learnable object prototypes to model object-level and patch-level visual consistencies.
- ObjectFormer combines frequency-domain forgery features with RGB features to generate multimodal patch embeddings for tampering detection and localization.Frequency features provide complementary information for subtle artifacts that are no longer visible in the RGB domain.
- Learnable object prototypes serve as mid-level representations for modeling object-level consistencies within images.Stacked object encoders alternately update object prototypes and patch embeddings.
- Refined patch embeddings capture patch-level visual consistencies through the interaction of object prototypes and patch representations.
- Extensive experiments on different datasets demonstrate the effectiveness of the proposed method.