Source-linked AI summary
Learning Rich Features for Image Manipulation Detection
Peng Zhou, Xintong Han, Vlad I. Morariu, Larry S. Davis
TL;DR
Image manipulation detection requires locating tampered regions through artifacts rather than semantic content alone, but existing datasets are small for deep-network training. The paper proposes an end-to-end two-stream Faster R-CNN that fuses RGB and SRM-based noise features with bilinear pooling. Across standard datasets, the fused streams improve performance and support detection of different tampering techniques.
Problem
Image manipulation detection must identify tampered regions using artifact-sensitive features, while standard datasets provide insufficient data for deep neural network training.
Method
An end-to-end two-stream Faster R-CNN combines RGB tampering-artifact features with SRM-derived local noise features through bilinear pooling.
Results
The fused two-stream network yields improved performance on four image manipulation datasets and detects different tampering techniques.
Takeaways & Limitations
Combining RGB artifacts with local noise inconsistencies provides complementary evidence for image manipulation detection.
Takeaways & Limitations
The approach relies on synthetic pre-training because current standard datasets lack enough data for deep neural network training.
Abstract
from arXiv · showhide
Image manipulation detection is different from traditional semantic object detection because it pays more attention to tampering artifacts than to image content, which suggests that richer features need to be learned. We propose a two-stream Faster R-CNN network and train it endto- end to detect the tampered regions given a manipulated image. One of the two streams is an RGB stream whose purpose is to extract features from the RGB image input to find tampering artifacts like strong contrast difference, unnatural tampered boundaries, and so on. The other is a noise stream that leverages the noise features extracted from a steganalysis rich model filter layer to discover the noise inconsistency between authentic and tampered regions. We then fuse features from the two streams through a bilinear pooling layer to further incorporate spatial co-occurrence of these two modalities. Experiments on four standard image manipulation datasets demonstrate that our two-stream framework outperforms each individual stream, and also achieves state-of-the-art performance compared to alternative methods with robustness to resizing and compression.
1. Introduction
Image manipulation detection must localize tampered regions by learning low-level visual and noise inconsistencies rather than relying only on semantic image content. The proposed end-to-end two-stream Faster R-CNN combines RGB artifacts with local noise features through bilinear pooling, improving detection across manipulation datasets.
- Motivation: Splicing, copy-move, and removal are common manipulations that can remain difficult for humans to localize after post-processing.Removal may be followed by inpainting, and Gaussian smoothing can be applied after tampering.
- Motivation: Image manipulation detection targets tampered regions, unlike semantic object detection, which detects objects across categories.The framework explores both RGB image content and image noise features for this task.
- Approach: The proposed framework models visual artifacts and local noise inconsistencies to detect regions across multiple tampering techniques.The two modalities are bilinearly pooled to identify tampered regions.
- Approach: RGB features capture boundary and contrast inconsistencies, while the noise stream uses SRM-derived features to analyze local noise patterns.The two-stream network is trained end-to-end within Faster R-CNN.
- Approach: Features from both streams are bilinearly pooled for each Region of Interest before tampering classification.The RGB and noise streams provide complementary evidence for each proposed region.
- Evaluation: A synthetic COCO-based dataset is used for pre-training because existing manipulation datasets contain only several hundred images.The model is then fine-tuned and tested on different datasets.
2. Related Work
Related work detects manipulation through low-level artifacts, learned filtering, segmentation, and boundary-focused modeling. These approaches generally target specific clues or patches, motivating broader feature learning for manipulation detection.
- Low-level artifact analysis: Image forensics methods analyze low-level artifacts such as double JPEG compression, CFA patterns, and local noise.These methods use region-level statistics and camera-imaging artifacts as manipulation clues.
- Noise-based methods: SRM-based methods extract local noise features from adjacent pixels to capture inconsistencies between tampered and authentic regions.Prior work also combines SRM features with quantization and truncation operations.
- Deep learning methods: Deep learning approaches have used low-pass or adaptive filtering, fully convolutional prediction, and boundary-guided segmentation for manipulation detection.These methods address filtering artifacts, tampering masks, or tampered edges.
3. Proposed Method
The proposed detector uses RGB and SRM-derived noise streams within Faster R-CNN, combining their RoI features to classify and localize manipulated regions. RGB features support proposal and regression, while noise features add evidence for artifacts that RGB appearance may conceal.
- Architecture: The network uses RGB and SRM-derived noise images in parallel streams, with multi-task manipulation classification and bounding box regression.The model is trained end-to-end, and the two streams share the RoI pooling layer.
- RGB Stream: RGB features drive the Region Proposal Network, which searches for regions likely to be manipulated rather than regions likely to contain semantic objects.The RPN also supports bounding box regression, while RGB features are used for manipulation classification in the RGB stream.
- Noise Stream: SRM filters extract local noise features that emphasize noise distributions and can reveal tampering artifacts hidden in RGB channels.The noise stream uses the same backbone architecture as the RGB stream and shares its RoI pooling layer.
- Bilinear Pooling: Bilinear pooling combines the RGB and noise RoI features while preserving spatial information before the fully connected manipulation classifier.The pooled output is processed with signed square root and L2 normalization; compact bilinear pooling reduces memory and training cost without decreasing performance.
- Training Objective: The total objective sums RPN loss, bilinear-feature tampering classification loss, and RGB-feature bounding box regression loss.Classification uses cross entropy, regression uses smooth L1 loss, and the RPN loss balances its terms with λ = 10.
- Training and Inference: The implementation resizes inputs to a 600-pixel shorter side and uses four anchor scales, three aspect ratios, augmentation, and NMS during inference.Training runs for 110K steps, with the learning rate reduced after 40K steps; the NMS threshold is 0.2.
4. Experiments
Experiments evaluate the two-stream network on a synthetic COCO dataset and four standard manipulation datasets, including comparisons with individual streams and alternative methods. The results support complementary RGB and noise features, effective bilinear fusion, and robustness to common attacks.
- Pre-training: The synthetic dataset contains 42K tampered and authentic image pairs generated by copying and pasting COCO objects across images.Training and testing backgrounds and tampered objects are separated to prevent overlap.
- Ablation and architecture: The two-stream network outperforms each individual stream on the synthetic COCO dataset, while RGB features are more suitable than noise features for region proposals.The compared architectures include RGB Net, Noise Net, RGB-N noise RPN, Noise + RGB RPN, and RGB-N.
- Datasets: The evaluation covers four standard datasets, with Columbia used only for testing a model trained on the synthetic dataset.The standard datasets include NIST16, CASIA, COVER, and Columbia.
- Standard-dataset comparison: Compared with conventional artifact-specific methods, the proposed approach achieves better F1 and AUC performance because those methods provide only partial localization information.The comparison uses F1 scores and pixel-level AUC across the standard datasets.
- Fusion and dataset effects: RGB-N improves the individual streams on every dataset except Columbia, where uncompressed splicing preserves noise differences sufficiently for the noise stream alone.Late fusion performs worse than RGB-N on all datasets, supporting the effectiveness of the proposed fusion approach.
- Robustness and augmentation: The method is more robust to resizing and JPEG-compression attacks than the compared methods, while image flipping improves performance more than JPEG or Gaussian-noise augmentation.The augmentation comparison reports F1/AUC scores for flipping, JPEG compression, and added Gaussian noise.
- Manipulation techniques: Splicing is easiest to detect, removal outperforms copy-move, and copy-move is most difficult because copied regions can retain similar noise distributions.Splicing commonly produces RGB and noise artifacts, whereas removal inpainting strongly affects noise features.
- Qualitative results: Qualitative examples show that RGB-N can perform well when either the RGB or noise stream fails and can classify different manipulation techniques on NIST16.The qualitative comparisons include COVER, Columbia, CASIA 1.0, and NIST16.
5. Conclusion
The proposed network combines RGB and noise streams to learn richer features for image manipulation detection. Fusing these streams improves performance, while the method also detects tampering artifacts and distinguishes among manipulation techniques.
- The network combines RGB and noise streams to learn rich features for image manipulation detection.
- An SRM filter layer adapted from steganalysis extracts noise features that capture inconsistencies between tampered and authentic regions.
- Fusing the two streams improves performance by combining complementary RGB and noise information.
- Experiments on standard datasets show that the method detects tampering artifacts and distinguishes between various tampering techniques.
- The authors identify JPEG compression and additional features as directions for future exploration.