Source-linked AI summary
Constrained R-CNN: A general image manipulation detection model
Chao Yang, Huizhou Li, Fangting Lin, Bin Jiang, Hao Zhao
TL;DR
Image manipulation detection is constrained by handcrafted features and by methods that overlook manipulation classification alongside localization. Constrained R-CNN learns unified manipulation features and uses a coarse-to-fine architecture with attention-guided region proposals and segmentation, achieving state-of-the-art performance with reported F1 gains on three benchmarks.
Problem
Existing image manipulation detectors rely on handcrafted or predetermined features and often address localization without manipulation-technique classification.
Method
Constrained R-CNN uses a learnable manipulation feature extractor, attention region proposal network, and coarse-to-fine segmentation architecture.
Results
The model achieves state-of-the-art performance, with F1 score increased by 28.4%, 73.2%, and 13.3% on NIST16, COVERAGE, and Columbia, respectively.
Takeaways & Limitations
The approach performs manipulation classification and tampered-region localization simultaneously across four manipulated-image benchmarks.
Takeaways & Limitations
Stage-1 is pre-trained using the COCO synthetic dataset because standard datasets lack sufficient data for deep neural network training.
Abstract
from arXiv · showhide
Recently, deep learning-based models have exhibited remarkable performance for image manipulation detection. However, most of them suffer from poor universality of handcrafted or predetermined features. Meanwhile, they only focus on manipulation localization and overlook manipulation classification. To address these issues, we propose a coarse-to-fine architecture named Constrained R-CNN for complete and accurate image forensics. First, the learnable manipulation feature extractor learns a unified feature representation directly from data. Second, the attention region proposal network effectively discriminates manipulated regions for the next manipulation classification and coarse localization. Then, the skip structure fuses low-level and high-level information to refine the global manipulation features. Finally, the coarse localization information guides the model to further learn the finer local features and segment out the tampered region. Experimental results show that our model achieves state-of-the-art performance. Especially, the F1 score is increased by 28.4%, 73.2%, 13.3% on the NIST16, COVERAGE, and Columbia dataset.
1. INTRODUCTION
Image manipulation detection must generalize across diverse content manipulations while addressing both what technique was used and where tampering occurred. Constrained R-CNN responds with a learnable, coarse-to-fine architecture for simultaneous manipulation classification and region segmentation.
- Content manipulation poses risks in fake news, bogus certificates, and malicious rumors because it can subtly change image content.The paper highlights splicing, copy-move, and removal as representative manipulation techniques.
- Existing methods are limited by handcrafted or predetermined features that are specific to manipulation types and constrain generalization.
- Manipulation forensics requires both identifying how an image was altered and localizing where the alteration occurred.A localization mask alone can confuse techniques such as copy-move and splicing.
- The proposed solution uses a learnable manipulation feature extractor and attention region proposal network within an end-to-end design.The architecture creates unified features from data and identifies manipulated regions for later classification and coarse localization.
- Constrained R-CNN combines manipulation-technique classification with manipulated-region segmentation in a coarse-to-fine architecture.Its contributions include a unified learnable feature representation and an attention regional proposal network.
2. PROPOSED METHOD
Constrained R-CNN uses a learnable feature extractor and a two-stage coarse-to-fine architecture to classify manipulation techniques, localize candidate regions, and segment tampered pixels.
- 2. PROPOSED METHOD: Constrained R-CNN combines learnable manipulation features with coarse classification/localization and fine pixel-level segmentation.The architecture uses LMFE, Stage-1 detection, and Stage-2 segmentation.
- 2.2. Learnable manipulation feature extractor: The constrained convolution layer adaptively learns manipulation features while retaining richer content and noise-inconsistency information than SRM filters.The constrained convolution is followed by ResNet-101 in the learnable manipulation feature extractor.
- 2.3. Coarse Manipulation Detection (Stage-1): Stage-1 performs manipulation-technique classification and coarse bounding-box localization using RoI features after RPN-A proposals.The prediction module applies fully connected and softmax layers to classification and bounding-box regression.
- 2.3. Coarse Manipulation Detection (Stage-1): RPN-A applies channel and spatial attention before proposing regions of interest corresponding to potential tampered regions.Its proposals treat tampered regions as foreground rather than whole objects.
- 2.4. Fine Manipulation Detection (Stage-2): Stage-2 uses Stage-1 boxes to refine local features and predict tampered regions at pixel level.A skip structure fuses detailed low-level Conv 3x features with high-level CBAM features before local refinement and decoding.
3. EXPERIMENTS
Experiments evaluate Constrained R-CNN against state-of-the-art methods on four benchmarks, using F1, AUC, AP, and multi-class classification comparisons. The model shows strong overall detection gains, with benefits attributed to its constrained convolution, skip structure, coarse-to-fine localization, and selected augmentation strategies.
- 3.2. Pre-trained Model: Constrained Conv significantly improves AP with only half the parameters, while Conv-C Net + CBAM-R improves AP by 6.1% over RGB-N.The pre-trained model experiments compare RGB-N and constrained-convolution architectures on the COCO synthetic dataset.
- 3.3. Experiments on Standard Datasets: F1 score increased by 0.205, 0.32, and 0.093 on NIST16, COVERAGE, and Columbia, corresponding to growth rates of 28.4%, 73.2%, and 13.3%.Tables 2 and 3 compare F1 and AUC against baseline methods on four benchmarks.
- 3.3. Experiments on Standard Datasets: Compared with Ours-Base, the full Constrained R-CNN significantly improves detection performance, supporting the effectiveness of the skip structure.Ours-Base uses the same architecture without the skip structure.
- 3.3. Experiments on Standard Datasets: Image flipping significantly improves performance on two datasets, while JPEG compression improves CASIA and produces AUC comparable to a state-of-the-art model.Table 4 reports F1/AUC scores for flipping, Gaussian noise, and JPEG compression augmentation.
- 3.4. Manipulation Techniques Classification Comparison: Constrained R-CNN achieves state-of-the-art performance on the entire NIST16 multi-class dataset, improving copy-move detection and matching RGB-N on splicing detection.Classification is evaluated at bounding-box level because NIST16 provides labels for splicing, copy-move, and removal.
- 3.3. Experiments on Standard Datasets: Qualitative comparisons show better localization and technique classification, with coarse-to-fine local-region detection reducing output-mask noise and improving segmentation accuracy.Figure 4 presents qualitative localization results, while Figure 5 compares manipulation-technique detection on NIST16.
4. CONCLUSIONS
Constrained R-CNN simultaneously performs manipulation technique classification and tampered-region segmentation, using clues learned directly from data without handcrafted components. Across four manipulated-image benchmarks, it achieves state-of-the-art performance in both classification and localization.
- Constrained R-CNN simultaneously performs manipulation technique classification and tampered-region segmentation.
- Across four manipulated-image benchmarks, the model achieves state-of-the-art performance in both manipulation classification and localization.
- The model captures manipulation clues directly from data without handcrafted components, supporting more general image forensics.