Source-linked AI summary
Image Forgery Localization Based on Multi-Scale Convolutional Neural Networks
Yaqi Liu, Qingxiao Guan, Xianfeng Zhao, Yun Cao
TL;DR
Forgery localization must identify tampered pixels rather than merely classify images as pristine or fake. The paper introduces MSCNNs, which combine multi-scale CNN detectors with superpixel-based segmentation fusion, and reports the best IFS-TC performance among methods using one clue, while noting remaining robustness gaps for real applications.
Problem
Forgery localization requires accurate pixel-level tamper detection, but is more difficult than binary forgery detection because the tampered areas must be located.
Method
MSCNNs use unified CNN detectors on color sliding windows at multiple scales and fuse their tampering possibility maps through a superpixel-level segmentation graph.
Results
MSCNNs achieve the best performance on the IFS-TC dataset among forgery-localization methods using only one clue for splicing detection.
Takeaways & Limitations
The results support multi-scale CNN analysis with segmentation-based fusion as an effective and efficient approach to forgery localization.
Takeaways & Limitations
Robustness against post compression, manipulation qualities, and camera models still needs further study before real applications.
Abstract
from arXiv · showhide
In this paper, we propose to utilize Convolutional Neural Networks (CNNs) and the segmentation-based multi-scale analysis to locate tampered areas in digital images. First, to deal with color input sliding windows of different scales, a unified CNN architecture is designed. Then, we elaborately design the training procedures of CNNs on sampled training patches. With a set of robust multi-scale tampering detectors based on CNNs, complementary tampering possibility maps can be generated. Last but not least, a segmentation-based method is proposed to fuse the maps and generate the final decision map. By exploiting the benefits of both the small-scale and large-scale analyses, the segmentation-based multi-scale analysis can lead to a performance leap in forgery localization of CNNs. Numerous experiments are conducted to demonstrate the effectiveness and efficiency of our method.
1. INTRODUCTION
Image forgery localization requires accurate pixel-level detection of tampered areas, making it more difficult than binary forgery detection. The paper proposes MSCNNs, combining multi-scale CNN detectors with segmentation-based map fusion, and reports strong performance using a single statistical clue.
- Forgery localization targets accurate tampered areas through pixel-level analysis, unlike forgery detection, which only distinguishes pristine from fake images.
- The method adapts and modulates state-of-the-art CNN architectures rather than designing an entirely novel CNN.
- The paper proposes MSCNNs, which apply CNNs to sliding windows at different scales and fuse their tampering possibility maps on a superpixel graph.A unified CNN architecture handles color patches, while segmentation-based fusion produces the final decision map.
- MSCNNs are designed as a set of weak multi-scale classifiers that exploit complementary small-scale and large-scale analyses.
- On the IFS-TC dataset, MSCNNs achieve the best performance among forgery-localization methods using only one clue for splicing detection.Methods scoring higher combine multiple clues, whereas MSCNNs use statistical features extracted by CNNs and can incorporate additional clues.
2. METHOD
The method combines unified CNN detectors across input scales with pixel-level possibility-map generation and superpixel-based graph fusion. This design smooths and integrates multi-scale evidence while reducing the computational burden of pixel-level graph optimization.
- 2.1. CNNs architecture: The CNNs use base-filter preprocessing for color patches, including fixed SRM kernels and constrained filters.Fixed SRM kernels are used as fixed base filters, while constrained filters impose a center weight of −1 and surrounding weights summing to 1.
- 2.1. CNNs architecture: The unified CNN architecture adapts to different input-patch scales by changing the final average-pooling parameter while keeping a 256-dimensional fully connected input.The architecture is modulated from prior CNNs and uses 90 base filters for color patches.
- 2.2. Maps generation: Sliding CNN detectors process s × s windows with stride st, producing possibility maps whose elements represent the probabilities that corresponding patches are fake.The initial map has dimensions hs × ws, with dimensions determined by image size, window scale, and stride.
- 2.2. Maps generation: Overlapping patch probabilities are aggregated into image-sized maps, edge pixels with no covering patches receive nearest valid probabilities, and mean filtering smooths mosaic artifacts.The smoothed map uses a filter whose size corresponds to the patch size.
- 2.3. Maps fusion: The method fuses multi-scale possibility maps by constructing a graph over SLIC-generated superpixels and minimizing an energy function for superpixel labels.Thousands of superpixels are used to limit information loss, and superpixel-level computation is much smaller than pixel-level graph optimization.
- 2.3. Maps fusion: Superpixel possibility maps are generated using compared aggregation strategies, including mean and maxa, before graph-based fusion.The mean strategy averages pixel probabilities within each superpixel; the passage also introduces maxa as an alternative strategy.
3. EXPERIMENTAL EVALUATION
Experiments on IFS-TC and RTD evaluate F1-score, robustness across training sets, and computational efficiency. Multi-scale CNNs improve localization performance over single-scale CNNs and provide a substantially faster alternative to SCRM+LDA.
- Experimental setup: Experiments evaluate localization with average F1-score on IFS-TC and RTD datasets using CNN variants, multi-scale CNNs, and competing methods.The IFS-TC evaluation uses testing sets with locally defined training and testing subsets, while RTD contains realistic object insertion and removal forgeries.
- IFS-TC evaluation: Mean filtering improves F1-scores by smoothing sliding-window mosaic artifacts, while CNN computation is 1/340 of SCRM+LDA at patch size 64 and stride 16.CNN-SRM is more efficient than constrained-filter variants because SRM base filters contain many zero values.
- IFS-TC evaluation: CNNs with input scales 64 and 96 outperform other single-scale settings, but no single-scale CNN exceeds an F1-score of 0.35.The multi-scale analysis combines detectors at different patch scales to improve performance.
- IFS-TC evaluation: Multi-scale analysis substantially improves CNN performance, and MSCNNs-maxa reaches an F1-score of 0.4063 versus 0.4072 for the IFS-TC challenge winner.The winner uses three different clues, whereas MSCNNs-maxa uses CNN-extracted features alone.
- RTD evaluation: When trained on IFS-TC and tested on RTD, all models show lower performance, indicating sensitivity to the training set.The RTD images may differ in capture conditions from the IFS-TC training images.
- RTD evaluation: When trained on RTD, MSCNNs outperform SCRM+LDA except on ND90, while requiring 34.62 s plus 30.36 s per image versus 2220.45 s for SCRM+LDA.The reported MSCNN timing separates five CNNs on GPU from CPU-based fusion.
4. CONCLUSIONS
The paper proposes MSCNNs with segmentation-based multi-scale analysis for forgery localization and reports effectiveness and efficiency on public datasets. It also identifies robustness against post-compression, manipulation quality, and camera-model variation as unresolved issues.
- CONCLUSIONS: MSCNNs use CNN forgery detectors for color patches at different scales, followed by segmentation-based multi-scale analysis to exploit their information.The method generates multi-scale analyses and uses segmentation to extract information from them.
- CONCLUSIONS: Experiments on publicly available datasets demonstrate the proposed method’s effectiveness and efficiency.
- CONCLUSIONS: Robustness against post compression, manipulation qualities, and camera models remains insufficiently studied for real applications.The paper identifies these robustness dimensions as future research needs.
- CONCLUSIONS: Future improvements may adopt more powerful CNNs.