Source-linked AI summary
Detection and Localization of Image Forgeries using Resampling Features and Deep Learning
Jason Bunk, Jawadul H. Bappy, Tajuddin Manhar Mohammed, Lakshmanan Nataraj, Arjuna Flenner, B. S. Manjunath, Shivkumar Chandrasekaran, Amit K. Roy-Chowdhury, Lawrence Peterson
TL;DR
Image forensics needs methods that can detect and localize manipulations using traces such as resampling artifacts. This paper proposes two deep-learning pipelines combining resampling features with Radon-transform or LSTM processing. Experiments show that both approaches are effective for detecting and localizing digital forgeries.
Problem
Digital manipulations such as scaling, rotation, and splicing leave resampling artifacts, motivating methods that detect and localize forged regions.
Method
The paper combines overlapping-patch resampling features with a Radon-transform/deep-learning pipeline and an LSTM-based patch-classification pipeline.
Results
Both CNN- and LSTM-based networks effectively exploit resampling features to detect tampered regions.
Takeaways & Limitations
The experiments support using resampling features with deep learning for detecting and localizing digital image forgeries.
Abstract
from arXiv · showhide
Resampling is an important signature of manipulated images. In this paper, we propose two methods to detect and localize image manipulations based on a combination of resampling features and deep learning. In the first method, the Radon transform of resampling features are computed on overlapping image patches. Deep learning classifiers and a Gaussian conditional random field model are then used to create a heatmap. Tampered regions are located using a Random Walker segmentation method. In the second method, resampling features computed on overlapping image patches are passed through a Long short-term memory (LSTM) based network for classification and localization. We compare the performance of detection/localization of both these methods. Our experimental results show that both techniques are effective in detecting and localizing digital image forgeries.
1. Introduction
The paper addresses image forgery detection by exploiting resampling artifacts and proposes two deep-learning-based methods for detecting and localizing manipulated regions.
- Motivation: Digital image manipulation has become easier as image distribution and editing tools have expanded.The paper notes widespread use of cameras, social media, and accessible editing software.
- Motivation: Resampling artifacts commonly arise from scaling, rotation, and splicing and provide a basis for forensic detection.
- Contributions: The paper proposes an end-to-end Radon-transform and deep-learning system for detecting and localizing digital manipulations.
- Contributions: It also proposes an LSTM approach combining probability-map resampling features to classify tampered patches.
2. Related Work
Prior work detects manipulation through resampling, compression, camera, and learned visual artifacts, but existing approaches have distinct robustness and task-specific limitations.
- Scope: Image-forensics research covers resampling, copy-move, splicing, object removal, JPEG artifacts, machine learning, and deep learning.
- Resampling detection: Resampling detectors analyze statistical correlations introduced by interpolation, often assuming linear or cubic resampling.
- Resampling detection: An EM-based resampling method is vulnerable to JPEG attacks, especially at quality factors of 95 or lower.
- Deep learning: Deep-learning methods have been applied to generic manipulation, resampling, splicing, bootleg, steganalysis, retouching, and region-forgery detection.
3. Detection and Localization
The proposed detection pipeline densely analyzes overlapping patches for resampling signatures, forms a multichannel heatmap, and segments manipulated regions from the resulting image-level representation.
- Feature extraction: The system extracts small overlapping patches and computes resampling-characterizing features at each patch location.
- Feature extraction: Dense extraction with stride 8 produces a multichannel heatmap with one channel per resampling characteristic.
- Localization: The pipeline postprocesses the heatmap to produce an image-level detection score and a binarized localization map.
- Localization: Otsu thresholding and Random Walk segmentation extract the manipulated regions from the processed heatmap.
- Design tradeoff: The 64x64 patch choice balances stronger resampling evidence in larger patches against poorer localization of small manipulated regions.
- Feature extraction: The feature extractor applies a 3x3 Laplacian, Radon transform, and FFT to identify periodic correlations in prediction errors.
3.1. Deep Neural Networks for Resampling Detection
The paper compares neural architectures for patch-level resampling detection, finding complementary strengths across handcrafted-feature and learned-feature models while evaluating patch-size effects.
- Architectures: A two-hidden-layer artificial neural network performed best as a binary classifier for characterizing resampling in patches.
- Architectures: The second architecture adds a learned convolutional layer to extract resampling features instead of relying solely on handcrafted features.
- Architectures: Including an FFT stabilized training, reduced loss noise, accelerated convergence, and improved the final score by 1%.
- Results: Architecture I performed slightly better for rotation, shearing, and downsampling, whereas Architecture II performed significantly better for upsampling and JPEG-quality differences.
- Results: Architecture I achieved ROC AUC scores of 0.87 for JPG quality and 0.89 for rescaling up.
- Patch-size effects: Upscaling detection reached AUC 0.922 on 64x64 patches and AUC 0.950 on 128x128 patches, while direct comparison requires matched datasets and tuned patch sizes.
3.2. Mask Filtering and Segmentation
The system filters noisy resampling feature maps to improve their suitability for segmentation, then thresholds and combines selected maps into a final manipulation mask.
- Mask Filtering: Bilateral filtering transforms noisy classifier heatmaps toward a bimodal histogram distribution useful for segmentation.The filter is motivated by edge-preserving semantic-segmentation methods and is tuned empirically from feature-map noise distributions.
- Mask Filtering: Only feature maps exhibiting a bimodal-like histogram after filtering are retained for segmentation.The remaining maps are excluded rather than contributing to the final segmentation.
- Thresholding: Otsu thresholding selects usable maps by identifying thresholds that minimize intraclass variance and maximize interclass variance.The threshold either lies near the distribution mean or separates the two modes of a bimodal distribution.
- Segmentation: Random-walker segmentation assigns pixels between histogram modes to classes and helps recover weak or missing boundaries.The resulting binary maps are combined with a bitwise-OR operation to produce the final mask.
- Mask Output: The implementation can produce a grayscale mask and confidence score by averaging binary-mask iterations and normalized nonzero pixel values.A normalized pixel value of 1 corresponds to a black or manipulated region.
3.3. Patch Classification Framework
The patch-classification framework uses resampling-feature correlations across image blocks and an LSTM network to classify manipulated patches and support localization.
- Framework: The framework exploits resampling features and LSTM cells to localize manipulated regions whose boundary statistics differ from natural image statistics.The LSTM learns correlations between blocks of resampling features rather than treating each block independently.
- LSTM Network: The resampling feature map is divided into 8×8 blocks, which are fed sequentially to LSTM cells to learn long-distance block dependencies.The cells correlate neighboring blocks with the current block, and the network uses three stacked layers.
- LSTM Network: LSTM gates control information flow through input, forget, and output gates whose values range from zero to one.The cell state and output state are updated through gated combinations of previous and candidate states.
- Patch Classification: The softmax layer predicts whether an image patch is manipulated or non-manipulated from features produced by the network.The framework describes patch labels as the output of the LSTM-based patch-classification task.
- Training: Patch classification is trained with cross-entropy loss using ground-truth patch labels.The loss uses an indicator for the correct class over the training patches.
4. Experimental Results
The experiments evaluate tampered-patch identification and manipulated-region segmentation on the NIST Nimble 2016 dataset. The LSTM approach performs better with resampling features than with raw pixels for patch classification.
- Experimental setup: The evaluation covers tampered-patch identification and segmentation of manipulated regions on the NIST Nimble 2016 dataset.The dataset includes copy-clone, removal, and splicing manipulations.
- LSTM patch classification: ROC curves compare true-positive and false-positive rates across thresholds for the LSTM patch classifier.The comparison includes both resampling features and raw patch pixels.
- LSTM patch classification: Resampling features boost LSTM patch-classification performance compared with using raw pixels.The paper reports that the ROC curve and Table 2 results attest to effective tampered-patch classification.
5. Conclusion
The paper concludes that both proposed deep-learning approaches can exploit resampling features to detect and localize manipulated image regions. It identifies combining the methods as future work.
- The paper presents two methods for detecting and localizing manipulated regions in images.
- CNNs and LSTM-based networks are effective in exploiting resampling features to detect tampered regions.
- Future work will investigate combining the CNN- and LSTM-based methods for image-forgery detection.