Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries

Jawadul H. Bappy; Cody Simons; Lakshmanan Nataraj; B. S. Manjunath; Amit K. Roy-Chowdhury

Source-linked AI summary

Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries

Jawadul H. Bappy, Cody Simons, Lakshmanan Nataraj, B. S. Manjunath, Amit K. Roy-Chowdhury

arXiv:1903.02495v1Mar 6, 2019cs.CV

TL;DR

Visually subtle image manipulations are difficult to localize at the pixel level. The paper combines resampling features and LSTM frequency-domain modeling with CNN encoder-decoder spatial processing, and reports high-confidence localization across three datasets.

Problem
Content-changing manipulations can alter an image’s semantic meaning while leaving manipulated regions visually indistinguishable from authentic content, making pixel-level localization difficult.
Method
A hybrid CNN-LSTM encoder-decoder uses resampling features for transformation artifacts, LSTM cells for frequency-domain patch correlations, and decoding for binary pixel masks.
Results
The proposed architecture localizes manipulated regions at pixel level with high confidence across NIST’16, IEEE Forensics Challenge, and COVERAGE datasets.
Takeaways & Limitations
The framework combines frequency-domain features and spatial context, while the introduced large masked splicing dataset supports training deep tamper-localization models.

Abstract

from arXiv · show

With advanced image journaling tools, one can easily alter the semantic meaning of an image by exploiting certain manipulation techniques such as copy-clone, object splicing, and removal, which mislead the viewers. In contrast, the identification of these manipulations becomes a very challenging task as manipulated regions are not visually apparent. This paper proposes a high-confidence manipulation localization architecture which utilizes resampling features, Long-Short Term Memory (LSTM) cells, and encoder-decoder network to segment out manipulated regions from non-manipulated ones. Resampling features are used to capture artifacts like JPEG quality loss, upsampling, downsampling, rotation, and shearing. The proposed network exploits larger receptive fields (spatial maps) and frequency domain correlation to analyze the discriminative characteristics between manipulated and non-manipulated regions by incorporating encoder and LSTM network. Finally, decoder network learns the mapping from low-resolution feature maps to pixel-wise predictions for image tamper localization. With predicted mask provided by final layer (softmax) of the proposed architecture, end-to-end training is performed to learn the network parameters through back-propagation using ground-truth masks. Furthermore, a large image splicing dataset is introduced to guide the training process. The proposed method is capable of localizing image manipulations at pixel level with high precision, which is demonstrated through rigorous experimentation on three diverse datasets.

1 INTRODUCTION

The paper targets pixel-level localization of visually subtle, content-changing image manipulations using a unified architecture that combines frequency-domain resampling features, LSTM modeling, and spatial encoder-decoder processing.

1 INTRODUCTION: Content-changing manipulations such as copy-move, splicing, and object removal can alter semantic meaning while remaining visually difficult to detect.The localization task is harder because only the manipulated region must be identified, even when it blends with surrounding content.
1 INTRODUCTION: The proposed framework combines resampling features, an LSTM network, and encoder-decoder architectures to learn pixel-level manipulation masks.Resampling features capture transformation artifacts, the LSTM learns frequency-domain correlations among patches, and the encoder-decoder captures spatial information and produces pixel-wise predictions.
1.2 Main Contributions: The network is trained end to end with ground-truth masks, and synthesized images augment the data for the data-hungry deep model.The paper introduces a large image-splicing dataset with binary masks for tamper-localization training.
1.2 Main Contributions: The method uses the whole image as input rather than isolated patches, allowing the architecture to exploit global, intra-patch, and inter-patch context.The encoder supplies spatial feature maps while LSTM outputs are combined before decoding into a binary mask.
1.2 Main Contributions: The architecture is evaluated on three datasets and is reported to localize manipulated regions at pixel level with high confidence.The evaluation uses NIST’16, the IEEE Forensics Challenge, and COVERAGE datasets.
1.2 Main Contributions: The work extends an earlier approach by adding encoder-based spatial context, LSTM frequency-domain features, whole-image input, resampling features, and a large splicing dataset.The encoder also provides larger receptive fields for identifying large manipulated regions.

2 RELATED WORK

Prior work addresses image-forgery detection through resampling analysis, JPEG and statistical artifacts, and deep-learning methods for visual segmentation and manipulation detection.

2 RELATED WORK: Existing media-forensics methods detect resampling, JPEG artifacts, and content-changing manipulations using signal-processing and statistical cues.Resampling approaches commonly exploit linear or cubic interpolation and periodic properties in transformed images.
2 RELATED WORK: Deep-learning research has applied CNNs and hierarchical features to semantic segmentation and other image-manipulation detection tasks.These methods include generic manipulation detection, resampling detection, splicing detection, and specialized forgery or retouching detection.
2 RELATED WORK: The paper distinguishes its localization goal from semantic segmentation because manipulated regions may not correspond to meaningful objects or visible visual concepts.This distinction motivates combining spatial CNN features with manipulation-specific cues.

3 NETWORK ARCHITECTURE OVERVIEW

The architecture combines resampling features and LSTM-based frequency-domain correlations with encoder-decoder spatial processing to produce pixel-level tamper masks. Hilbert ordering preserves patch locality, while the decoder converts coarse features into dense predictions and training uses weighted cross-entropy.

Architecture: The framework combines an LSTM with resampling features, an encoder, and a decoder to localize manipulated regions at pixel level.The LSTM analyzes frequency-domain relationships among patches, while encoder-decoder processing provides spatial features and dense masks.
Resampling Features: Radon-transform-based resampling features capture statistical differences between manipulated and non-manipulated patches caused by resampling artifacts.The method extracts features from non-overlapping patches and uses patterns in the Radon transform and its column sums.
LSTM Network: A third-order Hilbert curve orders 64 patches so neighboring image regions remain spatially coherent when processed sequentially by the LSTM.The 64 patches form an 8 × 8 grid, requiring three recursive divisions for the Hilbert ordering.
LSTM Network: The LSTM learns dependencies among neighboring patches, uses two stacked layers and 64 time steps, and reshapes projected outputs into an 8 × 8 × Nf feature map.Each time step produces a transformed feature for one patch before the outputs are reordered and reshaped spatially.
Encoder and Decoder: The encoder uses residual units and increasing feature-map depths, while the decoder upsamples, convolves, and batch-normalizes coarse representations into dense maps.The encoder uses 32, 64, 128, and 256 feature maps across four layers; decoder upsampling has no learnable parameters.
Training: Weighted cross-entropy training assigns greater weight to manipulated pixels to address the imbalance between manipulated and non-manipulated classes.Class weights are inversely proportional to their training-set frequencies and are normalized.

4 EXPERIMENTS

The proposed model is evaluated for segmentation of manipulated regions on three challenging datasets: NIST’16, IEEE Forensics Challenge, and COVERAGE.

The experiments evaluate segmentation of manipulated regions given an image.
The evaluation uses the NIST’16 dataset.
The evaluation also uses the IEEE Forensics Challenge and COVERAGE datasets.

4.1 Datasets

The paper creates a large synthesized manipulation dataset for training and evaluates the model on three datasets with ground-truth masks. Dataset preparation uses randomized splits, patch extraction, and end-to-end training.

Creation of Synthesized Data: Approximately 65k manipulated images are created to train the proposed Base-Model.The Base-Model is subsequently fine-tuned with the NIST’16 and IEEE Forensics Challenge datasets.
Creation of Synthesized Data: The synthesized dataset focuses mainly on object splicing through additions and subtractions.Segmented objects are obtained from MS-COCO using its ground-truth masks.
Creation of Synthesized Data: Figure 7 presents IEEE Forensics Challenge examples as input images, ground-truth masks, probability heatmaps, and predicted binary masks.
Creation of Synthesized Data: The synthesized data contains approximately 40k images from DRESDEN and 25k additional images using DRESDEN and NIST’16.The resulting scale is described as a hundred-fold increase over most comparable datasets, and the Base-Model predicts manipulated regions at pixel level.
Dataset Preparation: Evaluation uses NIST’16, IEEE Forensics Challenge, and COVERAGE, with NIST’16 containing copy-clone, removal, and splicing manipulations.These datasets provide ground-truth masks for manipulated regions.
Dataset Preparation: The data is randomly split into training, validation, and testing subsets of 70%, 5%, and 25%, respectively.Training data is augmented with patches extracted from image corners and centers, while validation and test sets are not augmented.

4.2 Experimental Analysis

The proposed LSTM-EnDec model is evaluated across three datasets using pixel-wise segmentation accuracy, ROC/AUC analysis, qualitative masks, and comparisons with baseline and existing methods. Results show gains over several alternatives, while global image context improves localization precision.

Experimental Setup: Experiments evaluate the model on NIST’16, IEEE Forensics Challenge, and COVERAGE using pixel-wise accuracy and ROC/AUC analysis.The setup also includes baseline comparisons, state-of-the-art comparisons, qualitative analysis, and an experiment on global context.
Performance of the Proposed Model.: Finetuning LSTM-EnDec-Base improves pixel-wise tamper labeling by 3.44% on NIST’16 and 2.95% on IEEE Forensics Challenge.The base model is trained on synthesized data, then finetuned on the NIST’16 and IEEE Forensics Challenge training sets.
Comparison against Existing Approaches.: The framework outperforms existing tamper-localization methods by a large margin on NIST’16 and exceeds Encoder-Decoder by 11.84%.The proposed architecture uses resampling features to predict manipulated regions.
Comparison against Existing Approaches.: After removing predicted bounding boxes with areas under 64, average precision rises from 0.825 to 0.923, compared with 0.934 for [91].The proposed model produces pixel-level segmentation maps and derives bounding boxes through contour approximation rather than predicting boxes directly.
ROC Curve: The model achieves AUC values of 0.7936, 0.7577, and 0.7124 on NIST’16, IEEE Forensics, and COVERAGE, respectively.The ROC curves measure binary pixel classification between manipulated and non-manipulated pixels.
Qualitative Analysis of Segmentation.: Qualitative masks localize manipulated regions with high probability and higher overlap than ground truth despite boundary effects from smooth image boundaries.The boundary issue is reported for NIST’16 examples.
Impact of Global Context: Localization precision degrades with smaller input patches, whereas full-image input performs more precisely because global context helps analyze manipulated images.The study compares image input with patch input to assess the effect of global context.

5 CONCLUSION

The paper presents a hybrid CNN-LSTM encoder-decoder for pixel-wise segmentation of manipulated image regions. It combines spatial feature maps, resampling-based patch transitions, and decoding to binary masks, and reports effective segmentation across several manipulation types.

5 CONCLUSION: The hybrid CNN-LSTM architecture classifies manipulated and non-manipulated regions using encoder feature maps, resampling features, LSTM transitions, and a decoder-produced binary mask.The encoder provides spatial maps, the LSTM models transitions between manipulated and non-manipulated patches, and the decoder maps encoded features to masks.
5 CONCLUSION: The paper introduces a large synthesized image dataset intended to support deep-network training in media forensics.The dataset includes a large number of images.
5 CONCLUSION: Experiments show that the approach can segment copy-move, object removal, and splicing manipulations.The reported outcome concerns pixel-level segmentation of these manipulation types.