Source-linked AI summary
Image Splicing Localization Using A Multi-Task Fully Convolutional Network (MFCN)
Ronald Salloum, Yuzhuo Ren, C. -C. Jay Kuo
TL;DR
Image splicing localization requires identifying manipulated pixels, whereas many existing detectors only determine whether an image has been spliced. This paper develops FCN-based single-task and multi-task models, finding that the proposed methods outperform existing algorithms and that MFCN produces finer localization than SFCN.
Problem
Many current splicing detectors decide whether an image was spliced without determining which pixels were manipulated.
Method
The paper uses an FCN-based SFCN and an MFCN with separate surface and edge branches, trained on CASIA v2.0 and tested across four datasets.
Results
The SFCN and MFCN outperform existing localization algorithms on four datasets, while edge-enhanced MFCN gives the finest localization among the proposed approaches.
Takeaways & Limitations
Learning surface and boundary information together supports finer spliced-region localization than the single-task approach.
Abstract
from arXiv · showhide
In this work, we propose a technique that utilizes a fully convolutional network (FCN) to localize image splicing attacks. We first evaluated a single-task FCN (SFCN) trained only on the surface label. Although the SFCN is shown to provide superior performance over existing methods, it still provides a coarse localization output in certain cases. Therefore, we propose the use of a multi-task FCN (MFCN) that utilizes two output branches for multi-task learning. One branch is used to learn the surface label, while the other branch is used to learn the edge or boundary of the spliced region. We trained the networks using the CASIA v2.0 dataset, and tested the trained models on the CASIA v1.0, Columbia Uncompressed, Carvalho, and the DARPA/NIST Nimble Challenge 2016 SCI datasets. Experiments show that the SFCN and MFCN outperform existing splicing localization algorithms, and that the MFCN can achieve finer localization than the SFCN.
1 Introduction
Image splicing can create misleading images, while many detection methods identify only whether an image was manipulated rather than which pixels were altered. The paper addresses this localization gap with FCN-based single-task and multi-task models.
- Splicing copies a region from a donor image and pastes it into a host image, potentially adding or removing objects.Such edits can create false impressions and have been used for political propaganda.
- Many existing splicing detectors determine whether an image was manipulated but do not localize the altered pixels.The paper frames localization as determining which image pixels changed because of splicing.
- The proposed solution uses an FCN based on VGG-16 with skip connections, batch normalization, and class weighting.These modifications form the paper’s FCN-based localization architecture.
- The paper first evaluates an SFCN trained on the surface mask, then proposes an MFCN that also learns spliced-region boundaries.The MFCN uses separate surface and edge output branches to refine localization.
- The study evaluates the networks on multiple datasets after training them on CASIA v2.0.The introduction identifies CASIA v1.0, Columbia Uncompressed, Carvalho, and Nimble SCI as test datasets.
2 Related Work
Prior localization methods rely on traces such as noise, CFA interpolation, or JPEG artifacts. The paper contrasts these feature-based approaches with neural networks that learn relevant features during training.
- Existing splicing localization algorithms are grouped by the traces used to distinguish manipulated from authentic regions.The three broad trace classes are noise patterns, CFA interpolation patterns, and JPEG-related traces.
- Noise-based methods assume donor and host images may have different noise patterns caused by cameras, capture settings, or post-processing.The differing noise pattern may help identify the spliced region.
- CFA-based methods exploit interpolation discontinuities caused by different cameras or by rescaling spliced regions.CFA interpolation reconstructs RGB channels from single-sensor image data.
- JPEG-based methods use quantization artifacts or compression-grid discontinuities to detect traces of splicing.These methods can rely on changes in compression characteristics after smoothing or resampling.
- The proposed neural-network solution avoids hand-designed feature extraction by learning relevant features during network training.The paper notes that relatively few published image-forensics methods had used deep learning at the time.
3 Proposed Methods
The proposed methods adapt FCNs for pixel-level splicing localization, first using an SFCN surface mask and then an MFCN that also learns spliced-region edges. Edge-enhanced inference combines both outputs to obtain finer localization.
- 3 Proposed Methods: FCNs are adapted for splicing localization without requiring manual feature extraction.Relevant features are learned automatically during network training.
- 3.2 Single-task Fully Convolutional Network (SFCN): The SFCN uses an FCN VGG-16 architecture with skip connections to predict a binary per-pixel surface mask.Batch normalization and median-frequency class weighting are also incorporated.
- 3.3 Multi-task Fully Convolutional Network (MFCN): The MFCN adds a second output branch that learns the edge or boundary of the spliced region alongside the surface label.Its architecture includes two output branches for multi-task learning.
- 3.4 Edge-enhanced MFCN Inference: Edge-enhanced inference thresholds the surface and edge probability maps, hole-fills the edge mask, and intersects the two masks.The resulting binary system output mask uses both surface and edge predictions.
- 3.4 Edge-enhanced MFCN Inference: Using both edge and surface probability maps produces finer localization of the spliced region than the coarse outputs seen in some SFCN cases.Figure 4 illustrates the edge-enhancement inference sequence and final output.
- 3.5 Training and Testing Procedure: The networks are trained on CASIA v2.0 and evaluated on CASIA v1.0, Columbia Uncompressed, Carvalho, and Nimble Challenge 2016 SCI datasets.Training uses Caffe with stochastic gradient descent and VGG-16 weights pretrained on ImageNet.
4 Performance Evaluation Metrics
Performance is evaluated by comparing thresholded binary output masks with ground-truth masks using per-pixel F1 and MCC scores. Thresholds are optimized separately for each output map and method, and dataset-level averages are reported.
- 4 Performance Evaluation Metrics: F1 and Matthews Correlation Coefficient (MCC) are used as per-pixel localization metrics for proposed and existing methods.Both metrics take a binary system output mask as input.
- 4 Performance Evaluation Metrics: Table 1 documents the training and testing image counts used in the evaluation setup.The table is titled “Training and Testing Images.”
- 4 Performance Evaluation Metrics: Each output map is thresholded, with the optimal threshold selected separately for each method before metric computation.The thresholded mask is compared against the corresponding ground-truth mask.
- 4 Performance Evaluation Metrics: F1 is computed from true positives, false negatives, and false positives in the binary system and ground-truth masks.True positives are correctly identified spliced pixels; false negatives and false positives represent the two corresponding classification errors.
- 4 Performance Evaluation Metrics: For each dataset and method, the reported F1 and MCC values are averages across the dataset.The metrics are therefore summarized at the dataset level rather than for a single image.
5 Experimental Results
Across the evaluated datasets, SFCN and MFCN outperform benchmark splicing-localization methods on F1 and MCC, while edge-enhanced MFCN provides finer masks. These advantages persist under JPEG compression and Gaussian blurring, despite small degradation under distortions.
- 5.1 Performance Comparison: JPEG-dependent ADQ2, ADQ3, and NADQ are evaluated only on CASIA v1.0 and Nimble 2016 SCI, with NA entries for Columbia and Carvalho.These algorithms require JPEG data directly extracted from compressed files.
- 5.1 Performance Comparison: SFCN and MFCN outperform benchmarking algorithms on all four datasets in both average F1 and MCC.The edge-enhanced MFCN performs best among the three proposed methods.
- 5.1 Performance Comparison: Edge-enhanced MFCN yields finer spliced-region localization than SFCN and MFCN without edge-enhanced inference.Figure 5 compares manipulated images, ground-truth masks, and binary outputs across CASIA v1.0 and Carvalho examples.
- 5.2 Performance on JPEG Compressed Images: SFCN and MFCN show small performance degradation after JPEG compression but remain better than existing methods on both compressed and original images.On Carvalho, compression used quality factors 50 and 70; proposed methods on compressed images exceed existing methods on the original uncompressed dataset.
- 5.3 Performance on Gaussian Blurred Images: Under Gaussian smoothing, proposed methods show slight degradation at σ = 2.0 but remain better than existing methods.The Carvalho images were filtered with Gaussian kernels having σ values of 0.5, 1.0, 1.5, and 2.0 pixels.
- 5.3 Performance on Gaussian Blurred Images: SFCN and MFCN on blurred images outperform existing methods evaluated on original unblurred images.This comparison is reported for the Carvalho dataset.
6 Conclusion
The paper applies fully convolutional networks to splicing localization and extends a surface-mask model with multi-task edge learning. The proposed methods improve over published techniques, while edge-enhanced inference produces finer localization than the SFCN and basic MFCN inference.
- 6 Conclusion: Applying FCNs to splicing localization yields large improvement over current published techniques.The network uses an FCN VGG-16 architecture with skip connections, batch normalization, and class weighting.
- 6 Conclusion: MFCN jointly learns surface and edge masks through two output branches to address coarse localization from SFCN.The surface mask classifies pixels as spliced or authentic, while the edge mask identifies spliced-region boundaries.
- 6 Conclusion: Basic MFCN inference outperforms SFCN-based inference, while edge-enhanced MFCN yields finer localization than both alternatives.Basic inference thresholds the surface probability map; edge-enhanced inference uses both surface and edge probability maps.
- 6 Conclusion: The proposed methods are evaluated on manipulated images using the described FCN-based localization approaches.