PaperlayerView exact revision on arXiv

Source-linked AI summary

PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization

Xiaohong Liu, Yaojie Liu, Jun Chen, Xiaoming Liu

arXiv:2103.10596v2cs.CV

TL;DR

Image manipulation detection and localization must handle diverse attacks and forged-region scales. PSCC-Net combines dense multi-scale features, progressive mask refinement, and spatio-channel correlations, reporting strong detection and localization results with 50+ FPS processing for 1,080P images.

  • Problem

    Forged regions vary in size, while prior architectures struggle to jointly leverage local and global features for scale variation.

  • Method

    PSCC-Net extracts multi-scale features and progressively predicts masks from coarse to fine, using SCCM to capture spatial and channel-wise correlations.

  • Results

    PSCC-Net achieves the best localization performance on four of five evaluated datasets and processes one 1,080P image in 0.019s.

  • Takeaways & Limitations

    The progressive design supports earlier inference termination for runtime savings while retaining progressively improving localization performance.

  • Takeaways & Limitations

    Some real-life manipulations remain incompletely localized when fabricators remove manipulation traces.

Abstract

from arXiv · show

To defend against manipulation of image content, such as splicing, copy-move, and removal, we develop a Progressive Spatio-Channel Correlation Network (PSCC-Net) to detect and localize image manipulations. PSCC-Net processes the image in a two-path procedure: a top-down path that extracts local and global features and a bottom-up path that detects whether the input image is manipulated, and estimates its manipulation masks at multiple scales, where each mask is conditioned on the previous one. Different from the conventional encoder-decoder and no-pooling structures, PSCC-Net leverages features at different scales with dense cross-connections to produce manipulation masks in a coarse-to-fine fashion. Moreover, a Spatio-Channel Correlation Module (SCCM) captures both spatial and channel-wise correlations in the bottom-up path, which endows features with holistic cues, enabling the network to cope with a wide range of manipulation attacks. Thanks to the light-weight backbone and progressive mechanism, PSCC-Net can process 1,080P images at 50+ FPS. Extensive experiments demonstrate the superiority of PSCC-Net over the state-of-the-art methods on both detection and localization.

I. INTRODUCTION

Image manipulation enables deceptive content, making reliable detection and pixel-level localization important. PSCC-Net addresses multiple manipulation types with a progressive, correlation-based architecture designed for scale variation and both detection and localization.

  • Manipulated images can support false information, insurance fraud, and Deepfake content, motivating reliable detection models.
  • Image manipulation includes splicing, copy-move, and removal, which respectively copy content from another image, copy target content, or refill removed objects.
  • Generic image manipulation detection and localization remains challenging because forged regions vary in size and prior architectures inadequately fuse local and global features.
  • PSCC-Net combines dense multi-scale feature extraction with a bottom-up path that progressively estimates manipulation masks and explicitly predicts image-level detection scores.
  • SCCM captures spatial and channel-wise correlations to provide holistic information and improve generalization across manipulation attacks.
  • PSCC-Net reports state-of-the-art results for both image manipulation detection and localization.

II. RELATED WORK

Related work progresses from manipulation-specific localization and detection methods toward generic models handling multiple forgery types. Existing approaches differ in feature design, computational cost, detectable-region size, and output granularity.

  • Image manipulation detection uses either implicit scores derived from predicted masks or explicit scores from dedicated classification modules.
  • Early methods target specific forgeries such as splicing, copy-move, or removal, but real-world manipulation types are often unknown in advance.
  • J-LSTM and H-LSTM capture boundary features but are time-consuming and limit detectable regions through patch-based designs.
  • RGB-N provides bounding boxes rather than segmentation masks, while ManTra-Net learns features for many known manipulation types through anomaly detection.
  • PSCC-Net architecture descriptions include a detection head and progressively refined masks, distinguishing its detection and localization outputs.

C. Progressive Mechanism

PSCC-Net uses a densely connected multi-scale design to refine manipulation masks from coarse to fine. Each earlier mask conditions the next prediction, while SCCM enriches features with spatial and channel-wise correlations.

  • C. Progressive Mechanism: Progressive mechanisms refine manipulation masks from small scales to large scales, using each predicted mask as a prior for the next-scale estimate.
  • C. Progressive Mechanism: SCCM captures spatial and channel-wise correlations, while spatial attention aggregates pixel-level features and channel-wise attention compares feature maps.
  • C. Progressive Mechanism: Conventional encoder-decoder and no-pooling structures have difficulty jointly leveraging local and global features across varying forged-region sizes.
  • C. Progressive Mechanism: The bottom-up path predicts a detection score and generates manipulation masks progressively with full supervision.
  • C. Progressive Mechanism: At scales 1–3, each current-scale feature is modulated by an upsampled previous mask before SCCM produces the next manipulation mask.

B. Spatio-Channel Correlation Module

SCCM captures spatial and channel-wise correlations to improve manipulation localization. It reduces spatial-correlation cost through reshaping and fuses both attentions with residual learning before generating a one-channel mask.

  • SCCM captures spatial and channel-wise correlations to distinguish forged regions from pristine regions and reduce overfitting to specific attacks.Spatial correlation aggregates similar local features, while channel-wise correlation enhances representations associated with manipulated areas.
  • The input feature is reshaped to reduce spatial-correlation size while preserving all feature information.With r = 4, a 256×256 feature produces a 4,096 × 4,096 correlation instead of 65,536 × 65,536.
  • Spatial and channel-wise attentions are applied to the same linear embedding for mutual accommodation and lower fusion cost.The attentions are formed through matrix multiplications and use Softmax-based similarity relationships.
  • The attention outputs are transformed by 1 × 1 convolutions and combined with learnable trade-off parameters through residual learning.The feature is expressed as Z = X + αs · ωs(Ys) + αc · ωc(Yc), with both αs and αc initialized as 1.
  • A Conv-ReLU-Conv-Sigmoid mask-generation block converts the fused features into a one-channel predicted manipulation mask.The convolution layers use 3 × 3 kernels.

C. Loss Function

PSCC-Net uses binary cross-entropy to supervise both image-level detection and multiscale localization masks.

  • Binary cross-entropy supervises the detection score and every predicted manipulation mask.The detection label and mask pixels use 0 for pristine and 1 for forged; ground-truth masks are downsampled to each prediction scale.

D. Training Data Synthesis

The authors construct a synthetic dataset covering splicing, copy-move, removal, and pristine images because no standard IMDL training dataset exists.

  • The synthetic dataset contains splicing, copy-move, removal, and pristine classes generated through manipulation-specific procedures.Splicing uses transformed regions selected from MS COCO and pasted into different images.
  • The dataset examples show generated images from different manipulation types together with their ground-truth masks.
  • Approximately 0.38M images comprise the synthetic dataset across the four classes.The class counts are 116,583 splicing, 100,000 copy-move, 78,246 removal, and 81,910 pristine images.
  • Training samples are uniformly drawn on-the-fly at 0.025M images per class per epoch.This forms a 0.1M-image training set per epoch, alongside a validation set of 4 × 100 images; all synthetic images are 256 × 256.

IV. EXPERIMENTS

Experiments evaluate localization across four standard datasets and one real-world dataset, while detection uses a balanced CASIA-D protocol. The model supports arbitrary-size inputs and is substantially lightweight relative to cited baselines.

  • Test data: Localization is evaluated on Columbia, Coverage, CASIA, NIST16, and the real-world IMD20 dataset.The datasets cover splicing, copy-move, removal, or combinations of these manipulation types.
  • Test data: CASIA-D contains 1,842 images with 50% forged and 50% pristine examples for manipulation-detection evaluation.It is created because no standard detection benchmark exists and CASIA uniquely pairs manipulated images with pristine counterparts.
  • Evaluation metrics: Localization uses pixel-level AUC and F1, while detection uses image-level AUC, F1, EER, and TPR1%.The EER threshold binarizes masks and detection scores for F1 computation.
  • Implementation: PSCC-Net has 2.0M top-down-path parameters and 1.6M bottom-up-path parameters, compared with 3.8M for ManTra-Net and 3.7M for SPAN.Within the bottom-up path, detection uses 0.9M parameters and localization uses 0.7M.
  • Implementation: The network accepts arbitrary-size images and resamples top-down features to fixed scales before mask prediction.Features are resampled to 256 × 256, 128 × 128, 64 × 64, and 32 × 32, with SCCM ratios 4, 2, 2, and 1.

B. Comparisons on Localization

PSCC-Net is evaluated against established IMDL baselines using pre-trained and fine-tuned models across standard and real-life datasets. It achieves the strongest overall localization results, with especially large gains on real-life manipulations and fine-tuned evaluations.

  • Pre-trained model: PSCC-Net achieves the best pre-trained localization AUC on Columbia, CASIA, NIST16, and IMD20, ranking second on Coverage.Its largest gain is 5.6% on real-life manipulated images, while it surpasses ManTra-Net by 2.8% on Coverage.
  • Pre-trained model: 5.6% ↑ is the most significant pre-trained-model gain, achieved on real-life manipulated images.This result supports PSCC-Net's reported generalization advantage over the compared methods.
  • Pre-trained model: 0.4% is PSCC-Net's fine-tuned AUC gain over SPAN on Coverage.The authors attribute the pre-trained Coverage shortfall to imperfect training data for similar-appearance copy-move cases.
  • Fine-tuned model: PSCC-Net surpasses all baselines in fine-tuned AUC, exceeding SPAN by over 2.4% on average.Fine-tuned models are initialized from the pre-trained weights and trained on dataset-specific splits.
  • Fine-tuned model: PSCC-Net exceeds SPAN by over 16.6% on average in fine-tuned F1 score.The authors use this result to validate the effectiveness of the overall network design.
  • Qualitative comparisons: Qualitatively, PSCC-Net produces more accurate masks, fewer false alarms, and effective localization across large, small, and real-life manipulations.The comparisons use ManTra-Net and SPAN alongside results on four standard datasets and IMD20.

C. Comparisons on Detection

PSCC-Net combines mask-based detection with a dedicated detection head and evaluates both detection behavior and SCCM attention responses. The reported results emphasize improved detection reliability and localization cues for forged regions.

  • Detection evaluation: PSCC-Net† achieves the best detection performance on all reported metrics on CASIA-D.PSCC-Net† uses the same mask-averaging score strategy as the compared baselines for fair comparison.
  • Detection evaluation: A tailored detection head dramatically improves performance by reducing false alarms on pristine images.Detection can precede localization so only detected forgeries are passed to the localization stage.
  • SCCM visualization: Spatial response maps usually assign high values to forged regions and low values to pristine regions, while authentic-image maps remain consistently low.The visualization examines selected pixels and their spatial correlations in SCCM.
  • SCCM visualization: Channel-wise attention consolidates forged regions in Yc relative to X and enhances no region for authentic images.The comparison visualizes the first channel of X and Yc.
  • Progressive masks: Predicted masks are visualized progressively from Scale 4 to Scale 1 across manipulated images from five datasets.The figure uses outputs from the pre-trained model.

E. Visualization of Predicted Manipulation Masks on Different Scales

PSCC-Net progressively refines manipulation masks from coarse to fine, using each preceding mask to guide the next scale. This produces gradually clearer localization with fewer false alarms while retaining runtime-saving early-exit options.

  • E. Visualization of Predicted Manipulation Masks on Different Scales: The progressive mechanism predicts a coarse manipulation mask first, then uses each previous mask to focus finer-scale predictions.Prediction continues until the finest-scale mask becomes the final output.
  • E. Visualization of Predicted Manipulation Masks on Different Scales: Localization performance gradually improves from Mask 4 to Mask 1, with lower false alarms and clearer boundaries.
  • F. Runtime Analysis and Ablation Study: 0.012s runtime for Mask 4 provides approximately 37% additional saving relative to the full model’s 0.019s processing time for one 1,080P image.Earlier termination does not interfere with predicting the mask at the selected scale.
  • F. Runtime Analysis and Ablation Study: Mask 3 is a practical stopping point because it already performs well under AUC and F1 scores.
  • F. Runtime Analysis and Ablation Study: Spatial and channel-wise attention both outperform the no-attention baseline, while spatial attention contributes more performance gain than channel attention.Feature sharing also improves SCCM results while slightly reducing runtime.

G. Robustness Analysis

PSCC-Net is evaluated under image distortions and on real-life manipulated images. It remains robust across tested distortions, but localization can still miss forged regions when manipulation traces are difficult to recover.

  • G. Robustness Analysis: PSCC-Net is more robust than ManTra-Net and SPAN for localization under resizing, Gaussian blur, Gaussian noise, JPEG compression, and mixed distortions.The analysis reports pixel-level AUC using pre-trained models.
  • G. Robustness Analysis: Resizing has the least impact on PSCC-Net among the compared methods, consistent with resampling manipulation features into fixed sizes.
  • G. Robustness Analysis: PSCC-Net is quite robust for image-level detection under distortions on CASIA-D, especially JPEG compression.
  • H. Limitations: Real-life forged regions can have diverse sizes and shapes, making pixel-level localization more challenging than image-level detection.
  • H. Limitations: PSCC-Net may fail to localize all or part of forged regions when fabricators have elaborately removed manipulation traces.The compared IMDL methods also have difficulty with these images, while PSCC-Net can still perform relatively better in some cases.
  • V. CONCLUSION: PSCC-Net outperforms state-of-the-art methods on both manipulation detection and localization in extensive experiments.
Loading 2103.10596v2…