PaperlayerView exact revision on arXiv

Source-linked AI summary

MVSS-Net: Multi-View Multi-Scale Supervised Networks for Image Manipulation Detection

Chengbo Dong, Xinru Chen, Ruohan Hu, Juan Cao, Xirong Li

arXiv:2112.08935v3cs.CVcs.AI

TL;DR

Image manipulation detectors need generalizable features that remain sensitive to tampering while avoiding false alarms on authentic images, particularly across independent datasets and degraded inputs. MVSS-Net jointly learns boundary-artifact and noise-view features with pixel, edge, and image supervision, and MVSS-Net++ replaces global max pooling with ConvGeM for image-level prediction. MVSS-Net++ performs best in within-dataset and cross-dataset evaluations and is more robust to JPEG compression, Gaussian blur, and screenshot-based re-capturing.

  • Problem

    Existing detectors often generalize poorly beyond aligned training and test data, while authentic-image scarcity leaves image-level specificity and false-alarm behavior insufficiently evaluated.

  • Method

    MVSS-Net jointly learns semantic-agnostic boundary-artifact and noise-view features using multi-scale pixel, edge, and image supervision; MVSS-Net++ uses ConvGeM for image-level prediction.

  • Results

    MVSS-Net++ performs best across within-dataset and cross-dataset scenarios and shows better robustness to JPEG compression, Gaussian blur, and screenshot-based image re-capturing.

  • Takeaways & Limitations

    Joint multi-view learning and multi-scale supervision provide a model that balances manipulation sensitivity with specificity across diverse evaluation conditions.

  • Takeaways & Limitations

    The authors identify future scope for adding components covering compression artifacts and modalities such as associated text.

Abstract

from arXiv · show

As manipulating images by copy-move, splicing and/or inpainting may lead to misinterpretation of the visual content, detecting these sorts of manipulations is crucial for media forensics. Given the variety of possible attacks on the content, devising a generic method is nontrivial. Current deep learning based methods are promising when training and test data are well aligned, but perform poorly on independent tests. Moreover, due to the absence of authentic test images, their image-level detection specificity is in doubt. The key question is how to design and train a deep neural network capable of learning generalizable features sensitive to manipulations in novel data, whilst specific to prevent false alarms on the authentic. We propose multi-view feature learning to jointly exploit tampering boundary artifacts and the noise view of the input image. As both clues are meant to be semantic-agnostic, the learned features are thus generalizable. For effectively learning from authentic images, we train with multi-scale (pixel / edge / image) supervision. We term the new network MVSS-Net and its enhanced version MVSS-Net++. Experiments are conducted in both within-dataset and cross-dataset scenarios, showing that MVSS-Net++ performs the best, and exhibits better robustness against JPEG compression, Gaussian blur and screenshot based image re-capturing.

1 INTRODUCTION

Image manipulation detection must identify copy-move, splicing, and inpainting while localizing tampered pixels, yet existing approaches struggle with diverse attacks, weakened forensic traces, poor cross-dataset generalization, and false alarms on authentic images. MVSS-Net addresses these challenges by jointly learning semantic-agnostic boundary and noise features with multi-scale supervision, while MVSS-Net++ improves image-level prediction through ConvGeM.

  • Motivation: Copy-move, splicing, and inpainting can be visually imperceptible and cause misinterpretation, motivating detection and pixel-level localization for media forensics.The task also supports trustworthy information sharing.
  • Challenges: Existing detectors struggle to generalize because semantic segmentation networks capture dataset-dependent content, especially when training and test data are not well aligned.Prior results show strong performance on homologous datasets but poor performance on non-homologous data.
  • Challenges: Authentic images are essential for evaluating image-level specificity, yet common protocols focus on manipulated images and pixel-level metrics, allowing serious false alarms.Pixel-wise segmentation losses give authentic examples only marginal influence during training.
  • Proposed approach: MVSS-Net jointly exploits RGB, explicitly extracted manipulation-boundary artifacts, and noise-view information to learn semantic-agnostic, more generalizable features.Its edge-supervised branch and noise-sensitive branch target complementary forensic clues.
  • Proposed approach: Multi-scale supervision uses pixel, edge, and image-level signals to learn effectively from authentic images and improve the balance between manipulation sensitivity and specificity.The edge and image labels are automatically extracted from pixel-level annotations, requiring no extra manual labeling.
  • Proposed approach: MVSS-Net++ replaces global max pooling with ConvGeM for image-level prediction, addressing gradient bottlenecks and sensitivity to the amount and spatial distribution of positive responses.The enhanced model is reported as superior to the state of the art across within-dataset and cross-dataset benchmarks.

2 RELATED WORK

Prior work learns semantic-agnostic manipulation cues through noise views, multi-view fusion, and edge artifacts, but commonly evaluates segmentation without authentic images. This motivates reporting detector specificity with authentic training and test images and a fixed threshold.

  • Noise-view feature learning: Recent methods target semantic-agnostic features for copy-move, splicing, and inpainting detection using noise views, constrained convolutions, or related representations.Examples include trainable high-pass filters, BayarConv, SRM noise maps, and modeled DCT distributions.
  • Multi-view feature fusion: Multi-view methods fuse RGB and noise representations through early or middle feature concatenation, while other approaches perform feature fusion within the network.Mantra-Net, SPAN, and JPEG-ComNet use early concatenation; CAT-Net uses middle-stage concatenation, and MM-Net uses another fusion strategy.
  • Edge-artifact modeling: Edge-aware methods exploit artifacts at tampered-region boundaries by jointly predicting regions and boundaries or refining edge features across network levels.MFCN symmetrically predicts the tampered area and its boundary, whereas GSR-Net uses an edge detection and refinement branch.
  • Specificity evaluation: Image manipulation detectors seldom report specificity because mainstream segmentation-based systems are trained and evaluated mainly on manipulated images.The absence of authentic images in training and testing raises concerns about responses to authentic content.
  • Specificity evaluation: The paper includes authentic images in training and testing and advocates selecting a default decision threshold of 0.5 instead of tuning it on test data.The authors present this evaluation setup as relevant to real-world deployment.

3 PROPOSED MODEL

MVSS-Net jointly learns semantic-agnostic manipulation features from RGB and noise views, while combining edge and holistic information through multi-scale supervision. Its ESB, NSB, trainable dual-attention fusion, and ConvGeM image-level prediction connect pixel evidence to image decisions.

  • MVSS-Net accepts RGB and noise views and jointly supervises pixel, edge, and image predictions to learn generalizable manipulation features.The network is designed to balance sensitivity to manipulations with specificity on authentic images.
  • 3.1.1 Edge-Supervised Branch: The edge-supervised branch progressively combines ResNet features, Sobel-enhanced edge patterns, and edge residual blocks to detect manipulation boundaries.Its output supplies feature maps for the main tasks and a predicted manipulation edge map.
  • 3.1.1 Edge-Supervised Branch: ESB produces more focused last-block responses near tampered regions than a standard ResNet without edge blocks or a GSR-like edge branch.The visualization covers inpainting, copy-move, and splicing examples.
  • 3.1.2 Noise-Sensitive Branch: The noise-sensitive branch processes a BayarConv-derived noise view with a second ResNet-50 backbone to capture noise inconsistency between tampered and authentic regions.BayarConv uses trainable prediction-error filters and is selected over SRM filtering.
  • 3.1.3 Branch Fusion by Dual Attention: A trainable Dual Attention module fuses ESB and NSB feature maps using parallel channel and position attention mechanisms.The fused representation is transformed and upsampled into the full-size segmentation map.
  • 3.2 ConvGeM for Image-Level Prediction: ConvGeM converts the pixel-level segmentation map into an image-level prediction by combining GeM and GeM(Conv) with a dynamically decayed weight λ.Compared with GMP, GeM allows more pixels to contribute to backpropagation and remains sensitive to neither the spatial distribution nor amount of positive responses without the Conv component.
  • 3.3 Multi-Scale Supervision: The edge loss uses Dice loss at reduced spatial resolution because edge pixels are overwhelmed by nonedge pixels.This treats manipulation edge detection as an auxiliary task rather than computing its loss at full image size.

4.1 Experimental Setup

The experiments use carefully separated training and test datasets, evaluate both pixel- and image-level detection, and include metrics designed to expose false alarms and threshold effects.

  • Datasets: DEF-84k and DEF-12k support ablation experiments, while CASIAv2 training and six test sets support state-of-the-art comparison.DEF-84k combines 64k manipulated and 20k authentic images; DEF-12k contains 6k of each, with source-image leakage avoided.
  • Evaluation Criteria: Pixel-level evaluation reports precision, recall, and F1, while image-level evaluation reports sensitivity, specificity, F1, and AUC.Authentic images are used only for image-level evaluation.
  • Dataset Repair: CASIAv1+ replaces 782 images shared with CASIAv2 to remove overlap between training and testing sources.Replacement images are randomly sampled from Corel, the original CASIAv1 data source.
  • Evaluation Protocol: A fixed default threshold of 0.5 is proposed for realistic evaluation because selecting thresholds separately per test set produces overly optimistic estimates.The practical motivation is that an operating point must be pre-specified and fixed.
  • Evaluation Criteria: Com-F1 is the harmonic mean of pixel-level and image-level F1, reaching zero when either component is zero.This makes the metric sensitive to the weaker detection level rather than allowing one level to compensate fully for the other.

4.2 Ablation Study

The ablation study progressively evaluates semantic backbones, supervision, multi-view branches, fusion, and non-trainable blocks, showing that MVSS-Net++ best balances sensitivity and specificity.

  • Influence of the semantic segmentation backbone: Semantic segmentation backbones are suboptimal for manipulation detection, motivating the FCN-16 with dual attention used as the Seg baseline.The comparison includes UNet, DeepLabv3, and DeepLabv3+.
  • Influence of the image classification loss: Adding image classification loss increases specificity but decreases sensitivity, making the model more conservative and reducing predicted manipulated areas.The sensitivity loss is also reflected by lower pixel-level F1.
  • On Trainable Components: Noise and edge supervision each improve the Seg+Clf setup, supporting the usefulness of the two complementary branches.The NSB and ESB comparisons independently report better performance after adding each branch.
  • On Trainable Components: The proposed ESB outperforms the GSR-Net edge branch, including on copy-move detection where ESB scores 0.405 versus 0.363 for GSR-Net.The larger gap on copy-move detection further distinguishes the proposed edge branch from the prior alternative.
  • On Trainable Components: Dual-attention fusion outperforms model averaging of the noise and edge branches, indicating complementary information is better combined within the network.The full MVSS-Net setup performs best among the compared fusion strategies.
  • Influence of Gclf: ConvGeM improves Com-F1 from 64.3 to 66.3 by balancing pixel-level and image-level performance, unlike GeM’s sensitivity loss.GeM raises pixel-level F1 but lowers image-level sensitivity from 79.7 to 63.1.
  • On Non-trainable Blocks: Removing Sobel lowers copy-move performance from 0.405 to 0.382, although its benefit is established only for the ResNet-based architecture.Whether Sobel helps more sophisticated billion-parameter vision models remains open.
  • On Non-trainable Blocks: Median filtering residual blocks raise sensitivity from 74.8 to 81.3 but lower specificity from 85.7 to 79.9, trading pixel-level accuracy for image-level detection.The result suggests high-frequency noise is manipulation-sensitive but insufficiently specific.

4.3 Comparison with State-of-the-art

Across cross-dataset tests, fixed-threshold image-level evaluation, processing distortions, and screenshot recapturing, MVSS-Net++ is consistently strongest while remaining imperfect on difficult cases.

  • Pixel-Level Manipulation Detection: MVSS-Net++ has the best overall pixel-level cross-dataset performance and surpasses same-training-data baselines on almost all test sets.The comparison trains models on CASIAv2 and tests them on COVER, Columbia, NIST16, CASIAv1+, IMD, and DEF-12k.
  • Evaluation Protocol: Pixel-level F1 drops from 68.8 to 21.4 for SPAN and from 73.2 to 38.7 for MVSS-Net++ when moving from optimal to realistic threshold conditions.The gap motivates the proposed fixed-threshold evaluation protocol.
  • Robustness Evaluation: Gaussian blur harms detection more than JPEG compression, especially at kernel sizes of 17 × 17 and above, while MVSS-Net and MVSS-Net++ remain more robust than baselines.Large blur kernels also reduce pictorial readability.
  • Image-Level Manipulation Detection: Under fixed threshold 0.5, MVSS-Net++ achieves image-level F1 of 68.0 averaged over five test sets, ahead of CAT-Net at 51.7 and MVSS-Net at 51.2.Multi-scale supervision improves specificity by learning from authentic images, while AUC remains competitive across operating points.
  • Robustness Evaluation: Screenshot recapturing affects all models, but MVSS-Net++ remains best; file format matters more than screenshot tool among the tested factors.The evaluation covers five variants from Snip&Sketch, Chrome, and Snipaste in PNG or JPEG formats.
  • Inference Efficiency: The MVSS-Net series run at 16 to 20 FPS on the tested GPUs, supporting real-time application according to the evaluation.Measurements use an NVIDIA Tesla V100 and a GeForce RTX2080ti.
  • Failure Case Analysis: Failure cases include tiny darkening traces, translucent overlays blended into mist, and background manipulations missed despite foreground-background inconsistency detection.These examples show that difficult or visually blended traces remain unresolved.

5 CONCLUSIONS

MVSS-Net++ combines multi-view feature learning with multi-scale supervision to improve generalizable manipulation detection and specificity. It outperforms prior methods across within-dataset and cross-dataset settings and is more robust to common image degradations, while leaving room to incorporate additional forensic information and modalities.

  • 5 CONCLUSIONS: Multi-view feature learning and multi-scale supervision together produce MVSS-Net++, a new state-of-the-art for image manipulation detection.The approach addresses semantic-agnostic feature learning while avoiding the pixel-level performance loss associated with image classification supervision alone.
  • 5 CONCLUSIONS: MVSS-Net++ outperforms current methods in both within-dataset and cross-dataset scenarios.
  • 5 CONCLUSIONS: MVSS-Net++ exhibits better robustness against JPEG compression, Gaussian blur and screenshot-based image re-capturing.
  • 5 CONCLUSIONS: Future network designs could incorporate additional information, such as compression artifacts, and modalities, such as associated text, for media forensics.

APPENDIX

The appendix reports additional detection measures and implementation details for the MVSS-Net series. It emphasizes balanced MCC performance and describes adding non-trainable MFR blocks to NSB in a shallow-to-deep arrangement.

  • APPENDIX: The MVSS-Net series clearly outperforms the baselines in terms of the well-balanced MCC.
  • APPENDIX: Table 9 measures detection performance using accuracy and MCC.
  • APPENDIX: Figure 12 shows non-trainable MFR blocks added to NSB in a shallow-to-deep manner similar to ESB.NRB is implemented in the same manner as ERB.
Loading 2112.08935v3…