Source-linked AI summary
Image Manipulation Detection by Multi-View Multi-Scale Supervision
Xinru Chen, Chengbo Dong, Jiaqi Ji, Juan Cao, Xirong Li
TL;DR
Image manipulation detection needs features that remain sensitive to novel tampering while avoiding false alarms on authentic images, a gap overlooked by much prior work. MVSS-Net combines noise- and boundary-based multi-view feature learning with multi-scale supervision, and experiments on five benchmark sets support its pixel- and image-level detection viability.
Problem
Image manipulation detection must generalize to novel tampering while remaining specific on authentic images, but prior methods often overlook specificity and struggle to learn from authentic examples.
Method
MVSS-Net jointly exploits noise distributions and boundary artifacts through multi-view feature learning, with multi-scale supervision for manipulation and authenticity-related learning.
Results
Experiments on five benchmark sets show that MVSS-Net compares favorably with state-of-the-art methods for pixel-level and image-level manipulation detection.
Takeaways & Limitations
Noise and edge information support semantic-agnostic detection, while combining multi-view learning with multi-scale supervision addresses sensitivity and specificity together.
Abstract
from arXiv · showhide
The key challenge of image manipulation detection is how to learn generalizable features that are sensitive to manipulations in novel data, whilst specific to prevent false alarms on authentic images. Current research emphasizes the sensitivity, with the specificity overlooked. In this paper we address both aspects by multi-view feature learning and multi-scale supervision. By exploiting noise distribution and boundary artifact surrounding tampered regions, the former aims to learn semantic-agnostic and thus more generalizable features. The latter allows us to learn from authentic images which are nontrivial to be taken into account by current semantic segmentation network based methods. Our thoughts are realized by a new network which we term MVSS-Net. Extensive experiments on five benchmark sets justify the viability of MVSS-Net for both pixel-level and image-level manipulation detection.
1. Introduction
Image manipulation detection must identify tampered pixels while generalizing across manipulation types and avoiding false alarms on authentic images. MVSS-Net addresses these goals through semantic-agnostic multi-view features and multi-scale supervision.
- The task targets copy-move, splicing, and inpainting, requiring both image-level discrimination and pixel-level localization of tampered regions.
- Semantic segmentation networks can become dataset-dependent because they capture semantic information, motivating semantic-agnostic features that remain sensitive while limiting false alarms.
- Noise-view methods exploit noise-distribution discrepancies but are ineffective for copy-move, whereas edge-supervised methods detect boundary artifacts around tampered regions.
- Specificity is often ignored because pixel-wise segmentation losses give authentic examples marginal influence during training and evaluation commonly focuses on manipulated images.
- MVSS-Net jointly uses noise and boundary information with multi-scale supervision, enabling learning from authentic images and improving specificity while targeting generalizable features.
2. Related Work
Prior work learns manipulation traces from noise or boundary artifacts, while MVSS-Net addresses limitations in edge-feature integration and explicitly considers detector specificity. Its supervision uses automatically derived labels rather than extra manual annotation.
- The taxonomy notes that edge and image labels are automatically extracted from pixel-level annotations, so multi-scale supervision requires no extra manual annotation.
- Deep learning approaches for copy-move, splicing, and inpainting detection commonly learn semantic-agnostic features, while low-level manipulations such as blur and compression are treated separately.
- Noise-based methods use fixed or trainable high-pass filtering to extract noise information, but relying on noise alone risks losing useful RGB information.
- Boundary-based methods predict tampered regions and edges, but using deeper or concatenated features can respectively affect segmentation or allow semantic-aware features to dominate.
- MVSS-Net uses an edge-supervised branch to resolve the trade-off between manipulation segmentation and edge detection identified in prior approaches.
- Specificity is seldom reported because mainstream detectors are trained and evaluated on manipulated images without authentic images in either stage.
3. Proposed Model
MVSS-Net combines RGB/noise views with edge-aware processing and multi-scale supervision to detect manipulated pixels and images while learning from authentic examples.
- Overall architecture: MVSS-Net uses a multi-head network to classify whether an image is manipulated and localize its manipulated pixels.The image-level probability is obtained by global max pooling over the pixel-level segmentation map.
- Overall architecture: The network combines RGB and noise views, supervising pixel, edge, and image outputs to balance sensitivity with specificity.The three supervision scales target pixel-level detection, semantic-agnostic edge features, and image-level specificity.
- Edge-supervised branch: Its edge-supervised branch progressively combines ResNet features through Sobel layers and edge residual blocks to detect boundary artifacts.The branch produces feature maps for the main task and a predicted manipulation edge map.
- Noise-sensitive branch: Its noise-sensitive branch uses a parallel ResNet-50 FCN with BayarConv to capture inconsistencies between tampered and authentic regions.BayarConv is adopted for noise extraction in this branch.
- Branch fusion: A trainable Dual Attention module fuses ESB and NSB feature maps using parallel channel- and position-attention mechanisms.The fused representation is transformed into the final segmentation map through convolution, bilinear upsampling, and sigmoid activation.
- Multi-scale supervision: The combined objective uses pixel-scale Dice, edge Dice, and image-scale classification losses, with authentic images contributing to the classification loss.Edge loss is computed at reduced spatial resolution because edge pixels are overwhelmed by non-edge pixels, while image-scale supervision addresses false alarms.
4. Experiments
Experiments evaluate MVSS-Net through controlled ablations and cross-dataset comparisons, measuring both pixel-level sensitivity and image-level specificity. The full model performs best overall, with strong cross-dataset generalization and real-time speed.
- Experimental Setup: The experiments use DEFACTO-84k and DEFACTO-12k for ablation, while CASIAv2 training supports comparison across five test sets.The setup avoids source-image leakage between training and test splits.
- Evaluation Criteria: Com-F1 is the harmonic mean of pixel-level and image-level F1, and becomes zero when either component is zero.This metric emphasizes balanced performance across localization and image-level classification.
- Ablation Study: Adding classification loss increases specificity but reduces sensitivity, making Seg+Clf more conservative and shrinking predicted manipulated regions.The change is reflected both in lower pixel-level performance and qualitative predictions.
- Ablation Study: Adding NSB or ESB improves performance, while removing Sobel from ESB reduces copy-move performance from 0.405 to 0.382.The ablations support the contribution of both branches and the Sobel operation.
- Ablation Study: The full setup with dual-attention fusion performs best and exceeds model averaging of the NSB and ESB variants, indicating complementary components and beneficial fusion.The ablation table caption identifies the full setup as the top-performing configuration.
- Comparison with State-of-the-art: MVSS-Net is best overall for pixel-level and image-level detection, with higher specificity on most test sets, best AUC scores, and 20.1 FPS for real-time application.For inpainting subsets, it scores 0.565 versus 0.284 on NIST16 and 0.391 versus 0.106 on DEFACTO-12k against HP-FCN.
5. Conclusions
Experiments on five benchmark sets support MVSS-Net as a new state-of-the-art for image manipulation detection. The conclusions emphasize semantic-agnostic feature learning, specificity, and the need to combine multi-view learning with multi-scale supervision.
- MVSS-Net combines multi-view feature learning with multi-scale supervision and achieves new state-of-the-art performance for image manipulation detection.The conclusion states that multi-view feature learning must be used together with multi-scale supervision.
- Noise and edge information both help learn semantic-agnostic features, but edge information is more effective when used alone.
- State-of-the-art methods have poor specificity, while image classification loss improves specificity at the cost of pixel-level detection performance.