PaperlayerView exact revision on arXiv

Source-linked AI summary

TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization

Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, Luisa Verdoliva

arXiv:2212.10957v3cs.CV

TL;DR

Image forgery detectors must handle diverse manipulations, post-processing, and rare manipulated images without excessive false alarms. TruFor combines RGB content with a self-supervised noise-sensitive fingerprint, anomaly localization, and confidence-aware image-level detection. Experiments report good generalizability, including localization of unknown DNN-based manipulations, and reliable image-level detection.

  • Problem

    Current detectors have limited generalization, robustness, and detection performance across out-of-distribution manipulations and image laundering, producing poor accuracy and high false-alarm rates.

  • Method

    TruFor combines RGB and Noiseprint++ traces to detect anomalies, localize forgeries, estimate confidence, and compute a global integrity score.

  • Results

    TruFor reports good generalizability, localization of unknown DNN-based manipulations, and reliable and robust image-level detection using its confidence map.

  • Takeaways & Limitations

    Confidence-aware anomaly analysis supports more reliable decisions for both image-level detection and pixel-level forgery localization.

  • Takeaways & Limitations

    The underlying Noiseprint approach has limited robustness to out-camera impairments, motivating Noiseprint++ to capture both in-camera and out-camera traces.

Abstract

from arXiv · show

In this paper we present TruFor, a forensic framework that can be applied to a large variety of image manipulation methods, from classic cheapfakes to more recent manipulations based on deep learning. We rely on the extraction of both high-level and low-level traces through a transformer-based fusion architecture that combines the RGB image and a learned noise-sensitive fingerprint. The latter learns to embed the artifacts related to the camera internal and external processing by training only on real data in a self-supervised manner. Forgeries are detected as deviations from the expected regular pattern that characterizes each pristine image. Looking for anomalies makes the approach able to robustly detect a variety of local manipulations, ensuring generalization. In addition to a pixel-level localization map and a whole-image integrity score, our approach outputs a reliability map that highlights areas where localization predictions may be error-prone. This is particularly important in forensic applications in order to reduce false alarms and allow for a large scale analysis. Extensive experiments on several datasets show that our method is able to reliably detect and localize both cheapfakes and deepfakes manipulations outperforming state-of-the-art works. Code is publicly available at https://grip-unina.github.io/TruFor/

1. Introduction

Modern editing tools enable cheapfakes and deepfakes, while current detectors remain insufficiently generalizable, robust, and accurate for in-the-wild use. TruFor addresses these challenges by combining multiscale evidence with confidence-aware detection and localization.

  • Deep learning and diffusion models make realistic image manipulation easier, including generated people and context-adapted edits.
  • Current state-of-the-art detectors struggle with out-of-distribution manipulations, laundering, and insufficient detection performance.
  • False alarms are especially problematic because manipulated images are rare and false positives can greatly outnumber true positives.
  • TruFor first determines whether an image is manipulated, then localizes forgeries in detected images using low- and high-level features.
  • TruFor outputs a global integrity score, anomaly-based localization map, and associated confidence map, with experiments reporting state-of-the-art detection and localization.

2. Related Work

Prior forensic methods combine low-level traces with high-level image features, but detection is often secondary to localization. TruFor explicitly uses anomaly and confidence maps for image-level decisions intended to reduce false alarms.

  • Low-level forensic traces include artifacts from sensors, lenses, color filter arrays, and JPEG quantization, often enhanced with high-pass filtering or denoising.
  • Existing methods fuse noise and RGB information through bilinear pooling, late fusion, early fusion, or middle fusion approaches.
  • Most state-of-the-art methods prioritize localization, deriving image-level detection from post-processed localization heatmaps.
  • TruFor explicitly designs a detection module that takes both anomaly-based and confidence maps as input.
  • Confidence maps are motivated by external uncertainty quantification to identify potentially unreliable regions in anomaly localization heatmaps.

3. Method

TruFor combines RGB semantics with a learned Noiseprint++ fingerprint to detect anomalies, localize suspected forgeries, estimate confidence, and derive a global integrity score. Noiseprint++ is trained on pristine images to capture camera and editing-history traces, while cross-modal fusion supports localization and confidence-weighted detection.

  • Framework: TruFor extracts a Noiseprint++ fingerprint from the RGB image, then jointly predicts anomaly and confidence maps from RGB and fingerprint features.The shared encoder produces dense features for separate anomaly and confidence decoders.
  • Framework: The anomaly map identifies suspected manipulated regions, while the confidence map helps distinguish valid forgery predictions from random pristine-image anomalies.For pristine images, anomaly maps should be discarded because they represent statistical anomalies rather than localized forgeries.
  • Noiseprint++: Noiseprint++ extends noiseprint by capturing traces from both in-camera processing and out-camera editing history, improving robustness to image impairments.Its contrastive training uses patches sharing origin, position, and editing history as similar examples and introduces varied editing operations.
  • Noiseprint++: The learned fingerprint enhances forensic artifacts, including JPEG 8×8 grid inconsistencies that can appear over forged regions.Examples show grid misalignment in spliced areas but not pristine areas, and stronger highlighting than standard residual filtering or Noiseprint.
  • Anomaly localization map: RGB and Noiseprint++ features are calibrated through cross-modal rectification and merged with cross-attention before decoding the anomaly map.The rectification filters noisy information from one modality using the other modality's features.
  • Confidence map and integrity score: Confidence-weighted maximum, minimum, average, and mean-square anomaly statistics are combined with corresponding confidence statistics to predict the integrity score.The confidence weights de-emphasize pristine areas with unusual statistics, reducing their influence on global detection.

4. Results

TruFor is evaluated across cheapfake, GAN-based, diffusion-based, and socially recompressed images, with strong localization, detection, and robustness results. Ablations show that confidence calibration and joint loss design materially support reliable image-level decisions.

  • Localization results: TruFor achieves the best average pixel-level F1 and ranks first or second across all evaluated datasets.It also performs well on GAN-based OpenForensics and diffusion-based CocoGlide local manipulations, where many methods fail.
  • Detection results: TruFor is the best performer on most datasets and has the best average image-level AUC and balanced accuracy.Unlike many competitors, it maintains almost 80% accuracy in the challenging fixed-threshold setting.
  • Robustness analysis: TruFor consistently outperforms the top three competitors on images compressed and resized through Facebook and WhatsApp.The performance gap widens significantly relative to CAT-Net v2 and MVSS-Net, despite IF-OSN targeting social-network images.
  • Qualitative comparisons: The confidence map helps avoid false positives from hot spots in erroneous anomaly localization maps during detection.Qualitative results present anomaly and confidence maps together for further forensic analysis.
  • Ablation study: Joint cross-entropy and dice losses become important on resized and compressed data, especially when combined with augmentation.On original data, F1 remains stable across the compared loss and augmentation settings.
  • Ablation study: The mean anomaly-map statistic is poor, while the maximum is nearly useless without calibration; TruFor’s feature vector provides a large accuracy advantage.The ablation compares image-level detection using the mean or maximum anomaly-map value against the proposed feature vector.

5. Conclusions

The conclusion presents TruFor as a reliable framework for image forgery detection and localization across challenging and unknown local manipulations. It also identifies fully generated images, separate training phases, and dependence on pixel-level supervision as limitations.

  • Conclusions: TruFor combines a learned noise-sensitive fingerprint with a confidence map to support reliable detection and localization under challenging conditions.The fingerprint enhances in-camera and out-camera artifacts, while the confidence map indicates possible false alarms on pristine areas.
  • Conclusions: The method generalizes to unknown local manipulations, including recent DNN-based ones, and provides robust image-level detection results.The conclusion attributes reliable image-level detection to the introduction of the confidence map.
  • Limitations and future work: TruFor cannot detect fully generated images and currently trains its anomaly map and detection score in separate phases with full pixel-level supervision.Future work proposes end-to-end training with partial image-level supervision and evaluation on newer generative models for local edits.

Supplementary Document

The supplementary document details the approach and datasets, adds robustness and detection results, presents qualitative maps, and reports failure cases.

  • Supplementary Document: The supplementary document covers implementation details, datasets, robustness and detection experiments, qualitative localization and confidence maps, and failure cases.It also notes that the code is publicly available.

A. Implementation details

The implementation uses a transformer-based architecture that fuses RGB and Noiseprint++ features, with separate training configurations for fingerprint, localization, and detection.

  • Architecture: The anomaly localization network uses a transformer-based segmentation encoder with RGB and Noiseprint++ branches connected by CM-FRM blocks.Each branch contains four Transformer blocks, with a CM-FRM block between successive Transformer blocks.
  • Architecture: CM-FRM combines semantic RGB and residual Noiseprint++ features through channel-wise and spatial-wise rectifications.Its weights are computed separately along channel and spatial dimensions, while the Feature Fusion Module uses cross-attention to merge feature maps.
  • Computational profile: TruFor contains 68.7M parameters and processes a 3.2-megapixel image in about 1.17 seconds during inference.The parameter count is lower than those reported for CAT-Net v2, MVSS-Net, and IF-OSN.
  • Datasets: CocoGlide supplies examples of real and manipulated images together with reference maps and the prompts driving synthetic generation.The dataset is used to illustrate diffusion-based synthetic manipulations.
  • Training setup: Noiseprint++ training uses 64 × 64 patches from five camera models and editing histories involving resizing, compression, and contrast or brightness adjustments.Training runs for 50 epochs with 8,800 steps per epoch.
  • Training setup: Localization and detection training uses pristine and fake images with reference maps, 512 × 512 crops, equal dataset sampling per epoch, and 100 training epochs.The equal sampling is used to avoid biases from unequal training-dataset sizes.

B. Datasets

The paper uses real-image controls for Noiseprint++ and evaluates TruFor across multiple manipulation datasets, including splicing and copy-move cases.

  • Noiseprint++ training uses unaltered images verified for consistent resolution, JPEG format, quantization, and metadata without editing software.
  • Training and testing cover CASIA v2, FantasticReality, IMD2020, and manipulated-image data involving splicing and copy-move operations.
  • OpenForensics and NIST16 are evaluated on test subsets of 2,000 and 160 images, respectively.

C. Additional robustness analysis

Additional experiments examine robustness under image degradation, social-media processing, and comparisons with prior localization methods.

  • Robustness to degradations: The CASIA v1 robustness analysis applies Gaussian blur, Gaussian noise, gamma correction, and JPEG compression at varying levels.Pixel-level F1 performance at the best threshold is reported.
  • Social-media platforms: Across Facebook, WhatsApp, Weibo, and WeChat, TruFor gains approximately 16%, 19%, 18%, and 18% over the second-best method on average.The exception is Columbia, where CAT-Net v2 performs better.
  • Comparison with prior work: TruFor is competitive or better than ObjectFormer across various test datasets and outperforms it on average.The comparison uses pixel-level localization values reported in prior work because ObjectFormer’s trained model is unavailable.

D. Additional detection results

Additional detection analyses show that confidence maps improve image-level decisions and help trade off false alarms against missed detections.

  • Confidence-map ablation: Confidence maps produce substantial improvements in image-level AUC and accuracy in the ablation study.The comparison evaluates detection with and without the confidence map.
  • Threshold analysis: Image-level accuracy is averaged over seven datasets using a fixed detection threshold of 0.5.Figure 10 also examines accuracy as a function of the detection-score threshold.
  • Threshold trade-offs: A fixed threshold reduces false alarms by around 80% and increases missed detections by around 30%, while improving average accuracy by 25%.The comparison concerns TruFor’s fixed-threshold results against its best-threshold setting.
  • Threshold trade-offs: State-of-the-art methods require best thresholds close to 1 and still exhibit high false-alarm rates in the seven-dataset average.

E. Qualitative results

Qualitative examples illustrate TruFor’s localization, confidence analysis, and integrity scoring, while also exposing failure cases on uniform or misleading image regions.

  • Confidence analysis: Confidence maps can correct false-positive predictions and produce more reliable image-level integrity scores on fake and pristine images.
  • Failure cases: Uniform and dark textures can mislead the confidence extractor, causing wrongly flagged false alarms or incorrect suppression of correctly localized forgeries.
  • Localization quality: Some localized forgeries appear sharper and more accurate than those produced by other methods in qualitative comparisons.
  • Confidence visualization: Dark boundary regions indicate lower confidence in assigning anomaly labels to intermediate regions around real forgeries.
  • False alarms: On pristine images, TruFor’s weaker responses and confidence filtering discard many false alarms caused by semantically relevant or highly saturated regions.
Loading 2212.10957v3…