TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization

• Deep learning and diffusion models make realistic image manipulation easier, including generated people and context-adapted edits.Source: §1 · p. 1
• Current state-of-the-art detectors struggle with out-of-distribution manipulations, laundering, and insufficient detection performance.Source: §1 · p. 2
• False alarms are especially problematic because manipulated images are rare and false positives can greatly outnumber true positives.Source: §1 · p. 2
• TruFor first determines whether an image is manipulated, then localizes forgeries in detected images using low- and high-level features.Source: §1 · p. 2
• TruFor outputs a global integrity score, anomaly-based localization map, and associated confidence map, with experiments reporting state-of-the-art detection and localization.Source: §1 · p. 2

• Low-level forensic traces include artifacts from sensors, lenses, color filter arrays, and JPEG quantization, often enhanced with high-pass filtering or denoising.Source: §2 · p. 2
• Existing methods fuse noise and RGB information through bilinear pooling, late fusion, early fusion, or middle fusion approaches.Source: §2 · p. 2
• Most state-of-the-art methods prioritize localization, deriving image-level detection from post-processed localization heatmaps.Source: §2 · p. 2
• TruFor explicitly designs a detection module that takes both anomaly-based and confidence maps as input.Source: §2 · pp. 2–3
• Confidence maps are motivated by external uncertainty quantification to identify potentially unreliable regions in anomaly localization heatmaps.Source: §2 · p. 3

Framework

• TruFor extracts a Noiseprint++ fingerprint from the RGB image, then jointly predicts anomaly and confidence maps from RGB and fingerprint features.The shared encoder produces dense features for separate anomaly and confidence decoders.Source: §3 · p. 3
• The anomaly map identifies suspected manipulated regions, while the confidence map helps distinguish valid forgery predictions from random pristine-image anomalies.For pristine images, anomaly maps should be discarded because they represent statistical anomalies rather than localized forgeries.Source: §3 · p. 3, §3.3 · p. 5

Noiseprint++

• Noiseprint++ extends noiseprint by capturing traces from both in-camera processing and out-camera editing history, improving robustness to image impairments.Its contrastive training uses patches sharing origin, position, and editing history as similar examples and introduces varied editing operations.Source: §3.1 · pp. 3–4
• The learned fingerprint enhances forensic artifacts, including JPEG 8×8 grid inconsistencies that can appear over forged regions.Examples show grid misalignment in spliced areas but not pristine areas, and stronger highlighting than standard residual filtering or Noiseprint.Source: §3.1 · p. 4, Figure 4 · p. 4, Figure 5 · p. 4

Anomaly localization map

• RGB and Noiseprint++ features are calibrated through cross-modal rectification and merged with cross-attention before decoding the anomaly map.The rectification filters noisy information from one modality using the other modality's features.Source: §3.2 · pp. 4–5

Confidence map and integrity score

• Confidence-weighted maximum, minimum, average, and mean-square anomaly statistics are combined with corresponding confidence statistics to predict the integrity score.The confidence weights de-emphasize pristine areas with unusual statistics, reducing their influence on global detection.Source: §3.3 · p. 5

Localization results

• TruFor achieves the best average pixel-level F1 and ranks first or second across all evaluated datasets.It also performs well on GAN-based OpenForensics and diffusion-based CocoGlide local manipulations, where many methods fail.Source: §4.2 · p. 6

Detection results

• TruFor is the best performer on most datasets and has the best average image-level AUC and balanced accuracy.Unlike many competitors, it maintains almost 80% accuracy in the challenging fixed-threshold setting.Source: §4.2 · p. 6

Robustness analysis

• TruFor consistently outperforms the top three competitors on images compressed and resized through Facebook and WhatsApp.The performance gap widens significantly relative to CAT-Net v2 and MVSS-Net, despite IF-OSN targeting social-network images.Source: §4.2 · pp. 6–7

Qualitative comparisons

• The confidence map helps avoid false positives from hot spots in erroneous anomaly localization maps during detection.Qualitative results present anomaly and confidence maps together for further forensic analysis.Source: §4.2 · p. 7, Figure 6 · p. 8

Ablation study

• Joint cross-entropy and dice losses become important on resized and compressed data, especially when combined with augmentation.On original data, F1 remains stable across the compared loss and augmentation settings.Source: Table 5 · p. 7, §4.3 · p. 8, Table 6 · p. 8
• The mean anomaly-map statistic is poor, while the maximum is nearly useless without calibration; TruFor’s feature vector provides a large accuracy advantage.The ablation compares image-level detection using the mean or maximum anomaly-map value against the proposed feature vector.Source: §4.3 · p. 8, Table 7 · p. 8

Conclusions

• TruFor combines a learned noise-sensitive fingerprint with a confidence map to support reliable detection and localization under challenging conditions.The fingerprint enhances in-camera and out-camera artifacts, while the confidence map indicates possible false alarms on pristine areas.Source: §5 · p. 8
• The method generalizes to unknown local manipulations, including recent DNN-based ones, and provides robust image-level detection results.The conclusion attributes reliable image-level detection to the introduction of the confidence map.Source: §5 · p. 8

Limitations and future work

• TruFor cannot detect fully generated images and currently trains its anomaly map and detection score in separate phases with full pixel-level supervision.Future work proposes end-to-end training with partial image-level supervision and evaluation on newer generative models for local edits.Source: §5 · p. 8

Supplementary Document

• The supplementary document covers implementation details, datasets, robustness and detection experiments, qualitative localization and confidence maps, and failure cases.It also notes that the code is publicly available.Source: Supplementary Document · p. 9

Architecture

• The anomaly localization network uses a transformer-based segmentation encoder with RGB and Noiseprint++ branches connected by CM-FRM blocks.Each branch contains four Transformer blocks, with a CM-FRM block between successive Transformer blocks.Source: A. Implementation details · p. 9
• CM-FRM combines semantic RGB and residual Noiseprint++ features through channel-wise and spatial-wise rectifications.Its weights are computed separately along channel and spatial dimensions, while the Feature Fusion Module uses cross-attention to merge feature maps.Source: A. Implementation details · p. 9

Computational profile

• TruFor contains 68.7M parameters and processes a 3.2-megapixel image in about 1.17 seconds during inference.The parameter count is lower than those reported for CAT-Net v2, MVSS-Net, and IF-OSN.Source: A. Implementation details · p. 9

Datasets

• CocoGlide supplies examples of real and manipulated images together with reference maps and the prompts driving synthetic generation.The dataset is used to illustrate diffusion-based synthetic manipulations.Source: Figure 7 · p. 9

Training setup

• Noiseprint++ training uses 64 × 64 patches from five camera models and editing histories involving resizing, compression, and contrast or brightness adjustments.Training runs for 50 epochs with 8,800 steps per epoch.Source: A. Implementation details · p. 9
• Localization and detection training uses pristine and fake images with reference maps, 512 × 512 crops, equal dataset sampling per epoch, and 100 training epochs.The equal sampling is used to avoid biases from unequal training-dataset sizes.Source: A. Implementation details · p. 9

• Noiseprint++ training uses unaltered images verified for consistent resolution, JPEG format, quantization, and metadata without editing software.Source: B. Datasets · p. 9
• Training and testing cover CASIA v2, FantasticReality, IMD2020, and manipulated-image data involving splicing and copy-move operations.Source: B. Datasets · p. 10
• OpenForensics and NIST16 are evaluated on test subsets of 2,000 and 160 images, respectively.Source: B. Datasets · p. 10

Robustness to degradations

• The CASIA v1 robustness analysis applies Gaussian blur, Gaussian noise, gamma correction, and JPEG compression at varying levels.Pixel-level F1 performance at the best threshold is reported.Source: C. Additional robustness analysis · p. 10, Figure 9 · p. 11

Social-media platforms

• Across Facebook, WhatsApp, Weibo, and WeChat, TruFor gains approximately 16%, 19%, 18%, and 18% over the second-best method on average.The exception is Columbia, where CAT-Net v2 performs better.Source: C. Additional robustness analysis · p. 10, Table 9 · p. 11

Comparison with prior work

• TruFor is competitive or better than ObjectFormer across various test datasets and outperforms it on average.The comparison uses pixel-level localization values reported in prior work because ObjectFormer’s trained model is unavailable.Source: C. Additional robustness analysis · p. 10, Table 10 · p. 11

Confidence-map ablation

• Confidence maps produce substantial improvements in image-level AUC and accuracy in the ablation study.The comparison evaluates detection with and without the confidence map.Source: D. Additional detection results · p. 11

Threshold analysis

• Image-level accuracy is averaged over seven datasets using a fixed detection threshold of 0.5.Figure 10 also examines accuracy as a function of the detection-score threshold.Source: D. Additional detection results · p. 11

Threshold trade-offs

• A fixed threshold reduces false alarms by around 80% and increases missed detections by around 30%, while improving average accuracy by 25%.The comparison concerns TruFor’s fixed-threshold results against its best-threshold setting.Source: D. Additional detection results · p. 11
• State-of-the-art methods require best thresholds close to 1 and still exhibit high false-alarm rates in the seven-dataset average.Source: D. Additional detection results · p. 11

Confidence analysis

• Confidence maps can correct false-positive predictions and produce more reliable image-level integrity scores on fake and pristine images.Source: E. Qualitative results · p. 11, Figure 11 · p. 12

Failure cases

• Uniform and dark textures can mislead the confidence extractor, causing wrongly flagged false alarms or incorrect suppression of correctly localized forgeries.Source: E. Qualitative results · p. 11, Figure 12 · p. 12

Localization quality

• Some localized forgeries appear sharper and more accurate than those produced by other methods in qualitative comparisons.Source: E. Qualitative results · p. 12

Confidence visualization

• Dark boundary regions indicate lower confidence in assigning anomaly labels to intermediate regions around real forgeries.Source: E. Qualitative results · p. 12

False alarms

• On pristine images, TruFor’s weaker responses and confidence filtering discard many false alarms caused by semantically relevant or highly saturated regions.Source: E. Qualitative results · p. 12