PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Adversarial Training incorporating Forgery Attention for Image Forgery Localization

Long Zhuo, Shunquan Tan, Bin Li, Jiwu Huang

arXiv:2107.02434v2cs.CVcs.MM

TL;DR

To localize forged regions amid limited labeled training data, the paper combines a coarse-to-fine network with CW-HPF-based forgery attention and self-adversarial training. It reports outperforming state-of-the-art solutions on de-facto benchmarks, including a fine-tuned F1 advantage over SPAN on NIST.

  • Problem

    Forgery localization faces ultra-limited training samples, while ordinary augmentation still yields limited additional data.

    Source: §I · p. 2, E. Self-Adversarial Training · p. 6

  • Method

    A coarse-to-fine localization network combines CW-HPF and spatial/channel forgery attention with SAT, which dynamically generates adversarial training examples.

    Source: §I · p. 2, §IV · p. 14

  • Results

    The approach outperforms other state-of-the-art solutions on de-facto benchmark datasets; after fine-tuning, it beats SPAN by 0.296 in F1 score on NIST dataset.

    Source: C. Quantitative Results Compared… · p. 13, §IV · p. 14

  • Takeaways & Limitations

    With 75 forged images from COVERAGE, the model shows superior performance to state-of-the-art approaches, indicating it does not rely on large-scale training data for decent performance.

    Source: C. Quantitative Results Compared… · p. 13

Abstract

from arXiv · show

Image editing techniques enable people to modify the content of an image without leaving visual traces and thus may cause serious security risks. Hence the detection and localization of these forgeries become quite necessary and challenging. Furthermore, unlike other tasks with extensive data, there is usually a lack of annotated forged images for training due to annotation difficulties. In this paper, we propose a self-adversarial training strategy and a reliable coarse-to-fine network that utilizes a self-attention mechanism to localize forged regions in forgery images. The self-attention module is based on a Channel-Wise High Pass Filter block (CW-HPF). CW-HPF leverages inter-channel relationships of features and extracts noise features by high pass filters. Based on the CW-HPF, a self-attention mechanism, called forgery attention, is proposed to capture rich contextual dependencies of intrinsic inconsistency extracted from tampered regions. Specifically, we append two types of attention modules on top of CW-HPF respectively to model internal interdependencies in spatial dimension and external dependencies among channels. We exploit a coarse-to-fine network to enhance the noise inconsistency between original and tampered regions. More importantly, to address the issue of insufficient training data, we design a self-adversarial training strategy that expands training data dynamically to achieve more robust performance. Specifically, in each training iteration, we perform adversarial attacks against our network to generate adversarial examples and train our model on them. Extensive experimental results demonstrate that our proposed algorithm steadily outperforms state-of-the-art methods by a clear margin in different benchmark datasets.

I. INTRODUCTION

The introduction frames content-changing forgery localization as difficult and data-constrained, then proposes a coarse-to-fine framework with forgery attention and self-adversarial training. It reports state-of-the-art results on primary benchmark datasets under a content-changed manipulation assumption.

I. INTRODUCTION

  • Image manipulation is inexpensive and often visually difficult to detect, while undetected forgeries are associated with harms including rumors, fraud, fake news, and academic cheating.The introduction cites an estimate of 3.8% of 20,621 biomedical research papers containing problematic figures with potential deliberate manipulation.Source: §I · p. 1
  • Forgery localization targets pixel-level detection of semantic changes such as splicing, copy-move, and removal, while excluding edits such as blur and JPEG compression.Earlier methods often focused on image-level classification or one manipulation type, and pixel-level localization remains challenging when tampering leaves few visual traces.Source: §I · p. 1
  • Earlier deep-learning localization methods include LSTM-based patch comparison and encoder-decoder approaches, while RGB-N is limited to rectangular tampered-region boxes.Mantra-Net and RGB-N also use high-pass filters to discover noise inconsistency between authentic and tampered regions.Source: §I · p. 2
  • Prior attention methods borrow object-detection components, which may overlook tampered regions that are not salient objects.The introduction argues for attention mechanisms adapted specifically to forgery localization.Source: §I · p. 2
  • With ultra-limited training samples, ordinary flipping and rotation still provide too few examples, motivating a strategy that expands training data dynamically.SAT generates adversarial examples during each training iteration.Source: §I · p. 2
  • The proposed framework combines CW-HPF, spatial- and channel-dependency forgery attention, a coarse-to-fine architecture, and self-adversarial training.CW-HPF extracts noise features and uses inter-channel relationships; forgery attention fuses dependencies across spatial and channel dimensions.Source: §I · pp. 2–3
  • The approach achieves state-of-the-art results on all primary benchmark datasets, assuming localization targets content-changed regions.The stated scope includes splicing, copy-move, and removal, not non-content-changing edits.Source: §I · pp. 1–2

II. PROPOSED APPROACH

The proposed approach predicts a coarse mask and refines it using features passed from the coarse network. Its CW-HPF block applies high-pass filters channel-wise to generate noise features and can be placed at different positions or branches.

A. Overview

  • The overview uses a coarse-to-fine network with a coarse mask and refined mask, and trains first on original pairs then on adversarial examples.The refined mask is the final output; the coarse network passes its complete feature map to the refined network rather than only its binary mask.Source: A. Overview · p. 3
  • The refined network attaches forgery attention, while the coarse network includes CW-HPF, VGG-style blocks, and dilated convolutions.The two subnetworks produce the coarse and refined predictions, respectively.Source: A. Overview · p. 3

B. CW-HPF Block

  • CW-HPF applies three SRM-initialized high-pass filters to each input channel and concatenates the outputs into a noise feature map of size H × W × 3C.The block leverages inter-channel relationships and can accept arbitrary input channels and occupy arbitrary positions or branches in the framework; its filters are handcrafted and unlearnable.Source: B. CW-HPF Block · p. 3

C. Forgery Attention

Forgery attention adapts attention to forgery traces by combining noise features and dependencies across spatial positions and channels. Its two CW-HPF-based branches produce features that are fused for refined localization.

C. Forgery Attention

  • Forgery attention targets tampered traces rather than salient objects, combining multimedia-security domain knowledge with spatial and channel-wise dependencies.The paper identifies salient-object focus and ignored color-channel dependencies as limitations of conventional attention for forgery localization.Source: C. Forgery Attention · p. 4
  • Two parallel CW-HPF-based branches process refined-net local features to model spatial and channel relationships in noise features.The spatial branch relates positions; the channel branch relates channels.Source: C. Forgery Attention · pp. 4–5
  • The spatial branch weights noise features at each position using similarities between pairs of positions, then adds the scaled result to input features.Its Spatial Attention Matrix represents similarity between spatial positions, and γ is initialized to zero and learned during training.Source: C. Forgery Attention · pp. 4–5
  • The channel branch weights channel features using their relationships, adds a scaled result to the input, and is fused with the spatial branch to form Forgery Attention Features.The two branches use sigmoid activations, and their element-wise sum is passed through a convolution layer.Source: C. Forgery Attention · pp. 5–6

D. VGG Block and Dilated Convolutional Block

The network uses VGG-style blocks for encoding and decoding, with dilated convolutions connecting the two and enlarging receptive fields.

D. VGG Block and Dilated Convolutional Block

  • VGG blocks contain three or four stacked 3 × 3 convolution layers and perform encoding in the first three blocks and decoding in the next two.Their architecture originates from Mantra-Net.Source: D. VGG Block and Dilated Convolutional… · p. 6
  • Max pooling downsamples features after the first two VGG blocks, whose outputs are skip-concatenated with the corresponding decoder outputs; upsampling restores feature-map size.The skip connections pair the first and fourth blocks, and the second and fifth blocks.Source: D. VGG Block and Dilated Convolutional… · p. 6
  • Four dilated convolution layers bridge encoders and decoders in both subnetworks, using dilation rates 2, 4, 8, and 16 to obtain larger receptive fields.The paper motivates larger receptive fields to avoid learning from narrow local regions.Source: D. VGG Block and Dilated Convolutional… · p. 6

E. Self-Adversarial Training

Self-adversarial training addresses limited forgery-localization training data by generating adversarial examples from the updated model during each iteration. The network is trained on both the original and generated examples.

E. Self-Adversarial Training

  • Each iteration first trains the network on forged images and ground-truth masks, then generates FGSM adversarial images and trains the same network on them with the same masks.A random ϵ in (0, 0.01] is used in each iteration; FGSM is selected for its speed relative to BIM and MI-FGSM.Source: E. Self-Adversarial Training · pp. 6–7
  • Common flipping and rotation augmentation does not use feedback from the target model, while forgery localization has ultra-limited training samples.The paper motivates a strategy to increase sample numbers by one or two orders of magnitude.Source: E. Self-Adversarial Training · p. 6
  • The adversarial examples are generated from the latest model gradients, dynamically expanding training data from limited original samples.The paper characterizes the supply of such adversarial samples as endless and says they can make the model more robust.Source: E. Self-Adversarial Training · pp. 6–7
  • Before adversarial-example training, the model predicts an approximately precise mask for the original image but a wrong mask for its adversarial version.The figure caption notes that the residual map is enlarged 20 times for visualization.Source: Figure 6 · p. 7, E. Self-Adversarial Training · p. 7

F. Visualization Analysis

The visualization analysis compares activation maps from conventional high-pass filtering and attention with the proposed CW-HPF and forgery attention.

F. Visualization Analysis

  • Activation maps compare noise features from normal HPF and CW-HPF, then attention maps from a normal module and forgery attention; red denotes higher and blue lower activation.The maps are superimposed on a forged image, and CW-HPF uses inter-channel information unlike normal HPF.Source: F. Visualization Analysis · p. 7

1) CW-HPF:

CW-HPF uses inter-channel relationships to make tampered-region noise features more precise and consistent. Forgery attention then aggregates spatial and channel dependencies to emphasize tampering traces rather than salient objects.

1) CW-HPF:

  • CW-HPF makes the fake flower more prominent than pristine high-frequency regions, and its noise-feature variance is one-ninth that of HPF.The smaller variance indicates more consistent noise features in tampered regions.Source: 1) CW-HPF · pp. 7–8

2) Forgery Attention:

  • Forgery attention emphasizes tampered traces over salient objects by aggregating global contextual noise information through spatial and channel dependencies.Its spatial branch weights noise features across pixels, while its channel branch weights features across channels.Source: 2) Forgery Attention · p. 8, Figure 9 · p. 8

III. EXPERIMENTS

  • Experiments compare the approach on benchmark datasets and evaluate robustness under resizing, JPEG compression, and adversarial attacks.The reported experiments use real-world datasets and hardware rather than numerical simulations.Source: §III · p. 8

A. Setup

The setup uses several forgery datasets, including synthesized and real-image benchmarks, and reports pixel-level F1 and AUC. Training uses 512 × 512 inputs and Adam optimization.

1) Datasets:

  • DEFACTO supplies 98,779 synthesized tampered images for ablation and pre-training, with a 9:1 training-testing split; Columbia, CASIA, COVERAGE, NIST, and PS-dataset provide additional evaluation data.For NIST, COVERAGE, and CASIA, the experiments follow RGB-N's training-testing ratios.Source: 1) Datasets · p. 9

2) Implementation Details:

  • Training resizes images to 512 × 512 and optimizes the TensorFlow framework with Adam at a learning rate of 0.002.Experiments run on a single Tesla P100 GPU.Source: 2) Implementation Details · p. 9

3) Evaluation Metrics:

  • Pixel-level F1 and AUC evaluate binary classification at each pixel, with higher scores indicating better performance.The authors consider F1 more accurate than AUC because false positives lower F1 substantially while AUC can remain high.Source: 3) Evaluation Metrics · p. 9

4) Loss Function:

The ablations assess the loss, comparison methods, and components of the proposed framework. They report strong DEFACTO results for the full model and favor dual spatial-channel attention with CW-HPF.

4) Loss Function:

  • The model minimizes the sum of binary cross-entropy losses for the coarse and refined masks against the ground-truth mask.The losses are applied to yM1 and yM, respectively.Source: 4) Loss Function · p. 9

5) Compared Methods:

  • The comparison includes unsupervised feature-extraction methods and deep-learning methods, including LSTM, RGB-N, Mantra-Net, and SPAN.The cited deep-learning methods use convolutional neural networks.Source: 5) Compared Methods · p. 10

B. Ablation Study

  • The progressive ablation study evaluates CW-HPF, forgery attention, and the coarse-to-fine architecture as successive components of the proposed model.The full coarse-to-fine model uses CW-HPF in its coarse net and a forgery-attention model as its refined net.Source: B. Ablation Study · p. 10, 1) Progressive Ablation Study · p. 10

1) Progressive Ablation Study:

  • The coarse-to-fine forgery-attention model was augmented with SAT, flipping, and rotation; for optimal performance, it achieved 0.998 AUC and 0.929 F1.Source: 1) Progressive Ablation Study · p. 10

2) Ablation Study for Forgery Attention:

  • Combining CAM and PAM outperforms either alone, while Softmax substitution and dual attention without CW-HPF decrease or fail to improve performance.The ablation supports the proposed dual attention, Sigmoid activation, and CW-HPF combination for forgery localization.Source: 2) Ablation Study for Forgery Attention · p. 10, Table II · p. 10

3) Ablation Study for SAT:

SAT dynamically generates adversarial training examples and improves robustness to Gaussian-noise attack relative to flipping and rotation. CW-HPF ablations favor using the block in both stages of the coarse-to-fine network.

3) Ablation Study for SAT:

  • 8% AUC and 10% F1 losses under Gaussian-noise attack with SAT are smaller than the 17% AUC and 18% F1 losses with flipping and rotation.SAT also adds 0.004 in AUC and 0.004 in F1 over flipping and rotation before attack.Source: 3) Ablation Study for SAT · p. 11, Table III · p. 11

4) Ablation Study for CW-HPF:

  • The no-high-pass-filter model scores 0.804 in AUC and 0.721 in F1, while adding CW-HPF improves noise features in both coarse and refined nets.The authors attribute the benefit of high-pass filters to tampering traces in the high-frequency domain.Source: 4) Ablation Study for CW-HPF · p. 11, Table IV · p. 11
  • Two CW-HPF blocks achieve the best results, increasing detection scores by 0.008 in AUC and 0.035 in F1 over CW-HPF in the refined net alone.Introducing one HPF module in either net improves AUC by 0.160–0.166 and F1 by 0.123–0.129 over RGB + RGB; using HPF in both nets improves them by 0.174 and 0.135.Source: 4) Ablation Study for CW-HPF · p. 11, Table IV · p. 11

5) Ablation Study for Dilated Convolutional Module:

The ablations favor dilated convolution in both coarse and refined nets and passing coarse-net features, rather than its binary mask, to the refined net.

5) Ablation Study for Dilated Convolutional Module:

  • Dilated modules in both coarse and refined nets improve performance; a refined-net-only module raises AUC 0.001 and F1 0.003 versus w/o Dilated + w/o Dilated, but trails both-net dilation by 0.002 AUC and 0.007 F1.Source: 5) Ablation Study for Dilated… · p. 12

6) Ablation Study for Coarse-to-Fine Connection:

5) Ablation Study for Dilated Convolutional Module:

  • A coarse-net-only dilated module slightly lowers performance versus using no dilated convolutional module.The authors interpret the results as favoring richer features in the refined net and dilation in both nets.Source: 5) Ablation Study for Dilated… · p. 12

6) Ablation Study for Coarse-to-Fine Connection:

  • The coarse net’s binary mask lacks semantics for refinement, so Feature Connection passes its last deconvolutional-layer features to the refined net.The authors state that this allows the refined net to be optimized along with those features.Source: 6) Ablation Study for Coarse-to-Fine… · p. 12

7) Trade-off Experiments:

The framework is evaluated across benchmark comparisons, a filter-count trade-off, and computational costs. The reported comparisons favor the framework, including after fine-tuning against DFCN, while 32 basic filters offer a better trade-off.

7) Trade-off Experiments:

  • On DEFACTO, F1 rises significantly from 8 to 32 basic filters and then increases more slowly, making 32 the reported better trade-off.The authors note that doubling the number of basic filters doubles network parameters and quadruples calculation resources.Source: 7) Trade-off Experiments · p. 12

C. Quantitative Results Compared against State-of-the-art Methods

  • With DEFACTO training and benchmark fine-tuning, the model achieves state-of-the-art performance; versus SPAN, it gains 0.296 in F1 on NIST and 0.048 in AUC on COVERAGE.The comparison covers NIST, COVERAGE, Columbia, and CASIA.Source: Table VII · p. 13, C. Quantitative Results Compared… · p. 13
  • With only 75 forged COVERAGE images, the model outperforms state-of-the-art approaches, especially on Columbia, where its F1 score beats all other methods.The authors state that it achieves decent performance without large-scale training data.Source: C. Quantitative Results Compared… · p. 13
  • After fine-tuning, the framework beats DFCN on three datasets, with about 10% higher F1 on PS-boundary and about 6% higher AUC on NIST.The comparison follows DFCN’s 512 × 512-patch setting and uses 100 PS-arbitrary, 100 PS-boundary, and 56 NIST forged-image samples for fine-tuning.Source: Table VIII · p. 13, C. Quantitative Results Compared… · p. 13

D. Computational Complexity Analyses

  • The framework takes roughly 126 ms per image on a Tesla P100 and uses less training and inference time than Mantra-Net and SPAN despite more parameters and FLOPs.The authors attribute the time comparison to their fully end-to-end training and validation, versus extra off-model operations in the comparison methods.Source: Table IX · p. 14, D. Computational Complexity Analyses · p. 14

E. Qualitative Results

Qualitative examples span three manipulation types, and the authors report accurate localization across tampered objects and backgrounds. The paper also reports robustness to several content-preserving manipulations and identifies future extensions.

F. Robustness Experiments

  • The framework is reported as quite immune to resizing, blur, noise, JPEG compression, and FGSM attacks on NIST.FGSM epsilon is 0–0.01 during SAT training and 0.02 at testing.Source: F. Robustness Experiments · p. 14

IV. CONCLUSION

  • The conclusion presents forgery attention for spatial and channel noise-feature dependencies, SAT for dynamic augmentation, and a coarse-to-fine CW-HPF framework; video localization and deepfake detection remain future directions.The authors report that experiments on benchmark datasets outperform other state-of-the-art solutions by a clear margin.Source: §IV · p. 14

E. Qualitative Results

  • Across splicing, copy-move, and removal examples, the framework localizes tampering in objects and backgrounds, including regions indistinguishable to people.The selected examples come from NIST, Columbia, COVERAGE, CASIA, and DEFACTO.Source: E. Qualitative Results · p. 14, Figure 12 · p. 15

Related papers

Loading 2107.02434v2…