PaperlayerView exact revision on arXiv

Source-linked AI summary

Forensic-Aware Continual Adaptation for Image Forgery Localization

Chenqi Kong, Song Xia, Anwei Luo, Peisong He, Alex C. Kot, Yuming Fang

arXiv:2609.38251v1cs.CRcs.AI

TL;DR

Emerging forgeries and domain shifts challenge IFL, while large-model retraining is impractical and continual adaptation remains underexplored. The paper establishes two continual-learning benchmarks and proposes FOCAL, reporting state-of-the-art localization and detection across protocols.

  • Problem

    IFL models have limited generalizability to emerging forgeries, while sequential adaptation is underexplored despite the impracticality of repeatedly retraining large models.

    Source: §1 · p. 2

  • Method

    FOCAL combines SMoFE and FEGDP for forensic representation adaptation with FLAG surgery to regulate LoRA updates and preserve prior knowledge.

    Source: §1 · p. 4

  • Results

    FOCAL achieves state-of-the-art performance in pixel-level forgery localization and image-level forgery detection across two continual-learning protocols.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The work establishes cross-dataset and cross-content benchmarks for evaluating continual adaptation in image forgery localization.

    Source: §5 · p. 24

    The CIFL setting assumes forensic data from different domains arrive sequentially and evaluates each stage on previously seen and newly introduced test sets.

    Source: §1 · p. 3, §3.1 · p. 6

Abstract

from arXiv · show

The rapid evolution of image manipulation techniques has raised growing public security concerns. Existing Image Forgery Localization (IFL) methods can accurately localize manipulated regions but are often unable to adapt to newly emerging forgeries. In real-world forensic scenarios, data typically arrive sequentially, yet continual model adaptation remains largely unexplored in IFL. To bridge this gap, we introduce the first continual learning framework for IFL and establish a comprehensive benchmark under two realistic data-evolution protocols: cross-dataset and cross-content continual learning. Evaluations of representative state-of-the-art IFL and continual learning methods reveal substantial performance degradation, highlighting two key challenges: (1) adaptively capturing intrinsic forensic traces from incoming data across unseen domains, and (2) preserving previously acquired forensic knowledge during sequential adaptation. To address these challenges, we propose a forensic-aware continual adaptation framework. First, a forensic trace mining module employs Spatial Mixture-of-Forensic-Experts (SMoFE) to dynamically route complementary forensic cues across spatial locations, together with Forensic Evidence-Guided Dense Prompting (FEGDP) to transform low-level forensic traces into structured localization evidence for SAM. Second, Fisher-weighted LoRA Gradient (FLAG) surgery identifies old-task-sensitive adaptation directions and suppresses conflicting updates, mitigating catastrophic forgetting while preserving plasticity for emerging forgery domains. Extensive experiments demonstrate state-of-the-art performance in both pixel-level forgery localization and image-level forgery detection across diverse continual learning scenarios.

1 Introduction

The introduction frames continual adaptation as necessary because emerging forgeries and domain shifts challenge existing IFL models, while repeatedly retraining large models is impractical. It benchmarks existing approaches and introduces FOCAL to adapt forensic representations while preserving prior knowledge.

  • New forgeries can leave fewer detectable traces and arise in substantially different image domains, limiting existing IFL models’ generalizability.Source: §1 · p. 2
  • Repeatedly retraining large IFL models on prior datasets as new forgeries emerge is computationally expensive and impractical for deployment.Source: §1 · p. 2
  • Existing IFL and continual-learning methods suffer significant performance degradation under the cross-dataset and cross-content protocols.The protocols use datasets ordered by release date and natural, document, and scientific image content, respectively.Source: §1 · p. 3
  • FOCAL combines spatial forensic-trace routing and evidence-guided SAM prompting with Fisher-weighted LoRA gradient surgery to adapt while preserving prior knowledge.SMoFE routes complementary forensic cues; FEGDP turns low-level artifacts into structured localization evidence; FLAG suppresses updates conflicting with old-task-sensitive directions.Source: §1 · p. 4
  • FOCAL achieves state-of-the-art performance in pixel-level forgery localization and image-level forgery detection across two continual-learning protocols.Source: §1 · p. 4

2 RELATED WORKS

The related work traces IFL from handcrafted forensic cues to learned representations, while emphasizing persistent challenges from weak traces and domain gaps. It also situates continual IFL as an underexplored dense-prediction problem.

2.1 Image forgery localization

  • Handcrafted noise, compression, demosaicing, and lens-distortion cues have been used to reveal local tampering evidence.Source: §2.1 · p. 4
  • Deep models span RGB segmentation, residual-guided, frequency-domain, and multi-branch approaches, while transformers and pretrained vision backbones offer stronger representations.Robust localization remains difficult with weak heterogeneous traces and domain gaps, motivating explicit evidence modeling and adaptation that preserves prior knowledge.Source: §2.1 · p. 5

2.2 Continual learning in forensics

  • Continual forensic learning has addressed classification tasks, but continual IFL remains largely unexplored despite requiring dense pixel-level localization.Source: §2.2 · p. 5

3.1 Problem Formulation

The formulation treats image-forgery localization as a continual-learning problem in which forensic data from different domains arrive sequentially. Each task has training samples and associated pixel- and image-level forgery labels.

  • CIFL differs from conventional IFL by assuming domain-specific forensic data arrive sequentially rather than being available together from a stationary distribution.Source: §3.1 · p. 6
  • The formulation describes an ordered stream of Q tasks and identifies N_q as the number of training samples in the current task.Source: §3.1 · p. 6
  • Each image has a pixel-level manipulation mask and a corresponding image-level forgery label.Source: §3.1 · p. 6

3.2 Framework Overview

The framework freezes the pretrained image encoder except for its LoRA modules, mines forensic traces, and regulates updates to preserve previously learned knowledge.

  • The pretrained image encoder is frozen, with only its LoRA modules updated during training.Source: §3.2 · p. 6
  • Forensic Trace Mining combines SMoFE, which integrates local anomalies from different forensic maps, with FEGDP, which converts forensic knowledge into structured prompts for SAM.Source: §3.2 · p. 6
  • Fisher-guided gradient surgery regulates parameter updates by their importance to previously learned forensic knowledge to mitigate catastrophic forgetting.Source: §3.2 · p. 6

3.3 Forensic-Aware Representation Adaptation (FARA)

FARA adapts forensic representations by routing local traces across experts and converting the resulting evidence into dense prompts for SAM.

3.3.1 Spatial Mixture-of-Forensic-Experts (SMoFE)

  • SMoFE spatially routes among aligned forensic trace maps so different image locations can emphasize different experts for localized, heterogeneous artifacts.The experts capture complementary forensic cues, and the routing can suppress noisy or domain-irrelevant responses.Source: §3.3.1 · p. 8

3.3.2 Forensic Evidence-Guided Dense Prompting (FEGDP)

  • FEGDP combines multi-scale local inconsistency, boundaries estimated from forensic and visual cues, and router entropy to represent suspicious regions, their extent, and evidence ambiguity.Local-reference statistics emphasize relative anomalies; visual context helps distinguish manipulation boundaries from benign image edges.Source: §3.3.2 · pp. 9–10
  • FEGDP composes heterogeneous forensic signals into a SAM-compatible dense prompt that encodes suspicious regions, manipulation boundaries, and uncertainty without relying on manipulation-specific semantic categories.The prompt is intended to suit continually evolving forgery domains.Source: §3.3.2 · pp. 9–10

3.4 Fisher-Weighted LoRA Gradient Surgery (FLAG)

FLAG uses Fisher-weighted gradient surgery in the LoRA adaptation subspace to protect previously acquired forensic knowledge while retaining flexibility for new forgery patterns.

3.4 Fisher-Weighted LoRA Gradient Surgery (FLAG)

  • FLAG addresses domain-shift-related forgetting by weighting adaptation directions according to their importance to previous tasks, preserving old knowledge while retaining flexibility for emerging patterns.Different domains can exhibit substantially different forensic cues and image contents.Source: §3.4 · pp. 10–11, §3.4.2 · p. 12

3.4.1 Fisher-Guided Adaptation Importance Modeling

  • For each prior task, FLAG estimates diagonal Fisher importance from stored samples and averages task weights; each previous-task memory buffer contains 256 representative samples.The resulting weights distinguish LoRA dimensions sensitive to previous forensic domains from less relevant dimensions.Source: §3.4.1 · p. 11

3.4.2 Importance-Aware Gradient Surgery

  • FLAG measures current-versus-prior gradient conflict in Fisher-weighted geometry and removes only the interfering current-gradient component when conflict occurs.The projected gradient updates LoRA parameters, while other trainable components retain their ordinary current-task gradients.Source: §3.4.2 · pp. 11–12

3.5 Overall Objectives

The training objective combines mask, image-classification, boundary, and mixture-of-experts losses, with evidence-map statistics also used for image-level manipulation scoring.

3.5 Overall Objectives

  • The overall objective contains mask prediction, image classification, boundary prediction, and mixture-of-experts losses.The mask and classification losses use binary cross-entropy against the manipulation mask and image-level label, respectively.Source: §3.5 · p. 12
  • Image-level manipulation scoring aggregates global visual and forensic features with evidence-map statistics, including each map’s mean and top-k mean.The aggregate features are used to predict the image-level manipulation score.Source: §3.5 · p. 12
  • Boundary loss supervises the predicted boundary against its label, while an additional consistency component links boundary evidence to the final SAM mask prediction.The expert-balance loss prevents the Top-1 router from collapsing to a single forensic expert.Source: §3.5 · pp. 12–13

4 Experiments

The experiments assess continual image-forgery localization across sequential dataset and content shifts, using eight baselines, image- and pixel-level metrics, and a consistent SAM-based training setup.

4.1.1 Protocols and Datasets

  • Protocol 1 sequences four manipulation datasets by release date, while Protocol 2 tests natural, scientific, and document content in all six task orders.Protocol 2 is designed to represent larger domain gaps than Protocol 1.Source: §4.1.1 · p. 13

4.1.2 Baseline Methods

  • The benchmark incorporates eight state-of-the-art continual-learning and image-forgery-localization baselines, including prompt-based, modular-adapter, and SAM+LoRA methods.SAM-Base with LoRA is used as a baseline, with only its LoRA modules updated.Source: §4.1.2 · pp. 13–14

4.1.3 Evaluation Metrics

4.1.4 Implementation Details

  • Experiments use SAM ViT-B with rank-32 LoRA, 512 × 512 inputs, and 10 training epochs per continual-learning task.Training uses Adam with batch size 64 and validates after each epoch on all seen validation sets.Source: §4.1.4 · p. 16

4.2 Experimental Results under Protocol 1

FOCAL achieves the strongest reported continual performance across dataset and content-shift settings. Qualitative localization and feature visualizations also indicate adaptation to new tasks alongside retention of earlier-task representations.

4.2 Experimental Results under Protocol 1

  • FOCAL achieves the best average performance across Protocol 1 sequential tasks in pixel-level localization and image-level forgery detection.It has the highest average F1 and IoU, and the best average AUC and ACC.Source: §4.2 · pp. 16–17

4.3 Experimental Results under Protocol 2

  • Across all six Protocol 2 task orders, FOCAL has the largest radar coverage, while competing methods fluctuate across orders.The radar charts report average F1 in the top row and average IoU in the bottom row across T1–T3.Source: §4.3 · pp. 17, 19, Figure 3 · p. 17

4.4 Continual Forgery Localization Results

  • After sequential training through Task 4, FOCAL produces compact localization masks across all four tasks, while baselines show under- or over-localization and occasional failures.The qualitative comparison includes input images, ground-truth masks, FOCAL results, and baseline results.Source: Figure 4 · p. 18, §4.4 · p. 19

4.5 T-SNE Distributions

  • FOCAL maintains clearer real/fake feature separation than SAM+LoRA across tasks, with the advantage especially evident on Tasks 3 and 4.The Task 1 test visualization after training through Tasks 1–4 also shows stronger final-stage separability for FOCAL than IFL and SAM+LoRA.Source: §4.5 · pp. 19–20, Figure 5 · p. 20, Figure 6 · p. 20

4.6 Ablation Experiments

Ablations show that FOCAL's performance and retention depend on its representation-adaptation and gradient-surgery components, its forensic experts, and sufficient LoRA rank.

4.6.1 Effectiveness on Different Backbones

  • FOCAL improves performance across SAM-Tiny, SAM-Large, and SAM-Base, with gains becoming more evident later in the task sequence.The compared vanilla backbones degrade on previously learned domains, while FOCAL better maintains those domains and adapts to new data.Source: §4.6.1 · pp. 21–22, Table 6 · p. 22

4.6.2 Effectiveness of the Designed Components

  • Removing either FARA or FLAG degrades performance; removing FLAG particularly harms previously learned domains while retaining competitive performance on the new domain.FARA targets robust forensic feature extraction, while FLAG is designed to preserve learned knowledge.Source: §4.6.2 · pp. 22–23

4.6.3 Effectiveness of the Forensic Experts

  • The full model performs best overall, especially at T4, and removing any forensic expert weakens performance.Removing the Bayer or SRM expert causes particularly large drops; SMoFE integrates complementary forensic cues.Source: Table 6 · p. 22, §4.6.3 · p. 23

4.6.4 Ablation Study on LoRA Rank

  • LoRA rank 32 gives the best overall final-stage performance; rank 64 is competitive but lower, while ranks 4 and 8 provide limited adaptation capacity.Further rank increases offer limited benefits, add redundant parameters, and may increase overfitting risk.Source: §4.6.4 · pp. 23–24, Table 7 · p. 23

5 Conclusion

FOCAL combines forensic-trace adaptation with Fisher-weighted LoRA gradient surgery and consistently outperforms existing methods in localization and detection across both protocols. The benchmark also exposes performance degradation under domain gaps, while leaving more complex real-world continual-learning settings for future study.

5 Conclusion

  • The benchmark introduces cross-dataset and cross-content protocols, where existing methods degrade severely on emerging forgery domains, particularly across substantial domain gaps.Source: §5 · p. 24
  • FOCAL consistently achieves superior performance in pixel-level forgery localization and image-level forgery detection under both evaluation protocols.The protocols are cross-dataset and cross-content continual learning.Source: §5 · p. 24
  • SMoFE integrates complementary forensic cues, FEGDP converts low-level artifacts into structured SAM localization evidence, and FLAG modulates LoRA gradients by prior-task importance.The framework combines forensic-aware representation adaptation with Fisher-weighted LoRA gradient surgery to mitigate forgetting while retaining plasticity for new forgery patterns.Source: §5 · p. 24
  • The benchmark does not fully capture real-world continual-learning complexity; future work could use more complex protocols and diverse datasets.Source: §5 · p. 24

Related papers

Loading 2609.38251v1…