PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Model the Artifacts Explicitly? Disentangle Artifacts via Pairwise Edit Relations for Image Manipulation Localization

Xuekang Zhu, Kaiwen Feng, Ruifeng Wang, Xiwen Wang, Xiaochen Ma, Bo Du, Changjiang Jiang, Chenfan Qu, Songyu Ye, Xia Du, Wentao Feng, Jian Liu, Ji-Zhe Zhou

arXiv:2610.07916v1cs.CV

TL;DR

Image manipulation localization estimates manipulated regions, but its artifacts are subtle and difficult to annotate directly, while end-to-end learning can entangle them with semantics. This paper uses edit relations and a source-anchored dataset to learn artifacts in a two-stage paradigm. It reports consistent improvements across diverse IML models and empirical evidence of feature disentanglement.

  • Problem

    Artifacts are subtle and difficult to annotate directly, while direct IML learning can be vulnerable to label noise and semantic shortcuts.

    Source: §1 · pp. 1–2

  • Method

    PAL uses edit relations and EditGroup-45K to disentangle artifacts, then SL uses the PAL-trained representation for localization.

    Source: §1 · pp. 2–3

  • Results

    PAL yields consistent improvements across diverse IML architectures, and empirical analyses support explicit artifact capture through feature disentanglement.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support a two-stage, architecture-general initialization strategy that decouples artifact extraction from final localization.

    Source: §6 · p. 10

    PAL does not customize the backbone, decoder, or localization head for particular models, which may limit model-specific optimization.

    Source: H Limitations · p. 25

Abstract

from arXiv · show

Image Manipulation Localization (IML) is commonly formulated as a fully supervised learning task that estimates the optimal manipulation mask $y$ for a given image $x$. In this work, we first reveal the latent nature of artifacts and thus reinterpret IML as a latent-variable problem, $P(y|x)=\int P(y|z)\,P(z|x)\,dz$, where $z$ denotes the artifacts. Following this interpretation, we pinpoint the cause for the current IML models' insufficiency as their implicit artifacts modeling strategy, highlighting the necessity of modeling $z$ in an explicit manner. Without direct labels, feature disentanglement is the most appropriate solution for this explicit modeling. Accordingly, we propose a two-stage learning paradigm with the Pairwise Artifacts Learning (PAL) and Standard Localization (SL) phases to estimate $P(z|x)$ and $P(y|z)$ via edit relations. To support our edit-relation-based learning, we further curate EditGroup-45K, a source-anchored dataset organized into edit groups for pair construction. Extensive experiments show that our PAL paradigm yields consistent improvements across diverse IML architectures, and empirical analyses further verify that PAL does capture artifacts explicitly through feature disentanglement. Code and dataset are available at https://github.com/venus-guangjian/PAL

1 Introduction

The introduction reframes image manipulation localization as latent-artifact modeling: artifacts are difficult to annotate directly and can be entangled with semantics in end-to-end learning. It proposes relational disentanglement, a two-stage training paradigm, and EditGroup-45K, reporting improvements across diverse IML models.

1 Introduction

  • Artifacts are subtle, largely imperceptible manipulation inconsistencies, making direct annotation infeasible and motivating a latent-variable view of IML.Existing direct P(y|x) learning is vulnerable to label noise and semantic shortcuts [61]; experiments report rigid human priors can compromise generalization.Source: §1 · pp. 1–2
  • Contrasting edit-related pairs with non-edit-related pairs suppresses semantic variation and disentangles latent artifacts without requiring artifact annotations.The resulting decomposition treats artifact estimation P(z|x) and artifact-based localization P(y|z) as separate processes.Source: §1 · p. 2
  • The proposed two-stage framework uses Pairwise Artifacts Learning (PAL) to model artifacts and Standard Localization (SL) to adapt the PAL-optimized encoder to pixel-wise masks.SL uses the PAL-optimized encoder as its backbone initialization and follows the standard IML training protocol.Source: §1 · pp. 2–3
  • EditGroup-45K provides 52K authentic and 232K manipulated images across 45K edit groups, supplying explicit edit relationships for pairwise artifacts modeling.Each group links at least one authentic image to its manipulated counterparts.Source: §1 · p. 3
  • PAL yields consistent improvements across diverse IML architectures and multiple benchmarks, with empirical analyses supporting explicit artifact capture through feature disentanglement.The reported gains are described as independent of network structure and supportive of cross-domain generalization.Source: §1 · p. 3

2 Related Works

Prior forgery methods explore varied forensic cues and architectures, while direct end-to-end localization can entangle artifacts with semantics. PAL targets that limitation, and EditGroup-45K supplies source-linked groups for pairwise artifact learning.

2.1 Forgery Methods

  • Forgery methods span forensic cues, transformers, sparse and mixture-of-experts modeling, and reasoning approaches, yet direct P(y|x) learning can entangle artifacts with semantics [61] and label noise.The surveyed methods include boundary and noise cues, spatial-channel correlations, and semantic-forensic feature combinations.Source: §2.1 · p. 3
  • PAL first approximates latent artifacts and uses the learned representation to initialize downstream detectors, addressing semantic interference in the two-stage strategy.The paper presents this as a general initialization paradigm rather than a new IML architecture.Source: §2.1 · p. 3

2.2 Forgery Datasets

  • Existing datasets often lack explicit original-to-edit links or focus on a single manipulation type; EditGroup-45K adds large-scale edit groups for paired supervision.Its group structure is intended to support learning robust, transferable artifact features.Source: §2.2 · p. 4

3 The EditGroup-45K Dataset

EditGroup-45K addresses the lack of large-scale source-anchored manipulation data with groups linking authentic images to edits. It combines manual and AIGC manipulations and is presented as a basis for relational artifact modeling.

3 The EditGroup-45K Dataset

  • EditGroup-45K is a source-anchored dataset organized into manipulation groups, addressing the lack of explicit authentic-source links in existing benchmarks.The paper positions these links as a basis for provenance tracking at scale and relational artifacts modeling.Source: §3 · p. 4

3.1 Construction and Organization

  • EditGroup-45K contains 45,546 aligned edit groups, comprising 52,074 authentic images and 232,520 manipulated variants; each manipulated image is physically linked to its specific source anchor(s).Authentic anchor images are curated from M3, considering image resolution and semantic diversity.Source: §3.1 · p. 4

3.2 Comparison with Existing Benchmarks

  • EditGroup-45K combines source-to-manipulation correspondence with manual and AIGC edits, including 140,137 manual edits and 92,383 AIGC edits.The dataset reports copy-move, splicing, inpainting, and full-image generation among its manipulation categories.Source: §3.2 · pp. 4–5

4 Method

The method treats artifacts as a latent variable, learning artifact-oriented representations from edit relations before adapting them to pixel-level localization. Its pair construction aims to suppress semantic cues through marginal matching.

4 Method

  • PAL first approximates P(z|x), then transfers its optimized encoder to standard localization, which learns P(y|z).The second stage follows standard pixel-level localization protocols.Source: §4 · p. 5, §4.3 · p. 6

4.1 Training Pairs Construction

  • Positive pairs link an authentic image to its directly derived manipulated counterpart; negatives lack that direct edit relation while approximating the manipulated-image marginal.Six negative types are sampled as a mixture for each positive pair to encourage reliance on the pairwise edit relation.Source: §4.1 · p. 6

4.2 Stage I: Pairwise Artifacts Learning

  • PAL encodes absolute feature differences between paired images, pools them into a relation vector, and trains a lightweight decoder with pairwise BCE.Under marginal matching, the objective estimates a conditional likelihood ratio and encourages artifact signals over semantic content.Source: §4.2 · p. 6

4.3 Stage II: Standard Localization

  • The PAL-optimized encoder initializes the target model, adapting the task from the entangled P(y|x) objective to P(y|z).The localization stage follows the standard pixel-level protocol.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate PAL through localization benchmarks, component ablations, artifact analyses, and transfer tests. Across tested backbones and existing methods, PAL initialization improves performance, while ablations and boundary analyses examine its behavior.

5.3 Analysis of Artifacts Approximation

  • PAL amplifies boundary activations by approximately 5×–7× across datasets.The analysis compares an ImageNet-pretrained ConvNeXt-Small with its PAL-optimized counterpart.Source: Table 4 · p. 7, §5.3 · p. 8

5.4 Transferability Analysis

  • PAL initialization improves every tested backbone and baseline; ConvNeXt-Small gains +10.68%, while TruFor rises 0.5623 →0.6095 (+8.39%).The evaluations cover CNNs, Transformers, foundation-style backbones, and existing IML methods.Source: Table 5 · p. 7, §5.4.1 · p. 9, §5.4.2 · p. 9

5.1 Experimental Setup

  • The evaluation uses PAL pairwise training before standard localization and reports pixel-level F1 at a 0.5 mask threshold.Localization training and evaluation use the specified datasets and Protocol-CAT setting.Source: §5.1 · p. 7, §5 · p. 7

5.2.1 Effect of Pairwise Artifacts Learning

  • PAL reaches the highest All-Avg of 0.6266 and raises Cross-Avg to 0.5868, versus 0.5320 for the baseline.Baseline-ABS reaches Cross-Avg 0.5292 despite excelling in In-Avg.Source: §5.2.1 · p. 8

5.2.2 Effect of Pair Representation Design

  • Removing explicit feature differences lowers Cross-Avg from 0.5868 to 0.5333; removing MaxPool lowers it to 0.5676.Removing AvgPool yields the lowest In-Avg, 0.7814.Source: §5.2.2 · p. 8

5.2.3 Effect of Negative Pair Composition

  • Without self-augmented negatives, In-Avg reaches 0.8521 while Cross-Avg falls to 0.5492; removing in-group Edited–Edited pairs yields the lowest Cross-Avg, 0.5386.The reported discrepancy for removing self-augmentation indicates source-domain shortcuts and reduced robustness to unseen artifacts.Source: §5.2.3 · p. 8

5.2.4 BCE vs. Contrastive Objective

  • BCE converges faster and reaches a higher performance ceiling than the contrastive objective for the heterogeneous artifacts in EditGroup-45K.The authors note that these artifacts do not naturally satisfy the class-level compactness assumption required by contrastive pulling.Source: §5.2.4 · p. 8

5.3 Analysis of Artifacts Approximation

  • PAL responses emphasize manipulated boundaries for manual local edits and broader regions for AIGC-based full-image edits.The authors interpret the patterns as manipulation-dependent artifacts rather than a uniform boundary cue.Source: §5.3 · p. 9, Figure 3 · p. 9

6 Conclusion

The conclusion frames IML as latent-variable learning and reports that relational contrast can disentangle artifacts from semantic content without pixel-level artifact annotation.

6 Conclusion

  • The paper frames IML through P(y|x) = ∫P(y|z)P(z|x)dz and reports that relational contrast disentangles artifacts from semantic content.The conclusion describes PAL with EditGroup-45K as practically superior to implicit artifact modeling without pixel-level artifact annotation.Source: §6 · pp. 9–10
  • The two-stage approach is presented as a generic initialization strategy, with gains across diverse models and a focus on generalization bottlenecks associated with semantic shortcuts.The authors position explicit, data-driven artifact representation as a direction for image forensics.Source: §6 · p. 10

A Theoretical Analysis

The analysis derives the Bayes-optimal pairwise BCE decision and explains how marginally matched negatives direct discrimination toward edit-induced artifacts rather than semantic shortcuts.

A.1 Bayes-Optimal Solution of the BCE Objective

  • The Bayes-optimal BCE score is the log-odds of the pairwise posterior, which reduces to a conditional likelihood ratio under the positive and negative pair distributions.Positive pairs follow qrel and negative pairs follow qneg, the non-related sampling strategy.Source: §A.1 · p. 15

A.2 Interpretation of Marginal Matching and Artifact-Focused Discrimination

  • If the negative-image marginal differs from the positive one, the discriminator can exploit semantic priors instead of modeling the edit relation.The analysis gives systematically cleaner negatives or different content domains as examples of such shortcuts.Source: §A.2 · p. 15
  • Matching negative and positive image marginals removes much of the label information in the second image alone, leaving conditional edit differences as the discriminative cue.The analysis interprets that remaining cue as latent artifacts associated with the causal editing process.Source: §A.2 · pp. 15–16

B BCE vs. Contrastive Objective

BCE-based PAL outperforms its contrastive variant in downstream localization, with faster convergence; the authors relate this difference to heterogeneous manipulation artifacts.

B BCE vs. Contrastive Objective

  • BCE-based PAL converges faster and reaches a higher downstream performance ceiling than PAL-Con.PAL-Con uses the same pair construction, shared encoder, feature difference, and pooling design, but replaces the binary classifier with a supervised contrastive objective.Source: B BCE vs. Contrastive Objective · p. 16
  • Contrastive same-label compactness can conflict with artifact heterogeneity across copy-move, splicing, inpainting, and full-image generation.The authors note that forcing these varied edit-related pairs together may introduce gradient conflicts.Source: B BCE vs. Contrastive Objective · p. 16
  • BCE distinguishes edit-related from non-edit-related pairs without imposing class-level compactness, matching the relational supervision in EditGroup-45K.The authors conclude this makes BCE more suitable for PAL.Source: B BCE vs. Contrastive Objective · p. 16

C Derivation and Interpretation of Artifacts Approximation

The appendix explains why residual subtraction is used to inspect PAL's artifact signal despite semantic information remaining in encoder features, and reports learning-rate sensitivity.

C Derivation and Interpretation of Artifacts Approximation

  • The raw pairwise feature difference mixes semantic changes needed for manipulation with the target manipulation artifacts.The encoder decomposition distinguishes shared semantic content from the manipulation-specific footprint.Source: C Derivation and Interpretation of… · p. 17
  • Residual probing empirically supports that PAL learns a structured artifact signal even though raw features remain dominated by backbone semantic bias.The probe subtracts a semantic baseline estimated with a fixed ImageNet-pretrained encoder from the model response.Source: C Derivation and Interpretation of… · p. 17

D.1 Impact of Learning Rate

  • At LR 1 × 10−4, PAL achieves optimal performance; LR 2 × 10−4 yields 0.8873, while LR 1 × 10−6 yields AUC 0.8862.The experiments use 12 epochs and a 1:1 sampling ratio for the learning-rate sweep.Source: Table 7 · p. 18, §D.1 · p. 18

D.2 Impact of Training Duration

Training duration and negative-sampling balance affect PAL's AUC, while EditGroup-45K spans repeated source edits, multi-source manipulations, and varied edit-group sizes.

D.2 Impact of Training Duration

  • At epoch 12, PAL reaches its peak AUC of 0.8926; 10 epochs yields 0.8747 and 15 epochs yields 0.8920.The authors select 12 epochs as the standard duration, noting a slight decline at 15 epochs.Source: Table 8 · p. 18, §D.2 · p. 18

D.3 Impact of Positive-to-Negative Ratio

  • A 1 : 1 positive-to-negative ratio yields AUC 0.8926, compared with 0.8837 at 1 : 2.The authors adopt 1 : 1 for performance and training efficiency.Source: §D.3 · p. 18, Table 9 · p. 19

E Dataset Statistics and Distribution Details

E.1 Editing Intensity

  • 58.49% of real images are reused as sources, including 31.78% edited 2–3 times; 2.81% are used for more than 30 edits.The remaining 41.51% of real images are used once.Source: Figure 4 · p. 19, §E.1 · p. 19

E.2 Composition Complexity

  • 11.31% of manipulated images use multiple sources, while 88.69% are single-source.The multi-source subset represents two-or-more-source splicing operations.Source: §E.2 · p. 19, Figure 5 · p. 20

E.3 Group Scale Distribution

  • 72.76% of edit groups contain 1–3 image pairs; 18.55% contain 4–10, 5.87% contain 11–30, and 2.8% exceed 30.Source: Figure 6 · p. 20, §E.3 · p. 20

E.4 Similarity and Metric Analysis

The SSIM analysis characterizes broad editing variation, while supplementary ablations show PAL's overall performance and test how pair representation and negative-pair composition affect robustness.

E.4 Similarity and Metric Analysis

  • SSIM categorization places 23.50% of pairs at 0.95 or higher, 38.11% between 0.80 and 0.95, and 38.40% below 0.80; its broad variance spans subtle to structural edits.Figures 7 and 8 show the categorical and continuous SSIM distributions, respectively.Source: §E.4 · pp. 20–21, Figure 7 · p. 21, Figure 8 · p. 21

F Supplementary Ablation Study

  • The ablations compare supervision forms, pair-representation designs, and negative-pair compositions using the same ImageNet-1K-pretrained ConvNeXt-Small and identical Standard Localization training.Metrics are final Protocol-CAT results; All-Avg averages six datasets, including five cross-domain datasets.Source: F Supplementary Ablation Study · p. 21, Table 10 · p. 22

F.1 Effect of PAL

  • 0.6266 All-Avg is PAL's highest overall score; on AutoSplice, PAL scores 0.4830 versus Baseline-ABS 0.3637, while Baseline-ABS reaches 0.8457 on CASIAv1.The comparison uses the pixel-level F1 score on Standard Localization under Protocol-CAT.Source: Table 10 · p. 22, §F.1 · p. 22

F.2 Effect of Pair Representation Design

  • Removing Diff drops Cross-Avg from 0.5868 to 0.5333 and COVERAGE from 0.6242 to 0.4260; removing AvgPool hurts CASIAv1, while removing MaxPool weakens cross-domain generalization.The reported CASIAv1 score without AvgPool is 0.7814.Source: §F.2 · p. 22

F.3 Effect of Negative Pair Composition

  • Removing IG-EE yields the lowest Cross-Avg (0.5386) and NIST16 score (0.2992), while removing G-EE gives the lowest CASIAv1 score (0.7790).The authors link IG-EE to distinguishing manipulation types within the same semantic context.Source: §F.3 · p. 23
  • Removing self-augmented negatives raises CASIAv1 to 0.8521 but lowers AutoSplice to 0.4169, versus PAL's 0.4830.The authors describe this trade-off as overfitting to source-specific shortcuts and reduced robustness to unseen manipulations.Source: §F.3 · p. 23

F.4 Robustness to Post-processing Perturbations.

PAL is more robust than the baseline under blur and JPEG compression, benefits from larger EditGroup-45K pre-training, and produces more accurate qualitative masks; the paper also outlines broader extensions and an architecture-specific limitation.

F.4 Robustness to Post-processing Perturbations.

  • Across blur and JPEG levels, Baseline+PAL outperforms Baseline and degrades more slowly, preserving higher F1 under aggressive perturbations.Both perturbations progressively suppress manipulation traces; PAL's relational modeling is described as less sensitive to these distortions.Source: §F.4 · p. 23

F.5 Scaling Behavior of EditGroup-45K

  • As EditGroup-45K pre-training grows, PAL gains consistently, whereas ABS saturates around 50% and slightly degrades at 100%.The authors attribute ABS label conflicts to source images that may already contain editing or generation traces; PAL learns relative edit differences within pairs.Source: Figure 9 · p. 24, §F.5 · p. 24

F.6 Qualitative Comparison between PAL and ImageNet Initialization

  • For copy-move, splicing, and inpainting examples, PAL initialization yields masks closer to ground truth with fewer false positives than ImageNet initialization.The authors characterize this as artifact-aware initialization that helps localization focus on manipulation-related rather than semantic or background regions.Source: Figure 10 · p. 24, §F.6 · pp. 24–25

G Broader Impact

  • PAL's artifact-oriented formulation is not inherently tied to RGB; proposed extensions include spectral and spatial remote-sensing traces, motion or pose inconsistencies, and 3D geometric deformation.The paper also identifies privacy protection, adversarial defense, and content authentication as possible extensions.Source: G Broader Impact · p. 25
  • Adaptive evidence selection and hierarchical organization are proposed as ways to organize and exploit learned artifact representations across manipulation types and domains.These are suggested directions, not evaluated extensions in the supplied discussion.Source: G Broader Impact · p. 25

H Limitations

  • PAL is a general artifact-aware initialization rather than a model-specific architecture, so it may lack backbone-, decoder-, or localization-head-specific optimization.The paper identifies dedicated architectures that integrate pairwise artifact learning more deeply as future work.Source: H Limitations · p. 25

Related papers

Loading 2610.07916v1…