PaperlayerView exact revision on arXiv

Source-linked AI summary

APT: Anchor-aligned Perturbations for Tamper Localization in Fully Regenerated Images

Suhyeon Ha, Woo Jae Kim, Joonsung Jeon, Sooel Son, Sung-eui Yoon

arXiv:2608.30656v1cs.CV

TL;DR

Existing proactive localization methods target spliced editing, but fully regenerated diffusion inpainting disrupts background signals and exposes a major gap in pixel-level localization. APT addresses this with anchor-aligned dense latent perturbations and alignment-based detection, achieving strong FR performance while remaining competitive on SP images. The paper positions APT as a practical framework for tampering types unknown at test time.

  • Problem

    Existing proactive localization methods assume splicing, while fully regenerated diffusion inpainting impairs background signals and has not been addressed by prior proactive localization work.

  • Method

    APT embeds dense vector-wise latent perturbations aligned to a predefined anchor, with hard negative mining and noisy perturbation improving signal uniformity and robustness.

  • Results

    APT* achieves an FR IoU of 0.92 versus WAM’s 0.84, while APT maintains an FR AUC of 0.97 and existing methods reach near-random AUC levels of ≈0.5.

  • Takeaways & Limitations

    APT provides a practical forensic framework that generalizes across tampering types unknown at test time while remaining competitive in the SP setting.

  • Takeaways & Limitations

    Copy-move forgeries within the same protected image can retain anchor alignment and potentially produce false negatives.

Abstract

from arXiv · show

Proactive tamper localization embeds an imperceptible signal into an image prior to distribution, enabling pixel-level manipulation detection. Existing methods assume a spliced (SP) setting, where synthesized regions are composited onto the original background, leaving embedded signals intact. However, real-world diffusion-based inpainting operates in a fully regenerated (FR) setting, where the entire image undergoes denoising, disrupting background signals and rendering existing frameworks ineffective. We propose APT, a semi-fragile latent-space perturbation that embeds a dense, vector-wise localization signal. By aligning each spatial feature vector toward a fixed anchor direction, APT localizes tampering via the alignment disparity between synthesized foreground and anchor-aligned background features after inpainting. The proposed hard negative mining loss and noisy perturbation branch further enforce uniform alignment. Experiments on COCO demonstrate that APT achieves an FR IoU of 0.92, outperforming the strongest baseline (WAM, 0.84), while existing methods collapse to near-random performance (AUC 0.5), establishing APT as a practical forensic framework generalizable across tampering types unknown at test time.

1 Introduction

APT addresses the overlooked fully regenerated inpainting setting by embedding a dense latent-space signal that remains aligned in reconstructed backgrounds but differs in synthesized foregrounds. It achieves localization across FR and SP manipulations, substantially outperforming existing methods under FR.

  • Problem: Diffusion-based fully regenerated inpainting disrupts embedded background signals, whereas spliced editing preserves them outside synthesized regions.Existing proactive localization methods primarily assume the SP setting, leaving a gap for native diffusion outputs.
  • Problem: Effective FR localization requires signals robust to background regeneration yet fragile to foreground synthesis.Existing methods struggle because their signals are vulnerable to VAE and pixel regeneration and often lack spatial correspondence.
  • Method: APT aligns dense spatial latent vectors toward a predefined anchor, using foreground-background alignment disparity as the localization cue.The anchor is sampled independently of the generative model’s learned distribution, while reconstructed backgrounds retain anchor-induced alignment.
  • Method: APT adds hard negative mining and a noisy perturbation branch to suppress weak alignment and improve robustness to VAE-induced distortions.These components enforce spatially uniform alignment during optimization.
  • Conclusion: APT is presented as a practical framework for tamper localization when the manipulation type is unknown at test time.The method produces localization masks under both FR and SP settings.

2 Related Work

Related work spans passive detection and proactive signal embedding, but existing proactive methods remain poorly suited to fully regenerated inpainting. The gap reflects both signal fragility under regeneration and insufficient spatial density for pixel-level localization.

  • Diffusion-based Inpainting: Modern inpainting generates coherent raw outputs without post-processing, creating the fully regenerated setting in which the entire image passes through diffusion.This makes tamper localization especially challenging when no explicit splicing occurs.
  • Passive Localization: Passive methods analyze boundaries, hierarchical features, noise anomalies, or multimodal representations, but struggle with photorealistic generative manipulations.They are described as effective against manual editing such as copy-move and splicing.
  • Proactive Localization: Proactive methods embed signals before alteration, including spatial templates, wavelet-domain secret images, localized watermarks, and VAE-aware extraction.The cited methods include MaLP, EditGuard, WAM, and OmniGuard.
  • Limitations of Existing Methods: Pixel- and frequency-domain signals are susceptible to the VAE bottleneck and background regeneration, while global signals lack direct spatial correspondence.Without locally dense signals, predictors may overfit to splicing boundaries and collapse under global FR distortion.

3 Preliminary

The preliminary distinction between SP and FR manipulations is whether the unmasked background is preserved. FR re-synthesizes the entire image, disrupting background equivalence and undermining existing proactive localization signals.

  • Spliced versus fully regenerated images: SP composites synthesized content with the original background, preserving unmasked pixels and embedded background signals.The SP formulation combines the generated image inside the mask with the original image outside it.
  • Spliced versus fully regenerated images: FR uses the raw generative output, so stochastic noise injection and iterative denoising re-synthesize regions outside the editing mask.Consequently, xFR ⊙(1 −m) ≠ x ⊙(1 −m).
  • Limitations of existing methods: Existing proactive localization becomes vulnerable in FR because generative priors, background disharmony, and VAE reconstruction errors prevent exact recovery of original details.These factors disrupt the background signals that SP-based methods rely on.
  • Anchor-aligned perturbation: Zero-bit watermarking constrains an image feature vector within a hypercone defined by a minimum cosine alignment with an anchor.The hypercone contains vectors whose cosine similarity with anchor a is at least cos(θ).

4 Method

APT embeds dense anchor-aligned signals by optimizing a latent perturbation, then verifies tampering from per-vector cosine similarity maps. Hard-negative mining, noisy optimization, and lightweight mask processing support spatially uniform and fine-grained localization.

  • Anchor-aligned perturbation: APT optimizes a VAE-latent perturbation so each spatial feature vector aligns with a predefined anchor direction.The dense feature map provides spatial correspondence between latent locations and image regions.
  • Tamper localization: APT verification extracts per-vector anchor-alignment maps and produces pixel-level masks through either training-free prediction or a shallow mask decoder.The decoder is optional and is intended to refine spatial boundaries.
  • Anchor-aligned perturbation: Generated foreground features tend to have low anchor similarity, while regenerated background features retain anchor-induced alignment, creating the localization disparity.This separation avoids requiring strict pixel-level background preservation.
  • Hard negative mining loss: Hard negative mining penalizes under-optimized spatial vectors by averaging the loss over the top-K vectors with the largest individual losses.The index set contains K = ⌈ρ · H′W′⌉ vectors selected by descending loss.
  • Noisy perturbation branch: The noisy perturbation branch injects Gaussian noise into randomly masked latent regions, then applies both hinge and hard-negative losses to the decoded image.This trains robustness against latent-space distortions introduced by VAE regeneration.
  • Optimization objective: The final objective jointly enforces alignment on perturbed and noisy images while minimizing perceptual distortion and clipping the perturbation to an ℓ∞ budget.The objective includes PSNR and LPIPS penalties before pixel-domain clipping.
  • Shallow mask decoder: The shallow decoder learns spatial contiguity from synthetic similarity maps generated from clean images, using alignment gain in background regions and collapse in tampered regions.Its training combines BCE and Dice losses, with two upsampling stages for mask prediction.

5 Experiments

Experiments show that APT maintains accurate localization under fully regenerated inpainting, transfers across models, remains robust to corruptions, and benefits from both auxiliary training components.

  • FR localization: 0.92 IoU: APT∗ outperforms WAM’s 0.84 IoU in fully regenerated localization, while StableGuard and OmniGuard approach random AUC.APT also achieves robust AUC of 0.97 in the FR setting.
  • FR localization: 0.93 average IoU: APT∗ maintains consistent transfer performance across diverse inpainting models in the FR setting.The evaluation includes BrushNet, ControlNet, and HD-Painter.
  • SP localization: 0.99 average AUC: APT and APT∗ generalize across inpainting architectures in the SP setting.Both variants remain competitive despite not being explicitly optimized for splicing boundaries.
  • Robustness to image corruptions: 0.90 average IoU: APT∗ remains robust in FR localization under brightness increase, Gaussian blur, and JPEG compression.In SP with unseen corruptions, APT and APT∗ maintain AUC above 0.96.
  • Alignment analysis: Layer 1 provides the sharpest spatial boundary, with foreground and background means of 0.124 and 0.265, respectively.Layer 0 is vulnerable to the VAE bottleneck, while deeper layers entangle tampering cues with surrounding background vectors.
  • Ablation study: Lnoise and Lhard independently improve localization, with IoU gains of +0.23/+0.24 and +0.29/+0.26 in SP/FR, respectively.The full model achieves the best performance.

6 Conclusion

APT addresses the fully regenerated inpainting setting through a semi-fragile perturbation framework. Its alignment-based signals support localization across tampering types unknown at test time.

  • APT addresses the fully regenerated (FR) inpainting setting overlooked by existing proactive localization methods.
  • APT embeds dense, vector-wise signals through latent perturbation for alignment-based semi-fragility.The signal is robust to background regeneration yet fragile to foreground synthesis.
  • Experiments across inpainting architectures show that APT significantly outperforms existing methods in FR while remaining competitive in SP.
  • APT is positioned as a practical forensic framework generalizable across tampering types unknown at test time.

Supplementary Material

The supplementary material expands evaluation, implementation context, and robustness analyses for APT and APT*. These results include segmentation-mask tests and larger-scale evaluation across FR and SP settings.

  • The supplementary material covers segmentation-mask evaluations, 500-image tests, cross-VAE and architecture transferability, signal disruptions, perturbation analyses, and implementation protocols.
  • APT denotes training-free prediction, whereas APT* uses a trained shallow mask decoder.
  • Both APT and APT* consistently outperform all baselines in FR localization on instance-segmentation masks.Qualitative results across four inpainting models further demonstrate superior localization.
  • APT and APT* maintain consistent localization when evaluation scales to 500 COCO validation images across FR and SP settings.

A.3 Transferability across Different VAEs and Architectures

APT* is evaluated for transferability beyond the protection-time SD-VAE and across diverse inpainting architectures. The supplementary comparisons also examine signal preservation and localization behavior under FR and SP.

  • APT* is tested on five additional inpainting models spanning LoRA-fine-tuned, SDXL, flow-matching, and LanPaint-based model families.
  • Transferability performance is reported in the FR setting using the same SD-VAE-optimized perturbation and shallow decoder for APT*.
  • APT* achieves AUC ≥0.97 and IoU ≥0.91 on recent inpainting architectures.The results indicate generalization to non-identical VAEs and deterministic ODE-based generation pipelines.
  • Under FR, WAM and OmniGuard undergo severe alterations in localization cues, while StableGuard and APT* maintain signals across FR and SP.

B.2 Layer-wise Perturbation Analysis

The supplementary analysis examines how perturbation structure and optimization choices affect visual quality, localization, and computational cost. It also documents the optimization and mask-generation procedures.

  • Layer-wise Perturbation Analysis: Layer 0 perturbations exhibit fine-grained horizontal and vertical patterns that appear as distinct crosses in the high-frequency spectrum.
  • Trade-offs: Varying target cosine similarity τ shifts the PSNR-IoU balance, while tightening ϵδ improves perceptual quality at a modest localization cost.
  • Trade-offs: Reducing optimization to 50 steps accelerates processing to 14.34 seconds per image while maintaining high localization performance.
  • Optimization Procedure: Algorithm 1 specifies the complete anchor-aligned perturbation optimization procedure.
  • Mask Generation: Shallow-decoder training uses equally likely unpaired instance masks and random rectangular masks with dimensions sampled from 10% to 50% of the image.

C.3 Evaluation

APT is evaluated for pixel-level localization using thresholded overlap metrics and threshold-independent discrimination, with particular attention to fixed-threshold usability across SP and FR settings. The evaluation also specifies the embedding optimization procedure, baseline calibration, inpainting setup, and a black-box verification scenario.

  • Metrics: F1, AUC, and IoU evaluate pixel-level localization, with F1 and IoU using threshold 0.5 while AUC measures threshold-independent class separation.AUC assesses inherent discriminative power, whereas F1 and IoU assess mask overlap at a fixed threshold.
  • Metrics: AUC is essential because predicting the entire image as perturbed can yield near-random AUC (≈0.5) despite misleadingly high F1 and IoU.This reflects the strong class imbalance between tampered regions and intact background.
  • Embedding optimization: APT optimizes a latent perturbation through clean and noisy branches, hard and hinge losses, perceptual losses, and iterative updates before clipping the protected image.The algorithm encodes the image with a VAE, applies random masked noise during optimization, and constrains the final perturbation budget.
  • Evaluation protocol: High IoU at a constant threshold is required because verifiers may not know whether an image was spliced or fully regenerated, and score distributions can shift between settings.AUC alone may remain high while a single threshold becomes suboptimal across SP and FR.
  • Evaluation protocol: Baseline embedding strengths are adjusted to match perceptual quality, while PSNR is computed directly between the original clean and protected images across methods.WAM and OmniGuard use watermark strengths of 3.0 and 2.0, respectively.
  • Evaluation protocol: All evaluated inpainting models use 50 inference steps and an empty text prompt to fill masked regions while blending with surrounding context.This standardizes the inpainting configuration across models.
  • Threat model: The defense assumes a black-box detector API whose internal anchor vectors remain hidden on the server as a secret key.The detection algorithm is publicly accessible for integrity verification, while the anchor vectors are kept private.

D.2 Limitation

APT's localization cue depends on alignment disparity between protected background and synthesized foreground, creating limitations for copy-move attacks and misuse on already manipulated images. The paper identifies point-of-capture defenses as a direction for mitigating the latter risk.

  • D.2 Limitation: Copy-move forgeries within the same protected image retain anchor alignment and can therefore produce false negatives.APT's alignment-disparity cue is absent when the copied region already carries the embedded signal.
  • D.2 Limitation: APT addresses fully regenerated inpainting but its practical forensic scope remains tied to this challenging localized image-synthesis setting.The paper presents it as a baseline for fully regenerated manipulation and image authentication in the wild.
  • D.2 Limitation: Applying APT to an already manipulated image can embed a valid signal that misleads verification into authenticating the forgery as intact.The paper suggests integrating proactive defenses at capture, such as camera hardware, to mitigate this misuse risk.
Loading 2608.30656v1…