PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Generated Image Detectors Overrely on Global Artifacts: Evidence from Inpainting Exchange

Elif Nebioglu, Emirhan Bilgiç, Adrian Popescu

arXiv:2602.00192v1cs.CVcs.AI

TL;DR

Reliable detection of locally inpainted content is difficult because detectors can exploit global VAE-induced spectral artifacts instead of the synthesized region. The paper introduces INP-X, which restores original background pixels while preserving the edited content, and finds that detector performance drops sharply under this intervention; training on INP-X improves robustness and localization.

  • Problem

    Detectors report high benchmark accuracy, but it remains unclear whether they identify locally synthesized content or global artifacts introduced by the inpainting pipeline.

  • Method

    INP-X restores original pixels outside the edited region while preserving synthesized content inside the mask, supported by theoretical and empirical analysis of VAE spectral attenuation.

  • Results

    Training on INP-X improves cross-distribution robustness and manipulation localization compared with reliance on standard inpainting artifacts.

  • Takeaways & Limitations

    The findings support content-aware, localization-based detection and evaluation with realistic post-edits such as INP-X.

  • Takeaways & Limitations

    The study focuses on VAE-based architectures; non-VAE models and subtle boundary effects remain areas for further analysis.

Abstract

from arXiv · show

Modern deep learning-based inpainting enables realistic local image manipulation, raising critical challenges for reliable detection. However, we observe that current detectors primarily rely on global artifacts that appear as inpainting side effects, rather than on locally synthesized content. We show that this behavior occurs because VAE-based reconstruction induces a subtle but pervasive spectral shift across the entire image, including unedited regions. To isolate this effect, we introduce Inpainting Exchange (INP-X), an operation that restores original pixels outside the edited region while preserving all synthesized content. We create a 90K test dataset including real, inpainted, and exchanged images to evaluate this phenomenon. Under this intervention, pretrained state-of-the-art detectors, including commercial ones, exhibit a dramatic drop in accuracy (e.g., from 91\% to 55\%), frequently approaching chance level. We provide a theoretical analysis linking this behavior to high-frequency attenuation caused by VAE information bottlenecks. Our findings highlight the need for content-aware detection. Indeed, training on our dataset yields better generalization and localization than standard inpainting. Our dataset and code are publicly available at https://github.com/emirhanbilgic/INP-X.

1. Introduction

The paper argues that inpainting detectors often rely on global VAE-induced artifacts rather than locally synthesized content. INP-X removes those background artifacts while preserving the edited region, exposing detector brittleness and improving content-aware evaluation.

  • Motivation: >90% benchmark accuracy reported by academic and commercial detectors motivates testing whether they recognize local synthesis or easier global artifacts.The paper frames reliable detection as important because accessible inpainting poses risks for misinformation and content authenticity.
  • Core problem: Detectors primarily exploit a subtle spectral shift spread across the image by processing it through a VAE encoder-decoder.This shortcut bypasses identifying the locally generated content.
  • INP-X: INP-X restores original pixels outside the edited mask while retaining synthesized content inside it.If detectors identify synthetic content, the preserved masked content should remain detectable after exchange.
  • Benchmark: 90K images across 4 datasets and 3 inpainting models form a benchmark with matched real, standard, and exchanged triplets.The benchmark evaluates detector behavior under the exchange intervention.
  • Findings: Commercial systems drop from >91% to ∼55% accuracy on exchanged images, approaching random chance.The reported systems are HiveModeration and Sightengine.
  • Implications: Training on INP-X improves cross-distribution robustness and manipulation localization while showing that exchanged images remain challenging to detect.The authors present this result as evidence for learning local content features rather than global shortcuts.

2. Related Work

Prior work develops fingerprint-based, generalizable, localized, and anti-forensic detectors, but reported success is increasingly questioned as shortcut learning and dataset bias emerge. INP-X extends partial-manipulation benchmarking to separate local synthesis from global reconstruction effects.

  • Detection and generalization: Early detection work identifies model-specific and frequency-domain fingerprints, while later methods seek generator-agnostic representations and invariant spectral patterns.The related work spans GAN fingerprints, upsampling artifacts, CLIP features, and spectral approaches.
  • Shortcut learning: Evidence of shortcut learning suggests detectors may rely on easy-to-learn fake-image biases rather than genuine real-image features.Related studies also report degraded performance as generators improve.
  • Localization and forensics: Localization methods analyze compression, lighting, noise, or sensor-specific PRNU inconsistencies beyond binary detection.The paper connects VAE attenuation of high-frequency sensor noise to differing behavior on standard inpainting and INP-X.
  • Anti-forensics: Post-processing and anti-forensic methods evade detectors by adding perturbations or degrading images, whereas INP-X preserves synthesized content while removing background artifacts.This distinguishes exchange from approaches based primarily on image degradation.
  • Benchmarks: Semi-Truths pairs real images with AI-inpainted counterparts and masks; applying INP-X extends this setup to disentangle global and local detection factors.The benchmark literature provides the foundation for the paper’s exchanged-image evaluation.

3. Theoretical Analysis

The analysis attributes global inpainting artifacts to VAE reconstruction, which attenuates high-frequency information across the image. INP-X copies the real background back into the result, reducing distributional divergence to the manipulated foreground.

  • 3.1. Latent Diffusion Inpainting: Latent diffusion inpainting encodes and decodes the full image, so reconstruction changes can affect unmasked pixels even when unmasked latents are preserved.The image is modeled as semantic signal plus stochastic sensor noise, with the decoder acting over the entire spatial domain.
  • 3.2. Spectral Bias in VAE Reconstruction: VAE reconstruction attenuates high-frequency sensor noise because the information bottleneck captures semantic signal while losing stochastic non-semantic variation.Theorem 3.2 formalizes lower reconstructed spectral power where noise dominates the signal.
  • 3.2. Spectral Bias in VAE Reconstruction: 8× spatial downsampling creates a 64× reduction in spatial dimensions, limiting the encoding of non-semantic high-frequency content despite modern hybrid losses.The strict MSE-dominance assumption is violated by perceptual and adversarial losses, but empirical validation and extensions to SDXL and FLUX.1 support persistent attenuation.
  • Empirical validation: r = 0.941 links VAE loss with high-frequency content, while r = 0.600 links VAE loss with inpainting loss at the image level.These correlations support VAE reconstruction error as a source of inpainting differences.
  • Empirical validation: Pixel-level correlations are weaker but positive across VAE, inpainting, and high-frequency signals, partly because generated masked content adds unrelated variance.Reported means include VAE↔Inpaint Pearson r=0.45±0.23 and Spearman ρ=0.55±0.21.
  • 3.3. INP-X and Divergence Minimization: INP-X reduces KL divergence relative to standard inpainting by making the background distribution identical to the real-image background distribution.The remaining divergence is conditional on the foreground, whereas standard inpainting incurs divergence across the entire image.

4. Experiments and Analysis

The experiments evaluate pretrained and fine-tuned detectors on standard inpainting and INP-X across classification, localization, corruption robustness, and spectral analyses. Results show that INP-X removes global artifacts, sharply degrades pretrained detection, and improves content-aware generalization and localization when used for training.

  • Dataset and evaluation: The 90K benchmark contains real, standard-inpainted, and exchanged images across four datasets, with paired masks for controlled evaluation.Inpainting uses three models, and evaluation covers pretrained detectors, commercial APIs, fine-tuning, localization, and corruption robustness.
  • Pretrained detectors: INP-X degrades pretrained detector performance across the board, with accuracy ranging from near-random levels to 0.604 for DNF.Frequency-based Corvi2023 also drops significantly, and low recall accounts for most degradation.
  • Fine-tuned detectors: Fine-tuned detectors achieve at most 0.753 accuracy on INP-X, while models trained on standard inpainting reach only 0.603 when evaluated on INP-X.Models trained on INP-X transfer better to standard inpainting, with ResNet-50 reaching 0.745 accuracy.
  • Commercial APIs and robustness: Commercial APIs remain accurate on standard inpainting but fall to approximately 55% accuracy on INP-X.Sightengine and Hive Moderation remain robust to blur, light-spot, and JPEG corruptions, making INP-X substantially more effective than those standard corruptions.
  • Spectral and mask-size analysis: VAE processing introduces periodic spectral artifacts that function as detector fingerprints, while INP-X suppresses them and improves spectral MSE by 11× on SUN-RGBD.The relationship between spectral artifacts and mask size is not trivial: SUN-RGBD has smaller masks than CelebA-HQ but more severe artifacts.

5. Limitations

The study’s dataset focuses on VAE-based architectures, while non-VAE architectures remain underanalyzed. Boundary effects may persist despite blending, and INP-X could facilitate detector evasion.

  • The dataset focuses on VAE-based architectures, leaving component-based and pixel-space models for deeper analysis.The paper notes these architectures are currently less efficient and less widely used.
  • Subtle boundary effects may still occur depending on the mask and blending method, and the authors do not claim perceptual quality.
  • INP-X could be used to evade existing detectors, a risk the authors frame as intrinsic to vulnerability analysis.

6. Conclusion

The conclusion identifies global VAE reconstruction artifacts as a fundamental detector vulnerability and reports that INP-X training improves robustness and localization. It points toward frequency-preserving reconstruction and spectrally consistent decoding as future directions.

  • AI detectors rely on global VAE encoding-decoding artifacts rather than local content, exposing a fundamental vulnerability.
  • Training models with INP-X improves robustness and localization performance.
  • Future work could investigate frequency-preserving VAEs and decoding strategies that explicitly enforce spectral consistency.

Impact Statement

The paper studies AI-generated image detection for inpainting and links the findings to misinformation. It presents the benchmark as a tool for developing more content-aware and localization-based detectors while acknowledging evasion risks.

  • The study examines AI-generated image detection in the context of inpainting, with implications for misinformation.
  • The findings reveal existing detectors’ vulnerability and are intended to promote more content-aware detection methods.
  • The released benchmark provides concrete tools for research on robust localization-based detectors.
  • Using the exchange operation could facilitate evasion attacks against existing detectors, motivating further work on robust localization-based detection.

A.1. Why Not Restore After Generation?

Full-image denoising and VAE reconstruction are needed during generation for coherent edited boundaries, but original pixels outside the mask can be restored afterward. INP-X implements this restoration, with blending available to mitigate seams.

  • Full-image denoising is required because latent-space locality is entangled and complete reconstruction supports smooth, coherent boundaries.
  • After generation, replacing pixels where M = 0 with the original image eliminates the global artifact without removing synthesized content.
  • Poisson editing can mitigate visible seams by blending gradients while preserving generated content and matching original boundary conditions.
  • Table 6 shows that the accuracy drop is not caused by edge artifacts.

A.2. Implementation Details

The study evaluates four pretrained detector architectures using standardized binary-classification training, validation, and saliency-based localization procedures.

  • Model Architectures: Four ImageNet-pretrained architectures are evaluated: ResNet-50, EfficientNet-B0, ViT-B/16, and CLIP ViT-B/32.Each model uses a two-class classification head; CLIP additionally uses staged linear probing and fine-tuning.
  • Training Configuration: CLIP uses AdamW with weight decay 0.01 during fine-tuning, after 3 epochs of linear probing.Its encoder and classifier use differential learning rates during fine-tuning.
  • Training Configuration: All non-CLIP models train for 3 epochs with Adam, learning rate 10^-4, batch size 32, and cross-entropy loss.The data use a stratified 90%/10% train-validation split.
  • Saliency Methods: Grad-CAM localizes CNN predictions, while Attention Rollout recursively aggregates transformer attention for patch-level interpretation.The methods target final convolutional features for CNNs and CLS-to-patch attention for transformers.
  • Localization Metrics: Localization is measured with mIoU and mAP after thresholding saliency maps at 0.5 against resized ground-truth masks.Masks are resized to 224 × 224 using nearest-neighbor interpolation.

A.3. Correlation Analysis

The analysis examines relationships among VAE loss, inpainting error, and high-frequency content while checking detector behavior across datasets and image sub-folders.

  • Correlation Analysis: Correlation matrices, pixel-level histograms, and boxplots characterize relationships among error metrics across datasets.These visualizations cover both cross-dataset structure and pixel-level correlation distributions.
  • Evaluation Context: Reported detector accuracies are not directly comparable to results on fully synthetic images because inpainting synthesizes only part of each image.The paper notes that partial image generation makes inpainting detection inherently more challenging.

A.4. Sub-Dataset Analysis

The analysis connects detector behavior and localization to VAE-induced frequency artifacts, evaluates robustness across datasets and architectures, and contrasts INP-X with edge-blurring alternatives.

  • Spectral Analysis: Standard inpainting produces grid-like high-frequency spectral artifacts, whereas INP-X removes them and more closely matches the spectrum of the original real image.The comparison is made among real, standard-inpainted, and exchanged images.
  • VAE-less Inpainting Methods: Pixel-space RePaint avoids VAE reconstruction artifacts in principle but takes approximately 9 minutes per 256 × 256 image on a P100 GPU.Latent Diffusion can generate higher-resolution outputs in less than 30 seconds with the same number of sampling steps.
  • Multiresolution Analysis: The wavelet attenuation theorem assumes mean-squared-error distortion, an entropy constraint H(Z) ≤ C, and a Gaussian high-rate rate-distortion approximation.The theorem models an autoencoder with latent spatial resolution reduced by r = 2^jc.
  • Multiresolution Analysis: Wavelet analysis predicts statistically significant background wavelet maxima for standard inpainting, while exact background preservation confines maxima to the mask boundary and foreground.The Fourier spectral gap corresponds to fine-scale wavelet attenuation, making the lost high-frequency information spatially localizable.
  • Edge-Artifacts Analysis: INP-X remains substantially harder to detect than standard inpainting even when compared with standard edge blurring.The comparison includes the best-performing detectors and commercial APIs, supporting the conclusion that edge artifacts do not drive the accuracy drop.
  • Localization Reliability: CNN detectors achieve mIoU 0.45–0.49, compared with mIoU 0.37–0.41 for ViT-based models across training conditions.The reported localization gap favors EfficientNet and ResNet-50.
  • Localization Reliability: The paper suggests CNN architectures may be preferable when manipulated-region localization is the primary objective.This conclusion is stated specifically for inpainting forensics.
Loading 2602.00192v1…