PaperlayerView exact revision on arXiv

Source-linked AI summary

From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering

Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Jing-Hao Xue, Hao Li, Salman Khan, Zhiqiang Shen

arXiv:2603.20193v1cs.CVcs.AIcs.LG

TL;DR

Existing benchmarks often use coarse masks that misalign with actual tampered pixels, missing off-mask edits and including untouched regions. The paper introduces PIXAR, a pixel-grounded, semantic- and language-aware benchmark and training framework, and reports improved localization and semantic consistency alongside failures of mask-only evaluation. Its scope includes a stage-wise benchmark that is continually expanded and excludes fully synthetic images as a separate category.

  • Problem

    Existing mask-based benchmarks misalign with true edits by labeling unchanged regions as tampered and overlooking consequential off-mask artifacts.

  • Method

    The paper derives thresholded per-pixel difference labels, builds PIXAR with diverse manipulation types and semantic annotations, and trains detectors for localization, classification, and descriptions.

  • Results

    PIXAR-7B-Lite raises IoU from 6.9% to 14.9% and Top-1 Acc from 10.6% to 29.5% over SIDA-7B, while lower τ improves localization.

  • Takeaways & Limitations

    Pixel-grounded evaluation exposes micro-edit and off-mask failures and provides a more realistic standard for tamper localization, semantic classification, and description.

  • Takeaways & Limitations

    The project is stage-wise, and images are continually expanded with more sources in further versions.

Abstract

from arXiv · show

Existing tampering detection benchmarks largely rely on object masks, which severely misalign with the true edit signal: many pixels inside a mask are untouched or only trivially modified, while subtle yet consequential edits outside the mask are treated as natural. We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task. First, we introduce a taxonomy spanning edit primitives (replace/remove/splice/inpaint/attribute/colorization, etc.) and their semantic class of tampered object, linking low-level changes to high-level understanding. Second, we release a new benchmark with per-pixel tamper maps and paired category supervision to evaluate detection and classification within a unified protocol. Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity, and further measure tamper meaning understanding via semantics-aware classification and natural language descriptions for the predicted regions. We also re-evaluate the existing strong segmentation/localization baselines on recent strong tamper detectors and reveal substantial over- and under-scoring using mask-only metrics, and expose failure modes on micro-edits and off-mask changes. Our framework advances the field from masks to pixels, meanings and language descriptions, establishing a rigorous standard for tamper localization, semantic classification and description. Code and benchmark data are available at https://github.com/VILA-Lab/PIXAR.

1 Introduction

The paper argues that coarse mask-based tampering benchmarks misrepresent true edit footprints and proposes PIXAR, a pixel-grounded benchmark with semantic supervision and more reliable evaluation.

  • Motivation: Mask-based labels include unchanged or trivially changed pixels and miss consequential edits extending beyond object boundaries.This misalignment conflates unedited pixels with tamper evidence and distorts detector training and measurement.
  • Motivation: Pixel-level difference maps expose untouched pixels falsely labeled as tampered and edited pixels incorrectly labeled as real.These errors can penalize genuine artifact detection while rewarding models that fit coarse shapes.
  • PIXAR formulation: PIXAR thresholds per-pixel differences with τ to create controllable labels that separate edit localization from edit intensity.Small τ emphasizes micro-edits, whereas larger τ emphasizes conservative, high-confidence changes.
  • Benchmark: PIXAR provides over 380K curated training pairs and a 40K-pair test set with pixel-level and semantic-level labels.The benchmark covers eight editing strategies and manually links tampered targets to semantic classes.
  • Contributions: The paper contributes a pixel-aware training framework and realistic metrics that evaluate localization, semantic classification, and tamper descriptions.Its re-evaluation reveals micro-edit and off-mask failures and establishes stronger baselines.

2 Related Work

Prior work spans broad real-versus-fake classification and increasingly fine-grained localization, but PIXAR organizes tampering evaluation around diverse, faithful, and precise pixel-level data.

  • Tampered image datasets: Earlier benchmarks primarily target full-image generation with binary real-versus-fake classification.Recent work has shifted toward fine-grained tampering localization as manipulations become subtler.
  • Tampered image datasets: SID-Set advances tampering localization using Stable Diffusion inpainting on social media images, while PIXAR addresses limitations of coarse masks.PIXAR emphasizes pixel-level and semantic-level labels rather than only region-based localization.
  • Tampered image detection: Tampering detectors use CNNs, Transformers, frequency-domain features, reconstruction learning, attention fusion, and CLIP representations.Methods with strong performance on seen generators can still generalize poorly to unseen generators.
  • PIXAR: PIXAR is designed around diversity, fidelity, and precision, incorporating eight manipulation types, fidelity checks, and precise true-tampering labels.Its dataset construction supports training and evaluation of pixel-grounded tampering detectors.

3 Benchmark Construction

PIXAR constructs a large, diverse tampering benchmark with pixel-level labels, semantic supervision, and rigorous checks for manipulation effectiveness, alignment, and fidelity.

  • Benchmark design: PIXAR contains over 380K curated training image pairs and a balanced 40K-pair test set with pixel-level and semantic-level labels.Each pair includes original and tampered images, metadata, and label information.
  • Manipulation diversity: The benchmark defines eight manipulation types to better reflect diverse and practical real-world tampering scenarios beyond inter-class replacement.The manipulation taxonomy was informed by analysis of large-scale Internet images.
  • Difficulty control: Tampered size and complexity are varied to test detection across subtle edits, larger artifacts, and sequential multi-object manipulations.Smaller edits leave subtler traces, while complexity captures compositional and iterative forgery structure.
  • Effectiveness checks: A filtering pipeline removes near-zero, unintended global, and semantically incorrect edits through geometric rectification, magnitude checks, and semantic correctness checks.Rectification aligns generated and original images before recomputing pixel differences.
  • Fidelity assessment: Automated scoring retains candidates rated at least 9/10 by Qwen3, followed by human review retaining samples with realism scores of at least 4/5.Ten human experts perform the manual review.
  • Fidelity assessment: PIXAR samples exhibit substantially higher fidelity than SID-Set samples, providing a more reliable foundation for tampered-image detection.The comparison is based on randomly sampled images from both datasets.
  • Label construction: Pixel labels are derived from absolute per-pixel differences and thresholded with τ, allowing sensitivity to micro-edits or conservative emphasis on high-confidence changes.The resulting Mτ provides controllable spatial support for edits and can be paired with semantic supervision.
  • Label construction: Each benchmark quadruple includes source and tampered images, a raw difference map, a recommended binary label Mτ, metadata, and detailed manipulation descriptions.Alternative labels for other τ values can be derived from the raw difference map.

4 Training Framework

PIXAR’s training framework jointly predicts pixel-level tampering, semantic object labels, image-level real/tampered status, and natural-language descriptions using multiple supervised objectives.

  • The detector produces a per-pixel tamper map, multi-label semantic predictions, and natural-language descriptions of tampering artifacts.
  • Semantic supervision supports one or multiple tampered objects, including paper-specific labels such as “Real / Tampered OBJ Head.”
  • Pixel-wise BCE and Dice losses supervise localization against the thresholded pixel label Mτ and refine connected-component mask boundaries.
  • A global detection head uses the ⟨CLS⟩ representation to classify images as real or tampered.
  • The framework generates human-readable tamper descriptions with a multimodal causal language model and combines five losses through weighted optimization.
  • The fully synthetic category from SIDA is excluded because its all-pixel tampering case is covered by pixel-level detection.

5 Experiments

Experiments evaluate PIXAR on balanced pixel-level and semantic labels, compare it with existing detectors, test generalization across generative sources, and analyze training choices. PIXAR improves localization and semantic alignment, while lower τ and balanced loss weights support performance.

  • 5 Experiments: The 40K-image test set is balanced across tampered classes, types, and sizes and includes pixel-level and semantic-level labels.
  • 5 Experiments: Evaluation combines pixel-level Recall, F1-Score, and AUC with semantic classification to assess localization and interpretive understanding.
  • 5.2 Evaluation on PIXAR: 14.9% IoU and 29.5% Top-1 Acc are achieved by PIXAR-7B-Lite versus 6.9% IoU and 10.6% Top-1 Acc for SIDA-7B.
  • 5.2 Evaluation on PIXAR: PIXAR models consistently outperform CnnSpot, AntifakePrompt, and SIDA baselines on binary classification and localization evaluation.
  • 5.2 Evaluation on PIXAR: 11.7% IoU and 26.0% Top-1 accuracy on GPT-Image-1.5 make it the most challenging evaluated generative source, while Qwen-generated images are easiest.
  • 5.3 Ablation Study: Higher training τ hampers localization, while consistent lower τ values yield superior performance and validate τ = 0.05 as a discriminative supervision signal.
  • 5.3 Ablation Study: λsem = 0.5 balances semantic supervision with other objectives, whereas lower and higher values marginally reduce Top-1 accuracy.
  • 5.3 Ablation Study: λtext = 3.0 balances text quality and core detection performance, while excessive weighting degrades semantic accuracy.

6 Conclusion

PIXAR reframes VLM tampering around controllable pixel-level labels, semantic understanding, and language descriptions, while showing that mask-only protocols mis-score detectors on subtle edits.

  • 6 Conclusion: PIXAR derives per-pixel difference maps and thresholds them with τ to obtain controllable labels Mτ for tamper localization.Small τ emphasizes micro-edits, whereas larger τ emphasizes conservative, high-confidence changes.
  • 6 Conclusion: The benchmark combines original and tampered images, metadata, raw difference maps, recommended Mτ labels, and language descriptions across 8 diverse manipulations.It contains more than 420K image pairs and supports flexible supervision.
  • 6 Conclusion: The framework adds pixel-aware localization, semantics-aware classification, and natural language descriptions to tamper detection and understanding.These components establish a standard beyond mask-only localization.
  • 6 Conclusion: State-of-the-art detectors are ill-scored by mask-only protocols, especially for micro-edits and off-mask changes.The reported failure modes motivate a more realistic and reliable evaluation standard.

B.1 Image Generation

PIXAR constructs diverse, realistic tampered images using high-fidelity generative models, multiple manipulation types, varied edit sizes, multi-object compositions, and quality-control checks.

  • B.1 Image Generation: Qwen-Image VLMs showed higher perceptual fidelity and precise editing than the compared open-source models, so PIXAR uses them for training-data generation.The comparison emphasizes texture realism, boundary coherence, and semantic consistency.
  • B.1 Image Generation: PIXAR integrates eight editing types, including mask-free object addition and background change, while also including challenging intra-class replacement.Intra-class replacement preserves object pose, scale, and contextual consistency, making it visually credible and difficult to detect.
  • B.1 Image Generation: The benchmark includes 387,810 training pairs and 41,781 test pairs, with three mask-conditioned training manipulations forming the Lite subset.The mask-conditioned manipulations are intra-class replacement, inter-class replacement, and object removal.
  • B.1 Image Generation: Tampered size is defined by the absolute number of tampered pixels and categorized as small below 23,000, medium from 23,000 to below 50,000, or large at least 50,000.Small edits induce subtle artifacts, whereas large edits typically introduce substantial semantic changes.
  • B.1 Image Generation: A multi-edit subset sequentially applies K ∈{2, 3} distinct manipulation types to reflect compositional forgery pipelines.This protocol increases complexity beyond single-object editing.
  • B.1 Image Generation: Global rectification aligns generated images to originals before pixel comparison, with boundary-artifact removal and fallback to unaligned images when homography estimation fails.Boundary filling is aborted when the detected boundary region exceeds 10%.
  • B.1 Image Generation: Samples with tampered size ≤2,480 or ≥184,500 are discarded because they usually represent negligible modifications or unintended global repainting.Only samples within [2,480, 184,500] are retained for this check.
  • B.1 Image Generation: Human experts retain only generated images receiving a realism score of at least 4 out of 5.Filtered and retained examples are visualized by manipulation type.

B.4 Label Reliability Checks

PIXAR checks agreement between pixel-level labels and semantic masks, while filtering labels that are misaligned, spatially dispersed, or structurally uninformative.

  • B.4 Label Reliability Checks: Similar colors and textures between replaced and original objects can make pixel labels diverge from semantic tampering annotations.This creates discrepancies between pixel-level and semantic-level labels in intra-class replacement.
  • B.4 Label Reliability Checks: PIXAR discards samples whose tampered-pixel overlap with the input mask falls below a predefined threshold.Higher overlap ratios correspond to improved consistency between pixel-level and semantic annotations.
  • B.4 Label Reliability Checks: Spatially dispersed pixel labels are filtered because background speckles and unstructured noise are structurally uninformative.The filtering uses concentration measures computed from the binary label map Mτ.
  • B.4 Label Reliability Checks: The grid coverage ratio rgrid measures the smallest fraction of 10 × 10 grid cells covering 80% of tampered pixels.Smaller rgrid indicates greater global compactness; larger values indicate dispersion.
  • B.4 Label Reliability Checks: The local density score rdens is the median of a 7 × 7 mean-filtered binary mask and measures local spatial coherence.Higher rdens indicates that tampered pixels are surrounded by other tampered pixels.
  • B.4 Label Reliability Checks: Maps are classified as Concentrated or Diverse using fixed cases based on rgrid, rdens, and a tie-break score, with Diverse samples discarded.The decision cases are specified in Table 13.

B.5 Text Description

PIXAR generates concise, semantically aligned descriptions from structured metadata, including ordered compositions for multi-edit samples.

  • B.5 Text Description: A template-based instruction design maps structured metadata to a single-sentence edit description for each tampered image.For multi-edit samples, descriptions are concatenated in the applied order.
  • B.5 Text Description: The constrained description design produces concise, semantically aligned text that is easier for models to condition on and humans to interpret.

C Balanced Test Data Construction

PIXAR constructs a balanced test set across tampered size, object class, and manipulation type, while using diverse generative models to assess generalization. The final test set contains over 40K image pairs and includes all eight manipulation types.

  • Balanced dimensions: The test set balances tampered size, object class, and manipulation type, with over 40K image pairs annotated at pixel level.It also incorporates samples generated by six state-of-the-art generative models for generalization evaluation.
  • Tampered size: Tampered areas follow an approximately 4:3:3 Small:Medium:Large distribution, preserving substantial representation of difficult small manipulations.Small, medium, and large areas are defined as <23,000, 23,000–50,000, and ≥50,000 pixels, respectively.
  • Dataset statistics: Figure 18 reports training and test distributions over manipulation size, manipulated object class, and manipulation type.The figure presents these distributions separately for the training and test partitions.
  • Tampered class: The dataset downsampled overrepresented COCO categories to reduce dominance by head classes such as person and car.The original person category alone accounts for approximately 30% of COCO; balancing aims to evaluate generalization across diverse classes.
  • Tampered type: The test set covers eight manipulation types, including replacement, removal, addition, color, motion, material, and background changes.The listed types include both intra-class and inter-class replacement; inter-class replacement and object removal have lower generation success rates because of complex context blending.

D Pixel Localization Metrics

PIXAR evaluates pixel-level tamper localization with complementary metrics covering sensitivity, precision–recall balance, discrimination, and spatial overlap. Together, these metrics compare predicted pixel labels with benchmark ground-truth tamper maps.

  • Metric suite: The benchmark evaluates localization with Recall, F1-Score, AUC, g-IoU, and IoU.These metrics jointly assess detection sensitivity, precision–recall trade-off, threshold-independent discrimination, and spatial alignment.
  • Recall: Pixel-level recall measures the fraction of truly tampered pixels that the detector correctly identifies.True positives and false negatives denote correctly detected and missed tampered pixels, respectively.
  • F1-Score: F1-Score provides a harmonic balance between precision and recall.Precision is defined from true positives and false positives, while recall accounts for true positives and false negatives.
  • AUC: AUC measures detector discriminability independently of a specific threshold by sweeping thresholds from 0 to 1.The sweep computes true-positive and false-positive rates from pixel-level predictions.
  • Spatial alignment: IoU directly measures spatial overlap between predicted and ground-truth tampered pixels, while g-IoU averages IoU across tampered samples.The implementation uses a small ε to avoid division by zero.
Loading 2603.20193v1…