Source-linked AI summary
DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection
Hai Ci, Ziheng Peng, Pei Yang, Yingxin Xuan, Mike Zheng Shou
TL;DR
AIGC benchmarks largely classify whole images, leaving systematic evaluation of diffusion-edit localization limited. DiffSeg30k provides a pixel-annotated dataset, and experiments show segmentation models face challenges in multi-turn semantic segmentation and image robustness while also performing strongly as whole-image classifiers and generalizing to unseen generators.
Problem
AIGC benchmarks largely classify whole images, while systematic evaluation of precise localization for diffusion-based edits remains limited.
Source: §1 · p. 2Method
DiffSeg30k is a 30k-image, pixel-annotated dataset built with a VLM-guided pipeline for selecting candidate regions and generating editing prompts.
Source: §1 · p. 2, §3.1 · p. 2Results
Across three baselines, multi-turn semantic segmentation remained challenging and models were sensitive to image transformations, yet segmentation models outperformed dedicated AIGC classifiers and generalized strongly to unseen generators.
Source: §1 · p. 2, §4.2 · p. 8Takeaways & Limitations
DiffSeg30k provides a foundation for systematic study of diffusion-edit localization and model attribution.
Source: §5 · p. 8The pipeline can produce low-quality edits from incomplete diffusion editing or mask errors, although severe artifacts are automatically discarded.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Diffusion-based editing enables realistic modification of local image regions, making AI-generated content harder to detect. Existing AIGC detection benchmarks focus on classifying entire images, overlooking the localization of diffusion-based edits. We introduce DiffSeg30k, a publicly available dataset of 30k diffusion-edited images with pixel-level annotations, designed to support fine-grained detection. DiffSeg30k features: 1) In-the-wild images--we collect images or image prompts from COCO to reflect real-world content diversity; 2) Diverse diffusion models--local edits using eight SOTA diffusion models; 3) Multi-turn editing--each image undergoes up to three sequential edits to mimic real-world sequential editing; and 4) Realistic editing scenarios--a vision-language model (VLM)-based pipeline automatically identifies meaningful regions and generates context-aware prompts covering additions, removals, and attribute changes. DiffSeg30k shifts AIGC detection from binary classification to semantic segmentation, enabling simultaneous localization of edits and identification of the editing models. We benchmark three baseline segmentation approaches, revealing significant challenges in semantic segmentation tasks, particularly concerning robustness to image distortions. Experiments also reveal that segmentation models, despite being trained for pixel-level localization, emerge as highly reliable whole-image classifiers of diffusion edits, outperforming established forgery classifiers while showing great potential in cross-generator generalization. We believe DiffSeg30k will advance research in fine-grained localization of AI-generated content by demonstrating the promise and limitations of segmentation-based methods. DiffSeg30k is released at: https://huggingface.co/datasets/Chaos2629/Diffseg30k
1. Introduction · 2. Related Work
DiffSeg30k addresses the gap between whole-image AIGC classification and precise localization of diffusion edits with a multi-turn, pixel-annotated dataset. Its preliminary findings show promise for segmentation-based detection while highlighting robustness challenges.
1. Introduction
- Whole-image AIGC benchmarks [6] [48] [80] and traditional forgery-localization benchmarks [20] [21] [32] [60] [79] leave diffusion-edit localization underrepresented.Localized diffusion edits can be subtle and realistic, challenging content verification and digital forensics [34] [47] [52].Source: §1 · p. 2
- DiffSeg30k provides 30k pixel-annotated diffusion-edited images spanning eight models, up to three edits per image, and context-aware additions, removals, and attribute changes.Its VLM pipeline identifies meaningful regions and assigns the diffusion model used for each edit, enabling semantic segmentation that localizes edits and identifies their models.Source: §1 · p. 2
- Preliminary experiments find that model capacity matters, multi-turn semantic segmentation remains challenging, and models are sensitive to resizing and JPEG compression.The work benchmarks FCN, SegFormer, and Deeplabv3+ on binary and semantic segmentation tasks.Source: §1 · p. 2
- Segmentation models serve as strong whole-image classifiers, outperform established classifiers, and generalize well to unseen generators.These findings suggest potential for more generalizable detection models.Source: §1 · p. 2
2.1. Image Forgery Localization
2.2. AIGC Detection
- Prior AIGC datasets and methods mainly classify whole images, while several local-edit datasets include at most three diffusion models and one edit per image.DiffSeg30k instead covers general images, eight diffusion models, and multi-turn edits, supporting semantic segmentation beyond first-classify-then-binary-segmentation approaches [21] [70].Source: §2.2 · p. 2
3. DiffSeg30k benchmark
DiffSeg30k is built through automated region selection, context-aware prompting, and sequential localized diffusion edits, with quality filtering and deliberate balancing across edits and image bases.
3.1. Dataset construction
- The pipeline uses Qwen2.5-VL [3] and Grounded-SAM [44] to select object masks, generate contextual prompts, and apply localized diffusion edits sequentially for up to three turns.It removes redundant masks with IoU greater than 70% and randomly selects one to three masks for editing.Source: §3.1 · pp. 2–3, Figure 2 · p. 3
- The collection balances object types, mask areas, three edit types, eight diffusion models, multi-turn edits, and real versus AI-generated base images.The first, second, and third editing rounds occur at an approximate 1:4:5 ratio; about half of the bases are real COCO images and half are AI-generated from COCO prompts.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
3.3. Dataset statistics
- Food was the most frequently edited category (14,308), and editing frequency decreased smoothly as object area increased across a broad mix of categories.Frequently edited categories also included vehicles, furniture, nature scenery, animals, humans, and signs.Source: Figure 3 · p. 4, §3.3 · p. 4
3.2. Sanity check
- The pipeline filters out edits with severe artifacts, using Qwen2.5-VL [3] to score quality from 0–5 and discard scores below 3.The authors attribute occasional low-quality edits to limited diffusion editing capability or Grounded-SAM mask errors, a tradeoff of broader, more complex scenarios than OnlineDet [16] and wFake.Source: §3.2 · p. 4
4. Benchmarking baselines
The benchmark shows that segmentation models can localize diffusion edits and generalize across generators, but semantic attribution and robustness to image transformations remain challenging. Repurposed as whole-image classifiers, they outperform dedicated AIGC detectors in the reported comparisons.
4.2. Results
- Deeplabv3+ and SegFormer achieve binary-localization mIoU of 0.974 and 0.961, respectively, while FCN-8s struggles on both localization tasks.The authors identify representational capacity as important for performance.Source: Table 2 · p. 5, §4.2 · p. 5
- JPEG compression and resizing substantially degrade both models, with semantic segmentation almost failing under JPEG compression.Deeplabv3+ shows partial resilience for binary segmentation, while SegFormer shows some robustness when resized to higher resolutions.Source: Table 3 · p. 5, §4.2 · pp. 5–6
4.1. Experimental setup
- The evaluation trains FCN-8s, SegFormer-B2, and Deeplabv3+ on an 8:2 train-validation split for binary localization and semantic attribution.Training uses 512×512 resolution and includes 5k unedited COCO images to help suppress false positives; reported results are on the validation set.Source: §4.1 · p. 5
4.2. Results
- Semantic segmentation is harder than binary localization: mIoU falls to 0.825 for SegFormer, 0.76 for Deeplabv3+, and 0.23 for FCN-8s.FCN-8s often predicts background, while SegFormer particularly struggles with Flux and Deeplabv3+ with Kolors and SDXL; Deeplabv3+ also often confuses SD2 and Flux.Source: §4.2 · p. 5, Figure 5 · p. 7
- On unseen generators, SegFormer’s mIoU stays above 0.9 and Deeplabv3+’s above 0.8 across three evaluated combinations.Both models were trained on six diffusion models and tested on two unseen models; the authors note they were not specifically designed for this generalization.Source: Table 4 · p. 7, §4.2 · p. 7
- SegFormer and Deeplabv3+ outperform dedicated AIGC classifiers when adapted for whole-image detection, with fine-grained supervision proposed as the advantage.An image is labeled edited when the non-background mask exceeds 1% of its area; the authors attribute the classifiers’ advantage to pixel-level supervision and training on images with multiple interleaved edits.Source: Table 5 · p. 7, Table 6 · p. 7, §4.2 · pp. 7–8
5. Conclusion
DiffSeg30k provides a multi-turn diffusion-editing dataset for studying edit localization and model attribution. The paper’s experiments highlight segmentation methods’ strengths and limitations, supporting further research in robust AIGC localization.
5. Conclusion
- DiffSeg30k provides a foundation for systematically studying diffusion-edit localization and model attribution.The dataset is multi-turn and derived from COCO.Source: §5 · p. 8
- Extensive experimental analyses highlight both the strengths and current limitations of segmentation methods.The authors frame these findings as groundwork for further research in robust AIGC localization techniques.Source: §5 · p. 8
- The dataset is distributed under the COCO license without additional restrictions.Source: §5 · p. 8
Detection · 6. Relation to global editing
Global editing regenerates whole images, unlike DiffSeg30k’s localized inpainting-style edits. This distinction motivates pixel-level detection, while segmentation models may also support whole-image detection of global edits.
6. Relation to global editing
- Global image-editing methods [56] [57] [65] [71] synthesize the entire output image from scratch, so they lack a local editing region.The paper groups these methods with standard (“vanilla”) image generation approaches because all pixels are fully regenerated.Source: §6 · p. 13
- DiffSeg30k focuses on localized inpainting-style edits, motivating a shift from whole-image classification to per-pixel attribution.The paper describes per-pixel attribution as a more general and fine-grained detection paradigm.Source: §6 · p. 13
- Experiments show that segmentation models can also serve as strong whole-image classifiers, laying a foundation for improving global-editing detection.Source: §6 · p. 13
7. Do LoRAs affect localization?
The baseline segmentation model generalizes strongly to SDXL edits made with Hyper-SD LoRA, with an approximately 0.03 mIoU drop.
7. Do LoRAs affect localization?
- The segmentation model generalizes strongly to SDXL-LoRA edits, with only an approximately 0.03 mIoU drop.Deeplabv3+ was trained on images edited by eight vanilla generators, including SDXL, and evaluated on SDXL- and SDXL-LoRA-edited validation masks; LoRA edits counted as correctly predicted when classified as SDXL.Source: Table 7 · p. 13, §7 · p. 13
8. Quality assessment
The dataset pipeline uses Qwen2.5-VL [3] to strictly filter low-quality edits, rejecting roughly half of generated samples. The assessment also notes that this strict filtering can discard some good edits when scores reflect the whole image rather than the edited region.
Quality assessment
- Qwen2.5-VL [3] evaluates each edited image, and images scoring below 3 are discarded, removing roughly 50% of generated samples; the intentionally strict threshold yields a higher-quality resulting dataset.Source: §8 · p. 13
- Typical low-quality edits stem from weak diffusion editing or inaccurate masks, and whole-image scoring can occasionally filter out higher-quality edits.The authors consider this acceptable under their preference for stricter filtering.Source: §8 · p. 13
9. Segmentation on different base images
Deeplabv3+ performs generally better on AI-based images than on real-image bases.
9. Segmentation on different base images
- Deeplabv3+ performs generally better on AI-based images than on real-image bases.The section reports this comparison for the real- and AI-based subsets.Source: §9 · p. 13
10. Additional visualization of edited images
Figures 9 and 10 visualize edited DiffSeg30k images, illustrating diversity in edit types and visual characteristics across diffusion models.
10. Additional visualization of edited images
- Figures 9 and 10 show edited DiffSeg30k images with diverse edit types and visual characteristics across diffusion models.Source: §10 · p. 13
COT 1
This section presents chain-of-thought examples for image-quality assessment, VLM ratings of low-quality edits, and additional paired examples from DiffSeg30k. It also identifies insufficient model editing capability as a limitation.
COT 1
- Figure 7 presents chain-of-thought examples for image-quality assessment.Source: Figure 7 · p. 14
- The section identifies insufficient model editing capability as a limitation.The passage provides no further explanation of this limitation.Source: COT 1 · p. 15
- Figure 8 shows typical low-quality edits alongside VLM ratings.Source: Figure 8 · p. 15
- Figures 9 and 10 show original images with red edit-region contours paired with editing results.Source: Figure 9 · p. 16, Figure 10 · p. 17