PaperlayerView exact revision on arXiv

Source-linked AI summary

See and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesis

Jaehyun Park, Minyoung Ahn, Minkyu Kim, Jonghyun Lee, Jae-Gil Lee, Dongmin Park

arXiv:2602.20951v2cs.CVcs.AI

TL;DR

AI-generated images still contain visual artifacts, and existing artifact-aware methods depend on costly, difficult-to-scale human-labeled datasets. ArtiAgent automatically synthesizes and curates artifact-injected images with rich annotations, enabling artifact-aware VLMs that improve perception and downstream generation and correction applications.

  • Problem

    Visual artifacts remain in diffusion outputs, while VLMs struggle to detect, localize, and explain them and existing datasets depend on costly human annotation.

  • Method

    ArtiAgent combines perception, synthesis, and curation agents to automatically inject plausible artifacts into images and generate training-ready annotations.

  • Results

    ArtiAgent-trained VLMs consistently improve detection, localization, and explanation, matching or exceeding proprietary systems and supporting artifact-free generation and correction.

  • Takeaways & Limitations

    Agentic synthetic data provides scalable supervision for artifact comprehension and supports diffusion sampling guidance and automated artifact correction.

Abstract

from arXiv · show

Despite recent advances in diffusion models, AI generated images still often contain visual artifacts that compromise realism. Although more thorough pre-training and bigger models might reduce artifacts, there is no assurance that they can be completely eliminated, which makes artifact mitigation a highly crucial area of study. Previous artifact-aware methodologies depend on human-labeled artifact datasets, which are costly and difficult to scale, underscoring the need for an automated approach to reliably acquire artifact-annotated datasets. In this paper, we propose ArtiAgent, which efficiently creates pairs of real and artifact-injected images. It comprises three agents: a perception agent that recognizes and grounds entities and subentities from real images, a synthesis agent that introduces artifacts via artifact injection tools through novel patch-wise embedding manipulation within a diffusion transformer, and a curation agent that filters the synthesized artifacts and generates both local and global explanations for each instance. Using ArtiAgent, we synthesize 100K images with rich artifact annotations and demonstrate both efficacy and versatility across diverse applications. Code is available at link.

1. Introduction

Modern diffusion models still produce plausible structural artifacts, while VLMs struggle to detect, localize, and explain them. ArtiAgent addresses the data-scaling gap with automated artifact synthesis and supports artifact-aware generation and correction.

  • Modern diffusion models produce structural artifacts such as six-fingered hands and fused entities, reducing realism and user satisfaction.
  • State-of-the-art VLMs show limited ability to detect, localize, or explain artifacts in AI-generated images.
  • Existing approaches target simple artifacts and rely heavily on costly human annotation, limiting coverage of modern diffusion failures.
  • ArtiAgent uses perception, synthesis, and curation agents to generate plausible artifact-injected images with binary labels, locations, and explanations.
  • ArtiAgent-generated data supports artifact-aware VLM evaluation and downstream applications in artifact-free generation and image correction.

2. Related Work

Prior artifact datasets provide supervision for detection, localization, explanation, and correction, but their human-annotation costs and focus on older degenerate artifacts limit scalability and modern relevance.

  • PAL, SynthScars, and DiffDoctor provide artifact annotations through 10K, 12K, and 25K-seed-based datasets, respectively.
  • Most prior datasets rely on human annotation and artifacts from earlier diffusion models, limiting scalability and relevance to modern failure modes.
  • Existing benchmarks evaluate artifact regions, explanations, and perception tasks, including RichHF-18K, LOKI, and SynthScars evaluation splits.
  • Prior modeling strategies use segmentation, feedback heatmaps, preference learning, and multimodal models for artifact handling.

3. Understanding Visual Artifacts

The paper defines structural visual artifacts as physically implausible object-structure defects and analyzes their prevalence and types in modern diffusion outputs. It categorizes them into duplication, omission, distortion, and fusion.

  • 3.1. Problem Scope: Structural visual artifacts distort an object’s inherent physical structure while preserving the prompt-specified content.
  • 3.1. Problem Scope: The definition excludes text-to-image misalignment, where the generated content itself differs from the prompt.
  • 3.2. Artifact Analysis in Modern Generative Models: The paper categorizes structural artifacts into duplication, omission, distortion, and fusion.
  • 3.2. Artifact Analysis in Modern Generative Models: Human inspection of images generated from 100 MS-COCO captions across five state-of-the-art diffusion models measures artifact frequency and type distribution.
  • 3.2. Artifact Analysis in Modern Generative Models: Modern diffusion models can produce visually high-quality pixels while compromising structural reliability through plausible artifacts.

4. Agentic Pipeline for Artifact Synthesis

ArtiAgent uses coordinated perception, synthesis, and curation agents to identify semantic regions, inject plausible artifacts through patch mappings and DiT inversion-injection, and produce filtered, explained training data.

  • Pipeline Overview: ArtiAgent coordinates perception, synthesis, and curation agents to select injection candidates, create artifacts, and refine outputs with local and global explanations.The pipeline produces paired real and artifact-injected images with metadata for downstream training.
  • Perception Agent: The perception agent decomposes images into entities and subentities, then uses Grounded-SAM masks and containment analysis to associate subentities with parent entities.Peripheral and intermediate subentities provide distinct semantic levels for artifact injection.
  • Synthesis Agent: The synthesis agent combines a toolbox for target-reference patch mappings with an inversion-injection module that applies mappings during image reconstruction.The toolbox supports add, remove, distort, and fuse operations, assigned according to subentity type and overlap.
  • Synthesis Agent: The inversion-injection module manipulates DiT self-attention positional and value embeddings so target patches borrow spatial semantics from reference patches while background semantics remain unchanged.PE injection controls denoising locations, while value injection supplies semantic content; both are restricted to early-to-middle layers to reduce shortcut learning.
  • Curation Agent: The curation agent contrasts paired original and injected images to filter low-quality artifacts and generate localized and global textual explanations.LPIPS-based filtering validates distortion artifacts, while VLM-based filtering validates duplication, omission, and fusion artifacts.
  • Dataset Construction: 50K pairs are collected from COCO, Caltech-101, 11K Hands, and CelebA-HQ, broadening the source distribution from diverse scenes to specific entities.Source images are reconstructed with inversion-restoration to reduce pairwise differences caused by diffusion-generated image traits.

5. ArtiBench: Artifact Detection Benchmark

ArtiBench addresses the limited recency, diversity, and task coverage of earlier artifact benchmarks by evaluating modern generative-model outputs with comprehensive human annotations.

  • Benchmark Design: ArtiBench is designed to reflect current artifact phenomena by including recent generative-model data and comprehensive annotations across multiple artifact-related tasks.The benchmark targets limitations in sample diversity, source recency, and task coverage.
  • Benchmark Design: ArtiBench contains 1K images generated by five diffusion models from prompts sampled from MS-COCO, PartiPrompts, and FuseCap.The models include Stable Diffusion 3.5, FLUX-schnell/dev, Qwen-Image, and Nano-Banana.
  • Annotation: Twelve human annotators label each image with artifact presence, bounding boxes for artifact regions, and concise descriptions of the artifacts.These annotations support artifact detection, localization, and explanation tasks.

6. Experiments

Experiments show that ArtiAgent-generated supervision improves artifact detection, localization, and explanation, scales with more data, and supports downstream artifact-free generation and correction.

  • Main Results: ArtiAgent-trained open-source VLMs consistently outperform vanilla counterparts and match or exceed proprietary systems across detection, localization, and explanation.The evaluation covers ArtiBench and three existing benchmarks.
  • Artifact Binary Detection: 26.5%: ArtiAgent improves InternVL3.5-8B accuracy on artifact binary detection.Overall ArtiBench accuracy remains low, reflecting the difficulty of recognizing subtle, structured failures in modern generated images.
  • Artifact Localization: ArtiAgent consistently enhances open-source VLM spatial grounding, while DiffDoctor’s strong LOKI accuracy does not generalize to newer benchmarks such as ArtiBench.ArtiBench captures artifacts that prior detection models struggle to identify.
  • Artifact Explanation: Fine-tuning on ArtiAgent data improves ROUGE and CSS explanation scores across all evaluated benchmarks.The gains strengthen VLM reasoning and description capabilities for visual artifacts.
  • Data Scaling Effect of ArtiAgent: Performance rises with synthesized-data scale across all three tasks; 1K samples already surpass GPT-5 for localization and explanation, while detection improves through 100K.The scaling pattern indicates sample-efficient supervision for localization and explanation, but larger and more diverse artifacts benefit detection.
  • Comparison with Human-Annotated Supervision: ArtiAgent’s synthetic supervision matches human annotation in detection, performs better in explanation, and is slightly worse in localization.The localization gap is attributed mainly to ArtiAgent’s patch-level label granularity, supporting its cost-effective scalability.
  • Reward-Guided Artifact-Free Generation: A Bradley-Terry reward model trained on clean-versus-artifact pairs steadily raises diffusion reward during search, guiding FLUX-schnell toward images with fewer artifacts.Later-round examples show clearer structures and reduced artifact patterns.
  • Image Correction: An ArtiAgent-trained VLM can localize artifact regions for FLUX inpainting, which produces natural and structurally consistent corrections verified by re-evaluation.The pipeline detects and localizes artifacts, inpaints the specified region, and checks whether the artifact was resolved.

7. Conclusion

ArtiAgent automatically synthesizes richly annotated visual artifacts through diffusion-transformer positional embedding manipulation and integrates perception, synthesis, and curation agents. Fine-tuned VLMs improve artifact understanding and support artifact-free generation and automated correction.

  • 7. Conclusion: ArtiAgent combines perception, synthesis, and curation agents to generate large-scale, richly annotated artifact datasets without human supervision.The framework synthesizes artifacts through positional embedding manipulation in diffusion transformers.
  • 7. Conclusion: Fine-tuned VLMs achieve substantial gains in artifact detection, localization, and explanation, while guiding artifact-free diffusion sampling and automated artifact correction.These results position agentic data synthesis as a pathway for perceiving and mitigating visual artifacts in modern generative models.

A.1. Implementation Details

The synthesis agent uses patch mappings and diffusion-time embedding injection to create localized structural artifacts, then filters and explains the resulting images.

  • Synthesis agent: The synthesis tools create target–reference patch mappings consumed by the inversion–injection module, with geometric priors tailored to duplication, omission, distortion, and fusion.Each mapping specifies where artifact changes occur and which reference semantics are injected there.
  • Artifact tools: The Add Tool selects a nearby, low-overlap perimeter location and duplicates an entire subentity there.Its score penalizes overlap with the source, same-entity foreground, and other same-subentity regions, while favoring smaller offsets.
  • Artifact tools: The Remove Tool maps each target patch to a nearby reference patch from a filtered neighborhood, producing omission artifacts by replacing the subentity with surrounding context.The neighborhood excludes invalid coordinates and can avoid same-subentity collisions.
  • Artifact tools: The Distort Tool applies shuffle, jitter, or strip kernels within the same foreground region, preserving global placement while perturbing local structure.The resulting one-to-one mapping is consumed by inversion–injection to reconstruct a context-consistent distorted object.

A.5. Ablation Studies

Ablations examine diffusion injection choices, while the artifact tools use explicit patch-grid algorithms to generate controlled duplication, omission, distortion, and fusion patterns.

  • Injection ablation: Changing PE injection steps and value-injecting blocks affects the quality of synthesized artifacts.These ablations motivate the selected synthesis-agent configuration.
  • Injection ablation: VLM performance peaks when artifacts are injected for 15 of 25 denoising steps.Fewer steps can fail to inject artifacts, whereas more steps degrade overall image quality.
  • Omission: The Remove Tool builds a local non-target neighborhood, filters same-subentity patches, and assigns each target patch its nearest reference under L1 distance.It outputs the resulting mapping for omission synthesis.
  • Distortion: The Distort Tool forms reference patches with shuffle, Gaussian jitter, or strip shifting before returning a one-to-one target–reference mapping.These kernels perturb internal structure while operating on the subentity region.
  • Fusion: The Fuse Tool constructs a band around overlapping entities, samples seeds, selects valid offsets into an opposite-side pool, and maps boundary patches across entities.This produces structurally implausible fusion along the interface while respecting valid patch locations.

B.1. Benchmark Datasets (§ 5)

The paper introduces ArtiBench and compares it with prior artifact benchmarks, emphasizing current structural artifacts and standardized evaluation across detection, localization, and explanation.

  • Existing benchmarks: Prior benchmarks include RichHF heatmaps, LOKI’s multimodal artifact tasks, and SynthScars’ pixel-level masks, explanations, and category labels.Their sizes and annotation formats differ substantially across datasets.
  • ArtiBench: ArtiBench represents structural artifacts that remain visible in recent, otherwise higher-quality generated images.The benchmark is positioned as a timely evaluation resource for modern diffusion failures.
  • ArtiBench: ArtiBench contains 1K images generated by modern diffusion models and annotated through classification, bounding-box labeling, explanation generation, and expert curation.Its images use five generation models and three prompt sources.
  • Evaluation: The evaluation uses accuracy and macro F1 for binary detection, IoU and pixel-wise F1 for localization, and ROUGE-L and CSS for explanations.Macro F1 averages positive- and negative-class F1 scores, while CSS measures sentence-embedding similarity.
  • Evaluation: All artifact-region representations are converted to pixel-wise binary maps to reduce differences among boxes, polygons, and heatmaps.This supports common localization metrics such as IoU and pixel-wise F1.

B.4. VQA Dataset Structure (§ 6.1)

The VQA dataset derives aligned supervision from clean–artifact image pairs, using separate conversations and supervised training procedures for artifact-aware VLMs.

  • Dataset construction: Each dataset instance is built from an ArtiAgent clean–artifact pair with synthesized annotations for artifact detection, localization, and explanation.The paired structure supports tightly aligned conversations about normal content and artifact regions.
  • Conversation design: Each ArtiAgent instance yields two conversations: one for the clean reconstruction and one for the corresponding artifact image.Tables 7 and 8 define the question-answer templates for the two settings.
  • Training: The VLM training uses two-stage supervised fine-tuning, initially freezing the vision encoder and later unfreezing it for continued training.Both stages use the same VQA dataset, with a smaller learning rate in the second stage.
  • Verifier: A Bradley-Terry artifact verifier is trained to score clean images higher than artifact-injected images.It uses a frozen ViT-B/16 encoder with a lightweight MLP head.
  • Test-time scaling: At inference, the verifier serves as a reward model in best-of-N sampling, retaining the highest-scoring image while exponentially expanding the candidate pool.Each round doubles the number of sampled latent noises.

C.2. Image Correction (§ 6.2.2)

The artifact-correction pipeline uses an ArtiAgent-supervised VLM to localize artifacts, generate clean-scene captions, and verify iterative FLUX inpainting results. Fixed-region, prompt-guided verification coordinates semantic guidance with pixel-level repair while preserving image semantics.

  • VLM-guided correction: The Qwen2.5-VL-7B model, fine-tuned with ArtiAgent supervision, handles artifact localization, clean captioning, and correction verification.The same VLM supports all interactions in the correction loop.
  • Localization: The localization prompt produces a single bounding box B in [x_min, y_min, x_max, y_max] format for deterministic parsing.The box is computed once and remains fixed during subsequent refinement iterations.
  • Captioning: The captioning prompt describes the intended clean scene without mentioning artifacts, providing semantic conditioning for FLUX inpainting inside region B.This guides synthesis of corrected content rather than reproducing the corrupted content.
  • Verification loop: After each inpainting step, the verification prompt checks only region B and returns Yes or No, determining whether the loop continues or terminates.Restricting verification to the localized region prevents drift toward unrelated image areas.
  • Correction procedure: The loop localizes B, generates caption c, repeatedly applies Inpaint(I, B, c), and returns the image when verification indicates no remaining artifact.The VLM supplies localization, semantic guidance, and validation, while FLUX performs pixel-level repair.
Loading 2602.20951v2…