Source-linked AI summary
AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation
Xiangfei Sheng, Weidong Zou, Tianjiao Gu, Zhichao Yang, Pengfei Chen, Leida Li
TL;DR
AGI defect diagnosis remains underexplored despite persistent structural and semantic defects in generated images. AGIDefect-4K addresses this gap with hierarchical annotations across 4,000 images from 15 models, while AGIDA jointly performs detection, localization, explanation, and quality prediction; benchmarking still reveals challenges in defect understanding.
Problem
Comprehensive diagnosis of structural and semantic defects in AI-generated images remains underexplored despite their importance to image reliability and authenticity.
Method
The paper introduces AGIDefect-4K and AGIDA, an MLLM-based baseline jointly addressing defect detection, localization, explanation, and quality prediction.
Results
Benchmarking reveals persistent challenges in AGI defect understanding, while AGIDA achieves the highest detection AUC of 0.65 among evaluated methods.
Takeaways & Limitations
AGIDefect-4K provides a resource for advancing AGI defect analysis and quality evaluation, and dedicated defect modeling can benefit quality assessment.
Abstract
from arXiv · showhide
Generative AI can now produce highly realistic images, yet current models still exhibit subtle but critical defects that undermine their reliability. While existing AI-generated image (AGI) evaluation benchmarks have made notable progress, comprehensive AGI defect diagnosis remains underexplored. To bridge this gap, we introduce AGIDefect-4K, a richly annotated dataset of 4,000 images from 15 state-of-the-art generative models spanning both open-source and closed-source systems. AGIDefect-4K features hierarchical defect annotations: (1) detection labels identifying whether defects exist, (2) pixel-level segmentation masks localizing defective regions, and (3) detailed textual explanations characterizing defect types and their perceptual impact. Each image is further annotated with an overall quality score. Building on this, we present AGIDA (AGI Defect Assistant), a baseline framework leveraging Multimodal Large Language Models (MLLMs) for joint defect detection, localization, explanation, and quality prediction. Comprehensive benchmarking on AGIDefect-4K reveals that AGI defect understanding remains challenging, underscoring the value of this dataset. The dataset is publicly available at https://github.com/sxfly99/AGIDefect-4K.
1 Introduction
AGIDefect-4K addresses the underexplored problem of comprehensive AGI defect diagnosis by introducing hierarchical annotations and an MLLM-based baseline for joint analysis.
- AGIs exhibit structural and semantic logic defects, with reported defect rates ranging from 4.5% to 15.2% across state-of-the-art generative models.
- Existing AGI evaluation methods generally treat defects as ancillary rather than conducting dedicated, comprehensive diagnosis.
- The proposed annotation framework covers defect detection, localization, explanation, and overall quality scoring.
- AGIDefect-4K contains 4,000 images from 15 open-source and closed-source models, with hierarchical annotations produced by at least three human experts.
- Benchmarking across MLLMs, defect detection models, and quality assessment models reveals persistent challenges in defect understanding.
2 Related Work
AGI evaluation has progressed from holistic quality scoring toward more granular feedback, but dedicated benchmarks for comprehensive defect diagnosis remain limited.
- Early AGI benchmarks established perceptual quality assessment through numerical ratings, with later datasets expanding scale and adding authenticity or preference-based evaluations.
- Recent work introduced artifact labels, heatmap localization, structural-problem taxonomies, and question-answer evaluations, yet comprehensive dedicated defect diagnosis remains underexplored.
3 AGIDefect-4K Dataset
AGIDefect-4K combines diverse image generation with rigorous hierarchical annotation to characterize AGI defects, their locations, explanations, and perceptual impact.
- Basic Principles: The dataset construction follows diversity and annotation-reliability principles across generative architectures and failure modes.
- Data Collection: A hybrid sample-and-generate strategy combines existing AGI datasets with images generated by multiple proprietary models.
- Annotation: Defect detection uses image-prompt pairs, multiple independent annotators, dispute review, and a 92.3% initial agreement rate.
- Annotation: Defective images receive segmentation masks, descriptions of defect type, location, and impact, plus overall quality scores through independent annotation and panel validation.
- Data Statistics: Over 85% of defective regions occupy less than 5% of image area, making precise localization challenging.
- Data Statistics: Defect descriptions average 50–100 tokens and extend to 200 tokens, indicating detailed annotations beyond simple labels.
- Data Statistics: Flawless images have higher, compact quality-score distributions, whereas defective images score lower with greater spread and partial overlap.
- Data Statistics: Body and face defects receive the lowest quality scores, indicating greater perceptual impact than object or logical defects.
4 AGIDA: A Baseline Model
AGIDA is a baseline MLLM framework that jointly performs defect detection, localization, explanation, and quality prediction. It uses task-specific tokens and complementary task heads and losses to produce these outputs.
- AGIDA jointly performs defect detection, localization, explanation, and quality prediction.
- Three task-specific tokens, <DET>, <SEG>, and <QUA>, enable simultaneous detection results, localization masks, quality scores, and textual descriptions.
- The framework predicts binary defective status from the <DET> token and defective-region masks from the <SEG> token with visual features from a frozen SAM encoder.
- Quality prediction enhances the <QUA> representation with defect-aware information from <DET> through cross-attention before producing a scalar quality score.
- AGIDA is trained end-to-end with four complementary losses for detection, masks, text generation, and quality prediction.
- LoRA adapts the multimodal LLM components while the SAM encoder remains frozen and the task-specific heads and projection components remain trainable.
5 Experiments
Experiments evaluate AGIDA and competing models across detection, localization, explanation, and quality prediction on AGIDefect-4K. AGIDA performs strongly on detection, explanation precision, localization, and quality assessment, while general-purpose MLLMs struggle with fine-grained defect understanding.
- Evaluation Metrics: AUC measures detection, F1, GIoU, and CIoU measure localization, GPT-4o scores explanation completeness and preciseness, and SRCC and PLCC assess quality prediction.Explanation scores average completeness and preciseness ratings from 0 to 2.
- Detection and Explanation Performance: 0.65 AUC is achieved by AGIDA, exceeding zero-shot MLLMs that mostly score 0.23–0.38 and Gemini-3.1-pro’s 0.49.The result is reported for a 4B-parameter model and attributed to task-specific fine-tuning.
- Detection and Explanation Performance: Completeness generally exceeds preciseness across models, indicating that accurately characterizing defect types and details remains difficult.AGIDA attains the highest preciseness and competitive completeness of 0.89, producing the second-best overall explanation score.
- Localization Performance: AGIDA substantially improves all localization metrics over fine-tuned IMDL methods and LISA through joint detection-localization training.IMDL methods have limited effectiveness, while LISA lacks binary detection capability and trains only on defective images.
- Quality Assessment: AGIDA achieves the highest PLCC and SRCC among fine-tuned quality-assessment methods, with cross-attention improving the quality branch over its ablated variant.The cross-attention mechanism injects defect detection features into quality prediction.
- Case Study: Case studies show general-purpose MLLMs missing defects or identifying the wrong regions, whereas AGIDA correctly detects fused fingers and a missing leg with aligned masks.AGIDA also correctly classifies a flawless image as defect-free, avoiding false positives.
6 Conclusion
The paper presents AGIDefect-4K for comprehensive AGI defect diagnosis and AGIDA as a joint baseline across four tasks. Benchmarking finds fine-grained defect understanding remains limited in current MLLMs, while defect modeling benefits quality assessment.
- Conclusion: AGIDefect-4K contains hierarchical annotations for detection, localization, explanation, and quality scoring across 4,000 images from 15 generative models.The dataset supports comprehensive AGI defect diagnosis.
- Conclusion: AGIDA jointly addresses detection, localization, explanation, and quality prediction using a cross-attention mechanism linking defect detection with quality prediction.The framework is presented as a baseline model.
- Conclusion: Benchmarking shows current MLLMs remain limited in fine-grained defect understanding, while dedicated defect modeling can benefit quality assessment.The conclusion positions AGIDefect-4K as a resource for advancing defect analysis and quality evaluation.