Source-linked AI summary
GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation
Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, Li Yuan
TL;DR
GPT-ImgEval addresses the need for comprehensive evidence about GPT-4o’s image-generation capabilities, weaknesses, and related safety issues. It evaluates generation, editing, and world-knowledge-informed synthesis, then analyzes architecture, failure patterns, multi-round editing, and detectability. GPT-4o shows strong performance across the three tasks, while the analyses identify recurring limitations and detectable artifacts.
Problem
Existing evaluation standards inadequately assess complex semantic understanding and world-knowledge integration, making systematic evaluation of GPT-4o’s generation capabilities and weaknesses necessary.
Method
GPT-ImgEval combines GenEval, Reason-Edit, and WISE evaluations with classifier-based architectural analysis, weakness and artifact studies, Gemini 2.0 Flash editing comparisons, and forensic detectability assessment.
Results
GPT-4o demonstrates strong performance across text-to-image generation, instruction-based editing, and world-knowledge-informed semantic synthesis, while its generated images remain detectable by state-of-the-art image forensic models.
Takeaways & Limitations
GPT-ImgEval provides a benchmark and empirical analyses intended to offer insight, support reproducibility, and guide future image-generation research.
Takeaways & Limitations
GPT-4o exhibits content-preservation inconsistencies, proportion-control and cropping problems, high-resolution refinement limits, complex-scene difficulties, color bias, and non-English text limitations.
Abstract
from arXiv · showhide
The recent breakthroughs in OpenAI's GPT4o model have demonstrated surprisingly good capabilities in image generation and editing, resulting in significant excitement in the community. This technical report presents the first-look evaluation benchmark (named GPT-ImgEval), quantitatively and qualitatively diagnosing GPT-4o's performance across three critical dimensions: (1) generation quality, (2) editing proficiency, and (3) world knowledge-informed semantic synthesis. Across all three tasks, GPT-4o demonstrates strong performance, significantly surpassing existing methods in both image generation control and output quality, while also showcasing exceptional knowledge reasoning capabilities. Furthermore, based on the GPT-4o's generated data, we propose a classification-model-based approach to investigate the underlying architecture of GPT-4o, where our empirical results suggest the model consists of an auto-regressive (AR) combined with a diffusion-based head for image decoding, rather than the VAR-like architectures. We also provide a complete speculation on GPT-4o's overall architecture. In addition, we conduct a series of analyses to identify and visualize GPT-4o's specific limitations and the synthetic artifacts commonly observed in its image generation. We also present a comparative study of multi-round image editing between GPT-4o and Gemini 2.0 Flash, and discuss the safety implications of GPT-4o's outputs, particularly their detectability by existing image forensic models. We hope that our work can offer valuable insight and provide a reliable benchmark to guide future research, foster reproducibility, and accelerate innovation in the field of image generation and beyond. The codes and datasets used for evaluating GPT-4o can be found at https://github.com/PicoTrex/GPT-ImgEval.
1 Introduction
GPT-ImgEval introduces a comprehensive evaluation of GPT-4o’s image generation across generation, editing, and world-knowledge-informed synthesis, alongside analyses of its architecture, limitations, artifacts, and safety. The benchmark reports strong performance across all three tasks while identifying recurring weaknesses and detectable generation artifacts.
- Benchmark scope: GPT-ImgEval evaluates GPT-4o across text-to-image generation, instruction-based editing, and world knowledge-informed semantic synthesis.The benchmark uses GenEval, Reason-Edit, and WISE for these three tasks.
- Benchmark scope: GPT-4o demonstrates strong performance across all three tasks, with accurate compositional reasoning, fine-grained attribute control, and nuanced real-world context understanding.The results are presented as superior image generation and comprehension capabilities over previous models.
- Architecture analysis: A classifier-based analysis consistently identifies GPT-4o’s generated images as diffusion-based, providing empirical evidence for a diffusion image-decoding head.The study also infers potential visual encoders and proposes complete candidate architectures.
- Weaknesses and artifacts: The evaluation identifies limitations in content preservation, image-proportion control, automatic cropping, high-resolution refinement, complex scenes, color bias, and non-English text generation.These findings are framed as areas for future model improvement.
- Comparison and safety: The study compares multi-round editing with Gemini 2.0 Flash and finds GPT-4o-generated images remain detectable by state-of-the-art image forensic models.The report links detectability to visible artifacts, including possible upsampling interpolation artifacts or watermark-like features.
2 GPT-ImgEval Evaluation Benchmark
GPT-ImgEval evaluates GPT-4o across text-to-image generation, instruction-based editing, and world knowledge-informed semantic synthesis using GenEval, Reason-Edit, and WISE. Across these tasks, GPT-4o shows strong compositional control, editing performance, and knowledge-grounded visual synthesis, while analyses identify architectural evidence and practical limitations.
- Benchmark Setup: GPT-ImgEval uses GenEval, Reason-Edit, and WISE to assess compositional text-to-image generation, instruction-based editing, and world knowledge-informed semantic synthesis.GenEval evaluates object co-occurrence, spatial arrangement, counting, and color consistency; Reason-Edit covers seven editing challenges; WISE tests cultural, temporal, spatial, and scientific knowledge.
- Text-to-Image Generation: 0.84 overall on GenEval, with 0.85 for counting, 0.92 for color recognition, 0.75 for spatial localization, and 0.61 for attribute binding.The reported results place GPT4o above both frozen text-encoder methods and LLM/MLLM-enhanced approaches.
- Text-to-Image Generation: GPT-4o produces accurate objects, counts, colors, positions, and attribute relationships across GenEval’s compositional generation examples.The qualitative examples include correctly rendered object numbers, object-specific colors, spatial layouts, and multi-object attribute binding.
- Image Editing: 0.929 on Reason-Edit, exceeding SmartEdit’s 0.572 by +0.357 and surpassing seven other image-editing methods.The benchmark uses GPT Score to assess instruction adherence and consistency of non-edited regions.
- Image Editing: GPT-4o’s Reason-Edit outputs show accurate replacement, removal, and attribute-specific substitution with strong spatial consistency and limited collateral modification.The reported qualitative comparison includes InstructPix2Pix, MagicBrush, and SmartEdit-7B, while noting inconsistencies in dimensions and color tone that GPT-eval may obscure.
- World Knowledge-Informed Semantic Synthesis: On WISE, GPT-4o significantly outperforms specialized T2I and unified MLLM approaches in overall WiScore, combining world-knowledge understanding with high-fidelity image generation.Qualitative examples include a bald eagle for a U.S. national symbol and an accurate enclosed medieval tournament helmet.
- Architectural Analysis: A classifier consistently labels GPT-4o images as diffusion-based, supporting the hypothesis that its decoder uses a diffusion-based head rather than a VAR-like architecture.The paper presents four architectural candidates that differ in visual encoder choice while sharing a diffusion-based image-decoding head.
3 Potential Architectures Behind GPT4o
The paper evaluates competing hypotheses for GPT-4o’s image-generation architecture and uses classifier-based evidence to support a hybrid autoregressive model with a diffusion-based head. It also speculates that GPT-4o uses continuous rather than discrete image tokens.
- Architectural hypotheses: Three architectural hypotheses are proposed to explain GPT-4o’s image-generation mechanism.The central architectural debate concerns the choice of visual generation head.
- Architectural hypotheses: The VAR-based hypothesis predicts images progressively from low to high resolution through next-scale refinement.This hypothesis is motivated by VAR-style generation and the model’s visible generation animation.
- Architectural hypotheses: The hybrid hypothesis combines a transformer-based autoregressive backbone with a diffusion-based head that decodes intermediate visual representations into pixels.Its proposed pipeline is token → transformer → diffusion → image pixels.
- Decoder analysis: The hybrid design is presented as consistent with GPT-4o’s image quality, texture diversity, semantic prompt alignment, and global shifts during localized editing.The proposed explanation links diffusion behavior to visual quality and editing-control difficulties, while the autoregressive stage supports language grounding.
- Decoder analysis: A classifier trained on 10,000 VAR-generated and 10,000 diffusion-head-generated images consistently classified GPT-4o outputs as diffusion-based.The images used identical GenEval prompts, providing empirical evidence for a diffusion head.
- Visual encoder analysis: The paper speculates that GPT-4o uses a continuous image tokenizer rather than a discrete VQ-based tokenizer.The speculation is based on imperfect reconstruction and concerns that VQ can impair comprehension; the exact architecture remains unavailable.
4 Weakness Analysis
The weakness analysis identifies recurring failures in GPT-4o’s image generation and editing, including fidelity, resolution control, localization, complex-scene coherence, and non-English text rendering. The section also frames these issues through failure-case and multi-round-generation comparisons.
- Overview: GPT-4o exhibits recurring artifacts that limit fidelity, consistency, and control in image generation and editing.The artifact summary extends beyond the three benchmark datasets and organizes the observed weaknesses into detailed categories.
- Image consistency: When asked to make no changes, GPT-4o may alter image dimensions through unpredictable aspect-ratio changes, edge cropping, or rescaling.These behaviors constrain applications requiring precise framing or spatial alignment with the original image.
- Resolution control: GPT-4o frequently enhances blurry or low-resolution prompts instead of preserving their requested appearance.The reported tendency suggests a bias toward super-resolution, high-frequency detail, or image enhancement.
- Localized editing: The brush tool can cause unintended global changes because GPT-4o regenerates the entire image even for small localized edits.The affected properties include texture, color, and fine details.
- Complex scenes: GPT-4o struggles with coherent multi-person scenes and object-human interactions, producing abnormal anatomy and spatially implausible overlaps.These failures reflect difficulty with spatial reasoning and consistency under high visual complexity.
- Non-English text: GPT-4o renders English text more clearly than Chinese text in complex scenes, where incorrect fonts and unintended traditional characters remain common.The paper attributes the gap to possible English–Chinese training-data imbalance and the structural complexity of Chinese characters.
5 More Discussion
The comparison with Gemini 2.0 Flash examines editing consistency, instruction comprehension, multi-turn interaction, and speed, while additional analyses assess detectability and safety safeguards. GPT-4o is stronger in editing continuity and consistency, whereas Gemini is faster.
- GPT-4o versus Gemini 2.0 Flash: The comparative evaluation measures editing consistency, instruction comprehension, multi-turn interactions, and computational efficiency.The study compares GPT-4o with Gemini 2.0 Flash across these four aspects.
- Consistency over edits: GPT-4o maintains significantly better consistency than Gemini as the number of edits increases.In a chair-color example, GPT-4o changed only the color, while Gemini also changed shape or position.
- Instruction comprehension: Neither model follows every instruction correctly, but Gemini’s computer-desk failure erased more unintended scene content than GPT-4o’s.GPT-4o removed a decorative wall panel instead of modifying the chair; Gemini removed the panel and additional objects.
- Multi-turn editing: GPT-4o supports continuous multi-turn image editing, whereas Gemini requires manual re-uploading of the previous image at each step.This contrast concerns native support for editing across multiple image states.
- Computational efficiency: Gemini 2.0 Flash is considerably faster than GPT-4o and is therefore more suitable for rapid-response applications.
- Detectability and safety: Most tested AI-image detectors, including Effort and FakeVLM, exceed 95% accuracy on GPT-4o-generated images.The result indicates that photorealistic outputs remain detectable by current state-of-the-art forensic models.
- Detectability and safety: GPT-4o’s consistent sharpening of blurry inputs suggests an internal super-resolution process that may contribute to detectability.The model produced a sharpened high-resolution image even when asked to return a blurry input unchanged.
- Detectability and safety: GPT-4o enforces safeguards against generating children, recognizable faces, and copyrighted materials such as logos.The paper presents these restrictions as part of its image-safety behavior.
6 Conclusion
GPT-ImgEval comprehensively evaluates GPT-4o across generation, editing, and comprehension-guided synthesis, then analyzes its architecture, weaknesses, editing behavior, and output detectability. The report is intended to support insight, reproducibility, and future image-generation research.
- GPT-ImgEval evaluates GPT-4o across generation quality, instruction-based editing, and comprehension-guided generation using GenEval, Reason-Edit, and WISE.
- The report combines benchmark evaluation with architectural analysis, weakness and failure-pattern studies, Gemini comparison, and detectability assessment.
- The authors intend GPT-ImgEval to provide insights, promote reproducibility, and stimulate innovation in image generation and related areas.
7 Appendix
The appendix provides supplemental GPT-4o generation and multi-round editing examples, alongside details of a discriminator-based investigation into GPT-4o’s potential image-generation architecture.
- Supplemental Visual Examples: Supplemental examples extend the visual evaluation with a GPT-4o image generated under the Reason-Edit benchmark.The example is referenced in Figure 12.
- Model-based Discriminator Between VAR and Diffusion: The architecture investigation trains a discriminator to distinguish diffusion-based from autoregressive-generated images.Flux and VAR-Infinity represent diffusion-based and VAR-based image generators, respectively, using default settings for image generation.
- Supplemental Visual Examples: Figure 13 compares multi-round generation between GPT-4o and Gemini-2.0 Flash.The appendix includes additional multi-round editing examples.