Source-linked AI summary
CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions
Chonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao
TL;DR
Complex creative image editing remains difficult to evaluate systematically because existing methods provide limited coverage and interpretability. CREval introduces a fully automated QA-based pipeline and CREval-Bench, then finds closed-source models generally lead while automated scores align strongly with human judgments.
Problem
Existing evaluation methods lack a systematic, human-aligned framework for assessing complex and creative image editing.
Method
CREval combines a fully automated QA-based pipeline with CREval-Bench, evaluating instruction following, visual consistency, and visual quality across creative editing tasks.
Results
Seedream 4.0 achieves the highest overall performance, while closed-source models generally outperform open-source alternatives on complex creative editing tasks.
Takeaways & Limitations
CREval provides a reliable foundation for benchmarking, model selection, and future research in creative image manipulation.
Abstract
from arXiv · showhide
Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question-answer (QA)-based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval's automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research.
1. Introduction
CREval addresses the difficulty of evaluating creative image editing under complex instructions, where models may fail in instruction following, visual consistency, and visual quality. It introduces an automated QA-based framework and benchmark, then finds Seedream 4.0 leads evaluated models overall while closed-source models generally outperform open-source alternatives.
- Complex creative editing remains difficult because models can misinterpret instructions, lose key visual characteristics, and produce artifacts or distortions.
- CREval introduces a fully automated, multidimensional evaluation pipeline for complex instruction-based creative image manipulation.
- CREval-Bench covers three categories and nine creative dimensions, with over 800 editing samples and 13K evaluation queries.
- The framework decomposes evaluation into Instruction Following, Visual Consistency, and Visual Quality using targeted QA pairs rather than opaque holistic MLLM scoring.
- Seedream 4.0 achieves the highest overall performance, Qwen-Image-Edit-2509 ranks second, and both outperform GPT-Image-1 in aggregate performance.
- User studies show strong consistency between CREval scores and human preference judgments, supporting the framework’s reliability and robustness.
2. Related Work
Prior image-editing benchmarks increasingly address evaluation quality and complexity, but many remain focused on common operations or structured difficulty rather than open-ended creative editing. CREval-Bench responds to this gap by targeting free-style creative challenges.
- Instruction-based image editing seeks semantic modifications while balancing instruction adherence, structural fidelity, and task generalization.
- The field has also shifted from single diffusion models toward multimodal Transformer and Flow Matching architectures for instruction-based editing.
- MagicBrush established large-scale manual annotations for common object-level operations such as addition, removal, and replacement.
- I2EBench spans high-level semantics and low-level details and uses user studies, while related methods employ manual annotation, detectors, or VQA-based correctness assessment.
- Benchmarks for multi-step reasoning and spatial complexity model difficulty logically, compositionally, or operationally, without fully capturing open-ended free-style editing.
3. CREval-Bench
CREval-Bench organizes creative editing into three levels and nine dimensions, using curated image–instruction pairs to represent diverse free-style tasks. CREval evaluates these tasks through QA-based metrics for instruction following, visual consistency, and visual quality, with a weighted aggregate score.
- Benchmark construction: The benchmark uses over 800 high-quality source images paired with detailed creative editing instructions spanning nine task categories.
- Creative editing taxonomy: Creative editing is organized into Customization, Contextualization, and Stylization, each subdivided into three dimensions for nine total dimensions.
- Creative editing taxonomy: Customization reconfigures object forms, while Stylization reinterprets images through artistic style, cultural identity, or materiality.
- Benchmark construction: Images are curated from public resources and open datasets, and representative examples guide GPT-4o in generating instructions aligned with each creative dimension.
- VQA-based evaluation: Instruction Following checks whether requested modifications are fully represented, Visual Consistency checks preservation of critical source elements, and Visual Quality evaluates perceptual quality.
- VQA-based evaluation: Evaluation supplies the source image, edited image, instruction, and generated questions to an MLLM, awarding points when predicted answers match references.
- VQA-based evaluation: The final score is computed as S = 0.4 * S_IF + 0.4 * S_VC + 0.2 * S_VQ, balancing metric dimensions while accounting for limited MLLM sensitivity to visual quality.
4. Experiments
CREval evaluates mainstream open- and closed-source image editing models across instruction following, visual consistency, and visual quality. Closed-source models generally lead overall, while visual consistency remains a major weakness and CREval scores closely correlate with human preferences.
- Current models perform well on complex creative editing but still struggle to maintain visual consistency with the source image.This weakness contributes to suboptimal overall performance.
- Qwen-Image-Edit achieves the strongest open-source instruction-following performance, while Seedream 4.0 and GPT-Image-1 approach 90 points among closed-source models.UniWorld-V1 and ICEdit average only 20–30% on Surreal Fantasy and Informationization and Narrative Expression tasks.
- Both open- and closed-source models obtain relatively low Visual Consistency scores, indicating difficulty identifying and preserving key visual elements from source images.UniWorld-V1 has the highest open-source Visual Consistency score, but qualitative inspection attributes this mainly to weak instruction execution.
- Closed-source models show strong Visual Quality performance, while Qwen-Image-Edit-2509 reaches a competitive level among open-source methods.Most models have similar metric scores, possibly because MLLMs overlook subtle artifacts such as twisted limbs or extra fingers.
- CREval scores closely correlate with human preferences across evaluations of six representative models using more than 200 image instances and 18 annotators.The study compared CREval with Aesthetic Score, VIEScore, and EditScore baselines.
- Using Qwen3-VL instead of GPT-4o slightly lowers Qwen-Image-Edit-2509’s ranking, but human evaluations find it nearly indistinguishable from GPT-Image-1.The authors state that relative ranking consistency preserves the validity of the conclusions.
5. Conclusion
The conclusion presents CREval and CREval-Bench as an automated framework and benchmark for creative image manipulation under complex instructions. It emphasizes QA-based evaluation, human and multi-MLLM validation, and applications to benchmarking, model selection, and future research.
- CREval evaluates complex creative image manipulation through automated QA-based metrics for instruction following, visual consistency, and visual quality.
- Human preference studies and evaluations using multiple MLLMs consistently support CREval’s reliability and robustness.
- CREval establishes a foundation for benchmarking, model selection, and future research in creative image manipulation.
Supplementary Material
The supplementary material provides additional analyses, quantitative results, QA examples, prompt templates, and visual comparisons supporting the main paper.
- The supplementary material analyzes final-score weight selection and presents additional quantitative and experimental results.
- It also provides CREval Question–Answer examples, detailed evaluation prompt templates, and further visual comparisons.
A. Weight sensitivity analysis.
The weight sensitivity analysis reduces the weight of Visual Quality and assigns higher weights to Instruction Following and Visual Consistency because these choices better align with human preferences.
- MLLMs perform suboptimally when evaluating Visual Quality, motivating a reduced Visual Quality weight.
- Instruction Following and Visual Consistency receive higher weights because they are described as more important and discriminative.
- The 4:4:2 weight ratio achieves better alignment with human preferences than the alternative settings tested.
B. More Experimental Details
Detailed experiments compare models across nine creative editing dimensions using IF, VC, and VQ, revealing task-dependent strengths and persistent difficulties with structural, identity, and contextual edits.
- Experimental analysis: Table S1 reports detailed IF, VC, and VQ scores across all nine creative dimensions for clearer model comparisons.The expanded analysis covers every editing task discussed in the main paper.
- Customization: Most models perform similarly on Derivative Character, where structural constraints are clear and modifications are limited.The open-source versus closed-source performance gap remains small for this task.
- Customization: Reimagined Representations and Surreal Fantasy remain difficult because structural changes and abstraction make it hard to preserve key source-image elements.
- Stylization: Identity & Cultural Transformation exposes failures in instruction following and preservation of essential visual features.These identity-related modifications are part of the Stylization category.
- Contextualization: Informationization & Narrative Expression and Commercial Design demand stronger semantic understanding and generative flexibility because they contain rich contextual information.Models show noticeable differences in handling and generating this contextual content.
C. VQA Examples
CREval provides example QA pairs for its MLLM-based VQA evaluation, using ‘Yes’ as the ideal answer for the listed perfect-editing cases.
- VQA examples: The examples illustrate QA pairs used by CREval’s MLLM-based VQA automatic evaluation.For every question listed in Table S3, the ideal answer for perfect editing is ‘Yes’.
D. Prompt Templates
The supplementary material documents prompt templates for dataset construction, metric-specific VQA generation, and evaluation answer generation.
- Dataset construction: After manually selecting high-quality images, GPT-4o generates editing instructions from category examples using the dataset-building prompt.
- VQA generation: CREval uses separate prompts to generate VQA pairs for Instruction Following, Visual Consistency, and Visual Quality.These prompts are designed with Chain-of-Thought reasoning to produce structured QA pairs.
- Evaluation: The evaluation prompt template specifies how answers are generated for the evaluation questions.
E. More Visual Comparisons
The supplementary material provides additional visual comparisons, example editing instructions, prompt templates, pipeline documentation, and QA examples spanning CREval’s evaluation process.
- Creative editing cases: The examples include highly detailed creative edits such as miniature robots, whimsical infographics, chibi stickers, and children’s treasure maps.
- Evaluation materials: The supplementary figures include VQ failure cases, weight ratios, and prompts for generating instructions, IF questions, VC questions, VQ questions, and evaluations.
- Pipeline documentation: Table S2 documents the core evaluation pipeline, while Table S3 presents QA examples whose ideal answers are ‘Yes’.
- Visual comparisons: Figures S9–S14 provide additional visual comparisons for open- and closed-source models across Customization, Contextualization, and Stylization.
- QA examples: Example QA pairs assess instruction-specific attributes, retained visual features, structural coherence, natural appearance, and artifacts such as malformed hands.