PaperlayerView exact revision on arXiv

Source-linked AI summary

WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing

Kaihang Pan, Weile Chen, Haiyi Qiu, Qifan Yu, Wendong Bu, Zehan Wang, Yun Zhu, Juncheng Li, Siliang Tang

arXiv:2512.00387v2cs.CV

TL;DR

Existing benchmarks provide limited coverage of the cognitive and creative demands of advanced image editing. WiseEdit evaluates challenging tasks across editing steps and knowledge types; experiments reveal persistent limitations in current models' reasoning and creative composition.

  • Problem

    Existing image-editing benchmarks often use simple instructions and do not holistically cover the task depth and knowledge breadth needed to evaluate cognition- and creativity-informed editing.

    Source: §1 · p. 1

  • Method

    WiseEdit evaluates Awareness, Interpretation, Imagination, and Complex editing tasks across Declarative, Procedural, and Metacognitive knowledge.

    Source: §1 · p. 2

  • Results

    Experiments reveal limitations in current models' knowledge-based cognitive reasoning and creative composition, with most models also declining on the Complex task relative to their average scores across the three WiseEdit tasks.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    WiseEdit's evaluation offers a reference for future advancement of image-editing research by revealing existing models' performance levels and potential weaknesses.

    Source: §5 · p. 9

    All models share an inability to solve logic-related problems such as matchstick tasks, defining a boundary of current capability.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Recent image editing models boast next-level intelligent capabilities, facilitating cognition- and creativity-informed image editing. Yet, existing benchmarks provide too narrow a scope for evaluation, failing to holistically assess these advanced abilities. To address this, we introduce WiseEdit, a knowledge-intensive benchmark for comprehensive evaluation of cognition- and creativity-informed image editing, featuring deep task depth and broad knowledge breadth. Drawing an analogy to human cognitive creation, WiseEdit decomposes image editing into three cascaded steps, i.e., Awareness, Interpretation, and Imagination, each corresponding to a task that poses a challenge for models to complete at the specific step. It also encompasses complex tasks, where none of the three steps can be finished easily. Furthermore, WiseEdit incorporates three fundamental types of knowledge: Declarative, Procedural, and Metacognitive knowledge. Ultimately, WiseEdit comprises 1,220 test cases, objectively revealing the limitations of SoTA image editing models in knowledge-based cognitive reasoning and creative composition capabilities. The benchmark, evaluation code, and the generated images of each model will be made publicly available soon. Project Page: https://qnancy.github.io/wiseedit_project_page/.

1. Introduction

Recent image editors can reason over complex instructions and create multi-concept edits, but existing benchmarks do not adequately assess these abilities. WiseEdit addresses this gap with challenging tasks across editing steps and knowledge types, then evaluates 1,220 cases to identify current model limitations.

1. Introduction

  • Existing benchmarks are often too simple, while reasoning-focused benchmarks mainly assess interpretation and omit other editing steps and metacognitive knowledge.The paper argues for holistic evaluation of advanced image editing capabilities.Source: §1 · pp. 1–2
  • WiseEdit introduces Awareness, Interpretation, Imagination, and Complex tasks, with free-form instructions and cases that may include multiple input images.The first three tasks target specific editing steps; in Complex, none of the steps is easy to complete.Source: §1 · p. 2
  • WiseEdit covers Declarative, Procedural, and Metacognitive knowledge across domains including natural science, cultural common sense, and spatio-temporal-logic reasoning.Each case is provided in Chinese and English to assess cross-lingual instruction following.Source: §1 · p. 2
  • WiseEdit contains 1,220 cases and evaluates 21 leading models using Knowledge Fidelity and Creative Fusion alongside conventional metrics.Experiments reveal limitations in knowledge-based cognitive reasoning and creative composition.Source: §1 · p. 2

2. Related Work

Image editing models have moved from simple localized changes toward unified systems that combine visual comprehension and generation. Existing benchmarks still have gaps in task coverage, input formats, and knowledge assessment, which WiseEdit is designed to address.

Image Editing Models

  • Early instruction-based editing models focused on localized changes, while newer unified systems integrate visual comprehension and generation for reasoning-based editing.Closed-source systems are also described as advancing cognition- and creativity-informed editing.Source: §2 · pp. 2–3

Image Editing Benchmarks

  • Table 1 compares open-source knowledge-informed reasoning-based image editing benchmarks.The supplied caption identifies the comparison's scope but does not state its individual entries.Source: Table 1 · p. 3
  • Existing benchmarks often use simple instructions, single-image inputs, and limited task coverage, with metacognitive knowledge and non-English instructions excluded.The paper contrasts these gaps with WiseEdit's broader evaluation.Source: §2 · p. 3

3. WiseEdit

WiseEdit evaluates cognition- and creativity-informed image editing through challenging task formats and broad knowledge coverage. Its evaluation pipeline combines established image-editing metrics with measures of knowledge fidelity and creative fusion.

3.1. Benchmark Construction

  • WiseEdit is designed to measure how models infer implicit intent from multimodal input and produce sophisticated edits that embody that intent.The benchmark breaks editing into interconnected steps and uses high-difficulty tasks to assess task depth.Source: §3.1 · p. 3, §3 · p. 3
  • Awareness tasks require locating targets without explicit spatial cues; Interpretation tasks require turning implicit instructions into executable changes, often using world knowledge.Awareness can involve comparative reasoning or reference-image correspondence; Interpretation may require accounting for cascaded effects.Source: §3.1 · p. 3
  • Imagination tasks require creative edits that preserve subject identity, while WiseEdit-Complex makes all three editing steps difficult to complete.The imagination cases include transformations and multi-object compositions; the complex task requires reasoning and creative generation together.Source: §3.1 · pp. 3–4
  • WiseEdit supports multiple input images with distinct target and reference roles, and each of its 1,220 test cases has English and Chinese instructions.Instructions may use ordinal terms to identify an image, whose role the model must infer.Source: §3.1 · p. 4

3.2. Knowledge-Intensive Evaluation in WiseEdit

  • The benchmark covers Declarative, Procedural, and Metacognitive knowledge, alongside cultural common sense, natural sciences, and spatio-temporal-logic reasoning.These domains are used to assess culturally appropriate, physically consistent, and logically coherent editing.Source: §3.2 · pp. 4–5

3.3. Evaluation Pipeline

  • GPT-4o evaluates Instruction Following, Detail Preserving, and Visual Quality, while Knowledge Fidelity and Creative Fusion assess knowledge use and creative qualities.Knowledge hints guide Knowledge Fidelity evaluation; the metrics are scored on a 1–10 scale and combined in a task-dependent average.Source: §3.3 · p. 5

4. Experiments

Across WiseEdit’s three tasks, closed-source models lead, while cognition- and creativity-informed editing remains difficult, especially for complex, multi-image cases. The analyses also identify weak knowledge fidelity, creative fusion, and metacognitive performance as persistent challenges.

4.1. Experimental Setup.

  • The evaluation covers 22 mainstream image-editing models across diffusion, unified comprehension-generation, and closed-source groups.Source: §4.1 · p. 5

4.2. Main Results on WiseEdit

  • Nano Banana Pro significantly surpasses the other three closed-source models; the lowest-scoring closed-source model beats the best open-source model by nearly 15 overall points.Source: Table 2 · p. 6, §4.2 · pp. 7–8
  • Unified comprehension-generation models outperform most diffusion-based models, with FLUX.2 Dev an exception; Qwen-Image-Edit scores 17 points higher than FLUX.1 Kontext on average overall.The paper attributes much of the Qwen-Image-Edit gap to its integration of Qwen-VL for visual comprehension.Source: §4.2 · pp. 5–7, Table 2 · p. 6

4.3. Main Results on WiseEdit-Complex

  • On WiseEdit-Complex, closed-source models outperform open-source unified models, which outperform diffusion models except FLUX.2 Dev; most models’ performance declines versus the three-task average.Some leading closed-source models score higher on CF and KF but lower on IF in the complex task.Source: Table 3 · p. 6, §4.3 · p. 8

4.2. Main Results on WiseEdit

  • Awareness and Interpretation IF and KF scores improve across model groups but generally remain below 80, while Imagination CF scores are poor and DP scores stay below 80.Many leading unified models show negligible performance change between Chinese and English instructions.Source: §4.2 · p. 8

4.4. In-Depth Analysis

  • The proposed evaluation protocol has the strongest human-rating/VLM-score correlation; radar results identify weak creative fusion, knowledge fidelity, and metacognitive performance.Knowledge fidelity further negatively impacts instruction following.Source: §4.4 · pp. 8–9, Figure 5 · p. 8
  • Multi-image inputs significantly lower average scores, and removing built-in thinking causes significant drops on WiseEdit and WiseEdit-Complex.The thinking-process comparison re-evaluates Bagel and DreamOmni2.Source: Figure 5 · p. 8, §4.4 · p. 9
  • Adding knowledge hints to 75 rewritten cases improves unified open-source models by about 15 points in Overall and 30 points in KF; closed-source models gain less.Qualitative comparisons also find that open-source models may interpret instructions correctly yet fail to execute them, while all models struggle with logic-related problems such as matchsticks.Source: Table 4 · p. 8, §4.4 · p. 9

5. Conclusion

WiseEdit introduces four challenging task types across diverse knowledge breadth to evaluate image-editing models’ reasoning and creative composition. Its results illuminate existing models’ performance and potential weaknesses.

5. Conclusion

  • WiseEdit evaluates multifaceted model abilities, including knowledge-based reasoning and creative composition.Source: §5 · p. 9
  • The benchmark includes Awareness, Interpretation, Imagination, and Complex tasks paired with diverse knowledge breadth.Source: §5 · p. 9
  • Its results illuminate existing models’ performance and potential weaknesses, offering a reference for future image-editing research.Source: §5 · p. 9

Appendix Overview

The supplementary material directs readers to benchmark, data-collection, evaluation, and additional experimental details.

Appendix Overview

A. Benchmark Details

This section explains the tasks and knowledge types incorporated into WiseEdit.

A. Benchmark Details

A.1. Task Type

WiseEdit organizes challenging editing tasks around Awareness, Interpretation, and Imagination, with a Complex category combining difficulties across all three steps.

A.1. Task Type

  • Awareness tasks require locating the edit target without explicit spatial information, using cues such as object properties, spatial relations, OCR, or reference images.Source: §A.1 · p. 12
  • Interpretation tasks translate implicit instructions into executable actions through world knowledge, rule-based, logical, conditional, or other reasoning.Source: §A.1 · pp. 12–13
  • Imagination tasks require creative, subject-driven visual generation while preserving subject identity, spanning transformations of appearance, pose, scene, style, and objects.Source: §A.1 · p. 13
  • WiseEdit-Complex combines difficult awareness, interpretation, and imagination steps in a single task, requiring complex reasoning and creative generation.Source: §A.1 · p. 13
  • Illustrative complex tasks include identifying a person by hair length, inferring a flag from a location, and combining clothing and pose references.Source: §A.1 · p. 13
  • Other examples combine object identification with functional or conditional reasoning to determine the person’s action and viewpoint.Source: §A.1 · pp. 13–14

A.2. Knowledge Type

WiseEdit distinguishes declarative knowledge (knowing what), procedural knowledge (knowing how), and metacognitive knowledge (knowing about knowing), across three knowledge domains.

A.2. Knowledge Type

  • Declarative knowledge covers explicitly stated facts and concepts, including rules, principles, definitions, and object properties.The Chinese chess example requires knowing each piece’s movement constraints and forbidden zones.Source: §A.2 · p. 14
  • Procedural knowledge is the expertise and skills for performing tasks, often dynamic and difficult to explain verbally.Examples include multi-step instruction following, logical reasoning, viewpoint transformation, and style transfer.Source: §A.2 · p. 14
  • Metacognitive knowledge governs when and why to apply declarative or procedural knowledge and how to combine them during editing.It supports conditional branching in complex instructions, such as selecting an object and changing a person’s pose based on its use.Source: §A.2 · p. 14
  • WiseEdit covers cultural common sense, natural sciences, and spatio-temporal-logical reasoning as knowledge domains.These domains concern culturally appropriate editing, physically consistent images, and coherent causal, spatial, and temporal relations.Source: §A.2 · p. 14

B. Data Collection

WiseEdit cases combine collected or generated images with human-authored, GPT-4-augmented instructions, followed by dual screening; the final benchmark contains 1,220 cases.

B. Data Collection

  • 1,220 test cases remained after Ph.D. experts and GPT-4 jointly screened out unreasonable or unachievable cases.Images were primarily collected from the Internet or generated with models, with a very small portion from existing datasets.Source: B. Data Collection · p. 14
  • Human annotators drafted instructions, and GPT-4 augmented them for clarity, reduced ambiguity, and diversity without changing their meaning.GPT-4 also translated instructions originally written in English into Chinese.Source: B. Data Collection · p. 14
  • The benchmark includes 362 Awareness cases, 317 Interpretation cases, 451 Imagination cases, and 80 WiseEdit-Complex cases.Interpretation uses only single images; all WiseEdit-Complex cases use multiple images, while Imagination ranges from one to five input images.Source: B. Data Collection · p. 14

C. Evaluation Details

WiseEdit evaluates diffusion, unified comprehension-and-generation, and closed-source image editing models using task-specific GPT-4o metrics and a defined score aggregation procedure.

C. Evaluation Details

  • The evaluation spans diffusion models, unified comprehension-and-generation models, and closed-source models accessed through APIs.The unified-model group includes quantized autoregressive, cascaded, and integrated architectures; closed-source models have no publicly available checkpoint weights.Source: C. Evaluation Details · p. 15
  • GPT-4o rates outputs from 1–10 using dimension-specific prompts across Instruction Following, Detail Preserving, Visual Quality, Knowledge Fidelity, and Creative Fusion.Awareness and Interpretation use the first four metrics; Imagination uses Instruction Following, Detail Preserving, Visual Quality, and Creative Fusion; WiseEdit-Complex uses all four applicable metrics.Source: C. Evaluation Details · p. 15
  • Scores are mapped linearly from 1–10 to 0–100, then averaged by task; the overall score averages Awareness, Interpretation, and Imagination task averages.Cases a model cannot handle receive zero across all metrics; some knowledge-informed cases include hints and a reference image for Knowledge Fidelity assessment.Source: C. Evaluation Details · p. 15

D. More Experimental Results

Across most models, performance averaged over all cases is lower than performance on single-image subsets. Qualitative comparisons also show changes associated with reasoning processes and instruction rewriting.

D. More Experimental Results

  • For the majority of models, average performance across all cases is lower than average performance on single-image inputs.Interpretation contains only single-image inputs, while WiseEdit-Complex contains only multi-image inputs, so separate single-image results are unavailable for those task subsets.Source: D. More Experimental Results · p. 15, Table 5 · p. 17, Table 6 · p. 18
  • Disabling built-in reasoning in Bagel and DreamOmni2 leads to a noticeable degradation in generation quality.The comparison illustrates the role of visual comprehension in visual generation in unified models.Source: D. More Experimental Results · p. 16, Figure 6 · p. 16
  • Instruction rewriting significantly improves Bagel’s editing results by adding knowledge, while its improvement for GPT-Image-1 is less pronounced.The passage attributes the smaller improvement to GPT-Image-1’s internal comprehension mechanisms already having acquired the externally introduced knowledge.Source: D. More Experimental Results · p. 16, Figure 7 · p. 19
  • Additional qualitative comparisons cover Awareness, Interpretation, Imagination, and WiseEdit-Complex across nine models.Outputs for models unable to process an input are marked with a prohibition symbol.Source: D. More Experimental Results · p. 16, Figure 8 · p. 20, Figure 9 · p. 21
Loading 2512.00387v2…