PaperlayerView exact revision on arXiv

Source-linked AI summary

EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing

Tianyu Chen, Yasi Zhang, Zhi Zhang, Peiyu Yu, Shu Wang, Zhendong Wang, Kevin Lin, Xiaofei Wang, Zhengyuan Yang, Linjie Li, Chung-Ching Lin, Jianwen Xie, Oscar Leong, Lijuan Wang, Ying Nian Wu, Mingyuan Zhou

arXiv:2509.13399v3cs.CVcs.AIcs.LG

TL;DR

Reference-based and VLM-only protocols have limitations for fine-grained, multi-turn image-editing evaluation. EdiVal uses object-centric analysis and multiple metrics, and its benchmark reveals model-specific strengths and weaknesses, with EdiVal-IF achieving 81.3% agreement with human ratings.

  • Problem

    Reference-based evaluation is constrained by the variety of acceptable edits and biases in synthesized references, while VLM-only judgments can be imprecise across editing dimensions.

    Source: §1 · p. 2

  • Method

    EdiVal decomposes images into objects, generates context-aware multi-turn instructions with evolving object pools, and evaluates edits with EdiVal-IF, EdiVal-CC, and EdiVal-VQ.

    Source: §1 · p. 3

  • Results

    81.3% human-rating agreement for EdiVal-IF outperforms zero-shot VLMs and CLIP-based baselines, while benchmark results show different models excel across editing axes and turns.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Contrasting multi-turn editing with complex single-shot prompts identifies complementary strengths and failure modes across model families.

    Source: §1 · p. 3

    Style change is omitted because style categories are inherently ill-defined, making instruction-following evaluation difficult to perform reliably.

    Source: §2.4 · p. 6

Abstract

from arXiv · show

Instruction-based image editing has advanced rapidly, yet reliable and interpretable evaluation remains a bottleneck. Current protocols either (i) depend on paired reference images, resulting in limited coverage and inheriting biases from prior generative models or (ii) rely solely on zero-shot vision language models (VLMs), whose prompt-based assessments of instruction following, content consistency, and visual quality are often imprecise. To address this, we introduce EdiVal, an automated and fine-grained evaluation framework grounded in an object-centric perspective, designed to assess not only standard single-turn but also multi-turn instruction-based editing with precision. Given an input image, EdiVal first decomposes it into semantically meaningful objects, then synthesizes diverse, context-aware editing instructions while dynamically updating object pools across turns. These two stages enable two novel object centric metrics tailored for multi turn evaluation and one global metric of visual quality: 1) EdiVal-IF, which measures instruction following by combining open vocabulary object detectors for symbolic checks with VLMs for semantic verification on detector guided crops; 2) EdiVal-CC, which evaluates content consistency by calculating semantic similarity of unchanged objects and background using the evolving object pools; and 3) EdiVal-VQ, which quantifies changes in overall visual quality with human preference models. Instantiating this pipeline, we build EdiVal Bench, a multi-turn editing benchmark covering 9 instruction types and 16 state-of-the-art editing models, spanning in-context, flow-matching, and diffusion paradigms. We demonstrate that EdiVal can be used to identify existing failure modes, thereby informing the development of the next generation of editing models.

1 INTRODUCTION

Image-editing evaluation must capture targeted changes, preservation, and realism across turns, but reference-based and VLM-only protocols have important limitations. EdiVal-Agent uses object-centric analysis and a multi-axis benchmark to evaluate editing systems and reveal model-specific strengths and failure modes.

1 INTRODUCTION

  • Reference comparisons are limited because acceptable edits vary and synthesized references can import prior models’ biases; similarity alone does not establish faithful edits or plausible outcomes.VLM-only evaluators can miss spatial relations, subtle localized changes, and generative defects, limiting fine-grained assessment.Source: §1 · p. 2
  • EdiVal-Agent decomposes images into objects and contextual relations, generates context-aware multi-turn instructions, and evaluates instruction following, content consistency, and visual quality.EdiVal-IF combines detector checks with VLM verification; EdiVal-CC evaluates unchanged objects and background, while EdiVal-VQ measures overall visual quality.Source: §1 · p. 3, Figure 2 · p. 3
  • EdiVal-Bench covers 9 instruction types and 16 editing models spanning in-context, flow-matching, and diffusion paradigms.The benchmark includes both closed- and open-source systems.Source: §1 · p. 3
  • Open-source Qwen-Image-Edit degrades in instruction following and visual quality as editing turns increase, while FLUX.1-Kontext-max and FLUX.1-Kontext-dev lag in instruction following.The paper also contrasts multi-turn editing with complex single-shot prompts to identify complementary strengths and failure modes.Source: §1 · p. 3
  • 81.3% human-rating agreement for EdiVal-IF exceeds zero-shot VLMs and CLIP-based baselines in instruction-following evaluation.The benchmark analysis also finds GPT-Image-1 strong in instruction following but near the bottom in content consistency, while Seedream 4.0 and Nano Banana perform strongly on both axes.Source: §1 · pp. 3–4

2 Æ EDIVAL-AGENT

EdiVal-Agent decomposes images into objects, generates context-aware multi-turn edits using evolving object pools, and evaluates those edits with object-centric measures. Its human-agreement study finds EdiVal-IF more aligned with human judgments than the reported VLM-only and CLIP-based baselines.

2.1 OVERVIEW

  • The pipeline has three stages: object decomposition, multi-turn instruction generation with evolving pools, and evaluation using EdiVal-IF, EdiVal-CC, and EdiVal-VQ.The instruction taxonomy covers nine types across subject-, attribute-, relational-, text-, count-, and global-level edits.Source: §2.1 · p. 4

2.2 STEP 1: DECOMPOSITION

  • A VLM extracts structured descriptions of visible foreground objects, then Grounding-DINO validates detections to seed instruction generation and evaluation.The method excludes 34 sensitive-content images from GEdit-Bench’s 606 images and retains 572.Source: §2.2 · p. 4

2.3 STEP 2: INSTRUCTION GENERATION

  • At each turn, the agent selects an unused instruction type and editable object, generates a scene-grounded prompt, and updates the object pool for the intended edit.The default chain has three turns with distinct instruction types; background-change turns disable background-consistency scoring and append an instruction to keep foreground objects unchanged.Source: §2.3 · pp. 4–5

2.4 STEP 3: EVALUATION

  • EdiVal-CC measures preservation of nontarget content, while EdiVal-VQ reports human-preference visual quality separately from the aggregate instruction-following and consistency score.EdiVal-CC averages feature-level similarities for background and unchanged objects; the aggregate combines EdiVal-IF and EdiVal-CC using a geometric mean.Source: §2.4 · pp. 5–6
  • Style change is omitted because style categories are ill-defined, making instruction-following evaluation difficult to perform reliably.The authors identify style-aware recognition as future work.Source: §2.4 · p. 6

2.5 MEASURING HUMAN AGREEMENT

  • 81.3% human agreement accuracy for EdiVal-IF exceeds Qwen2-VL’s 75.2% and thresholded CLIP dir’s 65.4%.The CLIP dir threshold is tuned separately for each task; inter-annotator agreement is 85.5%.Source: §2.5 · p. 6, Figure 4 · p. 7

3 BENCHMARKING MULTI-TURN EDITING

The benchmark compares multi-turn editors across instruction following, consistency, and visual quality, revealing model trade-offs and turn-dependent behavior. It also tests sequential versus complex edits and the robustness of the evaluation stack.

3 BENCHMARKING MULTI-TURN EDITING

  • Seedream 4.0 ranks first overall, while GPT-Image-1.5 leads EdiVal-O at T1/T2 and Qwen-Image-Edit degrades sharply over turns.Seedream 4.0’s EdiVal-IF is 75.93/41.59 at T1/T3; GPT-Image-1.5’s EdiVal-O is 84.29/71.41 at T1/T2.Source: Table 3 · p. 7, §3 · pp. 7–8

3.1 INSTRUCTION FOLLOWING

  • Seedream 4.0, Nano Banana, and FLUX.1-Kontext-max maintain relatively stable marginal success across turns, while several other models decline.The authors hypothesize that Qwen-Image-Edit’s sharper decline reflects exposure bias: small distribution mismatches can compound when models edit their own outputs.Source: Figure 5 · p. 8, §3.1 · p. 8

3.2 CONTENT CONSISTENCY

  • DINOv3 measures semantic consistency of unchanged objects and background, avoiding L1 sensitivity to small location shifts; FLUX.1-Kontext-dev ranks most consistent.GPT-Image-1 ranks near the bottom in consistency across turns.Source: §3.2 · pp. 8–9

3.1 INSTRUCTION FOLLOWING

  • Nano Banana performs relatively well on color and material changes but poorly on position and count changes, suggesting spatial and numerical reasoning weaknesses.Source: §3.1 · p. 8, Figure 6 · p. 9

3.3 VISUAL QUALITY

  • GPT-Image-1 has the highest aesthetic scores and largest EdiVal-VQ∆, while GPT-Image-1.5 reduces ∆ from 2.18 to 0.15.Gemini 2.0 Flash exhibits the least drift when preserving the base image’s appearance.Source: §3.3 · p. 9, Table 4 · p. 10

3.4 MULTI-TURN EDITING VS. COMPLEX EDITING

  • When exposure bias is limited, sequential edits tend to outperform complex prompts; when it is pronounced, a single complex prompt can perform better.Nano Banana benefits from multi-turn editing, whereas Qwen-Image-Edit can perform better with complex prompts.Source: §3.4 · p. 9, Figure 9 · p. 10

3.5 PARETO FRONT

  • The Turn 3 Pareto boundary compares all pairwise combinations of instruction following, content consistency, and visual quality to show their trade-offs.Source: Figure 10 · p. 10, §3.5 · p. 10

3.6 TOOL SWAP ANALYSIS AND PROMPT COMPRESSION STRATEGIES

  • Swapping the VLM, detector, threshold, or feature extractor has negligible impact on evaluation outcomes, while prompt-compression variants mildly affect complex-edit success.Source: §3.6 · p. 10

4 CONCLUSION

EdiVal-Agent is an automated, interpretable framework for evaluating instruction-based image editing. It combines object-centric decomposition, structured instruction generation, specialist tools, and vision-language reasoning for fine-grained assessment.

  • EdiVal-Agent provides an automated and interpretable framework for evaluating instruction-based image editing.Source: §4 · p. 10
  • Its pipeline combines symbolic object decomposition and structured instruction generation with specialist tools and vision-language reasoning models.Source: §4 · p. 10
  • The framework uses object-centric metrics and human preference models to assess editing quality.Source: §4 · p. 10

ETHICS STATEMENT

The framework and benchmark are intended to advance research in safe and trustworthy generative AI. The authors identify potential misuse and describe safeguards for the work.

  • The authors identify potential misuse of image editing, including misleading, deceptive, or inappropriate content.Source: ETHICS STATEMENT · p. 11
  • The benchmark and evaluation framework are intended solely to advance research in safe and trustworthy generative AI.Source: ETHICS STATEMENT · p. 11
  • The authors use publicly available datasets, apply safety filters to generated images, and encourage responsible use aligned with ethical standards and community guidelines.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

Prior editing work spans diffusion, flow-matching, and autoregressive systems, while editing evaluation has used reference comparisons and VLM judges. The paper situates its work within these approaches and states a scope covering single-turn, multi-turn, and complex single-shot editing.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement says prompting templates, pseudo-code, implementation details, API links, results, datasets, and metrics are documented, with code, data, and checkpoints slated for release upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 11

A RELATED WORK

  • Instruction-based editors progressed from diffusion models fine-tuned on synthesized edit pairs to larger real-image datasets, task-aware architectures, and flow-matching systems.Source: A RELATED WORK · p. 17
  • Autoregressive editors support in-context multi-turn refinement, and the paper reports the first systematic comparison with context-free editors across three evaluation dimensions.Source: A RELATED WORK · p. 17
  • Reference-based metrics face multimodal valid edits and possible bias from references synthesized by prior editors, while VLM judges can miss spatial, localized, and aesthetic details.Source: A RELATED WORK · p. 17
  • The paper compares single-turn, multi-turn, and complex single-shot editing in one framework, while leaving other scenarios such as prompt-to-prompt editing outside its scope.Source: A RELATED WORK · p. 17

B LIMITATION AND DISCUSSION

The discussion identifies limits in the benchmark’s object-centric instruction scope and in the appendix’s coverage of newly released models. Pareto plots show per-turn trade-offs among EdiVal dimensions.

B LIMITATION AND DISCUSSION

  • Object-centric prompts may miss creative requests involving stylistic changes or complex narrative elements; the framework also does not use evaluation scores to improve editing models.The authors suggest broader instruction coverage and future use of scores for Best-of-N selection or post-training.Source: B LIMITATION AND DISCUSSION · p. 18
  • Appendix comparisons for GPT-Image-1.5, Nano Banana 2, and FLUX.2-max may be incomplete because the models were released after initial submission and space and timing were limited.Source: B LIMITATION AND DISCUSSION · p. 18

C PARETO PLOT

D ABLATION STUDY ON COMPLEX EDITING COMPRESSION

The ablation tests whether alternative ways of compressing three-turn instructions into one complex-edit prompt affect editing success. The tested variants have only a very mild effect.

D ABLATION STUDY ON COMPLEX EDITING COMPRESSION

E TOOL SWAP ANALYSIS

Component swaps generally preserve model rankings while sometimes shifting absolute evaluation scores. Detector choice is a notable exception in human agreement: replacing Grounding DINO with OWL-ViT substantially reduces it.

E TOOL SWAP ANALYSIS

  • Across component swaps, evaluation outcomes remain highly correlated with the default stack, though some substitutions shift absolute scores.The analysis covers VLM, detector, detector threshold, and image feature extractor changes; HPSv3 is identified as the only preference model trained on images generated after SD3.5.Source: E TOOL SWAP ANALYSIS · p. 19

E.1 VLM SWAP

  • All Pearson correlations exceed 0.95 and Spearman correlations exceed 0.92 for VLM swaps, while editor rankings remain unchanged.Qwen2.5- and InternVL-based stacks tend to assign somewhat lower scores than the Qwen2-7B-VL stack.Source: §E.1 · p. 20, Table 7 · p. 20

E.2 GROUNDING DINO THRESHOLD AND LARGE-BOX FILTER SWAP

  • Global detector thresholds of 0.30 or 0.40, or disabling the large-box filter, change instruction-following metrics only slightly and leave model rankings unchanged.Source: §E.2 · p. 20, Table 8 · p. 20

E.3 DETECTOR SWAP

  • Grounding DINO offers a better trade-off for open-world editing than OWL-ViT or GLIP when ODinW performance is considered, despite OWL-ViT’s strong LVIS results.Grounding DINO outperforms GLIP on ODinW and has the best AP on COCO; LVIS has more than 1k categories.Source: §E.3 · p. 21, Table 9 · p. 21
  • Replacing Grounding DINO with OWL-ViT preserves model ordering but lowers absolute scores; EdiVal-IF human agreement drops from 81.30% to 53.67%.OWL-ViT misses visible objects, while GLIP produces denser, less precise boxes; detector swaps therefore change rather than eliminate some failure cases.Source: §E.3 · p. 21, Figure 13 · p. 22, Figure 14 · p. 22

E.4 IMAGE FEATURE EXTRACTOR SWAP: DINOV3 TO DINOV2

  • DINOv2 yields lower EdiVal-CC values than DINOv3 but preserves identical model ordering; per-turn correlations are Pearson 0.9987 and Spearman 1.0000.Figure 16 plots DINOv3 scores against DINOv2 scores for four representative models and confirms the ranking stability.Source: §E.4 · p. 23, Figure 16 · p. 23

F IMPROVEMENT TOWARDS COUNTING

  • For an example open-vocabulary count, CSRNet estimates 44.86 objects while Grounding DINO correctly predicts 3 bounding boxes.Density-map methods are typically trained for fixed categories and can fail when targets do not match their training distribution.Source: F IMPROVEMENT TOWARDS COUNTING · p. 23, Figure 17 · p. 24

G MORE METRICS FOR HUMAN AGREEMENT

H ARTIFICIAL ANALYSIS LEADERBOARD

On the overlapping systems, Artificial Analysis human-vote rankings match EdiVal’s exactly. The supplementary analyses also describe detector false positives in subject replacement and attribute multi-turn performance degradation primarily to accumulated sequential-edit errors.

H ARTIFICIAL ANALYSIS LEADERBOARD

  • Artificial Analysis human-vote rankings match EdiVal’s exactly on five overlapping systems, with Qwen-Image-Edit excluded.The comparison uses Seedream 4.0, Nano Banana (Gemini 2.5 Flash), GPT-Image-1 (GPT-4o), FLUX.1-Kontext-max, and Gemini 2.0 Flash.Source: Table 13 · p. 26, H ARTIFICIAL ANALYSIS LEADERBOARD · p. 26

I FAILURE CASE

  • Grounding-DINO false positives can incorrectly mark a visually correct subject replacement as a failure when source and target detections overlap.For this task, success requires the source object to be undetected and the target object detected; reducing false positives would improve evaluation fidelity.Source: I FAILURE CASE · p. 27, Figure 19 · p. 27

J DISCUSSION ON SINGLE-SHOT COMPLEX EDITING

  • Stable final-instruction success as complex prompt length increases, alongside multi-turn drops, supports accumulated sequential-edit errors as the primary degradation explanation.The passage frames this as an exposure-bias explanation rather than an intrinsic inability to handle multiple instructions in one prompt.Source: J DISCUSSION ON SINGLE-SHOT COMPLEX… · p. 28

K DISCUSSION ON VISUAL QUALITY

The section covers multi-turn editing analyses, including visual-quality tracking with luminance statistics and learned aesthetic scores, background-category checks, and reported success-rate tables.

K DISCUSSION ON VISUAL QUALITY

  • The analysis uses learned aesthetic scores and low-level image statistics; its 99.9% luminance quantile highlights extreme bright pixels while resisting single-pixel outliers.Luminance is computed from RGB using Rec. 709 coefficients, and the quantile is tracked across turns to surface over-exposure and bright streaks.Source: K DISCUSSION ON VISUAL QUALITY · p. 28
  • Qwen-Image-Edit and several flow-matching models show rising 99.9% luminance quantiles over turns, while GPT-Image-1 trends darker and several models remain stable.FLUX.1-Kontext-dev is the stated exception among the flow-matching models with pronounced increases; the brightening indicates greater over-exposure risk.Source: K DISCUSSION ON VISUAL QUALITY · pp. 28–29, Figure 21 · p. 28

L VLMS FAILING TO JUDGE VISUAL QUALITY

M.3 EVALUATION

  • Evaluation reports HPSv3 plausibility and aesthetics plus luminance metrics separately from the Overall score, while consistency uses unchanged-object and background comparisons.Object consistency uses DINOv3 feature similarity and pixel L1; background consistency uses masked L1 and is evaluated only when no earlier background change occurred.Source: §M.3 · p. 34

M.5 MODEL GENERATIONS

  • A VLM checks whether the background shows category g using a yes/no question.The supplied line does not specify further evaluation steps or outcomes.Source: §M.5 · p. 37
  • The table captions identify multi-turn image and overall task success rates across turns, and task success rates across five instruction types and three turns.The passages give no cell values or comparisons.Source: Table 15 · p. 38, Table 16 · p. 38

N ADDITIONAL EVALUATION RESULTS

The extended evaluation reports multi-turn and complex-editing results across instruction following, consistency, and quality, with separate consistency reporting for square and non-square inputs. It also describes the human-agreement study and identifies count change as the most challenging subtask.

N ADDITIONAL EVALUATION RESULTS

  • Multi-turn and complex editing are evaluated across instruction following, consistency, and quality, with results organized in Tables 15–26.For multi-turn consistency, square and non-square inputs are reported separately because resizing can leave padding that some models fill and others preserve; the authors say conclusions remain consistent.Source: N ADDITIONAL EVALUATION RESULTS · p. 38, Table 17 · p. 39, Table 18 · p. 39

O HUMAN AGREEMENT

  • The human study collected 4,576 binary instruction-following annotations on edits from four exemplary models, using original images, edited images, and instructions.Two ratings were collected per edit; raters were recruited online and each held at least a bachelor’s degree.Source: O HUMAN AGREEMENT · p. 42

P COUNTING

  • Count change is the most challenging subtask: even GPT-Image-1 remains below 25% success at turn 1, while most models are below 5%.Figure 23 illustrates changing the number of paper cups to five.Source: P COUNTING · p. 42, Figure 23 · p. 42

Q DIVERSITY OF OUR DATASET

The dataset diversity appendix describes varied real-world editing scenarios, illumination conditions, and object categories. It also provides qualitative examples of three-turn editing outputs across the evaluated models.

Q DIVERSITY OF OUR DATASET

  • The dataset draws on GEdit-Bench images and includes synthetic and real images selected for diversity, with varied scenes and illumination conditions.The described scenes include indoor and outdoor settings, people, special items, and artistic scenes; the illumination examples span very dark to very bright.Source: Q DIVERSITY OF OUR DATASET · p. 44
  • After decomposition, the dataset contains 724 distinct objects classified into 13 categories, ranging from everyday items such as furniture and kitchenware to special entities like vehicles and animals.Figure 26 presents categories and examples of objects in the base images.Source: Figure 26 · p. 44, Q DIVERSITY OF OUR DATASET · p. 45

R MORE QUALITY EXAMPLES

  • Qualitative examples show three editing turns across six models and then seven additional models, using the same pumpkin-color, background, and bow-removal instructions.The examples are presented row-wise, with input images and generations for the three turns.Source: Figure 27 · p. 46, Figure 28 · p. 47
Loading 2509.13399v3…