Source-linked AI summary
Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing
Yixuan Ding, Wei Huang, Ruijie Quan, Xiaojuan Qi, Yi Yang
TL;DR
Existing image editors often satisfy surface instructions while violating implicit physical, causal, or contextual constraints. The paper introduces RE-Edit and dimension-aligned evaluation, studies 12 editors, and explores EditRefine as a reasoning-guided post-edit baseline. Results show visually plausible edits can still lack reasoning correctness, while EditRefine consistently improves reasoning correctness across diverse backbones while preserving general editing behavior.
Problem
Image editors can produce visually plausible results while failing to satisfy implicit multi-dimensional logical constraints beyond surface instruction alignment.
Method
The paper introduces RE-Edit, a 1,000-sample benchmark with five reasoning dimensions and aligned evaluation criteria, and evaluates 12 editors alongside EditRefine.
Results
State-of-the-art editors produce visually plausible outputs that can violate physical, causal, or contextual logic, while EditRefine consistently improves reasoning correctness and preserves general editing behavior.
Takeaways & Limitations
Explicit reasoning can be incorporated as a lightweight, model-agnostic post-edit refinement step to partially mitigate logic-dependent editing failures.
Takeaways & Limitations
The evaluator defaults unrelated environmental or cultural edits toward penalization unless context remains perfectly intact, while intentionally unrealistic requested effects are exempt.
Abstract
from arXiv · showhide
Diffusion-based image editing has achieved strong visual fidelity under natural language instructions, yet most existing systems still operate at the level of surface instruction following, without reasoning about the implicit contextual constraints embedded in real user requests. This often leads to visually plausible but logically inconsistent edits. In this work, we introduce RE-Edit, a benchmark for REasoning-aware image Editing that evaluates image editing systems across five complementary reasoning dimensions: physical, environmental, cultural, causal, and referential. RE-Edit comprises 1,000 carefully curated samples, each designed such that visual plausibility alone is insufficient and correct editing requires satisfying implicit logical constraints. To support fine-grained analysis, we establish dimension-aligned evaluation criteria and conduct a comprehensive study of ten open-source and two commercial image editing models. Our results show that even advanced systems frequently struggle with implicit multi-dimensional reasoning despite producing high-quality visuals. We further present a lightweight reasoning-guided post-edit baseline as an initial exploration, illustrating how inserting explicit reasoning can help mitigate such failures in a model-agnostic manner.
1. Introduction
Existing image editors achieve strong visual fidelity but largely translate explicit language into appearance, missing implicit logical constraints in real requests. RE-Edit addresses this gap with a five-dimensional benchmark, aligned evaluation, broad model study, and a lightweight reasoning-guided refinement baseline.
- Diffusion-based image editing has improved visual fidelity and controllability but still predominantly performs surface-level instruction following.
- Implicit requests require reasoning about intended targets, physical plausibility, scene context, cultural conventions, and causal coherence.
- RE-Edit contains 1,000 curated samples spanning physical, environmental, cultural, causal, and referential reasoning, where visual plausibility alone is insufficient.
- The benchmark establishes dimension-aligned criteria and evaluates 12 state-of-the-art systems, including ten open-source and two commercial editors.
- EditRefine provides a lightweight post-edit framework that uses explicit reasoning to diagnose inconsistencies and generate corrective instructions without changing the underlying editor.
3. RE-Edit Benchmark
RE-Edit is a human-in-the-loop benchmark for implicit reasoning in image editing, organized around five dimensions and 1,000 balanced, rationale-annotated samples. It evaluates both reasoning correctness and general editing quality using dimension-aligned and conventional metrics.
- RE-Edit organizes implicit editing constraints into physical, environmental, cultural, causal, and referential dimensions.
- Its curation pipeline designs dimension-aligned seed cases, expands them into image-instruction-rationale triples, and validates their logical requirements.
- Images are synthesized at a fixed resolution of 1472 × 1104 to support controllable coverage of rare or complex editing scenarios.
- The resulting benchmark contains 1,000 balanced samples with original images, reasoning-intensive instructions, rationales, dimensions, and difficulty annotations.
- Evaluation Metrics: Reasoning correctness uses rationale-anchored binary Pass/Fail judgments and category passing rates, while SC and IF measure general editing quality on 0-10 scales.
4. Reasoning-Guided Post-Edit (EditRefine)
EditRefine is a plug-and-play second-pass module that diagnoses logical inconsistencies in initial edits and applies corrective re-editing. Its reasoning agent is trained with supervised fine-tuning and reinforcement learning using dimension-aware rewards.
- EditRefine takes an initial edited result and performs diagnostic reasoning before applying a corrective re-edit when necessary.
- The framework combines a multi-step Reasoning Agent with an off-the-shelf diffusion-based Execution Engine.
- Its implementation uses Qwen2.5-VL-7B as the Reasoning Agent and Qwen-Image-Edit as the Execution Engine.
- Training: The reasoning agent is trained through supervised fine-tuning followed by reinforcement learning.
- Training: A Max-Deviation reward prioritizes the most salient reasoning error, while a format reward encourages structured outputs.
5. Experiments
Experiments show that current editors often preserve visual quality while violating implicit physical, environmental, cultural, and causal constraints. EditRefine improves reasoning consistency across several backbones, while one-pass refinement outperforms iterative pixel-level editing.
- Evaluation: Ten open-source and two commercial editors are evaluated on RE-Edit across five reasoning dimensions and two general metrics.Evaluation uses Qwen3-VL-30B, with Executor-F and Executor-Q denoting FLUX.2 Dev and Qwen-Image-Edit.
- State-of-the-Art Results: Strong editors score better on Referential consistency but remain weak on implicit dimensions, including Environmental 14.8, Causal 15.0, and Cultural below 5.0 for many models.FLUX.2 Dev is one example of a strong editor with low Environmental and Causal consistency.
- EditRefine Results: EditRefine raises Qwen-Image-Edit Causal consistency by ↑4.5 and Cultural consistency by ↑2.8 points.Similar gains are reported for FLUX.2 Dev, FLUX.1.Kontext, and commercial Nano Banana.
- EditRefine Results: EditRefine also improves Nano Banana by Cultural ↑2.2, Causal ↑1.0, and Referential ↑1.5 while keeping IF and SC stable or slightly improved.The result supports plug-and-play use across a commercial editor as well as open-source backbones.
- Ablation Study: The full SFT+RL EditRefine pipeline generally achieves the strongest results, following gains from a reasoning module and further task alignment through SFT.The ablation attributes cumulative improvement to second-stage reasoning-guided refinement, explicit alignment, and subsequent optimization.
- Ablation Study: Iterative pixel-level refinement underperforms one-pass refinement, with FLUX.2 Dev dropping Physical consistency by ↓4.2 and Causal consistency by ↓5.5.The authors attribute degradation to accumulated artifacts, semantic drift, and over-editing across repeated edits.
- Robustness: Real-image counterparts exhibit the same reasoning failure patterns as the qualitative benchmark cases.This extends the observed failure pattern beyond synthetic benchmark images.
6. Conclusion
The paper identifies a gap between surface instruction compliance and satisfaction of implicit logical constraints in image editing. It introduces RE-Edit to characterize this gap and EditRefine as an initial model-agnostic mitigation, which improves reasoning correctness while preserving general editing behavior.
- Conclusion: RE-Edit organizes reasoning-aware image editing around five complementary dimensions inspired by human editing workflows.The dimensions are physical, environmental, cultural, causal, and referential.
- Conclusion: State-of-the-art editors can produce visually plausible results that violate physical, causal, or contextual logic.This demonstrates that visual plausibility does not guarantee reasoning correctness.
- Conclusion: EditRefine is a lightweight, model-agnostic post-edit refinement framework for mitigating reasoning failures.It is presented as an initial step toward improving reasoning-aware image editing.
- Conclusion: Across diverse backbones, EditRefine consistently improves reasoning correctness on RE-Edit while preserving general editing behavior.The conclusion frames this as evidence supporting further principled evaluation and modeling of reasoning in image editing.
7. Training Pipeline Implementation Details
EditRefine uses a two-stage reasoning-agent training pipeline: supervised fine-tuning initializes structured reasoning, then GRPO reinforcement learning optimizes dimension-aware rewards.
- The SFT dataset contains 1,600 curated OmniEdit samples augmented with synthesized structured reasoning chains.
- During rollouts, refined instructions are executed by frozen Qwen-Image-Edit and evaluated for dimension-specific feedback.
- The Max-Deviation Strategy selects the largest score deviation across reasoning dimensions to prevent reward dilution.
- The reward combines the selected reasoning signal with a rule-based format reward, while GRPO uses group advantages to update the policy.
- SFT uses AdaLoRA parameter-efficient adaptation, followed by one GRPO reinforcement-learning epoch.
8. Detailed Evaluator Comparison
The evaluator comparison combines human validation, the EditRefine training-pipeline overview, and GPT-4.1 benchmark results to assess model rankings and reasoning improvements.
- Ten annotators judged whether outputs from five editing models satisfied target reasoning requirements using binary labels.Each model contributed 50 cases per reasoning dimension, producing 250 cases per model.
- Figure 7 depicts EditRefine’s online loop, where a Reason Agent generates refined instructions that are executed and evaluated for policy-update feedback.
- Human scores were designed to verify whether human evaluation preserved the relative ranking of competing models rather than matching VLM absolute scores.
- Table 4 reports GPT-4.1 evaluations across five reasoning dimensions and highlights gains from adding EditRefine to corresponding backbones.
9. Inference Cost Analysis
EditRefine adds a reasoning stage and second editing pass, with limited reasoning overhead and stronger reasoning performance than a matched-budget multi-pass baseline.
- 9.1. Runtime Breakdown: The VLM-based reasoning stage accounts for only 1.8%∼3.7% of total runtime across representative settings.Most additional latency comes from the second editing pass rather than intermediate reasoning.
- 9.1. Runtime Breakdown: The full pipeline comprises an initial edit, VLM-based reasoning, and a second refinement edit.
- 9.2. Cost–Quality Trade-off: Under comparable computation budgets, EditRefine improves multiple reasoning dimensions over EditThinker-FLUX.1-Kontext, especially Environmental, Cultural, Causal, and Referential reasoning.
- 9.2. Cost–Quality Trade-off: Adding a second editing pass alone does not consistently improve reasoning performance, whereas EditRefine gains over both single-pass editing and the matched-budget baseline.
10. Generalization Across Original Image Sources
The generalization analysis tests RE-Edit across alternative synthetic generators and matched real-image cases, finding that its comparative conclusions remain broadly consistent.
- 10.1. Alternative Synthetic Source: Replacing Qwen-Image with FLUX.2 Dev while preserving the construction and evaluation protocols leaves relative model rankings largely consistent.
- 10.1. Alternative Synthetic Source: The benchmark therefore preserves relative model comparisons across different synthetic source-image generators.
- 10.2. Real-Image Counterparts: Real-image counterparts were constructed to match the original qualitative cases in scene semantics and editing intent.
- 10.2. Real-Image Counterparts: The same reasoning failure patterns appear in real-image settings, suggesting the gaps are not merely artifacts of synthetic benchmark construction.
11. Qualitative Examples from RE-Edit and EditRefine
The qualitative examples present RE-Edit cases alongside representative model outputs and EditRefine results, highlighting reasoning failures and corrected edits across benchmark challenges.
- The benchmark entries combine an original image, an implicit editing instruction, and a rationale explaining the latent logical constraint.
- Each qualitative example pairs a RE-Edit benchmark case with a representative editing-model output and an EditRefine result.
- Red annotations identify typical reasoning failures in baseline outputs, whereas green annotations mark corrections after EditRefine.
12. Prompt Templates
The prompt templates specify reproducible reasoning and evaluation procedures for EditRefine and the five RE-Edit dimensions, using strict structured outputs and binary judgments.
- 12. Prompt Templates: The appendix documents system prompts for the EditRefine reasoning agent and the dimension-specific RE-Edit evaluators.
- 12. Prompt Templates: General Semantic Consistency and Instruction Following evaluations follow the official VIEScore and UnicEdit prompt implementations.
- 12.2. Prompts for RE-Edit Automated Evaluator: The evaluator uses five dimension-specific prompts with Qwen3-VL-30B or GPT-4.1 to produce strict binary scores.
- 12.2. Prompts for RE-Edit Automated Evaluator: EditRefine receives the original description, edit instruction, and rationale, then outputs reasoning and imperative re-edit instructions in separate XML-style tags.
- 12.2. Prompts for RE-Edit Automated Evaluator: The EditRefine template permits multiple reasoning and re-edit tags, requires each tag on its own line, and forbids JSON, code fences, explanations, and extra text.
- 12.2. Prompts for RE-Edit Automated Evaluator: The physical evaluator checks lighting, contact, scale, materials, motion, gravity, and visual harmony, while treating impossible global color distortion as an immediate failure.
- 12.2. Prompts for RE-Edit Automated Evaluator: Physical evaluation defaults to NO when uncertain, borderline, questionable, ambiguous, or implausible, while remaining neutral toward intentionally unrealistic physical edits.
- 12.2. Prompts for RE-Edit Automated Evaluator: Environmental evaluation assesses consistency with time, weather, climate, surroundings, atmosphere, and overall scene logic under the same strict uncertainty policy.