PaperlayerView exact revision on arXiv

Source-linked AI summary

UniREditBench: A Unified Reasoning-based Image Editing Benchmark

Feng Han, Yibin Wang, Chenglin Li, Zheming Liang, Dianyi Wang, Yang Jiao, Zhipeng Wei, Chao Gong, Cheng Jin, Jiaqi Wang

arXiv:2511.01295v3cs.CV

TL;DR

Existing image-editing benchmarks underrepresent multi-object and rule-governed game scenarios and often rely on text-only references for evaluation. UniREditBench addresses these gaps with broader coverage and dual-reference assessment, while UniREdit-Data-100K and UniREdit-Bagel extend the work to data synthesis and model training. UniREdit-Bagel achieves the best overall performance on UniREditBench and strong out-of-distribution results, including a +17.08 gain in game-world scenarios.

  • Problem

    Current benchmarks emphasize single-object realistic edits and text-only evaluation, limiting assessment of complex reasoning across multi-object and game-world scenarios.

  • Method

    The paper introduces UniREditBench with 2,700 samples across 8 dimensions and 18 sub-dimensions, dual textual and ground-truth image references, and a multi-scenario synthesis pipeline producing UniREdit-Data-100K.

  • Results

    +17.08 in game-world scenarios; UniREdit-Bagel achieves the best overall UniREditBench performance and improves RISEBench results by 9.1 points over Bagel-Think.

  • Takeaways & Limitations

    The benchmark exposes strengths and weaknesses across diverse reasoning-based editing scenarios, while the resulting model narrows the gap between open- and closed-source performance.

  • Takeaways & Limitations

    The paper focuses on complex reasoning edits involving multi-object interactions and human-defined rules, rather than only simple perceptual instruction following.

Abstract

from arXiv · show

Recent advances in multi-modal generative models have driven substantial improvements in image editing. However, current generative models still struggle with handling diverse and complex image editing tasks that require implicit reasoning, underscoring the need for a comprehensive benchmark to systematically assess their performance across various reasoning scenarios. Existing benchmarks primarily focus on single-object attribute transformation in realistic scenarios, which, while effective, encounter two key challenges: (1) they largely overlook multi-object interactions as well as game-world scenarios that involve human-defined rules, which are common in real-life applications; (2) they only rely on textual references to evaluate the generated images, potentially leading to systematic misjudgments, especially in complex reasoning scenarios. To this end, this work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation. It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions. To improve evaluation reliability, we introduce multimodal dual-reference evaluation, providing both textual and ground-truth image references for each sample assessment. Furthermore, we design an automated multi-scenario data synthesis pipeline and construct UniREdit-Data-100K, a large-scale synthetic dataset with high-quality chain-of-thought (CoT) reasoning annotations. We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings. Through thorough benchmarking of both open-source and closed-source image editing models, we reveal their strengths and weaknesses across various aspects.

1 Introduction

UniREditBench addresses gaps in reasoning-based image-editing evaluation by covering diverse real- and game-world scenarios and adding ground-truth visual references. The work also introduces a synthetic CoT dataset and fine-tuned model to improve performance across in-domain and out-of-distribution benchmarks.

  • Existing benchmarks mainly target single-object attribute changes in realistic settings, leaving multi-object interactions and rule-governed game scenarios underrepresented.
  • Text-reference-only evaluation can systematically misjudge complex edits, such as awarding an inflated score when an edited image adds a faulty path.
  • UniREditBench covers 2,700 samples across 8 primary dimensions and 18 sub-dimensions, including real-world multi-object interaction and game-world logical or strategic reasoning.
  • UniREditBench adds ground-truth images alongside textual references, enabling direct visual comparison for more accurate and reliable assessment.
  • UniREdit-Data-100K provides high-quality VLM-generated step-by-step CoT annotations, and fine-tuned UniREdit-Bagel improves on UniREditBench and out-of-distribution benchmarks.

2 Related Work

Prior image-editing benchmarks assess instruction following and reasoning but largely emphasize single-object knowledge or attribute reasoning and often use text-only evaluation. UniREditBench broadens scenario coverage and combines real- and game-world cases with multimodal dual-reference assessment.

  • Existing reasoning benchmarks cover temporal, spatial, logical, factual, conceptual, and procedural editing, but primarily focus on single-object knowledge and attribute reasoning.
  • Many prior evaluations rely on CLIP-based metrics or VLM judges with text-only references, which can be insufficient or inaccurate for complex reasoning-intensive edits.
  • UniREditBench extends evaluation to multi-object interactions and human-rule-governed game scenarios across broader reasoning dimensions.
  • The benchmark uses a multimodal dual-reference protocol to support more reliable and accurate assessment across real-world and game-world editing scenarios.

3 UniREditBench

UniREditBench broadens reasoning-based image-editing evaluation across real- and game-world scenarios and combines textual with ground-truth visual references. Its VLM-based scoring assesses instruction following, visual consistency, and visual quality, supported by scenario-specific data synthesis.

  • 3.1 Overview: Existing benchmarks emphasize simple single-object real-world edits and text-only evaluation, limiting coverage of multi-object interactions, rule-based game scenarios, and complex reasoning assessment.The paper identifies these as two major challenges motivating UniREditBench.
  • 3.1 Overview: UniREditBench covers 2,700 samples across 8 primary dimensions and 18 sub-dimensions spanning real-world and game-world editing tasks.The benchmark uses a scenario-to-category hierarchy to represent diverse reasoning contexts.
  • 3.3 Dual-Reference Evaluation: The benchmark introduces multimodal dual-reference evaluation, comparing generated images with both textual edited-effect references and corresponding ground-truth images.This design supports direct textual and visual comparison for more reliable assessment.
  • 3.3 Dual-Reference Evaluation: Its scoring schema evaluates instruction following, visual consistency, and visual quality using a GPT-4.1 VLM evaluator and a weighted overall score.Visual consistency checks whether unrelated regions remain preserved, while visual quality checks realism, artifacts, distortions, and physical or logical plausibility.
  • 3.4 Multi-Scenario Data Synthesis: Scenario-specific synthesis uses a text-then-image strategy for real-world data and programmatic generation for rule-intensive game-world data, followed by VLM filtering and human inspection.The pipeline is designed to maintain logical and visual correctness across distinct scenario types.

4 UniREdit-Data-100K

UniREdit-Data-100K is a balanced, diverse reasoning-based editing dataset spanning real-world and game-world tasks with stepwise CoT annotations. Fine-tuning Bagel on this data supervises both textual reasoning and image generation, improving editing correctness and fidelity.

  • 4 UniREdit-Data-100K: UniREdit-Data-100K contains 100,421 samples spanning 8 reasoning dimensions and 18 categories.Each reasoning category contains more than 4,000 instances to support training across varied editing tasks.
  • 4 UniREdit-Data-100K: The dataset balances real-world object and multi-object editing with game-world puzzles and strategic planning under structured rules.These two scenarios provide diverse reasoning-based editing challenges.
  • 4.2 UniREdit-Bagel: Training samples pair an original image, editing instruction, stepwise CoT text, and target edited image for joint reasoning and editing supervision.Bagel receives the original image and instruction, then generates a reasoning trace and synthesizes the edited image.
  • 4.2 UniREdit-Bagel: The overall training objective combines text-generation loss with latent flow-matching image loss.The text loss supports explicit CoT reasoning, while the image loss improves correctness and fidelity of edited outputs.
  • 4.2 UniREdit-Bagel: UniREdit-Bagel demonstrates superiority in instruction following and visual quality compared with state-of-the-art closed-source and open-source models.The qualitative comparison reports advantages in both dimensions.

5 Experiment

Experiments benchmark closed- and open-source editors, evaluate UniREdit-Bagel, and test the reliability of the dual-reference evaluation protocol. Results show strong gains on complex game-world reasoning and close alignment between VLM and human scores.

  • Benchmarking Results on UniREditBench: GPT-4o leads closed-source models overall, while most open-source models remain weak on strategic game scenarios.Only GPT-4o and Nano Banana achieve average game-scenario scores above 60.
  • Comparison Results of UniREdit-Bagel: UniREdit-Bagel achieves the best overall UniREditBench performance, surpassing GPT-4o, with its largest gain in game-world scenarios at +17.08.On RISEBench, it also improves over Bagel-Think by 9.1 points and Gemini-2.0-Flash-exp by 5.0 points.
  • Comparison Results of UniREdit-Bagel: UniREdit-Bagel maintains instruction fulfillment and unrelated-content coherence in challenging box-pushing and maze-completion tasks.These qualitative cases highlight failures of other models on game actions and content preservation.
  • Evaluation Reliability: GPT-4.1 scores align closely with six human experts, with average MAE below 1 across evaluated models and score dimensions.The user study covers 200 samples from Qwen-Image-Edit, Nano Banana, GPT-4o, and UniREdit-Bagel.
  • Evaluation Reliability: The dual-reference protocol consistently yields the lowest human–VLM MAE compared with omitting either the image or text reference.This ablation supports using both textual and image references for evaluation.

6 Conclusion

The paper introduces a broad reasoning-based image-editing benchmark, a dual-reference evaluation protocol, and a synthetic CoT dataset. Fine-tuning Bagel on this dataset produces substantial quantitative and qualitative gains while benchmarking exposes model strengths and weaknesses.

  • Conclusion: UniREditBench broadens reasoning-based image-editing evaluation through wide dimension coverage and a robust dual-reference protocol.The benchmark is paired with real- and game-world reasoning scenarios.
  • Conclusion: UniREdit-Data-100K provides large-scale synthetic training data with high-quality CoT annotations for reasoning-based image editing.The paper uses this dataset to fine-tune Bagel into UniREdit-Bagel.
  • Conclusion: UniREdit-Bagel achieves substantial quantitative and qualitative gains, while comprehensive benchmarking reveals strengths and weaknesses across aspects and model types.The conclusion covers both open-source and closed-source image-editing models.

Supplementary Material

Supplementary analyses detail benchmark design, data filtering, score weighting, and additional model comparisons. They emphasize broader dual-reference evaluation and quality-controlled synthetic data construction.

  • Benchmark Design: UniREditBench provides reference images for every instance, combines text and image references, and covers real-world, rule-based game-world, and intra-game multi-task scenarios.The game setting evaluates image edits that execute actions while obeying explicit rules.
  • Evaluation Analysis: The weighting scheme (0.50, 0.30, 0.20) achieves the highest average Pearson correlation with human judgments and is adopted.The weights correspond to Instruction Following, Visual Consistency, and Visual Quality.
  • Data Construction: The synthesis pipeline removes redundancy through exact-match and semantic-similarity deduplication, improving dataset diversity.The pipeline targets both literal and paraphrastic duplicates.
  • Additional Analyses: Supplementary figures and tables provide additional benchmark visualizations, weighting results, and quantitative comparisons across closed- and open-source models.The cited materials include closed/open-source comparisons and KRISBench results.
  • Data Construction: Generated samples are filtered across six dimensions and retained only when they receive maximum scores on every dimension.Dimensions include hallucination, instruction adherence, content preservation, visual quality, image hallucination, and CoT quality.
  • Data Construction: Eight expert annotators perform final manual filtering and correction after automated quality checks.They remove severely erroneous samples and refine slightly incorrect textual-reference effects.

D More Quantitative Results

Additional KRISBench results show that UniREdit-Bagel achieves the strongest open-source performance and surpasses several closed-source models.

  • More Quantitative Results: UniREdit-Bagel achieves the best open-source performance on KRISBench and outperforms several closed-source models, including Gemini-2.0 and Doubao.The model was fine-tuned on UniREdit-Data-100K.

E Detailed Benchmarking Results

UniREditBench results are reported through detailed quantitative comparisons, with GPT-4.1 serving as the evaluator in the main benchmark table.

  • Detailed quantitative comparisons on UniREditBench are provided in Table 7 using GPT-4.1 as the evaluator.
  • Table 8 reports detailed UniREditBench comparisons using Qwen3VL-32B-Instruct as the evaluator.

F Open-source VLM Evaluation

Qwen3VL-32B-Instruct produces category-level UniREditBench scoring trends consistent with GPT-4.1.

  • Qwen3VL-32B-Instruct yields scoring trends consistent with GPT-4.1 across UniREditBench categories.

G Transferability of Game-world Data

The experiments examine transfer from game-world data, the role of chain-of-thought reasoning, and performance scaling with training data, while qualitative analyses expose rule-consistency failures.

  • Transferability of Game-world Data: Incorporating game-world data consistently improves performance across all four real-world evaluation categories, indicating positive cross-domain transfer.
  • Transferability of Game-world Data: CoTs provide clear causal chains that steer image editing toward the intended outcome, assessed through causal-reasoning and edit-alignment criteria.
  • Transferability of Game-world Data: Performance on UniREditBench consistently improves as UniREdit-Data training size increases from 20K to 100K samples.
  • Transferability of Game-world Data: UniREditBench reveals logical hallucinations in which models delete non-target gems or duplicate boxes, violating object-permanence and conservation rules.
  • Transferability of Game-world Data: Game-world scenarios impose explicit constraints covering elimination, movement, pushing, scoring, spatial connectivity, and valid puzzle solutions.
Loading 2511.01295v3…