Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

Benchmark design

• EditReward-Compass contains 2,251 preference pairs designed to simulate reward-model decisions during RL-based image editing optimization.This addresses the distribution mismatch between benchmark samples and edited images encountered during RL training, providing a realistic testbed for reward-guided optimization.Source: §1 · p. 2

Empirical findings

• Across 29 image editing models, the best proprietary model scores 3.99 overall, whereas the strongest open-source model, Qwen-Image-Edit [48], reaches 2.69.The evaluation includes proprietary systems such as Nano-Banana Pro, Wan2.7-Image, and Seedream 4.5 [33], alongside open-source models including Qwen-Image-Edit [48] and Joy-Image-Edit.Source: §1 · p. 2
• Native multimodal large language models outperform existing open-source reward models across the reward-model evaluation, including models explicitly trained on preference data.The result indicates that current reward models remain limited in assessing visual consistency and perceptual quality under complex editing scenarios.Source: §1 · p. 2

Motivation and findings

• Frontier image editing systems perform reasonably on shallow perception-level edits but still struggle with multi-image understanding, world knowledge, visual reasoning, and complex reasoning.These weaknesses motivate benchmarks with greater task difficulty and finer-grained protocols because existing benchmarks can diverge noticeably from human judgment for strong frontier models.Source: §1 · pp. 1–2

2.1 Benchmarks for Image Editing

• Existing image-editing benchmarks [56] [34] [54] [29] [53] [23] [59] [55] provide limited coverage or unstable evaluation, motivating Edit-Compass’s 36 fine-grained tasks across six categories and structured scoring.Early benchmarks emphasize narrow tasks and automated metrics, whereas recent MLLM-based judges often rely on simple prompts that can misalign with human judgment in complex scenarios.Source: §2.1 · p. 3

2.2 Benchmarks for Image Editing Reward Model

• Existing benchmarks [24] [51] [14] often use limited editing tasks or outputs from different models, unlike practical RL comparisons of same-model outputs under the same instruction.Source: §2.2 · p. 3

3.1 Task Taxonomy

• 36 diverse tasks span general, dynamic manipulation, world knowledge, algorithmic visual reasoning, multi-image, and complex editing in single- and multi-image settings.The taxonomy includes novel Copy, Multi-Image-Aware Editing, and Complex Paint tasks alongside global, local, dynamic, domain-reasoning, and compound-editing scenarios.Source: §3.1 · pp. 3–5, Figure 1 · p. 4
• Complex tasks evaluate coherent execution of multiple editing intents, with Complex Paint combining textual instructions and visual cues such as arrows, circles, and cross marks.This setting targets compositional and multimodal editing in interactive scenarios.Source: §3.1 · p. 5

3.2 Benchmark Construction

• Category-specific construction uses collected or generated images, executable instructions, algorithmic ground truth, unified templates, bilingual expert design, and multiple human reviews.General and Complex instructions are generated with Gemini 3 Pro and GPT-5.1 and human-verified; other categories use expert scenarios or Python-generated data.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 Evaluation Pipeline

• Evaluation uses an MLLM-as-judge pipeline across Instruction Awareness, Visual Consistency, and Visual Quality, producing scalar scores and fine-grained rationales.Reasoning-intensive tasks additionally receive expected ground-truth outcomes in the rationale for improved accuracy and interpretability.Source: §3.3 · p. 6
• Instruction Awareness measures explicit instruction following and implicit world-knowledge use, while Visual Consistency measures unedited-region and identity preservation.Visual Quality separately assesses plausibility, coherence, naturalness, structural fidelity, artifacts, distortion, and text legibility when applicable.Source: §3.3 · p. 6

4 EditReward-Compass

• EditReward-Compass contains 2, 251 preference pairs and uses Edit-Compass’s rubric-based judging framework for consistent reward-model assessment.The framework also supports examining the robustness and generality of the evaluation prompts.Source: §4 · p. 6

4.1 Sampling Stage

• Candidate outputs are sampled from six editing models using a FlowGRPO-inspired stochastic strategy, expanding to ten models for world-knowledge and complex-reasoning tasks.Denoising steps are controlled to ensure visually clear and valid results, while the expanded pool improves task diversity and coverage.Source: §4.1 · p. 6

4.2 Human Annotation Stage

• Eight experts select preference pairs through independent review, discussion of ambiguous cases, and unanimous five-annotator verification of task validity and preference correctness.The pipeline emphasizes instruction adherence and visual consistency alongside visual quality, and removes samples without consensus.Source: §4.2 · p. 6

5.1 Experimental Setup

• The evaluation covers 29 image-editing models across diffusion, unified multimodal, and proprietary systems, plus general-purpose, preference-trained, and proprietary reward models.The image-editing evaluation includes 25 open-source and 4 proprietary models.Source: §5.1 · p. 7

5.2 Main Results

• Qwen-Image-Edit [48] achieves the best overall performance among open-source models under English and Chinese instructions, while Nano Banana Pro performs best among closed-source models; substantial gaps remain on challenging categories.On World Knowledge Reasoning, Nano Banana Pro scores 3.89, compared with 1.74 for Qwen-Image-Edit [48].Source: §5.2 · p. 7

5.3 Analysis and Findings for Edit-Compass

• Edit-Compass agrees more strongly with human preferences than existing benchmarks, and its automatic scores correlate with human ratings across outputs from different models.Benchmark-level evaluation samples ImgEdit-Bench [53], GEdit-Bench [23], RISE-Bench [59], and Edit-Compass; model-level evaluation uses Pearson correlation.Source: §5.3 · p. 8
• Open-source models remain weak on Algorithmic Visual Reasoning, and closed-source models show only limited potential across its subtasks.The difficulty reflects both visual reasoning and faithful execution of the resulting edits.Source: §5.3 · p. 8

5.2 Main Results

• Qwen3.5-9B surpasses Qwen3-VL-32B [1] and matches much larger Gemma4-31B, while EditReward [51] outperforms EditScore [24] overall on the same Qwen2.5-VL backbone.The pointwise EditReward variant has a slight advantage over the pairwise variant under the evaluation protocol.Source: §5.2 · p. 8, Table 5 · p. 9

5.3 Analysis and Findings for Edit-Compass

• Open-source models handle basic visual perception such as Object Movement better than Object Swap and Complex Paint, while closed-source models lead on multi-image perception.Complex Paint requires interpreting textual instructions together with in-image visual annotations.Source: §5.3 · p. 8, Table 6 · p. 9

5.4 Analysis and Findings for EditReward-Compass

• EditReward-Compass system prompts improve every evaluation dimension, with the largest gain reaching 12.93% on Qwen3-VL-8B [1].The comparison uses corresponding single-image subsets and contrasts EditReward-Compass prompts with EditScore prompts.Source: Table 6 · p. 9, §5.4 · p. 9

5.3 Analysis and Findings for Edit-Compass

• Some models exhibit English-over-Chinese performance imbalance, whereas many advanced unified models show only marginal cross-lingual differences.The results suggest that robust multilingual editing depends on language-vision understanding and balanced multilingual training data.Source: §5.3 · p. 9

• Edit-Compass contains 2,388 annotated instances across 36 progressively challenging categories, using structured reasoning and fine-grained multidimensional scoring rubrics.The categories span general editing, world knowledge reasoning, visual reasoning, dynamic manipulation, and multi-image editing.Source: §6 · p. 10
• EditReward-Compass provides 2,251 preference pairs that simulate realistic reward-modeling scenarios during RL optimization.Source: §6 · p. 10
• Evaluations of 29 image editing models and 21 reward models reveal gaps between proprietary and open-source systems, weaknesses in reasoning-intensive and multi-image editing, and promise for native multimodal language models as reward models.Source: §6 · p. 10
• The evaluation relies on API-based multimodal language-model judges, so scores remain sensitive to judge capabilities and version updates despite improved rubric interpretability and human alignment.This dependence also limits accessibility.Source: §6 · p. 10
• Future work will develop a dedicated image-editing judge model for more stable and transparent evaluation without proprietary APIs.Source: §6 · p. 10

A Edit-Compass Data Construction

• The Edit-Compass data construction comprises three main components.Figure 2 presents the overall construction structure.Source: A Edit-Compass Data Construction · p. 14

A.1 General and Complex tasks.

• Real images are approved unanimously by five experts, while Gemini 3 Pro and GPT-5.1 generate candidate instructions that experts select for feasibility, clarity, and task alignment.Images come from permissively licensed sources and are reviewed for safety, quality, and editing suitability.Source: §A.1 · p. 14

Dynamic Manipulation, World Knowledge Reasoning, and Multi-Image Tasks

• Experts define each task, refine coarse source-image descriptions with Gemini 3 Pro, generate images, and retain only cases unanimously approved as feasible and valid.The pipeline combines expert discussion, model-assisted source-image generation, and multi-expert quality control.Source: §A.2 · p. 14

A.3 Algorithmic Visual Reasoning tasks.

• Experts define target editing problems, render source images and annotations with Python programs, and create category-specific instruction templates for structurally unambiguous instances.The templates are applied to corresponding cases so each task has a well-defined visual structure and expected outcome.Source: §A.3 · p. 14

A.3.1 Longest Word Discovery; A.3.2 Global Longest Word Discovery

• The word-search tasks generate challenging, verified instances: Longest Word Discovery embeds curated morphologically or semantically complex words, while global discovery searches all starting positions and retains only uniquely matching longest words.A Python reconstruction pipeline controls text content and layout; trie-based exhaustive verification ensures the target is reachable and that no longer valid word exists under the movement constraint.Source: §A.3.1 · pp. 14–15, §A.3.2 · p. 15

A.3.3 Knapsack Selection; A.3.4 Optimal Path Identification; A.3.7 Maximum Bonus Identification

• The optimization tasks require visual selection or annotation of exact solutions, covering budgeted knapsack selection, minimum-cost four-neighbor paths, and maximum-reward monotone paths.Knapsack instances use dynamic programming for optimal subsets; terrain-path instances use Dijkstra’s algorithm with road, grass, water, and impassable wall costs; bonus paths use dynamic programming and backtracking.Source: §A.3.3 · p. 16, §A.3.4 · p. 16, §A.3.7 · p. 18

A.3.5 Convex Hull Identification; A.3.6 Maximum Submatrix Sum Identification

• Convex Hull Identification asks models to recover the polygon enclosing sampled 2D points, while Maximum Submatrix Sum Identification asks them to locate the fixed-size rectangular submatrix with the greatest integer sum.The hull data are procedurally sampled from 8–15 points on a 10 × 10 canvas, whereas submatrix grids contain independently sampled integers from −9 to 9 with kernel dimensions between 2 and min(4, n − 1).Source: §A.3.5 · pp. 16–17, §A.3.6 · p. 17

A.3.8 Numberlink Path Identification

• Numberlink Path Identification hides intermediate cells and requires reconstructing non-intersecting paths connecting multiple same-colored endpoint pairs.Instances contain 3–5 path pairs on 6–9-sized grids, generated as sequential self-avoiding random walks whose accepted paths exceed a predefined minimum length.Source: §A.3.8 · p. 18

A.3.9 Word Path Recovery

• Word Path Recovery embeds a supplied target word along a monotone down/right path and requires recovering the corresponding highlighted cells.Target words vary in length to control difficulty, and local blocking with low-frequency letters reduces unintended continuations beyond the target.Source: §A.3.9 · pp. 18–19

A.3.10 Global Word Path Recovery

• Global Word Path Recovery removes the initial-letter cue, requiring models to infer the complete target-word path without explicit starting-letter guidance.It retains the prior generation strategy while increasing the search requirement and maintaining balanced difficulty.Source: §A.3.10 · p. 19

B EditReward-Compass Data Construction

• EditReward-Compass uses diverse model sampling, expert filtering, and FlowGRPO-based RL-like distributions to create preference pairs with clear differences and realistic optimization conditions.Candidates come from ten image-editing models, while Table 7 explores model-specific sampling configurations that preserve behavioral diversity.Source: B EditReward-Compass Data Construction · p. 19

C Detailed Design of Edit-Compass Categories

• Edit-Compass organizes image-editing tasks into six distinct categories with progressively challenging capabilities and representative examples.The categories receive detailed definitions in subsequent sections.Source: C Detailed Design of Edit-Compass… · p. 19

General Tasks

• General tasks cover subject addition, removal, replacement, extraction, color and size changes, material and style transfer, background changes, and Chinese or English visual-text editing.Variants impose spatial, semantic, attribute, copying, preservation, or knowledge-guided constraints, and style transfer spans more than 60 styles.Source: General Tasks · p. 19

Dynamic Manipulation Tasks

• Dynamic manipulation tasks edit spatial position, subject swaps, interactions, emotions, and actions while preserving relevant identity, appearance, or structure.The category includes object movement, object swap, object interaction, change emotion, and action tasks.Source: Dynamic Manipulation Tasks · p. 20

World Knowledge Reasoning Tasks

• World knowledge reasoning tasks require temporal, causal, mathematical, chemical, game, and algorithmic visual reasoning to guide image edits.Temporal reasoning includes forecasting future changes and reconstructing past appearance, while algorithmic visual reasoning is detailed in Section A.3.Source: World Knowledge Reasoning Tasks · pp. 20–21

Multi-Image Tasks

• Multi-image tasks transfer attributes across images, compose coherent scenes from multiple inputs, and perform virtual try-on while preserving the person’s identity and pose.Referenced attributes include action, color, function, garments, subjects, backgrounds, and human figures.Source: Multi-Image Tasks · p. 21

Complex Tasks

• Complex tasks combine multiple edits or multimodal signals, including English or Chinese instructions and visual annotations such as arrows, circles, and cross marks.They comprise Complex Instruction and English- and Chinese-language Complex Paint tasks.Source: Complex Tasks · p. 21

D Image Editing Model Evaluation

• The benchmark evaluates 29 mainstream image editing models spanning open- and closed-source systems and Chinese and English variants.Closed-source models are assessed through their official API services because their weights are unavailable.Source: D Image Editing Model Evaluation · pp. 21–22
• Edit-Compass scores outputs with Instruction Following, World Knowledge Awareness, Unedited Region Consistency, Identity Consistency, and Visual Quality.Instruction Awareness combines IF and WA, Visual Consistency combines URC and IC, and Gemini-3.1-Pro rates applicable metrics on a 1–5 scale.Source: D Image Editing Model Evaluation · p. 22
• The protocol uses category-specific aggregation and prompt templates tailored to task groups, including complex instruction, complex paint, multi-image, and other tasks.World Knowledge Awareness, Unedited Region Consistency, Identity Consistency, and Visual Quality receive specialized templates according to task requirements.Source: D Image Editing Model Evaluation · pp. 22–23
• Human studies show that the automatic evaluation correlates highly with expert ratings and is more preferred by annotators than comparison protocols.The studies sample 180 balanced Edit-Compass instances and additional instances from ImgEdit-Bench, GEdit-Bench, RISE-Bench, and Edit-Compass.Source: Figure 3 · p. 23, D Image Editing Model Evaluation · p. 23

Box 1: Instruction Following System Prompts for Complex Instruction Tasks

• Complex instruction judging decomposes compound requests into atomic tasks and requires each task to pass target, attribute, and spatial checks while ignoring aesthetics and unrelated accidental changes.Interaction tasks additionally require the target object to change in accordance with the subject’s action.Source: Box 1: Instruction Following System… · p. 41, Step 1: Instruction Decoupling · p. 41

Step 2: Strict Visual Comparison

• The visual-comparison procedure first documents source and edited states, then verifies attributes, spatial requirements, object interaction, extraction constraints, and visual-text style preservation.It prohibits redundant object generation or cloning, requires extracted objects on pure white #FFFFFF backgrounds with consistent orientation, and preserves substituted text style and color unless instructed otherwise.Source: Step 2: Strict Visual Comparison · pp. 41–42
• Instruction adherence is scored by decomposing edits into atomic target, attribute, and spatial requirements, then assigning 1–5 based on completeness and precision.Perfect adherence requires every atomic task, attribute, and spatial requirement to be satisfied without omissions or errors.Source: Step 2: Strict Visual Comparison · p. 42, Step 1: Analyze Edit Instruction… · p. 49

Box 2: Instruction Following System Prompts for Complex Paint Tasks

• Annotated paint tasks extract every visual marker, associated arrow, location, and instruction before evaluating each marked edit independently.The evaluator ignores aesthetics and unintended changes outside the marked regions while enforcing strict spatial compliance.Source: Box 2: Instruction Following System… · p. 44, Step 1: Instruction Extraction (Task… · p. 44

Step 2: Comparative Verification (Source vs. Edited)

• Comparative verification checks whether each extracted instruction occurred, including color changes, removals, attributes, quantities, states, actions, and materials.A task passes only when the extracted instruction is successfully executed in the edited image.Source: Step 2: Comparative Verification… · p. 44

Core Constraints

• Multi-image evaluation transfers reference objects or attributes onto the source subject while ignoring aesthetics, unintended changes, and subject identity consistency.Evaluation prioritizes semantic equivalence and fidelity of details such as text, logos, textures, patterns, orientation, color, and action.Source: Core Constraints · p. 46, Step 1: Attribute Sourcing &… · p. 46, Step 2: Reference Fidelity & Content… · p. 46

Scoring Analysis & Final Rubric

• Reference-fidelity scoring ranges from non-response and hallucinated or generic features to perfect transfer of unique details adapted to the source subject’s morphology, lighting, and perspective.Inserting a reference object when only its attribute was requested is a failure, while perfect fidelity requires native-looking semantic and visual detail transfer.Source: Scoring Analysis & Final Rubric · pp. 46–47

Step 1: Analyze Edit Instruction Requirements

• Compound-edit evaluation requires complete task execution, correct object interactions, no auxiliary objects or cloning, and exact attribute, orientation, background, text, and spatial compliance.For replacement, the new object must occupy the original position; cloning the interacted object or leaving it static is a major failure.Source: Step 1: Analyze Edit Instruction… · p. 48

Box 5: System Prompts for World Knowledge Awareness

• World-knowledge evaluation judges factual correctness, algorithmic validity, and physical consistency while excluding non-edited-region changes, aesthetics, realism, and art style unless explicitly requested.The protocol requires a T.C.R.V. logical reasoning pipeline for verification.Source: Box 5: System Prompts for World… · p. 50

1. T – Task Identification (Domain)

• The evaluator identifies the task’s domain and core problem type before deriving the objective requirements and strict world-knowledge constraints governing correctness.Examples include algorithmic path, convex-hull, optimization, chemistry, biology, game, and mathematical rules.Source: §1 · p. 50, §2 · p. 50

3. R – Requirement Definition (Goals)

• Requirements specify the desired visual outcome, optimization objective, and complete execution rather than partial completion.The framework distinguishes what the edited image should look like from how success is measured and whether the operation is fully carried out.Source: §3 · pp. 50–51

4. V – Verification & Evidence (Audit)

• Verification compares the ideal outcome with the actual image, checks constraint and requirement breaches, and assigns scores from 1 for rule violation to 5 for perfect execution.Scores 2–4 distinguish goal failure, weak visual execution, and correct-but-incomplete results; incomplete tasks cannot exceed 4.Source: §4 · p. 51

Box 6: Unedited Region Consistency System Prompts for the Complex Paint Task

• For complex paint tasks, consistency auditing extracts edit targets, checks local non-target changes, and verifies global geometry, lighting, color, perspective, and lens stability.The audit uses the source, annotation, and edited images together with a difference description, and treats residual annotations or identity drift as inconsistencies.Source: Box 6: Unedited Region Consistency… · p. 52, Step 1: Instruction and Target… · p. 52, Step 2: Local Non-Target Consistency… · p. 52

Step 4: Consistency Scoring (1–5 Scale)

• The complex-paint consistency score ranges from 1 to 5, with 5 requiring indistinguishable non-edited regions and 1 indicating severe scene reconstruction or total consistency loss.Intermediate scores reflect subtle differences, obvious identity or object changes, and multiple unexpected alterations.Source: Step 4: Consistency Scoring (1–5 Scale) · p. 53

Box 7: Unedited Region Consistency System Prompts for Other Tasks

• For other tasks, the evaluator isolates intended from unintended differences, discounts logically caused occlusion, fact-checks local object persistence and attribute leakage, and then assesses global structure and lighting.Changes hidden behind or underneath a newly added object are acceptable side effects rather than deletions.Source: Box 7: Unedited Region Consistency… · p. 54, Step 1: Align Differences with… · p. 54, Step 2: Verify Local Consistency (Fact… · p. 55

Scoring Criteria (1–5 Scale)

• The general consistency rubric scores 5 for unchanged non-edited regions and scene structure, declining to 1 when object identities, geometry, perspective, and continuity are largely lost.Scores 3 and 4 capture obvious inconsistencies and minor detectable flaws, respectively.Source: Step 3: Check Global Structure &… · p. 55, Scoring Criteria (1–5 Scale) · p. 55

Box 8: Identity Consistency System Prompts for Multi-Image Task

• Multi-image identity evaluation freezes all source-object features except authorized instruction changes, penalizes attribute leakage and unintended cropping, ignores background and image quality, and awards full preservation credit even when instruction execution fails.Reference-image features are transferred only for the explicitly requested target attributes, while authorized identity changes are exempt from penalties.Source: Box 8: Identity Consistency System… · p. 56
• Multi-image identity checks compare source and edited images at coarse and fine granularity, selecting 3–5 mandatory invariants and rating preservation from 1 to 5.The rubric spans perfect preservation, micro-drift, feature distortion, instance error, and structural collapse.Source: Box 8: Identity Consistency System… · pp. 56–57

Output Format (Strict JSON)

• The protocols require strict JSON outputs containing target metadata, authorized changes, invariant audits, reasoning, and integer scores.Identity outputs include coarse- and fine-grained leakage analyses, while quality outputs include realism, artifacts, text quality, reasoning, and score fields.Source: Output Format (Strict JSON) · p. 57

Box 9: Identity Consistency System Prompts for Other Tasks

• Identity auditing for other tasks exempts authorized identity changes, penalizes unauthorized attribute leakage, ignores execution and non-edited-area quality, and requires valid two-object swaps.Swap tasks fail when one object is replaced or cloned without the reciprocal exchange.Source: Box 9: Identity Consistency System… · p. 58

Box 10: Image Quality Assessment Expert

• The image-quality expert must not penalize low pixel count, but must penalize noticeable blur, heavy noise, compression artifacts, and quality failures in prominent text.Text is not applicable when no large or central text is present.Source: Box 10: Image Quality Assessment Expert · p. 60

Evaluation Dimensions

• Image quality is judged by visual realism, local artifacts, and prominent text, using a 1–5 scale from outstanding realism to unusable severe degradation.Small or incidental text is ignored, while blur, noise, compression artifacts, structural defects, and illegible large text are penalized.Source: Evaluation Dimensions · p. 60, Box 10: Image Quality Assessment Expert · p. 60

• The experiments ran on a distributed setup of four identical machines.Source: E Compute Resources · p. 61
• Each machine provided 8 NVIDIA H800 GPUs and 1000 GiB of system memory.Source: E Compute Resources · p. 61
• The main results are reproducible without additional compute beyond the reported experiments, excluding preliminary runs.Source: E Compute Resources · p. 61