PaperlayerView exact revision on arXiv

Source-linked AI summary

EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing

Keming Wu, Sicong Jiang, Max Ku, Ping Nie, Minghao Liu, Wenhu Chen

arXiv:2509.26346v2cs.CVcs.AIcs.CL

TL;DR

Open-source instruction-guided image editing lacks a reliable human-aligned reward model for scaling high-quality training data. The paper introduces expert-curated preference data, trains EDITREWARD, and evaluates it with new and established benchmarks. EDITREWARD achieves state-of-the-art alignment results and improves a downstream editor when used to select training data.

  • Problem

    Open-source image editing lacks reliable reward signals for scaling high-quality datasets, while existing rewards show weak human-preference alignment.

  • Method

    The paper constructs EDITREWARD-DATA with over 200K expert-annotated preference pairs, trains EDITREWARD, and introduces EDITREWARD-BENCH for multi-way preference evaluation.

  • Results

    EDITREWARD achieves state-of-the-art results across image-editing benchmarks, including 65.72% accuracy on GenAI-Bench and 63.62% on AURORA-Bench.

  • Takeaways & Limitations

    EDITREWARD enables selecting a high-quality subset from noisy image-editing data that improves Step1X-Edit training over the full set.

  • Takeaways & Limitations

    The dataset was constructed from public, nonsensitive benchmarks and filtered to remove personally identifiable information, but the resulting editing capabilities retain dual-use risks.

Abstract

from arXiv · show

Recently, we have witnessed great progress in image editing with natural language instructions. Several closed-source models like GPT-Image-1, Seedream, and Google-Nano-Banana have shown highly promising progress. However, the open-source models are still lagging. The main bottleneck is the lack of a reliable reward model to scale up high-quality synthetic training data. To address this critical bottleneck, we built EditReward, trained with our new large-scale human preference dataset, meticulously annotated by trained experts following a rigorous protocol containing over 200K preference pairs. EditReward demonstrates superior alignment with human preferences in instruction-guided image editing tasks. Experiments show that EditReward achieves state-of-the-art human correlation on established benchmarks such as GenAI-Bench, AURORA-Bench, ImagenHub, and our new EditReward-Bench, outperforming a wide range of VLM-as-judge models. Furthermore, we use EditReward to select a high-quality subset from the existing noisy ShareGPT-4o-Image dataset. We train Step1X-Edit on the selected subset, which shows significant improvement over training on the full set. This demonstrates EditReward's ability to serve as a reward model to scale up high-quality training data for image editing. Furthermore, its strong alignment suggests potential for advanced applications like reinforcement learning-based post-training and test-time scaling of image editing models. EditReward with its training dataset will be released to help the community build more high-quality image editing training datasets.

1 INTRODUCTION

Instruction-guided image editing has advanced rapidly, but open-source progress is constrained by reward models that poorly align with human preferences. EditReward addresses this gap with expert-curated preference data, a specialized reward model, and a challenging benchmark, achieving strong benchmark performance and improving downstream model training.

  • Motivation: Existing perceptual, feature-based, and general-purpose VLM rewards show weak alignment with human preferences for image editing.Perceptual scores miss instruction semantics, feature scores miss editing semantics, and VLM judges are not optimized for this task.
  • Contributions: EDITREWARD-DATA contains over 200K manually annotated preference pairs from diverse edits generated by seven state-of-the-art models across twelve sources.Annotations were curated by trained annotators using a rigorous, standardized protocol to reduce label noise.
  • Contributions: EDITREWARD is trained as a human-aligned reward model, while EDITREWARD-BENCH evaluates models using more difficult multi-way preference prediction.The benchmark is built from the paper’s high-quality annotations and is intended to assess future reward models rigorously.
  • Results: EDITREWARD reaches 65.72 on GenAI-Bench and 63.62 on AURORA-Bench, outperforming GPT-5 at 59.61 and OpenAI-GPT-4o at 50.81, respectively.The reported results establish state-of-the-art performance across several image-editing reward benchmarks.
  • Results: Selecting the top 20K ShareGPT-4o-Image examples with EDITREWARD significantly improves Step1X-Edit over training on the full dataset.This experiment tests EDITREWARD as a selector for filtering noisy synthetic training data.

2 EDITREWARD-DATA

EDITREWARD-DATA combines diverse candidate edits with expert multi-dimensional preference annotations, while the framework trains and evaluates EDITREWARD using dedicated ranking and benchmark pipelines. The annotation measurements support the reliability of the scoring process, and EDITREWARD-BENCH strengthens evaluation through multi-way comparisons.

  • EDITREWARD-DATA Construction: EDITREWARD-DATA aggregates 9,557 instruction–image pairs from six established editing benchmarks and an internal set.The sources include GEdit-Bench, ImgEdit-Bench, MagicBrush, AnyEdit, EmuEdit, and an internal set.
  • EDITREWARD-DATA Construction: Figure 1 separates the framework into a data pipeline for candidate generation and annotation and a model pipeline for reward-model training and inference.The model pipeline optimizes EDITREWARD on EDITREWARD-DATA with the Multi-Dimensional Uncertainty-Aware Ranking Loss.
  • Annotation Reliability: Krippendorff’s Alpha is used for the ordinal 4-point Likert annotations, yielding α=0.668 for Instruction Following and α=0.597 for Visual Quality.The lower Visual Quality agreement indicates greater subjectivity in that dimension than in Instruction Following.
  • EDITREWARD-BENCH Construction: EDITREWARD-BENCH contains 500 expert-annotated groups and evaluates ternary or quaternary tuples by requiring all pairwise relations within each tuple to be correct.Its challenging cases target small score differences, making the benchmark more discriminative than traditional pairwise accuracy.

3 EDITREWARD

EDITREWARD combines a VLM backbone with uncertainty-aware, multi-dimensional preference modeling for instruction-guided image editing. It uses separate Instruction Following and Visual Quality signals, aggregates them for ranking, and exploits informative ties during training.

  • 3.1 ARCHITECTURE: EDITREWARD scores each edited image by applying a multimodal backbone and reward head to the source image, prompt, and edited image.The backbone produces a latent representation, and the MLP head projects it to a scalar score.
  • 3.2 MULTI-DIMENSIONAL UNCERTAINTY-AWARE RANKING: The reward head predicts separate Gaussian score distributions for Instruction Following and Visual Quality rather than one holistic uncertainty distribution.This multi-task formulation models the two evaluation dimensions independently.
  • 3.2 MULTI-DIMENSIONAL UNCERTAINTY-AWARE RANKING: The model aggregates dimensional means using pessimistic minimum, balanced average, or direct summation before computing preference probabilities and ranking loss.The ranking objective minimizes the negative log-likelihood of the ground-truth preference.
  • 3.3 DISENTANGLING TIES VIA DIMENSIONAL PREFERENCE: EDITREWARD decomposes qualifying overall ties into two oppositely labeled preference samples when the images have complementary dimensional strengths.This doubles the utility of tie annotations and is reported to produce smoother training and more consistent validation gains.

4 EXPERIMENTS

Experiments evaluate EDITREWARD on three established benchmarks, EDITREWARD-BENCH, ablations, backbone scaling, data curation, and OOD tasks. The model achieves leading human-alignment results, improves Step1X-Edit when filtering training data, and remains competitive outside its training pool.

  • 4.3 EXPERIMENTAL RESULTS: ALIGNMENT WITH HUMANS: 65.72% on GenAI-Bench and 63.62% on AURORA-Bench establish EDITREWARD as a leading reward model, surpassing GPT-5’s 59.61 on GenAI-Bench.Its Qwen2.5-VL-7B variant also achieves a second-best ImagenHub Spearman correlation of 36.18.
  • 4.2 BENCHMARKS AND BASELINES: 38.42% on EDITREWARD-BENCH is the highest reported score, exceeding Gemini-2.5-Flash’s 38.02 and GPT-5’s 37.81.The benchmark evaluates multi-way preference ranking with held-out preference labels.
  • 4.6 ABLATION STUDIES: 23 points of GenAI-Bench improvement, from 40.48% to 63.97%, result when the training framework is applied to base Qwen2.5-VL-7B.Performance also improves with larger and stronger backbones, including MiMo-VL-7B.
  • 4.4 APPLICATION: EDITREWARD AS A REWARD: 7.086 on GEdit-Bench-EN G O is achieved by Step1X-Edit trained on the filtered Top 20K subset, versus 6.444 for the original model and 6.780 for the full noisy 46K set.The results indicate a trade-off between data quality and subset volume, with Top 20K reported as the best balance.
  • 4.5 OUT-OF-DISTRIBUTION GENERALIZATION ANALYSIS: EDITREWARD performs comparably to GPT-4o on Text-in-Image and Style Transfer OOD tasks while offering an open-source, cost-effective, faster-inference alternative.The OOD set contains 253 Text pairs and 185 Style pairs sourced from Open Images.
  • 4.6 ABLATION STUDIES: 63.97 on GenAI-Bench favors pair-wise uncertainty loss over point-wise regression at 49.62, multiple heads over shared heads at 60.17, and balanced mean aggregation across benchmarks.The final configuration uses pair-wise modeling, multiple heads, and mean aggregation.

5 RELATED WORKS

Instruction-guided image editing evolved from trajectory-based and dual-prompt diffusion methods toward more user-friendly single-instruction editing. Evaluation likewise progressed from perceptual and CLIP-based metrics to VLM-based judges, though semantic and human alignment remain limited.

  • Evolution of Instruction-Guided Image Editing: Early image editing methods used diffusion models with dual-prompt formulations, cross-attention manipulation, or inversion.The field later shifted toward single-instruction editing intended to be more user-friendly.
  • Evaluating Instruction-Guided Image Editing: LPIPS requires reference images and misses semantic alignment, while CLIP-based metrics show limited correlation with human judgment.VLMs subsequently enabled zero-shot evaluation, including proprietary and general-purpose judge models.

6 CONCLUSION

The paper addresses the lack of reliable human-aligned rewards for open-source instruction-guided image editing through a dataset, reward model, and benchmark. EDITREWARD both improves human-judgment correlation and supports downstream data curation gains.

  • The paper targets the lack of a reliable, human-aligned reward model for scaling high-quality instruction-guided image-editing data.
  • EDITREWARD-DATA contains 200K preference examples curated with rigorous expert annotation to reduce noise and bias in existing resources.
  • EDITREWARD is a dedicated image-editing reward model trained on the high-fidelity preference data.
  • EDITREWARD-BENCH introduces challenging multi-way preference tasks for more robust reward-model evaluation.
  • 7.1 vs. 6.7 overall, Step1X-Edit trained on a 20K EDITREWARD-filtered subset outperformed training on the full 46K noisy dataset.

ETHICS STATEMENT

The authors recognize that improved image-editing systems can enable creative expression but also lower barriers to deceptive or harmful content. They describe dataset filtering, constructive alignment, and licensing measures intended to mitigate misuse.

  • Advanced instruction-guided image editing has dual-use risks, including deepfakes, misinformation, and fraudulent documents.
  • The EDITREWARD-DATA dataset uses publicly available nonsensitive benchmarks and automated and manual filtering to remove PII and sensitive content.
  • EDITREWARD is trained to favor constructive, high-quality edits under a multidimensional rubric rather than harmful or malicious instructions.
  • Generated data and model outputs are released under CC-BY-NC-SA 4.0, prohibiting commercial use as a misuse mitigation.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility through documented training resources, baseline procedures, proprietary-model evaluation timing, and archived outputs. These details help preserve consistency despite evolving API-based models.

  • All reward models were trained on 8 NVIDIA A800 GPUs.
  • Baseline evaluations used official public codebases and their recommended configurations.
  • Proprietary-model APIs were accessed between April and June 2025, with specific outputs archived for consistency.

A.1 USE OF LLM

LLMs were used only for minor manuscript grammar and style refinement, while the authors made the major scientific contributions and retain responsibility for the work.

  • LLMs were used exclusively for minor grammar correction and stylistic refinement of the manuscript.
  • The authors state that LLMs played an auxiliary role and that they made all major scientific contributions.
  • The authors bear full responsibility for the content of the work.

A.2 DETAILS OF EDITREWARD-DATA CONSTRUCTION

EDITREWARD-DATA combines diverse, human-vetted editing instructions with candidates from multiple state-of-the-art models and expert ratings on instruction following and visual quality. Quality control and agreement analysis support the reliability of this multidimensional supervision.

  • Source Data Collection: 9,557 instruction-image pairs were collected from six established, human-vetted sources to provide broad task and style coverage.The sources include GEdit-Bench, ImgEdit-Bench, MagicBrush, AnyEdit, EmuEdit, and an internal set.
  • Candidate Generation: 12 candidate images were generated per source pair using six state-of-the-art editing models, multiple seeds, and model diversity.Seven candidates were then randomly sampled from each pool for human evaluation.
  • Multi-Dimensional Annotation: Annotators independently scored Instruction Following and Visual Quality on separate 4-point Likert scales.Instruction Following measures semantic accuracy, completeness, and avoidance of unprompted changes; Visual Quality measures plausibility, artifact absence, and aesthetics.
  • Quality Control: Annotation reliability was supported by pilot refinement, annotator training and calibration, and continuous random sampling with cross-checking.The protocol was designed to maintain consistency during formal labeling.
  • Quality Control: 0.668 for Instruction Following and 0.597 for Visual Quality were the Krippendorff’s Alpha agreement scores, indicating greater subjectivity for visual quality.The difference supports using a multidimensional rubric and multi-head reward model rather than one holistic score.

A.3 DETAILS OF EDITREWARD-BENCH CONSTRUCTION

EDITREWARD-BENCH evaluates fine-grained reward-model ranking with expert-labeled, difficult candidate groups and multi-way preference comparisons. EDITREWARD performs strongly across benchmarks, while curated-data training improves Step1X-Edit and failure analysis identifies specific judgment biases.

  • Benchmark Construction: 500 high-quality groups across seven editing categories were labeled by three independent expert groups on instruction fidelity and visual quality.Groups with small differences in average human scores were prioritized to increase difficulty and discrimination.
  • Multi-Way Preference Evaluation: Multi-way tuples require correct prediction of every constituent pairwise preference, providing a stricter test of ranking consistency than pairwise accuracy.The benchmark includes ternary and quaternary tuples evaluated with an all-or-nothing criterion.
  • Reward-Modeling Effectiveness: +23.5 points on GenAI-Bench resulted when Qwen2.5-VL-7B-Inst improved from 40.48% to 63.97% after EDITREWARD training.This comparison quantifies the effect of the reward-model training framework beyond the backbone model.
  • Data Curation Results: 7.086 on GEdit-Bench-EN Overall was obtained by Step1X-Edit trained on an EDITREWARD-curated subset, versus 6.444 for the original and 6.780 for full noisy-data training.The curated-subset result was also reported as comparable to or better than Doubao at 6.983.
  • Failure Modes: EDITREWARD sometimes favors vividness and may reward unnecessary background alterations despite explicit exclusivity training.These failure modes concern color or brightness bias and global consistency.
Loading 2509.26346v2…