PaperlayerView exact revision on arXiv

Source-linked AI summary

EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling

Xin Luo, Jiahao Wang, Chenyuan Wu, Shitao Xiao, Xiyan Jiang, Defu Lian, Jiajun Zhang, Dong Liu, Zheng liu

arXiv:2509.23909v3cs.CV

TL;DR

Online RL for image editing lacks a reliable, efficient, high-fidelity reward signal. The paper introduces EditReward-Bench and specialized EditScore models, then shows that EditScore supports robust RL training and substantial, consistent gains over a base model.

  • Problem

    Online RL for high-resolution image editing is hindered by the absence of a suitable reward function that reliably and efficiently scores edit quality.

  • Method

    The paper builds EditReward-Bench and trains 7B–72B EditScore reward models using multidimensional editing evaluation and generative inference-time ensembling.

  • Results

    EditScore establishes state-of-the-art open-source reward-model performance, surpasses leading proprietary VLMs, and enables substantial and consistent performance uplift through RL.

  • Takeaways & Limitations

    A specialized, high-fidelity reward model provides a practical foundation for Best-of-N selection and online RL in image editing.

  • Takeaways & Limitations

    Performance drops on subjective Portrait Beautification and complex Hybrid Edit tasks, although self-ensembling mitigates these weaknesses.

Abstract

from arXiv · show

Instruction-guided image editing has achieved remarkable progress, yet current models still face challenges with complex instructions and often require multiple samples to produce a desired result. Reinforcement Learning (RL) offers a promising solution, but its adoption in image editing has been severely hindered by the lack of a high-fidelity, efficient reward signal. In this work, we present a comprehensive methodology to overcome this barrier, centered on the development of a state-of-the-art, specialized reward model. We first introduce EditReward-Bench, a comprehensive benchmark to systematically evaluate reward models on editing quality. Building on this benchmark, we develop EditScore, a series of reward models (7B-72B) for evaluating the quality of instruction-guided image editing. Through meticulous data curation and filtering, EditScore effectively matches the performance of learning proprietary VLMs. Furthermore, coupled with an effective self-ensemble strategy tailored for the generative nature of EditScore, our largest variant even surpasses GPT-5 in the benchmark. We then demonstrate that a high-fidelity reward model is the key to unlocking online RL for image editing. Our experiments show that, while even the largest open-source VLMs fail to provide an effective learning signal, EditScore enables efficient and robust policy optimization. Applying our framework to a strong base model, OmniGen2, results in a final model that shows a substantial and consistent performance uplift. Overall, this work provides the first systematic path from benchmarking to reward modeling to RL training in image editing, showing that a high-fidelity, domain-specialized reward model is the key to unlocking the full potential of RL in this domain.

1 INTRODUCTION

The paper identifies the lack of a reliable, efficient, scalable reward signal as the central obstacle to online RL for high-resolution image editing. It addresses this gap with EditReward-Bench, EditScore, and experiments showing utility for selection and RL training.

  • Online RL for high-resolution image editing remains largely underexplored and a formidable challenge.
  • A suitable reward function must reliably and efficiently score edit quality given an instruction, while general-purpose VLMs are expensive to query at scale.
  • EditReward-Bench evaluates editing reward models across 13 subtasks and three dimensions—prompt following, consistency, and overall quality—with expert annotations and agreement checks.
  • EditScore provides 7B–72B specialized reward models whose data curation, filtering, and inference-time scaling establish state-of-the-art open-source performance, surpassing leading proprietary VLMs.
  • Best-of-N selection improves diverse editing models, while EditScore-based RL produces substantial and consistent performance uplift over the base model.
  • The work analyzes online RL success factors, including reward-signal accuracy and variance.

2 RELATED WORK

Prior work has emphasized reward modeling and evaluation for image generation, while reward modeling for image editing remains comparatively underexplored. Existing editing benchmarks and reward models have limited task or model coverage, motivating a broader public benchmark.

  • Image reward models have mainly targeted text-to-image generation, with approaches based on human-preference models and VLM-generated rewards.
  • Image-editing reward research remains relatively underexplored compared with text-to-image reward modeling.
  • Existing editing reward resources have narrow task and model coverage, are not open-source, and struggle to support online RL for current models.
  • EditReward-Bench addresses these gaps with 13 editing tasks and support for state-of-the-art editing models.
  • Recent editing models have expanded capabilities in stylization, hybrid editing, and text modification, increasing the need for accurate edit-reward evaluation.

3 EDITREWARD-BENCH

EditReward-Bench is designed as a broad, reproducible benchmark for image-editing reward models, combining diverse tasks and generators with multidimensional expert evaluation. Its annotation pipeline converts consensus tier rankings into high-fidelity pairwise preferences.

  • Overview: EditReward-Bench covers 13 diverse editing tasks and 11 heterogeneous editing models, spanning open-source baselines and proprietary state-of-the-art systems.
  • Benchmark design: Its task categories span increasing complexity, including subject, appearance, and other subtasks curated from established editing datasets.
  • Overview: The benchmark evaluates prompt following, consistency, and overall quality to make editing judgments multidimensional and interpretable.
  • Annotation pipeline: Two experts discuss each sample in real time, and the discussion protocol improves Consistency inter-annotator convergence by 12.12% at a 100% agreement threshold.
  • Annotation pipeline: Annotators rank five outputs in quality tiers, allowing ties among similar outputs before consensus determines the final ranking.
  • Annotation pipeline: Tiered rankings are decomposed into cross-tier pairwise preferences while within-tier comparisons are excluded as equal-quality judgments.

4 METHOD

EditScore formulates reward modeling as conditional textual generation and scores edits along semantic consistency and perceptual quality. It uses curated training data, VIEScore-based scoring, and inference-time aggregation to produce a granular reward signal.

  • Reward modeling: EditScore takes an instruction, input image, and output image as input and generates reasoning plus a scalar score as a specialized evaluator.
  • Reward modeling: Semantic Consistency measures instruction adherence and preservation of unmentioned regions, while Perceptual Quality measures photorealism and artifact absence.
  • Reward modeling: The final reward combines SC and PQ through their geometric mean, yielding a balanced and granular signal.
  • Inference-time ensembling: For a given triplet, inference-time ensembling generates K independent reasonings and aggregates their scalar scores with an arithmetic mean.
  • Inference-time ensembling: Aggregating scores from diverse reasoning perspectives is intended to improve evaluation accuracy and reward stability.
  • Data construction: The data pipeline selects images and instructions, generates candidate edits from five editing models, and filters samples using maximum scores and group standard deviation.

5 REWARD MODEL PERFORMANCE ON EDITREWARD-BENCH

EditScore establishes strong reward-model performance on EditReward-Bench, with specialized models outperforming open-source VLMs and inference-time self-ensembling improving efficiency and accuracy.

  • Proprietary VLMs achieve pairwise accuracies in the 0.7-0.75 range, while open-source Qwen2.5-VL-72B remains below 0.612 overall accuracy and performs worse than chance on Consistency.
  • EditScore-7B surpasses the 10x larger Qwen2.5-VL-72B, while EditScore-72B matches GPT-4.1 and establishes the state of the art on EditReward-Bench.
  • Avg@4 self-ensembling further boosts EditScore performance across model sizes by averaging scores over four forward passes.
  • Inference-time compute scaling yields greater marginal gains than parameter scaling, while latency grows sublinearly with ensemble size because attention shares the prefill stage.
  • Reasoning before scoring improves EditScore-7B pairwise accuracy by +0.038, from 0.621 to 0.659, while overly large score ranges reduce performance.

6 APPLICATION OF EDITSCORE IN IMAGE EDITING

EditScore is evaluated as a selector and online-RL reward across editing models, showing stable policy improvements, cross-backbone generality, and compatibility with stronger optimization.

  • Best-of-N selection: Best-of-N selection uses EditScore to choose among multiple outputs from OmniGen2, Flux.1-Kontext-dev, and Qwen-Image-Edit.
  • Stable RL: EditScore-7B stabilizes OmniGen2 RL training and surpasses GPT-4.1 on OOD EmuEdit CLIP-I, achieving 0.900 versus 0.884.
  • Stable RL: Self-ensembling with Avg@4 improves reward accuracy, accelerates convergence, and produces superior final policy performance compared with single-pass evaluation.
  • Generality across editors: EditScore-based RL improves Flux-Kontext-dev across metrics, including +0.57 in SC and +0.72 in Overall, indicating generality across editor backbones.
  • RL algorithm compatibility: Combining EditScore with TempFlow-GRPO achieves an Overall score of 7.21, demonstrating compatibility with an advanced RL algorithm.
  • Annotation sources and variance: Reward variance can matter more than annotator accuracy: GPT-4.1-labeled EditScore has 2.868 versus 2.533 score standard deviation and yields stronger post-RL policies.

7 CONCLUSION

The work addresses the lack of reliable reward signals for image-editing RL by introducing EditReward-Bench and EditScore, then validates a discussion-based annotation protocol for high-fidelity ground truth.

  • 7 CONCLUSION: EditReward-Bench and EditScore provide a benchmark and specialized reward-model family for future RL-based image-editing research.The benchmark evaluates reward models, while EditScore supplies a high-fidelity signal for Best-of-N selection and online RL.
  • 7 CONCLUSION: Two annotators discuss visual artifacts and resolve discrepancies by consensus rather than labeling samples independently.The protocol is intended to reduce annotation noise arising from subtle visual inspections.
  • 7 CONCLUSION: Convergence rate measures whether annotation units assign identical rankings to the same input.The metric applies to both individual experts and discussion groups.
  • 7 CONCLUSION: Agreement with majority vote measures how closely an annotation unit matches the collective expert consensus.The metric is used to assess whether a single unit can reliably represent the broader annotation pool.

B.3 RESULTS AND ANALYSIS

The two-annotator discussion protocol improves annotation reliability, especially for consistency judgments, while EditScore self-ensembling is designed to remain practical for inference and RL training.

  • Annotation reliability: 12.12%: strict agreement on Consistency increased from 82.88% to 95.00% with two-annotator discussion.The authors attribute the improvement to real-time cross-checking of subtle background distortions and structural artifacts.
  • Annotation reliability: Prompt Following and Consistency achieve higher strict agreement than Overall Quality, at approximately 93–95% versus 90%.The authors describe Overall Quality as more subjective and more prone to rater bias.
  • Annotation reliability: A single annotator pair aligns with the collective majority vote at over 97% across dimensions, with Consistency reaching 99.55%.This supports annotating each sample once with a single expert pair.
  • Inference efficiency: Self-ensembling does not increase memory consumption because all K queries share the same underlying model weights.Memory is determined by model parameters rather than ensemble size K.
  • Inference efficiency: With K = 4, EditScore’s wall-clock latency grows sublinearly because the prefill stage is shared across stochastic decoding paths.The configuration reduces throughput but does not multiply latency by four.
  • Inference efficiency: 576 reward queries complete in approximately 18 seconds on an 8×H100 cluster, representing about 20% of total training-iteration time.The measurement comes from the RL fine-tuning pipeline using EditScore-7B-avg4.

C.3 COST-NORMALIZED PERFORMANCE TRADE-OFF

Under a fixed 4-GPU budget, self-ensembling smaller EditScore models offers a favorable accuracy-throughput trade-off, while paired bootstrap tests assess whether reported improvements are statistically reliable.

  • Cost-normalized performance: 0.727 accuracy and 16.24 samples/s: EditScore-7B with K = 4 exceeds EditScore-32B with K = 1 at 0.680 accuracy and 9.48 samples/s.The comparison uses the same 4-GPU budget and benefits from hosting four parallel 7B instances versus two 32B instances.
  • Cost-normalized performance: 0.763 accuracy: EditScore-72B achieves the highest absolute accuracy but only 1.60 samples/s throughput.The 7B K = 4 setting is identified as the balance for high-volume RL applications.
  • Cost-normalized performance: Increasing K for smaller models is more cost-effective than increasing model size when abundant parallel compute is available.The self-ensemble trades parallel compute for quality improvements under the normalized hardware budget.
  • Statistical validation: Paired bootstrap testing uses 10,000 resampling rounds to estimate performance differences between baseline and proposed models.The analysis reports confidence intervals and one-sided p-values for each comparison.
  • Statistical validation: A 95% confidence interval strictly above zero indicates a significant improvement, while a one-sided p-value below 0.05 indicates statistical significance.These criteria are applied to the bootstrap distribution of performance differences.

D.2 SIGNIFICANCE OF EDITSCORE PERFORMANCE

EditScore’s gains over Qwen2.5-VL are statistically significant across model scales and evaluation dimensions, and its reward signal also produces significant downstream RL improvements on GEdit-Bench.

  • EditScore benchmark significance: p ≈ 0: EditScore’s improvements over Qwen2.5-VL are highly significant for the 7B and 32B models.The comparisons are conducted across EditReward-Bench metrics.
  • EditScore benchmark significance: p < 0.05: EditScore remains significant at 72B across all dimensions, with Consistency and Overall reaching p ≤ 0.0005.The baseline is stronger at the 72B scale, but the reported differences remain statistically significant.
  • Downstream RL significance: p ≈ 0.0002: EditScore reward training improves Semantic Consistency, Perceptual Quality, and Overall score over the baseline reward on GEdit-Bench.The comparison is between Qwen2.5-VL-72B-reward training and EditScore-7B-avg4-reward training.
  • RL hyperparameter analysis: Increasing inference steps from T ∈ {12, 16, 20} consistently improves Semantic Consistency, Perceptual Quality, and Overall metrics.The authors link larger step counts to higher-fidelity samples and more reliable reward signals during training.

E.2 IMPACT OF GROUP SIZE

The section examines how group size trades off advantage-estimation stability against prompt diversity under a fixed computational budget, identifying 12 as the best configuration.

  • E.2 IMPACT OF GROUP SIZE: Group size 8 exposes the model to 36 unique prompts but increases advantage-estimation variance, causing noisy and suboptimal policy updates.
  • E.2 IMPACT OF GROUP SIZE: Group size 16 stabilizes advantage estimation but reduces unique prompts to 18, hindering generalization and lowering the O score to 6.90.
  • E.2 IMPACT OF GROUP SIZE: Group size 12 yields the best performance by balancing stable advantage estimation with adequate prompt diversity.The comparison keeps total tokens per update fixed, so larger groups reduce the number of unique prompts.
  • G GRPO ON SDE FLOW MATCHING: The policy uses GRPO over trajectories sampled from an SDE-based stochastic policy, assigning terminal EditScore rewards to generated images.The stochastic policy is implemented through Euler-Maruyama transitions and optimized using GRPO.

I QUALITATIVE RESULTS

The paper reports qualitative results after reinforcement learning and describes the annotation interface used to collect multidimensional, tiered preference data for EditRewardBench.

  • I QUALITATIVE RESULTS: Qualitative examples across several image-editing tasks validate improved editing outcomes for OmniGen2 after reinforcement learning.
  • Annotation Interface: Annotators independently rank Instruction Following, Consistency, and Overall Quality to prevent visually pleasing but semantically incorrect edits from being favored.
  • Annotation Interface: Tiered ranking allows annotators to group outputs of equal perceived quality rather than forcing a strict linear order.Tied outputs are represented with a pipe separator in the ranking input.
  • Annotation Interface: Progress tracking and quality-control features support a smooth and reliable expert-annotation workflow.

L LIMITATIONS

EditScore remains strong on objective editing tasks but faces limitations on subjective beautification and complex compositional edits, where multiple failure modes remain.

  • L LIMITATIONS: EditScore performs strongly on objective tasks but is weaker on Portrait Beautification and Hybrid Edit subtasks.EditScore-7B reaches 0.284 accuracy on Beaut and 0.519 on Hyb, while Avg@4 raises them to 0.569 and 0.571.
  • L LIMITATIONS: In portrait beautification, EditScore misses subtle improvements such as skin smoothing and assigns a Semantic Consistency score of 0/25.
  • L LIMITATIONS: In a complex hybrid edit, EditScore recognizes radiator removal but fails to penalize an incorrect cat-color change, assigning 22/25.
  • L LIMITATIONS: The authors attribute these failures to limited pixel-level discernibility, disagreement in subjective data, and the scarcity of complex compositional samples.They identify more granular multi-step data and stronger base models as future directions.
  • Evaluation Prompts: The evaluation prompts separately score editing success and overediting on a 0–25 scale for two-image comparisons.
  • Evaluation Prompts: Perceptual Quality evaluates naturalness and artifacts through a self-contained single-image rubric.
Loading 2509.23909v3…