PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Verifier-Based Reinforcement Learning in Image Editing

Hanzhong Guo, Jie Wu, Jie Liu, Yu Gao, Zilyu Ye, Linxiao Yuan, Xionghui Wang, Yizhou Yu, Weilin Huang

arXiv:2604.27505v2cs.CV

TL;DR

Image editing lacks a robust reward model that can evaluate diverse instruction requirements beyond holistic scoring. Edit-R1 builds a verifier-based reasoning reward model and uses it in reinforcement learning, surpassing powerful models in evaluation and improving state-of-the-art editors.

  • Problem

    Image editing lacks a robust reward model that separately evaluates instruction fidelity, preservation of unedited regions, and overall quality rather than producing a single holistic score.

  • Method

    Edit-R1 trains a principle-decomposing chain-of-thought reasoning reward model with cold-start SFT and GCPO, then uses it as a verifier in GRPO.

  • Results

    Edit-R1’s 7B RL-RRM surpasses other reward models on EditRewardBench and delivers substantial gains to state-of-the-art editors including FLUX.1-kontext and Qwen-Image-Edit.

  • Takeaways & Limitations

    A verifier-based reasoning reward model can improve evaluation accuracy and instruction-following capabilities in downstream image editing models.

  • Takeaways & Limitations

    After cold-start SFT, the reward model can hallucinate or inaccurately judge the magnitude of edits.

Abstract

from arXiv · show

While Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm for text-to-image generation, its application to image editing remains largely unexplored. A key bottleneck is the lack of a robust general reward model for all editing tasks. Existing edit reward models usually give overall scores without detailed checks, ignoring different instruction requirements and causing biased rewards. To address this, we argue that the key is to move from a simple scorer to a reasoning verifier. We introduce Edit-R1, a framework that builds a chain-of-thought (CoT) verifier-based reasoning reward model (RRM) and then leverages it for downstream image editing. The Edit-RRM breaks instructions into distinct principles, evaluates the edited image against each principle, and aggregates these checks into an interpretable, fine-grained reward. To build such an RRM, we first apply supervised fine-tuning (SFT) as a ``cold-start'' to generate CoT reward trajectories. Then, we introduce Group Contrastive Preference Optimization (GCPO), a reinforcement learning algorithm that leverages human pairwise preference data to reinforce our pointwise RRM. After building the RRM, we use GRPO to train editing models with this non-differentiable yet powerful reward model. Extensive experiments demonstrate that our Edit-RRM surpasses powerful VLMs such as Seed-1.5-VL and Seed-1.6-VL as an editing-specific reward model, and we observe a clear scaling trend, with performance consistently improving from 3B to 7B parameters. Moreover, Edit-R1 delivers gains to editing models like FLUX.1-kontext, highlighting its effectiveness in enhancing image editing.

1 Introduction

Edit-R1 addresses biased, coarse image-editing rewards by replacing holistic scoring with a verifier-based reasoning reward model that decomposes instructions into principles and evaluates each one. It trains this RRM with SFT and GCPO, then uses it in GRPO to improve downstream editing models.

  • Motivation: Existing editing reward models often use a single holistic score, overlooking instruction fidelity, preservation of unedited regions, and overall quality.This can produce biased or hallucinated feedback.
  • Results: 82.22% accuracy: the final 7B model trained with SFT and GCPO surpasses the Seed-VLM baseline on the reward benchmark.The figure states that each training component contributes to the performance gain.
  • Verifier-based reward model: Edit-R1’s RRM decomposes editing instructions into verifiable principles, generates chain-of-thought analysis, and aggregates principle-level checks into structured feedback.The framework shifts reward modeling from direct scoring to explicit verification.
  • Downstream application: GRPO uses the trained non-differentiable RRM as a verifier to improve downstream image-editing models, including FLUX.1-kontext and Qwen-Image-Edit.The RRM’s discrete multi-step reasoning trace makes it incompatible with REFL-style differentiable methods, motivating the GRPO-based approach.
  • RRM training: GCPO aligns the pointwise reasoning RRM with pairwise human preferences by contrasting groups of winner and loser reasoning trajectories generated by the RRM.The algorithm is designed to refine the verifier’s reasoning capabilities and alignment with human judgments.

2 Related Works

Related work develops reward models from LLMs through regression-based and generative-based architectures, while image-editing alignment uses methods including DPO, REFL, and GRPO. However, visual-generation rewards remain challenging, motivating reasoning-based, principle-driven feedback for image editing.

  • Reward Models: LLM-based reward models primarily use regression-based or generative-based architectures, with generative methods leveraging models’ own assessment abilities.Regression-based methods add a scoring head, whereas generative-based methods use generation for evaluation.
  • Image-Editing Alignment: Image-editing preference alignment has used DPO and REFL, but DPO restricts policy exploration and REFL requires a differentiable reward model while remaining prone to reward hacking.DreamFuse adopts DPO, whereas other methods use REFL for preference alignment.
  • Verifiable Rewards: GRPO with verifiable rewards has been explored for visual generation because such rewards can support robust training and mitigate reward hacking.The passage connects this direction to the success of DeepSeek-R1 and recent visual-generation work.
  • Reasoning-Based Rewards: Defining verifiable rewards for visual generation remains challenging, so the paper extends visual GRPO with a reasoning-based reward model that provides structural, principle-driven feedback for image editing.This approach addresses the difficulty of specifying rewards for visual generation.

3 Method

Edit-R1 trains a verifier-based Reasoning Reward Model in two stages, first cold-starting it with CoT supervision and then aligning it with human preferences through GCPO. The resulting verifier decomposes edits into principles, produces fine-grained scores, and supplies a reward for GRPO-based editing-model optimization.

  • Cold-Start SFT: Edit-R1 cold-starts its RRM by constructing editing-specific supervised data from curated benchmark samples and externally verified reasoning traces.The dataset begins with 200K benchmark samples and selects the highest-accuracy Think + Score candidate for each generated quadruple.
  • Principle Decomposition: The method decomposes each editing instruction into Keep, Follow, and Quality principles that distinguish preserved content, required modifications, and edit quality.These principles form the set P used to structure sample-wise verification.
  • CoT Data Generation: It generates approximately 2 million quadruples by combining edited candidates from Flux-Kontext, Bagel, and SeedEdit3.0 with reference images, instructions, and principle sets.VLM pools then produce multiple CoT traces and weighted aggregates of principle-wise verification outcomes.
  • GCPO Preference Optimization: GCPO refines the fallible SFT-initialized RRM with human-annotated preference pairs, optimizing generated reasoning traces and final scores through inter-reward, intra-advantage learning.Human judgments prioritize instruction fidelity and overall image quality, while win/loss ratio rewards compare stochastic score sets and ignore ties.
  • Edit-aware GRPO: Edit-aware GRPO trains downstream editing models to optimize the RRM’s holistic reward while controlling updates with clipping and a KL-divergence penalty.The framework targets human-perceived quality and instruction fidelity captured by the verifier.

4 Experiments

Experiments show that Edit-RRM achieves strong human-preference prediction and consistently outperforms scorer baselines, with gains from principled SFT curation and GCPO. Using the resulting reward model for policy optimization improves editing quality, reward reliability, instruction adherence, and visual fidelity.

  • Reward Model Performance: 82.2% accuracy makes the 7B Edit-RRM outperform Seed-1.5-VL at 79.3% on human-preference prediction.The model uses the full two-stage training pipeline.
  • Verifier vs. Scorer: 78.2% accuracy is achieved by RL-RRM after GCPO, versus 73.3% for SFT-RRM and 65.9% for EditScore-7B on EditRewardBench.The independently constructed public benchmark indicates the improvement is not due to internal benchmark bias.
  • Data Curation and SFT: 75.4% accuracy with Think+Verify exceeds 68.9% with Think only for Qwen-7B, while VIESCORE-prompted SFT reaches 68.3%.These results support principled decomposition and rigorous filtering in the cold-start supervision pipeline.
  • GCPO and Scalability: GCPO-refined RRMs produce higher evaluation rewards and stricter evaluation while maintaining stable reward signals during editing-model optimization.RL-RRMs yield lower training rewards but higher evaluation rewards, indicating stricter and more reliable supervision.
  • Overall Performance: Optimizing FLUX.Kontext with RL-RRM raises Overall Score from 5.77 to 6.24 and Semantic Consistency from 6.27 to 6.86.The result is reported as exceeding the other compared systems in Table 3.
  • Qualitative Analysis: Models optimized with Edit-R1 show better instruction adherence and visual fidelity, especially for subject addition/removal, motion change, and localized color edits.The qualitative analysis reports successful instruction following where the baseline fails and precise target-object modification without introducing unwanted changes.

5 Conclusion

Edit-R1 enhances image editing through reinforcement learning from human feedback using a verifier-based reasoning reward model. Its cold-start SFT and GCPO training produce an RRM whose integration with GRPO improves instruction following in state-of-the-art editing models.

  • Edit-R1 is a framework designed to enhance image editing through reinforcement learning from human feedback.
  • Its verifier-based reasoning reward model is trained through a cold-start SFT phase and the GCPO algorithm.
  • The RRM achieves evaluation accuracy surpassing powerful proprietary models.
  • Integrating the RRM with GRPO substantially enhances instruction-following capabilities in state-of-the-art editing models.

A System prompt

The system prompts structure image-editing evaluation into diverse, category-balanced principles and guide a verifier-based reward model through detailed, holistic scoring. Together, they enforce explicit comparison rules, instruction priority, edit adequacy, and penalties for uncertainty, missing edits, borders, and unintended changes.

  • A.1 System Prompt for Decomposing Principles: The decomposition prompt uses few-shot expert examples to generate evaluation principles for each new image-editing task.It randomly samples 4 exemplars from a pool of 60 expert-authored examples to improve diversity and robustness.
  • A.1 System Prompt for Decomposing Principles: The prompt requires 10 evaluation points distributed across Instruction Following, Feature Preservation, and Image Quality.The requested allocation is 3–4, 3–4, and 2–3 points, respectively.
  • A.1 System Prompt for Decomposing Principles: Each generated principle is returned as a JSON-array element containing a question and category field.The new task is supplied through an edit-instruction placeholder.
  • A.2 System Prompt for Reward Model Evaluation: The reward-model prompt compares the original and edited images against decomposed principles while also requiring holistic analysis of editing performance.Its inputs include the original image, edited image, edit instruction, and evaluation points.
  • A.2 System Prompt for Reward Model Evaluation: Each evaluation point receives a binary 0 or 1 score, with uncertainty assigned 0 and significant changes required for range-based or positional edits.Relative left-right judgments use the viewer’s perspective when an object faces the camera.
  • A.2 System Prompt for Reward Model Evaluation: The final score ranges from 0 to 10 and is not determined solely by averaging point scores.The evaluator considers edit magnitude, direct usability, psychological expectations, newly revealed areas, and instruction contradictions.
  • A.2 System Prompt for Reward Model Evaluation: Near-duplicate outputs and edited images with white borders receive a score of 0.For single-edit instructions, insufficient similarity reduction indicates that the critical edit was not performed.
  • A.2 System Prompt for Reward Model Evaluation: The evaluator prioritizes the instructed edit and penalizes unreasonable changes to unmentioned subjects, objects, or backgrounds as hallucinations.Consistency is prioritized for the main subject, while removal tasks require complete object removal.

B Inference result of RRM

The RRM evaluates an edited image by reasoning through decomposed principles, assigning pointwise scores, averaging them, and producing a comprehensive final score. In the example, it gives 8/10 principles a score of 1, averages to 0.8, and assigns a final score of 7 because style and image quality succeed despite vehicle-detail changes.

  • Inference process: The RRM receives a source image, instruction, edited image, and decomposed principles, then generates detailed reasoning summarized into pointwise and comprehensive scores.Figure 5 depicts the input quadruple and final summary output, while Listing 4 provides the unabridged reasoning.
  • Pointwise evaluation: The white and black sports cars each score 0 because their appearance, shape, and details are not consistent with the original images.The white car changes from a convertible to a hardtop, while the black car’s model and styling change significantly.
  • Score aggregation: 8/10 = 0.8 is the average pointwise score across the ten evaluation principles.The scores are (1+1+1+1+1+0+0+1+1+1) /10.
  • Final assessment: The final score is 7 because the core Chinese-style transformation succeeds, image quality is good, integration is natural, and vehicle-detail changes are treated as minor issues.The reasoning considers the vehicles’ presence important but does not require preserving their details for the style-focused instruction.

C Category label in quantitative results

The quantitative results use 11 categories covering diverse image-editing operations, from background and color changes to text and tone transformations.

  • Category definitions: Categories 1–11 represent background change, color alteration, material modification, motion change, portrait beautification, style transfer, subject addition, subject removal, subject replacement, text modification, and tone transformation.The categories span visual attributes, subjects, styles, motion, text, and overall tone.

D Human Evaluation

The human evaluation tests whether GPT-based automatic metrics align with human perception by comparing RL-RRM-optimized FLUX.Kontext (7B) with the original baseline. Annotators use the Good-Same-Bad protocol, summarized by a higher-is-better GSB score.

  • Evaluation setup: The study compares FLUX.Kontext optimized by the RL-RRM (7B) with the original FLUX.Kontext baseline on identical inputs.The evaluation is designed to validate alignment between automatic GPT-based metrics and human perception.
  • Evaluation setup: Annotators classify each model output as better, comparable, or worse than the baseline.These judgments follow the Good-Same-Bad (GSB) protocol.
  • Scoring protocol: The GSB score is computed as (G − B)/(G + S + B), with higher values indicating better performance.G, S, and B correspond to good, same, and bad judgments, respectively.

E Qualitative Results for FLUX-Kontext

The section presents qualitative results for FLUX.Kontext, with examples shown in Figures 6, 7, and 8.

  • E Qualitative Results for FLUX-Kontext: Qualitative results are reported for FLUX.Kontext.The results are presented visually rather than through a quantitative metric in the supplied passage.
  • E Qualitative Results for FLUX-Kontext: Figure 6 shows qualitative results for FLUX.Kontext.
  • E Qualitative Results for FLUX-Kontext: Figures 7 and 8 also show qualitative results for FLUX.Kontext.

F Qualitative Results for Qwen-Edit

Qualitative comparisons show that Edit-R1 further improves Qwen-Edit on challenging edits, especially motion-related edits and fine-grained attribute changes. Across diverse editing categories and benchmarks, enhanced models better align outputs with user instructions than baselines.

  • Diverse editing categories: Edit-R1 produces stronger edits than FLUX.Kontext across text editing, color or material alteration, motion changes, and subject manipulation.The qualitative comparison covers subject addition and removal as forms of subject manipulation.
  • GEdit-Bench: On GEdit-Bench, FLUX.Kontext w. Edit-R1 consistently produces higher-quality edits that better align with user instructions than the FLUX.Kontext baseline.The comparison includes text modification and subject addition tasks.
  • Emu Edit Test Set: On the challenging Emu Edit Test Set, FLUX.Kontext w. Edit-R1 demonstrates strong capability on a wide variety of complex instructions.The examples are presented as qualitative results on the challenging test set.
  • Qwen-Edit: Edit-R1 further improves Qwen-Edit on challenging edits, especially motion-related edits and fine-grained attribute changes.Figure 9 compares Qwen-Edit with Qwen-Edit w. Edit-R1 on diverse instructions.
  • Verifier-guided correction: RRM-guided reinforcement learning corrects an SFT hallucination by changing the shirt to red while preserving the blue hat.The RRM penalizes the SFT output that incorrectly changes the hat color, while the RL output preserves it.

G Qualitative Analysis of RRM Judgments … 8 Score: 1

The qualitative case study shows that the RRM detects attribute leakage in SFT outputs and guides RL tuning toward instruction-faithful image edits. Its fine-grained judgments verify both the original failure and the corrected preservation of the hat.

  • G Qualitative Analysis of RRM Judgments: The RL-tuned model changes only the shirt color and preserves the hat, producing the instruction-following “winner” image.The RL-tuned output corrects the hallucination identified in the SFT output.
  • G Qualitative Analysis of RRM Judgments: The SFT model changes the shirt to red but also changes the light blue hat, producing attribute leakage or hallucination.The task requires changing only the shirt while preserving all other features.
  • 8 Score: 1: The RRM assigns a score of ‘1‘ to the SFT “loser” image on hat style preservation, correctly identifying its failure.This judgment is reported in Listing 5.
  • 8 Score: 1: The RRM assigns a score of ‘1‘ to the RL-tuned “winner” image on hat style preservation, confirming that the hallucination was resolved.Listing 8 states that the model learned to preserve the hat style correctly.
  • G Qualitative Analysis of RRM Judgments: The RRM provides precise, targeted feedback by evaluating the edited image against distinct instruction requirements rather than relying only on an overall score.The case study uses separate checks for shirt color, pose, hairstyle, clothing, background, structural quality, and scene blending.
  • 8 Score: 1: The verification checks also assess whether the generated image is free of significant structural problems and whether the red top blends naturally with the overall scene.These checks extend beyond the requested shirt-color change to broader image consistency and quality.
  • 8 Score: 0: The RL-tuned model’s “loser” output still fails on the specific evaluated point, according to Listing 7.Listing 7 reports a 29 Score: 1 while describing failure on that specific point.
Loading 2604.27505v2…