PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Verifier-Based Reinforcement Learning in Image Editing

Hanzhong Guo, Jie Wu, Jie Liu, Yu Gao, Zilyu Ye, Linxiao Yuan, Xionghui Wang, Yizhou Yu, Weilin Huang

arXiv:2604.27505v2cs.CV

TL;DR

Image editing lacks a sufficiently robust reward model, and holistic scores do not explicitly check distinct instruction requirements. Edit-R1 trains a CoT verifier that evaluates editing principles and uses it to reward downstream training; its evaluation accuracy surpasses powerful proprietary models.

  • Problem

    Image editing lacks a sufficiently robust reward model, while holistic single-score approaches provide limited evaluation of distinct editing requirements such as instruction fidelity and preservation.

    Source: §1 · pp. 1–2

  • Method

    Edit-R1 trains a CoT verifier that decomposes instructions into principles, evaluates each principle, and uses SFT, GCPO, and GRPO for reward-model and downstream editor training.

    Source: §1 · p. 2, §3 · p. 4, §3.1.1 · p. 4

  • Results

    Edit-RRM achieves evaluation accuracy that surpasses powerful proprietary models.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors report that using the RRM with GRPO substantially enhances instruction following in state-of-the-art editing models.

    Source: §5 · p. 10

    The cold-start RRM can hallucinate or misjudge the magnitude of edits, including whether an object moved far enough to satisfy an instruction.

    Source: §3.1.2 · p. 6

Abstract

from arXiv · show

While Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm for text-to-image generation, its application to image editing remains largely unexplored. A key bottleneck is the lack of a robust general reward model for all editing tasks. Existing edit reward models usually give overall scores without detailed checks, ignoring different instruction requirements and causing biased rewards. To address this, we argue that the key is to move from a simple scorer to a reasoning verifier. We introduce Edit-R1, a framework that builds a chain-of-thought (CoT) verifier-based reasoning reward model (RRM) and then leverages it for downstream image editing. The Edit-RRM breaks instructions into distinct principles, evaluates the edited image against each principle, and aggregates these checks into an interpretable, fine-grained reward. To build such an RRM, we first apply supervised fine-tuning (SFT) as a ``cold-start'' to generate CoT reward trajectories. Then, we introduce Group Contrastive Preference Optimization (GCPO), a reinforcement learning algorithm that leverages human pairwise preference data to reinforce our pointwise RRM. After building the RRM, we use GRPO to train editing models with this non-differentiable yet powerful reward model. Extensive experiments demonstrate that our Edit-RRM surpasses powerful VLMs such as Seed-1.5-VL and Seed-1.6-VL as an editing-specific reward model, and we observe a clear scaling trend, with performance consistently improving from 3B to 7B parameters. Moreover, Edit-R1 delivers gains to editing models like FLUX.1-kontext, highlighting its effectiveness in enhancing image editing.

1 Introduction

The introduction identifies holistic reward scoring as inadequate for image editing and proposes Edit-R1, which uses a reasoning verifier and tailored reinforcement-learning methods to improve reward modeling and downstream editing.

Motivation

  • Holistic VLM scores can miss instruction-specific requirements, balance editing criteria poorly, and produce biased or hallucinated feedback [20].Editing evaluation must account for instruction fidelity, preservation of unedited regions, and overall quality.Source: §1 · pp. 1–2

Results

  • 82.22% accuracy: the final 7B SFT+GCPO (RL-RRM) surpasses the Seed-VLM baseline and other reward models, including EditScore [37] on EditRewardBench [37].Figure 1 reports the 82.22% accuracy and Seed-VLM comparison; the introduction separately states that RL-RRM surpasses other reward models, including EditScore, on EditRewardBench.Source: Figure 1 · p. 2, §1 · p. 3

Downstream editing

  • Because the RRM samples a discrete reasoning trace and is non-differentiable, Edit-R1 uses it as a reward signal in a GRPO-based loop for downstream editing models.The introduction says REFL-style methods are inapplicable to this RRM and reports gains for FLUX.1-kontext and Qwen-Image-Edit.Source: §1 · pp. 2–3

RRM training

  • GCPO aligns the pointwise RRM with pairwise human preferences by contrasting groups of winner and loser reasoning trajectories generated by the RRM.The introduction identifies noisy CoT training data and the mismatch between pointwise reasoning outputs and pairwise preferences as verifier-training challenges.Source: §1 · p. 2

Verifier-based reward model

  • Edit-R1 replaces holistic scoring with an RRM that decomposes instructions into verifiable principles, checks each against the edit, and aggregates interpretable CoT feedback.The framework uses an external VLM as a quality-control judge to filter SFT data for plausible reasoning trajectories.Source: §1 · p. 2

2 Related Works

Prior work establishes LLM-based reward modeling and preference-aligned reinforcement learning for image editing, while leaving gaps in visual-specific, interpretable verification and robust verifiable rewards.

2.1 Reward model for generative models

  • LLM-based RMs use regression [34] [55] or generative [19] [26] [61] scoring and pointwise [59] [62] or pairwise [57] [61] inputs, with pairwise models lacking absolute single-response scores.This limits pairwise methods for direct quality assessment or filtering.Source: §2.1 · p. 3
  • Edit-RRM is a generative, pointwise image-edit verifier combining principle-decomposed CoT with a rationale-based SFT cold-start followed by reinforcement-learning optimization.Unlike prior principle-based CoT reward models such as DeepSeek-GRM [36], it is tailored to visual editing.Source: §2.1 · p. 3

2.2 Reinforcement Learning in Image Editing

  • DreamFuse [27] uses DPO [53], which limits policy exploration and risks suboptimal convergence, while REFL [64] used by [19] [43] can reward-hack and requires differentiable rewards.These limitations motivate reward approaches that permit exploration without requiring a differentiable reward model.Source: §2.2 · p. 3
  • Visual GRPO works [33] [66] draw on verifiable rewards, whose success in DeepSeek-R1 [21] motivates their use, but defining such rewards for visual generation remains challenging.Edit-R1 extends visual GRPO with a reasoning-based reward model that provides structural, principle-driven feedback for image editing.Source: §2.2 · pp. 3–4

3 Method

Edit-R1 trains a reasoning reward model in two stages—cold-start SFT followed by preference-based GCPO—and then uses it with GRPO to optimize downstream image-editing models.

3.1.1 Verifier-based Reasoning Reward Model with Cold-Start

  • Cold-start SFT begins with 200K benchmark samples, then generates approximately 2 million quadruples from edited candidates produced by multiple editing models.The 200K comprise 100K randomly selected edits and 100K GPT-4o-filtered hard edits involving complex or precise modifications.Source: §3.1.1 · p. 4
  • Each task is decomposed into Keep, Follow, and Quality principles, which guide pointwise verification and weighted aggregation into an overall score.VLMs produce multiple CoT-and-score candidates per quadruple; SeedVLM-1.5 rechecks their principle-level accuracy, and the highest-accuracy trace enters the SFT dataset.Source: §3.1.1 · pp. 4–5

3.1.2 Reasoning-Reinforced Reward Learning

  • After cold-start SFT, the RRM can still hallucinate or misjudge edit magnitude, motivating preference-based reinforcement learning of its reasoning and scores.Human annotators compare edited-image pairs for instruction fidelity and overall quality, producing approximately 10,000 preference pairs per context; ties are excluded.Source: §3.1.2 · p. 6
  • GCPO refines the reasoning reward model using human preference pairs, comparing multiple reasoning-score candidates across preferred and non-preferred images.It converts cross-group comparisons into win/loss rewards, computes advantages within each group, and optimizes the two groups with clipped surrogate losses without a KL term.Source: §3.1.2 · p. 6

3.2 Reinforcement Learning for Image Editing

  • Downstream GRPO uses the RRM’s holistic rewards to normalize advantages within generated image groups and update the editing policy.The clipped objective limits large policy updates, while a KL-divergence penalty regularizes the policy against a reference; the reward captures human-perceived quality and instruction fidelity.Source: §3.2 · p. 7

4 Experiments

Edit-R1’s verifier-based reward model outperforms the reported scorer and closed-source baselines, while GCPO-refined rewards improve image-editing results on FLUX.Kontext and Qwen-Image-Edit. The experiments evaluate human-preference prediction and editing quality, with evidence that principled SFT data curation and GCPO contribute to performance.

4.1 Experimental setups

  • The evaluation uses 5,000 reference-image instructions with manually annotated pairwise preferences, plus EditRewardBench and GEdit-Bench-EN metrics for editing models.GEdit-Bench-EN reports GPT-4.1-assessed semantic consistency, perceptual quality, and their geometric-mean overall score, including category SC scores.Source: §4.1 · p. 7
  • The RRM uses Qwen-VL-2.5, with 10k human preference pairs for GCPO—less than 1% of SFT-scale data—and GRPO applies it to FLUX.Kontext and Qwen-Image-Edit.Editing optimization uses Flow-GRPO with group size G = 24 and KL penalty coefficient β = 0.04.Source: §4.1 · pp. 7–8

4.3 Image Editing Performance

  • FLUX.Kontext’s Overall Score (O) rises from 5.77 to 6.24 and Semantic Consistency (SC) from 6.27 to 6.86 with RL-RRM (7B).Qwen-Edit’s O improves from 7.45 to 7.50; its Motion Change score rises from 4.01 to 4.62, a 15.2% relative gain. FLUX.Kontext also achieves a GSB score of +23.2 against its original baseline.Source: Table 3 · p. 9, §4.3 · p. 10

4.2 Reward Model Performance

  • For Qwen-7B, SFT accuracy increases from 68.9% with “Think” only to 75.4% with “Think+Verify”; VIESCORE-prompt SFT reaches 68.3%.The paper also reports that removing external “Verify” filtering significantly degrades accuracy and that reasoning traces provide supervisory signals.Source: §4.2 · p. 9
  • On public EditRewardBench, SFT-RRM scores 73.3% versus EditScore-7B’s 65.9%, and GCPO raises final RL-RRM accuracy to 78.2%.The benchmark is independently constructed from the internal pipeline.Source: §4.2 · p. 9, Table 4 · p. 10

4.3 Image Editing Performance

  • GCPO-refined RRMs produce lower training rewards but higher evaluation rewards than SFT counterparts, acting as stricter evaluators and improving editing-model supervision.Figure 4 reports stable reward signals and higher evaluation rewards after GCPO refinement.Source: Figure 4 · p. 8, §4.3 · p. 10
  • Qualitative examples show better instruction adherence and visual fidelity, including targeted Color Alter edits without global color shifts or artifacts.The passage also describes improved subject addition/removal and motion-change edits compared with baseline failures.Source: §4.3 · p. 10

5 Conclusion

Edit-R1 enhances image editing through reinforcement learning from human feedback, using a verifier-based reasoning reward model trained with cold-start SFT and GCPO and integrated with GRPO.

5 Conclusion

  • Edit-R1 combines a verifier-based reasoning reward model with GRPO-based reinforcement learning to enhance image editing.The framework is designed to improve instruction following in state-of-the-art editing models.Source: §5 · p. 10
  • The reward model uses cold-start SFT and GCPO, achieving evaluation accuracy that surpasses powerful proprietary models.Source: §5 · p. 10
  • Integrating the reward model with GRPO substantially enhances state-of-the-art editing models’ instruction-following capabilities.Source: §5 · p. 10

A System prompt

The system prompts structure principle decomposition, image-edit evaluation, and VLM-based verification and selection through explicit checks. They support pointwise and holistic reward judgments and require the verifier to return structured gold annotations as a single JSON object.

A.1 System Prompt for Decomposing Principles

  • A pool of 60 expert-authored examples supplies four randomly sampled few-shot exemplars per decomposition query to improve diversity and robustness.The system prompt defines the evaluator’s role, requirements, output format, and examples.Source: §A.1 · p. 16
  • The decomposition prompt requests 10 JSON evaluation points: 3–4 each for instruction following and feature preservation, and 2–3 for image quality.Each array element contains a question and category field.Source: §A.1 · pp. 16–17

A.2 System Prompt for Reward Model Evaluation

  • The reward model compares before-and-after images against each principle, scoring each point 0 or 1 and treating uncertain judgments as unmet.For unspecified changes in ranges or positions, the prompt requires a significant change before marking a point met.Source: §A.2 · pp. 17–18
  • The final 0–10 score is not merely an average: it weighs core directives, revision difficulty, instruction conflicts, usability, and unreasonable changes to unmentioned content.The prompt says deductions should reflect the effort or uncertainty involved in fixing unmet requirements.Source: §A.2 · pp. 17–18
  • The reward-model prompt assigns a score of 0 when no edit occurred or the image has white borders.It also cautions that perfect pointwise scores do not guarantee a perfect overall score.Source: §A.2 · p. 17

A.3 System Prompt for VLM Verification

  • A VLM verifier uses candidate reasoning and predictions as references, but checks the images and text itself to produce gold-standard 0/1 judgments; a selector then chooses the candidate with highest accuracy.The verifier supplies a one-sentence reason for each judgment, and references must not override the images and text.Source: §A.3 · pp. 18–19
  • The verifier must output only one JSON object, with no text outside it.The system prompt specifies this output constraint.Source: §A.3 · p. 19
  • Each evaluation-point annotation contains the point text, a binary gold label, and a brief reason that may include an approximate bounding box.The shown format uses question, gold, and reason fields.Source: §A.3 · p. 19
  • The output template labels each evaluation point with its text under the question field.The example uses “Evaluation point text 1” as the question value.Source: §A.3 · p. 19
  • The reason field is brief and can include an approximate bounding box.The example explicitly permits an approximate bbox in the reason.Source: §A.3 · p. 19
  • The gold array must preserve the input evaluation-point list’s length and order.Both its length and sequence are required to match the input list.Source: §A.3 · p. 19
  • The system prompt frames the model as a strict inspector producing gold-standard annotations.This is the stated purpose of the verifier prompt.Source: §A.3 · p. 19

B Inference result of RRM

The RRM evaluates an edited image against decomposed principles, explains its judgments in a chain of thought, and summarizes principle-level and comprehensive scores. In the example, it rates the edit 7 despite successful style change and minor vehicle-detail issues.

B Inference result of RRM

  • The RRM receives the source image, instruction, edited image, and principles, then generates detailed chain-of-thought analysis before summarizing scores for each principle and the image overall.The example checks style change, car presence and preservation, structural integrity, image quality, and harmony between the architecture and vehicles.Source: B Inference result of RRM · p. 19, Figure 5 · p. 21
  • The RRM gives the example edit a final score of 7, with a principle-score average of 0.8 (8/10).The style change succeeds, while vehicle details change; the reasoning treats those preservation issues as minor because preserving vehicle details was not the core instruction.Source: B Inference result of RRM · p. 20, Figure 5 · p. 21

D Human Evaluation

The human study compares FLUX.Kontext optimized with RL-RRM (7B) against the original baseline using annotators’ better, comparable, or worse judgments under the Good-Same-Bad protocol.

D Human Evaluation

  • The study scores annotator judgments with GSB = (G − B)/(G + S + B), where higher scores are better.Table 5 reports the human evaluation using the GSB protocol.Source: D Human Evaluation · p. 21, Table 5 · p. 21

E Qualitative Results for FLUX-Kontext

Qualitative results for FLUX.Kontext are presented in Figures 6–8.

E Qualitative Results for FLUX-Kontext

F Qualitative Results for Qwen-Edit

Edit-R1 improves Qwen-Edit on challenging edits, particularly motion-related edits and fine-grained attribute changes. A separate example shows RL correcting an SFT edit that changed an unrelated feature.

F Qualitative Results for Qwen-Edit

  • Edit-R1 further improves Qwen-Edit on challenging edits, especially motion-related edits and fine-grained attribute changes.The comparison shows outputs from Qwen-Edit and Qwen-Edit w. Edit-R1 across diverse instructions.Source: Figure 9 · p. 25
  • RL guided by the RRM preserves the blue hat when changing a shirt to red, correcting the SFT model’s erroneous hat-color change.The RRM penalizes the SFT failure; the RL model’s output preserves the other feature specified by the example.Source: Figure 10 · p. 26

G Qualitative Analysis of RRM Judgments · 7 Question: Does the generated image reference the light blue hat style from the source image?

The qualitative case shows an SFT edit hallucinating a red hat while changing the shirt correctly, then an RRM-guided RL edit preserving the hat. The RRM’s targeted judgments identify the preservation issue and illustrate feedback for correcting a specific weakness.

G Qualitative Analysis of RRM Judgments

  • The analysis examines a common SFT failure—hallucination—to illustrate how the verifier-based RRM guides reinforcement learning.The section presents a qualitative analysis of the RRM’s judgments.Source: G Qualitative Analysis of RRM Judgments · p. 27

7 Question: Does the generated image reference the light blue hat style from the source image?

G Qualitative Analysis of RRM Judgments

  • The case study presents the RRM’s fine-grained evaluation as targeted feedback that helps RL address specific model weaknesses and improve instruction-following.Here, the weakness is changing the hat color while editing the shirt.Source: G Qualitative Analysis of RRM Judgments · p. 27

8 Score: 1 … 8 Score: 1

The verifier distinguishes outputs that preserve the light-blue hat style from those that do not: it identifies the SFT loser’s failure and confirms that the RL-tuned winner learned to preserve it. The RL-tuned loser still fails on that point.

8 Score: 1

  • The verification questions check whether the double-braided hairstyle, light-brown lower garment, and green leaves match the source image.Source: §8 · p. 27
  • Other checks ask whether the blurred outdoor background and details such as stones remain consistent with the source image.Source: §8 · p. 27
  • The verifier correctly identifies the SFT model’s loser output as failing to preserve the hat style.Listing 5 reports a score of 1.Source: §8 · p. 27
  • The SFT model’s winner output also receives a score of 1 in the RRM verification.Listing 6 identifies the output as the SFT model’s winner.Source: §8 · p. 28

8 Score: 0

  • The RL-tuned model’s loser output still fails on the hat-style point.Listing 7 reports a score of 1, despite the stated failure on this specific point.Source: §8 · p. 29

8 Score: 1

  • The RL-tuned winner’s verification confirms that it learned to preserve the hat style.Listing 8 reports a score of 1.Source: §8 · p. 29

Related papers

Loading 2604.27505v2…