PaperlayerView exact revision on arXiv

Source-linked AI summary

Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback

Zongjian Li, Zheyuan Liu, Qihui Zhang, Bin Lin, Feize Wu, Shenghai Yuan, Zhiyuan Yan, Yang Ye, Wangbo Yu, Yuwei Niu, Shaodong Wang, Xinhua Cheng, Li Yuan

arXiv:2510.16888v3cs.CV

TL;DR

Instruction-based image-editing models can overfit supervised annotations and lack a universal reward model for diverse tasks. Edit-R1 addresses these gaps with DiffusionNFT, training-free logit-based MLLM feedback, and low-variance group filtering; UniWorld-V2 reaches state-of-the-art results on ImgEdit and GEdit-Bench while gains transfer across base models.

  • Problem

    Instruction-based image-editing models trained solely by supervised fine-tuning can overfit annotated patterns, while diverse editing tasks lack a universal reward model.

  • Method

    Edit-R1 combines DiffusionNFT policy optimization with continuous output-logit rewards from a training-free MLLM and filtering of noisy low-variance groups.

  • Results

    UniWorld-V2 achieves state-of-the-art performance on ImgEdit and GEdit-Bench, while Edit-R1 substantially improves UniWorld-V2, Qwen-Image-Edit, and FLUX.1-Kontext.

  • Takeaways & Limitations

    The framework is model-agnostic and provides gains across diverse image-editing base models and benchmarks.

  • Takeaways & Limitations

    Reward normalization can amplify insignificant differences when in-group MLLM rewards have high means and low variance, potentially misleading training.

Abstract

from arXiv · show

Instruction-based image editing has achieved remarkable progress; however, models solely trained via supervised fine-tuning often overfit to annotated patterns, hindering their ability to explore and generalize beyond training distributions. To this end, we introduce Edit-R1, a novel post-training framework for instruction-based image editing based on policy optimization. Specifically, we utilize Diffusion Negative-aware Finetuning (DiffusionNFT), a likelihood-free policy optimization method consistent with the flow matching forward process, thereby enabling the use of higher-order samplers and more efficient training. Another key challenge here is the absence of a universal reward model, resulting from the diverse nature of editing instructions and tasks. To bridge this gap, we employ a Multimodal Large Language Model (MLLM) as a unified, training-free reward model, leveraging its output logits to provide fine-grained feedback. Furthermore, we carefully design a low-variance group filtering mechanism to reduce MLLM scoring noise and stabilize optimization. \texttt{UniWorld-V2}, trained with this framework, achieves \textbf{state-of-the-art} results on the ImgEdit and GEdit-Bench benchmarks, scoring 4.49 and 7.83, respectively. Crucially, our framework is model-agnostic, delivering substantial performance gains when applied to diverse base models like Qwen-Image-Edit and FLUX-Kontext, demonstrating its wide applicability. Code and models are publicly available to support further research.

2 METHODOLOGY

Edit-R1 combines DiffusionNFT with training-free MLLM scoring to optimize image-editing models using continuous feedback and group-based noise filtering. Its pipeline samples edits, scores them, filters unreliable groups, and updates the diffusion velocity predictor toward higher-reward outcomes.

  • 2.1 PRELIMINARY: DiffusionNFT performs policy optimization directly on the flow-matching forward process, steering the velocity predictor toward high-reward policies and away from low-reward ones.The method uses a contrastive loss based on reward signals rather than a conventional policy-gradient formulation.
  • 2.2 TRAINING-FREE MLLM SCORING: Training-free MLLM scoring evaluates each original image, edited image, and instruction, using output logits to produce continuous rewards.The framework compares chain-of-thought and non-chain-of-thought scoring, as well as sampling-based and logit-based reward extraction.
  • 2.2 TRAINING-FREE MLLM SCORING: Edit-R1 uses non-chain-of-thought logit scoring over tokens 0 through 5, converting their probability distribution into an expected score and normalizing it to [0, 1].This approach preserves the MLLM’s confidence distribution instead of extracting only a discrete sampled score.
  • 2.3 LOW-STD GROUP FILTERING: Low-variance groups with high mean rewards can amplify insignificant scoring differences during normalization, so their gradients are discarded using mean and variance thresholds.Filtering targets groups whose mean reward exceeds τµ and whose variance falls below τσ.
  • 2.4 PIPELINE OF EDIT-R1: The Edit-R1 pipeline samples image groups, scores them with an MLLM, computes group rewards, and applies DiffusionNFT to move the policy toward higher-quality edits.Sampling uses DPM-Solver, while evaluation prompts combine base editing requirements with task-specific instructions.

3 EXPERIMENTS

Experiments evaluate Edit-R1 across diverse editing data, benchmarks, human preferences, ablations, reward alignment, and reward-model scaling. The method improves multiple base models, generalizes to out-of-domain editing, and benefits from larger reward models and low-variance filtering.

  • 3.1 DATASET: 27,572 instruction-based editing samples cover nine task types, while online learning requires only original images and editing instructions.The tasks include Replace, Adjust, Remove, Background, Hybrid, Action, Text Edit, Redbox Control, Reference, and Extract.
  • 3.3 MAIN RESULTS: 4.02 is the ImgEdit score for FLUX.1-Kontext [Dev] after finetuning, up from 3.71 and above FLUX.1-Kontext [Pro] at 4.00.Qwen-Image-Edit [2509] also rises from 4.35 to 4.48, while UniWorld-V2 achieves the best performance.
  • 3.3 MAIN RESULTS: 6.74 is the GEdit-Bench score for FLUX.1-Kontext [Dev] after finetuning, up from 6.00 and above the Pro version at 6.56.Qwen-Image rises from 7.54 to 7.76, and UniWorld-V2 outperforms all listed models on this benchmark.
  • 3.3 MAIN RESULTS: Users prefer UniWorld-FLUX.1-Kontext over FLUX.1-Kontext [Dev] across evaluation criteria, while official models retain slightly better image quality.The finetuned model receives more likes because of stronger instruction-following ability.
  • 3.4 ABLATION STUDY: DiffusionNFT outperforms Flow-GRPO baselines on ImgEdit, and group filtering raises Qwen-Image-Edit’s GEdit-Bench score from 7.54 to 7.76 after DiffusionNFT alone reaches 7.72.The ablation isolates gains from the policy optimization method and the filtering mechanism.
  • 3.5 ANALYSIS: 74.74% pairwise accuracy makes logit-based scoring the best-aligned evaluated reward mechanism with human preferences.Alignment remains consistent across diverse editing tasks.
  • 3.5 ANALYSIS: Larger reward models improve policy performance and mitigate reward hacking by maintaining reward variance for sustained exploration.The 32B model maintains high reward variance, whereas smaller models exhibit early variance collapse.

4 RELATED WORK

Related work traces image editing from global stylistic control toward spatially precise methods, then situates Edit-R1 within reinforcement-based alignment and MLLM judging. It highlights the challenge of converting MLLM evaluations into fine-grained rewards for image editing.

  • Image Editing: Early diffusion editing methods offered global stylistic control, while inversion-based techniques addressed the need for greater spatial precision.Image editing must alter specific attributes while preserving unedited regions.
  • Reinforcement Learning for Generation: RLHF-inspired approaches for generative image models typically train reward models on general human preferences or prompt-image alignment.This extends reinforcement-learning alignment beyond language models to text-to-image systems.
  • MLLM-based Evaluation: MLLM-as-a-Judge systems correlate with human judgments and have therefore been used as active rewards for optimizing generative models.The paradigm shifts MLLMs from passive evaluation toward optimization feedback.
  • MLLM-based Evaluation: Discrete MLLM scores are sparse and coarse for subtle visual improvements, motivating logit-based methods that derive continuous rewards from token distributions.The goal is a finer-grained signal for image-editing quality.

5 CONCLUSION

Edit-R1 addresses generalization limitations in instruction-based image editing through MLLM-based reward modeling and DiffusionNFT, achieving state-of-the-art benchmark performance across multiple base models.

  • Edit-R1 combines a training-free MLLM reward model with DiffusionNFT for post-training diffusion models on image editing.The reward model provides fine-grained continuous feedback from output logits, while DiffusionNFT performs likelihood-free policy optimization consistent with flow matching.
  • Edit-R1 achieves state-of-the-art performance on ImgEdit and GEdit-Bench while boosting UniWorld-V2, FLUX.1-Kontext, and Qwen-Image-Edit.The framework is evaluated across diverse editing benchmarks and base models.
  • The MLLM-derived reward signal highly correlates with human preferences, guides higher-quality outputs, and mitigates reward hacking.

6 CONTRIBUTORS

The paper lists its core contributors, additional contributors, corresponding author, and contact email addresses.

  • Core contributors are Zongjian Li, Zheyuan Liu, Qihui Zhang, and Bin Lin.
  • Additional contributors are Feize Wu, Shenghai Yuan, Zhiyuan Yan, Yang Ye, Wangbo Yu, Yuwei Niu, Shaodong Wang, and Xinhua Cheng.
  • Li Yuan is the corresponding author.
  • The listed contact emails are zongjianli25@stu. and yuanli-ece@pku.edu.cn.

A TRAINING SETTINGS

The training-settings appendix points readers to Table 5, which presents the key hyperparameters used in training.

  • The paper presents key training hyperparameters in Table 5.
  • The supplied passage does not specify the individual hyperparameter values.
  • Table 5 is titled “Key hyperparameters.”

B DETAILED HUMAN ALIGNMENT

The human alignment study evaluates how closely the proposed reward mechanism matches human judgment using pairwise comparisons and score-distribution similarity.

  • The study evaluates reward alignment through pairwise-comparison accuracy and score-distribution similarity.
  • It collects 800 edited images from 200 unique image-instruction pairs.
  • Three human evaluators annotate the results across two distinct dimensions.

B.1 REWARD METHOD DEFINITIONS

The evaluation compares continuous, discrete, binary, and pretrained reward mechanisms for scoring image edits. Score Logit uses MLLM score-token logits to produce a normalized expected reward, while other methods use sampled scores, binary probabilities, reasoning-enhanced variants, or a pretrained reward model.

  • Score Logit: Score Logit extracts logits for score tokens 0–5, applies softmax, computes their expected value, and normalizes the result to [0, 1].It is the primary reward mechanism evaluated in this section.
  • Sampling-based methods: Score Sampling parses a single MLLM-generated numerical score from 1 to 5 and normalizes it to [0, 1] as the reward.This approach directly uses an explicit discrete score.
  • Binary reward: Yes/No Logit reframes edit evaluation as binary classification and uses the softmax probability of “Yes” as the reward signal.The logits for “Yes” and “No” are converted into a probability distribution.
  • CoT variants: Score Sampling + CoT generates reasoning before producing a discrete score, while Score Logit + CoT extracts score-token logits after reasoning to compute a weighted-average reward.Both variants add Chain-of-Thought reasoning before the final reward calculation.
  • Baseline: Unified Reward uses a pretrained reward model's direct output without modification or retraining.It serves as a strong baseline for comparison.

B.2 ALIGNMENT IN PAIRWISE PREFERENCE

Pairwise preference alignment measures how often reward models agree with human judgments about which edited image is better. Score Logit achieves the strongest reported alignment, while adding Chain-of-Thought degrades continuous scoring performance.

  • Evaluation criterion: Higher pairwise accuracy indicates that a reward model's relative-quality judgments more closely resemble human perception.The evaluation uses human pairwise comparisons as ground truth.
  • Pairwise preference: 74.74% pairwise accuracy: Score Logit surpasses binary and discrete-score baselines in matching human preferences.Pairwise accuracy measures the percentage of image pairs for which the model selects the same preferred image as human annotators.
  • Effect of CoT: Incorporating Chain-of-Thought unexpectedly degrades continuous scoring methods.The authors attribute this to reasoning-induced bias that can shift attention from subtle visual details toward generated textual justifications.

B.3 ALIGNMENT IN SCORE DISTRIBUTION

Score-distribution alignment compares reward-model score tendencies with human absolute-quality scores. Score Logit's distribution most closely mirrors human evaluators and separates Good from Bad edits more strongly than other methods.

  • Evaluation criterion: The analysis evaluates alignment by comparing reward-model score distributions against human annotator distributions.Human annotators assign quality labels and absolute scores from 1 to 5 to edited images.
  • Distribution alignment: Score Logit's score distribution most closely mirrors the distribution produced by human evaluators.Distributional similarity indicates a comparable preference tendency and quality scale.
  • Category separation: Score Logit assigns significantly higher scores to Good edits and lower scores to Bad edits than competing reward methods.This supports stronger consistency with human judgments across absolute-quality categories.

B.4 ANNOTATION DETAILS

The annotation study builds human-alignment ground truth from edited-image comparisons and absolute quality ratings. Multiple evaluators, consensus rules, and standardized MLLM prompts support the pairwise and score-based evaluations.

  • Dataset and annotators: 800 edited images from 200 image-instruction pairs were annotated by five evaluators experienced in image editing.The study uses these annotations to construct ground truth for reward-model alignment.
  • Quality control: Each annotation initially received three evaluators, with labels accepted after at least two agreed; unresolved cases were reassigned to two remaining evaluators.This protocol was designed to improve annotation reliability and consistency.
  • Annotation tasks: For pairwise preference, annotators selected the better edit or marked equivalent quality, while absolute scoring evaluated each edit's quality.These tasks provide the two forms of human judgment used in the alignment analysis.
  • Yes/No evaluation: The Yes/No template asks whether the edit satisfies the instruction, requirements, and visual-quality criteria, returning a Yes or No judgment.The response format is explicitly restricted to a Yes/No judgment.
  • Score evaluation: The score template rates editing accuracy and quality from 0 to 5, with 0 denoting failure and 5 denoting a correct, high-quality result.The template defines endpoint criteria for the numerical score.
  • CoT score evaluation: The CoT score template requests concise reasoning before the final 0–5 score.Its response format contains separate reasoning and score fields.
Loading 2510.16888v3…