PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation-Verification Reward for Consistent Multi-Reference Image Editing

Yingmao Miao, Pengfei Zhang, Xiaochen Lv, Meng Yu, Lei Sun, Xiangxiang Chu, Chao Shen, Chenhao Lin

arXiv:2607.29025v2cs.CV

TL;DR

Multi-reference image editing still lacks reward models that assess relational consistency and visual harmony across references. The paper introduces Evaluation-Verification Reward (EVR) with reinforcement learning and a scalable data pipeline, achieving a 67% human-preference win rate over the base model while matching or surpassing NanoBanana.

  • Problem

    Existing reward models lack the fine-grained, multi-dimensional sensitivity needed to evaluate reference consistency and visual harmony in multi-reference editing.

  • Method

    EVR decomposes evaluation into five visual criteria, generates multiple evaluator hypotheses, and uses visual verification to produce grounded rewards for reinforcement learning.

  • Results

    67% human-preference win rate over the base model, with substantially improved consistency and visual harmony matching or surpassing NanoBanana.

  • Takeaways & Limitations

    EVR enables stable, human-aligned reinforcement-learning updates that enhance multi-reference editing without requiring ground-truth edits.

Abstract

from arXiv · show

While recent image editing models have made rapid progress, multi-reference editing remains challenging, particularly in maintaining visual consistency across references and ensuring overall visual harmony. Reinforcement learning has proven highly effective for text-to-image generation and single-image editing, but its extension to multi-reference editing is hindered by the absence of suitable reward models that capture multi-image relational constraints. Moreover, naively using multimodal large language models(MLLMs) as zero-shot evaluators faces a key tension between hallucination-prone long-form reasoning and the limited deductive power of short-form judgments. We address these issues with a Multi-dimensional Evaluation-Verification Reward(EVR). EVR decomposes evaluation into distinct visual criteria; for each criterion, an MLLM Evaluator generates multiple candidate hypotheses, and a Verifier grounds each claim in concrete visual evidence to accept or reject it, producing reliable and fine-grained reward signals. Together with a scalable data pipeline, our method enables RL fine-tuning of off-the-shelf editors without architectural changes. Extensive experiments show substantial gains over the base Qwen-Image-Edit, improving consistency and harmony to match or surpass NanoBanana.

1 Introduction

Multi-reference image editing requires evaluation that captures reference consistency and visual harmony, but existing metrics and naive MLLM evaluation lack sufficient fine-grained visual grounding. The proposed EVR framework combines multi-criteria evaluation, evidence-based verification, and reinforcement learning to produce reliable rewards, with experiments showing a 67% human preference win rate over the base model.

  • Motivation: Multi-reference editing evaluation must assess reference consistency and visual harmony beyond coarse text alignment and generic visual quality.Conventional metrics such as CLIPScore rely on global semantic embeddings and lack fine-grained sensitivity.
  • Motivation: Naive MLLM evaluators face a trade-off between reasoning depth and visual grounding, as overly long chains-of-thought can encourage hallucination.The paper identifies this issue through observations from Uniworld-v2(Edit-r1).
  • Method: EVR decomposes evaluation into five criteria and separates short-chain evaluation from evidence-grounded verification to reduce cognitive overload and curb hallucinations.The criteria are reference consistency, scene consistency, visual harmony, instruction consistency, and overall visual quality.
  • Method: The reinforcement learning framework has an Evaluator generate multiple hypotheses per criterion and a Verifier ground claims in visual evidence for stable, visually grounded rewards.These rewards support policy optimization for multi-reference image editing.
  • Results: 67% human preference win rate against the base model demonstrates improved consistency and visual harmony, matching or surpassing NanoBanana.The method is evaluated on test datasets and real-world examples using a scalable multi-reference data pipeline.

2 Related Works

Related work spans diffusion-based image generation and editing, reinforcement-learning methods for diffusion models, and vision-language reward models. EVR combines MLLM evaluation with visual verification to provide stable rewards for reinforcement-learning-based image editing.

  • Diffusion-Based Image Editing: Diffusion models established text-to-image generation as a cornerstone of generative AI and drove rapid advances in image editing.Early editing methods used inversion-based techniques or explicit controls such as Textual Inversion, ControlNet, and IP-Adapter.
  • Reinforcement Learning for Diffusion Models: RL advances in language-model alignment catalyzed related progress in text-to-image generation and editing.Subsequent work explored RL-based approaches for diffusion models and image editing.
  • Reinforcement Learning for Diffusion Models: DPO removed explicit reward models but remained limited by static offline datasets, while GRPO and variants enabled scalable online learning for diffusion architectures.The distinction centers on DPO’s lack of dynamic feedback versus the online-learning capability introduced by GRPO-based methods.
  • Vision-Language Reward Models: CLIP and related VLMs measure semantic alignment but lack explicit reasoning for complex compositional attributes, motivating MLLM-based reward evaluation.PickScore refines embedding-based evaluation using human preference data, while MLLMs provide a reasoning-oriented alternative.
  • Evaluation-Verification Rewards: EVR combines MLLM evaluation with visual verification to generate stable rewards that guide RL fine-tuning for precise image editing.The process consists of sampling, EVR, and RL.

3 Method

The method combines reinforcement learning with a Multi-dimensional Evaluation Verification Reward (EVR) and a scalable pipeline for semantically aligned multi-reference editing data. EVR evaluates candidates across multiple visual dimensions, verifies evaluator claims against visual evidence, and aggregates dimensions multiplicatively to discourage reward hacking.

  • 3.1 Method Overview: The framework uses a scalable pipeline to build semantically aligned reference-image and instruction tuples, then applies DiffusionNFT reinforcement learning to optimize image editing.Each iteration samples diverse candidate edits through random initialization before EVR evaluates them.
  • 3.2 Scalable Data Pipeline: Generic tuple construction can create severe semantic mismatches, so the pipeline uses one foreground object and one background scene in a representative N = 2 setting.This alignment provides well-posed environments for optimizing object consistency and visual harmony.
  • 3.2 Scalable Data Pipeline: The pipeline generates reference objects, compatible scenes, and grounded instructions for replacement and insertion tasks, producing 10K triplets and a 1K-triplet test set.It also includes over 300 real-world examples for out-of-domain evaluation and arbitrary-reference-count testing.
  • 3.3 Evaluation-Verification Reward: EVR decomposes reward computation into rationale-based multidimensional evaluation and visual-grounding verification to reduce hallucination noise in MLLM judgments.An MLLM Evaluator assesses each edited image using five key dimensions, including identity consistency and physical plausibility.
  • 3.3 Evaluation-Verification Reward: EVR uses a geometric mean rather than an arithmetic mean, making high total reward require strong performance across all editing dimensions.The multiplicative aggregation is intended to prevent reward hacking through improvement in one dimension while another fails.

4 Experiments

Experiments show that EVR improves multi-reference editing consistency, instruction following, and harmony over baseline and competing reward formulations. Ablations and out-of-distribution evaluations further show that dimension-wise verification, balanced reasoning, and fine-grained rewards are essential for robust editing.

  • Comparison with Baselines: EVR yields substantial gains in Reference Consistency, Harmony, and Instruction Consistency while maintaining stable Image Quality over Qwen-Image-Edit baselines.Scene Consistency dips slightly because baseline models can inflate it by copying the reference scene, whereas EVR prevents this shortcut.
  • Qualitative Evaluation: The RL-tuned model retains fine-grained reference details, disambiguates object and scene context, follows addition or replacement instructions, and harmoniously aligns inserted objects with scene lighting and geometry.Qualitative comparisons report fewer texture losses, context leaks, instruction errors, floating artifacts, disharmony, and geometric misalignment.
  • Reward-Model Comparison: EVR achieves superior alignment with human judgment compared with Joint CoT, Decoupled Direct, Decoupled CoT, CoT Averaging, and Logit Weighted rewards.The compared formulations differ in whether they jointly or separately score dimensions, use rationales, average reasoning trials, or weight discrete-score logits.
  • Ablation Study: Ablations show that removing the Verifier harms complex-reasoning metrics, long Thinking-model chains mislead evaluation, removing CoT harms deduction, and joint judgment collapses editing into scene copying.Joint evaluation artificially inflates Scene Consistency while eliminating editing capability, demonstrating the importance of dimension-wise fine-grained rewards.
  • Generalization to the Wild and N > 2: On real-world OOD data, the model outperforms the base model, Flux2, and Gemini2.5-Flash-Image (NanoBanana) across settings, including unseen N > 2 scenarios.The model was trained only on synthetic N = 2 data yet maintains robust instruction following and visual harmony for diverse real-world tuples with N ≥2 images.
  • User Study: 6,768 user-study feedback responses support significant improvements over baselines and superior consistency and harmony against leading open-source and closed-source methods.The study compares fine-tuned models with their baselines and competing multi-reference image-editing systems.

5 Conclusion · A Implementation Details · A.1 Setting

The paper proposes Evaluation-Verification rewards and a scalable synthetic data pipeline for reinforcement learning in multi-reference image editing. It also reports implementation resources and verifier settings intended to support stable, efficient training.

  • 5 Conclusion: The framework uses Evaluation-Verification rewards to suppress hallucinations in MLLM-based feedback for multi-reference image editing.Evaluator rationales are grounded in visual evidence.
  • 5 Conclusion: Grounded evaluator rationales enable stable, human-aligned policy updates without requiring ground-truth edits.The method combines this mechanism with a scalable synthetic data pipeline.
  • 5 Conclusion: The method significantly enhances reference consistency, context disambiguation, instruction consistency, and visual harmony in Qwen-Image-Edit.The conclusion reports these gains without architectural changes or additional quantitative values in the supplied passage.
  • A.1 Setting: Training hyperparameters are presented in Table 4.The supplied implementation passage identifies Table 4 as the location of the key hyperparameters.
  • A.1 Setting: The categories of objects included in the dataset are shown in Figure 11.The supplied passages identify the figure but do not enumerate its categories.
  • A.1 Setting: The evaluator outputs five candidate rationales and scores to balance verifier availability against computational overhead.This setting is described as a trade-off between verifier availability and computational overhead.
  • A.1 Setting: When verification fails, the method uses the mean candidate score as a fallback to maintain reward stability.The fallback applies upon verifier failure.

A.2 Ablation on the Number of Evaluation Hypotheses (K) · B Additional Analysis and Results

The ablation finds that generating more evaluation hypotheses improves the chance of obtaining a valid grounded claim, but correlated hallucinations make simple averaging unreliable. EVR therefore uses a visual Verifier and caps K at 5 to balance reward quality against computational and reasoning costs.

  • A.2 Ablation on the Number of Evaluation Hypotheses (K): 89% availability is reached at n = 5, up from 70% at n = 1, on 500 generated samples.Top-n availability measures the probability of obtaining at least one visually grounded and valid evaluation claim.
  • A.2 Ablation on the Number of Evaluation Hypotheses (K): The observed growth indicates that evaluation hypotheses are not independent and identically distributed.The empirical availability trend does not match the assumption of independent evaluations.
  • A.2 Ablation on the Number of Evaluation Hypotheses (K): The MLLM Evaluator systematically repeats similar hallucinated textual priors when handling complex edits.This behavior reflects systematic bias rather than purely random noise across independent generations.
  • A.2 Ablation on the Number of Evaluation Hypotheses (K): Simple averaging is inherently flawed under this mode collapse, making the visual Verifier essential for extracting valid reward signals.The non-i.i.d. evaluator behavior directly motivates EVR’s verification stage.
  • B Additional Analysis and Results: Table 6 reports reward statistics across different models.The supplied passage provides the table’s scope but no individual statistics.
  • A.2 Ablation on the Number of Evaluation Hypotheses (K): K is capped at 5 because increasing it would linearly increase computational overhead during online reinforcement learning.The restriction applies despite the upward trend in availability.
  • A.2 Ablation on the Number of Evaluation Hypotheses (K): Larger K values feed more claims to the Verifier simultaneously, substantially expanding its Chain-of-Thought (CoT) length.The longer context can overwhelm the MLLM, worsening text-induced bias and visual grounding capability.

B.1 Reward Distributions and qualitative analyses

EVR addresses reward inflation by verifying short evaluator hypotheses against concrete visual evidence, rejecting hallucinated praise and retaining grounded critiques. This behavior contrasts with baseline methods’ inflated, low-variance scoring distributions caused by context bias.

  • Qualitative reward verification: Baseline MLLM evaluators produced artificially inflated scores ranging from 4.4 to 5 through fluent reasoning hallucinations in a watch-insertion scenario.The cited baselines are Direct Scoring, CoT Mean, and Logit Weighted Scoring.
  • Qualitative reward verification: EVR generates multiple independent short hypotheses, then verifies each claim against concrete visual evidence before producing a final reward.This evaluator–verifier decomposition allows hallucinated praises and invalid perfect scores to be rejected.
  • Qualitative reward verification: EVR rejected an ungrounded evaluation after identifying an implausible screen orientation, producing a visually grounded final score of 3.The example demonstrates how factual verification prevents text-induced bias from determining the reward.
  • Reward distributions: Direct Scoring exhibited an overconfident mean of 0.95 and exceptionally low variance of 0.029, reflecting context bias and score inflation.The passage describes this behavior as essentially generating perfect scores regardless of actual edit quality.

B.2 Detailed results of User Study

A user study with 12 expert evaluators found that the EVR policy decisively outperformed base, generalist, and state-of-the-art systems. Its strongest dimensional gains were in Reference Consistency and Harmony, including against Gemini.

  • Study setup: The study randomly assigned 12 expert evaluators to compare EVR with Qwen-2509, Qwen-2511, Flux2, and Gemini-2.5-flash-image.The comparison included base models and state-of-the-art systems.
  • Overall comparison: 65.0% and 67.0% strict win rates against Qwen-2509 and Qwen-2511 established EVR’s advantage, including a nearly 6-to-1 win-to-loss ratio against Qwen-2509.EVR also won 57.2% against Flux2 and 58.0% against Gemini-2.5-flash-image.
  • Dimension breakdown: Against Qwen-2509, EVR achieved 62% win rate in Reference Consistency and 52% in Harmony.These dimension-level results support the paper’s multi-dimensional reward design.
  • Dimension breakdown: 59% win rate in Reference Consistency against Gemini indicated that EVR’s verification mechanism enforced complex relational constraints beyond baseline capabilities.The study identified Reference Consistency and Harmony as particular strengths.

B.3 Cross-Model and Strategy Analysis. · B.4 Discussion on Computational Efficiency

EVR remains effective across model scales and reasoning strategies, with verification compensating for limited backbone capacity. Its training benefits are achieved with low practical overhead through data-efficient convergence and asynchronous reward inference.

  • B.3 Cross-Model and Strategy Analysis.: EVR consistently outperforms unverified baselines on Consistency and Harmony across model scales and reasoning strategies.These dimensions are especially prone to text-induced hallucinations.
  • B.3 Cross-Model and Strategy Analysis.: Qwen3-VL-8B-Instruct with EVR matches or exceeds 32B/3.5-Plus baselines without verification on key dimensions.This shows that verification can compensate for limited model capacity.
  • B.3 Cross-Model and Strategy Analysis.: EVR delivers reliable rewards regardless of backbone scale through its verification pipeline.The comparison covers six configurations spanning 8B, 32B, and 3.5-Plus models, with verification versus CoT-only strategies.
  • B.4 Discussion on Computational Efficiency: The two-stage EVR process incurs minimal overhead through data efficiency and asynchronous inference optimizations.The implementation generates multiple hypotheses and verifies them while reducing queries and overlapping inference with training.
  • B.4 Discussion on Computational Efficiency: EVR converges with only 1,000 training samples, reducing reward queries required across the training trajectory.This data-efficiency result is reported in Fig. 13.
  • B.4 Discussion on Computational Efficiency: Asynchronous vLLM serving lets Evaluators and Verifiers generate rewards without blocking diffusion-model GPU updates.Inference overlaps with gradient computation and achieves real-time reward generation with negligible waiting time.
  • B.4 Discussion on Computational Efficiency: EVR’s substantial consistency and harmony gains come with a negligible marginal increase in per-step wall-clock time.The paper characterizes the practical cost as minimal.

B.5 Failure Case when N ⩾5

When integrating five or more references, base models exceed their operational capacity, causing severe image-quality degradation and incomplete denoising. EVR preserves relational and semantic placement but cannot fully restore low-level fidelity when the base model’s generative prior collapses.

  • Method limitation: EVR improves instruction alignment and relational constraints but remains bounded by the underlying base model’s generative prior.This bound becomes evident when five or more distinct references exceed the base models’ fundamental operational capacity.
  • Failure under extreme reference counts: For N ≥5, base models frequently exhibit feature entanglement across objects, severe image-quality degradation, and incomplete denoising.The task complexity overwhelms the diffusion architecture’s raw synthesis capabilities.
  • Failure under extreme reference counts: Despite similar noise artifacts, our method correctly places children’s clothing, the woman’s top, slippers, and headphones in appropriate contexts.Figure 14 also reports noticeable noise in the base outputs and residual noise in our result.
  • Method limitation: EVR provides reliable optimization gradients for relational alignment but cannot completely rescue low-level image fidelity after foundational prior collapse.The method consistently improves multi-image editing within the base model’s capability limits.

B.6 More Results · C Prompt Templates

The paper reports supplementary qualitative results showing EVR’s robustness in preserving reference identity and visual harmony across complex edits, and provides reproducible Evaluator and Verifier prompt templates. The templates decompose assessment into visual criteria, generate independent judgments, and validate claims against image evidence.

  • B.6 More Results: Supplementary results demonstrate consistent reference identity and visual harmony across diverse, complex editing instructions.These qualitative examples are presented in Figs. 16, 17, 19, and 20.
  • C Prompt Templates: The Evaluator prompt elicits structured reasoning across five distinct visual dimensions.The prompt templates are provided to support reproducibility and transparency for EVR.
  • C.1 Evaluator Prompt: The Evaluator generates multiple independent rationales and preliminary scores for edited-image assessment.The prompt instructs the base language model to act as the Evaluator.
  • C.1 Evaluator Prompt: Object-retention evaluation focuses on whether the edited image preserves the reference object’s identity and essential visual characteristics.The prompt distinguishes weak and partial matches when identifying features or critical attributes are missing, altered, or inconsistent.
  • C.1 Evaluator Prompt: Scene-consistency evaluation ignores the localized edited object and assesses the remaining environment, lighting, perspective, and human-object context.The prompt targets scene-level consistency rather than changed meaning or style caused by the edit.
  • C.1 Evaluator Prompt: Harmonization evaluation emphasizes physical plausibility, common usage, wearing patterns, orientations, and realistic object interactions.The criteria also consider lighting direction, shadow softness, and color grading when judging whether the inserted object fits its environment.
  • C.1 Evaluator Prompt: Instruction-compliance and aesthetic prompts assess execution of the requested edit alongside image quality factors such as focus, exposure, color, and composition.The instruction-compliance prompt first identifies reference and edited images, while the aesthetic rubric ranges from bad to good quality.
  • C.2 Verifier Prompt: The Verifier cross-checks Evaluator outputs against visual inputs by aggregating claims, verifying them as true, false, or unsupported, and returning ordered validity labels.An evaluation is marked valid only when all required conditions are met; otherwise it receives an invalid label.

D Additional Qualitative Results

Additional qualitative results compare EVR with alternative MLLM-based reward strategies, showing stronger performance than Edit-R1 and stable harmony and consistency across evaluator scales. They also illustrate reduced hallucinated reward signals and extend evaluation to in-distribution, OOD, and N > 2 editing cases.

  • Additional cases: Additional qualitative examples extend the analysis to in-distribution editing cases.These examples supplement the appendix’s evidence for the paper’s main claims.
  • Reward-strategy comparisons: EVR is substantially better than Edit-R1 and remains stable in harmony and consistency across models of different evaluator scales.The comparison also includes a setting without a verifier.
  • Reward verification: Visual verification suppresses hallucinated reward signals compared with other MLLM-based reward formulations.EVR’s evaluations are compared alongside the input images, instructions, and edited outputs.
  • Additional cases: Additional results evaluate EVR on OOD data and datasets with N > 2 references.The appendix includes separate qualitative results for both settings.
Loading 2607.29025v2…