PaperlayerView exact revision on arXiv

Source-linked AI summary

HP-Edit: A Human-Preference Post-Training Framework for Image Editing

Fan Li, Chonghuinan Wang, Lina Lei, Yuping Qiu, Jiaqi Xu, Jiaxiu Jiang, Xinran Qin, Zhikai Chen, Fenglong Song, Zhixin Wang, Renjing Pei, Wangmeng Zuo

arXiv:2604.19406v1cs.CVcs.AI

TL;DR

Human-preference alignment for diffusion-based image editing lacks scalable datasets and task-aware RL frameworks. HP-Edit addresses this gap with a VLM-based HP-Scorer, preference-data pipeline, and RL post-training, achieving stronger human-preference-aligned editing, including an overall score of 4.667 versus 4.472 for QwenImage-Edit-2509. The authors also identify dependence on designed system prompts as a limitation of HP-Scorer.

  • Problem

    Human-preference RL for diffusion-based image editing remains underexplored because scalable preference datasets and frameworks tailored to diverse editing tasks are lacking.

  • Method

    HP-Edit uses a pretrained VLM-based HP-Scorer to evaluate editing cases, construct hard-case preference data, and provide rewards for task-aware RL post-training.

  • Results

    HP-Edit achieves the best performance across almost all eight RealPref-Bench sub-tasks and reaches an overall HP-Score of 4.667 versus 4.472 for QwenImage-Edit-2509.

  • Takeaways & Limitations

    HP-Edit improves instruction faithfulness, realism, artifact reduction, and scene-structure preservation in image editing.

  • Takeaways & Limitations

    HP-Scorer is highly dependent on the designed system prompts for evaluating editing tasks.

Abstract

from arXiv · show

Common image editing tasks typically adopt powerful generative diffusion models as the leading paradigm for real-world content editing. Meanwhile, although reinforcement learning (RL) methods such as Diffusion-DPO and Flow-GRPO have further improved generation quality, efficiently applying Reinforcement Learning from Human Feedback (RLHF) to diffusion-based editing remains largely unexplored, due to a lack of scalable human-preference datasets and frameworks tailored to diverse editing needs. To fill this gap, we propose HP-Edit, a post-training framework for Human Preference-aligned Editing, and introduce RealPref-50K, a real-world dataset across eight common tasks and balancing common object editing. Specifically, HP-Edit leverages a small amount of human-preference scoring data and a pretrained visual large language model (VLM) to develop HP-Scorer--an automatic, human preference-aligned evaluator. We then use HP-Scorer both to efficiently build a scalable preference dataset and to serve as the reward function for post-training the editing model. We also introduce RealPref-Bench, a benchmark for evaluating real-world editing performance. Extensive experiments demonstrate that our approach significantly enhances models such as Qwen-Image-Edit-2509, aligning their outputs more closely with human preference.

1. Introduction

HP-Edit addresses the underexplored problem of human-preference alignment for image-to-image editing, where systems must satisfy both task accuracy and visual naturalness. It combines task-aware preference scoring, scalable hard-case data construction, and reinforcement-learning post-training, supported by RealPref-50K and RealPref-Bench.

  • Motivation: SFT-based editing models can misalign with real-world preferences because their data mixes sources such as cartoons and synthetic images.Constructing preference-aligned editing datasets also requires substantial annotation effort.
  • Motivation: Human-preference RL for image-to-image editing remains underexplored because editing requires both task accuracy and natural-looking results.The paper identifies a need for efficient preference-data construction and task-aware reward models across editing sub-tasks.
  • Contributions: HP-Edit unifies a VLM-based HP-Scorer, hard-case-focused dataset construction, and task-aware RL post-training.The framework is designed to align editing models with human preferences while preserving editing accuracy.
  • Contributions: RealPref-50K contains over 50K real-world-oriented cases spanning eight editing sub-tasks and balanced MS-COCO object categories.The tasks include addition, removal, background replacement, object swapping, color change, bokeh, relighting, and style transfer.
  • Contributions: RealPref-Bench evaluates preference-aligned editing on real-world images with manually verified preference instructions.It is intended to support rigorous model evaluation.
  • Related context: Diffusion and flow-based preference optimization methods have improved generation quality, but their application to diverse image editing remains limited.The related literature includes DPO-, score-matching-, and GRPO-based extensions for generative vision.

3. Preliminaries

Flow matching trains a continuous generative process by matching velocity fields along interpolations between base and data samples. Flow-GRPO enables reward-based policy optimization by introducing stochasticity, comparing grouped sample rewards, and updating toward above-average outcomes.

  • 3.1. Flow Matching: Flow matching trains continuous normalizing flows by aligning modeled velocity fields with those derived from data interpolations.Rectified flow defines intermediate samples through linear interpolation between a base sample and a data sample.
  • 3.1. Flow Matching: The target velocity field in rectified flow is the constant vector from the base sample x0 to the data sample x1.The model learns a neural velocity field vθ(xt, t) to approximate this target.
  • 3.2. Flow-GRPO: Flow matching can be formulated as an MDP in which the flow model generates a trajectory and receives a reward at the final step.The terminal reward R(xT, c) measures the quality of xT or its alignment with prompt c.
  • 3.2. Flow-GRPO: GRPO computes each sample’s advantage by normalizing its reward against the statistics of a group generated from the same prompt.This group-relative comparison supports lightweight, memory-efficient online reinforcement learning.
  • 3.2. Flow-GRPO: GRPO updates policy parameters by maximizing an objective that favors above-average rewards while maintaining training stability.The objective is evaluated over trajectories sampled from the previous policy.
  • 3.2. Flow-GRPO: Flow-GRPO converts deterministic flow-matching dynamics into an equivalent stochastic differential equation so GRPO can explore through sampling.The resulting SDE preserves the original ODE flow’s marginal probability density, while σt controls generation stochasticity.

4. Approach

HP-Edit is a three-stage framework that uses VLM-based human-preference scoring to construct a hard-case dataset and reward task-aware RL post-training. It introduces RealPref-50K and RealPref-Bench to support balanced real-world editing training and evaluation.

  • HP-Edit combines HP-Scorer, hard-case-focused dataset construction, and task-aware RL post-training for human-preference-aligned editing.The framework is designed to improve preference alignment while preserving editing accuracy.
  • HP-Scorer: Human annotators rate approximately 50–100 editing cases per sub-task on a 0–5 scale to calibrate task-specific VLM scoring prompts.The prompts are iteratively refined with task-specific reasoning questions until HP-Scorer results closely match human judgments.
  • Human preference data construction pipeline: The pipeline filters raw editing cases because pretrained models receive the maximum score on many easy examples, limiting their training value.The retained dataset focuses on cases that better capture preference signals for online post-training.
  • Task-aware RL Post-Training: Task-aware RL post-training uses HP-Scorer as the reward model and applies online Flow-GRPO to the filtered dataset.The final reward is normalized to [0, 1] with a sigmoid function using scaling and shift parameters α=2 and β=5.
  • RealPref-50K: RealPref-50K contains 55,795 real-world editing cases spanning eight common tasks with balanced coverage of object categories.Cases are formed from input images, instructions, and model-generated outputs, then scored by HP-Scorer to filter high-quality, hard editing cases.
  • RealPref-Bench: RealPref-Bench contains 1,638 real-world editing cases with approximately 200 instances per sub-task for preference-aligned evaluation.Its manually verified instructions and balanced MS-COCO object coverage align the benchmark with the training dataset.

5. Experiments

Experiments evaluate HP-Edit on RealPref-Bench, ablate its data and scorer components, test transfer to GEdit-Bench-EN, and compare HP-Scorer with user judgments. HP-Edit improves editing quality and preference alignment across diverse real-world tasks.

  • Evaluation: HP-Edit is evaluated against multiple editing models on RealPref-Bench using HP-Score computed on a 0–5 scale.The comparison includes Step1X-Edit, BAGEL, X2Edit, UniWorld-V1, OmniGen2, QwenImage-Edit, FLUX.1-Kontext-Dev, and Qwen-Image-Edit-2509.
  • Qualitative Analysis: Qualitatively, HP-Edit better follows instructions, preserves scene structure, and produces more realistic outputs with fewer artifacts than baseline methods.Baselines show distortions in challenging removal and background-replacement edits, while FLUX.1-Kontext-Dev sometimes produces painted or stylized results.
  • Quantitative Analysis: HP-Edit achieves an overall score of 4.667 versus 4.472 for QwenImage-Edit-2509 and ranks first across all eight editing categories.It also achieves state-of-the-art performance on GEdit-Bench, outperforming Step1X-Edit and other comparison methods.
  • Quantitative Analysis: The largest gains occur on realism-sensitive tasks including color change, bokeh, relighting, and background replacement.These tasks involve subtle semantic reasoning or complex visual adjustments where pretrained models often struggle.
  • Ablation Study: RealPref-50K filtering and HP-Scorer refinement jointly improve reward learning, with the full setting showing the most stable upward reward trajectory.The ablation compares BaseData + BaseScorer, RealPref-50K + BaseScorer, and RealPref-50K + HP-Scorer.
  • Ablation Study: The ablation scores rise from 4.391 with BaseData + BaseScorer to 4.577 with RealPref-50K + BaseScorer and 4.667 with the full framework.The unfiltered setting underperforms the pretrained baseline score of 4.472, whereas the full setting achieves the highest score.
  • User Study: A user study over more than 1k editing pairs finds that HP-Scorer’s score distribution closely matches human scores and that HP-Edit improves consistently across tasks.Five annotators rated instruction adherence and image quality on RealPref-Bench.

6. Conclusion and Limitation

The paper introduces HP-Edit, RealPref-50K, and RealPref-Bench for human-preference-aligned image editing. It also reports a limitation in mixed Chinese-English text editing inherited from base models.

  • HP-Edit is a post-training framework for human-preference-aligned editing.
  • RealPref-50K uses HP-Scorer for scalable hard-case filtering and contains high-quality scorer-generated pseudolabels.
  • HP-Edit still struggles with code-switching or mixed Chinese-English text editing, largely because of limitations inherited from base models.

Supplementary Material

The supplementary material expands the paper’s experimental, prompt, quantitative, qualitative, and dataset documentation.

  • S1. Experimental details: Section S1 provides additional experimental details for the main paper.
  • S2. System prompts: Section S2 documents the system prompts used by HP-Scorer for each task.
  • Additional material: Sections S3–S5 provide further quantitative comparisons, visual examples, and details of RealPref-50K and RealPref-Bench.

S1. Experimental details

The supplementary experimental details correct annotation mistakes in Figure 4 and provide complete instructions for its eight cases. The cases cover multiple editing operations, including addition, replacement, removal, recoloring, relighting, blurring, and style transfer.

  • Experimental details: Figure 4 contains annotation mistakes, and corrected complete instructions are provided for cases 1–8.
  • Editing cases: The corrected cases include object addition, background replacement, object removal, and color changes.
  • Editing cases: The remaining cases cover object replacement, relighting, background blurring, and Japanese Ukiyo-e style transfer.

S2. System prompts of HP-scorer for each task

HP-Scorer’s evaluation behavior depends strongly on the task-specific system prompts designed for each editing task. The supplementary material presents these prompts across multiple figures.

  • Prompt dependence: HP-Scorer is highly dependent on its designed system prompts when evaluating editing tasks.
  • Prompt dependence: The prompts are task-specific rather than a single shared evaluation instruction.
  • Prompt documentation: The system prompts are documented in Figures S2–S9.

S3. More quantitative comparisons

Additional experiments report consistent improvements for HP-Edit across RealPref-Bench, GEdit-Bench-CN, and DreamBench++, while HP-Scorer correlates strongly with human ratings. Comparisons also examine LoRA rank and DPO, with rank 32 and GRPO-based training supported by the reported results.

  • Ablations: LoRA rank 32 provides the best reported setting: performance improves from rank 8 to 32, then remains unchanged or declines beyond 32.
  • HP-Scorer validation: 0.89 average Pearson correlation coefficient links human and HP-Scorer ratings on a held-out GEdit-Bench set.The ratings concentrate along the diagonal, supporting HP-Scorer as a reliable evaluator and reward signal for RL.
  • Benchmark comparisons: HP-Edit shows obvious improvement across metrics over Qwen-Image-Edit-2509 on GEdit-Bench-CN.
  • Benchmark comparisons: HP-Edit improves over Qwen-Image-Edit-2509 on DreamBench++ under traditional metrics, Concept Preservation, and Prompt Following.The comparisons are reported in Tables S3 and S4.
  • Training comparisons: DPO improves over the base model but remains worse than GRPO with HP-Scorer and HP-Edit on the same RealPref-Bench subset.DPO uses offline winner/loser mining, whereas GRPO performs online sampling with HP-Scorer feedback.

S4. More visual comparison

Supplementary figures provide additional HP-Edit image-editing results across the paper’s editing tasks.

  • Figures S10–S17 present additional image-editing results generated by HP-Edit.

S5. Details of RealPref-50k and RealPref-Bench

RealPref-50K and RealPref-Bench cover eight editing tasks using curated images, VLM-generated instructions, task-specific filtering, and evaluation prompts. The supplementary materials provide task examples, prompts, and qualitative comparisons for these settings.

  • Dataset overview: RealPref-50K and RealPref-Bench report statistics across eight editing tasks.
  • Dataset construction: Style-transfer data combine content images with more than 30 style categories and instructions that preserve the original image structure.The styles include classical categories such as Impressionism and inkwash painting, alongside contemporary aesthetics such as anime.
  • Dataset construction: Bokeh data use aligned bokeh–non-bokeh pairs, VLM-generated region-specific instructions, and COCO object-class balancing.
  • Dataset construction: Other tasks combine high-quality image sources, VLM-generated instructions, and task-specific filtering rules to promote diversity and realism.These tasks include object swapping, object removal, background replacement, attribute modification, relighting, and composition editing.
  • Supplementary materials: Supplementary figures document system prompts and qualitative comparisons for object removal, object adding, object swapping, background replacement, bokeh, relighting, style changing, and color changing.
  • Evaluation prompts: The evaluation prompts assess whether edited images satisfy task requirements while remaining natural, realistic, and free of visible artifacts.Examples check foreground preservation for background replacement, non-blurred content for bokeh, realistic relighting, and localized color changes.
Loading 2604.19406v1…