Source-linked AI summary
HIVE: Harnessing Human Feedback for Instructional Visual Editing
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, Ran Xu
TL;DR
Instructional image editing can fail to reflect users’ instructions and preferences, while conventional on-policy human-feedback fine-tuning is difficult for diffusion models. HIVE collects human rankings, learns rewards, and incorporates them through scalable diffusion fine-tuning, with experiments showing preference over prior instructional editing methods.
Problem
Instructional image editing outputs may not accurately follow users’ instructions and selective editing preferences, while on-policy human-feedback fine-tuning remains challenging for diffusion models.
Method
HIVE trains a baseline editor, learns a reward model from human rankings of variant outputs, and fine-tunes diffusion models using reward-weighted or reward-conditioned objectives.
Results
HIVE’s two human-feedback fine-tuning variants improve performance over previous state-of-the-art instructional image editing methods.
Takeaways & Limitations
Human feedback is a viable way to align instructional image generation with human preferences and support scaling toward applications such as conversational image editing.
Takeaways & Limitations
Applying on-policy reinforcement learning to diffusion fine-tuning can be prohibitively expensive and makes gradient backpropagation to the U-Net challenging.
Abstract
from arXiv · showhide
Incorporating human feedback has been shown to be crucial to align text generated by large language models to human preferences. We hypothesize that state-of-the-art instructional image editing models, where outputs are generated based on an input image and an editing instruction, could similarly benefit from human feedback, as their outputs may not adhere to the correct instructions and preferences of users. In this paper, we present a novel framework to harness human feedback for instructional visual editing (HIVE). Specifically, we collect human feedback on the edited images and learn a reward function to capture the underlying user preferences. We then introduce scalable diffusion model fine-tuning methods that can incorporate human preferences based on the estimated reward. Besides, to mitigate the bias brought by the limitation of data, we contribute a new 1M training dataset, a 3.6K reward dataset for rewards learning, and a 1K evaluation dataset to boost the performance of instructional image editing. We conduct extensive empirical experiments quantitatively and qualitatively, showing that HIVE is favored over previous state-of-the-art instructional image editing approaches by a large margin.
1. Introduction
Instructional image editing enables human-readable edits but can miss users’ intended changes and preferences. HIVE addresses this gap by collecting human rankings, learning rewards, and integrating them into diffusion-model fine-tuning.
- Instructional image editing requires only human-readable instructions, unlike traditional editing, which uses both an input image and edited caption.
- InstructPix2Pix can produce edits that only partially align with instructions because its training-data generation lacks explicit image-instruction alignment.
- HIVE uses three stages: baseline supervised training, human ranking of variant outputs with reward-model learning, and reward-integrated diffusion fine-tuning.
- HIVE introduces two scalable human-feedback fine-tuning approaches designed to be computationally efficient and comparable in cost to supervised fine-tuning.
- The project contributes a 1.1M training dataset, a 3.6K reward dataset, a 1K evaluation dataset, and cycle-consistency augmentation for bidirectional editing.
2. Related Work
Related work spans diffusion-based text-to-image generation, human-feedback alignment, and concurrent preference-learning methods. The paper emphasizes that standard on-policy reinforcement learning is difficult to apply to diffusion fine-tuning.
- Recent diffusion models achieve state-of-the-art text-to-image generation, motivating fine-tuning rather than training a new model from scratch.
- Human feedback has improved models across language, robotics, and computer vision, but PPO-based on-policy alignment is unsuitable for diffusion-model fine-tuning.
- Concurrent methods align text-to-image generation through late-step reward backpropagation, fine-grained preference scores, or reward-weighted maximum-likelihood training.
3. Methodology
HIVE combines instructional supervised training, human-preference reward learning, and scalable diffusion fine-tuning to align edited images with user preferences.
- 3. Methodology: HIVE has three stages: instructional supervised training, reward-model learning from ranked outputs, and diffusion fine-tuning using estimated rewards.The framework first trains on instructional editing data, then learns a reward function from human rankings, and finally incorporates rewards during fine-tuning.
- 3.2. Instructional Supervised Training: The instructional training data combines a newly collected 1.1M dataset with InstructPix2Pix data and uses a pre-trained stable diffusion backbone.The training setup conditions diffusion on the original image and editing instruction; cycle consistency is an additional augmentation.
- 3.2.1 Cycle Consistency Augmentation: Cycle consistency augments single-direction editing with instruction reversion, forming a forward edit and reverse mapping loop.The reverse instruction can invert an edit, such as changing “add a dog” to “remove the dog”; invertible and non-invertible instructions are separated.
- 3.3. Human Feedback Reward Learning: The reward model scores an edited image from the original image, editing instruction, and edited image, using a multimodal vision-language encoder.Human annotators rank multiple edited outputs for each image-instruction condition, and the rankings train the reward function with pairwise comparisons.
- 3.4. Human Feedback based Model Fine-tuning: Reward maximization produces an instructional diffusion model that aligns with human preferences.The reward is integrated into diffusion-model fine-tuning rather than optimized through sampling-based on-policy reinforcement learning.
- 3.4. Human Feedback based Model Fine-tuning: HIVE offers two scalable reward-based fine-tuning methods: exponential reward weighting and conditioning diffusion on reward labels.The weighted method targets an exponential reward-weighted data distribution, while the condition-reward method treats preference as an additional condition; offline-RL regularization stabilizes reward maximization.
4. Experiments
HIVE is evaluated against InstructPix2Pix baselines using synthetic tradeoffs, real-image user studies, qualitative comparisons, and ablations. Across these evaluations, human-feedback training improves instruction alignment, image consistency, and preservation of uninstructed content, while cycle consistency and reward-loss variants are also examined.
- Ablation Study: The conditional reward loss performs slightly better than the weighted reward loss, although user preferences between the two losses are close.Cycle consistency augmentation also improves HIVE performance, while directly conditioning on the highest reward score receives 25.8% fewer votes than HIVE.
- Baseline Comparisons: HIVE improves image consistency over InstructPix2Pix at the same directional similarity value.On the synthetic evaluation dataset, IP2P-Ours improves over IP2P-Official, while HIVE further improves over IP2P-Ours.
- Baseline Comparisons: Around 25% more users favor HIVE than IP2P-Ours on the 1K real-image evaluation dataset.IP2P-Ours also receives around 30% more votes than IP2P-Official.
- Baseline Comparisons: HIVE produces edits that better align with instructions, including a door matching human perception and correctly recoloring a tie.The reward model improves alignment beyond the gains from using more training data alone.
- Baseline Comparisons: HIVE tends to preserve uninstructed image regions, whereas IP2P-Ours more often makes excessive modifications.A qualitative example shows HIVE naturally blending a pond into the original image while InstructPix2Pix alters uninstructed background content.
- Ablation Study: HIVE improves the success rate for the verb “add” from 23.5% to 28.7% compared with IP2P-Ours.The comparison evaluates the top five verbs.
5. Conclusion and Discussion
HIVE integrates human feedback as reward values into diffusion-model fine-tuning for instructional image editing. Its variants improve performance over previous state-of-the-art methods, while failure scenarios and inherited foundation-model risks remain.
- HIVE integrates human feedback, quantified as reward values, into the diffusion model fine-tuning process.
- Both HIVE variants improve performance over previous state-of-the-art instructional image editing methods.
- The authors identify failure scenarios and possible inherited bias or harmful content from pretrained foundation models.
- The paper expects future red teaming with human feedback to mitigate some of these risks.
A. Data Collection and User Study
The evaluation data and user studies use Amazon Mechanical Turk to collect diverse editing instructions and human preferences among edited images.
- The evaluation collects 200 real-world images and asks annotators to write five diverse instructions for each image.
- Annotators are encouraged to imagine varied edits, including unrealistic changes such as adding a horse in the sky.
- User studies present the original image, an instruction, and two edited images for annotators to select the better result.
- Three annotators label each sample, with majority votes determining the study result.
B. Implementation Details
HIVE combines reward-model training, reward-based diffusion fine-tuning, and evaluation infrastructure, while discussing the computational limits of direct sampling-based policy-gradient methods.
- Reward model: The reward model takes an input image, text instruction, and edited image, then outputs a scalar reward value.
- Reward model: The model uses a shared visual transformer for input and edited images, followed by an image-grounded text encoder and linear scalar head.
- Reward data: 3,634 ranking images each receive five edited variants, which annotators rank from best to worst while flagging nonrankable cases.
- Reward data: Only data without nonrankable options trains the reward model; 1,285 images are used for training and the remainder for validation.
- Reward-based fine-tuning: Exponential rewards weight diffusion fine-tuning in one experiment, while another converts rewards into text prompts for conditional reward loss.
- Sampling-based methods: The paper considers scalable sampling-based methods future work rather than concluding that sampling-based approaches are infeasible.
- Sampling-based methods: Direct sampling-based policy-gradient updates require the whole reverse diffusion chain, making their cost proportional to the number of diffusion steps.
C.2. Derivation for Eq.(2)
The derivation differentiates the objective with respect to the policy representation and uses the resulting expression to obtain an optimal conditional distribution.
- The derivation differentiates J(ρ) with respect to ρ to characterize the objective’s optimization.
- The resulting analysis yields the optimal conditional distribution ρ∗(˜x|c).
D. Additional Ablation Study
The ablations examine backbone choice, reward-model transfer, and reward-loss design. HIVE remains effective across backbones, while the conditional reward loss is slightly better than the weighted alternative.
- D.1. Backbone Analysis: The SD v2.1 ablation compares HIVE and IP2P-Ours with SD v1.5 and v2.1 on synthetic consistency plots.The reported comparison appears in Figure 17.
- D.2. Model Adaptation: The reward-model backbone comparison is reported in Figure 18.The figure contrasts SD v1.5-trained and SD v2.1-trained reward models.
- D.2. Model Adaptation: HIVE adapts a reward model trained with SD v1.5 to fine-tuning with SD v2.1.Users vote similarly for reward models trained on the two SD backbones.
- D.3. Reward Loss: Weighted and conditional reward losses have close performance, with conditional reward loss slightly better.The paper adopts conditional reward loss for all experiments.
D.4. Training with Less Data
The study evaluates HIVE under reduced training-data budgets. Moderate reduction has limited impact, whereas using only 10% substantially harms large image edits.
- D.4. Training with Less Data: Using 10% of the training data leads to worse ability to perform large image edits.The analysis compares 100%, 50%, 30%, and 10% data ratios.
- D.4. Training with Less Data: Figure 20 presents HIVE results for four training-dataset size ratios: 100%, 50%, 30%, and 10%.The figure is the basis for the reduced-data analysis.
- D.5. Subcategory Analysis: Figure 21 provides a subcategory analysis comparing IP2P and HIVE.The subcategory comparison is separate from the dataset-size ablation.
D.5. Subcategory Analysis
The paper analyzes instructional editing across several semantic categories. The category counts show that add/remove-object instructions dominate the evaluation grouping.
- D.5. Subcategory Analysis: The analysis covers global-style changes, main-object attribute adjustments, object addition or removal, object manipulation, and challenging cases.Challenging cases include zooming and camera-view changes.
- D.5. Subcategory Analysis: The add/remove-objects category contains 508 instructions, more than the global-style and main-object-attribute categories.Those categories contain 133 and 134 instructions, respectively.
- D.5. Subcategory Analysis: ChatGPT is used to assign each instruction to an editing subcategory.The reported category counts are based on this classification procedure.
D.6. Additional Visualized Results
Additional visualizations show instructional editing examples across varied object, attribute, style, and scene transformations. The examples are arranged as rows of three editing cases.
- D.6. Additional Visualized Results: Figures 22–26 present additional visualized results, with each row showing three instructional editing examples.The examples supplement the main qualitative comparisons.
- D.6. Additional Visualized Results: The examples include adding bisons, changing cloth color, and applying Japanese style while adding sunglasses.These instructions illustrate object insertion, attribute editing, and style-related edits.
- D.6. Additional Visualized Results: Other examples change a lighthouse into the Eiffel Tower and add boats to a scene.These cases involve object transformation and scene modification.