Source-linked AI summary
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
Sihao Wu, Xiaonan Si, Chi Xing, Jianhong Wang, Gaojie Jin, Guangliang Cheng, Lijun Zhang, Xiaowei Huang
TL;DR
Images generated or edited with diffusion models can misalign with text, depart from human aesthetic preferences, or include toxic or biased content, and comprehensive reviews of preference alignment remain limited. This survey synthesizes alignment methods and applications and identifies continuing challenges. It concludes that computational efficiency and preference modeling remain open issues for future work.
Problem
Diffusion-model image generation and editing face text-image misalignment, aesthetic deviations, and possible toxic or biased content, while comprehensive reviews of preference alignment remain scarce.
Source: §1 · p. 1Method
The paper surveys preference-alignment optimization techniques, applications in image generation and editing, and associated challenges.
Source: §1 · p. 1Results
The survey synthesizes preference-alignment methods and applications, while identifying computational efficiency and preference modeling as remaining challenges.
Source: §6 · p. 7Takeaways & Limitations
The paper points future work toward scalable algorithms, multimodal capabilities, and ethical considerations.
Source: §6 · p. 7Preference fine-tuning faces high computational demands, costly and time-consuming human-preference data collection, and difficulty balancing diverse objectives and modeling subjective preferences.
Source: §5 · p. 7
Abstract
from arXiv · showhide
The integration of preference alignment with diffusion models (DMs) has emerged as a transformative approach to enhance image generation and editing capabilities. Although integrating diffusion models with preference alignment strategies poses significant challenges for novices at this intersection, comprehensive and systematic reviews of this subject are still notably lacking. To bridge this gap, this paper extensively surveys preference alignment with diffusion models in image generation and editing. First, we systematically review cutting-edge optimization techniques such as reinforcement learning with human feedback (RLHF), direct preference optimization (DPO), and others, highlighting their pivotal role in aligning preferences with DMs. Then, we thoroughly explore the applications of aligning preferences with DMs in autonomous driving, medical imaging, robotics, and more. Finally, we comprehensively discuss the challenges of preference alignment with DMs. To our knowledge, this is the first survey centered on preference alignment with DMs, providing insights to drive future innovation in this dynamic area.
1 Introduction
Images generated or edited with diffusion models still face alignment, aesthetic, and safety-related problems, while existing reviews leave preference alignment in image generation and editing incompletely covered. This survey reviews optimization techniques, applications, and challenges in that area.
1 Introduction
- Diffusion-generated images can misalign with text, depart from human aesthetic preferences, or propagate toxic or biased content.These are the problems motivating preference alignment.Source: §1 · p. 1
- Existing surveys rarely comprehensively review how diffusion models integrate with reinforcement learning for image generation and editing.Earlier reviews tend to focus on RL applied to diffusion processes or broad generative-AI applications.Source: §1 · p. 1
- The survey reviews optimization techniques including RLHF and DPO, explores applications such as autonomous driving, medical imaging, and robotics, and discusses challenges.It presents itself as the first survey specifically focused on preference alignment with diffusion models in image generation and editing.Source: §1 · p. 1
2 Preliminaries
The preliminaries describe forward and reverse diffusion, including DDIM as a deterministic alternative. They then outline preference fine-tuning through supervised training and PPO-based reinforcement learning.
2.1 Diffusion Model
- The forward diffusion process adds Gaussian noise over T time steps as a Markov chain, with noise schedules determining how quickly the data degrades.The distribution at any step can also be computed directly.Source: §2.1 · p. 2
- The reverse process starts from pure noise and iteratively denoises to reconstruct x0; training minimizes a variational lower bound on negative log-likelihood.A learnable model performs the denoising.Source: §2.1 · p. 2
- DDIM replaces the Markov chain with a more efficient, deterministic non-Markovian process that supports faster generation, accurate reconstruction, and image and video editing.Source: §2.1 · p. 2
2.2 Reinforcement Learning
- Preference fine-tuning begins with supervised training to map prompts to images, then commonly uses PPO for reinforcement learning because of its stability and robustness.Source: §2.2 · p. 2
- PPO optimizes a clipped surrogate using a policy-probability ratio and GAE-estimated advantage, with epsilon controlling the clipping interval.Source: §2.2 · p. 2
- PPO alternates trajectory collection and optimization, balancing value-function loss with an entropy bonus through weighting coefficients.The overall objective combines the PPO losses and entropy term.Source: §2.2 · p. 2
3 Preference Alignment on DMs
The section organizes preference alignment for diffusion models by sampling, training strategy, feedback source, and modality, then surveys RLHF, DPO, and other approaches.
3.1 Framework
- The framework classifies methods by online or offline sampling, RL/DPO/other training, human or AI feedback, and textual, visual, or multimodal optimization.Online methods may be on-policy or off-policy; human and AI annotation can be combined in hybrid datasets.Source: §3.1 · pp. 2, 4
3.2 RLHF
- RLHF learns rewards from human comparisons before policy optimization; DDPO [Black et al., 2023] maximizes expected reward, while DPOK regularizes it.Other reviewed approaches use AI feedback for region correction or text-image evaluation, optimize prompts and text encoders, or backpropagate reward gradients directly.Source: §3.2 · pp. 4–5
3.3 DPO and its Variants
- DPO bypasses reward modeling and RL; Diffusion-DPO [Wallace and et al., 2024] adapts its objective, D3PO [Yang and et al., 2024] reduces overhead, and Diffusion-KTO [Li et al., 2024a] uses per-image binary feedback.Other variants use reward-weighted likelihoods, temporal discounting, or regression on reward differences between candidate images.Source: §3.3 · p. 5
3.4 Others
- GFlowNet approaches map denoising transitions to forward and backward policies; RLCM frames inference as an MDP, while SPIN-Diffusion removes the need for human preference data.The section suggests these approaches when reward functions are unknown, as generating high-reward samples beyond training data may be challenging.Source: §3.4 · pp. 5–6
4 Applications
The applications section surveys preference-aligned diffusion models across medical imaging, robotics, autonomous driving, and other domains, with domain-specific fine-tuning often integrated into denoising.
4 Applications
- Across application domains, fine-tuning modules are commonly integrated into the denoising process and conditioned on domain-specific requirements.The section frames preference alignment as useful across medical imaging, robotics, and autonomous driving.Source: §4 · p. 6
4.1 Medical Imaging
- Medical imaging work covers segmentation, privacy-conscious synthetic image generation, RLHF-based generation, and reconstruction.Methods include feature-map optimization, dual conditioning, stochastic mask sampling, and differential-privacy fine-tuning.Source: §4.1 · p. 6
4.2 Robotics
- In robotics, offline diffusion learning with online preference alignment improves quadrupedal locomotion, while diffusion models also support trajectory planning, policy learning, and world simulation.Preference-conditioned models are used for multi-task decisions, and fine-tuned models are applied to soft-robot design and control.Source: §4.2 · p. 6
4.3 Autonomous Driving
- Autonomous-driving applications use RLHF, SFT, and DPO to synthesize training data, and use preference optimization to address uncertainty in decision-making.Related work also targets faster denoising-based decisions, security, and diffusion-based world models.Source: §4.3 · pp. 6–7
4.4 Others
- Beyond the main domains, the survey describes diffusion models for protein design, game simulation, and preference-aligned creative content.The cited examples span protein stability and functionality, virtual worlds, and digital art and music.Source: §4.4 · p. 7
5 Challenges and Future Directions
The section identifies computational, data, preference-modeling, and safety challenges, and outlines research directions for more efficient, multimodal, interpretable, and deployable systems.
Challenges
- Preference fine-tuning faces high computational costs, expensive and time-consuming human feedback collection, difficulty balancing multiple objectives, and subjective preferences that complicate reward design.RLHF is particularly resource-intensive for both training and inference.Source: §5 · p. 7
Future Directions
- Future work should reduce fine-tuning compute and feedback needs, improve text-image integration, develop robust real-world evaluation, and make fine-tuning more interpretable and controllable.The section names parameter-efficient approaches such as LoRA and LLMs for interpreting human preferences as potential directions.Source: §5 · p. 7
Safety and Ethics
- As models become more powerful and widely deployed, the paper calls for safeguards addressing alignment evaluation, misuse, and privacy and security during fine-tuning.It presents these issues as milestones for responsible advancement.Source: §5 · p. 7
6 Conclusion
The conclusion synthesizes preference-alignment methods and applications, while identifying efficiency, preference modeling, scalability, multimodality, and ethics as continuing concerns.
6 Conclusion
- The survey reviews RLHF, DPO, and other preference-alignment methods, categorizes approaches and applications, and concludes that efficiency and preference modeling remain challenges.It calls for scalable algorithms, multimodal capabilities, and ethical considerations in future work.Source: §6 · p. 7