Source-linked AI summary
Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning
Chubin Chen, Sujie Hu, Jiashu Zhu, Meiqi Wu, Jintao Chen, Yanxun Li, Nisha Huang, Chengyu Fang, Jiahong Wu, Xiangxiang Chu, Xiu Li
TL;DR
The paper addresses Preference Mode Collapse, where reward optimization produces narrow high-scoring outputs and degraded diversity. It introduces DivGenBench to measure this problem and D2-Align to directionally correct biased rewards. The method achieves state-of-the-art results while improving human preference scores and generative diversity.
Problem
Preference Mode Collapse is a diversity-oriented form of reward hacking in which preference alignment drives models toward homogeneous outputs despite high reward scores.
Method
D2-Align learns a directional correction in reward-model embedding space with the generator frozen, then applies it during generator optimization.
Results
D2-Align achieves state-of-the-art performance while simultaneously improving human preference scores and generative diversity.
Takeaways & Limitations
DivGenBench provides a diversity-centric evaluation, while D2-Align breaks the usual fidelity–diversity trade-off within the reported experiments.
Takeaways & Limitations
The approach assumes reward-model intrinsic biases are a primary cause of collapse, while standardized diversity metrics remain difficult and computationally expensive.
Abstract
from arXiv · showhide
Recent studies have demonstrated significant progress in aligning text-to-image diffusion models with human preference via Reinforcement Learning from Human Feedback. However, while existing methods achieve high scores on automated reward metrics, they often lead to Preference Mode Collapse (PMC)-a specific form of reward hacking where models converge on narrow, high-scoring outputs (e.g., images with monolithic styles or pervasive overexposure), severely degrading generative diversity. In this work, we introduce and quantify this phenomenon, proposing DivGenBench, a novel benchmark designed to measure the extent of PMC. We posit that this collapse is driven by over-optimization along the reward model's inherent biases. Building on this analysis, we propose Directional Decoupling Alignment (D$^2$-Align), a novel framework that mitigates PMC by directionally correcting the reward signal. Specifically, our method first learns a directional correction within the reward model's embedding space while keeping the model frozen. This correction is then applied to the reward signal during the optimization process, preventing the model from collapsing into specific modes and thereby maintaining diversity. Our comprehensive evaluation, combining qualitative analysis with quantitative metrics for both quality and diversity, reveals that D$^2$-Align achieves superior alignment with human preference.
1. Introduction
The paper identifies Preference Mode Collapse as diversity-oriented reward hacking in text-to-image alignment and introduces DivGenBench and D2-Align to measure and mitigate it.
- Preference Mode Collapse causes aligned models to produce homogeneous images with monolithic styles, recurring visual features, or pervasive overexposure despite high reward scores.
- DivGenBench quantitatively measures the diversity loss that existing reward-hacking evaluations largely overlook.
- Existing approaches mainly modulate reward magnitude, while D2-Align corrects the reward direction in embedding space with the generator frozen before optimization.
- Experiments and human evaluations show D2-Align significantly outperforms existing approaches on generation quality and diversity.
2. Related Work
Prior diffusion-alignment methods improve reward scores through policy-gradient or differentiable-reward optimization, but reward hacking remains a central challenge.
- Diffusion reinforcement-learning methods use either policy gradients or direct fine-tuning with differentiable rewards to align text-to-image models.
- D2-Align overview: Figure 2 presents D2-Align as seeking an optimum that balances quality and diversity instead of converging to a narrow peak.
- Higher reward scores can accompany reward hacking, in which models optimize superficial cues that misalign with human preference.
- DanceGRPO combines reward models, Pref-GRPO attributes failures to reward-function imperfections, and MixGRPO uses hybrid inference to mitigate quality problems.
3. Preliminary
Diffusion models reverse a forward noising process, while alignment must evaluate clean images even though optimization operates on noisy latents.
- Diffusion generation reverses a forward process that transforms clean data into noisy states and integrates a learned flow field backward from the prior.
- Reward evaluation is difficult because optimization uses noisy latents whereas reward models require clean images, and one-step predictions can be inaccurate at high noise.
- Ground-truth noise constructs xt = αtx0 + σtϵgt, enabling high-fidelity differentiable reconstruction of x0 for reward evaluation.
- The reconstructed image provides a stable reward signal across timesteps, allowing uniform sampling of t from [0, 1].
4. Methodology
The methodology frames diversity loss during reward optimization as Preference Mode Collapse, then combines directional reward correction with DivGenBench’s diversity evaluation.
- 4.1. Preference Mode Collapse: Preference Mode Collapse is reward hacking from a diversity perspective, producing homogeneous outputs despite high reward scores.
- 4.1. Preference Mode Collapse: The paper attributes collapse to reward-model intrinsic preferences that optimization overfits, while noting that standardized diversity metrics are difficult and costly.
- 4.1. Preference Mode Collapse: Prompt perturbation can counteract reward-model style bias by producing a reward signal more aligned with human preference.
- 4.2. Directional Decoupling Alignment: D2-Align’s two stages learn a reward-correction direction with the generator frozen and then use it to guide generator updates away from specific modes.
- 4.2. Directional Decoupling Alignment: Figure 5 reports that D2-Align reaches higher scores in fewer steps, while DanceGRPO and Flow-GRPO require over 250 steps for similar performance.
- 5. Evaluation: Qualitative comparisons on HPDv2 and DivGenBench show superior fidelity, text-to-image alignment, and diversity relative to baselines.
- 4.3. DivGenBench: DivGenBench probes generative breadth across identity, style, layout, and tonal dimensions using keyword-driven prompts and four customized metrics.
5. Experiment
D2-Align is evaluated on FLUX.1.Dev using quality, diversity, and human-preference assessments. It consistently preserves fidelity and diversity while outperforming alignment baselines, with ablations supporting the learned directional correction and its robustness.
- D2-Align consistently outperforms baselines in fidelity, text-to-image alignment, and diversity on HPDv2 and DivGenBench.Qualitative comparisons show stronger concept preservation and fewer collapse artifacts.
- D2-Align achieves superior quality, text-to-image alignment, and semantic consistency while maintaining high fidelity.
- D2-Align achieves the best user-study win rate as a measure of alignment with human preference.
- Ablation Study: The learned directional vector becomes robust after approximately 2000 training steps, after which it yields significant performance improvements.
- Ablation Study: Setting the guidance scale ω to 1.5 yields superior results on both HPS-v2.1 and PickScore.
- Ablation Study: The learned vector consistently outperforms manually selected fidelity and aesthetic words and improves performance over the uncorrected reward signal across evaluated metrics.
- Ablation Study: Applying the learned vector to other PMC-prone alignment frameworks mitigates collapse while improving the balance between fidelity and diversity.
6. Conclusion
The paper identifies Preference Mode Collapse as diversity-oriented reward hacking and introduces D2-Align and DivGenBench to address and measure it. D2-Align improves human-preference scores and generative diversity simultaneously.
- D2-Align achieves state-of-the-art performance while simultaneously improving human-preference scores and generative diversity.
- DivGenBench measures Preference Mode Collapse, which the paper characterizes as reward hacking from a diversity perspective.
Supplementary Material
D2-Align separates reward-signal correction from policy alignment in two stages. It learns a directional correction with the generator frozen, then freezes that correction while optimizing the policy with the guided reward.
- D2-Align uses a two-stage process that decouples reward-signal correction from policy alignment.
- Stage 1: Directional Correction: Stage 1 freezes the policy model and optimizes the directional vector to create a guided reward that corrects reward-model biases.
- Stage 2: Guided Alignment: Stage 2 freezes the learned vector, unfreezes the policy, and updates generator parameters using the guided reward signal.
- Stage 2: Guided Alignment: The guided reward is computed by modifying text embeddings with positive and negative directional corrections before scoring the reconstructed image.
- The implementation uses FLUX.1.Dev, retrains baselines on HPD v2, and evaluates standardized HPS-v2.1 and HPS-v2.1-plus-CLIP reward combinations.
B.1. Baseline Implementation Details
The experiments standardize baseline training and reward functions for comparison with D2-Align. The supplementary setup specifies baseline schedules, D2-Align’s two-stage schedule, and keyword-driven prompt dimensions for diversity evaluation.
- Baseline comparisons use standardized training steps and aligned reward functions while following official implementations.
- DanceGRPO and Flow-GRPO are trained for 300 steps, whereas SRPO uses its official 20-step recommendation.
- D2-Align trains the correction vector for 3000 steps and aligns the policy for 20 steps in its two-stage schedule.
- The 3,200 keyword-driven prompts cover identity, style, layout, and tonal dimensions using templated attribute combinations.
C.2. Metric Calculation Details
DivGenBench measures generative breadth through dimension-specific metrics for identity, style, layout, and tonal diversity, alongside human preference evaluation across quality and diversity dimensions.
- Identity: Identity Divergence Score measures average pairwise cosine similarity among ArcFace identity embeddings, with lower values indicating greater identity diversity.Each generated face is represented by a 512-D identity embedding.
- Style: Artistic Style Coverage estimates synthetic style diversity relative to real WikiArt reference diversity, with higher adjusted scores indicating better coverage.The metric uses retrieval-based estimation and adjusts for the feature extractor’s measurement gap.
- Layout: Spatial Dispersion Index measures variation in object layouts across images sharing a prompt, with higher scores indicating greater layout diversity.Detected bounding boxes are matched using the Hungarian algorithm before computing average layout similarity.
- Tonal: Photographic Variance Score summarizes the spread of saturation, brightness, and contrast values across generated images.The component sets are formed from HSV-derived saturation and brightness plus grayscale contrast, and their standard deviations are summed.
- Human Evaluation: The evaluation compares D2-Align with FLUX, DanceGRPO, Flow-GRPO, and SRPO using human judgments of detail, color, image-text alignment, overall preference, and diversity.The study uses 100 HPDv2 prompts, shared random seeds, and 20 randomized blind evaluators; DivGenBench covers Identity, Style, Layout, and Tonal dimensions.
D.1.2. Analysis of Results
Human evaluations favor D2-Align over the base model and RL baselines on image quality and alignment, while DivGenBench directly tests whether alignment preserves diversity.
- HPDv2 Human Preference: D2-Align received 61.7% for Detail Preservation versus 16.5% for runner-up SRPO, and 52.2% for Image-Text Alignment.It also led Color Consistency with 29.0% of votes.
- HPDv2 Human Preference: 48.2% overall preference for D2-Align confirms its comprehensive lead in the HPDv2 human evaluation.D2-Align was preferred across the evaluated quality dimensions.
- DivGenBench Human Preference: The second user study uses 80 DivGenBench evaluation sets across Identity, Style, Layout, and Tonal dimensions to assess repetitive or homogeneous outputs.Twenty templates were sampled from each dimension, and evaluators selected the model best reflecting requested diversity.
D.2.2. Analysis of Results
D2-Align preserves diversity while improving preference alignment, and its learned corrective signal also strengthens DanceGRPO robustness without complex retraining changes.
- D.2.2. Analysis of Results: Existing RL baselines exhibit Preference Mode Collapse, with Flow-GRPO reaching 7.7% tonal and 8.6% style diversity preference versus FLUX’s 26.7% and 18.2%.The results associate naive reward optimization with narrow outputs such as overexposed or uniformly realistic styles.
- D.2.2. Analysis of Results: 33.7% tonal-diversity preference and 35.2% identity and 37.3% style preference show D2-Align leading across collapse-prone dimensions.It outperforms the approximately 23% runner-up on Identity and Style while maintaining robustness where baselines fail.
- D.2.2. Analysis of Results: Plugging the frozen Stage 1 correction into DanceGRPO changes only reward calculation while retaining its original training logic and hyperparameters.The guided reward uses the rectified text embedding constructed from the learned parameter, with guidance scale kept consistent.
- D.2.2. Analysis of Results: The evaluation compares FLUX, DanceGRPO, and corrected DanceGRPO on alignment, semantic consistency, and DivGenBench diversity metrics.Tables 6 and 7 cover human-preference and semantic-consistency metrics alongside IDS, ASC, SDI, and PVS.
- D.2.2. Analysis of Results: 20.1% lower IDS and 57.7% higher ASC show that the corrective signal expands DanceGRPO’s diversity beyond both its baseline and pretrained FLUX.The method filters out the low-diversity manifold and broadens the optimizer’s explored solution space.
E.3. Results on DivGenBench
On DivGenBench, D2-Align avoids the narrow identities, styles, layouts, and tonal distributions seen in competing methods while maintaining prompt-relevant image variation.
- E.3. Results on DivGenBench: The comparison targets two state-of-the-art methods that achieve high HPDv2 performance but the lowest DivGenBench diversity scores.This setup directly examines the quality–diversity trade-off identified in existing work.
- E.3. Results on DivGenBench: D2-Align generates distinct identities, broad artistic styles, diverse spatial layouts, and wide tonal spectra, contrasting with competitors’ monotonic distributions.The qualitative evaluation covers the Identity, Style, Layout, and Tonal dimensions in Figures 12–15.