PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization

Xinxin Liu, Ming Li, Zonglin Lyu, Yuzhang Shang, Chen Chen

arXiv:2604.24952v1cs.CVcs.AI

TL;DR

Holistic binary preference labels collapse multi-dimensional visual judgments into conflicting training signals for text-to-image alignment. Semi-DPO filters consensus pairs as clean data and iteratively pseudo-labels conflicting pairs, achieving state-of-the-art alignment with complex human preferences. The method requires no additional annotations or explicit reward models during training.

  • Problem

    Holistic binary labels compress multi-dimensional visual preferences into noisy, contradictory supervision for Diffusion-DPO.

  • Method

    Semi-DPO filters consistent pairs as clean data and uses timestep-conditional self-training to pseudo-label conflicting pairs.

  • Results

    Semi-DPO achieves state-of-the-art performance, consistently outperforming baselines across multi-reward, win-loss, object-focused, and compositional benchmarks.

  • Takeaways & Limitations

    Semi-DPO improves text-to-image alignment with complex human preferences without additional annotations or explicit reward models during training.

  • Takeaways & Limitations

    The multi-stage pseudo-labeling and retraining pipeline increases operational complexity despite not increasing computational costs.

Abstract

from arXiv · show

Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise: images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training. We will release our code and models at: https://github.com/L-CodingSpace/semi-dpo

1 INTRODUCTION

Holistic binary preference labels collapse multi-dimensional visual judgments, creating conflicting gradients that hinder Diffusion-DPO alignment. Semi-DPO addresses this as learning with noisy labels by filtering consensus-clean pairs and iteratively pseudo-labeling conflicted data with the diffusion model itself.

  • Motivation: Text-to-image diffusion alignment typically relies on separate reward models, creating significant computational bottlenecks, while Diffusion-DPO avoids that requirement.The passage introduces Diffusion-DPO as an adaptation of direct preference optimization for text-to-image diffusion models.
  • Motivation: Human judgments can favor different dimensions, such as semantic alignment, composition, aesthetics, or texture, yet datasets record only one binary preference label.The example contrasts Image A’s semantic alignment and composition with Image B’s texture and shows how forced annotation can hinge on a single dimension.
  • Problem: Collapsing multi-dimensional preferences into binary labels creates conflicting gradient signals across the dataset, leading to suboptimal Diffusion-DPO convergence.The paper formalizes this phenomenon theoretically and reframes it as learning with noisy labels.
  • Method: Semi-DPO treats dimensionally conflicted preference pairs as unlabeled data and applies iterative self-training after learning from a small clean set.This follows the learning-with-noisy-labels paradigm of training on clean data, generating pseudo-labels, and progressively refining the model.
  • Method: The diffusion model serves as an implicit preference classifier, generating fine-grained pseudo-labels for conflicted data without architectural changes.The DPO loss trains the model to distinguish preferred from dispreferred samples, enabling pseudo-label generation within the diffusion model.
  • Method: About 21% of Pick-a-Pic V2 is designated clean through unanimous agreement among diverse pretrained reward models, with the remainder assigned to a noisy unlabeled set.This is the first stage of Semi-DPO, called Multi-Reward Consensus; the second stage performs iterative self-training.

2 RELATED WORKS

Prior work established diffusion models for text-to-image generation but highlighted their limited alignment with human preferences. Related research also examined label noise in large-scale datasets and semi-supervised learning as a response to annotation errors.

  • Diffusion Models and Diffusion Alignment: Diffusion models have achieved notable success in text-to-image generation, yet they are not well-aligned with human preferences.These models are traditionally trained on large-scale text-image datasets scraped from the web.
  • Noise Data & Semi-Supervised Learning: Web scraping and crowdsourcing inevitably introduce label errors, which deep neural networks can memorize and which can degrade generalization.This motivates methods addressing noisy annotations, including semi-supervised learning.

3 METHOD

The method explains how Diffusion-DPO converts preference comparisons into timestep-wise policy updates, then shows that multi-dimensional label conflicts create unstable, opposing gradients. Semi-DPO addresses this by filtering consensus-clean pairs and iteratively pseudo-labeling conflicting pairs with timestep-conditional model predictions.

  • Direct Preference Optimization for Diffusion Models: Diffusion-DPO re-parameterizes the reward through policy and reference models, bypassing an explicit reward model while optimizing directly over diffusion denoising preferences.The loss averages over preference pairs and diffusion timesteps with noisy states sampled from the forward process.
  • Diffusion-DPO Gradient Formulation: The per-timestep Diffusion-DPO update follows the feature difference to increase the preferred sample’s log-probability relative to the dispreferred sample.This direction is effective for clean preference pairs but becomes problematic when preferences conflict across dimensions.
  • The Source of Conflicting Signals in Preference Optimization: A conflict set exists exactly when a dimensional preference opposes the holistic label, guaranteeing gradient updates that align with and oppose the oracle direction.The resulting inner-product measure directly quantifies conflicting-signal severity, while the lower-bound analysis ties instability to the coexistence of aligned and conflicting subsets.
  • Timestep-Conditional Pseudo-Labeling: The initial model generates timestep-conditional pseudo-labels for the noisy unlabeled set, accepting predictions according to logit confidence and using their sign as the learned preference.This mechanism re-labels conflicting pairs at the timestep level to address inflated gradient variance.
  • The Source of Conflicting Signals in Preference Optimization: These opposing signals cause parameter updates to oscillate, inefficiently cancel prior progress, and hinder consistent optimization toward convergence.The analysis identifies both learning inefficiency and suboptimal convergence as consequences of directional conflict.
  • Data Partitioning via Multi-Reward Consensus: Semi-DPO reframes alignment as semi-supervised learning, using multi-reward consensus to create a clean labeled subset for stable cold-start training.A pair enters the labeled set only when all reward models unanimously agree with its holistic label.

4 EXPERIMENTS

Experiments train Semi-DPO on Pick-a-Pic V2 with consensus-filtered preferences and evaluate it against released diffusion-alignment baselines using reward, generation, and multi-dimensional preference metrics. Results show improved text alignment and detail fidelity, while ablations support iterative self-training and larger consensus committees.

  • Datasets and Models: Semi-DPO is trained on 851,293 preference pairs spanning 58,960 prompts after excluding approximately 12% of ties from Pick-a-Pic V2.The experiments use SD1.5 and SDXL as base models.
  • Baselines and Evaluation: Comparisons use officially released checkpoints of SD1.5, SDXL, Diffusion-DPO, Diffusion-KTO, MaPO, and InPO.Evaluation on SD1.5 includes ImageReward, PickScore, HPS v2, LAION Aesthetics, CLIP Score, GenEval, and MPS.
  • Training Details: Consensus filtering with five proxy reward models produces 176,999 clean pairs, while the remainder is classified as noisy.The committee comprises PickScore, HPS v2, CLIP Score, the LAION Aesthetics Classifier, and ImageReward.
  • Qualitative Result: Qualitative comparisons show improved text alignment, detail fidelity, and aesthetics; Semi-DPO alone generates Pikachu wearing the specified chef’s hat.This example uses the prompt “a photo of Pikachu cooking at a restaurant, wearing a chef’s hat.”
  • Quantitative Result: Semi-DPO consistently outperforms baselines on SD1.5 and SDXL across multi-reward metrics and win-loss rates, with further advantages on GenEval and T2I-CompBench.GenEval evaluates object-focused generation, while T2I-CompBench evaluates compositional generation.
  • Iterations: Performance improves significantly from Iter0 to Iter1 and Iter2, but gains diminish and stabilize after the second iteration, making two self-training rounds sufficient.Iter0 uses only the clean dataset; Iter1 and Iter2 add successive rounds of pseudo-labeling and retraining.
  • Number of Reward Models for Consensus Filtering: Increasing the consensus committee from two to five reward models consistently improves all metrics and reduces individual-evaluator bias.Larger committees improve both filtering metrics and metrics not used for filtering.

HPS v2 · 5 CONCLUSION

The work shows that binary labels collapse multi-dimensional preferences into conflicting gradients that hinder Diffusion-DPO optimization. Semi-DPO addresses this through consensus-based clean-data selection, iterative noisy-label correction, and timestep-conditional pseudo-labels.

  • 5 CONCLUSION: Collapsing multi-dimensional preferences into binary labels generates conflicting gradients that hinder Diffusion-DPO optimization.
  • 5 CONCLUSION: Semi-DPO is proposed as a semi-supervised framework for resolving conflicting preference signals.
  • 5 CONCLUSION: Multi-Reward Consensus identifies clean data for the initial training subset.
  • 5 CONCLUSION: Iterative self-training corrects noisy labels without requiring additional annotation or explicit reward models during training.
  • 5 CONCLUSION: Timestep-conditional pseudo-labels enable the method to decouple conflicting preference dimensions.
  • 5 CONCLUSION: Experiments confirm that Semi-DPO effectively addresses the optimization problem caused by conflicting gradients.

ETHICS STATEMENT

The research improves text-to-image alignment using publicly available resources while acknowledging the societal implications and potential misuse of generative technologies.

  • ETHICS STATEMENT: The study relies on the publicly available Pick-a-Pic V2 dataset and SD1.5 and SDXL models, while recognizing generative technology’s broader societal implications and misuse risks.The resources are Pick-a-Pic V2, SD1.5, and SDXL.

6 APPENDIX · 6.1 DERIVATION OF THE DIFFUSION-DPO GRADIENT · 6.2 PROOF OF VARIANCE INFLATION

The appendix derives the per-timestep Diffusion-DPO gradient and proves that conflicting preference labels impose a conflict-dependent lower bound on gradient variance. The variance inflation is quadratic in the combined aligned and conflicting update magnitudes whenever conflict mass is nonzero.

  • 6.1 DERIVATION OF THE DIFFUSION-DPO GRADIENT: The appendix derives the gradient of the per-timestep Diffusion-DPO loss L(t)_DPO(θ) with respect to model parameters θ.The derivation introduces h(t) to simplify notation and applies the derivative identity for −log σ(x).
  • 6.2 PROOF OF VARIANCE INFLATION: For preference dimension k, each sample’s reward difference partitions the dataset into an alignment set A_k and conflict set C_k.The sets correspond to positive and negative reward differences, with probabilities p_a,k and p_c,k, respectively.
  • 6.2 PROOF OF VARIANCE INFLATION: The proof analyzes the per-sample gradient g(t) and defines conditional expected update magnitudes for aligned and conflicting samples.These conditional quantities are used to characterize the variance of ξ_t across alignment and conflict membership.
  • 6.2 PROOF OF VARIANCE INFLATION: By the law of total variance, Var(ξ_t) decomposes into within-group variance and variance between alignment and conflict groups.A random variable Z indicates whether a sample belongs to A_k or C_k.
  • 6.2 PROOF OF VARIANCE INFLATION: Because variance is non-negative, total variance is bounded below by the inter-group variance term based on conditional expectations.The proof computes Var(E[ξ_t | Z]) from the conditional aligned and conflicting expectations and uses p_a,k + p_c,k = 1.
  • 6.2 PROOF OF VARIANCE INFLATION: Any non-zero conflict mass p_c,k > 0 introduces a variance term that grows quadratically with the sum of conflicting and aligned update magnitudes.This establishes the conflict-dependent lower bound on total variance.

6.3 THE USE OF LARGE LANGUAGE MODELS

The authors used large language models solely to polish language and improve clarity, while retaining responsibility for all technical content and research outcomes.

  • 6.3 THE USE OF LARGE LANGUAGE MODELS: LLMs such as GPT were used only for language polishing and clarity improvements in the paper.They were not used to generate ideas, data, or experimental outcomes.
  • 6.3 THE USE OF LARGE LANGUAGE MODELS: The authors created the technical content, dataset design, experimental results, and analyses.

6.4 LIMITATIONS · 6.5 COMPARISON OF ONLINE AND OFFLINE DPO PARADIGMS IN IMAGE GENERATION

Semi-DPO requires iterative pseudo-labeling and retraining, yet noisy-data filtering makes it more efficient than the single-stage baseline. As an offline T2I DPO method, it uses fixed human-preference data, whereas online DPO dynamically generates and reward-model-evaluates new pairs at higher computational and procedural complexity.

  • 6.4 LIMITATIONS: Semi-DPO requires multiple cycles of pseudo-labeling and model retraining, following a pattern common to classical semi-supervised learning methods.The paper identifies iterative processing as a limitation, while noting that it does not increase computational costs.
  • 6.4 LIMITATIONS: 132 vs. 192 GPU hours: effective noisy-data filtering makes Semi-DPO more efficient than the single-stage baseline.The reported comparison indicates that iterative refinement does not entail higher computational cost.
  • 6.5 COMPARISON OF ONLINE AND OFFLINE DPO PARADIGMS IN IMAGE GENERATION: Offline sampling-based DPO in T2I, the paradigm used by Semi-DPO, trains on a static pre-collected human-preference dataset.The fixed dataset is used throughout the training process rather than being refreshed during training.
  • 6.5 COMPARISON OF ONLINE AND OFFLINE DPO PARADIGMS IN IMAGE GENERATION: Offline DPO fine-tunes the model in a single stage, providing computational efficiency, stability, and reproducibility.These properties arise from training on a fixed dataset for the entirety of the process.
  • 6.5 COMPARISON OF ONLINE AND OFFLINE DPO PARADIGMS IN IMAGE GENERATION: Online sampling-based DPO dynamically generates new preference data during the training loop by producing images and forming new preference pairs.The newly generated images are typically evaluated by an auxiliary reward model before continued training.
  • 6.5 COMPARISON OF ONLINE AND OFFLINE DPO PARADIGMS IN IMAGE GENERATION: Online DPO can learn continuously and potentially surpass the initial dataset’s quality, but it introduces significant computational overhead and complexity.Its additional generation and evaluation loop distinguishes it from the fixed-data offline paradigm.

6.6 COMPARISON WITH EXISTING METHODS · 6.7 FUTURE WORK: AN ONLINE EXTENSION OF THE SEMI-DPO PARADIGM · 6.8 MOTIVATION BY MULTI-REWARD SELECTION

Semi-DPO reframes conflicting preference pairs as noisy unlabeled data and uses the diffusion model itself as an implicit classifier and latent reward model. The paper extends this idea toward online, model-agnostic alignment and motivates multi-reward filtering through the dimensional specialization of existing reward models.

  • 6.6 COMPARISON WITH EXISTING METHODS: Semi-DPO separates clean labeled pairs from conflicting unlabeled pairs, leveraging both to address multi-dimensional preference noise and use existing datasets more fully.This semi-supervised formulation outperforms training on labeled data alone.
  • 6.6 COMPARISON WITH EXISTING METHODS: Theoretical and empirical analyses show that binary labels collapsing multi-dimensional preferences create noisy-label conflicts, while DPO-trained diffusion models can correct them as implicit classifiers.The model learns to distinguish preferred and dispreferred latent representations through the DPO objective.
  • 6.6 COMPARISON WITH EXISTING METHODS: Unlike explicit latent reward models requiring separate training and architectural modifications, Semi-DPO uses the original diffusion model as an efficient implicit latent reward model.This design requires no architectural changes.
  • 6.7 FUTURE WORK: AN ONLINE EXTENSION OF THE SEMI-DPO PARADIGM: Existing latent reward models are architecturally specific because shared VAE encoders make models trained for one generator incompatible with others such as SD1.5 and SDXL.This limits their plug-and-play versatility despite providing deep training signals across diffusion timesteps.
  • 6.7 FUTURE WORK: AN ONLINE EXTENSION OF THE SEMI-DPO PARADIGM: The proposed online extension would cold-start from a small consistent dataset, then generate and label new data using an ensemble of implicit reward models from consecutive training iterations.At iteration i+1, data generated by iteration i is labeled by models from iterations i and i −1.
  • 6.7 FUTURE WORK: AN ONLINE EXTENSION OF THE SEMI-DPO PARADIGM: This iterative self-training strategy is intended to eliminate bespoke latent reward models for each text-to-image architecture and enable model-agnostic alignment.The approach aims to retain latent-space rewards’ deep-signal benefits without their architectural constraints.
  • 6.8 MOTIVATION BY MULTI-REWARD SELECTION: Semi-DPO filters data with five proxy reward models because human preferences span aesthetics, detail quality, semantic alignment, and overall assessment.The committee comprises PickScore, HPS V2, CLIP, LAION Aesthetics Classifier, and ImageReward.
  • 6.8 MOTIVATION BY MULTI-REWARD SELECTION: Existing reward models specialize by preference dimension, whereas MPS shows the strongest correlation with multi-dimensional human preferences; its unavailable per-dimension scoring motivated five-model filtering.CLIP Score correlates strongly with semantic alignment, while Aesthetic Score correlates highly with aesthetics.

6.9 TRAINING DETAILS · 6.10 ADDITIONAL QUANTITATIVE RESULTS · 6.11 ADDITIONAL QUALITATIVE RESULTS

The paper details Semi-DPO’s training configuration, dynamic pseudo-labeling thresholds, and computational costs, while reporting additional quantitative and qualitative comparisons. Iteration 0 plus Iteration 1 offers the stated efficiency–performance balance, with unchanged deployment costs from the base architecture.

  • 6.9 TRAINING DETAILS: Semi-DPO training for SD1.5 uses 32 NVIDIA A100 40GB GPUs, global batch size 512, and β = 2500 across iterations.Iteration 0 uses learning rate 4 × 10−9 for 1,600 steps; Iterations 1 and 2 use 4 × 10−10 for 4,000 steps each.
  • 6.10 ADDITIONAL QUANTITATIVE RESULTS: The model’s prediction accuracy is additionally evaluated across the diffusion timeline on a clean test set.Table 7 presents accuracy variation across the diffusion timeline.
  • 6.9 TRAINING DETAILS: Dynamic pseudo-labeling partitions t ∈ [0, 999] into ten diffusion-time intervals and initially selects samples at each interval’s 80th confidence percentile.The initial thresholding strategy was evaluated on an accuracy-test portion containing 3,992 pairs.
  • 6.9 TRAINING DETAILS: 132 GPU hours are required for Iterations 0 and 1, compared with 192 GPU hours for the standard Diffusion-DPO baseline.The efficiency gain is attributed to dynamic thresholding filtering low-confidence samples from the noisy unlabeled dataset.
  • 6.9 TRAINING DETAILS: Extending training through Iteration 2 raises total cost to 228 GPU hours, while performance gains over Iteration 1 are marginal.The paper identifies the Iteration 0 + 1 configuration as the optimal balance between training efficiency and performance.
  • 6.9 TRAINING DETAILS: Because the final model preserves the base model’s exact architecture, deployment inference time and memory footprint incur zero additional computational cost.The paper states that inference and memory costs are identical to the baseline.
  • 6.10 ADDITIONAL QUANTITATIVE RESULTS: Additional win-rate comparisons are reported on Pick-a-Pic V2, HPS V2, and Parti-Prompt for both SD1.5 and SDXL baselines.Table 8 reports the comparisons as win rates in percent.
  • 6.11 ADDITIONAL QUALITATIVE RESULTS: A qualitative comparison presents Semi-DPO against baseline models.Figure 6 provides the additional qualitative comparison.
Loading 2604.24952v1…