PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributes

Chenye Meng, Zejian Li, Zhongni Liu, Yize Li, Changle Xie, Kaixin Jia, Ling Yang, Huanghuang Deng, Shiying Ding, Shengyuan Zhang, Jiayi Li, Lingyun Sun

arXiv:2601.04300v1cs.CV

TL;DR

Post-training alignment commonly uses reward scores or binary signals, unlike the fine-grained criteria of human expertise. The paper builds a hierarchical painting-attribute system and uses SFT with CPO to align diffusion models; experiments report improved quality and expert alignment over existing baselines.

  • Problem

    Post-training methods rely on reward-model scores or binary signals, whereas expert evaluation uses complex, fine-grained criteria.

    Source: §1 · p. 1

  • Method

    The paper constructs a hierarchical painting-attribute system, trains an expert model with SFT, and uses CPO to align diffusion models by decoupling positive and negative attributes.

    Source: §1 · p. 2, §5.1 · p. 5

  • Results

    On the primary SDXL group, SDXL-CPO (LCPO−S) reduced #A neg to 5.180 while achieving best FID (87.37) and joint-highest preference scores; CPO outperformed existing baselines.

    Source: §6.3 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that fine-grained evaluation supports future post-training paradigms.

    Source: §1 · p. 2

    CPO remains constrained by base-model limitations and has not yet achieved simultaneous improvement in positive-sample fitting and degradation in negative-sample fitting.

    Source: S13. Failure Cases and Limitation · p. 21

Abstract

from arXiv · show

Post-training alignment of diffusion models relies on simplified signals, such as scalar rewards or binary preferences. This limits alignment with complex human expertise, which is hierarchical and fine-grained. To address this, we first construct a hierarchical, fine-grained evaluation criteria with domain experts, which decomposes image quality into multiple positive and negative attributes organized in a tree structure. Building on this, we propose a two-stage alignment framework. First, we inject domain knowledge to an auxiliary diffusion model via Supervised Fine-Tuning. Second, we introduce Complex Preference Optimization (CPO) that extends DPO to align the target diffusion to our non-binary, hierarchical criteria. Specifically, we reformulate the alignment problem to simultaneously maximize the probability of positive attributes while minimizing the probability of negative attributes with the auxiliary diffusion. We instantiate our approach in the domain of painting generation and conduct CPO training with an annotated dataset of painting with fine-grained attributes based on our criteria. Extensive experiments demonstrate that CPO significantly enhances generation quality and alignment with expertise, opening new avenues for fine-grained criteria alignment.

1. Introduction

The paper argues that scalar rewards and binary preferences do not capture the hierarchical, fine-grained nature of expert evaluation. It develops a painting-specific attribute hierarchy and dataset, then proposes attribute-decoupled alignment to address this mismatch.

1. Introduction

  • CPO uses an SFT-trained expert model to guide a target diffusion model toward positive attributes and away from negative attributes.The expert model supplies winner and loser noise predictions based mainly on positive and negative attributes, respectively.Source: §1 · p. 2

2. Related Work

  • Prior preference datasets use binary wins and losses or monolithic aesthetic scores, while multi-objective methods aggregate rewards rather than use fine-grained image attributes.The related-work discussion also notes approaches refining preference optimization and approaches using multiple distinct rewards.Source: §2 · pp. 2–3

3. Domain-specific Fine-grained Evaluation

  • The painting evaluation system organizes seven root dimensions and 246 manually defined positive/negative attribute pairs in a five-level hierarchy.The dimensions include composition and color relations; the paper describes the evaluation as multidimensional, discrete, and non-equilibrium.Source: §1 · p. 2, §3 · p. 3
  • Expert criteria include coexisting negative and positive attributes, while some positive attributes are mutually exclusive, challenging utility-based post-training methods.The applicable attributes also vary across samples, so a single fixed signal does not capture every painting’s evaluation criteria.Source: §3 · p. 3
  • A domain-expert agent applies a Deconstruct-Structure-Quantify process to annotate 10,277 paintings with image-relevant positive and negative attributes.It activates a relevant subset of terminal symbolic labels from the hierarchy for each image; the authors report acceptable annotation accuracy by manual investigation.Source: §3 · p. 3

4. Preliminary

The preliminary section introduces denoising in diffusion models, then reviews preference optimization and classifier-free guidance. It presents the latter as inspiration for steering generation toward positive attributes and away from negative ones.

4. Preliminary

  • Diffusion models learn data distributions by reversing a gradual noising process, training noise prediction through expected reconstruction error and sampling through iterative denoising.The model predicts injected noise at a timestep conditioned on the prompt or other condition.Source: §4 · pp. 3–4
  • Classifier-Free Guidance extrapolates from an unconditional prediction toward a conditional prediction, inspiring attribute guidance toward positive and away from negative directions.The passage describes a fidelity–diversity trade-off for CFG and connects the extrapolation structure to the paper’s attribute steering.Source: §4 · p. 4
  • DPO directly optimizes a generative policy from preferred and dispreferred pairs, avoiding explicit reward-model training while regularizing against a reference distribution.The paper describes the objective as pushing the policy toward preferred outputs while limiting divergence from the reference.Source: §4 · pp. 4–5
  • Diffusion-DPO adapts preference optimization to diffusion by optimizing reverse trajectories and reducing training to a preference-weighted denoising criterion.Its practical loss uses model and reference noise predictions, with timestep weighting described as typically constant.Source: §4 · p. 5

5. Method

Existing post-training methods use simplified signals that miss fine-grained evaluation, so the paper proposes a two-stage framework for injecting and aligning with domain-specific criteria. It combines an attribute-aware expert model with preference optimization that separates positive and negative attributes.

5.1. Domain-specific Knowledge Learning

  • SFT trains expert model θ1 on image, prompt, and positive/negative attribute tuples to capture correlations between images and domain-specific attributes.The conditioning combines the prompt with both attribute sets, and training minimizes denoising loss.Source: §5.1 · p. 5

5.2. Complex Preference Optimization

  • CPO uses an SFT expert model as a dynamic reward oracle, generating ideal and non-ideal denoised predictions at each timestep to guide the target model.Unlike static preference pairs, these predictions provide fine-grained guidance for training.Source: §5.2 · p. 5
  • Deterministic positive and negative noise guidance steers trajectories toward virtual winning and losing samples, enabling process-level training guidance rather than endpoint-only comparisons.The final sample can be reconstructed back to the noisy sample at timestep t.Source: Figure 3 · p. 6, §5.2 · p. 6
  • The target model minimizes error against preferred noise and maximizes error against dispreferred noise, learning positive and negative attributes without attribute inputs at inference.The target model is conditioned only on the content prompt.Source: §5.2 · pp. 5–6

5.3. Stabilization of the Optimization

  • The authors attribute instability to the losing-term gradient growing while the winning-term gradient shrinks, and transform the loss to balance their contributions.The stabilized objective uses a surrogate convex term in place of the original concave term.Source: §5.3 · pp. 6–7

6. Experiment

Experiments on painting generation compare CPO with preference-optimization baselines using negative-attribute and general-quality metrics. Ablations assess reward granularity, training-data volume, and stabilization.

6.1. Dataset and Implement; 6.2. Evaluation and Baselines

  • The experiments use 10,277 paintings and compare CPO with fine-tuned SDXL and FLUX, Diffusion-DPO, SPO, and NPO-augmented variants across negative-attribute and image-quality metrics.The dataset is split 80/10/10; the evaluation includes #A neg, FID, PickScore, HPSv2, ImageReward, and Aesthetic Score.Source: §6.2 · p. 7, §6.1 · p. 7

6.3. Quantitative Result

  • SDXL-CPO (LCPO−S) reduces #A neg to 5.180 while achieving the best FID (87.37) and joint-highest preference scores; CPO also outperforms NPO on most metrics.With NPO further training, CPO has lower scores only on PickScore and ImageReward.Source: §6.3 · p. 7
  • FLUX-CPO reaches #A neg 3.780, versus 5.120 for FLUX and 4.400 for FLUX-DPO, and achieves the best preference scores.The authors report that fine-tuning raises FID, particularly for FLUX.Source: §6.3 · p. 8

6.4. Qualitative Result

  • In visual comparisons, CPO shows fewer negative attributes and tends toward higher preference scores, with strengths in composition, color harmony, light and shadow, and brushwork.Figure 4 annotates #A neg and PickScore on each image.Source: §6.4 · p. 8, Figure 4 · p. 8

6.5. Ablation Study

  • The complex, seven-dimensional reward performs best across all metrics against scalar-score and binary-preference reward designs.Scalar averages the seven dimensions; binary reduces each dimension to a winning or losing label.Source: §6.5 · p. 8
  • Performance improves significantly as the proportion of attribute-decoupled training data increases.Table 3 reports #A neg, PickScore, ImageReward, and LAION-Aesthetic across data proportions.Source: §6.5 · p. 8
  • Stabilization makes the winning and losing loss terms decrease more smoothly; without it, the loss oscillates and emphasizes unlearning losing samples over learning winning ones.The authors report that the stabilized method outperforms the non-stabilized objective in Table 1.Source: §6.5 · p. 8

7. Conclusion

The conclusion presents a fine-grained evaluation criterion with positive and negative attributes and a two-stage alignment approach with stabilization. The experiments report that CPO outperforms existing baselines.

7. Conclusion

  • The reward-design comparison distinguishes scalar-score and binary-preference optimization from fine-grained, attribute-decoupled optimization.Table 2 labels the latter design Complex.Source: Table 2 · p. 9
  • The paper proposes a two-stage alignment strategy with stabilization, based on a hierarchical, fine-grained criterion of positive and negative attributes.The criterion is intended to address reliance on simplified feedback in preference alignment.Source: §7 · pp. 8–9
  • CPO outperforms existing baselines in experiments.Figure 5 compares the win and lose loss curves and reports greater stability with stabilization than without it.Source: §7 · p. 9, Figure 5 · p. 9

S1. Description of the Fine-grained Hierarchical Evaluation

The supplementary material describes a hierarchical evaluation system, explains why complex preference learning needs more than a single superiority score, and reports a professional user study comparing CPO with DPO baselines.

S1. Description of the Fine-grained Hierarchical Evaluation

  • The evaluation system has five hierarchy levels, seven root dimensions, and 246 positive/negative attribute pairs, providing separate attribute sets for the same image.Its dimensions include Composition, Color Relationships, and Brushstrokes & Texture.Source: S1. Description of the Fine-grained… · p. 12

S2. Description of Complex Preference Learning Tasks

  • Because paintings can combine positive and negative attributes and criteria vary with content and style, the authors argue that a single superiority score is unsuitable for these tasks.Their examples contrast different strengths and weaknesses across images.Source: S2. Description of Complex Preference… · p. 12

S3. User Study

  • The user study sampled 150 test prompts, generated 600 images from four trained models, and asked 10 art- or design-trained participants to compare image pairs across seven root dimensions.The comparisons paired SDXL-DPO+NPO with SDXL-CPO+NPO and FLUX-DPO with FLUX-CPO.Source: S3. User Study · p. 12

S4. More Qualitative Results

The supplementary results compare CPO-generated painting details and examine stabilization’s effects on training behavior and gradients. They also identify a remaining gap between the optimization and the desired positive/negative separation.

S4. More Qualitative Results

  • CPO+NPO produces the fewest negative attributes and highest PickScore across examples, while CPO improves brushwork, light and shadow, and style consistency.Examples include a swirling Van Gogh sky, a Monet night-scene halo, and dramatic Baroque-portrait lighting.Source: S4. More Qualitative Results · p. 14

S5.1. Effectiveness Analysis

  • Stabilized training achieves a noticeably lower winning-term loss than the comparison, indicating stronger learning of positive attributes.The authors contrast this with conventional DPO-style margin enlargement, which can degrade fit on both positive and negative samples.Source: S5.1. Effectiveness Analysis · p. 14
  • The method does not yet fully achieve the ideal of improving fit on positive samples while deteriorating fit on negative samples.The authors identify achieving this separation as future work.Source: S5.1. Effectiveness Analysis · p. 14

S5.2. Gradient Analysis

  • The original losing-term gradient pushes the model away from the negative prototype, but its magnitude grows unbounded as negative attributes are unlearned, allowing it to dominate the winner term.The gradient analysis names the winner and loser terms in the original CPO objective.Source: S5.2. Gradient Analysis · p. 14
  • The stabilized objective introduces a surrogate target for the losing term and treats that target as fixed via stop-gradient.Source: S5.2. Gradient Analysis · p. 14

SDXL-DPO+NPO SDXL-CPO+NPO FLUX-DPO FLUX-CPO

The supplementary material reports a guidance-strength trade-off, visual comparisons, a user-study setup, and the stabilization strategy’s loss transformation. At ω = 2.0, the authors select a balance between negative-attribute count and preference metrics.

  • The user-study figure presents qualitative image comparisons and preference rates for CPO methods against DPO baselines across SDXL and FLUX.Source: Figure S8 · p. 15
  • CPO outperforms the baselines in both negative-attribute avoidance and PickScore in the visual comparisons.Figures S9 and S10 annotate #A neg and PickScore on the images.Source: Figure S10 · p. 16, Figure S9 · p. 16
  • The stabilization transformation makes the losing term convex while preserving its direction and matching its optimization magnitude to the winning term.The accompanying loss curves compare training with and without stabilization.Source: Figure S12 · p. 17, Figure S11 · p. 17

Ablation Study of the Dynamic Process Reward Parameter ω

  • 5.18 negative attributes at ω = 2.0 is the selected balance; increasing ω from 1.0 to 3.0 reduces that count from 5.26 to 4.87 but raises FID from 86.61 to 90.18.ImageReward and Aesthetic scores improve continuously with ω, while PickScore and HPSv2 have an optimal balance at ω = 2.0.Source: SDXL-DPO+NPO SDXL-CPO+NPO FLUX-DPO… · p. 17, Table S4 · p. 17

S7. Additional Details on CPO

CPO constructs positive and negative preference trajectories from a shared noisy state, then derives a training objective using the auxiliary model’s deterministic trajectories. The supplementary derivation also describes how this construction relates to diffusion-DPO’s path-based formulation.

S7.1. Trajectory Description

  • CPO uses the auxiliary model to construct deterministic, controllable latent-space preference trajectories, rather than relying only on final endpoints and approximations to an intractable reverse process.The text cautions that this does not necessarily mean smaller propagation error, but says the error becomes controllable and exploitable for more efficient training.Source: S7.1. Trajectory Description · pp. 17–18
  • From the same noisy state, CPO deterministically constructs a positive trajectory conditioned on the prompt and positive attributes and a negative trajectory conditioned on the prompt and both positive and negative attributes.The shared starting state lets CPO focus optimization on diverging steps and provide a positive or negative gradient at each timestep.Source: S7.1. Trajectory Description · p. 18

S7.2. Mathematical Derivations

  • Diffusion-DPO [40] uses an ELBO formulation over the full diffusion path, but its loss remains intractable; the derivation substitutes deterministic trajectories from θ1.Given the current noise, the predicted x̂0 can be used to reconstruct xt exactly, making the θ1 trajectory accessible.Source: S7.2. Mathematical Derivations · p. 18
  • The final CPO objective lowers positive-noise prediction error relative to the reference model while raising negative-noise error, with βTω(λt) scaling the preference score.The derivation relates the diffusion-model KL terms to mean-squared predicted-noise error and samples timestep t uniformly.Source: S7.2. Mathematical Derivations · p. 18

S8. Hallucination in Agent Behaviors

The appendix verifies automatically annotated painting attributes against human judgments and reports SFT-model performance under three prompting configurations. It notes residual deviation from human judgments and identifies Configuration A as strongest on the reported measures.

S8. Hallucination in Agent Behaviors

  • The human verification randomly sampled 100 complex-annotation images and asked 10 participants to check positive and negative attributes across seven dimensions.Source: S8. Hallucination in Agent Behaviors · p. 18
  • Annotation accuracy is 88.71% overall, ranging from 81.96% for Shape and Form to 96.18% for Color Relationship.Other dimension accuracies are Perspective and Space 91.93%, Edge Relationship 91.44%, Light and Shadow 94.49%, Brushwork and Texture 89.29%, and Composition 88.93%.Source: S8. Hallucination in Agent Behaviors · pp. 18–19, Table S5 · p. 19
  • The authors report slight deviation from human judgment, noting aesthetic subjectivity and prioritizing correct positive/negative attribute identification over exhaustive annotation coverage.Source: S8. Hallucination in Agent Behaviors · p. 19

S9. Reliability of the SFT Model

  • Among the three SFT-model prompting configurations, Configuration A has the highest positive-attribute IoU, lowest negative-attribute IoU, and best overall performance.All three configurations have high Apos IoU and low Aneg IoU.Source: S9. Reliability of the SFT Model · p. 19, Table S6 · p. 19

S10. Negative Noise Construction

The supplement explains how CPO uses attribute-conditioned noise targets, relates its objective to existing preference optimizers, and qualifies its scope with model, optimization, and social limitations.

S10. Negative Noise Construction

  • Positive labels encode an image’s complete attribute information, so negative noise guidance uses both positive and negative attributes rather than negative labels alone.For example, the negative label “absence of shape-breaking elements” lacks the compositional context supplied by “circular composition.”Source: S10. Negative Noise Construction · p. 19

S11. Differences from and Advantages over Inversion-Based DPO

  • CPO decouples positive and negative attributes for fine-grained guidance, unlike inversion-based DPO methods that rely on coarse winner/loser preferences.The criteria are hierarchical, multidimensional, discrete, and non-equilibrium.Source: S11. Differences from and Advantages… · p. 19
  • Approximately 10 GPU hours yield stable one-epoch SDXL convergence with CPO, versus around 138 GPU hours for comparable Inversion-DPO convergence; stabilization also smooths training.Source: S11. Differences from and Advantages… · p. 20

S12.1. The Special Variant of CPO

  • With a one-level, single-dimension attribute system, CPO becomes highly similar to DPO; unlike BCO, it directly optimizes the policy without an explicit classifier.Source: S12.1. The Special Variant of CPO · p. 20

S12.2. The reliability of CPO

  • CPO relies on fine-tuned θ1 as a surrogate to translate positive and negative labels into dynamic noise targets; direct, stable optimization from raw labels remains future work.Source: S12.2. The reliability of CPO · p. 20

S12.3. The Generalizability of CPO

  • The criteria are instantiated for painting with a five-level hierarchy, seven root dimensions, and 246 attribute pairs, while the authors propose the paradigm for other domains with similarly complex criteria.Source: S12.3. The Generalizability of CPO · p. 20

S13. Failure Cases and Limitation

  • CPO remains constrained by base-model defects and occasional missed positive attributes, while stabilization has not yet both improved positive fitting and degraded negative fitting.Reported failures include anatomical, quantity, scale, and spatial errors.Source: S13. Failure Cases and Limitation · p. 21, Figure S13 · p. 21

S14. Social Impact

  • Expert-defined criteria may support nuanced alignment in art and design, but narrow or homogeneous criteria could reduce diversity and marginalize unconventional styles.Source: S14. Social Impact · p. 21

Related papers

Loading 2601.04300v1…