iADD: Improving Alignment and Diversity in Diffusion Policy Optimization

1 Introduction

• Online reward optimization can cause mode collapse and quality degradation, while theoretically grounded study of alignment-diversity tradeoffs without preference data remains under-explored.Source: §1 · p. 2
• iADD combines early-timestep updates, structural regularization, and discrete-time Feynman-Kac guidance that prunes low-reward paths and branches promising ones.Source: §1 · p. 2
• The paper argues that early intervention and incremental sparse-timestep updates can preserve diversity while seeking higher alignment, contrary to late-timestep-only updates.Source: §1 · p. 2
• Across rare-prompt image generation, vanishing-point correction, and 3D scene generation, iADD improves alignment and generation diversity/quality over state-of-the-art approaches.Source: §1 · p. 3

2 Related Work

• Prior online approaches do not treat timestep selection with careful theory and under-exploit denoising paths, contributing to poorer alignment-diversity tradeoffs.Source: §2 · p. 3

3 Preliminaries

• DDPO models reverse diffusion as a multi-step MDP, but standard updates treat timesteps uniformly and rely on a single terminal reward, producing sparse credit assignment.Source: §3 · p. 4

4.2 Sparse Incremental Timestep Update

• For fixed N updated timesteps, uniform sampling across 1..T yields strictly greater total output perturbation ||δx0|| than late-stage sampling across 1..N.These are comparative predictions, not exact training-time bounds; the paper tests both under nonlinear finetuning dynamics in Sec. 5.5.Source: §4.2 · pp. 5–6
• Sparse updates over N ≪ T timesteps preserve prior probability volume better than updating all T, as measured by the global-Jacobian log-determinant.The paper interprets full updates as adding optimizer degrees of freedom that can yield reward-satisfying solutions at the cost of global prior collapse.Source: §4.2 · p. 6
• iADD begins with sparse high-leverage early timesteps for global mode discovery, then expands S to refine local manifold details after the distribution stabilizes.The overview depicts a progressive stage-wise increase in the cardinality of updated timesteps.Source: Figure 3 · p. 5, §4.2 · p. 6

4.3 Feynman-Kac Path Guidance

• FK guidance weights partial trajectories with intermediate potentials; branch-level terminal rewards set potentials used to prune branches unlikely to yield high reward.Sampling branches are completed to terminal samples before their rewards are used to score branches and intermediate nodes.Source: §4.3 · pp. 6–7

4.4 Rare Events and Rarity

• Rarity is the pretrained-model probability of a reward-at-least-η outcome; smaller ρ(c;η) means a rarer, harder condition, and the adapted model uses pθ instead of pϕ.Source: §4.4 · p. 7

5.1 Experimental Setup

• Stable Diffusion image experiments use LoRA fine-tuning and an incremental timestep schedule; vanishing-point correction reuses the image pipeline, while 3D synthesis uses MiDiffusion and 3D-FRONT.The comparison includes the pretrained model, DDPO, B2-DiffuRL, and additional guidance and GRPO-style baselines.Source: §5.1 · pp. 7–8

5.2 Evaluation Metrics

• The evaluation measures task-specific alignment, image diversity, and their reward–diversity trade-off, with rarity, intra-prompt LPIPS, and 3D collision and class-label metrics also reported.Models are evaluated at selected operating points; 3D comparisons use a common reward threshold.Source: §5.2 · p. 8

5.3 Qualitative Assessment

• Qualitative comparisons show iADD preserving prompt fidelity and visual quality on common and rare prompts, enforcing perspective geometry, and better preserving noncolliding 3D layouts.In a blind study of 4,800 judgments, raters preferred iADD more frequently than DDPO and B2-DiffuRL.Source: §5.3 · pp. 8–9, Figure 4 · p. 9, Figure 5 · p. 9

5.4 Quantitative Results

• Across all three tasks, iADD achieves the best alignment–diversity trade-off against DDPO and B2-DiffuRL at their best operating points.The comparison reports higher diversity at comparable or higher reward, alongside robustness on rare prompts.Source: Table 3 · p. 10

5.5 Ablation Studies

• At comparable reward levels, the sparse DDPO variant updating any five timesteps achieves higher diversity than standard 20-step Vanilla DDPO updates, supporting the reported timestep-selection finding.At matched compute (stage 27), sparse updates improve the full-generator log-volume by 1,605 nats over dense updates (−25,963 versus −27,568); the paired difference has a 437 SEM across 24 prompt–seed pairs.Source: §5.5 · pp. 10–11
• Adding an incremental schedule to FK improves alignment on common and rare prompts and diversity; FK branching also yields higher diversity, while the full incremental FK-branching recipe outperforms simpler component combinations.The reported comparison says naive combinations improve IS by only about 0.016; the figure reports additional diversity, collision, and human-preference results.Source: §5.5 · p. 11, Figure 7 · p. 11

6 Conclusions

• Output-sensitivity and Jacobian log-volume analyses motivate careful timestep selection, while discrete Feynman–Kac pruning and branching improve training-trajectory quality.The conclusion describes an incremental sparse-update curriculum and says the trajectory guidance and timestep curriculum also compose with GRPO-style optimization.Source: §6 · pp. 11–12

• Supplementary materials include theoretical proofs and algorithm and method details.Source: Supplementary Material for iADD:… · p. 15
• The supplement expands experimental setup across image, 3D, and vanishing-point settings, including reward-model details, metrics, training-time analysis, and additional comparisons and ablations.The overview names DPOK among the additional comparisons.Source: Supplementary Material for iADD:… · p. 15
• Extended qualitative results and discussions of limitations, future work, and broader impacts are also included.Source: Supplementary Material for iADD:… · p. 15

• For a fixed budget of N optimized timesteps, uniform sampling yields a strictly greater total output perturbation than updating only late timesteps.The paper argues that lower control authority can require the policy to work harder to meet the reward, leading to simplified generated structure.Source: S1 Theoretical Analysis and Proofs · p. 15
• A timestep’s effect on the clean sample scales with its cumulative Jacobian reach and noise coefficient; near the final denoising steps, the noise coefficient vanishes and the Jacobians contract.Source: S1 Theoretical Analysis and Proofs · p. 15
• For a fixed RL budget, sparse timestep updates preserve prior probability volume strictly better than full updates, as measured by the global Jacobian log-determinant.The analysis describes reward-tuned negative divergence contracting volume across fully updated steps, while frozen steps retain prior divergence properties.Source: S1 Theoretical Analysis and Proofs · pp. 15–16
• The proposed curriculum begins with a sparse set of high-noise timesteps, then expands the active subset for local refinement as reward gradients diminish.The analysis presents this as balancing early displacement against volume contraction.Source: S1 Theoretical Analysis and Proofs · p. 16

S2.1 Full Training Algorithm

• The section summarizes the full iADD training procedure in a pseudocode figure.The supplied passage identifies Fig. S1 as the training-procedure summary.Source: S2.1 Full Training Algorithm · p. 16, Figure S1 · p. 17

S2.2 Text-to-Image Alignment Reward

• CLIP (ViT-H-14) measures image–prompt alignment by matching each generated image with its paired text prompt.The procedure normalizes image and text features, computes pairwise similarity, and returns the diagonal cosine similarities as rewards.Source: S2.2 Text-to-Image Alignment Reward · pp. 16–17

S2.3 Vanishing Point Reward

• The Vanishing Point reward evaluates geometric coherence by finding dominant vanishing points and assessing line-segment consistency.The procedure extracts and orientation-clusters line segments, then uses RANSAC to select vanishing-point candidates and compute cluster rewards.Source: S2.3 Vanishing Point Reward · pp. 17–18
• Indoor-scene rewards combine collision avoidance, which ignores ceiling fixtures, with a user-defined constraint placing the TV stand in front of the bed.The non-penetration reward penalizes overlapping ground objects; the placement reward accounts for missing objects, distance, and bed-facing alignment.Source: S2.4 3D Indoor Scene Layout Rewards · p. 18

S3.1 Image Generation

• Image-generation experiments use Stable Diffusion v1.4 as the base model and LoRA for parameter-efficient fine-tuning, progressively expanding trained timesteps across four stages.Sampling uses DDIM with η = 1.0.Source: S3.1 Image Generation · p. 19

S3.2 Vanishing Point Detection

• The 3D-scene experiments fine-tune MiDiffusion’s continuous-domain component, which predicts bedroom layouts conditioned on floor plans.Scenes are unordered object sets with semantic, position, size, and rotation features; the experiments use LoRA and Feynman–Kac filtering with four particles.Source: S3.3 3D Scene Generation · pp. 19–20

S3.4 Branch Feynman-Kac Trajectory Rollout

• Feynman–Kac steering branches trajectories before resampling, creating multiple paths with shared early history that diverge later.The experiments use a max-potential form, which favors higher-reward particles; resampling is kept to selected timesteps because overly frequent resampling may waste computation without significant gains.Source: S3.4 Branch Feynman-Kac Trajectory… · p. 20

S3.5 Prompt Templates and Dataset

• Four prompt templates test animal-action compositionality, rare color-object binding, spatial relations, and geometric structures.The templates include 45, 40, 40, and 24 prompts, respectively.Source: S3.5 Prompt Templates and Dataset · p. 21

S4 Evaluation Metrics

• The section presents metrics for image generation and 3D indoor-scene generation.The supplied passage identifies the two evaluation areas but does not specify additional metric details for the latter section.Source: S4 Evaluation Metrics · p. 21

S4.1 Image Generation Metrics

• The Inception Score uses CLIP embeddings instead of the traditional Inception-v3 classifier.Higher scores indicate better image quality and greater diversity; each case is evaluated with 1080 samples.Source: S4.1 Image Generation Metrics · p. 21
• Collision rate measures the percentage of generated-scene objects involved in collisions, detected by Axis-Aligned Bounding Box overlap.Lower rates indicate more physically plausible layouts; ceiling-mounted objects are excluded because they do not interact with ground-level furniture.Source: S4.2 3D Indoor Scene Generation Metrics · p. 22

S5.1 Reward Distribution Across Prompts

• Pretrained rewards span a substantial range, with lower-reward prompts representing concepts the pretrained model finds challenging to represent accurately.Source: S5.1 Reward Distribution Across Prompts · p. 22

S5.2 Rare vs Common Prompt Definition

• Rare prompts are the bottom 25% of pretrained rewards, Common prompts the top 25%, and the remaining half are Intermediate.The grouping is defined separately for each template.Source: S5.2 Rare vs Common Prompt Definition · p. 22

S5.3 Quartile Analysis

• Ours improves over SD and competitive baselines across all pretrained-reward quartiles.The analysis aggregates evaluation prompts across templates and groups them by pretrained reward.Source: S5.3 Quartile Analysis · p. 22
• 16.7% is Ours’ reported increase over the SD baseline for Rare prompts in the third quartile.The passage identifies this as the most pronounced relative improvement.Source: S5.3 Quartile Analysis · p. 22
• 12.0% is Ours’ reported increase over the SD baseline for Intermediate (Q2) prompts.Source: S5.3 Quartile Analysis · p. 22
• 10.5% is Ours’ reported increase over the SD baseline for Common (Q3) prompts.Source: S5.3 Quartile Analysis · p. 22

S6.1 Comparison with DPOK

• At reward 0.38 and diversity 1.285, the controlled comparison finds that KL regularization improves robustness against reward hacking but does not match iADD’s performance.The authors suggest iADD balances reward maximization and diversity without an explicit KL constraint.Source: S6.1 Comparison with DPOK · p. 22

S7.1 Template-wise Tables for Image Generation

• Tables S4–S6 report quantitative results separately for image-generation templates T1, T2, and T3.The supplied passage identifies the template-wise scope but does not state table values.Source: S7.1 Template-wise Tables for Image… · p. 23

S7.2 Vanishing Point Evaluation

• On vanishing-point correction, iADD improves reward and Inception Score simultaneously, and evaluators consistently prefer its visual quality and alignment over alternatives.The authors caution that Inception Score may be unreliable for image quality under strict geometric constraints.Source: S7.2 Vanishing Point Evaluation · p. 23

S7.3 User Study Results for Image Generation

• The image-generation user-study plots break human preference rates down by both template and prompt.The passage describes the breakdown but gives no preference values.Source: S7.3 User Study Results for Image… · p. 23

S7.4 Quartile Reward Analysis

• +16.7% is iADD’s largest relative gain over SD, DDPO, and B2-DiffuRL in the rare-prompt quartile, with +12.0% in intermediate and +10.5% in common prompts.The analysis compares prompt-difficulty quartiles against the global pretrained Stable Diffusion baseline.Source: Figure S3 · p. 24
• Quartile reward plots examine the 25th, 50th, and 75th percentiles of prompt difficulty across templates T1–T3 and the vanishing-point task.The supplied description identifies the scope of the plots without reporting their individual values.Source: S7.4 Quartile Reward Analysis · p. 24

S9 Training Time Comparison

• Reaching a mean CLIP score of 0.388 takes iADD 7.43 GPU hours versus 2.55 GPU hours for DDPO, mainly because FK sampling requires repeated reward evaluations.Incremental timestep updates partially reduce the cost; inference-time cost is comparable to DDPO and avoids the extra overhead of inference-time FK steering.Source: S10 Limitations and Future Work · p. 25

S10 Limitations and Future Work

• Rewards inferred from denoised predictions of noisy latents can be unreliable at early timesteps, limiting improvements when 3D scene geometry is not recoverable from those states.The passage gives object size and position in indoor scene layout generation as examples of critical attributes that may remain inaccurate midway through denoising.Source: S10 Limitations and Future Work · p. 25
• CLIP can assign a high score to a weakly aligned image, such as cooked chicken near a chessboard for “chicken playing chess,” motivating stronger reward models.The authors also identify more accurate terminal-reward estimation from noisy intermediate states and video diffusion as future work.Source: S10 Limitations and Future Work · p. 25

S11 Broader Impacts.

• Task-specific alignment may better respect perspective or collision-free layout constraints, but controllable generation can also produce misleading synthetic content.The authors call for safeguards, responsible deployment, and robust detection and auditing tools.Source: S11 Broader Impacts · p. 26