PaperlayerView exact revision on arXiv

Source-linked AI summary

AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generation

Xu Wang, Zexian Li, Litong Gong, Tiezheng Ge, Zhijie Deng

arXiv:2604.28126v1cs.CVcs.AI

TL;DR

Few-step diffusion distillation can degrade generation quality, and combining it with RL can add complexity while fixed rewards risk reward hacking. AdvDMD unifies distillation and RL using an online adversarial discriminator reward and a revised training schedule. On SD3.5, its four-step model exceeds the reported 40-step baseline on DPG-Bench and GenEval.

  • Problem

    Distillation can reduce sampling steps, but performance degradation remains pronounced at very low step counts, while integrating RL with distillation can be complex.

    Source: §1 · p. 2

  • Method

    AdvDMD unifies distillation and RL by using an online-updated DMD2 discriminator as reward across intermediate and final denoising states, with decoupled updates and SDE backward simulation.

    Source: §1 · p. 3

  • Results

    84.65 on DPG-Bench for 4-step SD3.5 AdvDMD exceeds the 40-step SD3.5 baseline (84.55); on GenEval it scores 0.70 versus 0.69 for that baseline.

    Source: §1 · p. 3, §4.2 · p. 9

  • Takeaways & Limitations

    AdvDMD demonstrates that few-step models can match or exceed the reported full-step SD3.5 baseline on DPG-Bench and GenEval.

    Source: §4.2 · p. 9

    Optimizing a single reward proxy can produce reward hacking, improving PickScore slightly while HPS and DEQA decline.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

Diffusion models offer superior generation quality at the expense of extensive sampling steps. Distillation methods, with Distribution Matching Distillation (DMD) as a popular example, can mitigate this issue, but performance degradation remains pronounced when sampling steps are limited. Reinforcement learning (RL) has been leveraged to improve the few-step generation quality during distillation, with the potential to even surpass the performance of the teacher model. However, existing approaches are combinatorial in nature, merely integrating an RL process with the distillation process, which introduces unnecessary complexities. To address this gap, we propose AdvDMD, a method that seamlessly unifies DMD distillation and RL. Specifically, AdvDMD employs the adversarially trained discriminator from DMD2 as the reward model, which assigns low scores to generated images and high scores to real ones. It is trained on both intermediate and final states of the denoising process and updated online with the distilled model, enabling a holistic supervision of the sampling trajectories and mitigating reward hacking. We adopt a unified SDE backward simulation and a different training schedule for DMD and RL to enable a more stable and efficient training. Experimental results demonstrate that the 4-step AdvDMD outperforms the original 40-step model for SD3.5 on DPG-Bench, while achieving significant performance gains for SD3 on the GenEval. On Qwen-Image, our 2-step AdvDMD achieves superior performance over TwinFlow.

1 Introduction

Few-step distillation can lose quality against the teacher, while combining it with reinforcement learning can add complexity and expose reward hacking. AdvDMD unifies distillation and RL around an online adversarial reward, with training changes designed to improve stability and efficiency.

1 Introduction

  • AdvDMD unifies distillation and RL by using the DMD2 discriminator as a reward model, trained on intermediate and final denoising states and updated online.The online update with the distilled model is intended to mitigate reward hacking.Source: §1 · p. 3
  • The experiments evaluate SD-series and Qwen-Image models using GenEval and DPG-Bench as primary benchmarks.The supplied results highlight SD3.5 on DPG-Bench.Source: §1 · p. 3
  • 84.65 on DPG-Bench for 4-step SD3.5 AdvDMD surpasses both 4-step DMD2 and the original 40-step baseline; 2-step AdvDMD scores 84.11.The reported SD3.5 results are on DPG-Bench.Source: §1 · p. 3
  • Decoupled DMD and GRPO update frequencies and SDE-based backward simulation address unreliable early rewards and redundant sampling, respectively.DMD is updated multiple times between GRPO updates.Source: §1 · p. 3

2 Related Work

Prior work accelerates diffusion generation through distillation and improves models through reinforcement learning, but few-step settings and fixed reward preferences remain challenges. AdvDMD is positioned among methods that combine or adapt these approaches.

Distillation for Diffusion Models

  • DMD aligns student and teacher output distributions, while DMD2 adds a GAN-style objective to distinguish real images from synthesized samples.TwinFlow uses a self-adversarial objective and DMDR brings preference-model rewards into DMD.Source: §2 · p. 3

Reinforcement learning for Diffusion Models

  • Existing distillation and RL methods can rely on fixed teacher or reward preferences, and reward optimization may inflate scores without improving overall output quality.The related-work discussion identifies this as a reward-hacking concern in limited-inference-step settings.Source: §2 · pp. 3–4
  • Flow-GRPO uses SDE sampling for exploration and group-wise rewards, while Flow-DPO and DiffusionNFT offer distinct preference-optimization and online-RL approaches.These methods are described as presupposing sufficiently strong generative capabilities.Source: §2 · p. 4

3 Method

AdvDMD combines DMD distillation and GRPO reinforcement learning using a discriminator as an adversarial reward model. Its method specifies how the reward, sampling, and update schedule are integrated.

3.1 Preliminary: DMD and GRPO

  • DMD minimizes the divergence between generated and real image distributions, while GRPO applies online policy-gradient updates using group-normalized rewards.For DMD, the real and fake score functions are approximated by a pretrained teacher and an additionally trained diffusion model, respectively.Source: §3.1 · pp. 4–5

3.2 From Discriminator to Adversarial Reward

  • A frozen diffusion-model backbone supplies multiscale features to convolutional heads, whose averaged realism predictions form the discriminator output.The discriminator is trained to distinguish generated from reference images.Source: §3.2 · p. 5
  • The discriminator scores noisy intermediate states individually, providing timestep-specific rewards for GRPO rather than reusing the final image’s reward at every step.The paper says these per-timestep signals support fine-grained policy updates aligned with the target data distribution.Source: §3.2 · p. 6

3.3 AdvDMD: Adversarial Reward Meets DMD

  • AdvDMD repurposes the DMD2 discriminator as a GRPO reward model, unifying distillation and reinforcement learning without adding another model.The discriminator provides adversarial feedback throughout generation.Source: §3.3 · p. 6
  • SDE backward simulation lets DMD reuse GRPO-generated samples, avoiding redundant sampling; DMD updates are prioritized early while reward signals remain unstable.The authors report that using SDE sampling in DMD does not degrade performance, and defer full GRPO updates until synthesis and discrimination stabilize.Source: §3.3 · p. 6, Figure 3 · p. 7
  • The generator’s objective combines DMD and GRPO losses, with α balancing them and γ controlling the adversarial term.The fake model and discriminator use the DMD2 reconstruction and adversarial losses.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate AdvDMD on SD3 and SD3.5 benchmarks and examine implementation choices and reward-model ablations. Four-step AdvDMD outperforms the 40-step SD3.5 baseline on DPG-Bench and matches or slightly surpasses it on GenEval.

4.1 Implementation Details

  • Training uses SD3-medium and SD3.5-medium backbones, with 4-step and 2-step SD3.5 variants and a 4-step SD3 variant.Models train at 512 × 512 resolution on 8 NVIDIA H20 GPUs for approximately 18 hours.Source: §4.1 · p. 7

4 Experiments

  • Evaluation uses DPG-Bench and GenEval, and compares AdvDMD with teacher models and accelerated methods including Hyper-SD, TDM, DMD2, SD3.5-Flash, and SD3.5-Turbo.The benchmarks assess image fidelity, semantic alignment, compositional correctness, and adherence to human preferences.Source: §4.2 · p. 8

4.2 Main Results

  • 84.65: 4-step SD3.5 AdvDMD exceeds the 40-step baseline’s 84.55 on DPG-Bench; 4-step SD3 AdvDMD scores 84.25, the highest among 4-step SD3 methods.AdvDMD also outperforms DMD2* at 2 steps on SD3.5 and achieves a Global score of 92.42.Source: Table 1 · p. 8, §4.2 · p. 9
  • 0.70: 4-step SD3.5 AdvDMD matches or slightly surpasses the 40-step baseline’s 0.69 on GenEval; 4-step SD3 scores 0.68 versus 0.62 for its 25-step baseline.At 2 steps, SD3.5 AdvDMD scores 0.67 versus 0.65 for DMD2*.Source: Table 2 · p. 8, §4.2 · p. 9

4.3 Ablation Studies

  • At 4 steps, optimizing HPS raises its score from 0.282 to 0.324, while ClipScore falls from 0.389 to 0.369 and DEQA from 4.14 to 3.91.PickScore optimization also slightly improves PickScore (0.224 versus 0.223) while HPS and DEQA decline; the authors describe this as reward hacking.Source: §4.3 · pp. 9–10
  • Standalone 4-step Flow-GRPO with HPS or PickScore produces blurred outputs, and HPS training can degrade instruction following.At 1 or 2 steps, the model lacks the generation capability needed to produce valid images.Source: §4.3 · pp. 9, 11, Figure 5 · p. 11
  • On DPG-Bench, the discriminator alone scores 0.8406; adding HPS and ClipScore raises the overall score to 0.8465.The combined rewards gain on relation, attribute, entity, and global metrics.Source: §4.3 · p. 11, Table 5 · p. 11
  • On GenEval, the discriminator scores 0.7064 overall, increasing to 0.7070 with HPS and ClipScore despite slight declines on some dimensions.The authors report that the discriminator alone provides a strong, stable reward signal, with auxiliary rewards offering modest complementary improvements.Source: §4.3 · p. 11, Table 6 · p. 12

5 Conclusion

AdvDMD uses the GAN discriminator as a reward model for intermediate denoising steps. Its adversarial feedback guides the generator toward the data distribution, with the stated aim of promoting perceptual realism and structural fidelity.

5 Conclusion

  • AdvDMD uses the discriminator from the GAN objective as a reward model that provides signals for intermediate denoising steps.The conclusion describes adversarial reward as continuous discriminative feedback during training.Source: §5 · p. 12
  • The adversarial reward guides the generator toward the true data distribution and is intended to promote perceptual realism and structural fidelity.Source: §5 · p. 12
  • The authors identify better integration of distillation and reinforcement learning as future work for high-performance few-step generation.Source: §5 · p. 12

A Ablation Studies

The ablation compares ODE and SDE backward simulation on SD3.5 in a 4-step training setup; the reported DPG-Bench results indicate a slight image-quality improvement with SDE.

A Ablation Studies

  • SDE backward simulation slightly improves image quality over ODE in the SD3.5 4-step training comparison.The comparison reports results on DPG-Bench; Table 7 is captioned as covering performance improvements across DPG-Bench metrics.Source: A Ablation Studies · p. 15, Table 7 · p. 15

B Algorithm

Algorithm 1 alternates between generating N-step trajectories and updating either the generator or discriminator-related parameters. The generator path combines DMD, GAN, and GRPO losses, while the discriminator path updates the discriminator and fake-score parameter.

B Algorithm

  • The procedure takes a generator, real and fake scores, a discriminator, an image-prompt dataset, and a sampling-step count as inputs.Source: B Algorithm · p. 15
  • For each image-prompt pair, the generator produces N-step samples, including intermediate noisy samples via backward SDE simulation, and the discriminator supplies step-wise rewards.Source: B Algorithm · p. 15
  • On generator updates, the algorithm combines DMD and GAN losses with a GRPO loss; the alternative update computes discriminator and diffusion losses and adjusts the fake score and discriminator parameters.Source: B Algorithm · p. 15

C Additional Image Results

The additional figures show AdvDMD generations at different sampling-step counts and compare AdvDMD with other methods. The captions specify that the displayed generations were produced without classifier-free guidance.

C Additional Image Results

Related papers

Loading 2604.28126v1…