Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models

Motivation

• Pre-trained models can provide useful knowledge when large amounts of high-quality training data are difficult to access.The paper frames model-driven learning as an efficient way to support new tasks with a base model that can be further improved.Source: §1 · p. 2
• Knowledge transfer from diffusion models to implicit generators is difficult because implicit models lack the explicit multi-level score information used by diffusion models.Implicit generators are valued for flexibility and efficient generation, but receiving supervision from diffusion score networks is technically challenging.Source: §1 · p. 2

Method

• Diff-Instruct instructs arbitrary implicit generative models whose generated samples are differentiable with respect to their parameters.The framework can operate across model types, including GAN generators and fixed-time U-Nets for single-step diffusion generation.Source: §1 · p. 2
• Diff-Instruct directly minimizes Integral Kullback-Leibler divergence and connects to DreamFusion and generative adversarial training.The paper also identifies the SDS objective as a special case when the generator outputs a Dirac’s Delta distribution.Source: §1 · p. 2

Results

• Diff-Instruct achieves state-of-the-art single-step diffusion-based models on ImageNet 64×64 and consistently improves pre-trained GAN generators across settings.The diffusion results span diffusion distillation and direct training baselines.Source: §1 · p. 2

Diffusion models

• Diffusion models transform data distributions toward simple noise through a forward diffusion process and learn marginal score functions.Samples are generated by simulating a stochastic differential equation using the learned score network.Source: §2 · p. 3
• SDE simulation makes diffusion-model sampling significantly slower than sampling from implicit models.This speed difference motivates interest in transferring diffusion knowledge to more efficient generators.Source: §2 · p. 3

Generative adversarial networks

• GANs map easy-to-sample latent vectors to data-space samples, enabling efficient generation but making training challenging through adversarial optimization.A discriminator distinguishes real and generated samples, while generator and discriminator objectives are optimized alternately.Source: §2 · p. 3
• Certain GAN objectives correspond to minimizing divergences between generated and data distributions, including KL divergence.The paper later establishes an equivalence between KL-minimizing adversarial training and Diff-Instruct.Source: §2 · p. 3

Neural radiance fields

• Neural radiance fields map mesh coordinates to volume properties, and differentiable rendering permits parameter updates from rendered two-dimensional images.This illustrates how differentiable generated outputs can receive external instruction.Source: §2 · p. 3

Optimization

• The Diff-Instruct gradient updates generator parameters by integrating score-based instructions over diffusion time levels.The pipeline applies the resulting gradient to the generator, while the diffusion model acts as the teacher.Source: Figure 1 · p. 4, §3.1 · p. 5, §3.2 · p. 5

Instruction criterion

• IKL integrates KL divergence across diffusion time levels, combining multi-level knowledge from the pre-trained diffusion model.The divergence is defined using marginal densities from the same diffusion process and is nonnegative, with equality only when the distributions match almost everywhere.Source: §3.1 · p. 4
• IKL remains finite for some distributions with disjoint supports where vanilla KL divergence is infinite.This robustness motivates using diffusion processes to compare distributions whose supports are misaligned.Source: §3.1 · p. 5

Framework

• Diff-Instruct trains an implicit generator to match a pre-trained diffusion model’s distribution without using training data.The generator samples are differentiated with respect to model parameters, and the pre-trained model supplies distributional knowledge through marginal score functions.Source: §3.2 · p. 5, §3.4 · pp. 6–7

Connections

• For a generator producing a Dirac distribution, Diff-Instruct reduces to the score distillation sampling gradient used in DreamFusion.Under this assumption, no separate diffusion model is needed to estimate the generator’s marginal scores.Source: §3.3.1 · p. 6
• Unlike adversarial training, Diff-Instruct uses a pre-trained diffusion model instead of a discriminator and can distill knowledge without real or synthetic data.The paper also contrasts IKL with KL-based degeneration and discusses potential advantages regarding mode-drop issues.Source: §3.3.2 · p. 6, §3.4 · pp. 6–7

Universality

• Diff-Instruct supports diverse generator architectures, including CNN-, Transformer-, and U-Net-based generators, as well as GAN generators.Its experiments apply knowledge transfer to both U-Net-based and GAN generators.Source: §3 · p. 3, §3.4 · p. 7

4.1 Single-step diffusion distillation

• Diff-Instruct distills pre-trained EDM models into single-step generators on CIFAR10 and ImageNet 64×64, evaluating FID and IS.FID is lower-is-better, while IS is higher-is-better.Source: §4.1 · p. 7, §4 · p. 7
• On ImageNet 64×64, Diff-Instruct outperforms diffusion-based single-step models in FID, including methods using real data, synthetic data, or training from scratch.The comparison includes both diffusion distillation and direct-training methods.Source: §4.1 · p. 8
• Without synthetic or real datasets, Diff-Instruct achieves competitive distillation performance across FFHQ-64, ImageNet-64, and CIFAR10.Figure 2 presents non-cherry-picked samples from one-step generators on these datasets.Source: Figure 2 · p. 8, §4.1 · p. 9
• 1 NFE yields FID 4.19 versus 15.56 for the teacher at 10 NFEs, corresponding to at least 10+ times acceleration.The student uses the same UNet architecture as the teacher, so NFE represents inference-time cost.Source: §4.1 · p. 9
• Diff-Instruct FID converges within 7k iterations, whereas consistency distillation does not converge in fewer than 7k iterations.The authors suggest that the one-step student avoids learning across multiple time indexes.Source: §4.1 · p. 9, Figure 3 · p. 9

4.2 Improving generative adversarial networks

• Diff-Instruct consistently improves pre-trained StyleGAN-2 generators, reducing conditional FID from 2.42 to 2.27 and unconditional FID from 2.92 to 2.71.The experiments use pre-trained EDM models as instructors and StyleGAN-2 models initialized after adversarial training.Source: §4.2 · pp. 9–10

• Diff-Instruct enables data-free knowledge transfer from pre-trained diffusion models into generic generators.Source: §5 · p. 10
• Using multiple pre-trained diffusion models as instructors remains an uninvestigated direction.Source: §5 · p. 10
• The benefits of combining Diff-Instruct with real training data have not yet been explored.Source: §5 · p. 10
• With data but no pre-trained diffusion model, the framework could be adapted by training a teacher diffusion model from data while instructing the student.Source: §5 · p. 10

• IKL is more robust to misaligned density support than traditional KL divergence.Source: §A.1 · p. 16
• IKL remains finite and reliable for gradient-based training where traditional KL divergence fails to guide the generator to θ = 0.Source: §A.1 · p. 16
• For the example distributions Pθ and P0, a suitable weighting makes IKL a differentiable quadratic with its sole minimum at θ = 0.Source: §A.1 · p. 16
• The appendix compares IKL with KL divergence and Wasserstein distance, concluding that IKL better supports gradient-based optimization of θ.Source: §A.1 · p. 16, Table 5 · p. 17
• In practice, Diff-Instruct uses an inverted U-shaped weighting function also used for training the pre-trained diffusion models.Source: §A.1 · p. 16

• The generator distribution q(t) is represented by sampling z and Wiener-process trajectories, then evolving x0 = gθ(z) through the forward SDE.Source: §A.2 · p. 17
• The teacher distribution p(t) is defined through the pre-trained diffusion model’s score functions sp(t).Source: §A.2 · p. 17
• The proof decomposes the IKL derivative into terms and shows that term B vanishes.Source: §A.2 · pp. 17–18
• Differentiating under the integral requires existence of the parameter derivative and an integrable dominating function.Source: §A.2 · p. 18
• Common forward diffusion processes instantiate the trajectory with Gaussian noise as xt = α(t)x0 + β(t)ϵ.Source: §A.2 · p. 18

• When the generator distribution is a Dirac delta, its conditional and marginal distributions coincide, simplifying the score calculation.Source: §A.3 · p. 18
• The general Diff-Instruct gradient reduces to the SDS gradient when the generator outputs a single differentiable datum.Source: §A.3 · pp. 18–19
• Using xt = α(t)x0 + σ(t)ϵ and ϵp(x,t) := −sp(t)(xt)/σ(t), the objective recovers DreamFusion’s SDS gradient estimation.Source: §A.3 · pp. 18–19
• A fixed-view NeRF can be interpreted as a generator producing a Dirac delta distribution, linking DreamFusion to approximated Diff-Instruct distillation.Source: §A.3 · p. 19

• With an optimal discriminator, adversarial training minimizes the KL divergence between the generator and data distributions.Source: §A.4 · p. 19
• The resulting gradient uses the difference between data and generator score functions.The score functions are defined as sd(x)=∇xlogpd(x) and sg(x)=∇xlogpg(x).Source: §A.4 · p. 19
• This gradient is equivalent to Diff-Instruct with w(0)=1 and w(t)>0 for all t>0.The equivalence depends on the discriminator being trained to optimality.Source: §A.4 · p. 19

• Table 6 compares diffusion distillation methods using efficiency and flexibility as evaluation dimensions.Source: Table 6 · p. 19
• Efficiency denotes the training efficiency of diffusion models.Source: Table 6 · p. 19
• Flexibility denotes whether the student model must have the same input-output dimensions.Source: Table 6 · p. 19

Experimental setup

• EDMs are selected as instructors because they achieve state-of-the-art generative performance on benchmarks including CIFAR10 and ImageNet 64×64.Source: B More on experiments · pp. 19–20

Hyperparameters

• Table 7 lists the hyperparameters used for Diff-Instruct diffusion distillation.Source: Table 7 · p. 20

Visualization

• Figure 4 illustrates training with fixed latent vectors across changing generator weights and shows one-step samples distilled from a class-conditional CIFAR10 EDM.Source: Figure 4 · p. 20

Diffusion process

• Forward-process samples are generated by adding Gaussian noise to generator outputs as xt=x0+σ(t)ϵ.Here ϵ∼N(0,I).Source: B More on experiments · p. 20

Experimental setup

• The experiments transfer EDM knowledge to implicit generative models, specifically UNet and StyleGAN generators.Source: B More on experiments · p. 20

Evaluation

• FID and IS are computed using the StyleGAN2-ADA codebase, with ImageNet 64×64 preprocessing matched to EDM.Source: B More on experiments · p. 20

Diffusion distillation

• For UNet students, Diff-Instruct provides an alternative diffusion-distillation method competing with progressive and consistency distillation.Source: §B.1 · p. 20

Initialization

• The generator can be initialized from a pre-trained diffusion model's score network through a Tweedie-based data-prediction transform at fixed time t∗.Source: §B.1 · p. 21
• The initialized generator takes a zero-mean Gaussian latent vector with variance σ^2(t∗).Source: §B.1 · p. 21

Optimization

• Diff-Instruct alternates updates to an auxiliary diffusion model and the generator, keeping the other model fixed during each update.This makes the backpropagation graph memory cost nearly that of one model.Source: §B.1 · p. 21

Computational costs

• 10.40 over 9.55 peak GPU memory is reported for Diff-Instruct versus CD, while the method incurs only minor additional memory cost.Source: §B.1 · p. 21
• 0.0728 seconds versus 0.0489 seconds per 1K iterations is reported for Diff-Instruct versus CD, corresponding to about 1.5 times CD's wall-clock time per iteration.Source: §B.1 · p. 21

Results

• Figure 5 reports FIDs of 2.27, 2.71, 6.62, and 7.56 for the generated samples from left to right.Source: Figure 5 · p. 22

GAN improvement

• GAN-improvement experiments use StyleGAN-2 generators and pre-trained VP-architecture EDMs on CIFAR10 under conditional and unconditional settings.Source: §B.2 · p. 22