PaperlayerView exact revision on arXiv

Source-linked AI summary

A First-Step Bias Restores Lost Diversity in Diffusion Model Distillation

Wenwu Tang, Dong Wang, Yun Cheng, Olga Saukh

arXiv:2610.37110v1cs.CVcs.AI

TL;DR

Few-step distribution-matched students can produce similar outputs from different initial noises, limiting the variety users obtain. ReSeed adapts only the first denoising step of a frozen student, and its evaluation reports diversity gains across image and video models, including a 48.3% DINOv3 diversity increase on a 4-step FLUX.1-dev student.

  • Problem

    Distribution-matched distillation reduces generation to a few denoising steps, but students often produce similar outputs from different initial noises under the same prompt.

    Source: §1 · p. 2

  • Method

    ReSeed trains small bias adapters only for the first student step to match the teacher’s average velocity over that interval, while keeping student weights and later steps unchanged.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    DINOv3 diversity rises by 37.1%, 39.3%, and 48.3% on SD3-Medium, SD3.5-Medium, and SenseFlow-FLUX, respectively; video-level DINOv2 diversity rises by 27.1% on Wan2.1-T2V-1.3B DMD2.

    Source: §7 · p. 9

  • Takeaways & Limitations

    ReSeed offers a lightweight post-distillation correction that can improve diversity without retraining the student.

    Source: §1 · p. 2, §7 · p. 9

    Experiments use flow-matching transformers; non-flow-matching diffusion models and U-Net backbones remain untested, so whether the same first-step behavior holds for them is unclear.

    Source: §8 · p. 9

Abstract

from arXiv · show

Distribution-matching distillation enables high-quality image generation in only a few denoising steps. However, distilled student models often map different initial noise samples to similar outputs under the same prompt. To locate where diversity is lost, we compare the teacher and student along the sampling trajectory. The largest measured reduction in the response to different initial noises occurs during the first, high-noise step. When we replace the first step with the teacher's corresponding trajectory and leave the final three student steps unchanged, most of the output diversity returns. This suggests that the first step is a key point for restoring diversity. Motivated by this finding, we introduce \emph{ReSeed}, a post-distillation method that adds a seed-response bias adapter only to the first student step and aligns its velocity with the teacher's first-interval secant velocity. All student weights and later denoising steps remain unchanged. We evaluate ReSeed on three text-to-image models (SD3-Medium, SD3.5-Medium, and FLUX.1-dev) and one text-to-video model (Wan2.1-T2V-1.3B). For a distilled 4-step FLUX.1-dev model, \emph{ReSeed} requires only $0.00196\%$ additional trainable parameters relative to the base model and is trained on 8,000 prompt--noise pairs, while keeping the distilled student fully frozen and requiring no target images, perceptual losses, or adversarial losses. It improves DINO diversity by $48.3\%$ while maintaining competitive visual quality. These results show that substantial diversity can be recovered through a lightweight, data-efficient tuning of only the first denoising step, without finetuning the full student model.

1 Introduction

Few-step distribution-matched students can lose output diversity across initial noises, motivating a post-distillation repair that changes only the first denoising step. ReSeed applies a small first-step adapter while keeping student weights and later steps fixed, recovering diversity with few trainable parameters.

1 Introduction

  • Distilled students often produce similar outputs from different initial noises under the same prompt, limiting the range of results users can obtain.Distribution matching distillation enables generation in only a few denoising steps.Source: §1 · p. 2
  • The paper asks whether diversity can be recovered in an already distilled student by changing only its first denoising step while fixing its weights and later steps.Earlier approaches address initial-noise response, use different objectives for early and late steps, or use the base model for the first step.Source: §1 · p. 2
  • ReSeed trains small bias adapters for the first student step, matching its velocity to the teacher’s average velocity over the same interval while freezing original weights.At inference, biases are enabled only in step 1; later student steps remain unchanged and require no teacher calls.Source: §1 · p. 2
  • DINOv3 diversity rises by 48.3% on the 4-step FLUX.1-dev student, while FID decreases from 40.22 to 34.65.ReSeed uses 0.00196% of denoiser parameters and 8,000 prompt–noise pairs; other evaluated image and video models also show diversity gains, with trade-offs in some quality metrics.Source: §1 · p. 2
  • ReSeed repairs existing distilled models through lightweight post-hoc tuning, without repeating distillation or fine-tuning the whole student.The contribution is framed as a practical way to enhance diversity in existing few-step generative models.Source: §1 · p. 2

2 Preliminaries

The preliminaries frame sampling as integration of a learned velocity field and describe how distillation matches the student and teacher output distributions. They define seed response as the effect of initial noise on the first-step state and final output.

2 Preliminaries

  • Flow matching directly regresses a velocity field, while diffusion models derive velocity from their predicted score; rectified flow uses a straight path between clean latent and Gaussian noise.The path’s constant velocity is the difference between noise and the clean latent.Source: §2 · pp. 2–3
  • Sampling integrates the learned velocity from pure noise to data using Euler steps on a decreasing time grid.For a fixed prompt, the deterministic sampler’s output differences come from the initial noise.Source: §2 · p. 3
  • Distribution matching distillation trains a few-step student to match the teacher’s output distribution, using score estimates of re-noised samples rather than directly evaluating reverse KL.The update uses a frozen teacher score and an auxiliary model fitted online to student samples; reverse KL penalizes student samples the teacher finds unlikely.Source: §2 · p. 3
  • Seed response is the output change caused by changing initial noise, decomposed into the first-step mapping and the remaining sampling steps.This decomposition tests whether first-step correction can restore diversity while later steps stay unchanged.Source: §2 · p. 3

3 First-step correction restores diversity

On the 4-step SD3-Medium student, first-step replacement recovers diversity while leaving the final three student steps unchanged. Across checkpoints, better alignment of first-step seed responses accompanies higher final-image diversity.

3.1 First-step replacement restores diversity

  • DINOv3 diversity increases from 0.2142 to 0.3117 when the teacher replaces the first student step and the remaining three student steps stay unchanged.Replacing additional intervals brings smaller further gains, and the teacher replacement requires extra evaluations, serving as a diagnostic rather than an equal-cost alternative.Source: §3.1 · p. 3, Figure 2 · p. 4
  • The unchanged final three steps can produce diverse images from a different first-boundary state, identifying the first step as an effective repair point in this setting.For an Euler step, matching that state is equivalent to matching the student velocity to the teacher’s average velocity over the interval.Source: §3.1 · p. 3
  • Figure 2 reports that replacing only the first step yields 91% of the diversity gain, while CLIPScore changes little until later steps are replaced.With steps 2–4 fixed, teacher replacement raises diversity for 98 of 100 prompts and ReSeed moves them toward the teacher-prefix result.Source: Figure 2 · p. 4

3.2 Better seed-response alignment accompanies higher diversity

  • Mean seed-response error falls by a factor of 2.7 as final-image DINOv3 diversity rises from 0.213 to 0.290.Across checkpoints, lower error accompanies higher diversity for 90% of prompts; this is an association, not a final-image guarantee.Source: §3.2 · p. 4

4 ReSeed: a seed-response bias for the first step

ReSeed adds bias adapters to the first student step while freezing the backbone and leaving later steps unchanged. It trains that step toward the teacher’s average velocity over the same interval, with a bound connecting the training objective to first-endpoint seed-response error.

4 ReSeed: a seed-response bias for the first step

  • ReSeed adds block-output bias adapters only for the first student step, freezes the backbone, and disables the adapters in steps 2–4.The overview depicts the teacher and ReSeed retaining alternative seed paths where first-step contraction can merge trajectories.Source: §4 · p. 4, Figure 3 · p. 5
  • Training matches the biased student’s first-step velocity to the teacher’s average velocity over the same interval, then uses an Euler update for the adapted endpoint.The target is a displacement across the interval, not the teacher’s instantaneous velocity at its last step.Source: §4 · p. 4
  • The pointwise regression objective bounds expected same-prompt pairwise response error at the first endpoint, but does not bound final-image diversity.The paper tests the relationship to final-image diversity empirically; lower endpoint response error is associated with higher DINOv3 diversity.Source: §4 · p. 5
  • Figure 4 relates seed-response error to DINOv3 diversity on COCO5K, with a reversed error axis and checkpoint means shown in black.The paper reports that lower pairwise endpoint response error is associated with higher final-image DINOv3 diversity.Source: Figure 4 · p. 5, §4 · p. 5
  • For SD3-Medium, ReSeed uses 72,192 parameters, or 0.00356% of the 2,028,328,000 student parameters, in added zero-initialized residual vectors.The per-channel vectors are added to transformer block outputs; unlike BitFit, ReSeed does not unfreeze existing model biases.Source: §4 · pp. 5–6

5 Experiments and Results

Experiments evaluate ReSeed across image and video students, comparing quality and diversity against frozen distilled models and other baselines. Diversity improves across the evaluated models, with trade-offs in some quality measures and small training and inference costs.

5.1 Experimental setup

  • Evaluation covers three 4-step image students and a 4-step video student, using 5,000 COCO captions for image quality, 100 prompts with 10 noises for image diversity, and 320 VBench prompts for video.Main image results use three training seeds and video uses one.Source: §5.1 · p. 6

5.2 Text-to-Image Generation

  • DINOv3 diversity rises from 0.2142 to 0.2936 (+37.1%) on SD3-Medium and from 0.2356 to 0.3282 (+39.3%) on SD3.5-Medium, with FID improving on both.PickScore and HPS fall on both models, while aesthetic scores change little.Source: §5.2 · pp. 6–7, Table 1 · p. 7
  • DINOv3 diversity on SenseFlow-FLUX rises from 0.1575 to 0.2336 (+48.3%), recovering 89.8% of the measured teacher gap, while FID improves from 40.22 to 34.65.Most diversity recovery occurs within 300–400 updates; CLIP score also improves, while PickScore, HPS, and aesthetic score decrease slightly.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Extension to Text-to-Video Generation

  • Video-level DINO diversity rises from 0.3374 to 0.4290 (+27.1%), recovering 56.2% of the measured teacher–student gap, while the teacher remains higher at 0.5005.Video-level CLIP diversity and per-frame DINO diversity also rise; some quality measures decline, and diversity alone does not establish better prompt-consistent coverage.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 Training and inference overhead

  • Adapters train only 0.002–0.004% of denoiser parameters, and inference adds 0.14–0.41% per-sample latency with at most 1 MiB more peak activation memory.The bias is applied only at step 1, leaving the number of function evaluations unchanged.Source: §5.4 · p. 8, Table 4 · p. 9

5.5 Ablation Studies

  • Increasing bias strength from α = 0.5 to 1.5 raises diversity and lowers FID, but reduces CLIP score, PickScore, and HPS.Bias strength allows the quality–diversity balance to be adjusted without retraining.Source: §5.5 · p. 8
  • Increasing teacher guidance from 1 to 10 reduces DINOv3 diversity from 0.3132 to 0.2873 while generally increasing alignment and preference scores.The default uses the student’s distillation guidance rather than maximizing diversity.Source: §5.5 · p. 9

6 Related work

Related work addresses few-step generation, diversity loss, and parameter-efficient adaptation through distinct approaches. ReSeed is positioned as a post-distillation correction that changes only the frozen student’s first step.

6 Related work

  • Few-step distillation methods progressively halve sampler steps, learn consistency maps, match output or trajectory distributions, or add adversarial objectives.This paper studies diversity loss in students that have already been distilled.Source: §6 · p. 9
  • Prior diversity approaches modify distillation objectives, use the base model for the first step, correct video samples, or perturb conditioning at inference; ReSeed instead adjusts only the first step of a frozen student post-distillation.The cited base-model approach attributes part of diversity loss to premature structural commitment [Gandikota and Bau, 2025].Source: §6 · p. 9
  • Unlike LoRA, BitFit, or timestep-specific LoRA experts, ReSeed adds residual bias only during the first solver interval and uses it as both intervention and diagnostic.The passage cites LoRA [Hu et al., 2022], BitFit [Ben Zaken et al., 2022], and TimeStep Master [Zhuang et al., 2025].Source: §6 · p. 9

7 Discussion & Conclusion

ReSeed repairs a distilled student by tuning residual biases for its first denoising step while leaving later steps unchanged. Across image and video models, it improves measured diversity with a small post-distillation parameter update.

7 Discussion & Conclusion

  • ReSeed trains small residual biases to match the teacher’s average velocity over the first high-noise interval, leaving later student steps unchanged.Across SD3-Medium, SD3.5-Medium, and SenseFlow-FLUX, DINOv3 diversity rises by 37.1%, 39.3%, and 48.3%; Wan2.1 video-level DINOv2 diversity rises by 27.1%.Source: §7 · p. 9

8 Limitations

The evaluation is limited to flow-matching transformers, and the training procedure requires multi-step teacher trajectories.

8 Limitations

  • Experiments cover flow-matching transformers only; whether the same first-step behavior holds for non-flow-matching diffusion models or U-Net backbones remains unclear.ReSeed training also requires multi-step teacher trajectories.Source: §8 · p. 9

A Implementation and evaluation details

The appendix specifies the image and video models, ReSeed training targets, and evaluation protocols. Video training uses caption text and embeddings rather than target videos, while evaluation covers 320 prompts and eight seeds.

A Implementation and evaluation details

  • SD3-Medium uses a 4-step DMD student distilled from a 28-step teacher, while SD3.5-Medium and SenseFlow-FLUX use 4-step students paired with 40- and 28-step teachers.The SD3-Medium teacher uses guidance scale 7 and the student uses scale 1; the SD3.5-Medium comparison is at 1024 pixels and CFG 4.5.Source: A Implementation and evaluation details · p. 14
  • Image adapters are trained against teacher targets generated from the same prompt and initial noise, with reported checkpoints using 7,600, 8,400, and 8,000 prompt–noise pairs.Image evaluation uses fixed prompt–latent pairs for quality and a separate bank of 100 prompts with ten noises for diversity, with shared prompts, noises, and scorers within comparisons.Source: A Implementation and evaluation details · p. 15
  • Image diversity is scored with DINOv3 and video diversity with DINOv2-L, so their absolute values are not directly comparable across modalities.Reported overhead measurements use one H100; bias storage is at most 456 KiB.Source: A Implementation and evaluation details · p. 15

A.1 Video training data and evaluation protocol

  • Video training uses 4,000 caption texts and precomputed embeddings without target videos, and the 300-update checkpoint uses 1,200 prompt–noise pairs.Evaluation covers 320 unique VBench prompts, with eight seeds per method for 2,560 videos.Source: §A.1 · p. 15
  • Video ReSeed targets the teacher’s secant velocity over the first student interval, using 13 teacher Euler microsteps; the learned biases are enabled only for the first student step.The target differs from the 50-step UniPC sampler used to evaluate the teacher.Source: §A.1 · p. 15

B.1 Additional video results

Video evaluation finds diversity gains over DMD2 across categories and feature choices, alongside trade-offs against the teacher and DFD. Additional analyses describe category patterns, teacher-guidance trends, and the scope of bias-scaling controls.

B.1 Additional video results

  • ReSeed increases mean DINO diversity over DMD2 in every VBench category but remains below the teacher in every category.The largest increases are in lifestyle and human prompts, and the smallest is in scenery; these feature-space distances are not counts of semantic modes.Source: §B.1 · p. 16, Table 6 · p. 16
  • +0.0170 projected-CLIP diversity over DMD2 (95% interval [+0.0119, +0.0221]) is observed for ReSeed across the 320-prompt video set.The aggregate gain also holds with pre-projection CLS features: mean diversity is 0.1835 for DMD2 and 0.2105 for ReSeed, a paired increase of +0.0270.Source: §B.1 · p. 16
  • DFD’s DINO difference from DMD2 is −0.0016 (interval [−0.0090, +0.0059]), while its projected-CLIP difference is +0.0084 (interval [+0.0047, +0.0116]).ReSeed exceeds DFD in DINO diversity by 0.0932 (interval [+0.0830, +0.1035]), with text alignment lower by 0.0033 (interval [−0.0053, −0.0012]).Source: §B.1 · p. 16

C.1 Teacher guidance

  • With higher teacher guidance, diversity falls and alignment generally increases, while FID is best at the lowest guidance.The comparison holds the student, sampler, prompts, seed, and block range fixed while changing the guidance used to build the target.Source: Table 7 · p. 17

C.2 Bias strength and controls

  • Table 8 covers COCO5K quality and matched diversity for inference-time bias scaling, using one seed-42 ReSeed checkpoint and reference student and teacher models.Channel-shuffled bias and the blocks 4–23 variant have diversity evaluations only and are excluded from the table.Source: §C.2 · p. 17, Table 8 · p. 17

D Training dynamics

Training trajectories show early loss reduction and improving prompt-level diversity for FLUX.1-dev. The theoretical analysis connects first-endpoint velocity regression to pairwise seed-response error, while qualifying the bound’s finite-sample behavior.

D Training dynamics

  • The largest ReSeed training-loss decrease occurs early, followed by a slower decline; these curves measure target fitting, not final-image quality or diversity.Figure 8 also tracks bias magnitude and cosine similarity with the final trained bias.Source: D Training dynamics · p. 17, Figure 8 · p. 17
  • 95 of 100 FLUX.1-dev prompts improve in DINOv3 diversity, from 0.158 to 0.242 on average, while CLIP returns to its initial mean (0.33748→0.33753).The trajectories compare each prompt’s checkpoint value with the original student and show prompt means.Source: D Training dynamics · p. 17, Figure 10 · p. 18

E.1 From velocity regression to pairwise endpoint error

  • Expected squared velocity error and expected squared first-endpoint error differ by the known factor (t1−t0)^2.The pairwise bound follows because a seed-shared endpoint error cancels from pairwise responses, and the inequality drops a nonnegative mean-error term.Source: §E.1 · p. 20

E.2 Bound tightness and finite-sample estimates

  • The ratio of pairwise response error to its population upper bound is exactly 1 − s(c), the fraction of squared endpoint error not due to the mean error across seeds.With ten noises, the corrected empirical upper bound is 10 bB/9, so a finite-bank estimate can exceed the uncorrected bB without violating the population result.Source: §E.2 · p. 20

E.3 Empirical experiments and the relation to output diversity

On COCO5K, lower pairwise response error accompanies higher final-image diversity across ReSeed checkpoints, but the reported relationship is associational. The analysis describes how shared internal biases can alter seed responses and includes matched-noise qualitative comparisons.

E.3 Empirical experiments and the relation to output diversity

  • Mean pairwise response error falls by a factor of 2.7 as mean DINOv3 diversity rises from 0.213 to 0.290 (+36.5%), with diversity gains on 92% of prompts.Across checkpoints, lower response error consistently accompanies higher mean diversity; the within-prompt association is negative for 90% of prompts, not evidence of causation.Source: §E.3 · p. 21, Table 9 · p. 21

E.4 Why a shared bias can change seed-dependent responses

  • A shared internal bias can produce seed-specific endpoint changes because the response Jacobian depends on the input, but the local result does not guarantee improved alignment or diversity at the trained bias.The paper states that the link to final-image diversity comes from matched evaluations and the teacher-prefix intervention, not from either proposition alone.Source: §E.4 · pp. 21–22

F Qualitative comparisons

  • For the latte-art prompt, video-level DINO diversity is 0.122 for the teacher, 0.058 for DMD2, and 0.245 for ReSeed.The frame strips match seeds across the 50-step teacher, 4-step DMD2 student, and ReSeed student.Source: Figure 15 · p. 28

Related papers

Loading 2610.37110v1…