PaperlayerView exact revision on arXiv

Source-linked AI summary

SD3.5-Flash: Distribution-Guided Distillation of Generative Flows

Hmrishav Bandyopadhyay, Rahim Entezari, Jim Scott, Reshinth Adithyan, Yi-Zhe Song, Varun Jampani

arXiv:2509.21318v1cs.CVcs.AI

TL;DR

SD3.5-Flash addresses the computational cost of high-quality rectified-flow generation and the instability and prompt-alignment challenges of few-step distillation. It combines trajectory-based timestep sharing, split-timestep fine-tuning, and deployment optimizations; the authors report that it outperforms existing methods across hardware configurations. The authors also report a speed-quality trade-off in complex generation tasks.

  • Problem

    Rectified-flow models require substantial inference steps, memory, and time, while few-step distribution matching can corrupt gradient estimates and weaken prompt-image alignment.

    Source: §1 · p. 1

  • Method

    SD3.5-Flash distills rectified-flow models using student-trajectory timestep sharing, split-timestep fine-tuning, and text-encoder and quantization optimizations.

    Source: §1 · pp. 1–2

  • Results

    SD3.5-Flash consistently outperforms existing methods across diverse hardware configurations; in user studies it outperforms the 50-step teacher in image quality, with prompt-adherence differences under ±1.6% across methods.

    Source: §1 · p. 2, §5.3 · p. 7

  • Takeaways & Limitations

    The authors present tailored model variants for practical deployment across commodity hardware, from mobile devices to desktop computers.

    Source: §1 · p. 2

    The authors report a quality-and-diversity trade-off with inference speed on complex tasks; removing T5 also makes complex compositions harder, and the four-step model offers up to ∼18× teacher speed-up.

    Source: §6 · p. 10

Abstract

from arXiv · show

We present SD3.5-Flash, an efficient few-step distillation framework that brings high-quality image generation to accessible consumer devices. Our approach distills computationally prohibitive rectified flow models through a reformulated distribution matching objective tailored specifically for few-step generation. We introduce two key innovations: "timestep sharing" to reduce gradient noise and "split-timestep fine-tuning" to improve prompt alignment. Combined with comprehensive pipeline optimizations like text encoder restructuring and specialized quantization, our system enables both rapid generation and memory-efficient deployment across different hardware configurations. This democratizes access across the full spectrum of devices, from mobile phones to desktop computers. Through extensive evaluation including large-scale user studies, we demonstrate that SD3.5-Flash consistently outperforms existing few-step methods, making advanced generative AI truly accessible for practical deployment.

1 INTRODUCTION

SD3.5-Flash targets the computational demands and prompt-alignment challenges that limit few-step image generation on consumer devices. It combines timestep sharing, split-timestep fine-tuning, and pipeline optimizations; the authors report that it outperforms existing methods across hardware configurations.

1 INTRODUCTION

  • Rectified flow models’ demands of 25+ steps, 16GB+ VRAM, and 30+ seconds per image put them beyond everyday devices.The introduction frames the work as bridging this access gap, from mobile phones to gaming desktops.Source: §1 · p. 1
  • Timestep sharing computes distribution matching from student-trajectory samples, providing stable gradient signals at known noise levels and reliable ODE-trajectory predictions.The authors say this improves training stability and consequently model performance.Source: §1 · p. 1
  • Split-timestep fine-tuning branches the model across timestep ranges, then merges the branches into one checkpoint to address prompt alignment.The approach temporarily expands model capacity rather than asking compressed parameters to handle aesthetic quality and semantic fidelity simultaneously.Source: §1 · p. 1
  • Pipeline optimizations restructure text encoders and apply 16-bit-to-6-bit quantization to balance memory footprint and inference speed across device tiers.The encoder components include optional T5-XXL and necessary CLIP-L/G components, using encoder-dropout pre-training.Source: §1 · pp. 1–2
  • SD3.5-Flash consistently outperforms existing methods across diverse hardware configurations while maintaining the quality standards of larger, slower models.The authors describe extensive evaluation, including large-scale user studies.Source: §1 · p. 2
  • Figure 1 presents four-step samples illustrating prompt adherence, compositional understanding, anatomy, and multi-object composition on consumer hardware.The caption describes these as areas where previous distillation approaches often struggle.Source: Figure 1 · p. 2

2 RELATED WORKS

Prior work accelerates diffusion through fewer denoising steps, trajectory distillation, consistency models, distribution matching, and model or pipeline optimization. The SD3.5-Flash suite is presented as a user-preferred option across consumer compute budgets.

2 RELATED WORKS

  • Skip-step schedulers reduce inference timesteps with deterministic sampling, while distillation methods learn a more efficient denoising trajectory.The cited approaches include Song et al. (2020a), Karras et al. (2022), and Luhman & Luhman (2021).Source: §2 · pp. 2–3
  • The SD3.5-Flash suite is preferred by users over other models at varied consumer compute budgets while offering comparable latency and memory requirements.Figure 2 uses human-ranking ELO ratings for image quality; bubble size represents VRAM for GPUs and pipeline size on disk for mobile devices.Source: Figure 2 · p. 3
  • Progressive distillation aligns student and teacher trajectories and progressively skips steps, but approximation errors prevent it from learning extreme low-step inference such as two steps.The passage cites Salimans & Ho (2022) and Lin et al. (2024).Source: §2 · p. 3
  • Consistency models learn jumps to trajectory endpoints or intermediate points and support iterative refinement; score-matching approaches instead match teacher and student distributions.The passage distinguishes both approaches from Insta-Flow’s rectified-flow fine-tuning and SWD’s scale-wise distillation.Source: §2 · p. 3
  • Recent few-step methods supplement distillation objectives with adversarial training, comparing student-generated samples against real or teacher-generated samples.Teacher-based discriminative features and multiple lightweight discriminator heads are also described as ways to provide richer feedback and training stability.Source: §2 · p. 3
  • Other efficiency work reduces or replaces attention units or optimizes VAE-based latent decoding, separately from distillation.The passage also identifies text encoders and VAEs as components of the generation pipeline.Source: §2 · p. 3

3 BACKGROUND

The background contrasts score-based diffusion trajectories with flow matching’s directly learned ODE velocity fields. It then describes distribution matching distillation, whose score-based gradient requires estimating teacher and student scores from generated samples.

3 BACKGROUND

  • Diffusion models learn a noise-to-data trajectory, commonly described by an SDE or its probability-flow ODE reformulation, and estimate score functions at different noise levels.The score function is the gradient of the log probability density.Source: §3 · p. 4
  • Flow matching directly learns an ODE-based mapping by parameterizing a velocity field that transports samples from noise to data.Rectified-flow pipelines use a straight path between the data distribution and standard normal noise.Source: §3 · p. 4
  • Distribution Matching Distillation matches the student’s generated-sample distribution to the multi-step teacher’s distribution using KL divergence.The student sample is generated from noise drawn from a standard normal distribution.Source: §3 · p. 4
  • Because the probability densities are generally intractable, DMD substitutes score functions to compute the loss gradient.The score is defined as the gradient of the log probability density.Source: §3 · p. 4
  • DMD obtains teacher and student scores by re-noising generated samples to timestep t and using denoising signals from pretrained diffusion models.The student score is estimated using the student model.Source: §3 · p. 4
  • A multi-step proxy model estimates student scores because few-step models cover only a subset of timesteps; the pipeline also describes regression loss and DMD2’s biased proxy-update schedule and adversarial objective.The regression loss is an MSE between student and teacher images generated from the same noise.Source: §3 · p. 4

4 METHODOLOGY

The methodology combines trajectory guidance, distribution matching, and adversarial training to distill few-step students, then optimizes the inference pipeline for deployment across devices.

4.1 TRAJECTORY GUIDANCE

  • Trajectory guidance pre-trains the four-step student by sampling teacher-trajectory points that coincide with student timesteps and using the teacher velocity predictor.The objective compares the student and teacher velocities at those points.Source: §4.1 · p. 4

4.2 DISTRIBUTION MATCHING IN FLOW MODELS

  • Timestep sharing estimates distribution-matching gradients from partially denoised student-trajectory samples instead of re-noising endpoints, improving composition and generation quality.Training initially uses slightly noisier samples for each timestep, then switches back to the corresponding student-timestep samples after stabilizing.Source: §4.2 · p. 5

4.3 ADVERSARIAL LOSS

  • Adversarial training uses proxy-student features from multiple noise levels for discriminator heads, with teacher-generated samples as real data and periodically refreshed heads.The proxy student is trained with flow matching, while the generator uses distribution-matching and adversarial objectives.Source: §4.3 · pp. 5–6, Figure 3 · p. 5

4.4 TWO STEP AND FOUR STEP GENERATION

  • Two-step generation is obtained by distilling the teacher to four steps, then distilling that student to two steps with distribution matching, adversarial training, and a feature Gram-matrix loss.Trajectory-guidance pre-training aligns teacher and student trajectories before the four-step model is used as the two-step checkpoint.Source: §4.4 · p. 6

4.5 PIPELINE OPTIMIZATION

  • Pipeline optimization reduces the full 16-bit model’s 18 GiB memory requirement to about 8 GiB using 8-bit quantization and null embeddings, with a further 6-bit option for Apple Silicon.Table 1 compares inference latency across devices and lists VRAM or unified memory beneath device names.Source: §4.5 · p. 6, Table 1 · p. 6

5 EXPERIMENTS

The experiments evaluate SD3.5-Flash through user studies, automated metrics, qualitative comparisons, and component ablations. In user ratings, it leads in image quality, while prompt-adherence differences are marginal.

5.1 IMPLEMENTATION DETAILS

  • Synthetic training samples were generated with SD3.5 Large (8B) over 32 timesteps at CFG scale 4.0; after 2K pretraining iterations, 4-step and 2-step models trained 1,200 iterations each using 2.5B SD3.5M as teacher.Source: §5.1 · p. 6

5.2 QUALITATIVE COMPARISONS

  • In qualitative comparisons, 4-step SD3.5-Flash outperforms other 4-step pipelines in fidelity; its 2-step model beats SANA-Sprint but trails its own 4-step model.Comparisons include complex prompts, where competing 4-step pipelines show weaker alignment and composition.Source: §5.2 · p. 7, Figure 5 · p. 8

5.3 USER STUDY

  • SD3.5-Flash outperforms other few-step models and its 50-step teacher in image quality; prompt-adherence differences are marginal (< ±1.6%), and it ranks atop the ELO ladder across compute scenarios.The study used 124 annotators, 507 prompts, and four seeds.Source: §5.3 · p. 7, Figure 6 · p. 8

5.4 QUANTITATIVE COMPARISONS

  • SD3.5-Flash is competitive with SDXL-DMD2 and NitroFusion, surpasses SD3.5M on GenEval, AeS, and IR, but has worse FID.The comparison uses 30K COCO samples; the cited table caption identifies the automated-metric comparison but supplies no cell values.Source: §5.4 · p. 9, Table 2 · p. 9

5.5 ABLATIVE STUDIES

  • Removing adversarial training destabilizes generation; removing pre-training or timestep sharing harms color and composition, while removing discriminator refresh causes minor composition errors and oversmoothing.The ablations use the 16-bit, 4-step SD3.5M distillation setup and match the student model's training iterations.Source: Figure 7 · p. 8, §5.5 · p. 9

6 CONCLUSION

SD3.5-Flash trades some quality and diversity for faster inference in complex tasks. Removing T5 further challenges complex compositions, while the 4-step model is up to about 18× faster than its teacher.

6 CONCLUSION

  • Distillation trades some quality and diversity for inference speed in complex generation tasks.The paper describes this as a tradeoff common to distillation processes.Source: §6 · p. 10
  • Removing T5 reduces inference time and memory use but makes complex compositions harder to construct from weaker conditional context.The limitation is illustrated in Figure 4.Source: §6 · p. 10
  • The 4-step model offers up to ∼18× speed-up over the teacher and surpasses it in average performance in large-scale user studies across prompt complexity levels.Source: §6 · p. 10

A.1 TRAINING

The appendix details distillation from 50-step SD3.5 Medium to 4-step and 2-step models, including the training stages and discriminator schedule.

A.1 TRAINING

  • SD3.5 Medium is distilled from 50 steps to 4 steps, then further trained for 2-step generation; the 2-step model starts from the 4-step checkpoint.Denoising timesteps are distributed uniformly over [0, 1].Source: §A.1 · p. 14
  • The two-stage pipeline pre-trains the generator for 2K iterations, then trains the proxy student, generator, and discriminator; split-timestep fine-tuning adds 400 iterations to the 4-step checkpoint.The pre-training stage takes 17 hours on 8 H100s; stage two takes 6 hours, and split-timestep fine-tuning takes 4 hours.Source: §A.1 · p. 14
  • Discriminator heads are periodically re-initialized with probability p = 0.005 and updated ten times for each generator update.The heads use features from selected proxy-student layers and consist of eight MLP layers each.Source: §A.1 · p. 14

A.2 QUANTIZATION TRADEOFF

The appendix points to three visual analyses: memory versus performance on an M3 MacBook Pro, latency versus GenEval for 4-step pipelines, and prompt-adherence ratings across models.

A.2 QUANTIZATION TRADEOFF

  • Figure 8 analyzes the memory-performance tradeoff for quantized SD3.5-Flash on an M3 MacBook Pro with 32 GiB of memory.Source: §A.2 · p. 14, Figure 8 · p. 14
  • Figure 8 compares latency and GenEval scores for 4-step inference pipelines.The supplied caption does not state the outcome of that comparison.Source: Figure 8 · p. 14
  • Figure 9 presents user ratings of prompt adherence for different models.The supplied passage does not state which model receives higher ratings.Source: Figure 9 · p. 14

A.3 USER STUDY ANALYSIS

The user-study analysis examines 507 prompts, using GPT-4 to categorize them and score their image-generation complexity. It separates image-quality and prompt-alignment judgments to reduce conflation between those preferences.

A.3 USER STUDY ANALYSIS

  • GPT-4 categorized the 507 study prompts and scored their image-generation complexity; Figure 10 shows complexity by prompt length and categories, plus the 15 most frequent labels.The prompt analysis is described alongside the user-study prompt set.Source: §A.3 · p. 15, Figure 10 · p. 15
  • Participants judged image aesthetics with prompts hidden and prompt alignment while disregarding image quality, a more costly setup adopted for clearer results.The authors report that their ablations found it beneficial to disentangle the two preferences because participants otherwise tended to conflate them.Source: §A.3 · p. 15
  • Study participants had prior experience ranking generated images and needed no explicit instructions after multiple rounds of quality checks.Source: §A.3 · p. 15

A.4 ADDITIONAL QUALITATIVE ANALYSIS

The appendix adds qualitative comparisons and four-step model examples, alongside the prompts used for selected figures. These materials cover both comparisons with other few-step models and examples across different aspect ratios.

A.4 ADDITIONAL QUALITATIVE ANALYSIS

  • Additional figures show comparisons of the four-step and two-step models with other distilled models, plus four-step results at different aspect ratios.Figures 13 and 14 provide qualitative comparisons; Figure 15 shows four-step results in different aspect ratios.Source: §A.4 · p. 15, Figure 13 · p. 17, Figure 14 · p. 18

A.5 PROMPT LIST

  • The prompt list for Figures 1, 4, and 15 includes examples such as a cat wearing a hat labeled “Flash” and a fantasy bookstore in a glowing mushroom.Source: §A.5 · pp. 15, 20

Related papers

Loading 2509.21318v1…