PaperlayerView exact revision on arXiv

Source-linked AI summary

SenseFlow: Scaling Distribution Matching for Flow-based Text-to-Image Distillation

Xingtong Ge, Xin Zhang, Tongda Xu, Yi Zhang, Xinjie Zhang, Yan Wang, Jun Zhang

arXiv:2506.00523v2cs.CV

TL;DR

Vanilla DMD has difficulty distilling large flow-based text-to-image models, despite prior success on standard diffusion models. SenseFlow scales DMD with IDA and ISG and adds a vision-foundation-model discriminator; the authors report superior four-step results across diffusion and flow-matching teachers.

  • Problem

    Few distillation methods have demonstrated effectiveness across larger flow-based text-to-image models such as SD 3.5 Large and FLUX.1 dev.

    Source: §1 · p. 2

  • Method

    SenseFlow combines IDA to constrain divergence between the generator and fake model, ISG to relocate teacher timestep importance, and a vision-foundation-model discriminator.

    Source: §1 · p. 2

  • Results

    SenseFlow attains superior overall 4-step results on human-preference and compositional benchmarks across SDXL, SD 3.5 Large, and FLUX.1 dev.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors report stable and effective few-step generation across diffusion and flow-matching backbones.

    Source: §6 · p. 10

    The VFM discriminator raises FID-T, which the authors attribute to a semantic prior that may reduce sample variance in less-favored regions.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

The Distribution Matching Distillation (DMD) has been successfully applied to text-to-image diffusion models such as Stable Diffusion (SD) 1.5. However, vanilla DMD suffers from convergence difficulties on large-scale flow-based text-to-image models, such as SD 3.5 and FLUX. In this paper, we first analyze the issues when applying vanilla DMD on large-scale models. Then, to overcome the scalability challenge, we propose implicit distribution alignment (IDA) to constrain the divergence between the generator and the fake distribution. Furthermore, we propose intra-segment guidance (ISG) to relocate the timestep denoising importance from the teacher model. With IDA alone, DMD converges for SD 3.5; employing both IDA and ISG, DMD converges for SD 3.5 and FLUX.1 dev. Together with a scaled VFM-based discriminator, our final model, dubbed \textbf{SenseFlow}, achieves superior performance in distillation for both diffusion based text-to-image models such as SDXL, and flow-matching models such as SD 3.5 Large and FLUX.1 dev. The source code is available at \href{https://github.com/XingtongGe/SenseFlow}{https://github.com/XingtongGe/SenseFlow}

1 INTRODUCTION

DMD has worked well for standard diffusion models, but distilling larger flow-based models remains difficult. SenseFlow scales DMD with IDA and ISG, and combines them with a vision-foundation-model discriminator for reported gains across diffusion and flow-matching models.

1 INTRODUCTION

  • The paper builds on DMD’s prior success with standard diffusion models such as SD 1.5 and SDXL, while targeting larger flow-based models.Few methods had demonstrated effective distillation across larger models such as SD 3.5 Large and FLUX.1 dev.Source: §1 · p. 2
  • Vanilla DMD2 struggles to converge on large-scale text-to-image models even with TTUR, motivating implicit distribution alignment to constrain divergence between the generator and fake model.IDA is described as a lightweight proximal update after each generator step that nudges the fake model toward the generator.Source: §1 · p. 2
  • Intra-segment guidance relocates the teacher’s timestep-wise denoising importance to sparse supervised anchors, helping the generator approximate complex transitions.For each coarse timestep, the teacher denoises to a sampled intermediate timestep and the generator continues from there.Source: §1 · p. 2
  • SenseFlow achieves state-of-the-art distillation performance on large flow-matching models, including SD 3.5 and FLUX.1 dev, and diffusion-based models such as SDXL.The authors also highlight a more powerful vision-foundation-model discriminator with timestep-aware adversarial signals.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries describe diffusion sampling as solving a probability-flow ODE and frame DMD as matching a generator to a teacher distribution through an intermediate fake distribution.

2.1 DIFFUSION MODELS

  • Diffusion models perturb data toward Gaussian noise through a forward SDE, learn a score function, and generate samples by solving a probability-flow ODE.DDPM and flow-matching optimal transport fit within this framework through different choices of drift and diffusion parameters.Source: §2.1 · p. 3
  • Because the backward equation is intractable with a neural-network score, diffusion sampling usually requires time-consuming multi-step solvers.The paper focuses on distilling the backward-equation solution into another neural network.Source: §2.1 · p. 3

2.2 DISTRIBUTION MATCHING DISTILLATION

  • DMD trains a generator distribution to match a pretrained diffusion model’s distribution using score distillation.The teacher is represented by its learned score function and distribution.Source: §2.2 · p. 3
  • DMD introduces an intermediate fake distribution and alternates optimization of it and the generator; the fake distribution is parameterized by a score network.The generator is a clean-image network followed by a forward diffusion process, and its updates use gradient descent while the fake model uses score matching.Source: §2.2 · p. 3

3 METHOD: SCALING DISTRIBUTION MATCHING FOR GENERAL DISTILLATION

The method addresses vanilla DMD’s stability, timestep-sampling, and discriminator limitations with IDA, ISG, and a vision-foundation-model discriminator. The training loop combines these components with DMD and updates the fake model and discriminator alongside the generator.

3.1 BOTTLENECKS IN VANILLA DMD: STABILITY, SAMPLING, AND NAIVE DISCRIMINATOR

  • Vanilla DMD faces unstable convergence at large model scales, sparse timestep supervision that ignores varying timestep importance, and a discriminator with limited generality.TTUR that works for SD 1.5 and SDXL fails to converge stably on SD 3.5 Large and FLUX.Source: §3.1 · p. 4

3.2 IMPLICIT DISTRIBUTION ALIGNMENT (IDA) FOR FLOW-BASED MODELS

  • IDA applies a proximal alignment step after each generator update to keep the fake distribution near the generator, supporting an ε-best inner response.The paper states that increasing TTUR can be expensive and still oscillate at large scale; IDA with a relatively small 5:1 TTUR is reported to yield more stable convergence and reduce FID variance.Source: §3.2 · pp. 4–5
  • IDA transfers generator progress at a small set of anchor timesteps to the fake model, while samples from an SD 3.5 Medium generator remain consistent across 4, 8, and 16 Euler steps.The authors interpret this consistency as support for softly anchoring the fake model to generator parameters.Source: §3.2 · p. 5

3.3 INTRA-SEGMENT GUIDANCE: RELOCATING TIMESTEP IMPORTANCE

  • The normalized one-step reconstruction loss varies non-monotonically across timesteps, with pronounced local oscillations especially from t = 0.8 to 1.0.This variation means nearby timesteps need not be equally informative for training.Source: §3.3 · p. 5
  • ISG samples an intermediate timestep within each coarse interval, combines teacher denoising to that point with generator denoising afterward, and trains the generator to match that trajectory.The loss backpropagates through the direct generator path, while the target is stop-gradient; each anchor thereby absorbs information from its surrounding segment.Source: §3.3 · p. 5

3.4 GENERAL AND POWERFUL DISCRIMINATOR BUILT UPON VISION FOUNDATION MODELS

  • The discriminator uses a frozen vision-foundation-model backbone and trainable heads, combining generated-image features, real-image reference features, and CLIP text features to assess realism and semantic consistency.Its adversarial term is weighted by ω(t) = (1 − σ_t)^2, decreasing as noise increases; the training loop combines DMD, adversarial feedback, and ISG, then applies IDA after generator updates.Source: §3.4 · pp. 5–7

4 EXPERIMENTAL RESULTS

SenseFlow is evaluated as a 4-step distillation method across SDXL, SD 3.5 Large, and FLUX.1 dev, with setup and benchmark comparisons followed by ablations of IDA, ISG, and a VFM discriminator. The reported results cover benchmark performance, qualitative image quality, training consistency, and runtime overhead.

4.1 EXPERIMENTAL SETUP

  • Experiments use filtered high-resolution LAION-5B training images and evaluate on COCO-5K, GenEval, and T2I-CompBench across SDXL, SD 3.5 Large, and FLUX.1 dev.The evaluation reports FID, Patch FID, CLIP Score, HPS v2, ImageReward, and PickScore; baselines use their official 4-step configurations.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 TEXT TO IMAGE GENERATION

  • Across SDXL, SD 3.5, and FLUX, SenseFlow reports strong 4-step benchmark results on human-preference and compositional metrics, with qualitative gains in details and prompt fidelity.SD 3.5 variants rank first and second across the reported metrics; SenseFlow also performs strongly on SDXL and FLUX, while qualitative comparisons highlight faces, textures, and scene composition.Source: Table 1 · p. 8, §4.2 · p. 8, Figure 5 · p. 9

4.3 ABLATION STUDIES

  • IDA smooths FID reductions and stabilizes SD 3.5 Large training, including at a 5:1 TTUR ratio; adding IDA also improves all five reported ablation metrics.Without IDA, several TTUR settings oscillate, including at 20:1; the IDA comparisons report improvements in FID-T, HPSv2, PickScore, ImageReward, and AESv2.Source: §4.3 · pp. 8–9
  • ISG improves image quality, text-image alignment, and human-preference results; at 3k iterations, its generations are more consistent and semantically accurate than those without ISG.The comparison reports color shifts and degraded details without ISG, and the figure caption emphasizes improved consistency in early training.Source: §4.3 · pp. 9–10, Figure 6 · p. 9
  • With TTUR f=5, ISG adds 4.44% per-iteration time on SDXL, 3.23% on FLUX, and 6.16% on SD 3.5; IDA adds 3.97% on FLUX and 0.57% on SD 3.5.Both modules run only on generator steps in this setting.Source: §4.3 · p. 10
  • On SDXL, the VFM discriminator improves HPSv2, PickScore, ImageReward, and AESv2, while raising FID-T; the paper frames this as a quality–diversity trade-off.The authors prioritize human-preferred, semantically meaningful modes over exhaustive distribution coverage in few-step distillation.Source: Table 2 · p. 9, §4.3 · p. 10

5 RELATED WORK

Diffusion distillation methods include trajectory-based, consistency, and distribution-based approaches. SenseFlow builds on DMD2 to address its stability degradation on large-scale models.

5 RELATED WORK

  • Trajectory-based methods replicate denoising trajectories, while consistency methods enforce consistency across steps.Examples include Direct and Progressive Distillation for trajectory methods and Consistency Models for consistency methods.Source: §5 · p. 10
  • Distribution-based approaches match generative distributions and include GAN-based distillation, VSD variants, and adversarial methods using pretrained feature extractors.ADD and LADD are cited as examples of adversarial diffusion distillation with pretrained feature extractors.Source: §5 · p. 10
  • DMD2 performs strongly on standard diffusion models such as SDXL, but its stability degrades on large-scale models; SenseFlow targets SD 3.5 and FLUX.1 dev.SenseFlow builds on DMD2 and addresses this limitation with alignment and regularization strategies.Source: §5 · p. 10

6 DISCUSSION & CONCLUSION

SenseFlow scales distribution-matching distillation to large flow-based models and reports superior overall 4-step results across SDXL, SD 3.5 Large, and FLUX.1 dev. The appendix formalizes how the IDA inner objective tracks the generator distribution and how its best response relates to the teacher objective.

6 DISCUSSION & CONCLUSION

  • SenseFlow reports superior overall 4-step results on human-preference and compositional benchmarks across SDXL, SD 3.5 Large, and FLUX.1 dev.The paper also identifies 2-step and 1-step sampling and alternative vision backbones as future directions.Source: §6 · p. 10

A THEORY PROOFS, DISCUSSION, AND ADDITIONAL RESULTS FOR IDA

  • The generator induces timestep marginals along the forward-kernel flow-matching path, and the fake model is trained to track the corresponding generator-induced target field.The generator is time-conditional and can be evaluated at any t ∈ [0, 1], although training applies supervision only at sparse anchor timesteps.Source: A THEORY PROOFS, DISCUSSION, AND… · p. 14

A.1 DECOMPOSITION AND INNER BEST RESPONSE

  • For fixed generator parameters, the inner objective is maximized when the fake distribution matches the generator marginals, making its value the generator-to-teacher KL.The decomposition follows from cancellation of the generator entropy and non-negativity of KL divergence.Source: §A.1 · p. 15

A.2 BOUNDING THE TRACKING ERROR UNDER IDA UPDATE

The appendix bounds IDA's tracking errors under stated regularity and bounded-error assumptions, then relates field discrepancy to an approximate inner best response. Combining these results yields an outer-loop objective guarantee within ε.

A.2 BOUNDING THE TRACKING ERROR UNDER IDA UPDATE

  • Under bounded self-consistent error and vanishing generator parameter updates, IDA bounds the asymptotic target-field tracking error by the self-consistent-error bound.The analysis derives a field-gap bound and coupled one-step recursions for parameter tracking error and target-field tracking error.Source: §A.2 · p. 15

A.3 FROM VELOCITY FIELD ERROR TO KL GAP ALONG THE FM PATH

  • Under the stated regularity and Fisher-to-KL control assumptions, tracking and self-consistent field errors bound the inner KL gap and yield an ε-best-response bound.The score and velocity differences along the linear-Gaussian flow-matching path are related by a time-schedule scalar.Source: §A.3 · p. 16

A.4 COMBINING THEM TOGETHER

  • Combining the IDA tracking bound with the KL control makes the inner loop an ε-best response, so the outer loop approximately minimizes the generator-to-teacher KL within ε.This conclusion uses the decomposition of the min–max objective.Source: §A.4 · p. 16

A.5 OBSERVATIONS OF STEP CONSISTENCY AND EMPIRICAL BOUNDEDNESS OF THE SELF-CONSISTENCY ERROR

The appendix reports that samples remain close across 4, 8, and 16 Euler steps, supporting bounded self-consistent error, while IDA improves image quality without reducing step consistency. It also details implementation settings and the VFM discriminator design.

A.5 OBSERVATIONS OF STEP CONSISTENCY AND EMPIRICAL BOUNDEDNESS OF THE SELF-CONSISTENCY ERROR

  • Across 4, 8, and 16 steps, samples remain close, supporting a smooth velocity field and empirical boundedness of the generator’s self-consistent error.The SD 3.5 Medium setup fixes prompts and latent seeds across step counts.Source: §A.5 · pp. 17–18, Figure 7 · p. 17
  • IDA improves textures, reduces artifacts, and produces more coherent structures at all reported step counts while maintaining the same level of step consistency.The comparison is between DMD without IDA and DMD with IDA, both using a VFM discriminator and TTUR 5:1.Source: §A.5 · pp. 17–18, Figure 8 · p. 17

B.1 IMPLEMENTATION DETAILS

  • Coarse timestep anchors vary by model: SDXL uses discrete DDPM indices, SD 3.5 Large uses continuous times, and FLUX.1 dev uses shifted sigmas.The main experiments use TTUR ratio 5; large flow-based models use IDA, ISG, and the VFM discriminator, while SDXL omits IDA.Source: §B.1 · p. 18

B.2 DETAILED VFM-BASED DISCRIMINATOR DESIGN

  • The VFM discriminator combines DINOv2 image and reference features with a spatially projected CLIP prompt condition to compute logits.Input images are augmented before frozen DINOv2 feature extraction; lightweight spectral-normalized convolutional heads process the spatial features.Source: §B.2 · pp. 18–19, Figure 9 · p. 18

B.3 TRAINING-TIME OVERHEAD OF ISG AND IDA

The reported per-iteration overheads of ISG and IDA are small, and both components are applied only on generator updates under TTUR. The appendix also reports training durations for the three model runs.

B.3 TRAINING-TIME OVERHEAD OF ISG AND IDA

  • With TTUR frequency f=5, both components run only on generator updates, amortizing their per-iteration cost.The measurements use 8×A100 GPUs.Source: §B.3 · p. 19
  • The reported runs take 32 hours for SDXL, 56.3 hours for SD 3.5 Large, and 23.4 hours for FLUX.1 dev.The corresponding training steps are 18k, 27k, and 12k, respectively.Source: §B.3 · p. 19
  • ISG adds +4.44% on SDXL, +3.23% on FLUX.1 dev, and +6.16% on SD 3.5 Large; IDA adds +3.97% on FLUX and +0.57% on SD 3.5.The table reports training-time overhead for ISG and IDA.Source: Table 3 · p. 19, §B.3 · p. 19

B.4 DETAILED RESULTS AND DISCUSSION OF GENEVAL AND T2I-COMPBENCH

SenseFlow’s four-step models show strong compositional performance across SD 3.5, SDXL, and FLUX. Across the two compositional suites, the results support improved semantic alignment and compositional correctness without increased sampling cost.

B.4 DETAILED RESULTS AND DISCUSSION OF GENEVAL AND T2I-COMPBENCH

  • Across diverse teachers, the two compositional suites corroborate COCO-5K findings of improved semantic alignment and compositional correctness without increasing sampling cost.This is the paper’s stated overall interpretation of the results.Source: §B.4 · p. 20
  • On SD 3.5, the Euler variant has the highest GenEval score among four-step methods and leads five of six T2I-CompBench categories, ranking second on Spatial.The reported comparisons are within methods distilling the same teacher.Source: §B.4 · p. 19, Table 4 · p. 20
  • On SDXL, SenseFlow leads GenEval among SDXL distillation methods and performs best on Color, Shape, and Texture in T2I-CompBench.It remains competitive on Spatial, Non-spatial, and Complex-3-in-1.Source: §B.4 · p. 19, Table 4 · p. 20
  • On FLUX, the stochastic sampler has the highest GenEval score among four-step baselines, while Euler leads on Shape and ranks second on Texture, Spatial, and Complex-3-in-1.The two solvers use the same distilled weights.Source: §B.4 · p. 19, Table 4 · p. 20

B.5 ANALYSIS OF DIVERSITY

The appendix finds that SenseFlow maintains broadly similar diversity, extends to two-step generation across three model families, and improves ISG training stability and convergence speed on SD 3.5 Large.

B.5 ANALYSIS OF DIVERSITY

  • LPIPS-diversity changes little, while CLIP-diversity decreases for SDXL and increases slightly for SD 3.5 Large; the authors characterize the overall diversity change as mild.For SDXL, LPIPS-diversity is 0.6002 vs. 0.5960 and CLIP-diversity is 0.0802 vs. 0.0985; for SD 3.5 Large, the corresponding pairs are 0.5664 vs. 0.5659 and 0.0900 vs. 0.0879. In the qualitative example, DMD2-SDXL has more viewpoint variation, while SenseFlow produces more centered, compositionally consistent bears.Source: Figure 10 · p. 20, §B.5 · p. 21, Table 6 · p. 21

B.6 1–2-STEP GENERATION

  • At two steps, SenseFlow improves over strong baselines across SDXL, SD 3.5 Large, and FLUX; it also shows promising one-step performance after light tuning.At two steps, SD 3.5 Large reaches ImageReward 1.2022 and PickScore 22.88, exceeding SD 3.5-Large-Turbo; the one-step model reaches ImageReward 1.0651 and CLIP score 0.3332.Source: §B.6 · pp. 21–22, Table 7 · p. 22

B.7 IMPACT OF ISG ON TRAINING DYNAMICS

  • With ISG, SD 3.5 Large reaches FID-T ≈14.48 at 1.5k iterations, versus ≈138 without ISG, and remains lower at later reported checkpoints.At 9k iterations, FID-T is 16.18 versus 18.07; at 12k, it is 15.20 versus 19.06. Figure 12 plots FID and FID-T across training iterations.Source: Figure 12 · p. 23, §B.7 · p. 23, Table 8 · p. 23

B.8 COMPARISON TO DMD2 AND DETAILED COMPONENT ABLATIONS

The ablations show that the VFM discriminator, IDA, and ISG each contribute to SenseFlow’s results, while qualitative comparisons report improvements over strong baselines across SDXL, SD 3.5 Large, and FLUX.

B.8 COMPARISON TO DMD2 AND DETAILED COMPONENT ABLATIONS

  • On SD 3.5 Large, removing ISG and IDA degrades performance and raises FID-T from 13.38 to 43.84; the full method combines both components with the VFM discriminator.The ablation reports further drops in HPSv2, PickScore, and ImageReward when both IDA and ISG are removed.Source: Table 9 · p. 24, §B.8 · p. 24

B.10 MORE ABLATION STUDY RESULTS AND VISUALIZATION SAMPLES

  • For SDXL, setting λG = 0.5 improves most reported metrics and gives the best HPSv2, PickScore, and ImageReward scores among methods in Tab. 1.The tested lower weight is λG = 0.25; the table reports quantitative results for different adversarial loss weights.Source: Table 10 · p. 24, §B.10 · p. 25
  • VFM backbone scale does not yield monotonic metric gains: ViT-B has the best FID-T, ViT-S has higher CLIP Score and ImageReward, and ViT-L slightly leads on HPSv2 and PickScore.The paper uses ViT-L as its default discriminator backbone.Source: Table 11 · p. 24, §B.10 · p. 26

B.9 QUALITATIVE COMPARISONS

  • Under shared prompts, SenseFlow is reported to produce sharper details, cleaner structure, and more faithful quality than strong baselines across SDXL, SD 3.5 Large, and FLUX.The SD 3.5 Large comparison reports improved composition and text–image consistency; the FLUX comparison reports clearer structures and better textual alignment.Source: §B.9 · p. 24, Figure 14 · p. 28, Figure 15 · p. 29

Related papers

Loading 2506.00523v2…