PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Halves are More than One: Phase-wise Velocity Distillation for Fast and High-Quality Image Generation

Zhen Guo, Rongyuan Wu, Qiaosi Yi, Chenxi Xie, Xinyu Wei, Lei Zhang

arXiv:2610.08070v1cs.CV

TL;DR

Diffusion models require costly repeated backbone evaluations, while high-quality generation in a single full-backbone-forward budget remains challenging for monolithic students. PVD assigns separate half-sized experts to coarse and fine phases using average-velocity targets. Across class-conditional and text-to-image tasks, it outperforms a single full-size monolithic student under comparable computation, with T2I results competitive with teachers and lower active parameters and peak VRAM.

  • Problem

    Diffusion generation typically requires tens to hundreds of costly full-backbone passes, and a compact monolithic student can struggle to handle coarse structure and fine detail in one pass.

    Source: §1 · pp. 1–2

  • Method

    PVD divides generation into structural and refinement phases, assigning each a half-sized expert trained to approximate its phase's average velocity.

    Source: §1 · p. 2

  • Results

    Across C2I and T2I, two ordered phase-specific experts outperform a single full-size monolithic student under comparable computation; T2I models are teacher-competitive and reduce active parameters and peak VRAM.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The phase allocation separates structural composition from detail refinement while keeping cumulative computation equivalent to one full-backbone evaluation.

    Source: §5 · p. 10

    The optimal number of experts and partitioning strategies remain open; failures include dense object counting and uncommon spatial relations, which also challenge teacher backbones.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent diffusion-based image generation backbones have grown substantially in scale, making the network inference cost increase rapidly. While diffusion distillation techniques can reduce the number of inference steps, high-quality image generation within a single full-backbone-forward compute budget remains challenging. Existing one-step methods typically allocate this budget to a single evaluation of a monolithic student. However, approximating the heterogeneous coarse-to-fine transport with a single monolithic mapping is difficult and often leads to over-smoothed outputs. To address this issue, we propose Phase-wise Velocity Distillation (PVD), which partitions the generation timeline into a coarse and a fine phase, and models the transition within each phase via the average velocity. A dedicated half-sized expert is assigned to each phase, decoupling structural composition from detail refinement while keeping the cumulative computation equivalent to one full-backbone forward pass. We show that the use of two half-sized phase-specific experts outperforms a single full-size monolithic student. On class-conditional image generation, PVD achieves an FID of 1.48 on ImageNet 256 x 256. On more complex text-to-image (T2I) tasks, PVD-distilled models (Stable Diffusion 3.5-Medium, FLUX.1-dev, Qwen-Image) produce results competitive with their multi-step teachers, significantly outperforming prior distillation methods. Moreover, across the evaluated T2I backbones, PVD reduces active parameters by 49.10-50.89% and peak VRAM by 45.76-48.36% compared to the corresponding teachers. Source code and distilled models are available at https://github.com/PolyU-VCLab/PVD.

1 Introduction

Diffusion inference is costly, and a compact monolithic student can struggle to represent both coarse structure and fine detail in one pass. PVD divides generation into two phase-specific average-velocity transports, reporting strong results on class-conditional and text-to-image tasks within a single full-backbone-forward budget.

1 Introduction

  • Prior distillation methods include trajectory-based approaches that imitate or regress teacher transport and distribution-based approaches that align student outputs with teacher or data distributions.The paradigms use objectives such as self-consistency, average-velocity regression, adversarial supervision, and score or divergence matching.Source: §1 · pp. 1–2
  • A single compact student must resolve global layout and fine details together under a strict one-pass budget, a mismatch associated with over-smoothed outputs and possible one-step collapse on complex T2I tasks.Diffusion generation typically requires tens to hundreds of costly full-backbone-forward passes.Source: §1 · pp. 1–2
  • PVD partitions the trajectory into structural and refinement phases, assigning each a half-sized expert that predicts localized average velocity while keeping cumulative inference cost within one teacher forward pass.The experts replace one global noise-to-data jump with two shorter-horizon transport objectives.Source: §1 · p. 2
  • The proposed phase-wise experts use localized mean-velocity objectives to align model capacity with coarse-to-fine dynamics without increasing the compute budget.The paper presents this as a trajectory-decomposition framework under a single full-backbone-forward constraint.Source: §1 · p. 2
  • PVD outperforms existing distillation methods on C2I and reports teacher-competitive T2I quality with reduced active parameters and peak VRAM; C2I FID is 1.48 on ImageNet 256 × 256.The reported T2I models are Stable Diffusion 3.5-Medium, FLUX.1-dev, and Qwen-Image.Source: §1 · pp. 2–3

2 Related Work

Related work covers the progression from iterative diffusion and flow-based generation to few-step distillation, then reviews phase-specialized generation and phased distillation. The paper distinguishes PVD by targeting a strict single full-backbone-forward budget.

2 Related Work

  • DDPM and DDIM use iterative denoising, while flow matching and rectified flow learn continuous transport; these approaches still rely on multi-step integration.Transformer backbones have scaled these paradigms to large text-to-image models, including FLUX, Qwen-Image [6], and Z-Image [7].Source: §2 · p. 3
  • Few-step distillation spans trajectory-based methods that preserve teacher dynamics and distribution-based methods that align student outputs with teacher or data distributions.Examples include progressive distillation, consistency methods, long-horizon transport, adversarial supervision, and score- or divergence-based matching.Source: §2 · p. 3
  • Phase-specialized generation assigns different denoising regimes to experts, including ensemble, SNR-routed, and interval-specific LoRA designs [43] [10].Hierarchical Distillation [9] and Phased DMD also divide generation into stages, but operate in the few-step regime.Source: §2 · pp. 3–4
  • PVD’s overview combines half-sized backbone initialization, phase-specific LoRA experts trained by mean-velocity learning, and phase-wise discriminators for T2I refinement.The figure presents these as three stages of the framework.Source: Figure 3 · p. 4
  • Prior phased methods do not work under a strict single full-backbone-forward compute budget, the regime targeted by PVD.The paper contrasts this constraint with methods that reduce inference to a few steps.Source: §2 · p. 4

3 Method

PVD divides image generation into ordered structural and refinement phases, each handled by a half-sized expert predicting localized average velocity. A shared initialized backbone supports phase-specific training, with adversarial refinement added for T2I tasks.

3.2 Expert Initialization

  • A half-sized backbone is extracted from the teacher and fine-tuned over the full time horizon to provide a shared initialization for phase-wise experts.It retains the first and last three blocks, samples intermediate layers until depth is halved, and is trained by flow matching to mimic the teacher’s instantaneous dynamics.Source: §3.2 · p. 5

3.3 Phase-wise Distillation

  • PVD assigns half-sized experts to ordered phases, using the first for global layout and low-frequency structure and the second for local structure and high-frequency detail.Each expert predicts the average velocity over its assigned interval, enabling a one-step jump within that phase.Source: §3.1 · p. 4, §3.3 · p. 6
  • Phase-wise distillation trains each expert with a self-consistent stop-gradient target, and samples t′ = t to recover instantaneous matching and stabilize optimization.The localized phase decomposition is motivated by a bound that makes the marginal-shift bound scale with interval length.Source: §3.3 · pp. 5–6

3.4 Adversarial Refinement

  • For T2I, phase-wise discriminators provide complementary localized adversarial supervision, while teacher-anchored velocity loss remains the primary distillation objective.The adversarial penalty is integrated as a secondary marginal regularizer.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate PVD on ImageNet class-conditional generation and text-to-image generation distilled from three backbones. PVD reports strong generation quality alongside reduced deployment costs, while ablations and Unsplash further training examine expert allocation and perceptual quality.

4.1 Experiment Setup

  • The evaluation covers ImageNet 256 × 256 C2I and T2I distilled from SD3.5-Medium, FLUX.1-dev, and Qwen-Image, comparing against prior accelerated and multi-step methods.T2I assessment includes prompt-alignment, generation, aesthetic, perceptual, and efficiency measures.Source: §4.1 · pp. 6–7

4.2 Class-Conditional Image Generation

  • FID 1.48 and IS 295.89 on ImageNet: PVD outperforms iMF (1.72 FID, 282.00 IS) and FACM (1.76 FID).The results are highly competitive with multi-step references.Source: Table 1 · p. 7, §4.2 · p. 7
  • The class-conditional ablations favor a [0.4, 0.6] split, zero overlap, and ordered phase-specific experts; a shared student degrades from 12.89 to 26.62 FID and 126.70 to 77.12 IS.The study also reports that reducing overlap improves performance, while full overlap substantially degrades both metrics.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Text-to-Image Generation

  • Across three T2I backbone groups, PVD achieves most of the best results across evaluation dimensions while reducing active parameters and peak VRAM relative to teachers.On SD3.5-Medium, FLUX.1-dev, and Qwen-Image, active parameters fall by 50.89%, 49.75%, and 49.10%, and peak VRAM by 46.49%, 45.76%, and 48.36%, respectively.Source: Table 3 · p. 8, §4.3 · p. 8
  • PVD’s T2I outputs preserve clearer structure and finer detail than existing distillation methods under a single-forward budget, with quality close to multi-step teachers.The comparisons note that other methods often produce oversmoothed outputs and structural artifacts around object boundaries.Source: §4.3 · p. 8, Figure 4 · p. 9
  • T2I ablations find compact phase-wise experts outperform a monolithic full-size student even when it costs twice the computation; adversarial refinement adds a complementary quality gain.Expert reuse and order swaps weaken quality, consistent with learned mappings being directional and non-interchangeable.Source: §4.3 · pp. 8–9
  • Further training on Unsplash consistently improves aesthetic scores and diversity across all three T2I backbones, while GenEval scores decrease slightly.The authors attribute the GenEval decrease to weaker Unsplash supervision for object counts, attribute bindings, and spatial relations; visual comparisons show more natural lighting, richer textures, and stronger spatial depth.Source: §4.3 · pp. 9–10, Table 4 · p. 9, Figure 5 · p. 10

5 Conclusion

PVD assigns separate half-sized experts to coarse and fine phases, matching phase-specific average-velocity transports to a single full-backbone-forward budget. The conclusion reports competitive generation quality and reduced T2I resource use, while noting unresolved partition choices and difficult cases.

5 Conclusion

  • PVD achieves an FID of 1.48 on ImageNet 256 × 256 and remains competitive with multi-step teachers on three T2I backbones.The T2I models significantly outperform prior distillation methods; active parameters fall by 49.10–50.89% and peak VRAM by 45.76–48.36% versus the corresponding teachers.Source: §5 · p. 10
  • The conclusion attributes PVD’s quality–efficiency trade-off to two ordered half-sized experts that separate structural composition from detail refinement within one full-backbone evaluation.Each expert models its phase transition through average velocity, and the two experts outperform a single full-size monolithic student under comparable computation.Source: §5 · p. 10
  • The study uses two half-sized experts for simplicity; the optimal expert count and more effective partitioning strategies remain open questions.PVD can still fail on dense object counting and uncommon spatial relations, which are also challenging for teacher backbones.Source: §5 · p. 10

Appendix Contents

  • The appendix lists further discussions, mathematical analysis, training and inference procedures, detailed experimental settings, and additional C2I and category-wise T2I results.It also lists detailed cost measurements, more T2I visualizations, T2I ablations, and failure cases.Source: Contents · p. 15
  • The appendix contents also identify failure-case examples and a broader-impacts section.The failure cases correspond to the conclusion’s discussion of limitations.Source: Contents · p. 15

A Further Discussions on Related Distillation Methods

The discussion distinguishes PVD from methods that specialize timesteps, use average velocities, or factorize transport. Its defining combination is phase-restricted average-velocity learning, distinct compact experts, and sequential phase-specific transport under a one-backbone-equivalent budget.

A Further Discussions on Related Distillation Methods

  • Table 5 compares temporal localization, parameter specialization, average-velocity modeling, explicit phase-factorized inference, and capacity and computation allocation.It distinguishes temporal specialization from PVD’s compute-constrained transport factorization.Source: Table 5 · p. 16, A Further Discussions on Related… · p. 16
  • MeanFlow uses one shared model for interval-conditioned average velocities, while Hierarchical Distillation [9] uses MeanFlow to initialize and then refine a single-step student.Their phase hierarchy is not realized as sequential, temporally distinct inference operators.Source: A Further Discussions on Related… · p. 16
  • TimeStep Master and Glance specialize lightweight parameters but retain the backbone at each iterative denoising evaluation; Phased DMD uses multiple generator evaluations.These designs do not match PVD’s compact phase-local operators under a one-backbone-equivalent cumulative budget.Source: A Further Discussions on Related… · p. 16
  • PVD combines phase-restricted average-velocity learning, distinct compact experts, and sequential transport composition while keeping cumulative sampling cost approximately equal to one full-backbone forward pass.This factorizes both the transport horizon and active capacity per evaluation.Source: A Further Discussions on Related… · p. 16

B Mathematical Analysis on PVD

The analysis defines interval-average velocities and an initialization error for comparing teacher and backbone-induced transports. It gives an integration-error bound and a temporal marginal-shift bound under stated velocity assumptions.

B Mathematical Analysis on PVD

  • The teacher and trajectory-induced backbone average velocities are defined by integrating their instantaneous fields along a trajectory over a temporal phase.The directly parameterized phase expert is optimized separately and is not assumed to equal the trajectory-induced average velocity.Source: B Mathematical Analysis on PVD · p. 17
  • Theorem 3.1 bounds the discrepancy between teacher average velocity and trajectory-induced backbone average velocity by the temporal average of instantaneous prediction error.The analysis defines the error as the discrepancy between those two average velocities.Source: B Mathematical Analysis on PVD · p. 17
  • Under a uniformly bounded velocity field, the induced marginal distributions’ 2-Wasserstein discrepancy is bounded linearly by interval length: W2(pt, pt′) ≤ CΔt.The proof bounds the distance by expected trajectory length.Source: B Mathematical Analysis on PVD · p. 18

C Training and Inference Procedure of PVD

The procedure section points to algorithms for PVD optimization and sampling. Training samples a phase and trajectory inputs, while inference starts from noise, applies phase experts, and decodes the final state.

C Training and Inference Procedure of PVD

  • The training algorithm samples a phase index and, when adversarial training is enabled, freezes its discriminator and unfreezes the phase expert.It then draws data and noise samples, samples times within the selected phase, constructs an interpolated state, and evaluates the teacher velocity and phase expert.Source: C Training and Inference Procedure of… · p. 19
  • Training computes the phase expert’s average-velocity prediction and its Jacobian-vector product, then forms an updated velocity and a fake state.The algorithm indicates that adversarial training is optional.Source: C Training and Inference Procedure of… · p. 19
  • The inference algorithm samples initial noise, iterates through phase-specific updates, and decodes the final latent state into an image.The supplied procedure identifies the output as the generated image.Source: C Training and Inference Procedure of… · p. 19

D Detailed Experimental Settings

The section separates backbone initialization from phase-expert training and describes settings for class-conditional and text-to-image models. Additional class-conditional samples show coherent structures, clear boundaries, and rich textures.

D Detailed Experimental Settings

  • Training uses separate backbone-initialization and phase-expert stages, with experts assigned to early [0, 0.4] and late [0.4, 1] intervals.Experts are trained independently on samples restricted to their interval, with (Pmean, Pstd) rescaled accordingly.Source: D Detailed Experimental Settings · p. 20, Table 6 · p. 20
  • LightningDiT-XL/1 and SD3.5-Medium use full-parameter phase experts, while FLUX.1-dev and Qwen-Image use LoRA experts with ranks 64 and 32, respectively.Qwen-Image’s backbone is further trained using DiffusionNFT after initialization; phase-wise training uses BLIP3o-60k and Echo-4o.Source: D Detailed Experimental Settings · p. 20

E More C2I Visualizations

F Category-wise T2I Results

The section reports category-wise text-to-image benchmark results and detailed inference costs, alongside additional qualitative comparisons across three backbones. Qwen-Image examples illustrate the transition from coarse structure to refined detail.

F Category-wise T2I Results

  • Category-wise results are reported for GenEval, DPG-Bench, WISE, TIIF-Bench mini, and Qwen-Image-Bench.The section points to detailed benchmark tables but the supplied passages do not state their cell values.Source: F Category-wise T2I Results · p. 22

G Detailed Cost Measurements

  • Across three backbones, PVD keeps cumulative computation close to one full-backbone evaluation while reducing active parameters by 49.10–50.89% and peak VRAM by 45.76–48.36% relative to the corresponding teachers.Measurements use 1024 × 1024 resolution, one PPU, and BF16; peak VRAM excludes text encoders and the VAE.Source: Table 14 · p. 26, G Detailed Cost Measurements · p. 26

H More T2I Visualizations

  • Additional comparisons cover SD3.5-Medium, FLUX.1-dev, and Qwen-Image, while continued training on Unsplash data illustrates improvements in lighting, composition, and fine-grained details.Figure 8 marks models further trained on Unsplash images with an asterisk.Source: H More T2I Visualizations · p. 27, Figure 7 · p. 27
  • Qwen-Image outputs show the first phase establishing composition, silhouette, semantic layout, and regional color despite residual noise, with the second phase refining fine-scale, high-frequency content.Figure 9 pairs each intermediate first-phase image with its final second-phase output.Source: H More T2I Visualizations · p. 29, Figure 9 · p. 29

I Ablation Studies on T2I Task

The ablations compare PVD with shared-model and monolithic alternatives, test the discriminator’s contribution, and evaluate phase-expert specialization and execution order. PVD’s designated early-to-late expert composition achieves the strongest reported results among the tested order variants.

Shared MeanFlow vs. Phase-specific Experts

  • PVD obtains 0.88 GenEval with Nflops = 1.02, compared with the Qwen-Image teacher’s 0.87 at substantially higher inference cost.MeanFlow improves from 0.44 to 0.49 GenEval when sampling increases from 4 to 8 steps, while Nflops doubles from 4.00 to 8.00.Source: Table 15 · p. 30, I Ablation Studies on T2I Task · p. 30

Phase Decomposition and Adversarial Refinement

  • At nearly matched compute, PVD scores 0.6948 GenEval at 6.93 TFLOPs and 2.67 GiB peak VRAM, versus 0.4142 for the full-size student at 7.02 TFLOPs and 4.99 GiB.Doubling the full-size student’s Nflops to 2.00 raises its score to 0.5695 at 14.04 TFLOPs; removing PVD’s discriminator leaves inference cost unchanged but reduces GenEval from 0.6948 to 0.5599.Source: Table 16 · p. 30, I Ablation Studies on T2I Task · pp. 30–31

Expert Specialization and Execution Order

  • The designated early-to-late expert composition achieves the highest GenEval (0.6948) and ImageReward (0.4947); reusing the early expert lowers these to 0.6754 and 0.4598.Variants beginning with the late expert lead to significantly worse GenEval and ImageReward values.Source: Table 17 · p. 30, I Ablation Studies on T2I Task · p. 31

J Failure Cases

The failure cases show that recognizable, orderly images can still violate precise count and spatial-relation instructions. The broader-impacts discussion describes potential efficiency benefits as well as risks from inexpensive photorealistic image synthesis.

J Failure Cases

  • The failures illustrate limits in following precise instructions even when generated content appears coherent and realistic.The authors suggest targeted post-training, including preference-based fine-tuning or reinforcement learning with relative rewards, as possible complementary supervision.Source: J Failure Cases · p. 32
  • For a request for 35 macarons in five rows of seven, Qwen-Image produces a 5 × 5 grid (25) and PVD a 6 × 6 grid (36), so both miss the requested count and arrangement.Both outputs retain an orderly tray appearance.Source: J Failure Cases · p. 32, Figure 10 · p. 32
  • For the mug prompt, both teacher and student place the pencil inside the cup and through an additional hole in the mug wall, leaving the handle hole empty.The objects remain recognizable despite the contradictory spatial relation.Source: J Failure Cases · p. 32, Figure 11 · p. 32

K Broader Impacts

  • The authors say lower inference cost and VRAM use could support edge-device deployment and real-time interactive applications, while inexpensive photorealistic synthesis may increase deepfake and misinformation risks.They advocate watermarking and safety content filters as mitigations.Source: K Broader Impacts · p. 33

Related papers

Loading 2610.08070v1…