PaperlayerView exact revision on arXiv

Source-linked AI summary

Image-Free Timestep Distillation via Continuous-Time Consistency with Trajectory-Sampled Pairs

Bao Tang, Shuai Zhang, Yueting Zhu, Jijun Xiang, Xin Yang, Li Yu, Wenyu Liu, Xinggang Wang

arXiv:2511.20410v1cs.CV

TL;DR

Continuous-time consistency distillation still has substantial data and computational demands, while forward-diffusion training samples differ from the teacher’s inference trajectory. TBCM performs image-free latent-space distillation using samples from teacher trajectories. On MJHQ-30k, it reports 6.52 FID and 28.08 CLIP score for one-step generation, alongside reduced training time and GPU memory use.

  • Problem

    Continuous-time consistency distillation faces high data and training-cost demands, and its forward-diffusion target samples differ from the teacher’s inference trajectory.

    Source: §1 · p. 2

  • Method

    TBCM performs image-free consistency distillation in latent space by sampling along the teacher model’s inference trajectories.

    Source: §1 · p. 2

  • Results

    6.52 FID and 28.08 CLIP score are reported for one-step TBCM on MJHQ-30k, outperforming Sana-Sprint’s 7.04 FID and 28.04 CLIP score under the same training setup.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    TBCM’s latent-space, image-free formulation reduces training cost and enables efficient, scalable deployment across diffusion backbones while achieving strong one-step generation performance.

    Source: §6 · p. 8

    Without real-image supervision, TBCM depends on teacher capacity and biases, which may propagate and limit diversity or induce mild mode collapse.

    Source: §6 · p. 8

Abstract

from arXiv · show

Timestep distillation is an effective approach for improving the generation efficiency of diffusion models. The Consistency Model (CM), as a trajectory-based framework, demonstrates significant potential due to its strong theoretical foundation and high-quality few-step generation. Nevertheless, current continuous-time consistency distillation methods still rely heavily on training data and computational resources, hindering their deployment in resource-constrained scenarios and limiting their scalability to diverse domains. To address this issue, we propose Trajectory-Backward Consistency Model (TBCM), which eliminates the dependence on external training data by extracting latent representations directly from the teacher model's generation trajectory. Unlike conventional methods that require VAE encoding and large-scale datasets, our self-contained distillation paradigm significantly improves both efficiency and simplicity. Moreover, the trajectory-extracted samples naturally bridge the distribution gap between training and inference, thereby enabling more effective knowledge transfer. Empirically, TBCM achieves 6.52 FID and 28.08 CLIP scores on MJHQ-30k under one-step generation, while reducing training time by approximately 40% compared to Sana-Sprint and saving a substantial amount of GPU memory, demonstrating superior efficiency without sacrificing quality. We further reveal the diffusion-generation space discrepancy in continuous-time consistency distillation and analyze how sampling strategies affect distillation performance, offering insights for future distillation research. GitHub Link: https://github.com/hustvl/TBCM.

1. Introduction

Timestep distillation seeks to make diffusion generation more efficient, but continuous-time consistency distillation still faces data and cost demands and a mismatch between training samples and inference trajectories. TBCM distills from the teacher’s own trajectories in latent space and reports strong one-step results with lower training costs.

1. Introduction

  • Diffusion models often require dozens or hundreds of denoising steps, motivating timestep distillation into compact students that generate in only a few steps.The motivation identifies long inference times and high computational costs as barriers to real-world applicability.Source: §1 · p. 1
  • Continuous-time consistency distillation remains limited by high data requirements, training costs, and forward-diffusion targets that differ from the teacher’s inference trajectory.The passage says this mismatch constrains potential improvement in one-step generation quality.Source: §1 · p. 2
  • TBCM distills entirely in latent space by sampling along the teacher’s inference trajectories, removing VAE involvement and mitigating training–inference inconsistency.The image-free framework also avoids image data and preprocessing.Source: §1 · p. 2
  • TBCM investigates how sampled points from different trajectories affect distillation and reweights an unstable sCM loss term for a more balanced objective.The introduction describes the sampling scheme as pivotal to consistency-distillation performance.Source: §1 · p. 2
  • 6.52 FID and 28.08 CLIP score on MJHQ-30k are reported for one-step TBCM, with reduced training time and GPU memory versus Sana-Sprint and standard sCM.The comparison to Sana-Sprint and standard sCM is stated in the introduction; the approximately 40% training-time reduction is reported separately.Source: §1 · pp. 2–3

2. Related Work

Prior work accelerates diffusion through sampling advances and through timestep-distillation methods that either follow teacher trajectories or match generative distributions. Consistency Models impose self-consistency across adjacent timesteps, while distribution-oriented methods include adversarial and score-distillation approaches.

2. Related Work

  • Diffusion-model research has pursued faster, more stable sampling, while Flow Matching recasts diffusion as learning continuous flows between data and noise.SANA combines compressed DCAE with a linear architecture for efficient, high-quality generation.Source: §2 · p. 3
  • Trajectory-oriented distillation uses teacher ODE trajectories, and Consistency Models align x0 predictions across adjacent timesteps.The passage distinguishes these from distribution-oriented methods, which match overall generative distributions.Source: §2 · p. 3
  • Distribution-oriented approaches include adversarial pixel-domain ADD, latent-space LADD, and non-adversarial Variational Score Distillation.Later methods build on VSD to improve stability and effectiveness.Source: §2 · p. 3

3. Preliminaries

Diffusion formulations perturb data according to a noise schedule, while consistency models predict clean data consistently across noise levels. The section moves from discrete to continuous-time consistency and notes both optimization instability and a mismatch between forward and backward equivalent noise.

3.1. Different Formulations of Diffusion Models

  • Diffusion models reverse progressive noising; formulations differ mainly in interpolation schedules and vector-field parameterization.A perturbed sample is formed using a noise schedule.Source: §3.1 · p. 3

3.2. Continuous-Time Consistency Models

  • Consistency models predict clean data from noisy observations and keep that prediction consistent across noise levels along a probability-flow ODE trajectory.Source: §3.2 · p. 3
  • Taking the timestep interval to zero yields a smooth continuous-time consistency objective without discretization artifacts.Earlier consistency models use a loss between neighboring timesteps.Source: §3.2 · p. 3
  • The classical optimization objective is reported as unstable and difficult to scale to large models or datasets; TrigFlow changes its representation.Source: §3.2 · p. 3
  • Equivalent noise stays constant during forward diffusion but shifts noticeably during backward generation, reflecting training–inference inconsistency.Source: Figure 3 · p. 4

4. Method

TBCM replaces image-based consistency training with latent samples from the teacher’s inference trajectory. The method addresses resource bottlenecks, examines how trajectory sampling shapes distillation, and adds filtering and stability adjustments.

4.1. Trajectory-Backward Consistency Models

  • VAE encoding accounts for approximately 80% of memory use, while text encoding takes substantial training time and VAE encoding less than 1%.Figure 4 breaks down memory use above and training time below.Source: §4.1 · p. 4, Figure 4 · p. 4
  • Training samples differ substantially from inference samples: equivalent noise shifts during backward sampling from random noise toward patterns correlated with the prediction target.The paper also reports substantial inconsistency between forward- and backward-process sample distributions.Source: §4.1 · p. 5
  • TBCM distills directly in latent space without VAE encoding, generating multiple samples along each teacher trajectory for a prompt.This is intended to address both resource bottlenecks and training–inference inconsistency.Source: §4.1 · p. 5

4.2. Sampling Schemes Shaping the Sample Space

  • TBCM samples in generation space, where each training pair depends on both its timestep and the preceding inference trajectory; forward-sampled pairs define diffusion space.The paper identifies sample-space composition as a decisive factor in consistency-distillation effectiveness.Source: §4.2 · p. 6
  • Sampling design must account for the trajectory as well as timestep distribution, because the trajectory used to obtain samples can significantly affect training outcomes.The paper identifies sampling scheme as a factor in distillation quality.Source: §4.2 · p. 6

4.3. Additional Adjustments

  • Brightness filtering removes low-brightness latent samples; TBCM also explores settings and schedules for the sCM unstable-term weight r.The filtering uses the observed correspondence between dark pixel-space images and latent representations close to an all-black image.Source: §4.3 · p. 6

5. Experiments

TBCM is evaluated using image-free training prompts and one-step generation on MJHQ-30k, with ablations examining trajectory sampling, sampling-step count, and R scheduling. Its reported results pair strong fidelity and text–image alignment with lower training and memory costs.

5.1. Main Results

  • Training uses 1M randomly sampled text prompts without paired image data, with the Sana-Sprint 0.6B teacher, on 32 NVIDIA V100 GPUs.The experiments follow Sana-Sprint training configurations apart from a minor learning-rate adjustment for trajectory sampling.Source: §5.1 · pp. 6–7
  • 6.52 FID and 28.08 CLIP score: TBCM outperforms Sana-Sprint (7.04 FID, 28.04 CLIP score) in one-step generation on MJHQ-30k.The comparison follows the same training setup.Source: Table 2 · p. 7, §5.1 · p. 7
  • Over 40% lower training costs and more than 60% less GPU memory: TBCM reduces both versus Sana-Sprint and sCM.Training time was measured on 32 NVIDIA V100 GPUs; memory was measured on one A100 GPU with batch size 16.Source: Table 3 · p. 7, §5.1 · p. 7

5.2. Ablation Study

  • Reference Route achieves the best FID and CLIP performance, followed by Logit-Normal; both outperform Random in the reported comparisons.Reference Route assigns timesteps across subregions, reducing randomness relative to probabilistic sampling; the two strategies’ timestep distributions look visually similar.Source: Figure 7 · p. 7, Table 4 · p. 8, §5.2 · p. 8
  • Increasing the number of sampled steps improves FID, while CLIP score remains relatively stable.The authors relate the trend to fewer steps yielding lower-quality clean samples along the denoising trajectory.Source: Table 5 · p. 8, §5.2 · p. 8
  • R_final = 0.75 yields the most stable and effective training among the tested R values.The selection is evaluated in the R hyperparameter ablation.Source: Table 6 · p. 8, §5.2 · p. 8
  • Warmup–Cooldown outperforms direct Warmup-to-R_final scheduling.Warmup–Cooldown first warms up to 1, then cools down to R_final.Source: Table 7 · p. 8, §5.2 · p. 8

6. Conclusion

TBCM performs continuous-time consistency distillation entirely in latent space under image-free conditions, removing VAE involvement. The conclusion also identifies dependence on teacher quality and the design of the constructed sample space as important boundaries and open directions.

6. Conclusion

  • TBCM conducts continuous-time consistency distillation in latent space without image data or VAE involvement.The authors state that this substantially reduces training cost and supports efficient, scalable deployment across diffusion backbones.Source: §6 · p. 8
  • Without real-image supervision, student effectiveness remains constrained by teacher capacity and biases, which may propagate and limit diversity or induce mild mode collapse.The authors describe student performance as tightly coupled to the quality of the teacher’s synthetic trajectories.Source: §6 · p. 8
  • The authors identify more expressive, well-structured sample spaces as an open research direction because their structure shapes continuous-time distillation behavior.They suggest combining TBCM with complementary generative or regularization strategies to mitigate teacher-induced limitations.Source: §6 · p. 8

Supplementary Material

The supplement outlines five additions: equivalent-noise distribution visualizations, sampling-strategy comparisons on predicted x0, the full TBCM algorithm, a Brightness Filter illustration, and more multi-step results.

Supplementary Material

  • The supplementary material covers equivalent-noise distribution shifts, predicted x0 under different sampling strategies, the TBCM algorithm, the Brightness Filter, and additional multi-step results.These topics appear in Sections A through E, respectively.Source: Supplementary Material · p. 11

A. Distribution Shift of Equivalent Noise.

The supplement compares equivalent-noise distributions during diffusion and generation using t-SNE. It reports stable noise distributions during diffusion but changing patterns and declining similarity to initial noise during generation.

A. Distribution Shift of Equivalent Noise.

  • t-SNE visualizes the overall equivalent-noise distributions in diffusion and generation to test whether their differences are distributional rather than instance-specific.Figure 8 pairs timestep-specific cosine similarity to initial noise with t-SNE projections at selected timesteps.Source: A. Distribution Shift of Equivalent… · p. 11, Figure 8 · p. 11
  • During diffusion, equivalent-noise distributions remain consistent; during generation, their patterns change and similarity to initial noise gradually decreases.The reported diffusion pattern reflects the instance-level consistency shown in Figure 3.Source: A. Distribution Shift of Equivalent… · p. 11

B. Comparison of Sampling Strategies.

The section compares teacher x0 predictions across sampling schemes and inference steps, then describes how TBCM uses trajectory samples in its training algorithm.

B. Comparison of Sampling Strategies.

  • Reference Route yields x0 predictions closest to Flow Euler, followed by Logit-Normal and Random; better x0 predictions are treated as partial evidence of trajectory-wide sample quality.The comparison is shown in Fig. 9 and aligns with the reported Tab. 4 results.Source: B. Comparison of Sampling Strategies · p. 11, Figure 9 · p. 11
  • Increasing inference steps generally improves predicted x0 quality, but the differences diminish as the step count rises.This trend is consistent with the observations reported in Tab. 5.Source: B. Comparison of Sampling Strategies · p. 11

C. Algorithm of TBCM.

  • TBCM collects training samples along the teacher’s inference trajectory without image data, then uses them to compute the sCM loss.The algorithm also incorporates the brightness filter and stability hyperparameter adjustment.Source: C. Algorithm of TBCM · p. 12

D. Illustration of the Brightness Filter.

The Brightness Filter identifies low-quality latent samples using their similarity to a precomputed black-image latent, avoiding VAE decoding.

D. Illustration of the Brightness Filter.

E. Multi-Step Generation Results.

The section presents TBCM generation results for two and four inference steps, using a scheduler that maps back to x0 before adding noise at an intermediate timestep.

E. Multi-Step Generation Results.

Loading 2511.20410v1…