Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping

1. Introduction

• Diffusion models trade generation cost against sampling quality but need many function evaluations for satisfactory results.The paper frames slow inference as a constraint on score-based generative models.Source: §1 · p. 2
• LCM image quality at one or a few steps remains behind teacher-model convergence, and increasing inference iterations can reduce visual complexity and quality.The paper links multi-step sampling to accumulated discretisation error and notes that excessive noise addition and removal can lose detail.Source: §1 · p. 2
• TCD combines a trajectory consistency function that reduces parameterisation and distillation errors with strategic stochastic sampling that suppresses accumulated discretisation and estimation errors.The function broadens the consistency boundary conditions and supports transitions along the PF ODE trajectory.Source: §1 · pp. 2–3
• TCD enhances LCM’s image quality and can outperform the teacher model, such as SDXL with DPMSolver++, at sufficient iterations.The paper identifies 20 NFEs as an example of sufficient iterations.Source: §1 · p. 3

Diffusion Models

• Diffusion models have been applied across image, video, and audio generation, including text-to-image synthesis.Source: §2 · p. 3

Text-conditional Diffusion Models

• Text-conditional diffusion models reduce computational cost by operating in latent space or using separate super-resolution steps, while classifier-free guidance improves conditional samples at almost no added cost.Source: §2 · p. 3

Fast Sampling of DMs

• Diffusion models have strong generative capabilities but slow sampling, motivating numerical solvers and few-step distillation approaches.Distillation methods can require large datasets and lengthy training, and may converge slowly.Source: §2 · p. 3
• Exponential-integrator methods exploit diffusion models’ semi-linear structure to obtain exact analytical solutions for the linear term and accurate PF-ODE solutions.Source: §2 · p. 3

Consistency Models

• Consistency models map points on a PF ODE trajectory to data for one-step generation while retaining multistep sampling to trade computation for quality.Latent consistency models combine consistency distillation with latent diffusion for accelerated text-to-image synthesis.Source: §2 · p. 3

Parameter-Efficient Fine-Tuning

• LoRA freezes pretrained weights and injects trainable low-rank matrices, reducing modified parameters, computational load, and storage demands.Source: §2 · pp. 3–4

3.1. Diffusion Models

• A forward SDE progressively adds Gaussian noise to data, with drift and diffusion coefficients governing the process and its marginals moving toward a Gaussian prior.Source: §3.1 · p. 4
• A PF ODE shares the forward SDE’s marginal probability densities, and sampling can solve an empirical PF ODE from time T to 0 using a learned score model and ODE solver.The empirical score approximates the ground-truth score through score matching.Source: §3.1 · p. 4

3.2. Consistency Models

• Consistency models map points on a PF ODE trajectory to its origin, with the boundary condition f(x0, 0) = x0 and self-consistent outputs for points on the same trajectory.Source: §3.2 · p. 4
• Consistency distillation trains a consistency function by minimizing a self-consistency objective using adjacent timestep pairs and a one-step ODE-solver update.The paper focuses on the distillation objective, while consistency training is an alternative that does not require a pretrained model.Source: §3.2 · p. 5

3.3. Latent Consistency Model

• Latent Consistency Models extend consistency models to image latents and use a conditioned consistency function to predict the PF-ODE solution at t = 0.Their parameterisation is compatible with several prediction methods, including x-, epsilon-, and v-prediction.Source: §3.3 · p. 5
• LCM skipping-step distillation enforces self-consistency across timesteps k steps apart, approximating the earlier point with DDIM or DPM-Solver under classifier-free guidance.Source: §3.3 · p. 5

4.1. Background

• LCM’s DDIM-based consistency parameterization incurs discretization error from finite differences, while the consistency-loss bound also depends on the time interval spanned by the function.The paper reports that multistep LCM sampling loses image detail, whereas TCD addresses that issue.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2. Trajectory Consistency Function

• TCF uses a semi-linear exponential-integrator parameterization, and its upper error bounds match those of DPMSolver-k.The paper also reports that TCF reduces both parameterization and distillation error bounds.Source: §4.2 · pp. 6–8
• The paper proposes TCF(S+) to estimate the exponentially weighted integral of xθ directly, without omitting the residual term when k = 1.The second-order TCF expansion uses a midpoint u between t and s.Source: §4.2 · p. 7
• TCF broadens the consistency boundary from the origin to arbitrary trajectory intervals, while retaining self-consistency for points on the same PF-ODE trajectory.Its parameterization is described as satisfying both classical and broadened boundary conditions.Source: §4.2 · pp. 6–7

4.3. Strategic Stochastic Sampling

• Strategic Stochastic Sampling (SSS) introduces γ to control stochasticity at each step, adjusting both the denoising destination and the noise level in the diffuse step.Each step alternates an ODE-solver denoise sub-step with a Langevin-SDE diffuse sub-step.Source: §4.3 · p. 8
• When γ is low, estimation error matters more, so the optimal γ must be determined empirically.The paper also notes that TCF sampling retains a discretization error from its parameterization that cannot be eliminated.Source: §4.3 · p. 9

4.4. Extension to Large Text Conditional Models

• For text-conditioned models, TCF adds conditioning input c; for larger models such as SDXL, the paper uses LoRA to reduce distillation memory consumption.The paper describes TCD as fine-tuning on top of a pretrained model.Source: §4.4 · p. 9

5.1. Experimental Setup

• Experiments use SDXL with TCF(1) and γ=0.2 by default, comparing TCD qualitatively and quantitatively with Euler, DDIM, DPM-Solver++(2S), and LCM.The quantitative comparison is zero-shot generation on 5K COCO2017 validation captions.Source: §5.1 · p. 9, §5.2 · p. 9

5.2. Main Results

• At 20 steps, TCD produces more detailed images than LCM and even SDXL with DDIM, whose multistep samples are smoother and less detailed.At 4 steps, prior efficient numerical methods produce suboptimal images, while LCM performs relatively better and TCD further improves visual quality.Source: §5.2 · p. 9, Figure 4 · p. 10
• At 2 or 4 steps, TCD significantly surpasses numerical ODE solvers; at 8 or 20 steps, it outperforms its teacher in almost all metrics.The evaluation uses 5K COCO2017 validation captions and reports FID, Image Complexity Score, Image Reward, and PickScore.Source: §5.2 · p. 9, Table 1 · p. 11

5.3. Ablation Studies

• The authors report that stochastic sampling can reduce accumulated errors; at γ=0, the model’s inherent estimation error becomes more pronounced, leading to decreased quality as evaluated with FID (Figure 6a).Source: §5.3 · p. 11
• At four sampling steps, TCF(2) is slightly inferior to TCF(1), while TCF(S+) performs worse despite 43,000 training iterations.The authors attribute TCF(2)'s result to high-order solver instability and TCF(S+)'s result to difficulty distilling teacher information with additional parameters.Source: §5.3 · p. 11, Table 2 · p. 11
• TCD is applied to Animagine XL V3 2, Papercut XL, Depth and Canny ControlNets, and IP-Adapter, generating high-quality images in 2–8 steps.The authors describe this as evidence of versatility across community models, styled LoRA, ControlNet, and IP-Adapter.Source: §5.3 · p. 11

6. Conclusion

• TCD combines TCF, which tracks the PF ODE trajectory in semi-linear form, with SSS, which controls stochastic intensity and reduces accumulated errors through bijective traversal.The authors report that TCD outperforms LCM across sampling steps and numerical methods of the teacher model.Source: §6 · p. 11
• The authors identify enhanced super-resolution detail and a better intermediate manifold for editing as potential downstream insights from TCD.These are proposed applications rather than reported experimental results in the supplied conclusion passage.Source: §6 · p. 11
• High-order TCF instability and poor TCF(S+) convergence remain unresolved and require further analysis.The conclusion also calls for improved design to achieve fewer sampling steps, though the specific target is truncated in the passage.Source: §6 · p. 11
• TCD is reported to accelerate styled LoRA, ControlNet, and IP-Adapter generation in a few steps while retaining high quality.The conclusion identifies these model types as downstream applications.Source: §6 · p. 12

A. Discussion between CTM (Kim et al., 2023) and TCD

• The section frames CTM’s distillation-based integral estimation as implicit, while TCD retains explicit sampling and consistency-model definitions.Source: A. Discussion between CTM (Kim et al.,… · pp. 15–16
• Unlike CTM’s unified anytime-to-anytime CM/DM decoder and γ-sampling, TCD targets LCM with semi-linear consistency parameterization and explicit sampling.Source: A. Discussion between CTM (Kim et al.,… · pp. 15–16
• TCD parameterizes the PF-ODE’s semi-linear structure, supports x0, v, and ϵ parameterizations, and can use existing explicit samplers without designing new ones.It follows consistency-distillation targets and uses only distillation loss with skipping steps to pursue self-consistency and avoid mode collapse.Source: A. Discussion between CTM (Kim et al.,… · pp. 15–16

B.1. Algorithm Details of Trajectory Consistency Distillation

• TCD training samples data and time indices, forms a noisy input, computes a solver-based target, and updates the student with an EMA target model.Source: §B.1 · p. 16

B.2. The Difference between Multistep Consistency Sampling and Strategic Stochastic Sampling

• Unlike multistep consistency sampling’s fixed zero endpoint, strategic stochastic sampling uses γ to adjust noise during alternating denoising and diffusion steps.Source: §B.2 · pp. 16–17

C.1. Dataset

• Training used LAION5B High-Res images with aesthetic scores of at least 5.8, resized to 1024 pixels on the shorter side and randomly cropped to 1024 × 1024.Source: §C.1 · p. 17

C.2. Hyper-parameters

• TCF(1) and TCF(2) used batch size 256, learning rate 4.5e-6, and LoRA rank 64, while TCF(S+) used batch size 192 and learning rate 4.0e-6 without LoRA.TCF(S+) directly fine-tuned the U-Net and used EMA decay 0.95; TCF(1) and TCF(2) did not use EMA.Source: §C.2 · p. 17

C.3. Hardware and Efficiency

• TCF(1) trained for 3,000 iterations in 15 hours, whereas TCF(S+) took 43,000 iterations over 5 days 20 hours and had a low convergence rate.The authors suggest its additional parameters may account for the lower distillation efficiency and defer efficiency improvements to future work; models used eight 80G A800 units.Source: §C.3 · p. 17

• As NFEs increase, LCM image details tend to vanish, while TCD maintains or enhances them.Source: §D.1 · p. 18

• Visual complexity and image quality gradually improve as the stochastic parameter γ increases.Source: §D.2 · p. 18

D.3. More Comparisons Results

• With few NFEs, TCD generates high-quality images that outperform LCM; with sufficient NFEs, it produces more detailed, high-definition images than DPM-Solver++(2S).These are additional qualitative comparisons in Figures 10–14.Source: §D.3 · p. 18

D.4. More Samples from TCD’s Versatility Testing

• TCD is directly applied to Animagine XL V3, Papercut XL LoRA, Depth and Canny ControlNet, and IP-Adapter, accelerating their sampling with only 4 steps.The tested models share the same TCD LoRA parameters.Source: §D.4 · p. 18, Figure 17 · p. 28, Figure 18 · p. 28
• Figure 15 compares TCD with SDXL and Animagine XL V3 using the same TCD parameters, with all samples generated in 4 steps.The top row shows SDXL and the bottom row shows Animagine XL V3.Source: Figure 15 · p. 26
• Figure 16 compares TCD with and without Papercut XL LoRA at a LoRA scale of 1.0, using the same TCD parameters and 4 steps.The top row omits Papercut LoRA; the bottom row includes it.Source: Figure 16 · p. 27