PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis

Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, Xuefeng Xiao

arXiv:2404.13686v3cs.CV

TL;DR

Diffusion models are costly to run because high-quality generation typically requires many inference steps, while existing distillation methods either preserve or reformulate ODE trajectories with limitations. Hyper-SD combines segmented consistency distillation, human feedback learning, and score distillation to improve few-step generation, achieving state-of-the-art performance from 1 to 8 steps on SDXL and SD1.5. Its scope remains limited by eliminated classifier-free-guidance functionality and a focus on non-DiT architectures.

  • Problem

    Diffusion models still require many inference steps for high-quality images, creating substantial computational overhead.

  • Method

    Hyper-SD combines trajectory-segmented consistency distillation, human feedback learning, and score distillation in a unified framework for few-step generation.

  • Results

    Hyper-SD achieves state-of-the-art low-step performance for both SDXL and SD1.5, with unified LoRAs supporting inference from 1 to 8 steps.

  • Takeaways & Limitations

    The framework provides efficient few-step diffusion generation while preserving trajectory information and improving one-step generation.

  • Takeaways & Limitations

    Hyper-SD eliminates classifier-free-guidance functionality and leaves diffusion-transformer architectures for future work.

Abstract

from arXiv · show

Recently, a series of diffusion-aware distillation algorithms have emerged to alleviate the computational overhead associated with the multi-step inference process of Diffusion Models (DMs). Current distillation techniques often dichotomize into two distinct aspects: i) ODE Trajectory Preservation; and ii) ODE Trajectory Reformulation. However, these approaches suffer from severe performance degradation or domain shifts. To address these limitations, we propose Hyper-SD, a novel framework that synergistically amalgamates the advantages of ODE Trajectory Preservation and Reformulation, while maintaining near-lossless performance during step compression. Firstly, we introduce Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time-step segments, which facilitates the preservation of the original ODE trajectory from a higher-order perspective. Secondly, we incorporate human feedback learning to boost the performance of the model in a low-step regime and mitigate the performance loss incurred by the distillation process. Thirdly, we integrate score distillation to further improve the low-step generation capability of the model and offer the first attempt to leverage a unified LoRA to support the inference process at all steps. Extensive experiments and user studies demonstrate that Hyper-SD achieves SOTA performance from 1 to 8 inference steps for both SDXL and SD1.5. For example, Hyper-SDXL surpasses SDXL-Lightning by +0.68 in CLIP Score and +0.51 in Aes Score in the 1-step inference.

1 Introduction

Diffusion distillation methods reduce multi-step inference by either preserving the original ODE trajectory or reformulating it, but each has limitations. Hyper-SD combines both directions through segmented consistency distillation, human feedback learning, and score distillation.

  • Diffusion models require computationally expensive multi-step inference, motivating distillation methods that reduce the number of sampling steps.
  • Trajectory-preserving distillation maintains the original ODE trajectory so student models can make further flow predictions with fewer inference steps.
  • Trajectory-reformulating distillation supervises models with ODE endpoints or real images, constructing efficient trajectories while potentially causing output-domain inconsistencies.
  • Hyper-SD combines trajectory preservation and reformulation through trajectory segmented consistency distillation, human feedback learning, and score distillation.TSCD progressively enforces consistency within time-step segments; feedback learning modifies trajectories for few-step inference, while score distillation improves one-step generation.
  • Hyper-SD provides a unified LoRA supporting inference at all NTEs and achieves state-of-the-art low-step performance for both SDXL and SD1.5.

2 Preliminaries

Diffusion-model distillation methods differ in whether they preserve or reformulate the ODE trajectory, with representative approaches targeting teacher flows, endpoints, intermediate states, or real-image distributions. The preliminaries also introduce score distillation and human-feedback learning as mechanisms for improving accelerated generation.

  • The paper distinguishes trajectory-preserving methods from trajectory-reformulating methods for accelerating diffusion-model inference.
  • The distillation notation defines teacher and student models, noise, prompt conditions, an ODE solver, training and inference timesteps, noised trajectory points, and skipping steps.
  • Progressive Distillation trains a student to approximate subsequent flow locations determined by the teacher over a sequence of steps.The two-step illustration obtains successive target locations by repeatedly applying the ODE solver with teacher predictions.
  • Consistency Distillation maps a noised trajectory point directly to the endpoint, while CTM instead targets an intermediate trajectory point to reduce accumulated estimation and discretization errors.
  • Adversarial Diffusion Distillation bypasses the ODE trajectory and directly supervises the original state using an adversarial objective.
  • Score distillation uses score estimates and a fake-distribution simulator to calibrate the score direction, enabling one-step inference and refinement toward the teacher distribution.The gradient of the KL divergence between real and fake distributions is approximated for this refinement.
  • Human-feedback learning trains a reward model from preferred and unpreferred image pairs, then uses reward scores to fine-tune the diffusion model.The paper also uses aesthetic predictors, ImageReward, and perceptual models as feedback providers.

3 Method

Hyper-SD combines segmented trajectory-preserving distillation with feedback and score-based enhancement to improve low-step diffusion generation. Its staged consistency training preserves the original ODE trajectory while supporting flexible inference across steps.

  • Trajectory Segmented Consistency Distillation: TSCD divides the timestep range into segments, performs consistency distillation within them, and progressively reduces the segment count to achieve global consistency.Training proceeds from k = 8 to, with k = 1 matching standard CTM training.
  • Trajectory Segmented Consistency Distillation: Fine-grained segment distillation reduces fitting complexity and accumulated errors while preserving the original ODE trajectory.Models from each stage can be used for inference at corresponding step counts while closely mirroring the original model’s generation quality.
  • Human Feedback Learning: Human feedback learning combines aesthetic rewards with perceptual structure feedback to improve accelerated-model generation quality.The framework uses LAION and ImageReward for aesthetics and instance segmentation models for structural feedback.
  • Human Feedback Learning: Feedback learning is trained as a LoRA plugin so it can be merged with TSCD LoRAs to balance generation quality and output-domain similarity.The authors note that feedback learning may unintentionally alter the output domain.
  • One-step Generation Enhancement: Score distillation enhances one-step generation by combining teacher and student score functions with an MSE loss for training stability.The method applies Distribution Matching Distillation and integrates human feedback learning into the enhanced model.
  • One-step Generation Enhancement: The enhanced global consistency model supports inference from 1 to 8 steps with a unified one-step LoRA and the TCD scheduler.The approach avoids conversion to x0-prediction for implementing the one-step LoRA plugin.

4 Experiments

Experiments evaluate Hyper-SD on SD15 and SDXL using quantitative metrics, qualitative comparisons, user studies, and ablations. Results show strong low-step quality, benefits from TSCD and human feedback, and practical unified-LoRA and ControlNet compatibility.

  • Implementation Details: Experiments use SD15 and SDXL baselines, with CLIP, aesthetic, ImageReward, PickScore, and user-study evaluations.The training data includes subsets of LAION and COYO, while human-feedback learning uses COCO2017 annotations and captions.
  • Main Results: Hyper-SD produces high-quality SDXL images in one step and outperforms other methods in style, aesthetics, and image-text matching.The qualitative comparison includes LCM, TCD, PeRFLow, Turbo, and Lightning.
  • Main Results: User studies find significantly more preference for Hyper-SD, including more than a two-thirds advantage for Hyper-SD15 against the same architectures.SD21-Turbo is the stated exception in one-step inference, while the 1-Step UNet is preferred over LoRA among comparable UNet-based approaches.
  • Ablation Study: TSCD outperforms baseline TCD at lower inference steps, while performance approaches TCD as the step count increases.This supports the progressive segmentation strategy for alleviating training difficulties in the extremely low-step regime.
  • Ablation Study: Human feedback compensates for distillation-related performance degradation while maintaining similar CLIPScore and aesthetic evaluations across steps.The results are described as better matching the consistency-model objective.
  • Ablation Study: The unified one-step LoRA supports 1, 2, 4, and 8 inference steps with consistent image-text matching and slightly rising aesthetics as steps increase.The authors emphasize deployment convenience because one model can serve different efficiency requirements.
  • Ablation Study: Unified LoRAs remain compatible with ControlNet and different anime, realistic, and artstyle base models.ControlNet tests use scribble and canny conditions, while the lightweight LoRA reduces acceleration cost.

5 Conclusion

Hyper-SD combines trajectory-segmented consistency distillation with human feedback learning and variational score distillation to improve few-step diffusion generation. The framework achieves new SOTA performance on SDXL and SD15 while providing models for 1–8-step inference.

  • Conclusion: Hyper-SD combines trajectory-segmented consistency distillation, human feedback learning, and variational score distillation to improve few-step generation.Trajectory-segmented distillation preserves the original trajectory, while the other components improve low-step inference and efficiency.
  • Conclusion: The framework provides LoRAs for SDXL and SD15 supporting 1–8 inference steps, plus a dedicated one-step SDXL model.
  • Preliminaries: The diffusion process is modeled through a forward SDE and a reverse denoising process, with an equivalent probability-flow ODE used for generation.The forward process transforms data into an approximately Gaussian distribution, while the learned score estimates ∇x_t log p_t(x_t).
  • Motivation: Diffusion models require multi-step inference because image quality becomes suboptimal with relatively large ODE solver steps.The paper notes that some customized diffusion models still require 50 inference steps.
  • Method: TSCD divides timesteps into segments, enforces within-segment consistency, and progressively reduces the segment count toward all-time consistency.The method is implemented through an ODE solver, timestep sampling, TSCD loss optimization, and exponential-moving-average parameter updates.

C.1 SD15 Architecture with LoRA training

This section presents qualitative comparisons between Hyper-SD's LoRA-based approach and other LoRA-based approaches on the SD15 architecture.

  • C.1 SD15 Architecture with LoRA training: Qualitative comparisons evaluate LoRA-based approaches on the SD15 architecture.

C.2 SDXL Architecture with UNet training

This section presents qualitative comparisons between Hyper-SD's UNet-based approach and other UNet-based approaches on the SDXL architecture.

  • C.2 SDXL Architecture with UNet training: Qualitative comparisons evaluate UNet-based approaches on the SDXL architecture.

C.3 Unified LoRA

The unified LoRAs are evaluated qualitatively and are compatible with ControlNet and different base models. Their examples use scribble or canny conditioning.

  • C.3 Unified LoRA: Qualitative results are presented for unified LoRAs.
  • C.3 Unified LoRA: Unified LoRAs are compatible with ControlNet using either scribble or canny image conditioning.
  • C.3 Unified LoRA: LoRAs with different inference steps can be applied to different base models while consistently generating high-quality images.

D More ablation studies against TCD

Additional experiments show that TSCD consistently outperforms TCD under both TCD+RLHF and TCD+DMD training settings, indicating robustness across reduced training difficulty and accumulated-error conditions.

  • TSCD consistently outperforms TCD with both RLHF and DMD training settings.These experiments are reported in Table 5.

E Discussion and Limitation

The authors identify limitations involving classifier-free guidance, generic feedback models, and the current focus on conventional architectures, outlining corresponding directions for future improvement.

  • The framework is presented as promising for high-quality image generation with few inference steps and several downstream tasks.The cited applications include semi-supervised learning, domain adaptation, and retrieval.
  • Most diffusion acceleration methods, including Hyper-SD, eliminate classifier-free guidance and restrict negative-cue usage.The authors propose retaining negative cues to improve generation effectiveness and security.
  • Hyper-SD uses generic reward models for feedback learning rather than customized strategies for accelerated models.Customized feedback learning is identified as future work.
  • Future work will investigate Diffusion Transformer architectures for superior few-step generative diffusion models.
Loading 2404.13686v3…