PaperlayerView exact revision on arXiv

Source-linked AI summary

HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Acceleration

Desen Sun, Jason Hon, Jintao Zhang, Sihang Liu

arXiv:2603.07815v1cs.CVcs.AI

TL;DR

Large text-to-image diffusion models provide high-quality generation but incur heavy computation, while prior model-switching methods overlook differing computational demands within a timestep. HybridStitch combines large and small models through pixel- and timestep-level, region-aware stitching. It achieves up to 18.06% lower latency than a state-of-the-art mixture-of-models method while maintaining comparable quality.

  • Problem

    Large T2I diffusion models impose heavy computation overhead, and prior model-switching methods ignore computational differences among regions within a single timestep.

  • Method

    HybridStitch uses pixel- and timestep-level region-aware switching, with the small model processing full latents while the large model refines difficult regions and uses KV cache for missing context.

  • Results

    Up to 18.06% lower latency than the state-of-the-art mixture-of-models acceleration approach is achieved while maintaining comparable quality.

  • Takeaways & Limitations

    HybridStitch provides efficient text-to-image diffusion generation by preserving image quality while reducing the computation assigned to the large model.

Abstract

from arXiv · show

Diffusion models have demonstrated a remarkable ability in Text-to-Image (T2I) generation applications. Despite the advanced generation output, they suffer from heavy computation overhead, especially for large models that contain tens of billions of parameters. Prior work has illustrated that replacing part of the denoising steps with a smaller model still maintains the generation quality. However, these methods only focus on saving computation for some timesteps, ignoring the difference in compute demand within one timestep. In this work, we propose HybridStitch, a new T2I generation paradigm that treats generation like editing. Specifically, we introduce a hybrid stage that jointly incorporates both the large model and the small model. HybridStitch separates the entire image into two regions: one that is relatively easy to render, enabling an early transition to the smaller model, and another that is more complex and therefore requires refinement by the large model. HybridStitch employs the small model to construct a coarse sketch while exploiting the large model to edit and refine the complex regions. According to our evaluation, HybridStitch achieves 1.83$\times$ speedup on Stable Diffusion 3, which is faster than all existing mixture of model methods.

1 Introduction

Large diffusion models improve image quality but impose substantial latency, while prior model-switching methods overlook regional differences in computational demand. HybridStitch addresses this by switching models at pixel and timestep levels, using the small model broadly and the large model for difficult regions.

  • Motivation: Larger T2I diffusion models improve image quality but significantly increase execution latency through heavier computation.The passage contrasts Stable Diffusion model sizes and identifies latency as a barrier for latency-sensitive applications.
  • Illustration: 1.83× speedup over the large model is achieved by region-aware switching, compared with 1.55× for naive whole-image switching.Figure 1 attributes the larger speedup to switching some pixels while others continue with the previous model.
  • Motivation: Prior switching methods process the entire image uniformly, overlooking heterogeneous computational demands within a timestep.Full-image switching can either degrade quality by switching too early or increase latency by waiting for all pixels.
  • HybridStitch: HybridStitch switches models at pixel and timestep levels, allowing easy-to-render pixels to transition earlier while difficult regions remain with the large model.This region-aware strategy treats different image regions according to their rendering difficulty.
  • HybridStitch: HybridStitch uses the small model to process full latent states while the large model refines only difficult regions and combines both outputs between denoising steps.The large model stops processing once all pixels satisfy the switching condition, reducing computation while preserving coherent content.

2 Preliminaries and Related Work

Diffusion inference iteratively denoises Gaussian noise, and its computation grows with more advanced architectures and parameters. Related acceleration methods reduce this cost through caching, attention sparsity, or switching between models across denoising steps.

  • 2.1 Diffusion: Diffusion models iteratively remove noise from an initial Gaussian sample across a sequence of denoising processes.The inference trajectory changes x_T into x_0 over T iterations under a Markov-chain formulation.
  • 2.1 Diffusion: p_θ(x_{t-1}|x_t) models the transition from the current noisy state x_t to the preceding state x_{t-1}.More advanced model structures and larger parameter counts improve prediction correctness but add computation overhead.
  • 2.2 Efficient Diffusion Models: Diffusion acceleration methods reuse intermediate results, reuse latent states across requests, or exploit sparsity in attention kernels.These approaches target denoising computation without necessarily changing the model-switching schedule.
  • 2.2 Efficient Diffusion Models: Figure 2 compares absolute prediction differences between large and small models at denoising steps 10, 30, and 50.The distributions motivate examining model discrepancies across both timesteps and image regions.
  • 2.2 Efficient Diffusion Models: Mixture-of-model methods exploit unequal denoising-step importance by assigning larger or smaller models to different portions of the trajectory.MoDM uses larger models initially, whereas T-Stitch uses small models first and switches to a larger model for later refinement.

3 Method

HybridStitch performs region-aware model switching across denoising timesteps, using the large model for difficult regions and the small model for global drafting. Its staged design reduces computation by shrinking the large model’s mask before transitioning fully to the small model.

  • Compute reduction: The large model’s processed area decreases after each switch, while the small model processes the full image to construct the ongoing sketch.This design aims to lower latency relative to pure large-model inference while retaining large-model refinement where needed.
  • Region-aware stitching: HybridStitch switches models at both pixel and timestep levels, refining difficult regions with the large model while the small model processes the full latent.The large model’s masked output is combined with the small model’s full-image output to preserve coherent generation.
  • Three-stage pipeline: The method begins with large-model processing to construct the image layout, then enters a hybrid stage before finishing with small-model-only denoising.The three stages respectively use the large model, both models, and only the small model.
  • Mask construction: The mask selects top-K regions with the largest latent differences because larger changes indicate unstable areas requiring large-model refinement.The mask is updated during the hybrid stage as the difference tensor changes.
  • Adaptive switching: HybridStitch measures normalized L1 differences between adjacent latent steps and advances stages when the difference falls below a threshold.The latent and noise definitions determine the combined prediction used for stage switching.

4 Evaluation

The evaluation uses Stable Diffusion 3 models and compares HybridStitch with mixture-of-model baselines using standardized generation settings. The baselines differ in model ordering and switching strategy.

  • Evaluation setup: The evaluation uses Stable Diffusion 3.5 Large as the large model and Stable Diffusion 3 Medium as the small model.Experiments use 50 denoising steps, 5k COCO captions, one image per caption, and 768 × 768 resolution by default.
  • Baselines: HybridStitch is compared with T-Stitch and SRDiffusion for both efficiency and image quality.The comparison targets existing multi-model denoising approaches.
  • Baselines: T-Stitch uses a small denoiser during initial denoising steps and a large denoiser later, with a fixed switching step.Its model ordering reflects an assumption that later steps emphasize quality refinement.
  • Baselines: SRDiffusion uses the large model early and switches to the small model after several steps, with an adaptive prompt-dependent switching function.It was originally designed for video generation but is also applied to diffusion-based image generation.

Metrics

The evaluation measures visual distributional quality and prompt-image semantic similarity, while configuring baselines and HybridStitch mask sizes for efficiency-quality comparisons.

  • Quality metrics: FID measures distributional differences between generated outputs and ground-truth images, while CLIP score measures prompt-image cosine similarity.These metrics jointly assess visual quality and semantic alignment.
  • Evaluation configuration: T-Stitch uses the small model for the first 40 % of denoising steps and the large model for the remaining 60 %.SRDiffusion uses a 0.005 threshold, while HybridStitch evaluates mask sizes of 10 %, 20 %, 30 %, and 40 %.
  • Results table: Table 1 reports the quantitative evaluation of the compared methods.The table provides the study’s efficiency and quality comparison.

4.2 Main Results

HybridStitch improves quality and efficiency over prior acceleration methods while preserving visually consistent outputs. Its 30% mask configuration achieves the highest speedup without noticeable visual degradation.

  • HybridStitch beats T-Stitch and SRDiffusion on all reported quality metrics.
  • Up to 5% FID reduction over T-Stitch and 4.4% over SRDiffusion demonstrates improved distributional quality.
  • Up to 18.06% latency reduction over SRDiffusion is achieved while maintaining equivalent or better quality.
  • All methods produce high-quality images consistent with input prompts, with differences mainly appearing in finer generated details.
  • HybridStitch-30% achieves the highest speedup while preserving visual quality without noticeable degradation.

4.3 Ablation Study

The ablation study shows that updating masks during denoising supports quality, while processing the full image with the large model introduces bias.

  • StaticMask increases FID and LPIPS, indicating quality loss when the mask is not updated during denoising.
  • StaticMask lowers latency because its fixed mask cannot capture changes across denoising steps.
  • Table 2 evaluates StaticMask and FullLarge by measuring their effects on quality and latency.
  • FullLarge introduces considerable bias by shifting global feature statistics away from the real data distribution.

4.4 Sensitivity Study

HybridStitch remains effective across mask configurations and GPU types, with performance shaped by switching choices and hardware capability.

  • HybridStitch performance across multiple mask configurations is primarily influenced by switching steps rather than large-model denoising steps.
  • HybridStitch achieves at least 1.5× speedup over the original large model on both H100 SXM and A100 SXM GPUs.
  • Figure 5 compares speedup under H100 and A100 GPU types.
  • On average, HybridStitch achieves 1.11× speedup over T-Stitch and 1.1× over SRDiffusion on H100.
  • The speedup advantage over prior methods decreases on H100 because its greater power reduces the impact of intensive denoising computation.

Quality-Latency Tradeoff

Across hyperparameters, most HybridStitch configurations offer better quality–latency trade-offs than T-Stitch and SRDiffusion. At fixed target quality, HybridStitch consistently achieves lower latency than SRDiffusion.

  • Most HybridStitch configurations lie below or left of T-Stitch and SRDiffusion in the quality–latency comparison.FID is the visual-quality metric, and lower values indicate better quality.
  • For a fixed target quality, HybridStitch consistently achieves substantially lower latency than SRDiffusion.
  • Configurations with closely spaced switching thresholds can have insufficient masked refinement and degraded image quality.

Mask Size Analysis

Masked large-model latency rises with mask size, but remains substantially below full-model evaluation even at a 40% mask. HybridStitch’s earlier first switch, while retaining a comparable second switch, explains its lower latency than SRDiffusion.

  • Latency versus Mask Size: Latency increases monotonically with mask size, demonstrating that masking reduces computation relative to full-model evaluation.The measurements use 100 samples for each of four mask ratios.
  • Latency versus Mask Size: Even with a 40% mask, the masked large model remains substantially faster than evaluating the full model.A 10% mask does not equal 10% of full-model latency because mask indexing and irregular memory accesses add overhead.
  • Switch Step Analysis: HybridStitch’s first switching step occurs much earlier than SRDiffusion across all mask configurations, significantly reducing computation.The comparison records switching steps over 200 randomly selected samples.
  • Switch Step Analysis: HybridStitch’s second switching step is comparable to SRDiffusion, indicating that comparable generation quality can be maintained while only a subset is processed by the large model.The method therefore gains latency primarily from earlier first switching rather than earlier completion of the second switch.

5 Conclusion

HybridStitch is a pixel- and timestep-level stitching approach for efficient text-to-image diffusion generation. It uses regional discrepancy awareness and masked generation, with KV-cache context completion, to reduce latency while maintaining comparable quality.

  • 5 Conclusion: HybridStitch stitches models at the pixel and timestep levels for efficient text-to-image diffusion generation.The approach incorporates masked generation to account for regional differences between large and small models.
  • 5 Conclusion: HybridStitch uses regional discrepancy diversity to guide masked generation within the mixture-of-models paradigm.This explicitly accounts for differences between image regions during acceleration.
  • 5 Conclusion: KV-cache information from the previous denoising step supplies missing context during masked computation to preserve generation quality.The cache completes context omitted by masked processing.
  • 5 Conclusion: 18.06% lower latency than the state-of-the-art mixture-of-models acceleration approach is achieved while maintaining comparable quality.This is the reported maximum latency reduction relative to that baseline.
Loading 2603.07815v1…