Source-linked AI summary
Accelerating Diffusion via Hybrid Data-Pipeline Parallelism Based on Conditional Guidance Scheduling
Euisoo Jung, Byunghyun Kim, Hyunjin Kim, Seonghye Cho, Jae-Gil Lee
TL;DR
Diffusion inference remains expensive, and distributed acceleration can introduce artifacts while delivering sublinear speedups. The paper combines condition-based partitioning with adaptive parallelism switching, achieving reported latency reductions on SDXL and SD3 while preserving fidelity and generalizing across architectures.
Problem
Diffusion inference is computationally expensive, while distributed parallelism can suffer generation artifacts and fail to scale proportionally with GPU count.
Method
The framework partitions conditional and unconditional denoising paths and adaptively switches parallelism according to their denoising discrepancy.
Results
2.31× and 2.07× latency reductions are achieved on SDXL and SD3, respectively, while preserving fidelity and covering U-Net and DiT architectures.
Takeaways & Limitations
The framework provides a unified multi-GPU diffusion-parallelism approach with reported acceleration across conventional denoising and flow-matching generative frameworks.
Takeaways & Limitations
Support for parallelism greater than two GPUs for a single image is deferred to future work.
Abstract
from arXiv · showhide
Diffusion models have achieved remarkable progress in high-fidelity image, video, and audio generation, yet inference remains computationally expensive. Nevertheless, current diffusion acceleration methods based on distributed parallelism suffer from noticeable generation artifacts and fail to achieve substantial acceleration proportional to the number of GPUs. Therefore, we propose a hybrid parallelism framework that combines a novel data parallel strategy, condition-based partitioning, with an optimal pipeline scheduling method, adaptive parallelism switching, to reduce generation latency and achieve high generation quality in conditional diffusion models. The key ideas are to (i) leverage the conditional and unconditional denoising paths as a new data-partitioning perspective and (ii) adaptively enable optimal pipeline parallelism according to the denoising discrepancy between these two paths. Our framework achieves $2.31\times$ and $2.07\times$ latency reductions on SDXL and SD3, respectively, using two NVIDIA RTX~3090 GPUs, while preserving image quality. This result confirms the generality of our approach across U-Net-based diffusion models and DiT-based flow-matching architectures. Our approach also outperforms existing methods in acceleration under high-resolution synthesis settings. Code is available at https://github.com/kaist-dmlab/Hybridiff.
1. Introduction
Diffusion inference is costly because iterative denoising creates latency, while existing multi-GPU strategies face communication, synchronization, artifact, and quality limitations. The paper proposes condition-based partitioning with adaptive parallelism switching to pursue beyond-linear acceleration while preserving fidelity.
- Motivation: Iterative denoising creates substantial diffusion-inference latency and computational bottlenecks as model sizes grow.
- Motivation: Existing multi-GPU approaches distribute input data or model components, but communication and synchronization costs make practical speedups sublinear.
- Motivation: 2.3× speed-up: the hybrid approach with two GPUs versus 1.2× for data parallelism and 1.3× for model parallelism.
- Limitations of Existing Hybrid Strategies: Patch-and-pipeline combinations can introduce patch-boundary artifacts and asynchronous denoising errors that degrade generation quality.
- Proposed Approach: Condition-based partitioning separates conditional and unconditional CFG paths while each partition covers the entire image, preserving global consistency and reducing feature-aggregation communication.
- Proposed Approach: Adaptive parallelism switching selects execution modes from denoising discrepancy between conditional and unconditional estimates, aligning pipeline execution with conditional guidance.
- Scope and Results: The framework reports acceleration across conventional diffusion and flow-matching models, including U-Net and DiT architectures, while preserving generation quality.
2. Related Work
Related work accelerates diffusion on one device through fewer sampling steps or architectural changes, and on multiple GPUs through data- or model-parallel execution. Patch-based and transformer-specific approaches remain constrained by artifacts, scalability, or architectural scope.
- Single-GPU Diffusion Acceleration: Single-device diffusion acceleration reduces sampling steps, optimizes architectures, or uses mathematical reformulations and approximations.
- Multi-GPU Diffusion Acceleration: DistriFusion divides images into independent patches for data-parallel denoising, while AsyncDiff partitions the U-Net into asynchronously connected layer segments.
- Multi-GPU Diffusion Acceleration: AsyncDiff uses stride-based scheduling to balance parallel execution and reduce latency.
- Multi-GPU Diffusion Acceleration: PipeFusion and XDiT combine patch-level parallelism with ring attention, while CFG-based data-parallel methods remain limited to inter-image processing.
- Multi-GPU Diffusion Acceleration: Transformer-specific ring-attention schemes show limited scalability and inconsistent performance across general diffusion architectures.
3. Preliminaries
The preliminaries describe denoising diffusion, classifier-free guidance, and flow matching as foundations for the paper's multi-GPU inference framework. CFG provides conditional and unconditional predictions, while flow matching defines sampling through an ODE.
- Denoising Diffusion Model: Denoising diffusion defines a forward noising process with variance schedule {β_t} and learns a parameterized reverse denoising process.
- Denoising Diffusion Model: The reverse process is optimized through a variational lower bound.
- Classifier-Free Guidance: Classifier-free guidance predicts conditional noise ε_θ(x_t, t, c) and unconditional noise ε_θ(x_t, t, ∅), then uses guidance scale w during sampling.
- Flow Matching: Flow matching defines an ODE between target distribution q(x) and base distribution p_0(x), with sampling obtained by integrating ẋ = v_θ(x, t) from t = 1 to t = 0.
4. Method
The framework combines conditional/unconditional branch partitioning with adaptive switching among warm-up, parallelism, and fully connecting stages. Switching boundaries are determined from denoising discrepancy to balance acceleration and generation quality across diffusion and flow-matching models.
- Framework Overview: The framework feeds isotropic noise into conditional and unconditional denoising branches, combining condition-based partitioning with adaptive parallelism switching.The conditional path uses a prompt, while the unconditional path omits it; devices are assigned either branch alongside model subsets.
- Denoising Discrepancy: Denoising discrepancy rel-MAE_t(ϵ_c, ϵ_u) measures the timestep-wise difference between conditional and unconditional noise predictions.Larger discrepancy indicates stronger conditional influence on the denoising trajectory.
- Three-Stage Scheduling: The denoising process is divided into Warm-Up [T, τ1], Parallelism (τ1, τ2), and Fully Connecting [τ2, 0] stages.The discrepancy follows a U-shaped trend, with stable branch alignment during the parallelism interval.
- Generality and Scope: The discrepancy-based three-stage framework extends to flow matching by replacing predicted noise with predicted velocity and supports diverse U-Net and DiT architectures.The framework is described as robust when sequential denoising permits measuring conditional and unconditional influence.
- Adaptive Switching: τ1 is selected from the recent discrepancy slope and bounded by τcap, while τ2 is set to τ1 + k.The safety cap handles late or undefined τ1 selections; increasing k accelerates inference but can reduce generation quality.
- Generality and Scope: Supporting parallelism greater than two for a single image is deferred to future work.The framework can be applied to single-sample generation, but finer partitioning may reduce acceleration efficiency and quality.
5. Experiments
Experiments show that the hybrid framework accelerates SDXL and SD3 while preserving generation quality, outperforming distributed baselines across qualitative, ablation, trade-off, and high-resolution evaluations.
- Main Results: 2.31× acceleration over the single-GPU baseline is achieved on SDXL while slightly improving image fidelity.The method also reduces communication cost by 19.6× compared to AsyncDiff.
- Main Results: 2.07× speed-up is achieved on SD3 with negligible communication cost and comparable or superior generation quality.The method outperforms DistriFusion, AsyncDiff, xDiT-Ring, and Parastep.
- Qualitative Results: The method preserves global coherence and fine-grained details similar to the original model, avoiding boundary artifacts and spatial inconsistency observed in baselines.Qualitative comparisons use 1024×1024 SDXL generations.
- Ablation Study: 1.78× speed-up from condition-based partitioning increases to 2.31× with adaptive parallelism switching while maintaining comparable image quality.The ablation attributes the additional efficiency to the pipeline component.
- Ablation Study: Smaller parallelism intervals k preserve higher fidelity, whereas larger values provide greater acceleration; k=5 offers a balance of quality and speed.The interval can be selected flexibly to adjust the efficiency–fidelity trade-off.
- High-Resolution Generation: 2.72×, 1.54×, and 1.62× speed-ups are reported at 1024×1024, 2048×2048, and 2560×2560 resolutions, respectively, on SDXL with NVIDIA H200 GPUs.The method consistently achieves superior acceleration over existing distributed inference frameworks across these resolutions.
6. Conclusion
The paper introduces a hybrid diffusion-inference framework combining condition-based partitioning with adaptive parallelism switching, achieving acceleration on SDXL and SD3 while preserving fidelity across U-Net and DiT architectures.
- Conclusion: The framework integrates condition-based partitioning and adaptive parallelism switching using a denoising discrepancy criterion.It adaptively switches parallelism modes to minimize redundant communication.
- Conclusion: 2.31× and 2.07× latency reductions are achieved on SDXL and SD3, respectively, while preserving fidelity.The framework generalizes across U-Net and DiT architectures.
A. Evaluation of Hybrid Parallelism
The evaluation measures speed-up, image quality, and model generality while analyzing denoising discrepancy and the algorithmic transitions among warm-up, parallelism, and fully connected execution.
- Evaluation Protocol: Speed-up is measured relative to SDXL baseline latency, while image quality is evaluated using FID scores from the main SDXL results.Reported latencies are 13.53 seconds for DistriFusion, 12.54 seconds for AsyncDiff, and 7.12 seconds for the proposed method.
- Denoising Discrepancy: Figure 8 measures average denoising discrepancy rel-MAE_t(ε_c, ε_u) over 5,000 MS-COCO validation prompts using SDXL.The shaded region denotes the ±2σ range, and the red dot denotes τ_cap.
- Adaptive Parallelism Switching: Algorithm 1 receives latent noise, a prompt, denoising steps, a window, slope threshold, safety cap, and interval, then returns the denoised output and switching thresholds.It computes conditional and unconditional noise estimates at each denoising step.
- Adaptive Parallelism Switching: The algorithm begins with warm-up, switches to parallelism after the selected threshold, and uses fully connecting execution during the intermediate interval.The transitions are determined by the denoising-discrepancy criterion and interval k.
D. Derivation of Score-Based Interpretation of Denoising Discrepancy
The paper derives denoising discrepancy from score decomposition, showing that the conditional–unconditional denoiser difference represents conditional information relative to the unconditional data prior.
- The preconditioned score is decomposed into an unconditional data score and a conditional information-flow term.
- The conditional–unconditional denoiser difference corresponds to the conditional gradient scaled by σt.
- rel-MAE_t quantifies the relative magnitude of conditional information compared with the unconditional data prior.
E. Robustness of Determine τ1 under Stochastic Denoising Noise
The method stabilizes τ1 detection against stochastic denoising fluctuations by smoothing discrepancy slopes over a finite timestep window, with larger windows reducing false detections.
- Random sampling causes predicted noises and observed discrepancy values to fluctuate, so G_t may reach approximately zero prematurely.
- A finite-difference slope over L timesteps smooths stochastic perturbations, while L, δ, a, and b define the detection bound and value range.
- As L increases, estimated-slope variance decreases and false-detection probability decreases exponentially.
- τ1 is detected as the earliest timestep satisfying 0≤G_t<g_slope and t≤τ_cap within an empirically stable regime.
- The framework extends from two GPUs to batch-level and layer-wise many-GPU configurations, with layer-wise scaling trading some efficiency and quality for fine-grained scheduling.
- Experiments use DDIM with T=50 and 1024×1024 images on NVIDIA GeForce 3090 GPUs, with model-specific adaptive-switching parameters.
H. Quantitative Results on the Parallelism Interval k
The experiments quantify a speed–quality trade-off controlled by the parallelism interval k: smaller intervals preserve fidelity, while larger intervals provide stronger acceleration.
- Smaller parallelism intervals k preserve higher fidelity, whereas larger k values yield stronger acceleration.
I. Additional Qualitative Results
Additional SDXL results at 1024×1024 compare the proposed method with prior approaches and examine how the interval k affects visual fidelity.
- The comparison includes Original, DistriFusion, AsyncDiff, and Ours alongside generated examples for two prompts.
- The method achieves the best acceleration and FID performance while producing visuals most similar to the original at 1024×1024 resolution.
- Smaller k values preserve higher visual fidelity, while larger k values gradually blur local detail and fine-grained conditional attributes.