Source-linked AI summary
PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, Zhenguo Li
TL;DR
Text-to-image diffusion models need both efficient high-resolution sampling and fine-grained control. PIXART-δ combines LCM acceleration with a Transformer-specific ControlNet design, generating high-quality 1024px images in 1 second while enabling precise control.
Problem
Text-to-image generation requires faster high-resolution sampling and effective controllability, but conventional ControlNet is designed for UNet architectures rather than Transformers.
Method
PIXART-δ integrates LCM into PIXART-α and introduces ControlNet-Transformer, which applies trainable copies to initial Transformer blocks and adds their outputs to corresponding frozen blocks.
Results
PIXART-δ generates 1024px images in 1 second with high quality and fine-grained control, while reaching 0.5 seconds per 1024 × 1024 image on an A100.
Takeaways & Limitations
PIXART-δ supports faster, high-resolution, controlled image generation suitable for real-time applications.
Abstract
from arXiv · showhide
This technical report introduces PIXART-δ, a text-to-image synthesis framework that integrates the Latent Consistency Model (LCM) and ControlNet into the advanced PIXART-α model. PIXART-α is recognized for its ability to generate high-quality images of 1024px resolution through a remarkably efficient training process. The integration of LCM in PIXART-δ significantly accelerates the inference speed, enabling the production of high-quality images in just 2-4 steps. Notably, PIXART-δ achieves a breakthrough 0.5 seconds for generating 1024x1024 pixel images, marking a 7x improvement over the PIXART-α. Additionally, PIXART-δ is designed to be efficiently trainable on 32GB V100 GPUs within a single day. With its 8-bit inference capability (von Platen et al., 2023), PIXART-δ can synthesize 1024px images within 8GB GPU memory constraints, greatly enhancing its usability and accessibility. Furthermore, incorporating a ControlNet-like module enables fine-grained control over text-to-image diffusion models. We introduce a novel ControlNet-Transformer architecture, specifically tailored for Transformers, achieving explicit controllability alongside high-quality image generation. As a state-of-the-art, open-source image generation model, PIXART-δ offers a promising alternative to the Stable Diffusion family of models, contributing significantly to text-to-image synthesis.
1 INTRODUCTION
PIXART-δ integrates LCM and ControlNet into PIXART-α to combine faster inference with controllable, high-quality text-to-image generation. Its Transformer-specific ControlNet architecture addresses limitations of directly adapting UNet-based ControlNet.
- PIXART-δ incorporates LCM and ControlNet into PIXART-α, a high-quality 1024px diffusion-transformer text-to-image model.
- 0.5 seconds per 1024 × 1024 image on an A100 yields a 7× inference-speed improvement over PIXART-α using approximately 4 steps.LCM enables high-quality inference in only 2∼4 steps by treating reverse diffusion as an augmented probability-flow ODE.
- ControlNet-Transformer adapts controllable generation to Transformers after direct replication of the UNet-based architecture proved less effective.The proposed architecture achieves explicit controllability while retaining high-quality image generation.
2 BACKGROUND
The background introduces consistency-based methods for accelerating diffusion sampling and ControlNet for conditioning text-to-image outputs. These foundations motivate combining rapid latent-space sampling with controllable generation in PIXART-δ.
- Consistency Models accelerate sampling by maintaining self-consistency along Probability Flow ODE trajectories and mapping points back to their origins.
- LCM extends consistency-based acceleration to latent diffusion text-to-image models by directly predicting augmented PF-ODE solutions in latent space.Its approach combines one-stage guided distillation and skipping-step techniques for rapid inference and fast training convergence.
- ControlNet conditions text-to-image diffusion outputs through a trainable copy of a frozen UNet whose layer outputs are integrated with skip connections.
- ControlNet supports conditioning controls including edges, depth, segmentation, and human pose while preserving the quality of pretrained UNet models.
3 LCM IN PIXART-δ
PIXART-δ uses latent consistency distillation, guidance and noise-schedule modifications, and Transformer-compatible control integration to improve training and inference efficiency. It produces 1024px images rapidly across hardware while supporting controlled generation.
- 3.1 ALGORITHM AND MODIFICATION: The training pipeline uses teacher, student, and EMA denoisers together with an ODE solver for latent consistency distillation.
- 3.1 ALGORITHM AND MODIFICATION: A constant classifier-free guidance scale improves PIXART-δ performance over CFG embeddings while simplifying implementation.
- 3.1 ALGORITHM AND MODIFICATION: Larger batch sizes positively influence FID and CLIP scores, although smaller batches can still converge quickly with comparable image quality.
- 3.1 ALGORITHM AND MODIFICATION: PIXART-δ adapts the LCM noise schedule to PIXART-α’s higher-logSNR schedule and supports a broader range of noise distributions.
- 3.2 TRAINING EFFICIENCY AND INFERENCE SPEEDUP: Distillation is successfully conducted within 32GB GPU memory while retaining the same batch size and supporting image resolutions up to 1024 × 1024.The resulting efficiency enables training on a broad range of consumer-grade GPU specifications.
- 3.2 TRAINING EFFICIENCY AND INFERENCE SPEEDUP: 0.5 seconds is sufficient for 1024 × 1024 generation on an A100, while PIXART-δ takes 3.3 seconds on a T4 and 0.8 seconds on a V100 at batch size 1.It uses 4 steps, compared with 14 for PIXART-α and 25 for SDXL standard, and requires less than 8GB GPU VRAM with 8-bit inference.
4 CONTROLNET IN PIXART-δ
PIXART-δ introduces ControlNet-Transformer, a Transformer-specific design that preserves controllability while matching Transformer data flow. Ablations show it outperforms ControlNet-UNet, with N = 13 selected for the final balance between control quality and computation.
- 4.1 ARCHITECTURE: The architecture addresses the mismatch between UNet ControlNet’s skip connections and Transformers’ lack of explicit encoder and decoder blocks.The proposed design is tailored to Transformer architectures rather than directly replicating the UNet-based structure.
- 4.1 ARCHITECTURE: ControlNet-Transformer applies trainable copies to the initial N frozen Transformer blocks and merges each copy through a zero linear layer.The merged output is passed into the subsequent frozen block, preserving PixArt’s data flow.
- 4.3 ABLATION STUDY: ControlNet-Transformer generally outperforms ControlNet-UNet, with faster convergence and improved overall performance.ControlNet-UNet imposes information flow between nonexistent encoder and decoder stages, unlike the Transformer’s natural processing pattern.
- 4.3 ABLATION STUDY: N = 13 is the final copied-block choice, balancing computational burden and performance across control conditions.N = 1 suffices for many scenes and objects, while difficult human outline edges improve as N increases.
- 4.4 CONVERGENCE: Most edges achieve satisfactory results around 1,000 training steps, while additional steps improve human face and body outline quality.The model also exhibits sudden convergence between 300 and 1,000 steps, followed by continued detail improvement.
- 4.5 VISUAL RESULTS: PixArt-ControlNet produces high-resolution images with granular geometric control, including fidelity at the level of individual strands.The report uses visualizations to demonstrate precise control over generated image composition.
5 CONCLUSION
PIXART-δ combines LCM-based acceleration with a Transformer-specific ControlNet design for fast, controlled text-to-image generation. The report presents high-quality 1024px generation and fine-grained controllability, supporting real-time applications.
- PIXART-δ integrates LCM for 4-step sampling acceleration while maintaining high-quality generation.
- The proposed Transformer-based ControlNet enables precise control over generated images.
- PIXART-δ generates high-quality 1024px images and fine-grained controllable images in 1 second.
- The reported speed and controllability are presented as capabilities that unlock real-time image-generation applications.