Source-linked AI summary
PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, Zhenguo Li
TL;DR
Advanced T2I models demand costly computation and produce substantial CO2 emissions, limiting access to high-quality generation. PIXART-α decomposes training, streamlines a DiT-based Transformer, and uses dense auto-captions; it reports competitive quality with substantially lower training cost and resource use. Remaining weaknesses include target-count control, human-hand details, and text generation.
Problem
Advanced T2I training requires immense computational resources and CO2 emissions, creating significant barriers for researchers and entrepreneurs.
Method
PIXART-α decomposes training into pixel learning, text-image alignment, and aesthetic enhancement, while combining an efficient T2I Transformer with dense auto-captioned data.
Results
753 A100 GPU days and $28,400 deliver reported superior image quality and semantic alignment, with less than 1.25% of SDv1.5’s training-data volume.
Takeaways & Limitations
PIXART-α provides insights for building high-quality, low-cost T2I models for the AIGC community and startups.
Takeaways & Limitations
The model struggles with target-count control, specific details such as human hands, and text generation because its data contains few font- and letter-related images.
Abstract
from arXiv · showhide
The most advanced text-to-image (T2I) models require significant training costs (e.g., millions of GPU hours), seriously hindering the fundamental innovation for the AIGC community while increasing CO2 emissions. This paper introduces PIXART-$α$, a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-art image generators (e.g., Imagen, SDXL, and even Midjourney), reaching near-commercial application standards. Additionally, it supports high-resolution image synthesis up to 1024px resolution with low training cost, as shown in Figure 1 and 2. To achieve this goal, three core designs are proposed: (1) Training strategy decomposition: We devise three distinct training steps that separately optimize pixel dependency, text-image alignment, and image aesthetic quality; (2) Efficient T2I Transformer: We incorporate cross-attention modules into Diffusion Transformer (DiT) to inject text conditions and streamline the computation-intensive class-condition branch; (3) High-informative data: We emphasize the significance of concept density in text-image pairs and leverage a large Vision-Language model to auto-label dense pseudo-captions to assist text-image alignment learning. As a result, PIXART-$α$'s training speed markedly surpasses existing large-scale T2I models, e.g., PIXART-$α$ only takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU days), saving nearly \$300,000 (\$26,000 vs. \$320,000) and reducing 90% CO2 emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training cost is merely 1%. Extensive experiments demonstrate that PIXART-$α$ excels in image quality, artistry, and semantic control. We hope PIXART-$α$ will provide new insights to the AIGC community and startups to accelerate building their own high-quality yet low-cost generative models from scratch.
1 INTRODUCTION
PIXART-α addresses the computational and environmental burden of training advanced text-to-image models with a lower-cost Transformer-based approach. Its design combines decomposed training, an efficient T2I Transformer, and more informative captions to retain competitive generation quality.
- Motivation: Training advanced T2I models requires substantial computational resources, costs, and CO2 emissions, creating barriers to research and entrepreneurship.SDv1.5 requires 6K A100 GPU days and about $320,000, while RAPHAEL requires 60K A100 GPU days and about $3,080,000.
- Contribution: PIXART-α reduces training demands while maintaining competitive image-generation quality relative to current state-of-the-art generators.The paper presents PIXART-α as targeting near-commercial image-generation quality with substantially lower computational demands.
- Core designs: Training strategy decomposition separates pixel-distribution learning, text-image alignment, and aesthetic enhancement into streamlined subtasks.A low-cost class-condition model initializes pixel learning, followed by pretraining on information-dense pairs and fine-tuning on aesthetically superior data.
- Core designs: The efficient T2I Transformer adds cross-attention for text conditioning, streamlines the computation-intensive class-condition branch, and reuses ImageNet-derived initialization.Re-parameterization enables the adjusted text-to-image model to load the original class-condition model’s parameters directly.
- Core designs: High-informative data uses LLaVA to auto-label SAM images with dense captions, addressing incomplete and long-tailed captions in existing datasets.The resulting text-image pairs are intended to provide richer information for text-image alignment learning.
- Results: 753 A100 GPU days and $28,400 yield reported superior image quality and semantic alignment while using less than 1.25% of SDv1.5’s training-data volume.Compared with RAPHAEL, training costs are reported as 1%, saving approximately $3,000,000.
2 METHOD
PIXART-α addresses slow T2I training by decomposing generation into staged learning, adapting DiT for efficient text conditioning, and constructing denser captioned data. These designs reduce training difficulty while supporting effective alignment and aesthetic high-resolution generation.
- Training Strategy Decomposition: T2I training is slow because pixel dependency, text-image alignment, and aesthetic quality are entangled in one pipeline.The paper separates these aspects into distinct training stages.
- Training Strategy Decomposition: Three stages learn pixel dependencies, text-image alignment, and high-resolution aesthetic generation using progressively specialized data.Stage 1 uses an ImageNet-pretrained class-conditional model; later stages use information-rich text-image pairs and aesthetic data.
- Training Strategy Decomposition: The staged strategy significantly alleviates training difficulties and achieves highly efficient training.The third-stage adaptation converges significantly faster because of prior knowledge established in earlier stages.
- Efficient T2I Transformer: PIXART-α adds cross-attention to DiT for text conditioning and replaces class-condition processing with the more efficient adaLN-single design.adaLN-single computes shared time-conditioned scales and shifts through a global MLP and layer-specific embeddings, while re-parameterization preserves pretrained-weight compatibility.
- Efficient T2I Transformer: Global time-conditioning parameters and cross-attention preserve generative abilities while reducing model size.The design replaces layer-specific MLPs with a global MLP and layer-wise embeddings for time-step information.
- Dataset Construction: LLaVA-generated captions and diverse SAM images increase concept density, improving the information available for text-image alignment learning.Compared with LAION, SAM-LLaVA contains 328M total nouns and averages 30 nouns per image; LLaVA-labeled captions also increase valid noun proportion and average noun count.
3 EXPERIMENT
The experiments evaluate PIXART-α using fidelity, compositional alignment, human preference, and ablations, showing strong quality with substantially lower training-resource use and efficient architecture choices.
- Fidelity Assessment: PIXART-α achieves a zero-shot COCO FID of 7.32 using 753 A100 GPU days and 25M training images.This corresponds to 12% of the training time and 1.25% of the training samples of the second most efficient method.
- Alignment Assessment: PIXART-α performs outstandingly on nearly all T2I-CompBench alignment metrics, including attribute binding, object relationships, and complex compositions.It succeeds on 5 of 6 evaluation metrics.
- User Study: In a 300-prompt user study, PIXART-α receives higher preference for both image quality and text-image alignment than compared models.Fifty participants ranked models on perceptual quality and prompt-image alignment.
- Ablation Study: The adaLN-single design reduces GPU memory from 29GB to 23GB and parameters from 833M to 611M while producing visual results on par with adaLN.These reductions correspond to 21% lower memory use and 26% fewer parameters.
- Ablation Study: Removing re-parameterization produces distorted target images and missing details across the test set.The comparison adds 200K iterations to compensate for the omitted pretraining iterations.
4 RELATED WORK
The paper situates its approach within diffusion-based image generation, latent diffusion, and Diffusion Transformer research.
- Related Work: Related work covers denoising diffusion probabilistic models, latent diffusion models, and Diffusion Transformers.The review identifies these as the paper’s three main related-work areas.
5 CONCLUSION
The conclusion presents PIXART-α as a Transformer-based T2I diffusion model combining high image-generation quality with substantially reduced training costs and CO2 emissions.
- Conclusion: PIXART-α achieves superior image-generation quality while significantly reducing training costs and CO2 emissions.The conclusion attributes this outcome to training strategy decomposition, an efficient T2I Transformer, and high-informative data.
- Conclusion: The authors report near-commercial image-generation quality and position PIXART-α as a low-cost basis for building T2I models.They specifically identify the AIGC community and startups as potential users.
A.1.1 DENOISING DIFFUSION PROBABILISTIC MODELS
The section describes diffusion models as iterative denoising systems for image generation and contrasts them with earlier generative-model families.
- Denoising Diffusion Probabilistic Models: Diffusion models use an iterative denoising process to transform Gaussian noise into images.The passage contrasts this process with earlier GAN, VAE, and Flow-based generative models.
A.1.2 LATENT DIFFUSION MODEL
Latent Diffusion Model improves diffusion modeling by operating in image latent space and using cross-attention for control. Its results have influenced subsequent text-to-image research and low-cost adaptation methods.
- Latent Diffusion Model applies score-matching in image latent space rather than directly in image space.
- Cross-attention-based controlling enables the model to condition image generation on text.
- Stable Diffusion and its variants have supported later text-to-image improvements, low-cost fine-tuning, and customization techniques.
A.1.3 DIFFUSION TRANSFORMER
Transformer architectures have achieved broad success in language modeling and are promising for computer vision. Their applications include classification, object detection, and semantic segmentation.
- Transformer architecture has achieved major success in language models.
- Recent work shows Transformers are promising across computer vision tasks.
- Reported computer vision applications include image classification, object detection, and semantic segmentation.
A.2 PIXART-α vs. MIDJOURNEY
Figure 7 compares PIXART-α with Midjourney using randomly sampled online prompts while hiding which method produced each image. Readers are invited to judge the results from the prompts.
- Figure 7 compares PIXART-α and Midjourney using randomly sampled prompts from online sources.
- The figure conceals each image’s generating method from readers.
- Readers are encouraged to assess the images using the provided prompts.
A.3 PIXART-α vs. PRESTIGIOUS DIFFUSION MODELS
PIXART-α is evaluated against prestigious diffusion models through captioning, implementation analysis, quantitative metrics, visual quality, customization, and failure-case studies. The comparisons report strong generation quality and versatility, while identifying metric and control limitations.
- Visual comparisons: PIXART-α is compared with powerful generative models using RAPHAEL-selected prompts, with results reported as comparable to or better than existing models.
- Data and implementation: LLaVA-generated detailed captions are used to produce high-information-density image-text pairs for training.
- Data and implementation: The reported training accounting excludes VAE training, T5 training, and LLaVA auto-labeling time.The excluded components require approximately 25 hours on 64 V100 GPUs for VAE training and 24 hours on 64 V100 GPUs for LLaVA annotation.
- Quantitative evaluation: PIXART-α shows marginal FID improvement over SDv1.5 and consistent performance across classifier-free guidance scales on T2ICompBench.
- Generation quality: PIXART-α generates high-fidelity images up to 1024 × 1024 pixels with rich detail and arbitrary aspect ratios.
- Evaluation caveats: FID may not accurately reflect visual quality because its ImageNet-pretrained feature extractor has limited overlap with text-to-image data.
- Customization: PIXART-α supports DreamBooth and ControlNet customization, producing high-fidelity outputs and personalized extensions from images, prompts, and control signals.
- Architecture discussion: Transformer-based fusion improves compositionality over U-Net generators by modeling long dependencies between multimodal information.