PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation

Motivation

• Training top-tier text-to-image models requires substantial computational resources, creating barriers for researchers with limited resources.Training SD1.5 from scratch requires about 6000 A100 GPU days.Source: §1 · p. 1

Weak-to-Strong Training

• Weak-to-strong training builds PixArt-Σ on PixArt-α’s pre-trained foundation and evolves the weaker baseline into a stronger model through efficient training.The strategy includes replacing the VAE, scaling from low to high resolution, and adding KV compression.Source: §1 · p. 3

Higher-Quality Training Data

• The training dataset contains 33M high-resolution images, including 2.3M images around 4K, paired with dense captions from Share-Captioner.All images exceed 1K resolution, and the text encoder’s token length is extended to improve textual-visual alignment.Source: §1 · p. 3

Efficient Token Compression

• Key-value token compression uses stride-2 group convolutions for local aggregation, reducing high-resolution training and inference time by ∼34%.A specialized initialization scheme adapts compression smoothly from a pre-trained model without KV compression.Source: §1 · p. 3

Results

• PixArt-Σ directly generates high-quality 4K images with 0.6B parameters and uses only 9% of PixArt-α’s GPU days for fine-tuning.The model also demonstrates fine-grained textual prompt alignment and aesthetic quality comparable to leading text-to-image products.Source: §1 · p. 4

Diffusion Transformers

• PixArt-α achieved high-quality 1024px image generation through efficient text-to-image training schemes within the Diffusion Transformer framework.Related models include DiT, UViT, GenTron, and later Transformer-based diffusion systems.Source: §2 · p. 4

Comparisons

• Qualitative comparisons report improved portrait realism and semantic analysis over PixArt-α, plus stronger instruction following than SDXL.The figure highlights prompt keywords in blue.Source: Figure 3 · p. 5

PixArt-Σ

• PixArt-Σ is presented as the first work to use a Transformer architecture for direct 4K ultra-high-resolution image generation.The approach addresses the computational complexity caused by long token sequences.Source: §2 · p. 5

High-Resolution Generation

• Prior high-resolution methods commonly add super-resolution networks or multiple diffusion stages, which can introduce cumulative errors.Imagen, GigaGAN, Stable Diffusion, and Stable Cascade exemplify these combined-model solutions.Source: §2 · p. 6

Efficient Transformer Architecture

• Transformer self-attention has quadratic computational complexity in token count, motivating sparse, local, and low-rank attention alternatives.PVT v2 condenses key and value spaces with convolution, while Swin Transformer restricts computation to windows.Source: §2 · p. 7

Higher Aesthetic and higher Resolution

• As image resolution increases, model fidelity and semantic alignment improve according to FID and CLIP Score.The paper uses this observation to motivate training directly on higher-resolution data.Source: §3.1 · p. 8

Efficient DiT Design

• KV compression merges spatially adjacent key and value tokens while retaining all query tokens, reducing attention computation for high-resolution generation.The module uses convolutional aggregation and compresses keys and values rather than queries.Source: §3.2 · pp. 9–10, Figure 7 · p. 9
• The Conv Avg Init convolution starts as an average operator, enabling coarse initial outputs during fine-tuning while adding only 0.018% parameters.This initialization uses group convolution with weights initialized to 1/R^2.Source: §3.2 · p. 10

Weak-to-Strong Training Strategy

• Weak-to-strong training combines VAE replacement, low-to-high-resolution scaling, and progression from uncompressed to KV-compressed models.The strategy is illustrated as accelerating convergence during these transitions.Source: §3.3 · pp. 10–11, Figure 8 · p. 10

Image Quality Assessment

• PixArt-Σ improves portrait realism and semantic analysis over SDXL and PixArt-α while adhering more closely to user instructions than SDXL.These findings come from qualitative comparisons with open-source models.Source: §4.2 · p. 12
• PixArt-Σ produces photo-realistic images and adheres closely to user instructions, performing competitively with contemporary closed-source products.The comparison is reported in Fig. 4 against current commercial T2I products.Source: §4.2 · p. 12

High-resolution Generation

• PixArt-Σ directly generates 4K images without post-processing and follows complex, detailed, and long user prompts.The paper states that satisfactory results do not require prompt engineering.Source: §4.2 · p. 12
• The paper argues that direct 4K generation avoids cascade-related cumulative errors and learns text alignment from 4K image distributions.This contrasts with post-processing and super-resolution pipelines described in the discussion.Source: §4.2 · p. 12

Performance Comparisons

• Human and GPT-4 Vision preference studies show PixArt-Σ surpassing PixArt-α and matching or exceeding Stable Cascade on image quality and instruction following.Human evaluators ranked quality and prompt matching, while GPT-4 Vision voted on image quality and image-text alignment.Source: Figure 9 · p. 12, §4.2 · p. 13

Experimental setup

• Unless specified, KV-compression ablations use 512px generation, with detailed experimental settings provided in the appendix.FID and CLIP-Score are used to compare generated-image distribution and prompt alignment.Source: §5 · pp. 13–14, §5.1 · p. 14

Compression positions

• KV compression in deep Transformer layers achieves superior performance to compression in shallow or intermediate layers.The authors attribute this to deep layers encoding higher-level semantics, which are less affected by compression than shallow texture features.Source: §5.2 · p. 14

Compression operators

• Conv 2×2 outperforms random discarding and average pooling when merging 2×2 tokens into one token.The result supports using a learnable kernel to reduce redundant features.Source: §5.2 · p. 14

Compression ratios on different resolutions

• The experiments vary compression ratios across resolutions and report speed using seconds per iteration for batch size 384.The study evaluates compression ratio, position, operator, resolution, and speed.Source: §5.2 · p. 14

• PixArt-Σ directly generates high-quality images at 4K resolution.Source: §6 · p. 15
• Weak-to-strong training upgrades PixArt-α by incorporating higher-quality data and efficient token compression.Source: §6 · p. 15
• PixArt-Σ produces high-fidelity images while adhering closely to textual prompts.Source: §6 · p. 15

• DMD accelerates PixArt inference through distribution matching distillation and a one-step generator.The generated image is represented as x0 = Gθ(T, text).Source: §A.1 · p. 16
• The denoising timestep T was initially set to 999, matching the training denoising timesteps.Source: §A.1 · p. 16
• Figure 10 reports Base model + DMD performance over T.Source: Figure 10 · p. 16
• Table 4 compares PixArt + DMD with PixArt + LCM at 512x512 resolution and batch size 1.Source: Table 4 · p. 16

A.2 Training Details of PixArt-Σ

• PixArt-Σ training stages record image resolution, training samples, steps, batch size, learning rate, and GPU-day compute.Source: §A.2 · p. 16
• KV token compression reduces 512px-to-1024px fine-tuning time from 50 to 20 V100 GPU days.Source: §A.2 · pp. 16–17
• Figure 11 provides a visual comparison of Base model + DMD and Base model + LCM.Source: Figure 11 · p. 17
• KV token compression reduces 2K training time from 20 to 14 A800 GPU days and 4K training time from 25 to 20 A800 GPU days.Source: §A.2 · p. 17

• Ablations evaluate FID on 30,000 curated prompts against images from the High-Quality Evaluation Dataset.Source: §A.3 · p. 17
• CLIP scores compare model-generated images with the original 30,000 prompts.Source: §A.3 · p. 17
• GPU-day counts exclude offline VAE-feature and T5-text-feature extraction because both features are prepared before training.Source: Table 5 · p. 18
• Testing-image count affects FID, with 10,000-image datasets typically yielding higher scores.Source: §A.3 · p. 18
• Compression-position and operator experiments use images generated at 512px resolution.Source: §A.3 · p. 18

High-Quality Evaluation Dataset

• 30,000 high-quality text-image pairs form the proposed evaluation dataset for assessing visually appealing images.The dataset emphasizes superior aesthetic quality and precise text-image alignment.Source: §A.4 · pp. 18–19

Comparisons with T2I Products

• PixArt-Σ produces high-quality, photo-realistic images with rich details comparable with Firefly 2, Imagen 2, Dalle 3, and Midjourney 6.The comparison is presented through visual results in Figures 14 and 15.Source: §A.8 · p. 20, Figure 14 · p. 23, Figure 15 · p. 24

Generated Samples

• The authors report that PixArt-Σ samples closely match provided textual prompts and exhibit high fidelity.These qualities are illustrated in Figures 16, 17, and 18.Source: §A.9 · p. 21

Limitations

• PixArt-Σ still struggles with specific scenes and objects, especially text and hands, and may fail on complex prompts or produce flawed faces.The paper also notes that sensitive content may be generated.Source: §A.10 · p. 22

Generated Samples

• PixArt-Σ generates high-quality images with fine-grained details and diverse aspect ratios.The samples illustrate variation in both visual detail and image proportions.Source: Figure 16 · p. 25

4K Generation

• PixArt-Σ directly generates 4K HD images at 3840×2560 while preserving fine-grained details.Figures 17 and 18 provide high-resolution sample outputs.Source: Figure 17 · p. 26, Figure 18 · p. 27