PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models

Dengyang Jiang, Ruoyi Du, Zhennan Chen, Dongyang Liu, Zanyi Wang, Mingzhe Zheng, Xiangpeng Yang, Huanqia Cai, Aiming Hao, Yuming Jiang, Peng Gao, Harry Yang, Steven Hoi

arXiv:2608.16887v1cs.CV

TL;DR

Large-scale pixel-space diffusion lacks a clear training recipe comparable to latent-space models. This paper studies latent-to-pixel adaptation and derives a practical recipe that maintains competitive performance while delivering 3.18×–4.75× faster end-to-end inference.

  • Problem

    Large-scale training recipes for pixel-space text-to-image diffusion remain unclear, with prior work concentrated on class-conditioned or small-scale settings.

  • Method

    The paper compares pixel- and latent-space diffusion at scale, then systematically studies latent-to-pixel adaptation and its key design choices.

  • Results

    3.18×–4.75× faster end-to-end inference accompanies competitive performance against latent-space counterparts and better performance than prior latent-to-pixel methods on most benchmarks.

  • Takeaways & Limitations

    The findings provide practical guidance for scalable, high-quality, and efficient pixel-space diffusion.

  • Takeaways & Limitations

    Resolution-based SNR matching is only a theoretical reference and does not necessarily determine the optimal noise scale for latent-to-pixel adaptation.

Abstract

from arXiv · show

This paper investigates an increasingly important topic in generative modeling: pixel-space diffusion models. Although numerous studies have explored this topic, most focus on small-scale or class-conditional settings. Consequently, a practical recipe for training pixel-space models that rival or exceed well-established latent-space counterparts remains elusive. Through a comprehensive empirical study, we first observe that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This observation motivates a latent-to-pixel strategy that acquires generative priors efficiently in latent space and transitions to pixel space during post-training. We then systematically investigate the key design choices governing this transition, including weight initialization, data composition, prediction target, decoder architecture, and noise schedule, and identify a practical recipe that makes the resulting pixel-space models match or outperform their latent-space counterparts while delivering 3.18 to 4.75 times end-to-end inference speedups. We hope that our findings provide useful empirical insights and practical guidelines for future research on pixel-space generation.

1. Introduction

The introduction motivates pixel-space diffusion as an efficient alternative to latent-space models but highlights limited large-scale understanding and slower pre-training convergence. The paper addresses this gap with latent-to-pixel adaptation and a practical recipe achieving comparable performance with faster inference.

  • Motivation: Pixel-space generation avoids VAE reconstruction limits and decoding overhead while enabling a flexible efficiency–quality trade-off through larger patch sizes.It learns directly from RGB images, generates pixels directly, and can reduce sequence length without increasing image resolution.
  • Problem: Large-scale pixel-space text-to-image training remains poorly understood because prior work largely targets class-conditioned ImageNet or small-scale synthesis with limited data.Latent-space diffusion has received substantially more extensive study and scaling effort.
  • Contribution: Under identical large-scale pre-training settings, pixel-space diffusion converges substantially more slowly than its latent-space counterpart.This finding motivates acquiring generative knowledge through latent-space pre-training before introducing pixel-space learning.
  • Contribution: The paper systematically studies latent-to-pixel adaptation at scale and establishes a practical recipe for effective pixel-space post-training.The investigation revisits foundational training choices for building an efficient, competitive pixel-space text-to-image model.
  • Contribution: 3.18×–4.75× faster end-to-end inference is achieved while pixel-space models deliver comparable overall performance to established latent-space counterparts.The result is reported for the final recipe developed through the paper’s latent-to-pixel adaptation study.

2. Related Work

Related work contrasts efficient but reconstruction-limited latent diffusion with increasingly capable pixel-space diffusion, while noting that large-scale text-to-image studies of pixel-space generation remain scarce.

  • Latent-space diffusion: Latent diffusion reduces computational and memory costs by denoising in an autoencoder-derived latent space.Its development spans architectures, prediction paradigms, training and inference methods, and distributed systems.
  • Latent-space diffusion: Autoencoder reconstruction bottlenecks can discard fine-grained details and limit sample fidelity in latent diffusion.Large-scale autoencoder training also demands substantial data, computation, and empirical tuning.
  • Pixel-space diffusion: Pixel-space diffusion has narrowed its performance gap through increased compute, improved network designs, and advances in training objectives.On moderate-scale datasets such as ImageNet, pixel-space methods achieve performance comparable to or better than latent-space methods.
  • Pixel-space diffusion: Large-scale studies of pixel-space diffusion for generation-centric models remain scarce, especially for challenging text-to-image generation.The paper addresses this gap with empirical insights and practical training guidelines.

3. Large-Scale Pixel-Space Pre-training Converges More Slowly Than Latent-Space

Under matched large-scale pre-training settings, latent-space diffusion converges faster and performs better than pixel-space diffusion. The gap is especially pronounced early in training and persists under the same compute budget, motivating alternatives to direct pixel-space pre-training.

  • The comparison holds architecture, data, compute, and training configuration constant while varying only the prediction space.
  • Latent-space pre-training consistently outperforms pixel-space pre-training, with the largest gap during early learning of object structure and text–image alignment.The pixel model improves but does not close the gap under the same compute budget.
  • A VAE likely helps by compressing raw pixels into a lower-dimensional, perceptually structured representation that suppresses redundancy and high-frequency variation.This representation preserves information relevant to image semantics and appearance while simplifying the distribution the diffusion model must learn.
  • Pixel-space training must learn global structure, fine-grained local statistics, and denoising directly from high-dimensional signals, increasing optimization difficulty even with more than 20 billion image–text pairs.
  • Under identical large-scale pre-training settings, latent-space diffusion learns image structure and text–image alignment substantially faster, making direct pixel-space pre-training potentially suboptimal.

4. A Carefully Designed Latent-to-Pixel Transition Offers a Better Trade-off

The paper advocates pre-training in latent space before post-training directly in RGB space, using transferred generative priors to accelerate pixel-space optimization. It finds that self-generated and real data, x-prediction, the DiP decoder, and empirical noise calibration together provide an effective quality–efficiency trade-off.

  • Transition strategy: Latent-to-pixel training first pre-trains a latent diffusion model, then introduces pixel-space supervision during post-training to obtain a direct RGB generator.This avoids the VAE reconstruction bottleneck and additional decoding stage while leveraging faster latent-space pre-training.
  • Weight initialization: Latent-space weight initialization accelerates pixel-space convergence and enables earlier formation of recognizable objects, coherent layouts, and stable text–image alignment.It consistently yields higher GenEval and DPG scores than training from scratch under identical settings.
  • Training data: Self-generated data enables rapid adaptation, but combining it with high-quality real images provides the best trade-off by correcting inherited artifacts and preserving stable convergence.Self-generated samples align with the source model’s conditional distribution, whereas real-image supervision recovers information lost through latent compression.
  • Prediction target: x-prediction consistently outperforms v-prediction for pixel-space post-training, including when initialization comes from a latent-space v-prediction model.The results indicate that latent initialization stabilizes the transition between prediction objectives.
  • Decoder architecture: 2.25B decoder parameters and 19,731 GFLOPs make PiT impractical, whereas DiP achieves the best quality–efficiency trade-off with 10.09M parameters and 834 GFLOPs.DiP’s lightweight convolutional decoder also produces smoother transitions across neighboring patches.
  • Noise schedule: γ= 2 achieves the best performance on both GenEval and DPG, while the theoretically motivated resolution-derived value γ= 8 is suboptimal.Resolution-based SNR analysis provides a reference, but empirical calibration is necessary because the latent-to-RGB distribution shift exceeds spatial-resolution differences.

5. Optimizing Efficiency with Larger-Patch Adaptation and Step Distillation

Progressive patch-size adaptation reduces visual tokens while preserving quality better than direct larger-patch training, and step distillation further improves pixel-space inference efficiency. The combined approach achieves substantial end-to-end latency reductions without the latent-space VAE decoding bottleneck.

  • Larger-Patch Adaptation: Direct ps32 training reduces token count by 4× but converges slowly and produces visible local artifacts.Unlike ps16, direct ps32 simultaneously changes prediction space and coarsens the token grid, making local visual-prior transfer more difficult.
  • Larger-Patch Adaptation: ps32-adapt16 converges substantially faster than direct ps32 training and achieves comparable GenEval and DPG performance to ps16.Separating the pixel-space transition from spatial-granularity changes transfers stable visual and local-detail priors while largely removing local artifacts.
  • Larger-Patch Adaptation: 4× fewer visual tokens are achieved by adapting from ps16 to ps32, with only a small quality gap.The paper identifies preserving fine-grained details under more extreme token compression as an important direction for future work.
  • Step Distillation: 0.20 s per image is required by the distilled pixel-space model versus 0.95 s for its latent-space distilled counterpart, yielding a 4.75× latency reduction.Pixel-space generation removes the fixed VAE decoding bottleneck, allowing NFE reductions to translate more directly into end-to-end speedups.

6. System-Level Comparison

The final pixel-space recipe is evaluated against latent-space counterparts and prior pixel-space adaptations on two model families at 100 and 4 NFE. It delivers competitive or better performance with substantially faster inference, including 4.41×–4.75× speedups for Z-Image and 3.18×–3.29× for FLUX2-klein.

  • Evaluation setup: At 100 NFE, the comparison includes latent-space counterparts and L2P for Z-Image, plus AsymFlow for FLUX2-klein; at 4 NFE, distilled models are compared with official latent-space counterparts.The evaluation uses the final models obtained after scaling mixed training data and extending training.
  • Z-Image: 4.41× and 4.75× speedups at 100 and 4 NFE accompany comparable or better overall Z-Image performance than latent-space counterparts.The pixel-space models also outperform L2P across all four benchmarks with substantially lower latency.
  • FLUX2-klein and generalization: 3.18×–3.29× speedups for FLUX2-klein accompany better results than both latent-space baselines and AsymFlow, indicating transfer beyond the primary Z-Image family.Across two model families, the recipe provides a practical approach to efficient, high-performing pixel-space models and outperforms prior latent-to-pixel methods on most benchmarks.

7. Conclusion

The study finds that direct large-scale pixel-space pre-training converges substantially more slowly than latent-space pre-training, motivating a latent-to-pixel strategy. It then examines the transition’s key design choices to derive a practical recipe.

  • Direct pixel-space pre-training converges substantially more slowly than latent-space pre-training.
  • The latent-to-pixel strategy acquires generative priors efficiently in latent space before adapting the model to pixel space during post-training.
  • The study examines the key design choices underlying the latent-to-pixel transition and derives a practical recipe.

Appendix · A. Additional Implementation Details

This appendix section documents implementation details for the main-paper experiments, including configurations summarized in Tables 4 and 5. Unless specified otherwise, component-wise comparisons hold the backbone, text conditioning, preprocessing, optimization, and evaluation protocol fixed.

  • A. Additional Implementation Details: The appendix provides additional implementation details for the experiments presented in the main paper.It frames the section as supplementary experimental documentation.
  • A. Additional Implementation Details: Configurations used in the component-wise studies are summarized in Tables 4 and 5.These tables collect the configurations used for the component-wise comparisons.
  • A. Additional Implementation Details: Unless otherwise specified, comparisons keep the backbone architecture, text-conditioning modules, data preprocessing, optimization strategy, and evaluation protocol fixed.Only the configurations under study vary by default.

A.1. Large-Scale Pixel Versus Latent-Space pre-training

The controlled comparison uses identical transformer and text-conditioning components for pixel- and latent-space models, while varying only the prediction space under a shared large-scale pre-training recipe. The latent model uses Flux-AE features with 16× effective spatial compression, whereas the pixel model uses ps16.

  • Both models share the same Z-Image Transformer backbone and pre-trained Qwen-3-4B text-conditioning stack.
  • The latent-space model uses Flux-AE features with spatial downsampling factor 8 and latent patch size 2, yielding 16× effective spatial compression.For 1024^2 image generation, this corresponds to a 64 × 64 token grid.
  • The pixel-space model uses ps16 in the controlled comparison.
  • Both models train on more than 20B image–text pairs with the original Z-Image recipe, varying only the prediction space.The shared recipe includes progressive resolution from 256^2 to 512^2, batch size, learning rate, and optimizer.

A.2. Latent-to-Pixel Transition · A.3. System-Level Comparison

The paper uses latent-pretrained initialization and controlled pixel-space post-training to identify a practical transition recipe, then applies it in a final system-level comparison to Z-Image and FLUX2-klein. The recipe combines mixed self-generated and real data, x-prediction, the DiP decoder, γ=2, and progressive patch-size adaptation.

  • A.2. Latent-to-Pixel Transition: The default pixel-space setup initializes from a Z-Image latent checkpoint and trains the full model directly at 1K resolution with batch size 128 and learning rate 5e-5.The default configuration also uses mixed self-generated and real data, x-prediction, the DiP decoder, γ=2, and ps16.
  • A.2. Latent-to-Pixel Transition: The initialization study compares latent-pretrained and from-scratch models while holding architecture, data, optimization, learning-rate schedule, and update count constant.The latent-initialized model transfers Transformer and conditioning weights, while pixel-specific input and output modules are initialized as in the from-scratch baseline.
  • A.2. Latent-to-Pixel Transition: The data study evaluates real images only, self-generated samples only, another model’s samples, and mixed self-generated and real images.Real images come from the latest Z-Image SFT data, while self-generated samples use the same latent Z-Image checkpoint as initialization.
  • A.2. Latent-to-Pixel Transition: The prediction study compares v-prediction with x-prediction under identical flow-matching, timestep-sampling, noise-scale, backbone, and decoder settings.x-prediction estimates the clean RGB target x0, whereas v-prediction estimates velocity along the interpolation path.
  • A.2. Latent-to-Pixel Transition: The decoder study changes only the feature-to-RGB module, comparing JiT’s linear projection, DiP’s lightweight convolutional U-Net, and PiT’s decoder design.All variants share the latent-initialized Transformer backbone, mixed data, x-prediction, noise scale, and ps16 tokenization; decoder-specific modules start from scratch.
  • A.2. Latent-to-Pixel Transition: The noise-schedule study evaluates γ∈{1, 2, 4, 8} with RGB images normalized to [-1,1], selecting γ=2 for subsequent experiments based on GenEval and DPG.Training and sampling use the same γ, while timestep distribution, prediction target, loss weighting, initialization, and optimization schedule remain fixed.
  • A.3. System-Level Comparison: The final system-level comparison scales the mixed dataset and extends pixel-space post-training before applying the recipe to Z-Image and FLUX2-klein.For FLUX2-klein, initialization uses FLUX2-klein-9B-Base with a synthetic-to-real ratio of 1:1; other recipe details are retained except for an architectural change described afterward.
Loading 2608.16887v1…