PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelDiT: Pixel Diffusion Transformers for Image Generation

Yongsheng Yu, Wei Xiong, Weili Nie, Yichen Sheng, Shiqiu Liu, Jiebo Luo

arXiv:2511.20645v2cs.CV

TL;DR

Latent diffusion is efficient but its autoencoder can introduce lossy reconstruction and separate-stage optimization issues. PixelDiT addresses this with end-to-end pixel-space diffusion using dual-level transformers and efficient pixel modeling, achieving strong class- and text-conditional results. The approach reaches image quality on par with latent diffusion within the reported tasks, while complex object generation and training stability remain limitations.

  • Problem

    Latent diffusion’s autoencoder can remove high-frequency details and create misalignment between reconstruction and downstream generative objectives.

  • Method

    PixelDiT uses a single-stage, fully transformer-based pixel-space diffusion model with patch-level semantic modeling, pixel-level texture refinement, pixel-wise AdaLN, and token compaction.

  • Results

    PixelDiT achieves high image quality in class-conditional and text-conditional generation, significantly outperforming existing pixel-space models and approaching state-of-the-art latent diffusion models.

  • Takeaways & Limitations

    Effective and efficient pixel modeling is identified as the key to practical pixel-space diffusion.

  • Takeaways & Limitations

    The 1.3B-parameter text-to-image model sometimes struggles with geometrically and texturally complex objects, while velocity prediction can produce loss spikes in deeper or lengthy training runs.

Abstract

from arXiv · show

Latent-space modeling has been the standard for Diffusion Transformers (DiTs). However, it relies on a two-stage pipeline where the pretrained autoencoder introduces lossy reconstruction, leading to error accumulation while hindering joint optimization. To address these issues, we propose PixelDiT, a single-stage, end-to-end model that eliminates the need for the autoencoder and learns the diffusion process directly in the pixel space. PixelDiT adopts a fully transformer-based architecture shaped by a dual-level design: a patch-level DiT that captures global semantics and a pixel-level DiT that refines texture details, enabling efficient training of a pixel-space diffusion model while preserving fine details. PixelDiT achieves 1.61 FID on ImageNet 256 and 1.81 FID on ImageNet 512, surpassing existing pixel generative models. We further extend PixelDiT to text-to-image generation and pretrain it at the 10242resolution in pixel space. It achieves 0.74 on GenEval and 83.5 on DPG-bench, approaching the best latent diffusion models. Code: https://github.com/NVlabs/PixelDiT

1. Introduction

PixelDiT addresses the practical difficulty of pixel-space diffusion by separating global semantic modeling from dense pixel-level detail refinement in a single-stage transformer. Its design targets efficient training while preserving fine visual details and achieves strong image-generation results without an autoencoder.

  • Motivation: Pixel-space diffusion must capture dense per-pixel interactions and high-frequency details, unlike coarse patch tokens that mainly encode semantic structure.Per-pixel modeling improves texture fidelity but is computationally expensive.
  • Motivation: Aggressive patchification reduces attention cost but weakens pixel-token modeling, while near-pixel designs incur long-sequence quadratic attention or heavy decoder costs.Cascaded pipelines reduce some costs but add stages and may accumulate errors.
  • Approach: PixelDiT is a single-stage, fully transformer-based model trained and sampled end-to-end directly in pixel space without a separate autoencoder.This design explicitly structures pixel modeling rather than relying on latent-space compression.
  • Approach: A dual-level architecture assigns global layout and content to a patch-level DiT and local texture refinement to a pixel-level DiT.The patch pathway uses short token sequences for long-range attention, while the pixel pathway performs dense token modeling.
  • Approach: Pixel-wise AdaLN conditions pixel tokens on semantic context, while token compaction keeps global attention efficient during dense pixel modeling.Together, these mechanisms support faster convergence and preserve fine details.
  • Results: 1.61 FID on ImageNet 256 × 256 demonstrates high-quality pixel-space generation without autoencoders, while extensions cover text-to-image generation at 1024^2 resolution.The text-to-image model achieves competitive scores against state-of-the-art latent diffusion models and avoids VAE reconstruction artifacts in editing.

2. Related Works

Related work established latent diffusion as an efficient standard while exposing reconstruction and optimization trade-offs. Pixel-space research pursued direct denoising, cascades, simplified networks, hierarchical flows, neural fields, and alternative two-stage or hybrid designs.

  • Latent diffusion: Latent diffusion denoises compressed representations, substantially reducing compute and memory and enabling higher-resolution training with larger backbones on fixed budgets.Most latent diffusion models use a variational autoencoder that trades reconstruction fidelity against compression rate.
  • Latent diffusion: Prior work explored stronger autoencoders, tokenizers, and end-to-end VAE–diffusion alignment to improve latent representations.Representation autoencoders have also been used without explicit variational modeling.
  • Latent diffusion: Autoencoder-based latent diffusion inherits a reconstruction bottleneck: aggressive compression can remove high-frequency details and limit sample fidelity.The autoencoder’s reconstruction objective may also be misaligned with downstream generation.
  • Pixel-space diffusion: Pixel-space methods include direct denoising, cascaded multi-scale diffusion, simplified convolutional networks, fractal designs, hierarchical flow models, and neural-field layers.These approaches target image quality, high-resolution synthesis, memory efficiency, long-range structure, or efficient pixel-space modeling.
  • Pixel-space diffusion: Other concurrent approaches combine self-supervised pretraining with generative fine-tuning, normalizing flows with autoregression, or plain Transformers with x_0-prediction.These methods differ from PixelDiT’s purely transformer-based architecture trained directly at 1024^2 resolution.

3. Method

PixelDiT performs diffusion directly in pixel space with a dual-level transformer: patch-level processing captures global semantics, while pixel-level processing refines local details efficiently.

  • Overview: PixelDiT directly performs denoising in pixel space using a transformer architecture designed to make pixel token modeling computationally efficient.The model is trained and sampled end-to-end without a separate autoencoder.
  • Patch-level architecture: The patch-level pathway processes coarse patch tokens with global attention to capture image semantics and long-range layout.Global conditioning parameters are broadcast across patch tokens at this level.
  • Pixel-level architecture: Patch-level semantic tokens combined with the timestep embedding condition the pixel-level pathway, which performs dense pixel-token modeling for detail refinement.The pixel-level DiT uses PiT blocks and receives pixel tokens together with s_cond.
  • Pixel-level architecture: Pixel-wise AdaLN expands each semantic token into distinct scale, shift, and gating parameters for every pixel, enabling spatially varying updates.Unlike patch-wise modulation, it does not repeat one parameter set across all pixels in a patch.
  • Pixel-level architecture: Pixel token compaction compresses the p^2 pixels within each patch before global attention and expands them afterward, reducing the sequence length by p^2.With p=16, the attention sequence receives a 256× reduction while preserving per-pixel updates.
  • Text-to-image extension: For text-to-image generation, MM-DiT blocks fuse text and image semantics in the patch-level pathway, while the pixel-level pathway remains conditioned through semantic tokens and the timestep.No text tokens are routed directly to the pixel stream.
  • Training objectives: Training uses a pixel-space Rectified Flow velocity-matching loss together with an alignment objective based on frozen DINOv2 features.The combined objective is L = L_diff + lambda_repa L_repa and is used for both class- and text-conditional models.

4. Experiments

Experiments show that PixelDiT scales across ImageNet class-conditioned and text-to-image generation while preserving quality and improving pixel-space efficiency. Ablations attribute these results to dual-level pixel modeling, pixel-wise AdaLN, token compaction, and compact attention.

  • Class-conditioned Image Generation: PixelDiT-XL achieves 1.61 gFID on ImageNet 256×256, surpassing PixelFlow-XL, PixNerd-XL, and EPG-XXL/16.At 320 epochs, it obtains gFID 1.61 versus 1.98, 1.93, and 1.81 for the cited pixel-space models.
  • ImageNet 512×512: PixelDiT achieves 1.81 gFID, 5.61 sFID, 278.6 IS, 0.78 precision, and 0.67 recall on ImageNet 512×512.It outperforms prior pixel-space models and achieves the highest recall among pixel methods at this resolution.
  • Text-to-Image Generation: PixelDiT-T2I reaches GenEval 0.74 and DPG 83.5 at 1024^2 resolution, surpassing many latent diffusion models on GenEval.At 512×512, it achieves GenEval 0.78 and DPG 83.7 while outperforming recent pixel-space models.
  • Efficiency: PixelDiT-T2I reaches 0.33 samples per second at 1024^2 resolution, comparable to latent diffusion models despite direct pixel-space denoising.At 512×512, throughput is 1.07 samples per second and exceeds PixelFlow and PixNerd.
  • Ablation Studies: Pixel-level token modeling is necessary for fine details, while removing pixel-pathway attention reduces GFLOPs but worsens gFID and IS.Ablations show that pixel-wise AdaLN and token compaction improve convergence, and compact global attention remains necessary for local-update alignment.

5. Conclusion

The paper argues that pixel-space diffusion can largely close its quality gap with latent diffusion through suitable architectural and training design. PixelDiT supports this claim by separating semantic learning from fine-grained detail refinement with efficient pixel modeling.

  • 5. Conclusion: Pixel-space methods face greater data and noise complexity and lack a mature training recipe spanning architecture, objectives, scheduling, and optimization.These challenges are presented as likely sources of the quality gap from latent diffusion models.
  • 5. Conclusion: PixelDiT uses a dual-level transformer with pixel-wise AdaLN and pixel token compaction to disentangle semantic modeling from detail refinement.The architecture is described as enabling efficient pixel modeling for both class-conditional and text-conditional generation.
  • 5. Conclusion: The paper reports that appropriate architectural design can largely close the gap between pixel-space and latent-space diffusion quality.The conclusion identifies effective and efficient pixel modeling as central to practical pixel-space diffusion.

A. Architecture and System Details

The appendix describes PixelDiT configurations and its text-to-image architecture. The patch-level pathway handles model scaling and text fusion, while the pixel-level pathway remains dedicated to pixel modeling.

  • Architecture and System Details: PixelDiT experiments use Base, Large, and Extra Large variants, with XL as the default ImageNet configuration.N and M denote patch-level and pixel-level pathway depths, while D and Dpix denote their hidden dimensions.
  • Architecture and System Details: The T2I variant adds MM-DiT blocks to the patch-level pathway to fuse text embeddings while leaving the pixel-level pathway unchanged.The architecture emphasizes separated streams, conditioning flow, and pixel-wise modulation into the pixel-level pathway.

B.1. Ablation of Solvers

Solver experiments compare FlowDPMSolver, Euler, and Heun under fixed sampling conditions and support FlowDPMSolver as the default sampler. Sampling-budget and guidance studies further show that benefits depend on training stage and configuration.

  • Ablation of Solvers: FlowDPMSolver provides the best combined trade-off across gFID, sFID, IS, precision, and recall in 100-step ImageNet 256×256 comparisons.It is therefore used as the default sampler for the main ImageNet and text-to-image evaluations.
  • Inference Steps: For well-trained models, increasing FlowDPMSolver’s budget improves gFID, with 100 steps reaching approximately 1.61 and gains beyond 50 steps becoming small.At 100K iterations, additional steps provide only modest improvements because the model is undertrained.
  • Classifier-Free Guidance: The best guidance setting shifts with training: scale 3.25 over [0.10, 1.00] gives gFID 2.36 at 80 epochs, while scale 2.75 over [0.10, 0.90] gives gFID 1.61 at 320 epochs.The 320-epoch setting also retains recall 0.64.

C.1. Ablation on Depth N and M

Depth allocation and token compaction materially affect pixel-space diffusion efficiency and quality. Moderate pixel-pathway depth and aggressive compaction provide the strongest reported trade-offs, while finer patch tokenization remains important at higher resolution.

  • Depth allocation: Even a shallow pixel pathway, such as N=28 and M=2, reduces final gFID to around 2.1 and improves convergence.The comparison uses a fixed total budget of roughly N+M≈30 layers.
  • Depth allocation: A 26-layer patch pathway and 4-layer pixel pathway reaches gFID 2.34 by 300K iterations and 1.94 at 1M iterations.This configuration outperforms both 28/2 and 22/8 allocations; the deeper pixel pathway converges more slowly early.
  • Patch size: At 120 epochs on ImageNet-512, p=16 achieves FID 2.23 compared to 3.78 for p=32.The larger patch reduces computation but consistently underperforms the finer tokenization.
  • Editing preservation: PixelDiT achieves substantially better background consistency than FLUX and SD3, including 6.0× lower MSE and higher SSIM than FLUX.The evaluation uses background regions outside editing bounding boxes across 281 FlowEdit source-target pairs.

E. FLOPs Estimation and Comparison

PixelDiT-XL narrows the compute gap between pixel- and latent-space generation. Its reported ImageNet-256 quality is achieved at compute close to latent models and below most prior pixel-space models.

  • Compute comparison: PixelDiT-XL obtains 1.61 FID with 311 GFLOPs on ImageNet 256×256.Latent models use around 240–290 GFLOPs, while many prior pixel models require several hundred to several thousand GFLOPs.
  • Resolution scaling: At fixed patch size, compute scales roughly quadratically with image resolution because self-attention operates over the patch-token sequence.This scaling is summarized in the GFLOPs breakdown across resolutions and patch sizes.
  • Patch-size efficiency: At 1024² resolution, increasing patch size from p=8 to p=16 reduces GFLOPs by 7.4×, and p=32 reduces them by a further 3.1×.The reductions demonstrate the compute benefit of larger patches, although patch-size quality trade-offs are evaluated separately.

F. Extended Training Iterations

Longer training improves PixelDiT-XL on ImageNet, with the largest gains occurring before the longest schedules. Extended high-resolution fine-tuning also improves generation quality.

  • Extended pre-training: Extending ImageNet-256 training from 320 to 800 epochs lowers gFID from 1.61 to 1.54.Performance improves steadily through 320 epochs, while the additional 800-epoch gains are smaller.
  • Extended pre-training: From 80 to 320 epochs, gFID decreases from 2.36 to 1.61 and recall rises from 0.57 to 0.64.sFID and IS also improve over this interval.
  • High-resolution fine-tuning: Longer fine-tuning at 512×512 improves generation quality when starting from the 320-epoch ImageNet-256 checkpoint.The study evaluates 40- and 530-epoch fine-tuning durations.
  • Alternative strategies: Factorized AdaLN did not match full pixel-wise AdaLN performance despite reducing modulation parameters from O(P^2 · C) to O(K · C).The low-frequency basis limits the spatial resolution of the modulation signal.

G.3. Training Stability and Optimization

PixelDiT’s optimization behavior depends on representation alignment and carefully chosen architectural components. Several alternatives reduce cost or stabilize training but do not improve the reported baseline quality.

  • Training stability: Removing the DINOv2-based representation alignment loss makes training unstable and eventually divergent.The alignment loss encourages patch-level representations to remain aligned with frozen DINOv2 features.
  • Prediction target: X-prediction mitigates loss spikes but does not outperform direct velocity prediction and requires additional tuning.Its velocity-space loss uses a stability threshold ε=0.05.
  • Pixel representation: A compact pixel representation, such as dimension 16, is sufficient when paired with adequate cross-patch attention dimension.Smaller dimensions perform poorly, while larger ones increase memory without proportional quality gains.
  • Modulation design: Sharing AdaLN modulation does not substantially degrade shallow models but becomes limiting for deeper architectures.Different layers in deeper networks may benefit from distinct modulation patterns.
  • Positional embeddings: Fixed sinusoidal and fully learnable positional embeddings yield comparable generation quality, so fixed embeddings are used in the final model.The choice favors simplicity and resolution generalizability.

I. Limitations

PixelDiT’s current limitations concern complex text-to-image content and training stability. The authors identify model capacity, data quality, and loss spikes as remaining constraints.

  • The 1.3B-parameter text-to-image model sometimes struggles with geometrically and texturally complex objects, including human hands and intricate architectural scenes.
  • Velocity prediction can produce loss spikes, especially in deeper architectures and during long training runs.
  • Future work targets larger models, higher-quality training data, and improved understanding and mitigation of pixel-space diffusion instabilities.
Loading 2511.20645v2…