Source-linked AI summary
PixNerd: Pixel Neural Field Diffusion
Shuai Wang, Ziteng Gao, Chenhui Zhu, Weilin Huang, Limin Wang
TL;DR
PixNerd addresses accumulated errors and decoding artifacts from VAE-based two-stage diffusion, as well as the complexity of pixel-space alternatives. It replaces linear patch decoding with implicit neural fields in a single-stage, end-to-end pixel-space diffusion model, achieving strong ImageNet and text-to-image benchmark results.
Problem
VAE-based two-stage training introduces accumulated errors and decoding artifacts, while pixel-space diffusion alternatives require complicated pipelines and increased token complexity.
Method
PixNerd predicts patch-specific neural-field MLP weights from diffusion-transformer hidden states and uses coordinate encodings with noisy pixel values to predict diffusion velocity.
Results
PixNerd achieves 2.15 FID on ImageNet 256 × 256, 2.84 FID on ImageNet 512 × 512, and text-to-image scores of 0.73 on GenEval and 80.9 on DPG.
Takeaways & Limitations
PixNerd provides a single-scale, single-stage, end-to-end pixel-space alternative that does not rely on a VAE or complex cascade pipeline.
Abstract
from arXiv · showhide
The current success of diffusion transformers heavily depends on the compressed latent space shaped by the pre-trained variational autoencoder(VAE). However, this two-stage training paradigm inevitably introduces accumulated errors and decoding artifacts. To address the aforementioned problems, researchers return to pixel space at the cost of complicated cascade pipelines and increased token complexity. In contrast to their efforts, we propose to model the patch-wise decoding with neural field and present a single-scale, single-stage, efficient, end-to-end solution, coined as pixel neural field diffusion~(PixelNerd). Thanks to the efficient neural field representation in PixNerd, we directly achieved 2.15 FID on ImageNet $256\times256$ and 2.84 FID on ImageNet $512\times512$ without any complex cascade pipeline or VAE. We also extend our PixNerd framework to text-to-image applications. Our PixNerd-XXL/16 achieved a competitive 0.73 overall score on the GenEval benchmark and 80.9 overall score on the DPG benchmark.
1 Introduction
PixNerd addresses the artifacts and complexity of VAE-based or pixel-space diffusion by using neural fields for patch-wise decoding in a single-stage pixel-space model. It reports strong ImageNet and text-to-image results without a complex cascade pipeline.
- Motivation: VAEs introduce optimization challenges, accumulated errors, and decoding artifacts through adversarially trained two-stage pipelines.Joint training can address these limitations but incurs substantial computational costs.
- Motivation: Pixel-space diffusion avoids VAE decoding but requires substantially more image tokens and impractical computational resources at comparable patch sizes.Large patch sizes also make simple linear projection inadequate for capturing fine details.
- Method: PixNerd replaces linear patch decoding with implicit neural fields whose MLP parameters are predicted from diffusion-transformer hidden states.Coordinate encodings and noisy pixel values are processed by patch-specific neural-field MLPs to predict diffusion velocity.
- Results: 2.84 FID is maintained by PixNerd-XL/16 on ImageNet 512 × 512.The reported result uses similar computation demands to its latent counterpart.
- Results: 0.73 overall GenEval and 80.9 average DPG scores are reported for PixNerd-XXL/16 text-to-image generation.The framework is extended beyond class-conditional ImageNet generation.
2 Related Work
Related work contrasts latent diffusion's computational advantages and VAE complications with pixel-space approaches that remain harder to scale. Existing pixel-space methods use long residuals or multiscale decompositions to reduce computational burden.
- Latent Diffusion Models: VAEs reduce latent spatial dimensions and ease learning and computation, but their adversarial and perceptual supervision complicates the diffusion pipeline.Insufficient VAE training can produce decoding artifacts.
- Pixel Diffusion Models: Pixel diffusion models face greater learning difficulty and computational demands because raw pixel space is substantially larger than latent space.Prior approaches reduce burden by splitting diffusion across resolution scales, while some retain long residuals.
3 Method
PixNerd replaces the diffusion transformer's pixel-patch linear projection with patch-wise implicit neural fields, while retaining the transformer backbone. Its design choices balance generation quality, convergence, and computational efficiency across class- and text-conditioned experiments.
- Patch-wise Neural Field: PixNerd predicts each patch neural field's MLP weights from the diffusion transformer's final hidden states, replacing linear projection for pixel-space decoding.Local pixel coordinates are transformed into coordinate encodings and combined with the noisy pixel values before neural-field prediction.
- Patch-wise Neural Field: The neural field decodes pixel-wise velocities by feeding coordinate encodings and noisy pixel values into an MLP for each patch.The formulation applies row-wise normalization to neural-field parameters to improve performance and stabilize training.
- Neural Field Design: The default neural-field normalization strategy achieves optimal performance and convergence speed by normalizing weights and output features.The evaluated alternatives normalize only the first weight or both neural-field weights.
- Neural Field Design: A 2-layer neural-field MLP is selected as the default because deeper networks improve performance but increase inference latency and training cost.The study compares 1, 2, and 4 MLP layers and selects the 2-layer configuration as the efficiency-performance compromise.
- Inference Scheduler Design: A CFG value of 3.5 is used by default after sweeping values from 3.0 to 5.0; the best FID10k results occur at 3.4 or 3.6.The selected scheduler uses interval guidance over the interval [0.1, 1].
5 Discussion
PixNerd is positioned as a single-stage pixel-space generative model that avoids VAEs, contrasting with prior neural-field and latent-diffusion approaches.
- PixNerd differs from INFD and DDMI by applying neural fields to diffusion models rather than using them to enhance VAEs.
- DenoisedWeights trains independent neural weights per image before learning a generative model over the collected weights.
- PixNerd avoids VAE-dependent two-stage training and its associated cascaded errors, adversarial-loss instability, and artifact risks.
- The discussion presents PixNerd as a simple single-stage pixel-space model, while describing pixel generative modeling as having future potential.
6 Conclusion
PixNerd returns diffusion modeling to pixel space through a neural field in a single-scale, single-stage, end-to-end framework, with reported image-generation and text-to-image results.
- PixNerd is introduced as a single-scale, single-stage, efficient, end-to-end pixel-space diffusion model with a neural field.
- 2.15 FID on ImageNet 256 × 256 and 2.84 FID on ImageNet 512 × 512 are reported without a complex cascade pipeline.
- 0.73 overall on GenEval and 80.9 average on DPG are reported for PixNerd-XXL/16 in text-to-image generation.
- Current PixNerd shows unclear details in some cases and still has gaps with its latent counterparts.