Source-linked AI summary
PixelFlow: Pixel-Space Generative Models with Flow
Shoufa Chen, Chongjian Ge, Shilong Zhang, Peize Sun, Ping Luo
TL;DR
PixelFlow asks whether image generation can operate directly in raw pixels without the usual latent-space VAE or fragmented pixel-space stages. It introduces an end-to-end cascade flow model that progressively denoises across resolutions, achieving competitive class-conditional and text-to-image performance. The model reaches 1.98 FID on 256 × 256 ImageNet, while its final full-resolution attention remains a major inference bottleneck.
Problem
High-resolution pixel-space generation is computationally unaffordable, while prior cascaded approaches require separate networks for different stages.
Method
PixelFlow uses unified parameters and Flow Matching to progressively denoise raw-pixel samples from low to high resolution without separate VAEs or upsamplers.
Results
PixelFlow achieves competitive performance on class-conditional and text-to-image benchmarks, including 1.98 FID on 256 × 256 ImageNet class-conditional generation.
Takeaways & Limitations
PixelFlow demonstrates that direct raw-pixel generation can provide simplicity and end-to-end trainability while retaining competitive image-generation capabilities.
Takeaways & Limitations
The final stage requires full-resolution attention that accounts for roughly 80% of total inference time, and convergence slows as sequence length increases.
Abstract
from arXiv · showhide
We present PixelFlow, a family of image generation models that operate directly in the raw pixel space, in contrast to the predominant latent-space models. This approach simplifies the image generation process by eliminating the need for a pre-trained Variational Autoencoder (VAE) and enabling the whole model end-to-end trainable. Through efficient cascade flow modeling, PixelFlow achieves affordable computation cost in pixel space. It achieves an FID of 1.98 on 256$\times$256 ImageNet class-conditional image generation benchmark. The qualitative text-to-image results demonstrate that PixelFlow excels in image quality, artistry, and semantic control. We hope this new paradigm will inspire and open up new opportunities for next-generation visual generation models. Code and models are available at https://github.com/ShoufaChen/PixelFlow.
1. Introduction
PixelFlow addresses the computational cost and fragmented training of pixel-space generation with an end-to-end cascade flow framework. It progressively denoises from low to high resolution and achieves competitive results across class-conditional and text-to-image benchmarks.
- Pixel-space diffusion becomes computationally unaffordable at high resolution because it must process substantial per-pixel correlations.
- Prior cascaded pixel-space methods reduce this cost by generating low-resolution images before upsampling, but require separate networks for different stages.
- PixelFlow eliminates separate VAEs and upsamplers by using a unified, end-to-end model for raw-pixel generation across cascading resolutions.
- During inference, PixelFlow starts from Gaussian noise at low resolution and progressively denoises and upscales until reaching the target resolution.
- 1.98 FID is achieved on the 256 × 256 ImageNet class-conditional generation benchmark.
- PixelFlow achieves 0.64 on GenEval and 77.93 on DPG-Bench for text-to-image generation, with qualitative results indicating strong visual fidelity and text-image alignment.
2. Related Work
Related work contrasts latent-space efficiency with pixel-space fidelity and complexity. PixelFlow follows the pixel-space direction while using progressive generation to address high-resolution inefficiency.
- VAEs map visual data into lower-dimensional latent spaces for efficient training and inference, but can compromise high-frequency details and introduce low-level artifacts.
- Early pixel-space diffusion models generated images in a single stage but proved challenging and inefficient for high-resolution generation.
- Cascaded pixel-space models subsequently generated low-resolution images and used super-resolution models to reach higher resolutions.
- Recent pixel-space approaches explore streamlined architectures, noise schedules, and recursive generative modules for high-resolution synthesis.
3. PixelFlow
PixelFlow performs end-to-end image generation directly in pixel space using a unified flow-matching model across progressively increasing resolutions. It combines multistage denoising, raw-pixel tokenization, and resolution-aware conditioning for efficient generation across image and text-to-image tasks.
- Preliminary: Flow Matching: Flow Matching transforms samples along linear paths from a prior distribution toward the target data distribution by predicting the intermediate velocity.The prior is typically standard normal, and the model guides transformation from the intermediate sample to the real data sample.
- Multi-Scale Generation in Pixel Space: Each resolution stage starts by upscaling the noisy output from the preceding stage, enabling progressively more refined samples.The generation procedure is partitioned into successive resolution stages.
- Multi-Scale Generation in Pixel Space: PixelFlow progressively increases image resolution through S cascaded stages, using lower-resolution samples early and reaching target resolution at the final stage.Each stage operates over a defined time interval, and the single-stage case reduces to pixel-space flow matching.
- Multi-Scale Generation in Pixel Space: PixelFlow constructs training states by resizing images to successive scales, adding Gaussian noise, interpolating adjacent noisy scales, and training velocity prediction end-to-end.Training examples are uniformly sampled across all resolution stages.
- Model Architecture: The model uses a Transformer-based DiT architecture that directly tokenizes raw pixels and packs sequences from varying resolutions for joint attention.PixelFlow replaces latent tokenization with raw-pixel patch embedding and uses sequence packing across resolutions.
- Model Architecture: PixelFlow distinguishes resolutions with a resolution embedding and supports text conditioning through cross-attention after each self-attention layer.The resolution signal is added to the timestep embedding, while cross-attention aligns visual features with text at every generation stage.
- Multi-Scale Generation in Pixel Space: During inference, PixelFlow starts from Gaussian noise at the lowest resolution and progressively denoises and upscales toward the target resolution.Sampling can use either the Euler discrete sampler or the Dopri5 solver, depending on the speed–accuracy trade-off.
4. Experiments
PixelFlow is evaluated on class-conditional and text-to-image generation, with experiments covering model design, inference settings, and benchmark comparisons. The results show that cascade choices can preserve quality while reducing computation, and PixelFlow performs competitively across image-generation benchmarks.
- Experimental Setup: PixelFlow is evaluated on ImageNet-1K class-conditional generation at 256×256 and on text-to-image benchmarks up to 1024×1024.The evaluation reports FID, IS, sFID, and Precision/Recall for class-conditional generation, plus T2I-CompBench, GenEval, and DPG-Bench for text-to-image generation.
- Model Design: 8×8 kickoff sequences match or slightly improve FID relative to the 32×32 baseline, whereas 2×2 degrades performance.The 8×8 setting is adopted as the default because it balances quality and computational efficiency.
- Model Design: Larger 8×8 patches improve computational efficiency but cause a noticeable quality drop at 256×256, even after 1600K training iterations.At 64×64, 4×4 patches perform similarly to 2×2 patches and eliminate the highest-resolution stage required by the smaller patch.
- Inference Schedule: 30 inference steps per resolution stage produce the best overall performance, after which gains saturate or slightly decline.FID, sFID, and IS improve consistently as the number of steps increases up to this point.
- Benchmark Results: 1.98 FID on ImageNet 256×256 places PixelFlow ahead of LDM, DiT, and SiT, while text-to-image results include 0.64 on GenEval and 77.93 on DPG-Bench.The reported comparisons describe competitive or superior performance against latent-space and recent pixel-based models, alongside strong visual fidelity and text-image alignment.
5. Conclusion
PixelFlow directly operates in raw pixel space, using a unified model across resolution stages to support simplicity and end-to-end trainability. It demonstrates competitive image generation capabilities while retaining important efficiency limitations at full resolution.
- PixelFlow directly operates in raw pixel space and transforms between resolution stages with a unified parameter set.This design avoids separate VAEs or upsamplers and supports end-to-end training.
- PixelFlow demonstrates competitive capabilities on both class-conditional image generation and text-to-image benchmarks.
- Roughly 80% of total inference time is spent on full-resolution attention in the final stage.Training convergence also slows as sequence length increases, creating efficiency and scalability challenges.