Source-linked AI summary
Advanced Pixel Diffusion Model with Guided Sparse Global Refinement
Weiyi You, Jinhua Zhang, Xingyu Zhou, Wei Long, Junyu Lou, Shuhang Gu
TL;DR
Pixel-space diffusion is computationally challenging because natural images occupy an extremely high-dimensional space, while existing efficient approaches compromise fine details or restrict refinement within patches. PixSGR combines a supervised low-channel bottleneck with coarse-guided sparse global refinement, achieving a state-of-the-art FID of 1.51 on ImageNet at 256×256.
Problem
Pixel-space diffusion must model extremely high-dimensional image distributions, while large-patch tokenization and intra-patch refinement limit fine-detail preservation, cross-patch continuity, and long-range interactions.
Method
PixSGR uses a low-channel bottleneck to capture compact global structure, then applies coarse-to-fine refinement with coarse attention maps preselecting globally relevant fine-scale interactions.
Results
1.51 FID: PixSGR achieves state-of-the-art generation quality among compared pixel-space models on ImageNet at 256×256 after 320 training epochs.
Takeaways & Limitations
PixSGR indicates that architectural design can support effective and efficient pixel-space diffusion through compact global modeling and sparse global refinement.
Abstract
from arXiv · showhide
Pixel-space diffusion has recently emerged as a promising direction for high-fidelity image generation by modeling images directly in the original pixel domain. However, pixel-space diffusion is computationally demanding due to the extremely high dimensionality of natural images. For efficiency, existing pixel diffusion models either compromise fine details with large-patch tokenization or confine subsequent refinement within individual patches. Such intra-patch refinement inevitably restricts structural continuity across patch boundaries and long-range token interactions, limiting refinement quality. To address these issues, we propose PixSGR, a novel Pixel diffusion framework with Sparse Global Refinement tailored for modeling the distribution of natural images directly in pixel space. PixSGR starts from a supervised low-channel bottleneck to efficiently capture the low-dimensional manifold of natural images. It then progressively expands the channel dimensionality and spatial resolution to recover increasingly fine-grained structures. At the spatial refinement stage, coarse-scale attention maps preselect globally relevant interactions to pre-sparsify fine-scale attention, enabling non-local refinement beyond isolated patches without the quadratic cost of dense attention. Extensive experiments on ImageNet validate the effectiveness of PixSGR. It achieves an FID of 1.51 at 256$\times$256 and maintains performance when scaled to 512$\times$512, attaining an FID of 1.60.
1 Introduction
Pixel-space diffusion avoids latent reconstruction loss but remains difficult because natural images are extremely high-dimensional. PixSGR addresses this with a supervised bottleneck and guided sparse global refinement that recovers details while preserving non-local interactions.
- Pixel-space diffusion avoids information loss from VAE reconstruction but faces the computational burden of extremely high-dimensional image modeling.
- Large-patch tokenization improves global structure modeling but weakens fine-grained detail preservation and patch-isolated refinement limits boundary continuity and long-range interactions.
- PixSGR combines a low-channel bottleneck with fine-scale refinement to model the natural-image manifold and recover increasingly detailed spatial structure.
- Coarse-scale attention maps preselect globally relevant fine-scale tokens, enabling sparse non-local refinement beyond isolated patches without dense O(N^2) attention.
- PixSGR uses auxiliary bottleneck supervision to strengthen compact representations before refinement and validates its framework and components through ImageNet experiments.
2 Related Work
Diffusion research commonly uses VAE-compressed latent spaces for efficiency, but compression can discard high-frequency detail and constrain reconstruction fidelity. Pixel-space approaches revisit direct image modeling, while PixSGR analyzes bottleneck representations and attention concentration to motivate its design.
- Latent Diffusion Models reduce computation by denoising in VAE-compressed latent spaces, but reconstruction quality remains bounded by autoencoder capacity.
- PixSGR’s bottleneck visualization compares supervised and unsupervised features, while attention visualizations examine concentration across Transformer blocks and timesteps.
- Aggressive latent compression can discard high-frequency details and fine structures, motivating renewed interest in pixel-space diffusion.
3 Methodology
PixSGR uses a coarse-to-fine pixel-space architecture that begins with a supervised low-channel bottleneck, expands channel capacity, and refines spatial details with sparse global attention. Cross-scale guidance selects globally relevant fine-scale tokens, while auxiliary bottleneck prediction and convolutional upsampling support final reconstruction.
- Reconstruction: The convolutional upsampling head maps the refined feature to the final clean-image prediction and adds local spatial aggregation after global refinement.The method uses two x-prediction outputs: a bottleneck prediction and a final prediction.
- Architecture: PixSGR extends pixel-space diffusion with a low-channel bottleneck, channel expansion, and fine-scale spatial refinement to recover local details.The design targets compact representation learning before higher-resolution refinement.
- Bottleneck supervision: The bottleneck pathway is supervised through clean-image prediction to encourage structured compact representations for subsequent refinement.The auxiliary loss aligns the bottleneck pathway with the clean-image prediction objective.
- Sparse global refinement: Sparse attention initializes each block from preceding attention maps, using coarse-scale patterns to select globally relevant fine-scale tokens under a limited budget.At the highest-resolution stage, coarse attention is mapped to fine tokens because the token counts differ.
- Sparse global refinement: Coarse-to-all preserves all fine children of selected coarse tokens, whereas coarse-to-one keeps one relative-position representative, trading fine-scale preservation against long-range coverage.Hybrid mapping applies coarse-to-all to the most relevant coarse tokens and representative mapping to the remainder.
4 Experiments
Experiments evaluate PixSGR’s design choices and component contributions under controlled ImageNet settings, showing gains from supervised bottlenecks and sparse global refinement.
- Experimental setup: PixSGR ablations use controlled training budgets, with variants trained for 50 epochs and evaluated under specified sampling conditions.The analyses cover bottleneck supervision, internal guidance, sparse attention, top-k selection, and convolutional upsampling.
- Bottleneck supervision: λbot = 0.5 achieves the best bottleneck-supervision performance, while larger weights may over-constrain bottleneck prediction and weaken subsequent refinement.The auxiliary loss should remain a constraint for coarse representation learning rather than dominate denoising.
- Sparse attention: Top-k sparse attention outperforms Swin and neighborhood attention under comparable GFLOPs, supporting adaptive non-local selection over fixed local windows.The comparison concerns high-resolution refinement while keeping computation affordable.
- Sparse attention: The hybrid top-k mapping performs best by applying coarse-to-all mapping to the top 75% of selected coarse tokens and coarse-to-one mapping to the remaining 25%.This preserves fine tokens in the most relevant regions while extending long-range coverage elsewhere.
- Component ablation: 16.9% fewer GFLOPs and 12.5% fewer parameters result from replacing JiT’s first two blocks with low-channel bottleneck blocks, while FID improves from 78.9 to 77.9.Adding Lbot further reduces FID by 3.2 points to 74.7.
- Component ablation: 15.3 points lower FID than JiT is achieved by the complete design, with sparse attention, upsampling, and internal guidance contributing successive improvements.Coarse-guided sparse attention lowers FID to 70.3, upsampling to 68.4, and internal guidance to 63.6.
5 Conclusion
PixSGR addresses the efficiency challenge of pixel-space diffusion by combining compact global modeling with sparse global refinement. On ImageNet, it achieves a state-of-the-art FID of 1.51 at 256×256 after 320 training epochs.
- Pixel-space diffusion avoids latent-compression loss, but efficiently modeling high-dimensional image distributions remains challenging.
- PixSGR combines a supervised low-channel bottleneck for global structural representations with coarse-guided sparse refinement across patch boundaries.Coarse-scale attention patterns pre-sparsify fine-scale attention for efficient cross-patch interactions.
- 1.51 FID is achieved by PixSGR on ImageNet at 256×256 after 320 training epochs among the compared pixel-space models.
A Implementation Details
The implementation follows the public JiT codebase and summarizes PixSGR’s backbone architecture and sampling configurations.
- PixSGR’s implementation closely follows the public JiT codebase.
- Table 4 summarizes the backbone architecture and sampling configurations used in the experiments.
B.1 ImageNet 512 × 512 Results
PixSGR scales to ImageNet 512 × 512 class-conditional generation while achieving strong guided image quality and semantic diversity against listed baselines.
- 1.60 gFID: PixSGR-H/32 outperforms all listed latent- and pixel-space baselines on ImageNet 512 × 512.It reaches this result after 300 training epochs with classifier-free guidance.
- 0.09 lower gFID: PixSGR improves over FrequencyBooster-H/16 while using fewer parameters and a shorter training schedule.PixSGR uses 879M parameters versus 1.1B and trains for 300 rather than 400 epochs.
- 317.9 IS: PixSGR exceeds JiT-H/16 and DeCo-XL/16, although FrequencyBooster-H/16 achieves a higher IS of 332.6.
B.2 ImageNet 256 × 256 Results without CFG
Without classifier-free guidance, PixSGR-H/16 improves ImageNet 256 × 256 generation metrics under the same 320-epoch configuration used for the guided comparison.
- Without classifier-free guidance, the evaluation provides a more direct view of the distribution learned by each model.
- 3.79 gFID and 199.5 IS: FrequencyBooster-H/16 is the strongest reproduced baseline in the unguided comparison.
- 2.22 gFID and 237.4 IS: PixSGR-H/16 achieves these results without classifier-free guidance.The evaluation uses the same 320-epoch configuration as the guided comparison.
B.3 Training Objective Analysis
PixSGR uses a cumulative training objective that combines robust regression, perceptual supervision, and representation alignment.
- The final PixSGR objective combines pseudo-Huber, LPIPS, and REPA losses.LPIPS uses λlpips = 0.1 and REPA uses λrepa = 0.05; the terms are introduced progressively.
C Visual Results
PixSGR produces class-conditional samples at 256 × 256 and 512 × 512 with recognizable class structure, varied appearances, and preserved high-resolution detail.
- Visual Results: Figures 6–9 provide additional class-conditional samples, with each block corresponding to a labeled class or containing 16 samples from that class.
- Visual Results: At both resolutions, samples exhibit recognizable class-specific structures across animals, natural scenes, vehicles, and other objects.The visualizations include varied poses, viewpoints, backgrounds, and appearances.
- Visual Results: At 512 × 512, generated samples preserve fine-grained textures and object details.