Source-linked AI summary
L2P: Unlocking Latent Potential for Pixel Generation
Zhennan Chen, Junwei Zhu, Xu Chen, Jiangning Zhang, Jiawei Chen, Zhuoqi Zeng, Wei Zhang, Chengjie Wang, Jian Yang, Ying Tai
TL;DR
Training pixel-space diffusion models from scratch is prohibitively expensive and can lag behind latent models in semantic quality. L2P transfers pretrained LDM priors into pixel space using synthetic data and limited training, achieving near-source benchmark performance while enabling native 4K generation.
Problem
Training state-of-the-art pixel-space text-to-image models requires prohibitive computation and data, while their semantic and compositional quality can trail pretrained LDMs.
Method
L2P replaces the VAE with large-patch pixel tokenization, freezes core LDM layers, and trains shallow components using synthetic images from the source LDM.
Results
L2P achieves 86.00 on DPG-Bench versus 84.86 for its source LDM and retains approximately 93.6% of the source model’s GenEval performance.
Takeaways & Limitations
L2P lowers the resource barrier for developing pixel-space diffusion models while transferring pretrained latent priors and supporting native 4K generation.
Takeaways & Limitations
Because L2P relies on synthetic images from the source LDM, its semantic and compositional capabilities are fundamentally bounded by the source model’s priors.
Abstract
from arXiv · showhide
Pixel diffusion models have recently regained attention for visual generation. However, training advanced pixel-space models from scratch demands prohibitive computational and data resources. To address this, we propose the Latent-to-Pixel (L2P) transfer paradigm, an efficient framework that directly harnesses the rich knowledge of pre-trained LDMs to build powerful pixel-space models. Specifically, L2P discards the VAE in favor of large-patch tokenization and freezes the source LDM's intermediate layers, exclusively training shallow layers to learn the latent-to-pixel transformation. By utilizing LDM-generated synthetic images as the sole training corpus, L2P fits an already smooth data manifold, enabling rapid convergence with zero real-data collection. This strategy allows L2P to seamlessly migrate massive latent priors to the pixel space using only 8 GPUs. Furthermore, eliminating the VAE memory bottleneck unlocks native 4K ultra-high resolution generation. Extensive experiments across mainstream LDM architectures show that L2P incurs negligible training overhead, yet performs on par with the source LDM on DPG-Bench and reaches 93% performance on GenEval.
1 INTRODUCTION
L2P addresses the prohibitive cost and VAE memory bottleneck of pixel-space diffusion by transferring pre-trained LDM knowledge into an efficient pixel-space model. It uses large-patch tokenization, frozen source architecture components, synthetic data, and only 8 GPUs while enabling native 4K generation.
- Motivation: Pixel-space diffusion avoids VAE memory constraints that hinder scaling to ultra-high resolutions, but training state-of-the-art models from scratch requires hundreds of high-end GPUs and billions of curated image-text pairs.These costs contribute to gaps in semantic comprehension and compositional quality relative to established LDMs.
- Method: L2P discards the VAE, applies large-patch tokenization to pixel inputs, and uses a lightweight U-Net while leaving the DiT architecture unmodified for latent-to-pixel transfer.The framework is designed to bridge the representation gap between latent and pixel spaces at low cost.
- Efficiency and resolution: 8 GPUs enable L2P to harness pre-trained LDM priors for pixel-space diffusion while unlocking native 4K ultra-high-resolution generation.The approach leverages the smooth manifold of pre-trained LDMs to bypass costly from-scratch training.
- Training data: Zero real-data cost is achieved by generating synthetic training pairs from a comprehensive, multi-dimensional prompt dataset.The synthetic corpus supports highly efficient training.
- Results: L2P robustly inherits source-LDM generative priors, maintaining near-lossless semantic alignment while delivering exceptional visual fidelity at native 4K resolution.These validations are reported across standard benchmarks and native ultra-high-resolution generation.
2 RELATED WORK
Related work spans latent diffusion models (LDMs), which reduce the computational cost of image generation through latent compression but face VAE decoding constraints, and pixel diffusion models, which remain bottlenecked by quadratic complexity at high resolution.
- Text-to-Image Generation: LDMs dominate text-to-image generation by compressing images into a compact latent space, avoiding the exorbitant costs of early pixel-space models.The related work identifies latent compression as the main efficiency advantage of LDMs over early pixel-space approaches.
- Text-to-Image Generation: The VAE’s severe quadratic memory footprint during spatial decoding makes native 4K generation practically intractable for standard LDMs.This limitation imposes rigid hardware constraints on ultra-high-resolution generation.
- Pixel Diffusion Models: Early pixel diffusion models such as DDPM and ADM are severely constrained on high-resolution images by a quadratic complexity bottleneck.Later approaches including JiT and PixelGen introduce novel prediction targets, while PixNerd, DeCo, PixelDiT, and DiP pursue more efficient modeling strategies.
3 METHOD
L2P transfers pretrained LDM priors into pixel space by training on source-LDM-generated synthetic images, replacing the VAE with large-patch tokenization and selectively tuning shallow components. The method preserves source-model optimization while enabling efficient 4K generation through adaptive patchification and noise scheduling.
- Synthetic Dataset Pipeline: L2P constructs a large-scale, scene-diverse synthetic corpus from the source LDM, fitting its smooth data manifold to accelerate convergence without real-world data collection.The pipeline uses hierarchical categories, LLM-generated prompts, automated filtering, and source latent T2I image synthesis.
- Architectural Adaptation: L2P discards the VAE and uses 16×16 input patches to preserve sequence length and computational efficiency comparable to the original VAE-compressed latent space.The pixel-space architecture applies patchification directly to input images.
- Architectural Adaptation: A lightweight U-Net Detailer Head replaces the final projection layer, decoding DiT representations into dense pixel semantics and restoring high-frequency details.This bypasses the VAE decoder bottleneck for high-fidelity pixel-level generation.
- Selective Optimization: L2P freezes most intermediate DiT blocks and updates only the input projection, first and last n blocks, and Detailer Head, reducing training overhead while preserving semantic priors.The optimization objective remains consistent with the source LDM, mitigating catastrophic forgetting and supporting diverse LDM frameworks.
- Ultra-High-Resolution Generation: 97.67% lower single-step inference latency and 38.81% lower peak GPU memory enable efficient native 4K generation versus the source latent baseline.For 4K inputs, L2P expands the patch size to 64×64 and increases the noise shift to ensure sufficient corruption and robust global generation.
4 EXPERIMENTS
Experiments show that L2P transfers latent priors effectively with minimal overhead, achieving strong benchmark performance, native 4K synthesis, and robust zero-shot generalization. Ablations further attribute its efficiency to source-model data, shallow tuning, and limited synthetic data.
- Quantitative Experiment: 86.00 on DPG-Bench validates L2P’s efficient migration of latent priors to pixel space despite discarding the VAE and requiring minimal training overhead.The evaluation uses DPG-Bench and GenEval to assess semantic alignment and generation quality.
- Qualitative Experiment: L2P renders unseen Korean text after transfer on only 20K English/Chinese samples, demonstrating zero-shot generalization without catastrophic forgetting.Qualitative comparisons report stronger text alignment than open-source pixel baselines, which often fail on complex attribute binding and text rendering.
- 4K Quantitative Experiment: L2P achieves the lowest FID and FIDpatch and the highest Inception Score among evaluated 4K solutions, indicating strong fidelity, local coherence, and diversity.The 4K evaluation measures global quality, local details, and generation diversity using FID, FID-patch, and IS.
- 4K Qualitative Experiment: L2P mitigates over-smoothing and artifacts while natively generating 4K images, unlike the source LDM, whose upsampled 1K outputs blur high-frequency details.The source LDM also fails to generate semantic content directly at 4K, whereas L2P raises the resolution ceiling with minimal training overhead.
- Ablation Studies: Shallow tuning preserves source-model priors, while full-layer tuning stagnates and degrades quality; source-generated data converges faster and performs better than real-data training.The default shallow-tuning configuration trains 5 layers, and source data is compared with a random 20k subset of UltraHR-100K as the real-data baseline.
- Ablation Studies: Performance gains are substantial from 10k to 20k synthetic samples but saturate beyond 20k, demonstrating L2P’s data efficiency and rapid adaptation.The scaling ablation compares 10k, 20k, and 100k synthetic samples.
5 CONCLUSION
L2P is a Latent-to-Pixel transfer paradigm that overcomes VAE-induced LDM limitations and avoids the prohibitive cost of training pixel-space models from scratch. It migrates deep semantic priors into pixel space through large-patch tokenization, frozen intermediate layers, and a smooth synthetic data manifold using only 8 GPUs.
- L2P overcomes VAE-induced limitations in latent diffusion models while bypassing the prohibitive costs of training pixel-space models from scratch.
- L2P discards the VAE in favor of large-patch tokenization and freezes core intermediate layers to migrate deep semantic priors into pixel space.
- 8 GPUs are sufficient for L2P to construct a multi-dimensional prompt dataset that fits a smooth synthetic data manifold.
A MORE IMPLEMENTATION DETAILS
L2P preserves the source LDM’s DiT structure while replacing the VAE with large-patch tokenization and modifying only the input stage for pixel-space processing. Transfer freezes intermediate layers and trains shallow layers with AdamW under specified optimization settings.
- DiT Architecture: L2P preserves the source LDM’s DiT structural configuration, modifying only input channels for RGB images and replacing the VAE with large-patch tokenization.Frozen intermediate transformer blocks process pixel-space inputs while retaining the source model’s latent priors.
- Detailer Head Architecture: The Detailer Head uses a symmetric encoder-decoder for spatial downsampling and upsampling, with its bottleneck dimension adapted to the DiT backbone.This adaptation enables concatenation and fusion of features from frozen intermediate layers.
- Optimization: 5 × 10^-5 learning rate and 0.01 weight decay are used with AdamW while freezing massive intermediate layers and training only shallow layers.Optimization uses batch size 8 with no gradient accumulation and steps set to 1.
B MORE DATA CONSTRUCTION DETAILS
The data construction pipeline uses explicit system prompts for generation and automated filtering, emphasizing diverse, well-structured, high-quality scene descriptions and safety compliance.
- Prompting and filtering: Figures 10 and 11 document the system prompts for General Prompt Generation and Automated Prompt Filtering.These prompts provide transparency into the data construction pipeline.
- Prompting and filtering: The generation template enforces prompt diversity, structural formatting, and text-rendering syntax.These constraints are intended to elicit complex, high-quality scene descriptions from the LLM.
- Prompting and filtering: The filtering mechanism automatically discards prompts that fail to meet the specified safety requirements.
C MORE EXPERIMENTAL RESULTS · C.1 QUANTITATIVE RESULTS
The 4K noise-shift ablation shows that increasing the parameter improves FID through shift 4, while shift 5 slightly degrades performance from over-corruption. This confirms that biasing the noise schedule toward higher noise levels is essential for fully corrupting dense 4K image signals.
- C.1 QUANTITATIVE RESULTS: FID drops significantly as the noise-shift parameter increases from 1 to 4 for 4K generation.The ablation identifies increasing noise shift as beneficial up to parameter 4.
- C.1 QUANTITATIVE RESULTS: Shift parameter=5 slightly degrades performance because the image signals become over-corrupted.Performance peaks before the parameter reaches 5.
- C.1 QUANTITATIVE RESULTS: Skewing the noise schedule toward higher noise levels is essential to fully corrupt dense 4K image signals.The ablation supports this noise-schedule design for 4K generation.
C.2 VISUALIZATION RESULTS
L2P produces visually faithful text-to-image generations at 1K resolution and enables native 4K generation by eliminating the VAE memory bottleneck. Its pure pixel-space formulation also supports zero-shot extrapolation to 8K while preserving global structure and fine details.
- Visual fidelity and scalability: L2P demonstrates diverse text-to-image generations at 1K resolution, validating transfer of the source LDM’s generative priors.Extended qualitative results assess visual fidelity and scalability.
- Native ultra-high resolution: L2P unlocks native 4K ultra-high-resolution generation by eliminating the VAE memory bottleneck.The pixel-space model renders detailed images at extreme resolutions without prohibitive computational overhead.
- Zero-shot resolution extrapolation: L2P enables zero-shot resolution extrapolation beyond its training resolution, generating 8K images with consistent global structure and faithful micro-details.This capability benefits from the pure pixel-space formulation and elimination of VAE-induced coupling.
D LIMITATIONS AND FUTURE WORK
L2P’s reliance on synthetic images from the source LDM makes its semantic and compositional capabilities fundamentally bounded by the source model’s priors. Real-world data could theoretically address this bottleneck, but would restore dependence on data curation and quality.
- Source-model dependence: Synthetic training images fundamentally upper-bound L2P’s semantic and compositional capabilities by the source LDM’s priors.The limitation follows directly from relying on images generated by the source LDM.
- Future work: Real-world datasets could theoretically circumvent this knowledge bottleneck.
- Future work: Using real-world datasets would reintroduce a strong dependency on data curation and quality.