PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion

Yifan Lu; Qi Wu; Jay Zhangjie Wu; Zian Wang; Huan Ling; Sanja Fidler; Xuanchi Ren

Source-linked AI summary

PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion

Yifan Lu, Qi Wu, Jay Zhangjie Wu, Zian Wang, Huan Ling, Sanja Fidler, Xuanchi Ren

arXiv:2605.23902v1May 22, 2026cs.CV

TL;DR

Reconstruction-oriented latent decoders struggle to synthesize missing high-resolution detail efficiently. PiD instead uses conditional pixel diffusion to unify decoding and upsampling, achieving competitive visual quality at substantially lower sampling cost, including 7.1 seconds per sample—1.87× faster than PixelDiT and 14.3× faster than FLUX.2.

Problem
Reconstruction-oriented latent decoders can lose fine details, pass through artifacts, and under-specify low-level appearance in semantically rich latents.
Method
PiD reformulates latent decoding as latent-conditioned pixel diffusion, unifying high-resolution decoding and spatial upsampling in one generative process.
Results
7.1 s per sample makes FLUX.2(512²) + PiD 1.87× faster than PixelDiT and 14.3× faster than FLUX.2, with competitive or better reported visual quality.
Takeaways & Limitations
PiD provides a practical alternative to cascaded or native high-resolution generation by combining latent structure with pixel-space detail synthesis.
Takeaways & Limitations
For small-text reconstruction, the student improves perceptual similarity but does not maximize pixel-wise alignment, with teacher variants achieving higher PSNR and SSIM.

Abstract

from arXiv · show

Most practical high-resolution text-to-image systems, including latent diffusion and autoregressive models, perform generation in a compact latent space, and a decoder maps the generated latents back to pixels. Yet the latent-to-pixel decoder is reconstruction-oriented, optimized to invert the encoder rather than synthesize more details, and becomes increasingly costly at megapixel scale. This drawback calls for a more expressive and efficient decoding paradigm. Motivated by recent progress in scalable pixel-space diffusion, we introduce PiD, a Pixel diffusion Decoder that reformulates latent decoding as conditional pixel diffusion, unifying decoding and upsampling into one generative module. By denoising directly in high-resolution pixel space, PiD synthesizes $4\times$ and even $8\times$ upscaled images with low latency. For latent conditioning, a lightweight sigma-aware adapter injects noise-corrupted latents into the pixel diffusion backbone, enabling PiD to decode partially denoised latents and terminate the latent diffusion process early. To further improve efficiency, we distill the model using DMD2, reducing inference to just 4 steps. PiD applies to both conventional VAE latents and semantic latents (e.g., SigLIP, DINOv2) used in recent RAE-based models. PiD decodes latents of $512 \times 512$ images into $2048 \times 2048$ pixels in under 1 second with 13 GB peak memory on a consumer RTX 5090, and as fast as 210 ms on a GB200 GPU, about $6\times$ faster than cascaded diffusion-based super-resolution pipelines with better visual fidelity.

1. Introduction

PiD reformulates latent decoding as conditional pixel diffusion, combining decoding and high-resolution upsampling in one generative module. It uses sigma-aware noisy-latent conditioning and four-step distillation to enable fast, high-resolution decoding across VAE and semantic latents.

Motivation: Latent diffusion and autoregressive image generators rely on decoders to map compact latents or visual tokens back to pixels.Latent-space generation is computationally efficient, but decoding remains the pathway to image pixels.
Limitations: Conventional VAE decoders lose fine-grained details and may pass through or amplify artifacts in generated latents.They are optimized to recover stored latent information rather than add missing visual detail.
PiD: PiD reformulates latent decoding as conditional pixel diffusion, unifying visual decoding and super-resolution in one generative module.The latent provides structural conditioning, while pixel-space diffusion supplies high-frequency appearance details.
Generality: 4× or 8× higher resolution: PiD directly denoises in target pixel space and extends from VAE latents to semantic representations such as DINOv2 features.This makes it applicable to RAE-style diffusion models.
Efficiency: 4 denoising steps: DMD2 distillation accelerates PiD, while a lightweight sigma-aware adapter injects noise-corrupted latents and supports partially denoised inputs.This conditioning enables early termination of the base latent diffusion process.
Results: 2048×2048 images: PiD decodes them from 512×512 latents in under 1 second using 13 GB peak memory on an RTX 5090, or in 210 ms on a GB200 GPU.The method is reported to provide superior speed and visual quality compared with the cascaded baseline.

2. Related Work

Prior work uses latent representations, diffusion decoders, cascaded super-resolution, and direct pixel-space generation to improve image synthesis efficiency, fidelity, and resolution. PiD builds on these directions by unifying latent decoding and spatial upsampling through conditional pixel diffusion.

Latent representations for image generation: Latent diffusion reduces diffusion-network training and inference cost by generating images in an autoencoder’s latent space, with VAEs, VQ-VAEs, and newer designs improving compression and reconstruction.These systems remain centered on reconstruction-oriented latent representations.
Diffusion decoders: Diffusion decoders replace conventional reconstruction decoders with iterative pixel-space denoising, while SSDD distills diffusion decoding into a fast single-step model.DALL-E 3 also reports using a diffusion decoder on top of an LDM latent space.
Diffusion decoders: Existing diffusion decoders remain primarily reconstruction-oriented, focusing on same-resolution decoding, relatively low-resolution evaluation, and separate super-resolution stages.PiD instead incorporates a pixel diffusion generative prior and combines decoding with upsampling as one conditional generation task.
High-resolution image synthesis: Cascaded super-resolution decomposes high-resolution synthesis into a base model and one or more upsampling stages, using adversarial, perceptual, or diffusion objectives.Diffusion-based cascades leverage large generative priors for higher-fidelity texture synthesis.
High-resolution image synthesis: Direct high-resolution methods improve scalable synthesis through training, architecture, compression, and pixel-space diffusion transformers that generate fine-grained detail from raw pixels.PiD contrasts with these approaches by unifying latent decoding and spatial upsampling within a single latent-conditioned process.

3. Pixel Diffusion Decoder

PiD reformulates latent decoding as conditional pixel diffusion, unifying decoding and upsampling in one target-resolution generative process. A pixel diffusion prior, latent adapter, noisy-latent conditioning, and four-step distillation enable efficient high-resolution decoding and early termination of base latent diffusion.

Pixel Diffusion Decoder: PiD directly generates target-resolution pixels from sampled latents, replacing separate decoding and super-resolution stages with one conditional pixel-diffusion process.The latent supplies global structure and semantic hints, while pixel diffusion synthesizes fine detail at 4× or 8× upsampling.
Pixel-space generative prior: The decoder builds on a text-conditioned PixelDiT pixel diffusion prior and converts it into a latent-conditioned decoder with a lightweight ControlNet-style adapter.At each denoising step, the model uses noisy target-resolution images, text, and latent conditions to predict pixel-space rectified-flow velocity.
Noisy latent conditioning: Noise-corrupted latent conditioning teaches PiD to reduce reliance on unreliable latents and decode partially denoised latents for early-exit inference.The latent noise level is provided to injection modules so conditioning strength can adapt to latent reliability.
Latent projection and injection: Latent features are spatially aligned to the 16 × 16 image-patch grid, projected into transformer hidden dimensions, and injected every two backbone blocks.Sigma-aware gating applies per-token, per-channel modulation, with weaker injection encouraged as latent noise increases.
Few-step distillation: Four sampling steps are achieved by DMD2 distillation, which also incorporates classifier-free guidance into the student and retains noisy latent conditioning.The base latent diffusion model can stop early and pass its partially denoised latent directly to PiD for pixel-space decoding.

4. Experiments

Experiments show that PiD improves perceptual quality across diverse latent spaces and generators while enabling efficient, high-resolution decoding. Ablations and reconstruction studies support its pixel prior, sigma-aware conditioning, partial-denoising compatibility, and distilled four-step inference.

Quantitative results: PiD achieves the best scores on most metrics across six latent settings, including lower NIQE on VAE latents and improved MUSIQ, DEQA, and Unipercept-IAA on SigLIP.On SigLIP, MUSIQ rises from 73.68 to 74.03, DEQA from 4.00 to 4.17, and Unipercept-IAA from 59.95 to 64.9.
Decoding Step Analysis: The distilled four-step student surpasses multi-step teacher variants on IQA metrics, while favoring perceptual similarity over pixel-wise alignment for small-text reconstruction.The teacher is evaluated at 50, 25, 12, 8, and 4 steps; the student achieves the lowest LPIPS but lower PSNR/SSIM than multi-step teachers for small text.
Reconstruction and termination: PiD preserves sharper details in real-image latent reconstruction, can recover corrupted small text, and generates additional content or sharper details when decoding partially denoised latents.With all 28 LDM steps, PiD remains closer to the original latent; at intermediate termination steps, it synthesizes more detail.
Super-resolution comparison: PiD yields richer details with lower latency than diffusion-based super-resolution baselines when decoding 512^2 FLUX.1 latents.The comparison uses PiD (24/28) against cascaded super-resolution methods.
Ablation study: Removing the text-to-image prior raises NIQE from 5.43 to 7.79 and lowers VisualQuality-R1 from 4.649 to 2.587, while removing sigma-aware gating worsens perceptual quality and fidelity.The ablation confirms that high-resolution pretraining and explicit control of decoder reliance on noisy latents are important design choices.
LDM + PiD vs. native 2K generation: 7.1 s per sample makes FLUX.2(512^2) + PiD 1.87× faster than PixelDiT and 14.3× faster than FLUX.2, with stronger prompt following and sharper details.PiD uses four steps on a 512^2 FLUX.2 latent, whereas FLUX.2 and PixelDiT use 50 sampling steps.

5. Conclusion

PiD replaces reconstruction-oriented latent decoding with conditional pixel diffusion, unifying decoding and super-resolution in one generative module. It supports VAE and RAE-style latents, including partially denoised latents for earlier base-diffusion termination while preserving high-fidelity outputs.

Conclusion: PiD replaces reconstruction-oriented latent decoding with conditional pixel diffusion.This reframes decoding as a generative pixel-space process.
Conclusion: PiD unifies decoding and super-resolution in a single generative module for high-resolution image synthesis.The formulation improves both speed and quality from VAE and RAE-style latent spaces.
Conclusion: PiD decodes partially denoised latents, enabling earlier termination of base latent diffusion while preserving high-fidelity outputs.This capability applies during latent diffusion rather than requiring fully denoised latents first.

A. MLLM Judgment Details

The evaluation uses closed-source MLLM pairwise judgments of perceptual quality and detail, with explicit criteria, mandatory verdicts, and safeguards against position bias. Method win rates are computed from valid judgments, while consistency measures stability under swapped input order.

Judgment criteria: Each paired sample asks a closed-source MLLM to choose the image with better perceptual quality and detail.The comparison covers sharpness, fine texture, noise, compression artifacts, ringing, blocking, over-smoothing, over-sharpening, and edge halos.
Judgment criteria: The judge anchors decisions on fine textures, edges, smooth regions, and repetitive patterns.These regions are assessed for texture fidelity, clean transitions, artifact-free flat areas, and the absence of moire or aliasing.
Bias control: Each image pair is evaluated twice with swapped input order to reduce position bias, and raw verdicts are converted into method-level preferences.The first round shows our result as A and the baseline as B; the second reverses those positions.
Output and parsing: Each judgment must select A or B without ties, and invalid or unparseable responses are excluded from the final win-rate calculation.The output parser accepts only VERDICT: A and VERDICT: B, with the verdict required on the first line.
Metrics: For each baseline and judge model, our win rate divides selections of our result by all valid non-failed judgments, while consistency counts matching preferences across both valid rounds.Each pair can contribute up to two valid votes; higher consistency indicates less sensitivity to input order.