PaperlayerView exact revision on arXiv

Source-linked AI summary

DiP: Taming Diffusion Models in Pixel Space

Zhennan Chen, Junwei Zhu, Xu Chen, Jiangning Zhang, Xiaobin Hu, Hanzhen Zhao, Chengjie Wang, Jian Yang, Ying Tai

arXiv:2511.18822v3cs.CV

TL;DR

Diffusion models must balance generation quality with computational efficiency, but LDMs can lose information through VAE compression and pixel-space models scale poorly at high resolution. DiP uses a large-patch DiT for global structure and a co-trained Patch Detailer Head for local detail, achieving strong ImageNet results with efficient pixel-space generation. Its main reported result is FID 1.79 without a pre-trained VAE, alongside lower inference latency and training costs.

  • Problem

    Diffusion models face a trade-off between generation quality and computational efficiency, while LDMs risk information loss and pixel-space models become computationally impractical at high resolution.

  • Method

    DiP combines a large-patch DiT backbone for global structure with a co-trained lightweight Patch Detailer Head for high-frequency local details in pixel space.

  • Results

    DiP achieves FID 1.79 on ImageNet after 600 epochs without a pre-trained VAE, surpassing DiT-XL, SiT-XL, and PixelFlow-XL/4.

  • Takeaways & Limitations

    DiP provides an end-to-end VAE-free pixel-space framework with competitive efficiency, lower inference latency, and lower training costs on ImageNet.

  • Takeaways & Limitations

    LDMs remain subject to potential information loss during VAE compression and a non-end-to-end training pipeline.

Abstract

from arXiv · show

Diffusion models face a fundamental trade-off between generation quality and computational efficiency. Latent Diffusion Models (LDMs) offer an efficient solution but suffer from potential information loss and non-end-to-end training. In contrast, existing pixel space models bypass VAEs but are computationally prohibitive for high-resolution synthesis. To resolve this dilemma, we propose DiP, an efficient pixel space diffusion framework. DiP decouples generation into a global and a local stage: a Diffusion Transformer (DiT) backbone operates on large patches for efficient global structure construction, while a co-trained lightweight Patch Detailer Head leverages contextual features to restore fine-grained local details. This synergistic design achieves computational efficiency comparable to LDMs without relying on a VAE. DiP is accomplished with up to 10$\times$ faster inference speeds than previous method while increasing the total number of parameters by only 0.3%, and achieves an 1.79 FID score on ImageNet 256$\times$256.

1. Introduction

Diffusion models trade generation quality against computational efficiency, while LDMs introduce VAE-related information loss and pixel-space models become impractical at high resolution. DiP addresses this trade-off with large-patch global modeling and a lightweight local detail module in an end-to-end pixel-space framework.

  • 1. Introduction: LDMs reduce denoising cost through VAE compression but may lose information and require a non-end-to-end training pipeline.The VAE-based latent-space approach is presented as an efficiency solution with these limitations.
  • 1. Introduction: Existing pixel-space Transformers use small patches that make sequence length grow quadratically with image resolution, rendering high-resolution training and inference computationally intractable.This scalability problem creates a barrier to practical application.
  • 1. Introduction: DiP combines a DiT backbone on large patches with a co-trained Patch Detailer Head to construct global structure and restore high-frequency details.The DiT reduces sequence length, while the lightweight head adds local detail synthesis without serving as post-processing.
  • 1. Introduction: DiP is designed to retain pixel-space, end-to-end training while using large patches to achieve computational efficiency comparable to LDMs.The large-patch DiT captures global layout and semantic content, while local refinement addresses the resulting loss of fine detail.
  • 1. Introduction: DiP achieves state-of-the-art ImageNet performance with the lowest inference latency and low training costs.The contribution summary identifies these as the framework’s benchmark outcomes.

2. Related Work

Latent diffusion models improve scalability by performing diffusion in a VAE-compressed latent space, but the paper seeks a more principled pixel-space alternative. DiP separates global structure construction from local detail refinement to provide an efficient pixel-space solution.

  • 2. Related Work: LDMs reduce memory and computation by performing diffusion in a VAE-learned compressed latent space, with DiT architectures further improving scalability.The efficiency comes from latent-space processing rather than direct pixel-space diffusion.
  • 2. Related Work: DiP separates global structure construction from local detail refinement to provide a more principled and efficient pixel-space diffusion framework.The framework uses a global Transformer and a dedicated local module rather than relying solely on a compressed latent representation.

3. Methods

DiP combines a global DiT backbone with a lightweight Patch Detailer Head to preserve efficient large-patch processing while restoring local pixel detail. The framework addresses DiT’s weak within-patch modeling through local inductive bias and supports strong ImageNet performance.

  • Motivation: DiT-only reconstruction captures global layout and color but misses fine textures and sharp edges because it lacks local inductive bias.The single-image overfitting experiment produces blurry pixel-space reconstructions, motivating a dedicated local-detail module.
  • Framework: DiP uses a DiT backbone for global structure and a lightweight Patch Detailer Head for patch-level high-frequency detail refinement.The backbone models long-range dependencies, while the head processes each patch using global context and noisy pixels.
  • Global Structure Construction: 16×16 patches reduce the pixel-space model’s sequence length, keeping its computational footprint comparable to latent-space DiT models.The resulting global features are produced from non-overlapping patches before local refinement.
  • Patch Detailer Head: The Patch Detailer Head introduces local inductive bias with a low parameter count, improving reconstruction and accelerating convergence.Its design targets the within-patch spatial information that flattened DiT tokens compress.
  • Experimental Evaluation: DiP achieves the best FID score and the lowest latency among compared pixel diffusion models on ImageNet 256×256.Table 1 reports comparisons using the Euler solver and classifier-free guidance.
  • Placement Strategies: All three Patch Detailer Head placement strategies provide comparable gains over DiT-only, while post-hoc refinement simplifies implementation and may reuse pretrained DiT checkpoints.Post-hoc refinement leaves the standard DiT backbone unchanged by placing the head at the end.

4. Experiments

Experiments show that DiP combines efficient large-patch global modeling with local detail refinement to achieve strong ImageNet quality at reduced computational cost. Ablations indicate that suitable local inductive bias and large patches are central to this trade-off.

  • Performance: 1.79 FID on ImageNet 256×256 surpasses DiT-XL, SiT-XL, PixelFlow-XL/4, ADM, and VDM++ without a pre-trained VAE.With 160 epochs, DiP still reaches 2.16 FID and outperforms DiT-XL trained for much longer.
  • Computational Cost: DiP reaches best performance with 320 epochs, over 4× more training-efficient than DiT-XL and SiT-XL, while using 631M parameters.The parameter count is substantially below VDM++ and Farmer.
  • Patch Detailer Head vs. Scaling Up DiT: Increasing DiT depth from 26 to 32 layers improves FID only from 5.28 to 4.91, whereas widening to hidden dimension 1536 reaches 2.83 at substantially higher cost.Widening increases parameters by 74.9%, training cost by 77.4%, and inference latency by 69.3%.
  • Patch Detailer Head: The Convolutional U-Net Patch Detailer Head provides the best quality–cost balance, reflecting the value of convolutional local inductive bias.Convolutions preserve local texture and edge continuity for patch-level refinement.
  • Performance on ImageNet 512×512: At 512×512, DiP achieves the best FID and maintains a performance margin over a DiT-only baseline using smaller patches.The result supports efficient high-resolution pixel-space synthesis.
  • Impact of Patch Size and Head Configuration: Small patches improve performance but increase computation, while DiP uses large patches to shorten sequences and retain LDM-comparable efficiency.Increasing Patch Detailer Head depth helps until gains diminish, whereas blindly increasing width does not sustain improvement.

5. Conclusion

DiP addresses the quality–efficiency trade-off in pixel diffusion through synergistic global-local modeling. Its large-patch DiT backbone and lightweight Patch Detailer Head support efficient, VAE-free high-quality generation.

  • 5. Conclusion: DiP combines a large-patch DiT backbone for global structure with a co-trained Patch Detailer Head for high-frequency details, avoiding a VAE.The framework is trained end to end in pixel space.
  • 5. Conclusion: ImageNet experiments report superior FID scores with lower inference latency and training costs.The paper identifies future applications in text-to-image and text-to-video generation.

A. Why Patch Detailer Head: A Theoretical Perspective

The analysis models DiT as effective at learning low-frequency global signals but insufficient for high-frequency details, motivating refinement that injects all-frequency raw information. Under stated distributional and effective-information assumptions, Theorem A.6 characterizes the resulting adaptive correction.

  • DiT primarily captures low-frequency global signals, whereas local refinement is introduced to recover high-frequency image details.The analysis attributes this gap to DiT’s limited effectiveness for local details and high-frequency signals.
  • Natural-image patches remain correlated through smooth boundaries and DiT attention, so patch estimates can use information beyond the target patch.The analysis contrasts smooth neighboring regions with abrupt transitions and notes that attention provides coarse information from other patches.
  • The refinement structure injects all signals from the initial data, expanding DiP’s effective information beyond DiT’s local patch and low-frequency cross-patch signals.This is the formal mechanism proposed for improving high-frequency estimation.
  • Under the Gaussian data-distribution, eigenvalue-decay, and DiT effective-information assumptions, Theorem A.6 compares near-optimal DiT and DiP estimates.The assumptions separate low- and high-frequency components and formalize DiT’s access to local and cross-patch information.
  • Theorem A.6’s interpretation assigns DiT strong adaptive fitting of low-frequency components but insufficient representational capacity for high-frequency components.The denoising analysis separates low-frequency fitting from the remaining high-frequency terms and correction effects.

B. Proof of Theorem A.6

The proof derives the conditional statistics and covariance structure needed to compare DiT and DiP’s patch-level vector-field estimates. It then uses the assumed eigenvalue decay to control cross-frequency terms and complete the theorem.

  • The proof uses Cov(x_t) = (1 −t)^2Σ + t^2I as the covariance of the noised input.This covariance follows from x_t = (1 −t)x_0 + tϵ.
  • The conditional-expectation identity converts covariance statistics into the required predictor estimates for the noised patch.The proof applies E[Y |X] = EY + Cov(Y, X)Cov(X, X)^−1(X −EX).
  • The proof separately derives the second predictor term using A = tI_d −(1 −t)Σ and the assumed DiT effective-information structure.The derivation then constructs corresponding conditional quantities for the compared estimates.
  • Fast eigenvalue decay makes cross-frequency covariance corrections o(1), enabling the first-order Taylor expansion used to finish the comparison.The proof bounds λ_p/λ_q for low- versus high-frequency indices before applying the expansion.

C. More Implementation Details

DiP is trained with flow matching and representation alignment, evaluated with a common 100-step Euler–Maruyama ODE sampler, and studies patch-level detail injection against image-level alternatives. The toy experiment favors patch-level input for preserving fine structure.

  • DiP uses flow matching with representation-alignment regularization, while adversarial or perceptual losses are identified as possible future improvements.The stated objective follows DDT’s training objectives.
  • The default sampler is Euler–Maruyama ODE with 100 sampling steps, using the same inference hyperparameters for DiT-only and DiP.This provides a shared sampling configuration for the two models.
  • The input-format study asks whether patch-level or image-level signals inject high-frequency information more effectively into the refinement structure.The motivation is that DiT already captures long-distance dependencies, leaving refinement focused on specific high-frequency signals.
  • The toy experiment shows patch-level input tightly follows intricate branching patterns and sharp boundaries, whereas image-level processing over-smooths fine details.The comparison is presented in Figure 12(a) and Figure 12(b).

E. Alternative Patch Detailer Head (PDH).

The alternative PDH variants underperform because they lack the spatial modeling advantages of the convolutional U-Net design. The comparison is documented in the alternative-PDH figure.

  • The convolutional U-Net’s inductive biases and hierarchical architecture better capture spatial detail and preserve local continuity than the alternative PDH variants.The alternatives either lack spatial information or model local patterns less effectively.

F. More Visualization Results

The visualization appendix presents ImageNet class-conditional samples at 256×256 and 512×512 resolutions, all generated with CFG = 4.0.

  • All displayed samples use classifier-free guidance with CFG = 4.0.
  • The remaining 256×256 examples depict teddy bear, cauliflower, potpie, and bolete classes.
Loading 2511.18822v3…