PaperlayerView exact revision on arXiv

Source-linked AI summary

Back to Basics: Let Denoising Generative Models Denoise

Tianhong Li, Kaiming He

arXiv:2511.13720v2cs.CV

TL;DR

Diffusion models usually predict noise or a noised quantity, although clean data and these targets occupy fundamentally different geometric spaces. The paper therefore uses direct clean-image prediction in simple large-patch pixel Transformers, showing competitive ImageNet results without tokenizers, pre-training, or extra losses. JiT remains effective where ϵ- and v-prediction can fail catastrophically, while the paper leaves further gains from extra losses or pre-training for future work.

  • Problem

    Diffusion models commonly predict noise or noised quantities even though natural data lies on a low-dimensional manifold and these targets are off-manifold, creating a high-dimensional prediction challenge.

  • Method

    JiT directly predicts clean images with a plain Vision Transformer operating on large raw-pixel patches, without a tokenizer, pre-training, or auxiliary loss.

  • Results

    JiT produces strong ImageNet results at 256 and 512 resolutions with patch sizes 16 and 32, while ϵ- and v-prediction fail catastrophically.

  • Takeaways & Limitations

    Direct clean-data prediction enables apparently under-capacity Transformers to operate effectively in very high-dimensional spaces and supports a self-contained diffusion-plus-Transformer design.

  • Takeaways & Limitations

    The approach does not use extra losses or pre-training, and the paper leaves possible gains from these additions for future work.

Abstract

from arXiv · show

Today's denoising diffusion models do not "denoise" in the classical sense, i.e., they do not directly predict clean images. Rather, the neural networks predict noise or a noised quantity. In this paper, we suggest that predicting clean data and predicting noised quantities are fundamentally different. According to the manifold assumption, natural data should lie on a low-dimensional manifold, whereas noised quantities do not. With this assumption, we advocate for models that directly predict clean data, which allows apparently under-capacity networks to operate effectively in very high-dimensional spaces. We show that simple, large-patch Transformers on pixels can be strong generative models: using no tokenizer, no pre-training, and no extra loss. Our approach is conceptually nothing more than "Just image Transformers", or JiT, as we call it. We report competitive results using JiT with large patch sizes of 16 and 32 on ImageNet at resolutions of 256 and 512, where predicting high-dimensional noised quantities can fail catastrophically. With our networks mapping back to the basics of the manifold, our research goes back to basics and pursues a self-contained paradigm for Transformer-based diffusion on raw natural data.

1. Introduction

Diffusion models commonly predict noise or a noised quantity rather than clean images, but the manifold assumption makes these targets fundamentally different. The paper proposes direct clean-image prediction and shows that simple large-patch pixel Transformers can work effectively without auxiliary components.

  • Motivation: Directly predicting clean images is fundamentally different from predicting noise or noised quantities because clean data lies on a low-dimensional manifold whereas noisy targets occupy the high-dimensional space.This distinction motivates revisiting what the network should predict, rather than treating prediction targets as interchangeable through loss reweighting.
  • Motivation: High-dimensional noise prediction requires preserving all noise information, whereas limited-capacity networks can predict clean data by retaining low-dimensional structure and filtering noise.
  • Motivation: Latent spaces alleviate the difficulty of noise prediction but hide the problem and make diffusion less self-contained, while pixel-space models remain vulnerable to the curse of dimensionality.
  • Approach: JiT applies a plain Vision Transformer to large raw-pixel patches while directly predicting clean images, using no tokenizer, pre-training, or auxiliary loss.The approach is presented as a self-contained “Diffusion + Transformer” paradigm on native data.
  • Results: On ImageNet at 256 and 512 resolutions, JiT with patch sizes 16 and 32 produces strong results with x-prediction, while ϵ- and v-prediction fail catastrophically.The analysis also finds that network width need not match patch dimensionality, and a bottleneck can even help.
  • Implications: The self-contained design is intended to support broader use of diffusion Transformers on natural domains where tokenizers are difficult to design.

2. Related Work

Related work traces diffusion’s evolution from denoising toward ϵ- and v-prediction, while connecting these choices to denoising, manifold learning, and pixel-space architectures. The paper positions JiT against high-dimensional diffusion methods that rely on architectural specialization or pre-training.

  • Diffusion Models and Their Predictions: Diffusion models evolved from denoising formulations to ϵ-prediction and then v-prediction, with flow matching interpretable as a form of v-prediction.v combines data and noise, and diffusion and flow-based models are now often treated within one framework.
  • Denoising Models: Denoising methods and denoising autoencoders use sparsity, low dimensionality, or the manifold assumption, linking classical representation learning to modern score-based diffusion.
  • Manifold Learning: Manifold learning commonly uses bottlenecks to pass only useful information, while latent diffusion can be viewed as applying manifold learning in an initial representation stage.
  • Pixel-space Diffusion: Pixel-space diffusion has traditionally used dense convolutions, over-complete channels, and long skip connections, whereas ViT-based pixel diffusion faces high-dimensional token spaces.Standard 16×16 patches can produce token dimensions comparable to or larger than the Transformer hidden dimension.
  • High-dimensional Diffusion: Prior approaches address high-dimensional diffusion with hierarchical architectures, large hidden sizes, or self-supervised pre-training, while this work achieves it using just Transformers without pre-training.
  • High-dimensional Diffusion: ViT-style diffusion models have been reported to degrade rapidly and catastrophically as per-token dimensionality increases, in both pixel and latent spaces.
  • x-prediction: Although x-prediction is an old formulation traceable to DDPM [23], this work revisits it as an overlooked issue specifically for high-dimensional data with low-dimensional manifolds.

3. On Prediction Outputs of Diffusion Models

The paper separates diffusion models by their direct prediction output and loss space, showing that these choices are mathematically distinct even when outputs can be transformed for sampling. Under the manifold assumption, experiments indicate that directly predicting clean data is more robust in high-dimensional settings.

  • Diffusion and Flows: A noisy sample is formed as z_t = t x + (1 −t) ϵ, and the flow velocity is the time derivative of z.Sampling transforms the learned output to v-space and solves an ODE from noise to data, using a 50-step Heun solver by default.
  • Prediction and Loss Spaces: Reparameterizing between prediction spaces changes the effective loss weighting, so prediction space and loss space are separate design choices.For example, v-loss with x-prediction becomes a reweighted x-loss by a factor of 1/(1−t)^2.
  • Prediction and Loss Spaces: The network can directly predict x, ϵ, or v, while the loss can be defined in any of these spaces, yielding nine valid but mathematically non-equivalent combinations.When prediction and loss spaces differ, the network output is transformed before computing the loss.
  • Toy Experiment: The result follows the manifold assumption: clean data lies on a low-dimensional manifold, whereas noise and velocity are off-manifold, making x-prediction more tractable for limited-capacity networks.The under-complete MLP can discard information in the ambient space while retaining the lower-dimensional structure needed for the clean output.
  • Toy Experiment: In the toy setting, only x-prediction remains reasonable as observed dimension increases; ϵ- and v-prediction fail catastrophically at D=512 with a 256-dimensional MLP.The experiment embeds d=2 data into D-dimensional space using an unknown random column-orthogonal projection and evaluates x-, ϵ-, and v-prediction under v-loss.

4. “Just Image Transformers” for Diffusion

JiT applies a plain ViT directly to raw pixel patches and uses x-prediction, enabling diffusion models to handle high-dimensional patches without matching network width to patch dimension. On ImageNet, x-prediction remains effective where ε- and v-prediction fail, while bottlenecks and proportional noise scaling can improve or preserve performance.

  • 4. “Just Image Transformers” for Diffusion: JiT is a plain ViT that embeds non-overlapping raw pixel patches, processes them with Transformer blocks, and projects tokens back to pixel patches for x-prediction.JiT/16 uses 768-dimensional patches on 256×256 images, while JiT/32 uses 3072-dimensional patches on 512×512 images.
  • 4.2. What to Predict by the Network?: Only x-prediction performs well for JiT-B/16 on 256×256 ImageNet, whereas ε- and v-prediction fail catastrophically across loss spaces.The comparison uses 768-dimensional patches and FID-50K; at 64×64 with 48-dimensional patches, all prediction-loss combinations perform reasonably well.
  • 4.2. What to Predict by the Network?: Adjusting noise levels helps an already effective x-prediction model but cannot fix ε- or v-prediction, whose failure stems from inability to propagate high-dimensional information.The experiments shift the logit-normal sampler parameter µ; the authors use µ = –0.8 in other ImageNet 256×256 experiments.
  • 4.2. What to Predict by the Network?: x-prediction works with under-complete networks because the clean-data target can be implicitly low-dimensional, unlike the high-dimensional noised quantities.The authors report that increasing hidden units is not necessary, including for patches substantially larger than the hidden size at higher resolutions.
  • 4.2. What to Predict by the Network?: A low-rank bottleneck patch embedding can improve JiT quality, with bottleneck dimensions from 32 to 512 improving FID by up to approximately 1.3.The embedding uses two sequential linear layers, reducing a 768-dimensional raw patch to an intermediate dimension before expanding to the Transformer hidden size; even dimension 16 avoids catastrophic failure.
  • 4.4. “Just Advanced” Transformers: Adding general-purpose Transformer components improves the reported JiT-B/16 FID from 7.48 to 5.49, while removing SwiGLU and RMSNorm degrades it to 7.89.The cited configurations add RoPE and qk-norm, then in-context class tokens; the strongest listed result is 5.49 for JiT-B/16.

5. Comparisons

JiT remains effective at high resolutions and scales across model sizes while avoiding the compute and dimensionality problems highlighted for pixel-based diffusion. Its minimalist design also leaves possible gains from extra losses unexplored.

  • High-resolution generation on pixels: JiT performs decently at resolutions up to 1024 while avoiding the curse of observed dimensionalities.Patch dimensions reach 3072 or 12288, yet comparable models retain similar parameter counts and computational cost across resolutions.
  • Scalability: Larger JiT models improve with scale, and JiT-G achieves lower FID at 512 resolution than at 256.The resolution gap narrows as models grow, although large-model FID is sensitive to overfitting.
  • Qualitative results: JiT-H/32 produces selected qualitative ImageNet samples at 512×512 resolution.The figure caption identifies the displayed examples but does not establish a quantitative comparison.
  • Reference results: JiT uses plain Transformers for pixel-based generation and avoids quadratic compute scaling when resolution doubles.The reference comparisons mark external pre-training components and report JiT as compute-friendly relative to other pixel-based methods.
  • Scope: Because JiT omits extra losses and pre-training, those components remain possible sources of further gains for future work.The paper explicitly leaves these directions open rather than evaluating them in the main approach.

6. Discussion and Conclusion

The discussion frames x-prediction as a capacity-aligned return to modeling natural data rather than noise. JiT’s minimalist Transformer design is intended to support self-contained diffusion beyond image pixels.

  • Discussion: Neural networks can use limited capacity more effectively on natural data than on noise, supporting the observed advantage of x-prediction.The authors interpret the result as consistent with the distinction between natural data and noise.
  • Conclusion: JiT reduces domain-specific inductive biases to encourage generalization to raw high-dimensional data in domains where tokenizers are difficult to obtain.The authors identify scientific applications as a particularly desirable target.
  • Reference results: Reference evaluations on ImageNet 256×256 and 512×512 report FID and IS for 50K generated samples, while tracking external pre-training components.The parameter counts include the generator and tokenizer decoder but exclude other pre-trained components.
  • Conclusion: The authors envision a general-purpose Diffusion + Transformer paradigm as a potential foundation for other scientific domains.This proposed extension follows the paper’s self-contained treatment of raw natural data.

A. Implementation Details

The implementation uses standard Transformer-based components with resolution-scaled patches, noise schedules, conditioning, regularization, and evaluation choices tailored to stable ImageNet experiments.

  • Configurations: The experiments follow public DiT and SiT codebases, with configurations summarized in Table 9.The implementation section describes the remaining training and evaluation choices separately.
  • Time distribution: Training samples logit-normal noise times with default µ = –0.8 and σ = 0.8 across the reported ImageNet resolutions.The mean controls the noise level.
  • Resolution scaling: JiT/32 at 512×512 preserves the 256-patch sequence length of JiT/16 at 256×256 while increasing each patch from 768 to 3072 dimensions.All other computations and costs are reported as exactly the same between these settings.
  • Resolution scaling: Noise is scaled by 2× at 512 resolution and 4× at 1024 to approximately preserve signal-to-noise ratios without other configuration changes.The 512 setup uses JiT/32, while the 1024 setup uses JiT/64.
  • Regularization: Dropout is applied to JiT-H and G models, and early stopping is used for G models when monitored FID begins degrading around 320 epochs.Dropout covers both attention and MLP blocks in the middle half of Transformer blocks.
  • EMA and CFG: The study searches CFG scales from 1.0 to 4.0 and EMA decays across three candidates, selecting settings with 8K samples before evaluating 50K samples.Multiple EMA copies are maintained during training with negligible computational overhead.
  • Evaluation: ImageNet evaluation computes FID and IS on the same 50K generated images, with 50 samples per each of 1000 classes.FID is measured against the ImageNet training set.

B.1. Training Loss and Denoised Images

The appendix attributes the failure of noise- and velocity-prediction to high-dimensional output demands, while showing that x-prediction lowers loss and avoids artifacts. Pre-conditioners remain insufficient, whereas an auxiliary classification loss offers a limited exploratory improvement.

  • Training Loss and Denoised Images: The failure of ϵ- and v-prediction is attributed to limited-capacity networks predicting high-dimensional outputs.This explanation is supported by the training-loss curves.
  • Training Loss and Denoised Images: v-prediction has about 25% higher loss than x-prediction under the same v-loss, while ϵ-prediction has about 3× higher and unstable loss.The comparison is legitimate because both parameterizations are evaluated in the same loss space.
  • Training Loss and Denoised Images: Single-step artifacts from v-prediction can accumulate through the multi-step ODE solver, producing catastrophic generation failure.The denoised v-prediction images visibly contain artifacts corresponding to the higher training loss.
  • Pre-conditioner: Pre-conditioned variants fail catastrophically, although they outperform ϵ- and v-prediction when they become more similar to x-prediction as t→0.The comparison uses ImageNet 256 JiT-B/16 settings from Table 10.
  • Classification Loss: Adding a classification head and cross-entropy loss produces a decent improvement, but the paper excludes this loss from its other experiments.The exploration uses a 1000-class ImageNet classifier attached after a Transformer block.

B.4. Cross-resolution Generation

Cross-resolution generation tests whether JiT models can transfer across image sizes through simple upsampling or downsampling. Downsampling the 512-resolution model remains competitive at 256, whereas upsampling the 256-resolution model substantially degrades 512-resolution quality.

  • Cross-resolution Generation: The comparison uses JiT/16 at 256 and JiT/32 at 512 with comparable parameters and compute, making cross-resolution performance directly comparable.Table 12 evaluates the models after upsampling or downsampling their generated images.
  • Cross-resolution Generation: FID@256 is 1.84 after downsampling the 512 model, close to the 256-resolution expert’s 1.82 at similar computational cost.The 512 model additionally retains the ability to generate at 512 resolution.
  • Cross-resolution Generation: FID@512 rises to 2.45 when upsampling the 256 model, versus 1.78 for the 512-resolution expert.The degradation is attributed to loss of higher-frequency details during upsampling.

B.5. Additional Metrics

The paper additionally reports precision and recall on ImageNet 256×256. These metrics are presented alongside comparisons with DiT, SiT, and RAE baselines.

  • Additional Metrics: Precision and recall are reported on ImageNet 256×256 as additional evaluation metrics.The results appear in Table 13.
  • Additional Metrics: Table 13 compares the ImageNet 256×256 precision and recall results with DiT and SiT baselines.
  • Additional Metrics: The additional-metrics evaluation also includes the latest RAE as a comparison baseline.

C. Qualitative Results

The paper provides additional uncurated qualitative examples from ImageNet 256×256. These examples are collected in Figures 8–11.

  • Qualitative Results: Figures 8–11 provide additional uncurated examples generated on ImageNet 256×256.
  • Qualitative Results: The qualitative examples are presented as supplementary visual evidence for ImageNet 256×256 generation.
  • Qualitative Results: The examples are explicitly described as uncurated rather than selected demonstrations.
Loading 2511.13720v2…