Source-linked AI summary
Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation
Alan Baade, Eric Ryan Chan, Kyle Sargent, Changan Chen, Justin Johnson, Ehsan Adeli, Li Fei-Fei
TL;DR
Latent diffusion achieves strong image generation but sacrifices end-to-end modeling through information-losing tokenization and a separate decoder. Latent Forcing jointly processes deterministic latents and pixels with ordered noise schedules, using latents as an intermediate scratchpad. On ImageNet, it reports state-of-the-art pixel diffusion transformer results at its compute scale while retaining lossless reconstruction.
Problem
Latent diffusion improves generation quality but loses information during encoding, requires a separately trained decoder, and models an auxiliary latent distribution rather than raw pixels.
Method
Latent Forcing trains one diffusion model over pixels and latent representations with separate schedules that reveal latents before pixels.
Results
Latent Forcing achieves state-of-the-art conditional and unconditional pixel diffusion transformer results on ImageNet at the authors’ compute scale.
Takeaways & Limitations
Generation order, rather than information loss alone, is a key design factor, enabling lossless pixel modeling with latent-diffusion-like convergence benefits.
Takeaways & Limitations
Latent Forcing exhibits cascaded error, requiring latent noise during pixel-generation steps to address errors from earlier generation.
Abstract
from arXiv · showhide
Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, conditional versus unconditional generation, and how tokenizer reconstruction quality relates to diffusability. Applied to ImageNet, Latent Forcing achieves a new state-of-the-art for diffusion transformer-based pixel generation at our compute scale.
1. Introduction
Latent Forcing addresses the information loss and extra decoder required by latent diffusion by ordering latent and pixel denoising so latents provide intermediate structure before pixels. The paper identifies ordering as central to performance and reports state-of-the-art conditional and unconditional pixel generation on ImageNet at its compute scale.
- Motivation: Latent diffusion improves generation quality but discards information through encoding and requires a separately trained decoder.This creates a tradeoff between tokenizer reconstruction quality and downstream generation.
- Motivation: Pixel-space diffusion preserves end-to-end modeling but tends to generate low-frequency details before higher-level semantic structure.The paper characterizes this ordering as less helpful than revealing semantics first.
- Latent Forcing: Latent Forcing jointly diffuses latents and pixels with separate time variables, revealing latents first so they act as a scratchpad for natural-image generation.The generated latent is discarded after denoising, while the model operates directly on pixels.
- Analysis: The paper finds that latent-versus-pixel ordering drives performance in both conditional and unconditional generation.It also uses ordering to analyze self-supervised representations, tokenizer distillation, and reconstruction-generation behavior.
- Results: Latent Forcing achieves state-of-the-art conditional and unconditional pixel diffusion transformer results on ImageNet at the authors’ compute scale.The method uses a standard diffusion transformer with matched compute to existing approaches.
2. Related Work
Related work spans latent diffusion, pixel-space diffusion, multi-time scheduling, and generation-ordering methods. Latent Forcing combines these directions by separately diffusing latent conditions and pixels in a non-autoregressive setting.
- Latent Diffusion: Latent diffusion uses learned latent spaces and commonly faces a reconstruction-generation tradeoff when designing tokenizer representations.State-of-the-art latent models often use low-PSNR tokenizers to favor generation quality.
- Pixel Diffusion Models: Pixel diffusion simplifies pretraining by modeling natural images directly, while recent methods improve its scalability and generation quality.These approaches avoid the dramatically reduced tokenizer dimension that is a core latent-diffusion benefit.
- Time Scheduling: Multi-time diffusion can modify denoising schedules while optimizing a joint distribution, but prior Diffusion Forcing targets autoregressive temporal data and still tokenizes inputs and outputs.Latent Forcing instead jointly generates pixels and deterministic latent representations in a non-autoregressive setting.
- Generation Order: Generation order affects discrete diffusion and can improve conditioning by generating latent structure before visual content.Latent Forcing extends this idea to full latent representations while removing extra architectural components.
3. Ordering the Diffusion Process
The paper orders a joint diffusion process over multiple modalities by assigning each modality its own time variable and schedule. For deterministic latents derived from pixels, non-overlapping schedules factor generation into latent generation followed by pixel generation while preserving the raw-pixel likelihood.
- Multiple Modalities: The model diffuses multiple modalities, typically latents and pixels, each with its own time variable and scheduled denoising trajectory.At training and inference, the schedules are chosen to produce favorable generation orderings.
- Multiple Modalities: For each modality, the model predicts its denoising direction from the jointly provided noised inputs and time variables.The multi-modality objective uses separate outputs and loss weights for the modalities.
- Time Schedules: Inference sets per-modality times as non-decreasing functions of a global time, allowing a chosen ordering through the schedule.The schedule starts all modalities at noise and ends all modalities at data.
- Probabilistic Interpretation: With non-overlapping schedules and deterministic latents derived from pixels, the joint process factors as P(Y)P(X|Y) while optimizing the raw-data distribution P(X).This formalizes latent-first conditioning without requiring a separate decoder distribution.
- Scaling as Scheduling: Scaling a modality changes its signal-to-noise ratio and is informationally equivalent to shifting its noise schedule, so tokenizer scale implicitly selects generation order.The paper motivates an SNR-preserving time-shift function for exploring such schedules.
4. Latent Forcing
Latent Forcing jointly generates pixels and latent embeddings with separate time schedules, using latents to guide pixel generation while retaining a pixel-space model. Experiments show that denoising latent structure before pixels improves generation and that this ordering explains structural differences across timestep combinations.
- Architecture: The architecture adds latent and pixel patch embeddings, a second time-embedding MLP, and optionally splits the final four layers into two output experts.The second time-embedding MLP increases parameters by roughly 0.5%, while the optional expert split adds no parameters or FLOPs.
- Method: Latent Forcing jointly models pixel and latent embeddings with separately scheduled time variables, using latent features as intermediate conditioning before pixel generation.The method operates on raw images and discards the generated latent scratchpad after denoising.
- Training and inference schedules: The Multi-Schedule Model samples independent modality timesteps to explore trajectories, whereas the Single-Schedule Model trains and infers along one fixed trajectory for baseline comparisons.The authors use the Multi-Schedule Model to hold model quality constant while varying ordering, then use the Single-Schedule Model for baseline comparisons.
- Multi-Schedule Diffusion: Latent features should denoise before pixels across multiple trajectories, with most FID gains appearing early, after DINOv2 reaches tDINO = 0.15.The schedule analysis uses FID-10K values for joint DINOv2 and pixel-space diffusion trajectories.
- Multi-Schedule Diffusion: Ordering benefits extend beyond DINOv2: downsampled pixels help before full-resolution generation, and latent structure helps across different latent embedding models.The reported comparison includes additional latent spaces and the downsampled-pixel ordering used to bootstrap high-resolution generation.
- Multi-Schedule Diffusion: At matched reconstruction quality, DINOv2 predictions preserve more spatial structure than pixel predictions, while pixel features improve DINOv2 predictions across nearly all noise levels.DINOv2 features do not inform pixel generation when tpixel ≥ 0.75, and pixel predictions retain greater structural uncertainty in the matched-PSNR comparison.
5. Single-Schedule Modeling
Single-schedule experiments show that denoising latents before pixels is the strongest ordering, with schedule design and training-time noise improving cascaded generation. The resulting Latent Forcing models outperform pixel-diffusion baselines in conditional and unconditional ImageNet generation.
- 5.1. Time Sampling: Latent-before-pixel ordering performs best in the single-schedule experiments.The evaluated cascaded schedule denoises latents entirely before pixels.
- 5.1. Time Sampling: Adding low-timestep pixel sampling moderately improves performance.Ten percent of samples use a pixel timestep from U[0, 0.5] instead of the logit-normal distribution.
- 5.2. Improving Cascaded Generation: Cascaded error causes earlier latent-generation errors to compound into later pixel outputs.The method adds minor latent noise during pixel steps to address this issue, but the benefit appears only during training.
- 5.3. Time Schedules: The variance-shift schedule achieves an unguided FID-10K of 18.57, while cascaded generation performs best among the evaluated single schedules.Inference uses 50 Heun steps, equally spaced along each schedule trajectory.
- 5.5. Distillation vs Ordering: Latent Forcing reduces unguided FID-50K by 1.9× versus JiT with REPA and by 2.5× versus JiT.These results distinguish gains from information ordering from gains attributed to distillation.
- 5.6. Conditioning as Ordering: Unconditional generation shows a 1.8× decrease in guided FID relative to REPA distillation, while improvements transfer across tokenizers.Data2Vec2-based unconditional generation outperforms DINOv2 distillation despite only 150 ImageNet training epochs.
- 5.7. Discussion: Towards Rethinking Compression: Latent Forcing uses the least-ever compressed ImageNet-256 input space while maintaining lossless reconstruction and improving diffusability.The default DINOv2 configuration uses six floats per pixel and outperforms existing pixel-space approaches in conditional and unconditional generation.
6. Conclusion
The paper argues that diffusion generation order is a critical design choice and introduces a practical way to reorder the trajectory while retaining pixel-space likelihood modeling. Experiments and ablations examine how ordering interacts with tokenization and representation learning.
- Generation order is a critical, underexplored component of diffusion models.
- Using multiple tokenizers and time variables to order the diffusion trajectory significantly improves performance.
- The approach is lossless, directly optimizes pixel-distribution likelihood, remains end-to-end at inference, and requires minimal architectural changes.
- Ablations compare any-order and order-specific generation and distinguish tokenizer feature incorporation from REPA-style distillation.
- The authors position the method as a practical and scalable alternative to latent diffusion and a starting point for rethinking tokenization.
A.1. Implementation Details
The implementation details include a configuration table for Latent Forcing.
- Table 12 reports the configuration of Latent Forcing.
A.2. AutoGuidance
The AutoGuidance implementation evaluates how model scale affects generation by training several model and output-expert configurations.
- The study sweeps AutoGuidance across four model configurations differing in base size and added ViT/L output experts.
- All evaluated configurations use fewer parameters and FLOPS than ViT/L, reducing inference FLOPS relative to CFG.
- The authors retain checkpoints at regular intervals and sweep FID-2K and FID-8K to select checkpoints and guidance schedules.
A.3. CFG on a Limited Interval
The CFG-Interval comparison uses different guidance strategies for latent and pixel timesteps, with a specialized schedule and DINOv2 time shift for system-level evaluation.
- CFG-Interval is applied only for system-level comparison to reduce the sweep required for multiple-tokenizer latent diffusion.
- CFG-Interval is best on latent timesteps, while interval-free AutoGuidance is best on pixel timesteps.
- Without interval guidance, AutoGuidance performs best for both latent and pixel timesteps and is used in the reported tables except Table 11.
- The system-level comparison uses interval [0.06, 1.0], guidance value 3.0, and a DINOv2 inference-time shift of α = 0.575.