PaperlayerView exact revision on arXiv

Source-linked AI summary

Level-of-Token Diffusion

Kiyohiro Nakayama, Brian Chao, Jan Ackermann, Hansheng Chen, Federico Tombari, Leonidas Guibas, Lior Yariv, Gordon Wetzstein

arXiv:2610.05816v1cs.CVcs.AI

TL;DR

Diffusion models allocate equal computation to all regions even when detail requirements vary spatially and can be anticipated before generation. LoT Diffusion adapts pretrained transformers to multiresolution token layouts while preserving full-resolution flow prediction. Across image and video generation, it offers favorable quality-efficiency tradeoffs with speedups that depend on the layout's token budget.

  • Problem

    Image and video diffusion models process regions uniformly despite prior knowledge that can indicate where fine detail is needed and computation can be reduced.

    Source: §1 · p. 2

  • Method

    LoT Diffusion uses variable-size rectangular tokens in a content-adaptive layout and adapts pretrained diffusion transformers to predict full-resolution flow from a reduced token sequence.

    Source: §1 · p. 2

  • Results

    Across image and video generation, LoT offers favorable quality-efficiency tradeoffs, with speedups up to 2.04× for images and 3.53× for videos on average relative to full-resolution pretrained models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LoT layouts can turn planned spatial importance into explicit computational allocation, and could serve as an editable intermediate representation for refining visual plans and budgets.

    Source: §6 · pp. 9–10

    Scenes with fine structures or textures throughout offer limited token-reduction opportunities, and aggressive compression can degrade fidelity.

    Source: §A.14 · p. 39

Abstract

from arXiv · show

Image and video diffusion models allocate equal computation to every region, even when the intended scene calls for varying levels of detail. The spatial distribution of detail can often be anticipated before generation, indicating where computation can be reduced. We introduce Level-of-Token (LoT) Diffusion, a framework that turns this knowledge into an explicit multiresolution token layout (Level-of-Token layout) for adaptive and efficient generation. Tokens represent rectangular patches of varying sizes and shapes, allocating finer tokens where detail is needed and coarser tokens elsewhere. We adapt pretrained diffusion transformers to LoT layouts through a patch-wise asymmetric flow parametrization and embeddings for multiresolution tokens, preserving full-resolution flow prediction at every denoising step while processing only a reduced token sequence. LoT Diffusion enables layout-adaptive generation while preserving pretrained generative priors. We demonstrate LoT with layouts derived from semantic masks, bounding boxes, texture variance, and depth-of-field cues, as well as agentic plans. Across image and video generation, LoT offers favorable quality-efficiency tradeoffs, with significant speedups determined by the layout's token budget. Our project website is at https://georgenakayama.github.io/lotdiffusion/.

1 INTRODUCTION

Diffusion models spend equal computation across regions despite predictable differences in required detail, while existing spatial conditioning and efficiency methods do not combine adaptive detail allocation with reduced computation. LoT Diffusion adapts pretrained transformers to multiresolution layouts and delivers favorable quality-efficiency tradeoffs across image and video generation.

1 INTRODUCTION

  • Spatial conditioning methods retain full-resolution processing, while token-reduction methods do not use prior knowledge of detail distribution to control allocation.The paper's goal is to use that prior knowledge for both adaptive detail and computation allocation.Source: §1 · p. 2
  • LoT represents images or videos with rectangular tokens of variable extent and adapts pretrained diffusion transformers to predict full-resolution flow at every denoising step from a reduced token sequence.The adaptation uses patch-wise asymmetric flow parameterization and embeddings and positional encoding for multiresolution tokens.Source: §1 · p. 2
  • Up to 2.04× for images and 3.53× for videos on average, LoT speeds up generation relative to full-resolution pretrained models while offering favorable quality-efficiency tradeoffs against token-reduction methods.The speedup depends on the layout's token budget.Source: §1 · p. 2
  • LoT is demonstrated with semantic masks, bounding boxes, texture variance, depth-of-field cues, and agentic plans.Generated content adapts to the specified token layout.Source: §1 · p. 2

2 RELATED WORK

Prior efficiency methods reduce computation along spatial, attention, or denoising axes, while layout-control methods specify content placement without allocating computation by detail. Content-adaptive tokenization addresses spatial allocation, but existing approaches lack some combination of user control, spatial redistribution, or compatibility with pretrained models.

Methods for efficient generation

  • Efficiency methods reduce cost through token merging or pruning, sparse attention, feature caching, fewer denoising steps, or progressive upsampling, but do not use spatial detail distribution to allocate computation.These methods operate along spatial, attention, or timestep axes without exploiting prior knowledge of where content requires computation.Source: §2 · p. 3
  • Existing content-adaptive tokenization methods variously lack user control, spatial budget redistribution, or compatibility with pretrained diffusion models, whereas LoT combines spatial adaptivity, user control, and pretrained-model compatibility.The paper also describes LoT as complementary to other efficiency techniques.Source: §2 · p. 3

Layout control for diffusion models

  • Layout-control methods specify what appears where, but do not allocate different amounts of computation to regions; LoT uses layout cues to allocate computation spatially.The passage gives empty sky and detailed typography as an example of regions still processed on the same dense token grid by prior methods.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define a full-resolution latent grid and flow-based generation, then introduce variable-size token partitions and the flow and positional-encoding concepts used by LoT. These foundations establish how a reduced token sequence can represent a full-resolution image or video grid.

3 PRELIMINARIES

  • A pretrained DiT patchifies a full-resolution latent grid into a uniform token layout, while the notation extends to video by adding a temporal axis.The latent grid is denoted x ∈ R^(H0×W0×D′), with D′ the channel dimension.Source: §3 · p. 3, §3.1 · p. 4

3.1 LEVEL-OF-TOKEN LAYOUT

  • A LoT layout partitions the uniform token grid into disjoint, axis-aligned rectangular regions whose variable extents are represented by one token each.The resulting sequence has L tokens; compression relative to the pretrained DiT is HW/L for images and THW/L for videos.Source: §3.1 · p. 4

3.2 FLOW-BASED IMAGE AND VIDEO GENERATION

  • Flow matching couples a data sample with Gaussian noise, trains a model to predict velocity from the state, timestep, and optional text conditioning, and generates by solving an ODE.The state and velocity remain on the full-resolution latent grid.Source: §3.2 · p. 4
  • Asymmetric flow matching provides a parameterization for adapting pretrained flow models, which LoT uses to generate a full-resolution latent grid from a reduced token layout.The paper describes this as enabling efficient, content-aware generation.Source: §3.2 · p. 4
  • Because standard RoPE encodes token positions but not spatial support, LoT must give tokens a consistent coordinate system and separately encode their extents.The passage states that this lets a DiT operate on a level-of-token layout.Source: §3.2 · p. 5

4 METHOD

LoT reduces diffusion-transformer sequence length using multiresolution layouts while adapting pretrained models to retain full-resolution flow prediction. Patch-wise asymmetric flow and token extent and position embeddings support generation with heterogeneous patches.

4 METHOD

  • LoT layouts reduce the DiT sequence from HW tokens to L while retaining compatibility with pretrained models and extending naturally to per-frame video layouts.The method combines patch-wise asymmetric flow matching with a minimally modified DiT.Source: §4 · p. 5

4.1 FLOW MATCHING WITH LEVEL-OF-TOKENS

  • Patch-wise asymmetric flow projects each region into an extent-matched basis, then recovers dense flow across the full-resolution latent grid.This addresses the reduced sequence’s mismatch with the full-resolution latent grid required by the pretrained VAE decoder and flow prediction.Source: §4.1 · pp. 5–6

4.2 LEVEL-OF-TOKEN DIT

  • Extent-dependent input and output heads, patch-shape conditioning, and finest-grid-aligned RoPE encode token size and location while reusing pretrained DiT weights.The RoPE change is query-independent, uses standard dense attention, and recovers pretrained positional encoding for unit patches.Source: §4.2 · pp. 6–7

5 RESULTS

LoT generates image and video content that adapts to user-specified token layouts. Across matched-budget comparisons, it offers speedups and favorable quality tradeoffs, while ablations show the full design performs best on nearly all reported metrics.

5 RESULTS

  • Image models were fine-tuned at 1024 × 1024, while video models used 81-frame 720P clips and 200 held-out text-video pairs.Layout-controlled models were trained on mixed-source layouts and support different token extents for image and video generation.Source: §5 · p. 7

5.2 RUNTIME AND QUALITY ANALYSIS OF IMAGE AND VIDEO GENERATION

  • 1.32–2.04× image speedup with LoT-Flux2-4B, leading baselines on most metrics; Wan2.1-14B reaches 1.58–3.53× and best aesthetic and imaging scores among compressed methods.Video results remain competitive on the other evaluated metrics.Source: Table 1 · p. 8, Table 2 · p. 8, §5.2 · pp. 8–9

5.1 LAYOUT-ADAPTIVE IMAGE AND VIDEO GENERATION

  • Across bounding-box, semantic-mask, texture-variance, and depth-derived layouts, finer-token regions retain intricate detail while coarser regions favor smoother or defocused content.Figure 6 illustrates user-specified layouts using LoT-Flux2-9B.Source: §5.1 · p. 8, Figure 6 · p. 8

5.3 ABLATION STUDY

  • The full model performs best on all ablation metrics except ImageReward, with particularly large gains over direct HR prediction and binary grids.The matched-token-budget variants remove patch-size conditioning, replace patch-wise asymmetric flow recovery, or restrict token extents to 1 × 1 and 2 × 2.Source: §5.3 · p. 9

6 CONCLUSION

LoT uses level-of-detail information to guide multiresolution image and video generation, with computation reduced selectively where detail requirements vary spatially.

6 CONCLUSION

  • LoT achieves favorable quality–efficiency tradeoffs over token-reduction baselines while adapting generation to specified layouts.Its speedups depend on the layout's token budget.Source: §6 · p. 9
  • Scenes requiring fine structures or textures throughout offer limited token-reduction opportunities, and aggressive compression can degrade fidelity.LoT is most advantageous when detail requirements vary spatially.Source: §6 · p. 9
  • Agentic plans provide an initial step toward turning planned spatial importance into explicit computational allocation.The authors suggest layouts could become editable representations for refining visual plans and computational budgets, with video applications in robotics and interactive simulation.Source: §6 · pp. 9–10

A APPENDIX

The appendix's main image-generation results use LoT-FLUX.2-4B and corresponding 4B baselines unless specified otherwise.

A APPENDIX

  • Unless otherwise specified, image-generation results use LoT-FLUX.2-4B and corresponding 4B baselines; Figure 6 uses LoT-FLUX.2 9B.Additional LoT-FLUX.2 9B results are on the supplementary website.Source: A APPENDIX · p. 16

A.1 SUPPLEMENTARY EVALUATION MATERIAL

The supplementary website presents examples pairing source cues and token layouts with generated image and video content.

A.1 SUPPLEMENTARY EVALUATION MATERIAL

  • The supplementary website demonstrates layouts from semantic masks, bounding boxes, texture variance, and depth maps, with video playback showing allocation and detail over time.It also includes agent-assisted generation, user-painted layouts, and typography examples.Source: §A.1 · p. 16

A.2 EXTENT-DEPENDENT SUBSPACE FITTING AND DATA SCALING

Extent-dependent fitting adapts token representations and data scaling to heterogeneous patch sizes while retaining a shared flow timestep.

A.2 EXTENT-DEPENDENT SUBSPACE FITTING AND DATA SCALING

  • For each extent, aligned dense patches and extent-matched VAE tokens are used to fit Procrustes bases and associated scales.The bases and scales are fit on a randomly sampled subset of the training split.Source: §A.2 · p. 16
  • A scalar scale is fit for each extent by matching the root-mean-square norms of projected dense patches and corresponding tokens.The scaling addresses the fact that projection alone does not preserve pretrained token magnitude.Source: §A.2 · pp. 16–17
  • During training, the extent-specific scale is broadcast over dense sites, and the patch-wise asymmetric flow is applied in scaled data space.The model predicts asymmetric velocity, from which dense velocity and a clean-data estimate are recovered using a clamped noise-level-weighted clean-data loss.Source: §A.2 · p. 17
  • At inference, flow is integrated in scaled data space and mapped back to pretrained latent space; the fitted VAE-based tokens, projections, and scales remain fixed.The multi-scale VAE tokens are used only to fit the projections and scales.Source: §A.2 · p. 17
  • The method avoids extent-dependent timestep calibration, which would give tokens different internal timesteps and depart from shared-timestep conditioning.Scaling clean data before adding standard Gaussian noise lets all tokens use the same timestep; orthonormal projections preserve standard Gaussian projected noise.Source: §A.2 · p. 17

A.3 DERIVING TOKEN LAYOUTS FROM DIFFERENT SOURCES

The section derives spatial token layouts from scene cues, assigning finer tokens where detail is needed and coarser tokens elsewhere. Its procedures cover semantic regions, luminance variation, depth of field, and user- or agent-specified detail maps.

A.3 DERIVING TOKEN LAYOUTS FROM DIFFERENT SOURCES

  • The same spatial allocation rules apply to images and video frames, and the resulting layouts use lattice-aligned quadtree tiling.Tiling starts from 8×8 blocks and recursively splits blocks; a high-detail cell prevents its containing block from remaining coarse.Source: §A.3 · pp. 17–18

Semantic masks and bounding boxes

  • Semantic masks and bounding boxes assign finer token levels to selected regions than to the background, with finer levels taking precedence where regions overlap.Levels 0–3 correspond to token extents from 8×8 down to 1×1.Source: §A.3 · p. 17

Texture variance (Variable-Rate Shading)

  • Texture-variance layouts coarsen candidate regions along directions whose estimated luminance-variation error falls below a tolerance.Sensitivity and ambient terms tune overall detail; larger values permit coarser tokens.Source: §A.3 · p. 17

Depth maps and depth of field

  • Depth-of-field layouts use depth relative to a selected focal plane to assign fine tokens near focus and coarser tokens farther from it.A region is merged only if its smallest blur radius supports the larger token size, so a small in-focus part can preserve detail around the subject and its boundaries.Source: §A.3 · p. 18

Custom detail maps

  • Users or agents can specify a detail map, which is converted to a detail field where larger values request finer tokens.The map can be smoothed and adjusted with a gain before being clipped to [0, 1].Source: §A.3 · p. 18

A.4 IMAGE GENERATION TRAINING DETAILS

The image model is fine-tuned from FLUX.2-klein-base-4B with frozen text encoder and VAE, using layouts derived from several visual cues. Training adapts selected components while retaining most pretrained parameters.

Training setup

  • The FLUX.2-klein-base-4B text encoder and VAE remain frozen, while LoRA adapters and LoT-specific heads and normalization are fine-tuned.The LoRA rank is 256 with dropout 0.05; other pretrained parameters remain frozen.Source: §A.4 · p. 18

Training data and source mixture

  • Training uses Aesthetic-Train-V2 images with a 95%/5% training and held-out split, deriving layouts from masks, boxes, luminance variation, and depth maps.Images are resized while preserving aspect ratio and center-cropped to 1024×1024.Source: §A.4 · p. 18

Training setup

  • The 4B model is trained for 50,000 steps with AdamW, batch size 32, and learning rate 10^-4.The asymmetric-velocity predictor uses the clean-data loss, with its noise weighting bounded near zero noise.Source: §A.4 · p. 19

9B model

  • A separate 9B model uses three million LAION-3M images for four layout branches and bokeh-rendered Aesthetic-Train-V2 images for depth of field.The five branches are sampled equally; qualitative results use the 24,500-step EMA checkpoint.Source: §A.4 · p. 19

A.5 IMAGE GENERATION BASELINE COMPARISON DETAILS

The supplementary sections detail image and video evaluation, additional layout-adaptive results, controllability tests, ablations, and limitations. Across these comparisons, layouts guide where detail is allocated, while quality and achievable savings depend on token budget and scene detail.

A.5 IMAGE GENERATION BASELINE COMPARISON DETAILS

  • Image evaluation uses 5,261 held-out prompts and reports preference, fidelity, image-quality, and runtime measures; additional qualitative baseline examples appear in Figures 8 and 9.The VRS layouts are derived from corresponding held-out reference images; runtime includes prompt encoding, sampling, and VAE decoding but excludes preprocessing and model loading.Source: §A.5 · p. 19, §A.6 · p. 19, Figure 8 · p. 20

A.7 ADDITIONAL QUANTITATIVE COMPARISON: TOKEN BUDGET AND IMAGE QUALITY

  • Lower token budgets retain recognizable subjects and scene structure while simplifying fine textures; higher budgets support finer spatial detail.The examples use semantic-mask and texture-variance layouts, illustrating a controllable budget–detail trade-off.Source: §A.7 · p. 22, Figure 10 · p. 23, Figure 11 · p. 24
  • At matched token budgets, LoT-FLUX.2-4B achieves lower FID and pFID and higher MUSIQ and TOPIQ scores than the token-matched FLUX.2 baseline.The baseline generates uniformly at a lower resolution and upsamples, whereas LoT generates at 1024×1024 with spatially varying token extents.Source: §A.7 · p. 22, Figure 14 · p. 27

A.8 ADDITIONAL LAYOUT-ADAPTIVE IMAGE GENERATION RESULTS

  • Bounding boxes and semantic masks concentrate fine tokens in selected regions, while texture-variance and depth-derived layouts allocate detail spatially or around focal regions.Additional examples are shown in Figures 15–18.Source: §A.8 · p. 28, Figure 15 · p. 29, Figure 16 · p. 30

A.9 LAYOUT CONTROLLABILITY COMPARED WITH DENSE GENERATION

  • With target boxes withheld, LoT yields higher target-region coverage under both layouts, gaining 15.79 and 21.89 percentage points for 4B and 9B, respectively; subject detection is slightly lower at 4B and comparable at 9B.Table 4 reports all values as percentages.Source: §A.9 · p. 33, Table 4 · p. 33

A.10 VIDEO GENERATION TRAINING DETAILS

  • Video training uses 81-frame clips, while evaluation generates 81-frame 1280×720 videos at 15 fps for 200 held-out prompts using layouts derived from generated reference videos.The evaluated models receive layouts, not reference RGB videos; the original held-out clips were mostly static.Source: §A.10 · p. 36, §A.11 · p. 36

A.12 ADDITIONAL VIDEO RESULTS

  • Video demos show static background and keyframed detail maps for moving subjects, alongside qualitative examples using semantic masks, boxes, texture variance, and depth of field.The baseline comparison setup is described in Section A.11.Source: §A.12 · pp. 36–37

A.13 ADDITIONAL ABLATION STUDY RESULTS

  • The ablation study varies asymmetric-flow recovery, layout token levels, and patch-size embeddings across token budgets.Table 5 extends the main-paper ablation results beyond a single threshold setting.Source: §A.13 · p. 37, Table 5 · p. 37
Loading 2610.05816v1…