PaperlayerView exact revision on arXiv

Source-linked AI summary

ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Features

Jay Mahajan, Chang Liu, Rauf Makharov, Viraj Shah, Alexander Schwing, Svetlana Lazebnik

arXiv:2609.04649v1cs.CV

TL;DR

Fine-grained spatial and motion control remains challenging for text-conditioned diffusion, especially across image and video generation. ReaDiT uses a lightweight predictor and inference-time guidance from a single DiT block to control spatial targets and motion, achieving competitive or improved guidance while using fewer parameters. Its main limitation is increased inference time and difficulty with fast motion.

  • Problem

    Text prompts provide limited control over nuanced spatial targets, while existing video-control adapters can be computationally demanding, data-intensive, or task-specific.

  • Method

    ReaDiT predicts spatial targets from a single DiT block's features and guides latent generation for depth, pose, edge, camera, and motion control.

  • Results

    ReaDiT achieves competitive or improved guidance accuracy across spatial and video-control evaluations while using substantially fewer parameters than prior approaches.

  • Takeaways & Limitations

    A compact feature-based controller can unify spatial and motion control across image and video DiT generation without modifying the base model.

  • Takeaways & Limitations

    Inference-time latent optimization increases generation time, and motion guidance struggles with fast motion because temporal upsampling is lacking.

Abstract

from arXiv · show

We present DiT Readout (ReaDiT) Guidance, a lightweight framework for controlling generation with Diffusion Transformer (DiT) models via their internal feature representations. ReaDiT Guidance uses features from a single DiT block to steer the generative process according to spatial targets - like depth, pose, or edge maps - provided at test time. Furthermore, since modern text-to-video models are largely built on DiT backbones, ReaDiT Guidance naturally extends to video generation, enabling camera and motion control. Experimental results demonstrate that our approach achieves competitive or improved results compared to existing feature-based and off-the-shelf adapter-based approaches while requiring fewer parameters.

1. Introduction

Fine-grained spatial control remains difficult for text-conditioned diffusion, while existing adapters can be costly, data-intensive, or task-specific. ReaDiT addresses these limits by using features from a single DiT block to unify spatial and motion control across images and videos.

  • Text prompts alone struggle to guide generation toward nuanced depth, pose, and edge targets.
  • ControlNet and similar adapters provide effective localized control but require substantial parameters and paired training data.
  • Extending adapter-based control to video is computationally and data intensive, while lightweight alternatives are often limited to specific tasks.
  • ReaDiT uses a lightweight predictor and guidance strategy based on features from a single DiT transformer block.
  • The framework supports depth, pose, edge, camera, and motion control across image and video generation using a small training dataset.
  • Experiments report competitive or improved guidance accuracy against feature-based and adapter-based methods with substantially fewer parameters.

2. Related Work

Diffusion control methods commonly use external adapters, while Readout Guidance instead steers generation from learned diffusion features. ReaDiT extends this feature-based approach from U-Net models to DiTs and video generation.

  • Latent diffusion models generate images and videos by denoising latent representations conditioned on inputs such as text.
  • External adapters such as ControlNet and T2I-Adapter inject spatial conditioning into frozen diffusion models for depth, pose, and edge control.
  • Readout Guidance predicts spatial targets from frozen diffusion features and uses gradient updates on latents to steer generation.
  • ReaDiT extends this feature-control paradigm to Diffusion Transformers and demonstrates its use for video generation.
  • Table 1 compares DiT control approaches by supported modalities, parameter overhead, training-data requirements, and implicit model conditioning.
  • Prior feature studies largely examine U-Net models, leaving the learned representations of increasingly important DiTs less understood.

3. ReaDiT Guidance

ReaDiT predicts spatial targets from frozen DiT features and uses those predictions to guide image and video generation. Its lightweight readout architecture relies on a single transformer block, timestep-conditioned multi-scale decoding, and task-specific training and guidance objectives.

  • Guidance framework: ReaDiT extracts a feature from a frozen DiT at timestep t and predicts a guidance target used to optimize the diffusion latent.The same readout formulation applies to image and video generation.
  • DiT feature selection: Features from a single DiT block generally suffice for compelling spatial readouts because DiT features are similar across blocks and encode global representations.The method therefore uses |L| = 1 rather than concatenating features from multiple layers.
  • Training: Training adds noise to an input latent, extracts intermediate DiT features, and learns to predict task targets produced by task-specific models.The general objective minimizes a task-dependent loss over sampled timesteps and inputs; dense depth, pose, and edge tasks use mean squared error.
  • Training: Log-based timestep sampling emphasizes lower-noise steps while retaining exposure to other timesteps.The sampling distribution is log t ∼ U(log t_min, log t_max).
  • Readout architecture: The readout conditions features on timestep t, projects them into multi-scale maps, progressively fuses and upsamples them, and applies a task-specific head.The architecture includes timestep-conditioned normalization, channel projection, reshaping into a spatial grid, and multi-resolution branches.
  • Video guidance: ReaDiT extends to video by using spatio-temporal DiT features for spatial guidance and optical-flow guidance of specific point trajectories.For motion guidance, descriptors from a reference frame and another frame produce predicted flow, and an ℓ1 guidance loss steers trajectories.

4. Experiments

Experiments show that ReaDiT provides effective spatial and motion control for images and videos, consistently following targets while preserving quality and prompt adherence. It outperforms RG across reported image and video comparisons, while remaining competitive with dedicated methods.

  • Experimental setup: ReaDiT is evaluated on depth, pose, edge, spatial-video, and optical-flow guidance using SD3, FLUX, and CogVideoX.Image training uses approximately 16K PascalVOC images with labels from off-the-shelf depth, pose, and edge models; video experiments use CogVideoX.
  • Spatial prediction results: ReaDiT produces reasonable depth, pose, and edge predictions from frozen DiT features despite lightweight training on a small dataset.Its predictions capture overall spatial structure, including fine edge details and accurate human poses, though they do not always surpass task-specific models.
  • Spatial image guidance: ReaDiT consistently follows spatial targets and produces natural, high-quality images across depth, pose, and edge tasks, improving upon RG.RG frequently fails to follow guidance or introduces stretched and distorted objects.
  • Spatial image guidance: ReaDiT outperforms RG across all reported image-guidance tasks and metrics under both generated-image and real-image settings.The evaluation reports RMSE for depth, PCK@0.2 and mAP for pose, and ODS for edge; ReaDiT is also comparable or better than ControlNet with significantly fewer parameters.
  • Dual guidance: ReaDiT combines depth and pose guidance, including conflicting targets from different images, while producing coherent images that follow both spatial constraints.For inconsistent targets, pose guidance covers the full image while depth guidance is restricted outside an automatically detected pose mask.
  • Spatial video guidance: ReaDiT successfully translates spatial video inputs into coherent videos, whereas RG’s performance degrades significantly under spatial video inputs.ReaDiT shows higher spatial-control adherence than RG quantitatively and qualitatively.
  • Motion video guidance: ReaDiT guides videos toward target motion and supports camera movements including panning and zooming, but its motion performance falls short of dedicated motion-specific methods.The authors attribute the gap primarily to limited temporal upsampling for fast dynamics.

5. Conclusion

ReaDiT Guidance controls Diffusion Transformers through a single DiT feature and supports depth, pose, edge, spatial, camera, and motion control across images and videos. Its main limitations are increased inference time and difficulty with fast motion.

  • Conclusion: ReaDiT Guidance controls Diffusion Transformers through a single DiT feature for image and video generation.The method covers depth, pose, edge, spatial, and motion-guided generation.
  • Limitations: Inference-time latent optimization increases generation time, with guided images taking roughly 1 minute versus 30 seconds unguided.A guided 16-frame video takes approximately 10 minutes versus 1.5 minutes unguided on a single A40.
  • Limitations: Motion guidance has difficulty with fast motion, motivating future work on improving fast dynamics and reducing inference cost through distillation.

A.1. ReaDiT Architecture Details

ReaDiT uses timestep-conditioned feature modulation, multiscale fusion, and task-specific heads to decode spatial or correspondence outputs from DiT features. It extracts features from one backbone block for each supported model.

  • Feature encoder: Timestep-conditioned AdaLN modulates DiT features to account for timestep-dependent distributions and suppress extreme outlier activations.The design preserves timestep information while improving downstream spatial prediction.
  • Task-specific outputs: A small convolutional task-specific head maps fused latent representations to dense spatial targets, while correspondence descriptors are used directly for optical-flow prediction.
  • Feature selection: ReaDiT extracts features from a single backbone block: block 12 in SD3-Medium, joint block 18 in FLUX.1 dev, and block 15 in CogVideoX.

B. PCA Analysis

PCA and prediction experiments show that DiT block features are highly redundant, unlike diverse U-Net features, and that a single DiT block generally suffices for spatial readouts. Redundant multiblock inputs can instead hurt generalization.

  • Feature comparison: U-Net features vary more across layers, whereas DiT features remain similar across blocks because transformers preserve constant token dimension and spatial resolution.
  • Feature comparison: An average R2 score of 0.7 between one DiT block and the remaining blocks confirms substantial redundancy across DiT representations.The analysis attributes this redundancy to similar token representations across transformer blocks.
  • Single-block validation: Single-block ReaDiT models generally achieve lower depth RMSE than models trained on the last ten blocks, supporting single-block readouts.The comparison is evaluated on SD3-Medium and SD3-Large base models.
  • Single-block validation: Redundant multiblock inputs may destabilize learned weights and worsen generalization on unseen data despite containing at least as much information as one block.

C.1. Additional Results on Spatial Prediction

ReaDiT produces cleaner and more structured depth, pose, and edge predictions than RG, despite being trained on a much smaller dataset. The authors attribute this advantage to its time-conditioned architecture and log-based training strategy.

  • ReaDiT achieves competitive prediction performance against task-specific models despite training on a much smaller dataset.
  • Across depth, pose, and edge tasks, ReaDiT consistently produces cleaner and more structured predictions than RG.RG outputs are noisier, with weaker subject boundaries, depth details, skeleton structure, and edge quality.

C.2. Additional Results on Spatial Image Guidance

ReaDiT provides stable spatial image guidance across scenes, styles, prompts, and target modalities while preserving visual quality. Its readout and training strategies outperform RG under comparable guidance settings, and balanced dual-task weighting best preserves both depth and pose adherence.

  • Spatial image guidance: ReaDiT guides images to follow spatial targets while maintaining natural, high-quality outputs across diverse prompts and conditions.
  • Spatial prediction: A single DiT block predicts depth more accurately than using the last 10 blocks together.This comparison is reported for SD3-Medium and SD3-Large using RMSE.
  • Spatial prediction: ReaDiT consistently outperforms RG in qualitative depth, pose, and edge prediction quality.
  • Comparison with RG: ReaDiT’s readout and training strategies contribute beyond the guidance strategy, allowing it to perform best on most tasks and metrics.RG improves under ReaDiT’s guidance strategy, but ReaDiT still outperforms RG on most evaluations.
  • Prompt and target interaction: ReaDiT follows spatial targets without the visible artifacts observed in RG and remains effective when prompts and targets partially conflict.Detailed prompts also produce richer background and lighting details.
  • Dual guidance: Dual guidance improves pose adherence while keeping depth accuracy comparable to depth-only guidance.The depth signal provides complementary geometric information, particularly for incomplete or unusual poses.
  • Dual guidance: When depth and pose loss weights are roughly balanced, overall performance is best across both tasks.Increasing one task’s weight improves its adherence while reducing adherence to the other; extreme ratios degrade one target.

C.5. Additional Results on Video Guidance

ReaDiT extends spatial and motion guidance to video generation while maintaining temporal consistency and visual quality. Guided and unguided generations also show nearly identical diversity and text-alignment scores.

  • Spatial video guidance: ReaDiT adheres to spatial signals for depth, pose, and edge guidance while maintaining temporal consistency and visual quality across video tasks.
  • Comparison with RG: ReaDiT demonstrates stronger spatial control than RG across depth and pose conditions while generating consistent video sequences.
  • Diversity and text alignment: ReaDiT preserves diversity and text alignment during guided generation, with nearly identical CLIP-T and pairwise LPIPS scores to unguided generation.CLIP-T measures text alignment, while pairwise LPIPS measures diversity.

E. Evaluation Setup

The evaluation uses real and generated image settings, alongside synthetic video prompts and reference motions, to assess spatial and motion guidance. The experiments span diverse scenes, styles, objects, actions, and environments using established pretrained models and predictors.

  • Image evaluation: Image evaluations use MSCOCO validation data and synthetic data, with control maps obtained from real images or generated images.
  • Image evaluation: Generated image evaluations use 250 diverse prompts and extract depth, pose, or edge targets with task-specific predictors.
  • Video evaluation: Video motion evaluation uses six reference motions and 20 prompts covering varied scenes, objects, and compositions.
  • Video evaluation: Video spatial evaluation uses more than 100 prompts and extracts depth, pose, or edge targets from unguided CogVideoX videos.
  • Prompt diversity: The prompt set spans object materials, human actions, lighting conditions, environments, scene complexity, styles, and semantic categories.
  • Model licensing: The evaluated pretrained models have different licensing conditions, including Apache 2.0 and custom non-commercial research licenses.
Loading 2609.04649v1…