Latent Space Jam: Layout Guided Video Synthesis

1 Introduction

• The paper frames explicit control of spatial composition within and across frames as necessary for structured visual storytelling.Recent text-based video generators can produce diverse and creative results from free-form prompts, but lack explicit mechanisms for spatial composition control.Source: §1 · p. 1
• JAM controls spatial composition and motion with bounding boxes across frames, without training or fine-tuning the pretrained generator.The method optimizes the latent input using selected cross-attention layers, then injects the resulting guide video at an early diffusion timestep for refinement.Source: §1 · pp. 1–2
• The approach consistently outperforms prior methods in complex, dynamic, and multi-object scenarios while preserving pretrained-backbone fidelity and requiring no additional training.The introduction reports support from subjective human evaluations and objective quantitative metrics.Source: §1 · p. 1
• A guide-generation stage pushes layout adherence, then a refinement stage uses the diffusion model to improve coarse results and address the layout–visual-quality trade-off.The authors identify poorer visual quality under stronger layout regularization as the trade-off motivating the two stages.Source: §1 · p. 1
• The method intervenes in only a small subset of cross-attention layers and diffusion steps, making single-GPU execution possible with minimal overhead relative to backbone inference.Video control is challenging because temporal consistency is harder to enforce and optimizing across every layer is impractical.Source: §1 · p. 1

2 Related Work

• Image-generation work uses cross-attention for test-time semantic grounding, token presence, object saliency, spatial layouts, regions, and compositionality without retraining.The paper presents these methods as establishing attention as a controllable structure for semantic and spatial constraints.Source: §2 · p. 2
• Many video-control methods add adapters or control branches for structural signals, enabling motion, layout, or viewpoint control but requiring paired supervision, retraining, and model modification.Signals include edges or depth, pose or trajectories, combinations of signals, and camera parameters [Hu and Xu 2023; Hu 2024; Karras et al. 2023; Wang et al. 2024; Yang et al. 2024; Jiang et al. 2025; Wang et al. 2023; Bai et al. 2025].Source: §2 · p. 2
• Training-free video methods vary in control strategy: Peekaboo and TrailBlazer align foreground motion through cross-attention, FreeTraj also guides latent noise, and LVD uses LLM-generated box trajectories.SwitchCraft and TempoControl modify contemporary DiT video models for temporal control.Source: §2 · p. 2
• JAM uses optimization-based attention alignment to scale to complex spatiotemporal layouts while preserving motion diversity and visual fidelity, unlike direct attention and latent-noise manipulation.The paper reports quantitative-metric and human-evaluation support for this comparison.Source: §2 · p. 2

Video Diffusion Models

• Diffusion learns to reverse gradual noising: a clean sample x_0 is noised over T steps, and a neural network predicts noise for iterative denoising from pure noise.Source: Video Diffusion Models · p. 2
• Latent video diffusion encodes frames into compact representations with a variational autoencoder and performs diffusion in latent rather than pixel space.Video models extend image diffusion to sequences of frames.Source: Video Diffusion Models · p. 2
• Diffusion Transformers replace convolutional U-Nets with spatiotemporal latent-token processing using self-attention.The paper says this design better captures long-range spatial and temporal dependencies, improving temporal coherence and scalability.Source: Video Diffusion Models · p. 2
• The model formulation uses z_t for the noisy latent at timestep t and c for a conditioning signal.Source: Video Diffusion Models · p. 2

Test-Time Latent Optimization via Cross-Attention

• Cross-attention maps represent how conditioning concepts influence spatial regions during denoising by relating text tokens to visual latent tokens.Source: Test-Time Latent Optimization via… · p. 2
• Test-time optimization defines a differentiable loss over model activations for a target constraint, then backpropagates gradients to update the latent rather than model weights.The formulation uses cross-attention activations A(·), conditioning input c, and step size η.Source: Test-Time Latent Optimization via… · pp. 2–3
• Treating cross-attention as an interface lets constraints steer inference through latent updates while model parameters remain fixed.Source: Test-Time Latent Optimization via… · p. 3

4 Latent Space Jam

• A two-stage pipeline separates layout adherence from visual quality: a coarse guide encodes the desired layout, and video diffusion refinement improves appearance while maintaining layout and motion.The guide can be injected at an early diffusion timestep; early steps primarily govern global layout, while later steps refine details.Source: §4 · p. 3, §4.1 · p. 3, Figure 2 · p. 4

4.1 On The Importance of a Guiding Prior

• A low-fidelity guide can yield a high-quality refined video that preserves the intended layout, so the guide need not be visually detailed if it adheres closely to that layout.The approach addresses the tradeoff between layout adherence and visual quality by focusing on layout in the guide stage and enhancing it afterward.Source: §4.1 · p. 3, §4 · p. 3

4.2 Are Text-to-Image Models Suitable for Guidance?

• Frame-by-frame text-to-image guidance can match or outperform prior methods on refined-video overlap and trajectory metrics, but independent generation does not preserve stable object structure or realistic interactions.The baseline generates frames independently and inserts separately generated objects into the scene.Source: §4.2 · p. 3

4.3 Utilizing Video Models for Better Guidance

• The proposed guide optimizes latent inputs of a frozen video diffusion model to align selected cross-attention maps with the target layout; stronger constraints improve layout adherence but reduce appearance quality.The guide is injected at an early diffusion step so a second pass can restore visual fidelity.Source: §4.3 · p. 3, Figure 2 · p. 4
• The framework achieves superior spatial alignment and trajectory consistency across metrics, with Wan 2.2 strongest at guide stage and LTX-2 comparable or stronger after refinement.Table 1 compares training-free layout-guided methods across datasets using overlap, trajectory, and visual-quality metrics.Source: Table 1 · p. 6

5 Results

• Evaluation spans about 100 Static, Dynamic, and Multi layouts with binary video-mask guidance and complementary measures of layout, trajectory, video quality, and prompt alignment.The three categories contain approximately 40, 40, and 20 layouts, respectively.Source: §5.0.1 · p. 6, §5.1 · pp. 6–7

5.1 Quantitative Results

• Across all datasets and metrics, the method strongly outperforms other approaches in alignment and video quality, while same-backbone TrailBlazer underperforms it across stages.The comparison includes Static, Dynamic, and Multi settings.Source: §5 · p. 6, §5.1 · p. 7

5.2 Qualitative Results

• Figure 7 shows stronger layout alignment and video quality, with strict guide-stage adherence trading off visual fidelity that refinement later recovers.The guide-stage optimization strength is adjustable through the learning rate.Source: §5.2 · p. 7
• Direct cross-attention-map manipulation can produce artifacts or weak layout control that refinement often cannot recover.The reported issue applies to the Wan 2.2 TrailBlazer implementation and prior benchmarks on earlier models.Source: §5.2 · p. 7
• In a 250-response study of 15 videos, the method leads all baselines in layout adherence and is competitive in visual quality and prompt adherence, though T2V is stronger on those latter measures overall.On some datasets, the method also exceeds T2V in visual quality or prompt alignment.Source: §5.2 · p. 7

5.3 Ablation Analysis

• Early layer ranges 0–10, 10–20, and 0–20 yield higher guide-stage IoU and AP50 and lower CD than later ranges, with the advantage persisting after refinement.The study randomly samples layers from the selected range at each diffusion timestep.Source: §5.3 · p. 7
• Higher refined resolution benefits results, while lower-resolution guide generation greatly reduces inference time with only a slight performance drop.Wan performs strongest during guidance; after refinement, LTX has higher IoU and AP50 with comparable trajectory metrics.Source: §5.3 · p. 7

5.4 Additional Applications

• Generating a prior from an existing video supports attribute editing, and optimizing style-related prompt parts produces different styles in that video.Source: §5.4 · p. 7

• Bounding boxes specify where content appears over time but not whether changes come from object movement or camera transformations, which can cause ambiguous or unintended behavior.Source: §6 · p. 7
• Performance may decrease with detailed motion, multiple interacting objects, or crowded layouts, which remain challenging settings for the approach.Source: §6 · p. 7

7 Conclusion

• The method enables explicit control over spatial composition and object motion without additional training.It uses minimal intervention at inference time while preserving the generative strength of modern video diffusion models.Source: §7 · p. 7
• JAM consistently improves over prior methods across static, dynamic, and multi-object scenarios.The comparisons include dynamic layouts and multi-object scenes where competing methods show artifacts or fail to maintain spatial and temporal consistency.Source: §7 · p. 7, Figure 7 · p. 9
• In the user study, JAM achieves the strongest layout adherence while remaining competitive in prompt alignment and visual quality.T2V excels in prompt adherence and visual quality but shows poor layout adherence.Source: Table 2 · p. 8
• The layer ablation on Wan 2.2 indicates that earlier layers perform better for layout adherence.Source: Table 3 · p. 8
• For the refined block of the Wan model, at the highest resolution (480 × 81), generating the respective guide at the lowest resolution (row 6) is preferable for trajectory accuracy (CD, I, I-AP50), with negligible decrease in overlap accuracy (IoU, AP50).Source: Table 4 · p. 8