Source-linked AI summary
LooseControlVideo: Directorial Video Control using Spatial Blocking
Shariq Farooq Bhat, Niloy J. Mitra, Kalyan Sunkavalli
TL;DR
Precise control of complex multi-object video events remains difficult because language is spatially imprecise and dense frame-level guidance is impractical. LooseControlVideo uses sparse oriented 3D primitives to specify high-level choreography while generating secondary dynamics, achieving a 1.2-3× reduction in Trajectory Error across diverse benchmarks.
Problem
Current video controls struggle to specify synchronized multi-object events because language lacks spatial precision and dense frame-level guidance is impractical to author.
Method
LooseControlVideo uses oriented 3D primitives and DNOCS to let users author high-level layout and motion while the generative model infers object deformations and depth.
Results
1.2-3× reduction in Trajectory Error, with consistently best motion alignment and trajectory accuracy on complex human-object interactions.
Takeaways & Limitations
Sparse oriented 3D primitives provide a sufficient geometric prior for orchestrating complex multi-object events while decoupling choreography from fine-grained dynamics.
Takeaways & Limitations
The method does not explicitly bind specific visual identities to individual 3D boxes, complicating identity preservation in complex multi-character scenes.
Abstract
from arXiv · showhide
Precise 3D spatial orchestration in text-to-video generation remains a significant challenge, particularly for multi-object scenes where semantic layout and temporal dynamics are often entangled. While existing depth-conditioned models achieve good structural fidelity, they necessitate dense, frame-accurate guidance that is labor-intensive to author for dynamic events involving deformable objects. We present LooseControlVideo, a framework that enables intuitive and expressive control by using sparse, oriented 3D boxes as a "blocking" proxy. This allows users to author high-level layout and trajectory while leveraging a video generative model to generate realistic occlusions, dynamics and interactions. We achieve this by fine-tuning a Wan 2.2 backbone on a video dataset annotated with DNOCS, a novel encoding for 3D size, orientation and depth-ordered occlusions. Furthermore, our method allows for localized refinement, such as adjusting a jump trajectory or adding an interaction, with minimal disruption to the global scene context. Extensive evaluations on the nuScenes, HO-3D, and BEHAVE benchmarks demonstrate that LooseControlVideo significantly outperforms existing 2D-box and flow-based baselines. Our findings indicate a 1.2x to 3x improvement in Trajectory Error; 2x improvement in Rigid Motion Consistency; and a 1.5x to 2x increase in Occlusion Accuracy over current state-of-the-art layout-conditioned models, demonstrating that oriented 3D primitives provide good geometric prior for complex, multi-agent video authoring.
1 Introduction
LooseControlVideo addresses the difficulty of directing complex, synchronized multi-object video events by replacing imprecise language and labor-intensive dense controls with sparse, oriented 3D blocking. Its DNOCS-conditioned Wan 2.2 model infers fine-grained dynamics from high-level choreography and outperforms existing baselines across multiple benchmarks.
- Problem: Current video models lack directorial controls for choreographing intricate spatio-temporal synchronization and complex multi-object interactions.This creates a gap between generative capability and creative intent.
- Problem: Natural language cannot specify exact spatial trajectories, while dense per-frame depth or edge maps are impractical to author for dynamic scenes.Frame-accurate guidance would require capturing fine-grained events such as every wing-beat and skeletal contraction.
- Method: LooseControlVideo lets users define high-level layout, motion paths, and timing by manipulating sparse, oriented 3D primitives while the model infers secondary dynamics.The approach draws on cinematographic blocking, using coarse spatial proxies to orchestrate movement.
- Method: DNOCS jointly encodes local orientation and global depth in a rendered 2D control signal for conditioning the Wan 2.2 DiT architecture.An automated pipeline extracts temporally tracked 3D oriented boxes without manual 3D annotation.
- Evaluation: 1.2-3× reduction in Trajectory Error and 1.5-2× increase in Occlusion Accuracy are achieved over state-of-the-art baselines across nuScenes, HO-3D, and BEHAVE.The evaluations include navigation and articulated interactions, alongside novel metrics for spatial blocking adherence.
2 Related Work
Prior work spans loose 2D spatial controls, evolving video-generation architectures, and increasingly 3D-aware scene composition. LooseControlVideo addresses the gap between intuitive sparse control and 3D-aware choreography for deforming, interacting multi-object events.
- Image edit metaphors and handles: Loose boxes reduce the need for exact shape guidance but do not support temporal changes from deformation and complex interactions.
- Image edit metaphors and handles: Unlike 2D-centric methods, LooseControlVideo uses oriented 3D primitives to combine sparse keyframe flexibility with loose 3D scene blocking.
- Video generators and control paradigms: Video generation has progressed from inflated-attention U-Nets to Diffusion Transformers, alongside proprietary and open-weight alternatives.
- 3D-Aware Video Synthesis/Editing: Recent 3D-aware video work includes LLM-as-director frameworks that translate scripts into spatial coordinates and world models focused on global 3D consistency.
3 Method
LooseControlVideo converts sparse, oriented 3D boxes and optional camera trajectories into image-space DNOCS controls that expose geometric cues for video generation and editing. The unified formulation supports per-frame and per-object control while preserving visible input-video regions during interpolation.
- Control formulation: Oriented 3D boxes encode time-varying layouts for multiple objects, with optional static or dynamic 3D cameras defining the generated video’s viewpoint.The model represents control as b := {b_t}^T_t=1 and cameras as c := {c_t}^T_t=1.
- Control formulation: Rendering boxes through the given cameras produces image-space controls that explicitly expose depth ordering, occlusion, and perspective cues without modifying the pretrained video model architecture.Direct parameter conditioning leaves these cues implicit and creates a mismatch between abstract 3D controls and the model’s 2D image domain.
- DNOCS representation: DNOCS combines depth-independent orientation hue with depth-dependent brightness in a single 3-channel signal, encoding object orientation, depth ordering, and occlusion relationships.The representation uses fixed L = 0.55 and a = 0.35 for orientation coloring, with βmin = 0.08 and k = 2.0 for brightness falloff.
- Generation and editing: The editing pipeline composites rendered DNOCS into input-video frames using estimated input depth, enabling one formulation for generation and editing with per-frame and per-object control.Input depth is estimated with VideoDepthAnything, and users can select which control form to apply for each frame and object.
- Training data and construction: 10k real videos are annotated through an automated pipeline using GroundingDINO tracks, SAM-based masks, and monocular-depth-derived per-object point clouds.Training randomly uses box-only renders (70%) or mixtures of partial real-video segments and spatially composited mixtures (30%).
- Generation and editing: During interpolation, the model preserves visible input-video regions while generating remaining content consistent with the box controls and text prompt.This behavior applies when vctrl combines the input video with DNOCS renderings.
4 Evaluation and Results
LooseControlVideo is evaluated on real-world driving and human-object interaction datasets using metrics for spatial grounding, trajectory accuracy, motion consistency, and occlusion ordering. It outperforms weaker 2D controls in quantitative and qualitative evaluations, while retaining limitations in identity assignment and manual interaction timing.
- Datasets: The evaluation covers nuScenes, HO-3D, and BEHAVE, spanning real-world driving scenes and complex human-object interactions.nuScenes provides dynamic vehicles, pedestrians, and cyclists; HO-3D and BEHAVE involve human-object interactions with rotations and close-contact motion.
- Evaluation Metrics: The proposed metrics measure containment, trajectory error, occlusion accuracy, and rigid motion consistency from generated masks, trajectories, depth ordering, and scene flow.Containment measures whether generated pixels remain within control regions; TrajErr measures trajectory deviation; OcclAcc measures depth-order adherence; RMC compares rigid motion with estimated scene flow.
- Quantitative Results: LooseControlVideo achieves the lowest trajectory error and highest occlusion accuracy on nuScenes, with DNOCS oriented 3D boxes providing geometric priors absent from 2D flow and box controls.The 2D box baseline has high containment from tight image-space constraints but performs worse on motion alignment and trajectory accuracy because it lacks orientation and depth information.
- Quantitative Results: LooseControlVideo consistently achieves the best GMFA, RMC, and TrajErr performance on HO-3D and BEHAVE, demonstrating stronger motion structure than 2D flow or box controls.These datasets contain complex human-object interactions with rotations and close-contact motion; the control-free baseline instead achieves the highest perceptual quality by receiving ground-truth frames.
- Qualitative Results: Qualitative results show complex interactions, avoided collisions, occlusion-driven disappearances and reappearances, and orientation changes, while editing preserves scene identity, appearance, geometry, and secondary physics-based effects.Figure 4 demonstrates individual-object motion editing, including a skidding car and soccer-ball edits, without disrupting the overall scene.
- Limitations: The main limitations are that 3D boxes are not explicitly bound to visual identities and users must still author the temporal rhythm of interactions.Future directions include assigning multi-view reference images to 3D anchors and allowing sparse spatial trajectories with inferred timing.
5 Conclusion
LooseControlVideo introduces a simple, effective control mechanism that uses oriented 3D bounding boxes to bridge high-level creative intent and low-level video synthesis. Its blocking-based approach uses sparse, 3D-aware primitives as a loose geometric prior for orchestrating complex multi-object events.
- 5 Conclusion: LooseControlVideo bridges high-level creative intent and low-level video synthesis through oriented 3D bounding boxes.The framework is presented as a simple-but-effective control mechanism.
- 5 Conclusion: Sparse, 3D-aware primitives provide a sufficient yet loose geometric prior for orchestrating complex multi-object events.The approach adopts a blocking metaphor from cinematography and game design.
- 5 Conclusion: The method applies the blocking metaphor commonly used in cinematography and game design to video control.This metaphor supports authoring through sparse, oriented 3D primitives.
Supplementary Material · A User Study
The user study compares LCV with weaker control signals using pairwise preference judgments across generation and editing scenarios. Participants consistently preferred LCV, with the strongest results associated with oriented 3D box controls and DNOCS.
- A User Study: The study compares LCV against Depth Only, Optical Flow, and 2D Bounding Boxes for motion following and visual plausibility.The evaluation targets perceptual preference under different control signals.
- A User Study: Participants used a two-alternative forced choice protocol to select between approximately 5-second generated videos.They viewed the conditioning input and, for editing cases, the original video alongside two method outputs.
- A User Study: The study covered both generation and editing scenarios, with original-video context provided only for editing cases.Generation inputs contained rendered control signals, while editing inputs additionally showed the original video.
- A User Study: 16 participants completed the study, with 64 votes collected for each method pair.Table A1 reports the overall completed-session pairwise preference matrix.
- A User Study: 78.1%, 87.5%, and 92.2%: LCV won these shares of overall comparisons against Depth Only, Optical Flow, and 2D Boxes, respectively.The overall matrix reports the percentage preferring the row method over the column method.
- A User Study: 84.4%, 90.6%, and 84.4%: LCV won these shares of editing comparisons against Depth Only, Optical Flow, and 2D Boxes, respectively.Table A2 reports editing-only completed-session pairwise preferences.
- A User Study: 71.9%, 84.4%, and 100%: LCV won these shares of generation comparisons against Depth Only, Optical Flow, and 2D Boxes, respectively.Table A3 reports generation-only completed-session pairwise preferences, supporting users’ preference for DNOCS-based control.
- A User Study: Participants consistently preferred videos generated with DNOCS over methods relying on weaker control signals.The study’s overall conclusion attributes this preference to oriented 3D control signals and richer structural representation.