Source-linked AI summary
Boximator: Generating Rich and Controllable Motions for Video Synthesis
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, Hang Li
TL;DR
Fine-grained motion control remains a challenge in video synthesis. Boximator adds hard and soft box constraints to existing video diffusion models and uses self-tracking to learn box-object correlations; it improves reported motion-control alignment and users favor its results over the base model.
Problem
Video synthesis needs fine-grained motion control, while prior controls such as trajectories do not specify object shape and size.
Source: §1 · p. 1, §4.3 · p. 4Method
Boximator uses hard and soft boxes to select objects and constrain their motion, adding a control module while freezing original model weights and training with self-tracking.
Source: §1 · pp. 1–2, §4.1 · p. 3, §6 · p. 8Results
1.9–3.7x higher AP on MSR-VTT and 4.4–8.9x higher on ActivityNet accompanied user preference over the base model (+18% video quality, +74% motion control); FVD also improved on MSR-VTT.
Source: §1 · pp. 2–3Takeaways & Limitations
Boximator is presented as a general plug-in for video diffusion models, with design and training techniques the authors believe can extend to controls such as human poses and key points.
Source: §6 · p. 8AP is not equivalent to motion-control success rate, and detector errors can affect its absolute values, so the authors emphasize differences between methods.
Source: §5.2 · p. 6
Abstract
from arXiv · showhide
Generating rich and controllable motion is a pivotal challenge in video synthesis. We propose Boximator, a new approach for fine-grained motion control. Boximator introduces two constraint types: hard box and soft box. Users select objects in the conditional frame using hard boxes and then use either type of boxes to roughly or rigorously define the object's position, shape, or motion path in future frames. Boximator functions as a plug-in for existing video diffusion models. Its training process preserves the base model's knowledge by freezing the original weights and training only the control module. To address training challenges, we introduce a novel self-tracking technique that greatly simplifies the learning of box-object correlations. Empirically, Boximator achieves state-of-the-art video quality (FVD) scores, improving on two base models, and further enhanced after incorporating box constraints. Its robust motion controllability is validated by drastic increases in the bounding box alignment metric. Human evaluation also shows that users favor Boximator generation results over the base model.
1. Introduction
Boximator uses hard and soft boxes to provide fine-grained motion control, including object selection and approximate or precise constraints on future positions, shapes, and paths. It plugs into video diffusion models and uses self-tracking to simplify learning box-object correlations.
1. Introduction
- Hard boxes precisely delineate object bounds, while soft boxes specify a region the object must occupy, from a tight bound to the frame boundary.Boxes are linked across frames by unique object IDs.Source: §1 · p. 1
- For image-conditioned generation, users can select objects by drawing hard boxes, a visually grounded alternative to describing every object in language.Boximator can also control foreground and background motion and alter larger-object poses by adjusting smaller components.Source: §1 · p. 1
- For frames without user boxes, algorithm-generated soft boxes approximate motion paths from paired user boxes or a hard box plus a specified path.Boximator is a plug-in for existing video diffusion models; its box encoding includes coordinates, an object ID, and a hard/soft flag.Source: §1 · p. 1
- Figure 1 illustrates hard-box endpoint control, motion paths, camera rotation, pose changes, and control of multiple objects’ trajectories and proximity.Dotted boxes mark first-frame constraints, solid boxes mark last-frame constraints, and arrows indicate motion paths.Source: Figure 1 · p. 2
- Self-tracking first trains the model to generate colored boxes aligned with constraints, then trains it to stop visibly generating them while retaining their internal representation.This addresses the challenge of learning visual box-object correlations without textual grounding.Source: §1 · p. 2
- The control module is trained on 1.1M highly dynamic clips and 2.2M annotated objects generated from WebVid-10M.The authors train Boximator on PixelDance and ModelScope base models.Source: §1 · p. 2
- AP was 1.9–3.7x higher on MSR-VTT and 4.4–8.9x higher on ActivityNet; users favored Boximator over the base model by +18% for video quality and +74% for motion control.The paper also reports improved FVD on MSR-VTT and further video-quality improvement with box constraints.Source: §1 · pp. 2–3
2. Related Work
Prior video-generation work adds controls such as sketches, depth, poses, trajectories, and conditional images. The cited approaches leave gaps in selecting precise or composite objects and in controlling their shape and size.
2. Related Work
- Existing controllability methods use sketches, depth maps, motion vectors, human poses, trajectories, or conditional images to guide video generation.Text-to-video systems can also generate a reference image before producing a video conditioned on that image.Source: §2 · p. 3
- Trajectory controls do not precisely define objects or capture shape and size, limiting control of composite objects and changes such as spreading an arm or approaching.Source: §2 · p. 3
- TrailBlazer and FACTOR require text descriptions for boxed objects; neither supports soft boxes or studies the associated training challenges.TrailBlazer edits attention maps, while FACTOR adds a box-control module to Phenaki.Source: §2 · p. 3
3. Background: Video Diffusion Model
Video diffusion models iteratively predict noise in noisy video inputs, gradually transforming Gaussian noise into video frames. Their 3D U-Net combines spatial processing of individual frames with temporal information exchange.
3. Background: Video Diffusion Model
- The 3D U-Net iteratively predicts noise in noisy video inputs, gradually transforming Gaussian noise into high-quality video frames.The network receives a noisy input, timestamp, and conditions, and is optimized with a noise-prediction loss.Source: §3 · p. 3
- The noisy input z_t is a Gaussian-noise-transformed version of ground-truth video z_0, with ᾱ_t denoting a predefined constant sequence.The supplied equation gives the forward noising form for z_t.Source: §3 · p. 3
- Alternating convolution and attention blocks combine frame-wise spatial processing with temporal information exchange across frames.Spatial attention typically applies self-attention followed by cross-attention, which conditions generation on a text prompt.Source: §3 · p. 3
4. Boximator: Box-guided Motion Control
Boximator adds box-based motion control to video diffusion models while freezing their original parameters. Its pipeline combines box-conditioned attention, self-tracking training, and interpolated soft boxes at inference.
4.1. Model Architecture
- Boximator freezes the original model parameters and adds a self-attention layer to spatial attention blocks for box-conditioned control.Box tokens encode normalized box coordinates, object ID, and a hard/soft flag through Fourier embedding and an MLP.Source: §4.1 · pp. 3–4
4.2. Data Pipeline
- A filtered dynamic WebVid subset supplies 1.1M clips, annotated with boxes for 2.4M objects and projected into random crops.Noun chunks from first-frame descriptions prompt grounding and tracking; boxes outside a crop become border line segments for controlling object entry or exit.Source: §4.2 · p. 4
4.3. Self-Tracking
- Self-tracking trains the model to generate colored boxes alongside objects, simplifying box-object alignment into box generation and constraint-following tasks.After this phase, training removes target-frame boxes; the model stops visibly generating them while retaining box alignment ability.Source: §4.3 · pp. 4–5
4.4. Multi-Stage Training Procedure
- Training first uses hard boxes, then replaces 80% with expanded soft boxes, and finally continues without self-tracking.Soft-box expansion margins are sampled independently per direction from Beta(1, 8); both initial stages use self-tracking.Source: §4.4 · p. 5
4.5. Inference
- At inference, interpolated boxes are expanded and marked soft; a hard box paired with a motion path can instead slide along that path.These constraints guide approximate trajectories while leaving the model flexibility for variations.Source: §4.5 · p. 5, Figure 5 · p. 5
5. Experiments
Experiments evaluate Boximator on video quality, box alignment, human preference, design ablations, and complex motion-control cases. Results show improved box-constrained video quality and alignment, with human raters favoring Boximator over the base model.
5.1. Experiment Settings
- The experiments use PixelDance and ModelScope on MSR-VTT and ActivityNet, evaluating FVD, CLIPSIM, and detector-based box AP.For ActivityNet, 796 clips with noticeable object motion were manually annotated across all 16 frames; MSR-VTT reference boxes were generated automatically.Source: §5.1 · pp. 5–6
5.2. Quantitative Evaluation
- Box constraints improve MSR-VTT FVD from 237 to 174 for PixelDance and from 239 to 216 for ModelScope in text-to-video generation.The reported FVD gains are smaller with first-frame conditioning: PixelDance 113 to 102 and ModelScope 142 to 132.Source: §5.2 · p. 6, Table 1 · p. 7
- ActivityNet mAP with boxes is 4.4-8.9x higher than without boxes, versus 1.9-3.7x on MSR-VTT; Table 2 reports ActivityNet box-alignment results.The ActivityNet test videos were selected for significant object movement; AP is not equivalent to motion-control success because detector errors affect it.Source: §5.2 · p. 6, Table 2 · p. 7
5.3. Human Evaluation
- In a blind comparison of 100 samples, raters favored Boximator over PixelDance for video quality and motion control.Boximator excelled in motion control in 76% of cases and was outperformed in 2.2%; video quality preference was +18.4%.Source: §5.3 · p. 6, Table 3 · p. 7
5.4. Ablation Study
- Ablations show that removing self-tracking or relaxed soft boxes reduces box alignment, while training all U-Net parameters adds no benefit.The all-parameter model had similar FVD, similar MSR-VTT control precision, and lower ActivityNet precision than the default model.Source: Table 4 · p. 7, §5.4 · p. 7
5.5. Case Study
- Case studies illustrate box-guided object counts, frame-spanning motion, composite-object movement, and introducing an object from the frame edge.Examples include placing four pigs, moving a baby across the frame, shifting a man on a horse, and guiding a man from the left edge to center.Source: Figure 6 · p. 8, §5.5 · p. 8
6. Conclusion
Boximator is presented as a general approach to controlling object motion in video synthesis, using boxes to select objects and define motion without extra text.
6. Conclusion
- Boximator uses hard and soft boxes to select objects and define their motions without extra text, while preserving original model weights and adding a self-tracking training method.The authors propose that its design could be adapted to controls such as human poses and key points.Source: §6 · p. 8
Ethical and Social Risks
The paper notes that video generation technologies, particularly advanced video diffusion models, carry ethical and social risks.
Ethical and Social Risks
- Advanced video diffusion models carry risks including deepfakes, misinformation, privacy violations, bias, and effects on intellectual property and creative industries.The passage calls for developers and users to be aware of these risks and use the technologies responsibly.Source: Ethical and Social Risks · p. 9
A. More Implementation Details
Implementation details specify the control module's Fourier embeddings and MLP, training configuration, and inference procedure.
A. More Implementation Details
- Fourier features encode normalized box coordinates, object IDs, and hard/soft flags; the combined 128-dimensional embedding passes through a three-layer 512-dimensional MLP to produce a 1024-dimensional control token.The Fourier features use sine and cosine functions over the input.Source: A. More Implementation Details · p. 12
- Training uses 16-frame, 256x256 videos at 4 FPS, up to eight objects, and three stages of 50k, 50k, and 10k iterations.A quarter of training videos omit box annotations, and half include the first frame as a condition.Source: A. More Implementation Details · p. 12
- Inference uses 50-step DDIM with classifier-free guidance scale 9, substituting null control tokens for negative conditions.Source: A. More Implementation Details · p. 12
B. Results on UCF-101
On UCF-101, box constraints significantly increased AP but had minimal impact on FVD; the authors attribute these patterns in part to noisier, lower-quality boxes.
B. Results on UCF-101
- The evaluation generated 10,240 16-frame videos from 2,048 UCF-101 test videos and used automatically annotated boxes for reference and detection.The original videos supplied real FVD features; DINO+DEVA detected boxes in generated videos for AP evaluation.Source: B. Results on UCF-101 · p. 12
- UCF-101 prompts are longer than those for MSR-VTT and ActivityNet, yielding more boxes per video that are noisier because object names are extracted from prompts.The authors identify lower-quality box annotations as a probable reason for UCF-101's lower absolute AP.Source: B. Results on UCF-101 · p. 13
- On UCF-101, box constraints significantly increased AP across scenarios but had minimal impact on FVD; absolute AP was lower than on other datasets.The authors suggest noisy boxes may explain the minimal FVD impact and lower-quality annotations may explain lower absolute AP.Source: Table 5 · p. 12, B. Results on UCF-101 · p. 13
C. Human Evaluation Details
Four raters compared randomized, anonymized pairs of regenerated videos for quality and motion control, using different criteria for each judgment.
C. Human Evaluation Details
- Four human raters compared randomized regenerated-video pairs from 100 high-quality WebVid videos excluded from training: standard PixelDance used captions and first frames, while PixelDance+Boximator additionally used endpoint bounding boxes.The videos featured prominent camera or object movements, and raters chose which video was better or indicated no preference.Source: C. Human Evaluation Details · p. 13
- For video quality, raters considered visual defects and temporal inconsistencies, and favored richer content when both videos were otherwise free of issues.A more dynamic video could be preferred over one that remained mostly stationary.Source: C. Human Evaluation Details · p. 13
- For motion control, raters preferred videos satisfying the bounding-box constraints at the initial and final frames; if both or neither did, they were expected to choose no preference.Source: C. Human Evaluation Details · p. 13
- Figure 7 places Boximator samples above base-model samples and reports votes as Boximator/no preference/base model; Figures 8 and 9 show further sample videos.Source: C. Human Evaluation Details · p. 13, Figure 7 · p. 14, Figure 8 · p. 15