PaperlayerView exact revision on arXiv

Source-linked AI summary

Mojito: Motion Trajectory and Intensity Control for Video Generation

Xuehai He, Shuohang Wang, Jianwei Yang, Xiaoxia Wu, Yiping Wang, Kuan Wang, Zheng Zhan, Olatunji Ruwase, Yelong Shen, Xin Eric Wang

arXiv:2412.08948v2cs.CVcs.CL

TL;DR

Text-to-video models struggle to control motion direction and intensity, despite the need for user-aligned motion in generated videos. Mojito combines training-free cross-attention guidance for trajectories with a learned motion-intensity module. The authors report high-quality motion-controlled videos, generalizable and orthogonal to existing approaches.

  • Problem

    Text-to-video models often struggle to generate large motion magnitudes or adaptable user-aligned intensities, and lack dedicated motion mechanisms and detailed motion annotations.

    Source: §1 · p. 1

  • Method

    Mojito combines training-free cross-attention guidance for user-specified object trajectories with a training-based motion-intensity module that uses optical flow.

    Source: §1 · pp. 1–2

  • Results

    Mojito produces high-quality motion-controlled videos while remaining generalizable and orthogonal to existing approaches.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Directional control does not require training, and intensity control does not require additional motion-data annotations.

    Source: §6 · p. 8

    The authors note that generated videos may be misused and raise concerns about fairness and potential biases in outputs.

    Source: Impact Statement · p. 8

Abstract

from arXiv · show

Recent advancements in diffusion models have shown great promise in producing high-quality video content. However, efficiently training video diffusion models capable of integrating directional guidance and controllable motion intensity remains a challenging and under-explored area. To tackle these challenges, this paper introduces Mojito, a diffusion model that incorporates both motion trajectory and intensity control for text-to-video generation. Specifically, Mojito features a Directional Motion Control (DMC) module that leverages cross-attention to efficiently direct the generated object's motion without training, alongside a Motion Intensity Modulator (MIM) that uses optical flow maps generated from videos to guide varying levels of motion intensity. Extensive experiments demonstrate Mojito's effectiveness in achieving precise trajectory and intensity control with high computational efficiency, generating motion patterns that closely match specified directions and intensities, providing realistic dynamics that align well with natural motion in real-world scenarios.

1. Introduction

Mojito addresses the difficulty of controlling both motion direction and intensity in text-to-video generation, where models lack dedicated motion mechanisms and annotated motion data are scarce. It combines training-free directional guidance with learned intensity modulation and reports high-quality video generation with effective, efficient motion control.

1. Introduction

  • Existing text-to-video models struggle to produce large motion magnitudes and user-aligned motion intensity, while motion labels are scarce and detailed-annotation training is computationally costly.Diffusion models also lack dedicated motion modeling and can produce flickering, trajectory inconsistencies, and artifacts.Source: §1 · p. 1
  • Directional Motion Control (DMC) steers user-specified trajectories or directions at test time without training and generalizes across U-Net- and DiT-based architectures.It guides attention during forward diffusion using inputs such as bounding boxes.Source: §1 · p. 2
  • A temporal smoothness function updates latent features during forward diffusion to improve coherence across frames under directional control.The method is introduced to address object inconsistency in generated videos.Source: §1 · p. 2
  • The training-based Motion Intensity Modulator (MIM) uses optical flow, and combining motion-intensity embeddings with text embeddings performs best among three explored designs.Source: §1 · p. 2
  • Mojito generates high-quality videos comparable to state-of-the-art models while providing effective and efficient motion control.The framework integrates trajectory/direction and intensity control in one text-to-video architecture.Source: §1 · p. 2

2. Related Works

Mojito follows the Stable Diffusion video-generation approach of combining spatial processing with temporal layers. For controllable generation, it contrasts with earlier video methods by using training-free attention guidance for direction and adding motion-intensity control.

2.1. Diffusion-based Text to Video Generation

  • Stable Diffusion-based video models extend the framework with temporal layers for frame consistency while inheriting spatial parameters from pretrained SD U-Net; Mojito follows this architectural approach.Source: §2.1 · p. 3

2.2. Controllable Text-to-Visual Generation

  • Controllable image generation uses text semantics and structural conditions such as attention maps and bounding boxes; video control adds the complexity of motion.Source: §2.2 · p. 3
  • Earlier video-control methods use training-based trajectory control or designed diffusion-process masking, whereas Mojito uses training-free attention maps for directional control and also provides motion-intensity control.Source: §2.2 · p. 3

3. Preliminaries

The preliminaries describe latent video diffusion as extending image latent diffusion with temporal dynamics and a noise-prediction objective. They define motion direction, intensity, and trajectories in terms of object positions across frames.

3. Preliminaries

  • Latent video diffusion extends latent image diffusion by incorporating temporal dynamics, with an autoencoder encoding images and a denoising network operating in latent space.The denoising network transforms noise into a coherent image representation.Source: §3 · p. 3
  • The denoising network is trained to predict added noise, while the latent state follows a Markov process with Gaussian noise added progressively from the initial state.The condition is supplied through cross-attention, and β_t controls noise strength at step t.Source: §3 · pp. 3–4
  • Video motion has two components: direction, derived from trajectories or bounding boxes, and intensity, reflecting movement speed and amplitude.Source: §3 · p. 4
  • A motion trajectory is a sequence of object positions (x_i, y_i) across L frames, with x and y bounded by the frame width W and height H.Source: §3 · p. 4

4. The Mojito Video Generation Framework

Mojito combines trajectory guidance and motion-intensity conditioning in a video-generation framework, using training-free directional control alongside a learned intensity pathway.

4.1. Methodology Overview

  • Mojito’s two motion-control modules coordinate directional guidance and motion intensity with generated video content.The framework introduces training-free Directional Motion Control (DMC) and the Motion Intensity Modulator (MIM).Source: §4.1 · p. 4

4.2. Training-Free Directional Motion Control

  • DMC steers object trajectories at inference by modifying spatial-transformer cross-attention and aligning object-token attention with frame-specific bounding boxes.The method alternates gradient updates with denoising; a temporal smoothness function penalizes large attention changes between consecutive frames.Source: §4.2 · pp. 4–5

4.3. Motion Intensity Modulator

  • During training, MIM conditions diffusion on optical-flow intensity levels 1–10 encoded as text-like embeddings; inference combines motion and text embeddings.Optical-flow magnitude is normalized and discretized, and the intensity representation is compatible with CLIP’s text encoder.Source: §4.3 · p. 5

4.4. Model Backbone

  • Mojito’s Conv-Spatial-Temporal Transformer uses temporal transformers to capture dependencies across frames and spatial transformers to model within-frame relationships.DMC operates within the Spatial Transformer, whose cross-attention aligns generated content with external context.Source: §4.4 · p. 6

5. Experiments

The experiments evaluate Mojito’s data, video quality, motion alignment, and implementation, then examine directional-control behavior and alternative intensity-modulation designs. They report qualitative trajectory control and identify design tradeoffs for temporal smoothness, guidance, and MIM conditioning.

5.1. Datasets

  • The evaluation uses Panda-70M, InternVID, and Mira for training, with a 10M-video Panda-70M subset and 1,000 Panda-70M videos for motion-intensity evaluation; LaSOT supports directional evaluation.Directional alignment is measured using bounding-box IoU and normalized centroid distance; intensity alignment compares generated and target average optical flow.Source: §5.1 · p. 6, §5.2 · p. 6

5.2. Evaluation Metrics; 5.3. Implementation Details

  • Video quality is evaluated with FVD on 300 videos of 16 frames and CLIPSIM, while implementation uses a VAE-based latent U-Net with trajectory adjustments in its mid-block and initial upsampling layers.The reported training setup uses DeepSpeed, FusedLamb, batch size 4, and a base learning rate of 1.0 × 10−5.Source: §5.2 · p. 6, §5.3 · p. 6

5.6.2. ALTERNATIVE CHOICES FOR MOTION INTENSITY MODULATOR

  • Combining motion-intensity embeddings with text embeddings achieves the best overall balance of video quality, semantic similarity, and motion alignment among the evaluated MIM designs.Global conditional input shows good motion alignment but significantly degrades video quality.Source: Table 3 · p. 7, §5.6.2 · p. 8

5.4. Performance Evaluation

  • Mojito dynamically guides named objects along user-defined paths at inference without extra training, unlike Tora, and its DMC module also works with DiT-based models.The human evaluation compares Mojito with OpenSora, OpenSora plan, and VideoCrafter2 across 2,400 comparisons on 400 video pairs, with five ratings per pair.Source: §5.4 · p. 7, Table 1 · p. 7

5.5. Visualization of Attention Maps

  • Attention maps for the target red ball begin with random patterns and progressively align with the frame-specific bounding boxes as denoising steps advance.The paper presents this refinement as supporting adherence to the user-defined trajectory.Source: §5.5 · pp. 7–8

5.6.1. DESIGNS IN DIRECTIONAL MOTION CONTROL

  • Without temporal smoothness (λ = 0), the sailboat video shows changing views and inconsistent appearances across frames.The guidance-strength sweep finds that η = 100 aligns the trajectory well while maintaining good visual quality, whereas η = 10000 produces severe quality degradation and square-pattern artifacts.Source: Figure 5 · p. 7, §5.6.1 · p. 8
  • With guidance steps increased from t = 1 to t = 10, trajectory alignment generally improves without significantly affecting video quality; at t ≥30, video quality deteriorates substantially.Figure 6 provides qualitative examples of the effect.Source: Figure 6 · p. 7, Table 2 · p. 7, §5.6.1 · p. 8

6. Conclusion

Mojito combines trajectory and intensity control in text-to-video generation through DMC and MIM. The conclusion reports high-quality controlled videos, efficient training, and generalizability while distinguishing the modules’ training requirements.

6. Conclusion

  • Mojito produces high-quality motion-controlled videos and is reported to be generalizable and orthogonal to existing approaches.The conclusion characterizes the framework as enabling integrated control over motion trajectory or direction and intensity.Source: §6 · p. 8
  • DMC is training-free, while MIM does not require additional motion-data annotations, supporting the paper’s claim that Mojito can be trained efficiently.Source: §6 · p. 8
  • Mojito specifies object trajectories and motion intensity during inference and explicitly models motion intensity in text-to-video generation.Source: §6 · p. 8

Impact Statement

The impact statement flags possible misuse and bias concerns for generated videos. The architecture description identifies a VAE-based latent diffusion U-Net with spatial and temporal transformers and motion-conditioned cross-attention.

Impact Statement

  • Generated videos may be misused for misleading or harmful content, and outputs raise fairness and bias concerns.The statement names watermarking, harmful-video filters, and training-data bias mitigation as possible safeguards.Source: Impact Statement · p. 8

A. Model Architecture

  • Mojito’s VAE-based latent diffusion framework places spatial and temporal transformers within a U-Net to process within-frame information and temporal coherence.Source: A. Model Architecture · p. 12
  • Motion-intensity and text embeddings enter cross-attention, while convolutional layers handle temporal features such as fps and timesteps.Source: A. Model Architecture · p. 12

B. Additional Qualitative Results with Cosmos

Additional examples apply DMC to Cosmos and visualize how attention maps change during sampling. The appendix also points readers to the DMC algorithm framework.

B. Additional Qualitative Results with Cosmos

C. Visualizations on Attention Map

D. Algorithm Framework

E. Dataset

Mojito is trained for motion intensity control using Panda 70M, InternVID, and Mira. Panda 70M clips undergo semantic-consistency filtering and quality post-processing.

E. Dataset

  • Mojito is trained for motion intensity control on Panda 70M, InternVID, and Mira, using the official processing setups for InternVID and Mira.Source: E. Dataset · p. 12
  • Panda 70M videos are split into clips, with artificial cuts every 5 seconds for long, uncut footage, and clips with ImageBind feature distance above 1.0 are filtered out.Source: E. Dataset · p. 12
  • Consecutive Panda 70M clips with boundary-feature distance ≤0.6 are merged, while post-processing filters short or low-motion clips and trims clips longer than 60 seconds.Further filtering targets semantic diversity and trims the first and last 10% of each clip; the result is 70,817,169 clips averaging 8.5 seconds.Source: E. Dataset · p. 12

F.1. Training Details

Training combines a pretrained VAE with spatial and temporal transformers and the MIM module. The paper also specifies the training schedule, DMC algorithm inputs, and video and text-conditioning configuration.

F.1. Training Details

  • The Mojito backbone uses a pretrained VAE encoder with spatial and temporal transformers alongside the MIM module for spatial consistency and controlled motion.Source: §F.1 · p. 12
  • Training runs for a maximum of 5,000,000 steps with batch size 1 and gradient accumulation over 2 steps; checkpoints are saved every 1,000 steps, with additional 5,000-step saves for evaluation and best-checkpoint selection.Source: §F.1 · p. 12
  • The training-free DMC algorithm takes text, an object phrase, a user-defined trajectory, guidance strength, temporal smoothness weight, and a final guidance timestep, and returns a generated video.Its listed steps initialize a latent, iterate from T to the guidance timestep, and return the generated video.Source: §F.1 · p. 13
  • Training uses 16 nodes with 128 A100 GPUs, DeepSpeed Stage-2, 32-bit precision, a step-based learning-rate schedule, and gradient clipping at 0.5.The learning rate decays to 1% of its initial value over 20,000 steps.Source: §F.1 · p. 13

F.2. Model Architecture Configuration

  • The model processes 320 × 512 frames into 40×64 latents with four channels, uses a frozen OpenCLIP text embedder, and incorporates FPS information as additional conditioning.Source: §F.2 · p. 13

G. Human Evaluation Interface

The human-evaluation interfaces ask participants to rate perceived motion intensity and trajectory alignment. The section also presents examples and describes failure cases for directional control.

G. Human Evaluation Interface

  • Amazon Mechanical Turk participants rate perceived motion intensity on a predefined scale and assess how well generated motion aligns with specified trajectories.Source: G. Human Evaluation Interface · p. 13

H. Failure Cases

  • When bounding boxes move much faster than the frame rate, multiple objects may appear, so a second taxi can occupy a later box instead of the first taxi moving there.Source: H. Failure Cases · p. 13
  • Similar objects in a prompt, such as a sunflower and a sun, can lead directional guidance to switch focus and produce unintended transitions.Source: H. Failure Cases · p. 13
  • Additional examples show the DMC module following bounding-box guidance and trajectory constraints on the DiT-based Cosmos backbone.Source: Figure 7 · p. 14
  • The cross-attention maps for a red ball evolve from random patterns toward alignment with the bounding boxes specified for each video timestep.Source: Figure 8 · p. 15
Loading 2412.08948v2…