MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

1. Introduction

• Text-based motion control limits precision and expressiveness, while camera and object motion can be ambiguous when users specify them together.Dragging a character’s arm might indicate camera panning, character movement, or arm movement.Source: §1 · p. 2
• MotionCanvas represents motion using 2D bounding boxes and point trajectories, avoiding reliance on costly, labor-intensive video labels such as camera parameters and object motion.These signals can be automatically and reliably estimated, allowing training on a wider variety and quantity of videos.Source: §1 · p. 2
• Users plan motion in 3D scene space, but video models use 2D screen-space conditions, so their intentions must be translated into 2D control signals.The Motion Signal Translation module performs this translation before the signals reach the video generation model.Source: §1 · p. 2
• MotionCanvas jointly controls camera and object motion in a scene-aware manner, translating users’ motion designs into screen-space signals for video diffusion models.Its Motion Signal Translation module decomposes motion into hierarchical transformations and can rectify potentially erroneous user input.Source: §1 · p. 2

2. Related Work

• Earlier image-animation methods use reconstructed geometry or estimated depth for novel views, but are limited to camera effects on static photographs.Cinemagraph methods animate selected image regions, often with subtle motion restricted to specific elements such as clouds or smoke.Source: §2 · pp. 2–3
• Image-to-video models generate realistic videos from a single image and text prompt, but typically provide limited or no control over generated motion.This limits their applicability to user-directed motion design.Source: §2 · p. 3
• Bounding-box methods control object motion without camera control, making it ambiguous whether the object or the whole scene should move.Point-trajectory methods offer drag-based control but lack 3D-aware object motion and are often restricted to few trajectories.Source: §2 · p. 3
• Camera-control methods use explicit 3D camera parameters, but their training data is largely restricted to static scenes and curated domains.The passage says these methods consequently support few object motions and lack generalizability.Source: §2 · p. 3
• MotionCanvas combines object-level and camera-level control, using 2D signals for training while enabling 3D-aware controls through depth-based synthesis.The related-work passage characterizes prior methods as lacking this combination of scene-aware object control and camera control.Source: §2 · p. 3

3.1. Motion Design Module – Capturing User Intents

• Users specify 3D camera paths, scene-anchored object boxes, local point traces, and timelines for coordinating camera and object motion.Camera poses can be interpolated from key moments or assembled from base patterns; box trajectories are interpolated from key boxes.Source: §3.1 · pp. 3–4

3.2. Motion Signal Translation Module

• Camera movement is translated into screen-space point tracks sampled from static background regions and warped using depth and the designed camera path.Likely moving-object regions are excluded using a YOLOv11-generated mask.Source: §3.2 · p. 4
• Scene-anchored bounding boxes are lifted to 2.5D and reprojected using camera pose and depth, including reference-depth or perspective-consistency choices for later boxes.The initial box uses average depth within its semantic mask.Source: §3.2 · pp. 4–5
• Local point trajectories are translated with camera and global-motion transforms, assuming local motion causes negligible depth change relative to global motion.Under this assumption, local points retain their initial depth.Source: §3.2 · p. 5

3.3. Motion-conditioned Video Generation

• A pretrained DiT-based image-to-video model is fine-tuned with screen-space motion conditions, alongside image and text conditioning, to generate video latents.The conditions include point trajectories and bounding boxes; generated latents are decoded into RGB frames.Source: §3.3 · p. 5

3.4. Generating Variable-length Videos via Autoregression

• MotionCanvasAR uses overlapping 16-frame clips as conditioning context for autoregressive generation, supporting arbitrary video length with independent motion control at each step.Training-free autoregressive generation from a single image often produces noticeable motion discontinuities; the clip context is used to address them, and motion signals are recomputed using user intent and back-traced motions.Source: §3.4 · p. 6

4.1. Shot Design with Joint Camera and Object Control

• Users can independently vary object motion while keeping a camera path fixed, or vary camera motion while preserving scene-anchored object behavior.The examples include keeping a car still or moving it forward or backward as the camera changes.Source: §4.1 · p. 7, Figure 5 · p. 7
• A specification-generation loop combines keyframes and interpolation to create smooth complex trajectories and generate extended videos in autoregressive chunks.The examples hold camera controls constant across two videos while changing the object controls.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2. Object Local Motion Control

• Users can draw object-local point trajectories that are transformed into screen-space controls accounting for camera and global object motion.This coordination supports consistent local motion under different camera and object dynamics.Source: §4.2 · p. 7, Figure 6 · p. 8

4.3. Additional Applications: Simple Video-based Editing

• The framework transfers object and camera motion from source videos to similar input images, and propagates extracted motion into edited images.The examples include transferring an apple’s rotation to a lion’s head and propagating edits to an initial frame through the resulting video.Source: §4.3 · pp. 7–8, Figure 7 · p. 8

5.1. Implementation Details

• Training used approximately 600K videos with motion annotations; the video-generation module was fine-tuned from a pretrained image-to-video DiT model.The reported setup includes 100K training steps, 32- and 64-frame videos, and 640×352 resolution.Source: §5.1 · pp. 8–9

5.2. Camera Motion Control Quality

• On RealEstate10K test set (1K), the method outperforms the compared methods across all metrics in a zero-shot setting, and follows intended camera motions better, especially complex Dolly-Zoom shots.Source: Table 1 · p. 9, Figure 8 · p. 9, §5.2 · p. 10

5.3. 3D-Aware Object Motion Control Quality

• MotionCanvas outperformed DragAnything, MOFA-Video, and TrackDiffusion in both object-control accuracy (ObjMC) and frame quality (FID).The authors attribute precise position, size, and pose control to spatiotemporal bounding-box conditioning.Source: Table 2 · p. 9, Figure 9 · p. 9, §5.3 · p. 10

5.4. Joint Camera and Object Control

• MotionCanvas outperformed DragAnything and MOFA-Video across motion adherence, motion quality, and frame fidelity in a study of 35 participants.The visual comparison reports that the baselines fail to capture complex global, local, and camera motion jointly, while MotionCanvas follows all three control types.Source: Table 3 · p. 10, §5.4 · p. 10, Figure 10 · p. 10

5.5. Ablation Studies

• The ablations report that DCT trajectory coding uses few coefficient tokens while supporting robust intrinsic and extrinsic camera control; the spatiotemporal color-coding map outperforms coordinate-token box conditioning.The authors note that Euclidean coordinate tokens are difficult to fuse with visual tokens, leading to low ObjMC.Source: Table 4 · p. 10, §5.5 · pp. 10–11, Figure 11 · p. 11

6. Conclusion

• MotionCanvas converts intuitive 3D motion plans into 2D control signals without relying on 3D annotations, broadening the available training-data sources.The conclusion describes flexible joint control of camera and object motion for cinematic shot design.Source: §6 · p. 11
• Comprehensive evaluations showed MotionCanvas generating diverse, high-quality videos that faithfully reflect user motion intent.Source: §6 · p. 11

A. Additional Details

• MotionCanvas training used 16 nodes of NVIDIA H100 (80GB) GPUs, with 8 GPUs per node; on one H100 GPU, generating a 32-frame 352×640 video clip with 50 denoising steps took approximately 32 seconds.Source: A. Additional Details · p. 14

B. User Interface

• Camera controls combine base motion patterns with configurable direction and speed.The listed patterns include trucking, panning, dolly, pedestal, tilt, roll, zoom, orbit, circle, and static.Source: B. User Interface · p. 14
• For object-global motion, users draw bounding-box trajectories and may assign reference depth points or use the scene-space sequence directly.The interface interpolates between boxes; scene-space boxes are noted as particularly effective for follow shots and dolly shots.Source: B. User Interface · pp. 14–15, Figure 12 · p. 15
• Users draw local point trajectories that are transformed into screen space according to camera and object-global motion, and can preview controls in both spaces.The global-motion transformation applies when a trajectory’s starting point lies within the object’s semantic region.Source: B. User Interface · p. 15

C. Essentiality of Camera-aware and Camera-object-aware Transformations

• The hierarchical Motion Signal Translation module converts scene-space user intentions into screen-space signals while accounting for intertwined camera and object motions.Figure 13 distinguishes scene-space inputs from the transformed screen-space signals.Source: C. Essentiality of Camera-aware and… · pp. 15–16, Figure 13 · p. 16
• Camera-aware transformation calibrates screen-space bounding boxes so people running forward move naturally right on screen during a trucking-left camera motion.The paper reports that the resulting animation appears accurate and more natural.Source: C. Essentiality of Camera-aware and… · p. 16
• Without camera-object-aware transformation, local-motion videos can look unnatural and incorrect; the transformation produces more life-like and accurate video.Mapping a local trajectory such as putting down hands accounts for both the object's global movement and the camera's pedestal-up motion.Source: C. Essentiality of Camera-aware and… · p. 16

D. More Details of MotionCanvasAR

• MotionCanvasAR recomputes screen-space signals by combining user intent with back-traced motion to support smoother, more consistent generation.The model uses 16 conditioned frames to generate 64 frames autoregressively and is fine-tuned for 120K iterations from a 32-frame motion-conditioned model.Source: D. More Details of MotionCanvasAR · p. 16

• The study used 15 representative Internet image cases, shuffled method results under the same input conditions, and standardized clips to 14 frames at 8 FPS.The resulting videos were approximately two seconds long.Source: E. User Study · p. 16
• The study was expected to take 7–15 minutes based on 15 cases, three questions per case, and 10–20 seconds per judgment.Source: E. User Study · p. 16
• Participants judged motion adherence, motion/dynamic quality, and frame fidelity; 35 valid responses remained after excluding comparisons completed within three minutes.Each participant evaluated 15 groups of video comparisons.Source: E. User Study · p. 17

F.1. Effect of Point Track Density on Camera Motion Control

• Higher-density point tracks improve adherence to orbit camera motion; low-density tracks leave motion ambiguous and can produce trucking-left motion instead.Source: §F.1 · p. 17

F.2. Effect of Text Prompt

• More detailed text prompts do not significantly affect camera motion control but can add dynamics such as raining or turning around.The experiments use simple text descriptions.Source: §F.2 · p. 17

G. Limitations and Future Work

• MotionCanvas provides holistic motion controls for cinematic shot design from a single image.Source: G. Limitations and Future Work · p. 17
• Video diffusion synthesis takes approximately 35 seconds for a 2-second video, currently limiting real-time applications.The authors identify more efficient generative models as a direction for future work.Source: G. Limitations and Future Work · p. 17
• The local-motion approximation assumes each object lies on a frontal parallel depth plane, which may not suit extreme close-up or macro shots with substantial within-object depth variation.Source: G. Limitations and Future Work · p. 17
• Motion design and text prompts are not explicitly harmonized, so their motion signals can conflict.The authors suggest explicit motion-aware prompt harmonization as future work.Source: G. Limitations and Future Work · p. 17

H. Camera Motion Legend

• Figure 15 presents the camera-motion legend, while Figure 16 shows the designed user-study interface.Source: H. Camera Motion Legend · p. 17, Figure 16 · p. 18