PaperlayerView exact revision on arXiv

Source-linked AI summary

LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis

Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Qifeng Chen, Yujun Shen, Limin Wang

arXiv:2412.15214v2cs.CV

TL;DR

Two-dimensional trajectory inputs can be ambiguous for out-of-plane motion because one image path may represent multiple 3D trajectories. LeviTor adds depth to clustered object-mask control points and uses them to fine-tune a video-generation model. The authors report better quantitative and qualitative performance than existing approaches, with accurate 3D trajectory control where previous baselines fail.

  • Problem

    A 2D path can correspond to multiple 3D trajectories, making it ambiguous for precise object-motion control.

    Source: §1 · p. 2

  • Method

    LeviTor fine-tunes a pretrained video-generation model using control signals that combine depth information with K-means-clustered object-mask points.

    Source: §1 · p. 2

  • Results

    LeviTor demonstrates superior quantitative and qualitative performance to existing approaches and achieves accurate 3D trajectory control where previous baselines fail.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach can control object occlusion and complex movements such as orbiting, which cannot be generated with only 2D trajectories.

    Source: §4.2 · p. 7

    LeviTor is constrained by SAM segmentation and user-provided trajectories, and does not understand physical laws to generate motion without trajectory controls.

    Source: §5 · p. 8

Abstract

from arXiv · show

The intuitive nature of drag-based interaction has led to its growing adoption for controlling object trajectories in image-to-video synthesis. Still, existing methods that perform dragging in the 2D space usually face ambiguity when handling out-of-plane movements. In this work, we augment the interaction with a new dimension, i.e., the depth dimension, such that users are allowed to assign a relative depth for each point on the trajectory. That way, our new interaction paradigm not only inherits the convenience from 2D dragging, but facilitates trajectory control in the 3D space, broadening the scope of creativity. We propose a pioneering method for 3D trajectory control in image-to-video synthesis by abstracting object masks into a few cluster points. These points, accompanied by the depth information and the instance information, are finally fed into a video diffusion model as the control signal. Extensive experiments validate the effectiveness of our approach, dubbed LeviTor, in precisely manipulating the object movements when producing photo-realistic videos from static images. Our code is available at: https://github.com/ant-research/LeviTor.

1. Introduction

Image-plane trajectories cannot uniquely specify 3D motion, while extracting and entering valid 3D paths is difficult. LeviTor adds depth to clustered object-mask points and offers an interactive way to control those paths.

1. Introduction

  • Trajectory control in video generation has applications in graphics, virtual reality, and interactive media.The paper cites character animation, physical simulation, and visual effects as example uses.Source: §1 · pp. 1–2
  • A 2D trajectory can correspond to multiple 3D paths, leaving front-versus-behind motion ambiguous and making precise control difficult.Accurate 3D trajectory extraction is also difficult in scenes with occlusions or complex object interactions.Source: §1 · p. 2
  • LeviTor fine-tunes pretrained video-generation models with a 3D trajectory control signal combining depth information and K-means-clustered object-mask points, encoding trajectory attributes and occlusion/depth changes without explicit 3D trajectory estimation.Source: §1 · p. 2
  • Users draw trajectories on 2D images and adjust point depths interactively, which the system interprets as 3D object paths.The paper describes this inference pipeline as accessible without extensive 3D modeling or animation expertise.Source: §1 · p. 2
  • LeviTor reports superior quantitative and qualitative performance over existing approaches, with accurate 3D trajectory control where previous baselines fail.The authors identify use of the SA-V dataset for training and describe the method as the first 3D object trajectory control approach for image-to-video synthesis.Source: §1 · p. 2

2. Related Work

Video diffusion models generate clips from text or image prompts, while some precise-control methods use dense per-frame signals. Trajectory-based control offers a simpler alternative, but image-space trajectories remain ambiguous in 3D scenes.

2.1. Video Diffusion Models

  • Video diffusion models include text-to-video and image-to-video frameworks, with image-to-video animating a still image into a clip that retains its visual content.The passage names SVD, VideoCrafter2, and CogVideoX as models supporting both frameworks.Source: §2.1 · p. 2
  • Some video-generation methods use pose, depth, or sound conditions for precise control, but require dense per-frame signals that are cumbersome in real-world applications.Trajectory-based control is presented as an alternative combining simplicity with precision.Source: §2.1 · pp. 2–3

2.2. Trajectory Control in Video Generation

  • Prior trajectory-control methods use recurrent networks, optical flow, bounding boxes, vector maps, sparse strokes, or latent trajectory embeddings.DragAnything and TrackGo add entity-level representations, but still use 2D image-space trajectories that are ambiguous in 3D scenes.Source: §2.2 · p. 3

3. Method

LeviTor builds 3D trajectory controls from object masks, estimated depth, and user input. Its training and inference pipelines turn these controls into video-generation signals while representing object movement, occlusion, and size changes.

3.1. Problem Formulation

  • LeviTor derives approximate 3D trajectories by clustering object-mask pixels and sampling estimated relative depth at each control point.The resulting controls avoid explicit 3D annotations; users can select points on a 2D image and adjust their depths.Source: §3.1 · p. 3

3.2. Training Pipeline

  • Training assigns each object a mask-area-based number of K-means points, then uses depth, instance information, and trajectory heatmaps as ControlNet signals for Stable Video Diffusion.When the maximum-to-minimum mask-area ratio exceeds 10, k is at least 3; k is capped at 8.Source: §3.2 · p. 4

3.3. Inference Pipeline

  • Inference shifts selected mask pixels in 3D and renders them back to 2D masks, from which K-means derives multi-point controls for video synthesis.The pipeline assumes fixed camera intrinsics and a stationary camera; using moderate numbers of trajectory controls helps represent motion variation and non-rigid detail, unlike direct dense-mask control.Source: §3.3 · pp. 4–5

4. Experiments

Experiments compare LeviTor with DragNUWA and DragAnything on DAVIS and examine depth-dependent motion, occlusion, and conditioning ablations. The authors report FID/FVD advantages over baselines, while identifying an ObjMC trade-off and a balance between motion amplitude and generation quality.

4.1. Experiment Settings

  • LeviTor trains on SA-V with 16-frame 288×512 clips and is tested on 16-frame DAVIS clips, using FVD/FID for quality and ObjMC for controllability.The training setup uses SVD as its base model.Source: §4.1 · pp. 5–6

4.2. Comparison with Other Approaches

  • LeviTor controls tornado occlusion and depth-driven size changes, planet motion relative to the lens, and a bowl orbiting a vase; the 2D-only baselines misinterpret or blur these motions.The comparisons use the same 2D control trajectory across models.Source: §4.2 · p. 7
  • On DAVIS, LeviTor has a significant advantage over baselines in FID and FVD, but performs worse than DragNUWA on ObjMC.The authors attribute the ObjMC result to not using tracking methods to obtain complete point trajectories and requiring generated videos to match them perfectly.Source: Table 1 · p. 7, §4.2 · p. 7

4.3. Ablation Studies

  • Removing either depth or instance information can confuse occlusion and blur results; instance information matters more because it identifies which object each control point belongs to.The authors note that depth is partly implicit in point clustering, reducing its relative impact.Source: §4.3 · p. 8, Table 2 · p. 8
  • Few inference control points increase motion amplitude but risk deformation or blur, whereas too many can preserve shape yet prevent the desired movement.Users can adjust the point count to suit the desired generation result.Source: §4.3 · p. 8, Figure 8 · p. 8

5. Conclusion

LeviTor uses depth information and K-means clustered points as control signals for 3D object trajectories without explicit 3D trajectory estimation. Its inference pipeline lets users draw trajectories in 2D and adjust point depths, while generation remains bounded by segmentation, user-provided controls, tracking-data absence, and the base model.

5. Conclusion

  • LeviTor combines depth information with K-means clustered points to control 3D object trajectories without explicit 3D trajectory estimation.Users draw trajectories on 2D images and adjust point depths in the inference pipeline.Source: §5 · p. 8
  • The method is constrained by SAM segmentation and user-provided trajectories, and does not generate object movements without trajectory controls or understand physical laws.It also cannot control internal pose changes because it was not trained with tracking data.Source: §5 · p. 8
  • Current generation results are limited by the base model SVD; future work aims to use more advanced models for deformable objects and intricate dynamics.Source: §5 · p. 8

A. Comparison with more methods

The qualitative comparison reports that SG-I2V and MOFA-Video fail to follow complex trajectories or produce proper depth variation. Figure S1 presents the comparison with both methods.

A. Comparison with more methods

B. More Ablations on the Number of Control Points for Inference

The inference ablation compares LeviTor with its default control-point count and a denser configuration. The reported examples show that overly dense controls restrict non-rigid motion, supporting flexible adjustment of clustered control points.

B. More Ablations on the Number of Control Points for Inference

C. Comparison with Single-point Control

The single-point alternative uses an object-mask center point with depth changes, but the reported comparison finds it unable to represent 3D motion well. The authors identify clustering and dispersion of multiple control points as central to their 3D motion representation.

C. Comparison with Single-point Control

D. Bad Case Analysis

LeviTor’s bad cases include difficulties with small faces, large motions, and similar object parts. The authors attribute these artifacts to the underlying Stable Video Diffusion model and plan to integrate more advanced video models.

D. Bad Case Analysis

  • LeviTor struggles to reconstruct small faces and generate scenes with large motions, and it may confuse similar object parts.Examples include blurry horse faces and unnatural legs, unnatural running feet, and an elephant leg changing from front to back before regenerating.Source: D. Bad Case Analysis · p. 10
  • The authors attribute these artifacts to Stable Video Diffusion’s difficulty reconstructing small faces and tendency to produce artifacts with large-scale movements.They also note unnatural movement in a person’s feet while running.Source: D. Bad Case Analysis · p. 10
  • The authors plan to integrate more advanced video models to better capture deformable objects and complex dynamics, including large-scale and non-rigid motions.Source: D. Bad Case Analysis · p. 10
Loading 2412.15214v2…