PaperlayerView exact revision on arXiv

Source-linked AI summary

TrajectoryMover: Generative Movement of Object Trajectories in Videos

Kiran Chhatre, Hyeonho Jeong, Yulia Gryaditskaya, Christopher E. Peters, Chun-Hao Paul Huang, Paul Guerrero

arXiv:2603.29092v3cs.CV

TL;DR

Existing methods do not provide a way to move an object's 3D trajectory while preserving its relative motion and plausible scene interactions, partly because suitable paired videos are scarce. The paper introduces TrajectoryAtlas for synthetic paired data and fine-tunes TrajectoryMover for trajectory movement, achieving stronger quantitative and qualitative performance than baselines while retaining limitations in strict long-horizon control and real-video performance.

  • Problem

    Existing video-editing methods can prescribe trajectories but lack a simple method for moving an object's 3D trajectory while preserving motion and plausible scene interactions; paired training data are scarce.

  • Method

    TrajectoryAtlas generates synthetic paired videos with moved object trajectories, and TrajectoryMover is fine-tuned as a video-to-video generator using bounding-box controls and interleaved training.

  • Results

    TrajectoryMover achieves the best results across background preservation, foreground identity, and trajectory adherence, including SSIMbg 0.92 and IoUtraj 0.27.

  • Takeaways & Limitations

    The framework enables generative movement of object trajectories without requiring users to manually specify detailed updated trajectories or interactions.

  • Takeaways & Limitations

    Strict trajectory control remains limited: on the 40-video synthetic test set, endpoint success is Re = 0.26, and real-video performance has a gap relative to synthetic videos.

Abstract

from arXiv · show

Generative video editing has enabled several intuitive editing operations for short video clips that would previously have been difficult to achieve, especially for non-expert editors. Existing methods focus on prescribing an object's 3D or 2D motion trajectory in a video, or on altering the appearance of an object or a scene, while preserving both the video's plausibility and identity. Yet a method to move an object's 3D motion trajectory in a video, i.e., moving an object while preserving its relative 3D motion, is currently still missing. The main challenge lies in obtaining paired video data for this scenario. Previous methods typically rely on clever data generation approaches to construct plausible paired data from unpaired videos, but this approach fails if one of the videos in a pair can not easily be constructed from the other. Instead, we introduce TrajectoryAtlas, a new data generation pipeline for large-scale synthetic paired video data and a video generator TrajectoryMover fine-tuned with this data. We show that this successfully enables generative movement of object trajectories. Project page: https://chhatrekiran.github.io/trajectorymover

1 Introduction

TrajectoryMover reframes trajectory editing as translating an object's motion while preserving plausible scene interactions, avoiding the need for users to specify detailed paths. TrajectoryAtlas supplies paired synthetic data for training, enabling more accurate and plausible trajectory translation than explicit-trajectory methods.

  • Motivation and task: Trajectory movement simplifies editing by avoiding manual specification of complete 2D or 3D paths and interaction-specific motions.Explicit trajectory design can be time-consuming, expertise-dependent, and difficult for scene interactions such as a basketball bouncing off a rim.
  • Motivation and task: TrajectoryMover lets users move an object's trajectory by specifying source and target bounding boxes in the first frame.The model automatically adjusts motion to preserve physically plausible scene interactions.
  • Data and model: TrajectoryAtlas addresses the lack of paired videos by synthesizing scene-consistent video pairs with one object's trajectory moved.Physics simulation and optional scene modification support diverse, plausible object trajectories and interactions.
  • Data and model: TrajectoryMover formulates trajectory movement as video-to-video generation and fine-tunes a video model with alternating real-video generation and synthetic task training.The strategy teaches the new control signal while retaining the pretrained prior for realism and physical plausibility.
  • Results: Extensive experiments show more accurate and plausible trajectory translation than methods requiring explicit trajectory specifications.The paper presents this as the principal outcome of TrajectoryMover.

3 Method

The method generates paired videos in which an object follows an offset version of its original 3D trajectory unless that offset would create implausible motion. TrajectoryMover uses bounding-box conditioning and concatenated latent streams to generate the target video.

  • Data generation: The pipeline generates instructive pairs showing the same scene and object with a moved trajectory for supervised training.Physics simulation and online scene modification help preserve plausible interactions and provide clearer trajectory-preservation examples.
  • Training strategy: Interleaved training alternates unconditional real-video generation with the synthetic trajectory-movement task to retain the generator's prior.This preserves realism and physical plausibility while teaching the new control signal.
  • Data generation: TrajectoryAtlas creates target trajectories by offsetting each source position as y_i = x_i + δ, except when the result would be implausible.The pipeline produces paired videos and binary masks for bounding-box control.
  • Data generation: TrajectoryAtlas samples scenes, camera poses, objects, initial placements, scales, and trajectory types including static, roll, drag, drop, and throw.Initial placements are checked for visibility, support, and non-intersection, while scale settings vary between close-up and regular views.
  • Model architecture: The architecture concatenates trajectory, source-video, and bounding-box latent streams before denoising.The control image marks the source box in red and target box in green.
  • Model architecture: TrajectoryMover maps a source video to a target video using an object-displacement control image containing source and target bounding boxes.The conditioning signal identifies which object to move and where to place it in the target frame.

4 Implementation Details

The implementation uses Blender and PyBullet to generate synthetic trajectory-shifted videos, then trains on resized, fixed-length sequences with bounding-box controls and null text conditions.

  • Data generation: TrajectoryAtlas uses Blender Cycles for rendering and PyBullet for physics, with curated indoor scenes and 119 foreground objects.Generated pairs enforce visibility, support, and intersection checks before simulating task-specific motion.
  • Training: Training resizes frames to 832 × 480 and limits sequences to 81 frames, yielding 21 latent frames for the Wan2.1-T2V-1.3B backbone.Bounding boxes are extracted from ground-truth masks to construct the control signal.
  • Training: The model is trained with empty text conditions, so test-time users do not need to provide text prompts.The conditioning instead uses bounding boxes indicating the object's source and target locations.

5 Results

TrajectoryMover is evaluated against repurposed video-editing baselines on synthetic and real videos using identity, preservation, trajectory-adherence, and plausibility measures. It performs strongly overall, while strict long-horizon trajectory control remains limited.

  • Quantitative Evaluation: TrajectoryMover achieves the best results across all three quantitative metrics, improving SSIMbg from 0.71 to 0.92 and DINOfg from 0.39 to 0.45 over ATI.Compared with SFM, IoUtraj improves from 0.23 to 0.27 while SSIMbg and DINOfg reach 0.92 and 0.45.
  • Qualitative Evaluation: Across complex drop, throw, and roll scenarios, TrajectoryMover more faithfully tracks intended trajectories while preserving foreground identity and background structure.The translated object remains temporally coherent with minimal texture collapse or shape drift across novel depths and changing support surfaces.
  • Qualitative Evaluation: Baseline methods exhibit distinct failure modes, including reconstruction artifacts, rigid scene-insensitive control, composition conflicts, and long-term propagation drift.SFM can fragment or duplicate objects; ATI and DaS may produce implausible scene-dependent motion; VACE and I2VEdit show composition or propagation failures.
  • User Study: TrajectoryMover ranks first in synthetic motion-plausibility analyses by both participants and InternVL, with utilities of 1.25 and 0.56, respectively.The study used 250 participant votes, while InternVL plausibility was scored over 10 runs.
  • Real Videos: On 37 real web videos, TrajectoryMover achieves the best foreground preservation, better target-start matching, and first-place InternVL ranking, despite a gap versus synthetic performance.The real-video evaluation reports DINOfg, Rs, and InternVL plausibility because full ground-truth trajectories are unavailable.
  • Limitations: On the synthetic test set, strict trajectory control remains weaker at long horizons and after contact, with Rs = 0.93, Re = 0.26, and Rc = 0.48.The reported limitations include occlusion-related tracking degradation, depth ambiguity, and difficulty with multiple simultaneously moving objects.

6 Conclusion

TrajectoryMover moves an object’s 3D motion trajectory while preserving complex spatial transformations through synthetic paired data and balanced fine-tuning. The authors identify broader motion-preservation controls as an open direction.

  • TrajectoryMover is presented as the first framework to move an object’s 3D motion trajectory in a video.
  • TrajectoryAtlas uses physics simulations and online scene modification to create plausible, diverse trajectory-shifted video pairs.
  • Balanced fine-tuning preserves the original backbone’s generative prior while learning complex spatial transformations.
  • Future work includes controls for arbitrary-frame relocation, trajectory rotation, scaling, speed changes, transfer between objects, and motion retargeting.

A Overview

The supplementary material documents baseline repurposing, TrajectoryAtlas construction, ablations, and qualitative comparisons. It also provides project-page videos covering source–target pairs, baselines, real videos, and ablation variants.

  • Supplementary Section B details baseline repurposing, shared 3D trajectory extraction, method-specific controls, and excluded baselines.
  • Supplementary Section C describes TrajectoryAtlas trajectory simulation, online scene modification, implementation settings, and dataset statistics.
  • The project webpage provides qualitative videos of ground-truth pairs, baseline and real-video comparisons, and ablation comparisons.

B Baseline Repurposing Details

Baselines are externally adapted for source-to-target relocation without changing their original architectures or training procedures. Their controls use extracted or edited trajectories, while simple transfer does not model new scene interactions.

  • Setup: Each baseline receives external controls for relocation while retaining its original model architecture and training procedure.
  • Common 3D trajectory extraction: ATI, DaS, and VACE obtain source 3D trajectories, offset the foreground motion, and convert the result into each method’s native control format.
  • Method-specific controls: ATI uses projected 2D trajectories, DaS uses edited 3D tracks, VACE uses rendered layout controls, and I2VEdit uses an edited first frame.
  • Common baseline repurposing pipeline: Trajectory overlays in the baseline pipeline show source localization, target localization, and source versus re-anchored motion elements.
  • Limitations: Simple trajectory transfer does not explicitly reason about new scene interactions after relocation, which can reduce plausibility under collisions.
  • TrajectoryAtlas generation: TrajectoryAtlas samples scenes, cameras, objects, and tasks, simulates source and target runs, generates masks, and retains samples passing canonical checks.

C.1 Object trajectory details

TrajectoryAtlas generates object motions using rigid-body simulation, procedural paths, and motion-specific initialization. Paths vary in shape, orientation, extent, curvature, speed, and endpoint placement, with special parameters for pendulum and door-swing motions.

  • Simulation: Rigid-body dynamics in Bullet generate trajectories from initial velocities, gravity, and optional elastic forces, beginning at visible, intersection-free placements.
  • Initial velocity sampling: Throw motions sample horizontal speed from a three-component Gaussian mixture, while roll motions use motion-specific velocity sampling with zero vertical velocity.
  • Procedural path sampling: Path-controlled objects follow procedurally sampled references through a damped spring controller with orientation alignment.
  • Path families: The simulator includes S-, O-, U-, C-, 8-, Z-shaped, spiral, sinusoidal, falling-leaf, pendulum, and door-swing paths.
  • Path parameters: Path motion is controlled by speed, start position, end position, orientation, extent, and curvature parameters.
  • Additional parameters: Pendulum and door-swing trajectories additionally sample length and swing angle, or radius, rotation angle, and hinge height.
  • Scene modification: Non-structural scene objects may be removed to clear clutter that would obstruct motion, while structural objects such as walls and floors remain.

C.3 Implementation details

The data-generation system uses Blender Cycles for rendering and Bullet for physics simulation, with parallel GPU-cluster execution and reusable collision caches.

  • Blender Cycles renders the synthetic data, while Bullet performs physics simulation.
  • Runs execute in parallel on a GPU cluster with one render worker per GPU.
  • Reusable collision caches support both scene and object geometry.

C.4 Dataset statistics

TrajectoryAtlas contains 33,290 paired samples built from curated indoor scenes and 119 foreground assets, with diverse trajectory templates and hit/no-hit annotations.

  • 33,290 paired samples comprise 66,580 RGB videos generated from curated Evermotion indoor scenes and 119 foreground assets.The assets include 98 Objaverse objects and 21 primitives with Bullet proxy variants.
  • The dataset includes straight, C-path, leaf-fall, figure-8, wiggly-line, U-path, stick-slip-slide, hover-orbit, and zigzag trajectory pairs.
  • Drag templates contain 3,151 O, 3,184 S, 3,202 spiral, and 1,078 point-to-point drag pairs.
  • Hit annotation includes 13,462 hit pairs and 16,974 no-hit pairs.

D Qualitative Ablation Examples

Qualitative ablations show that the full training recipe best balances trajectory fidelity, identity preservation, and scene-aware motion across varied interactions.

  • The full model gives the strongest combined performance in trajectory fidelity, identity preservation, and scene interaction realism.
  • Using only primitives most clearly degrades identity through blob-like shapes, weaker texture, and lower temporal consistency in difficult frames.
  • Only scene modification improves obstacle avoidance in no-hit settings but reduces placement accuracy in complex scenes.
  • Removing scene modification improves some hit interactions but weakens long-range unobstructed motion and increases trajectory misalignment after changed initialization.
  • Drop-only training transfers poorly to roll and drag motions because its motion coverage is too narrow.
  • The qualitative results support mixed object sources, mixed hit and no-hit data, and diverse motion tasks.

E User Study Interface

The user-study interface compares anonymized A/B videos from the same source-target setup using standardized criteria for motion and scene plausibility.

  • Participants compare two anonymized videos from the same source-target setup and select the more natural and coherent object motion.
  • The guidance text standardizes evaluation around motion stability, object consistency, trajectory reasonableness, and scene interaction plausibility.
  • The same interface format is used across all user-study experiments in the project.
  • The qualitative ablation figure compares the full model with primitive-only, scene-modification, no-scene-modification, and drop-only variants.
Loading 2603.29092v3…