Click to Move: Controlling Video Generation with Sparse Motion

• Prior motion-conditioned video methods mainly address single-object scenes, where disentangling and controlling each object’s movement is less complex.Source: §1 · p. 1
• C2M lets users select objects in complex scenes and specify their desired 2D locations, conditioning a generated sequence on the initial frame and its segmentation map.Source: §1 · p. 1
• Its appearance, motion, and generation modules use image features and user inputs to infer object displacements and produce a synthesized frame sequence.Source: §1 · pp. 1–2
• C2M outperforms previous video-generation methods on two publicly available datasets, with the authors attributing the result to its GCN’s modeling of object interactions.Source: §1 · p. 2
• Unlike prior work, C2M uses semantic information to select objects and models motion across multiple frames; it also follows a two-stage flow-estimation and frame-refinement procedure inspired by [29].Source: §1 · p. 2

• Earlier controllable-video approaches use motion transfer, pose, driving vectors, or learned motion representations, but mostly generate videos of a single object.Source: §2 · p. 2
• Future-frame prediction research spans deterministic, adversarial, and variational formulations, with methods also exploring optical-flow warping, long-term prediction, and motion latents.Source: §2 · p. 2
• For multi-object settings, Wu et al. [45] combine frames, optical flow, instance maps, and semantic information, while Hao et al. use sparse trajectories but omit instance segmentation and object interactions.Source: §2 · pp. 2–3
• Segmentation-guided methods [29] estimate flow and refine warped frames; C2M builds on this approach with a GCN that models object interactions and uses sparse user motion vectors.Source: §2 · p. 3

3. Click to Move framework

• The framework encodes the initial frame, segmentation, and instance map, predicts motion from user vectors and image features, and combines appearance and motion to generate each output frame.At test time, the input includes the initial frame, instance segmentation, and user-provided 2D displacements for selected objects.Source: §3 · p. 3

3.1. Object motion estimation with GCNs

• A fully connected object graph uses region-pooled image features and user-control flags to propagate information, keeping supplied motions fixed while refining displacements for objects with unknown motion.The GCN is embedded in a VAE: training uses an encoder and decoder to reconstruct unknown motions, while test-time sampling passes latent variables through the decoder to predict object displacements.Source: §3.1 · pp. 3–4

3.2. Motion encoding

• For each time step, the motion encoder derives object locations by warping the initial location map with object displacements, and encodes additional motion variability not represented by object motion and appearance.At test time, the GCN-predicted displacements warp the initial map, and the additional latent motion code is sampled from a unit Gaussian.Source: §3.2 · p. 5

3.3. Generation module and training objectives

• The generation module warps initial-frame features using predicted optical flow, masks occluded regions, and processes the features to produce video frames.Training objectives include motion reconstruction and KL terms, forward-backward flow consistency, flow smoothness, supervised flow from FlowNet2, and video-generation losses.Source: §3.3 · p. 5

4. Experiments

• Evaluation uses Cityscapes and KITTI 360 against Sheng et al. variants, with Oracle, shifted Custom, and all-object settings; FVD, NDE, and detector accuracy measure outcomes.Custom inputs scale the ground-truth displacement by λ = 1.5; the all-object setting provides all future positions and bypasses the GCN.Source: §4 · p. 6

4.1. Ablation Study

• Without the motion estimator, the model cannot infer object positions and fails to move the object; FVD worsens from 289 to 369.The ablation removes the motion-estimation module and instead feeds the controlled-object location tensor into the appearance encoder.Source: Table 1 · p. 6, §4.1 · p. 6

4.2. Comparison with State-of-the-art

• In Oracle, NDE falls 47% on Cityscapes and 53% on KITTI 360; FVD falls 22.79% on Cityscapes but is slightly worse on KITTI 360.C2M also outperforms the compared methods on object control in both datasets.Source: Table 2 · p. 7, §4.2 · p. 7

4.1. Ablation Study

• Without inter-object edges, the controlled object still moves, but video quality worsens and other objects move inconsistently.NDE and accuracy remain similar to the full model in this ablation.Source: Table 1 · p. 6, §4.1 · p. 7
• Removing flow supervision worsens NDE and FVD, though FVD remains better than when object interactions are removed.The comparison is between the no-flow-supervision variant and the interaction-removal ablation.Source: Table 1 · p. 6, §4.1 · p. 7

4.2. Comparison with State-of-the-art

• Under Custom inputs, C2M improves object control and video quality, and achieves the best results when all objects are user-controlled.The Custom setting shifts the requested displacement away from ground truth; the all-object setting supplies every future position.Source: §4.2 · p. 7, Table 2 · p. 7
• In Cityscapes examples, C2M moves the requested pedestrian, car, and van to specified positions, while baselines leave targets static or misplaced.The figures show pedestrian and car movement across predicted frames, and van displacement at λ = 1 and λ = 1.5.Source: Figure 3 · p. 7, Figure 4 · p. 8, §4.2 · p. 8

5. Conclusions

• C2M outperforms state-of-the-art approaches and its GCN architecture allows better motion control.The authors plan to extend the approach to generate videos of variable length.Source: §5 · p. 8
• Users specify selected objects’ positions in the last frame, and the generated video starts from the supplied frame and respects those motion constraints.At test time, the inputs include the initial frame and corresponding instance segmentation maps.Source: §5 · p. 8
• The GCN uses message-passing rules to model object interactions while accounting for user inputs.The paper notes that objects in a scene are often not independent.Source: §5 · p. 8
• Generating videos of variable length is identified as future work.Source: §5 · p. 8

A. Network architectures

• The flow decoder has FlowD and OccD heads alongside shared FeatD layers.Source: A. Network architectures · p. 10
• Table 3 provides architecture details for the trajectory encoder Es and flow predictor D.Source: A. Network architectures · p. 10
• Architecture details for the other networks are available in the supplementary material’s attached code.Source: A. Network architectures · p. 10

A.1. Convergence issue

• The original update rule caused all nodes to have identical features in preliminary experiments.Source: §A.1 · p. 10
• The authors added a residual update, which helped objects converge to better features.Source: §A.1 · p. 10
• The residual update acts like a skip connection, allowing gradient information to pass through GCN updates and mitigating vanishing-gradient problems.Source: §A.1 · p. 10

B. Implementation.

• Training takes about one day for Cityscapes and two days for KITTI 360.Source: B. Implementation · p. 10
• ADAM uses a learning rate of 2e-4 for the Generation module and 1e-4 for Motion estimation.Source: B. Implementation · p. 10
• The implementation uses PyTorch 1.7.0 and PyTorch Geometric for the graph neural network, and experiments run on two Nvidia RTX 2080Ti GPUs.Source: B. Implementation · p. 10
• The architecture notation defines Z as motion-vector dimension, K as kernel size, S as stride, and P as padding.CONV, UPCONV, 3D-CONV, BN, and SKIP denote convolution, upsampling, 3D convolution, batch normalization, and skip connection.Source: Table 3 · p. 11

C. Additional qualitative results

• C2M moves the requested car with good frame quality, while comparison methods leave it static or blurry in higher-resolution Cityscapes results.The additional model uses 256×128-pixel frames without changing hyperparameters and is shown on two initial frames.Source: C. Additional qualitative results · p. 11, Figure 5 · p. 12, Figure 6 · p. 13
• Additional KITTI360 figures show predictions at t + 1, t + 3, and t + 5, with the moved object's initial position highlighted in red.The figures provide additional visual results; the supplied text does not report a specific outcome for them.Source: C. Additional qualitative results · p. 11, Figure 8 · p. 15, Figure 9 · p. 15
• Without object-interaction modeling, the highlighted object still moves, but other objects have unrealistic motion.This qualitative example supports the reported quantitative finding that interactions matter for temporally consistent movement.Source: C. Additional qualitative results · p. 11, Figure 10 · p. 16