PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising

Assaf Singer, Noam Rotstein, Amir Mann, Ron Kimmel, Or Litany

arXiv:2511.08633v1cs.CVcs.AIcs.GRcs.LGcs.MM

TL;DR

Precise motion control remains limited in video generation, and many motion-conditioned approaches require costly, model-specific tuning. Time-to-Move turns crude animated references into guidance for image-conditioned video diffusion using region-dependent denoising. It performs strongly on object and camera motion benchmarks, including against training-based baselines.

  • Problem

    Video generation remains difficult to control precisely: prompt-driven motion is often unreliable and coarse, while many motion-conditioned methods require costly, model-specific fine-tuning.

    Source: §1 · p. 2

  • Method

    TTM uses crude animated references as motion guidance for image-conditioned video diffusion, combining region-dependent dual-clock denoising with full-reference conditioning for motion and appearance control.

    Source: §1 · pp. 2–3

  • Results

    TTM consistently ranks among the best methods on object and camera motion benchmarks, outperforming training-based baselines while matching standard generation speed.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Full-reference conditioning supports simultaneous motion and visual-attribute control, such as changing an object's color along a specified trajectory.

    Source: §1 · p. 2

    Identity preservation is limited to content visible in the first frame, and the dual-clock scheme still requires tuning of (tweak, tstrong).

    Source: §5 · p. 9

Abstract

from arXiv · show

Diffusion-based video generation can create realistic videos, yet existing image- and text-based conditioning fails to offer precise motion control. Prior methods for motion-conditioned synthesis typically require model-specific fine-tuning, which is computationally expensive and restrictive. We introduce Time-to-Move (TTM), a training-free, plug-and-play framework for motion- and appearance-controlled video generation with image-to-video (I2V) diffusion models. Our key insight is to use crude reference animations obtained through user-friendly manipulations such as cut-and-drag or depth-based reprojection. Motivated by SDEdit's use of coarse layout cues for image editing, we treat the crude animations as coarse motion cues and adapt the mechanism to the video domain. We preserve appearance with image conditioning and introduce dual-clock denoising, a region-dependent strategy that enforces strong alignment in motion-specified regions while allowing flexibility elsewhere, balancing fidelity to user intent with natural dynamics. This lightweight modification of the sampling process incurs no additional training or runtime cost and is compatible with any backbone. Extensive experiments on object and camera motion benchmarks show that TTM matches or exceeds existing training-based baselines in realism and motion control. Beyond this, TTM introduces a unique capability: precise appearance control through pixel-level conditioning, exceeding the limits of text-only prompting. Visit our project page for video examples and code: https://time-to-move.github.io/.

1 INTRODUCTION

Fine-grained motion control remains limited in video generation, while existing motion-conditioned methods often require costly, model-specific fine-tuning. Time-to-Move uses crude animated references, image conditioning, and region-dependent denoising to control motion and appearance, and performs strongly on object and camera motion benchmarks.

1 INTRODUCTION

  • Existing motion controls are often unreliable and coarse, while learned control methods require costly fine-tuning and may compromise model quality or require architecture changes.The goal is precise local motion with realistic, temporally coherent output while preserving the input image.Source: §1 · p. 2
  • Conditioning on full reference frames enables joint control of motion and appearance, including attributes such as color, shape, and style.The paper illustrates changing an object's color while animating it along a specified trajectory.Source: §1 · pp. 2–3
  • TTM converts crude animations, such as cut-and-drag edits or depth-based reprojections, into motion guidance using SDEdit-style noise injection and image conditioning.The crude references provide coarse motion structure, while image conditioning preserves appearance.Source: §1 · p. 2
  • Dual-clock denoising applies strong alignment in motion-specified regions and weaker alignment elsewhere, allowing unspecified areas to develop plausible dynamics without retraining.The region-dependent strategy uses two distinct noise schedules.Source: §1 · p. 2
  • TTM consistently ranks among the best methods on object and camera motion benchmarks, outperforming training-based baselines while matching standard generation speed.The method is validated across three I2V backbones.Source: §1 · p. 3

2 RELATED WORK

Prior work ranges from motion controls learned into model architectures to training-free approaches that manipulate attention, noise, or reference content. TTM is presented as an image-conditioned pipeline that uses a mask and motion signal to vary denoising across regions.

2 RELATED WORK

  • Video generation has advanced in visual quality and temporal coherence, but prompt-only or single-frame conditioning still offers limited control over accurate motion.Source: §2 · p. 3
  • Many learned motion-control methods inject trajectories through network modules or features, while other approaches use explicit motion cues; Go-with-the-Flow shares an architecture-agnostic goal but extensively fine-tunes its base model.Examples include multi-scale trajectory fusion, LoRA modules, motion-patch tokens, auxiliary branches, and warped diffusion noise.Source: §2 · p. 3
  • Training-free T2V methods manipulate attention or noise, but their text-linked boxes can limit part-level motion, precise appearance control, and in-place animation of a given image.Other motion-transfer methods rely on a suitable driving video, while several I2V approaches retain backbone-specific or design-specific limitations.Source: §2 · p. 3
  • Prior heterogeneous-denoising work includes methods adapted with LoRA or trained with per-pixel or per-token noise levels, whereas TTM imposes region-specific schedules at inference without training.RePaint is training-free but denoises only the mask and repeatedly re-noises unmasked regions; TTM denoises the whole image without those resampling loops.Source: §2 · pp. 3–4
  • The illustrated TTM pipeline combines an input image, a mask-marked motion region, and a generated motion signal, then denoises masked and unmasked areas from different noise levels.The masked region starts at lower noise to enforce specified motion; the background starts at higher noise for natural deviations.Source: Figure 2 · p. 4

3 METHOD

TTM guides image-to-video diffusion with a coarse warped reference and a mask that identifies where motion is specified. It injects motion through SDEdit-style noising, preserves appearance with image conditioning, and uses region-dependent denoising to balance adherence and natural dynamics.

3 METHOD

  • The method takes an input image, a coarse warped reference video, and a binary mask, and aims to generate a realistic video that preserves image fidelity while following prescribed motion.The mask identifies, frame by frame, where the reference supplies motion guidance.Source: §3 · p. 4

3.1 MOTION SIGNAL

  • Users select a region and drag it along a trajectory to create a piecewise-smooth displacement field and warped reference, with identity mapping outside the mask.Disocclusion holes are filled by nearest-neighbor inpainting; depth-based pixel warping, rotation, scaling, and appearance changes can also construct references.Source: §3.1 · pp. 4–5

3.2 SDEDIT ADAPTATION FOR MOTION INJECTION

  • TTM treats the warped video as an explicit motion instruction and noises it to an early denoising timestep, while conditioning on the clean first frame preserves identity and appearance.Noising alone in a text-conditioned model would lose input-image fidelity because the model sees appearance only through the noisy reference.Source: §3.2 · p. 5

3.3 REGION-DEPENDENT DUAL-CLOCK DENOISING

  • Figure 4 shows that competing methods exhibit artifacts, whereas TTM achieves clean placement and appearance consistency; Table 1 reports quantitative results for MC-Bench object motion control.Source: Figure 4 · p. 6, Table 1 · p. 6
  • A single noise level trades motion adherence against realistic dynamics: low noise can freeze unmasked backgrounds, while high noise can drift from the intended motion.The boat example requires the unmanipulated wake to change naturally as the boat moves.Source: §3.3 · pp. 5–6
  • Dual-clock denoising initializes from the warped reference at tweak and overrides masked regions with the reference noised to t−1 until tstrong, then resumes standard sampling.This keeps the masked trajectory constrained while allowing the background to denoise more freely; the qualitative comparison describes realistic dynamics without artifacts.Source: Figure 3 · p. 5, §3.3 · p. 6
  • The sampling modification adds no computation or latency over regular video diffusion and is training-free and plug-and-play for image-conditioned models.Experiments integrate it with three backbones.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments evaluate TTM on object motion, camera motion, joint motion–appearance editing, and plug-and-play adaptation across I2V backbones. Results show strong motion control and video quality, while qualitative examples demonstrate pixel-level appearance control.

4.2 CAMERA MOTION CONTROL

  • Against the best GWTF variant, TTM reduces pixel MSE by 33% and FID by 15.5% in camera-motion control.Optical-flow MSE also decreases, indicating better temporal alignment across the scene.Source: Table 2 · p. 7, §4.2 · p. 8

4 EXPERIMENTS

  • The evaluation covers object motion, camera motion, and joint motion–appearance editing, with quantitative benchmarks for the first two and qualitative results for appearance editing.It also examines plug-and-play generality across I2V backbones and dual-clock ablations.Source: §4 · pp. 6–7

4.1 OBJECT MOTION CONTROL

  • TTM exceeds training-based and training-free baselines on most object-motion metrics while remaining training-free.Across both backbones, it attains the lowest CoTracker distance, excluding SG-I2V; on CogVideoX it outperforms GWTF on motion adherence and nearly all video-quality metrics.Source: §4.1 · pp. 7–8

4.3 APPEARANCE CONTROL

  • TTM jointly controls motion and per-pixel appearance, including color, object insertion, and shape changes in qualitative examples.For the chameleon example, TTM preserves motion and appearance where GWTF with optical flow and a text prompt fails to satisfy both constraints.Source: §4.3 · pp. 8–9, Figure 7 · p. 9

4.4 PLUG-AND-PLAY MODEL ADAPTATION

  • TTM adapts to WAN 2.21 with brief adaptation for local object and explicit camera-motion control, while GWTF requires fine-tuning CogVideoX-5B.GWTF's warped-noise training requires approximately 7,680 A100-80GB GPU-hours.Source: §4.4 · p. 9

5 CONCLUSIONS, LIMITATIONS AND FUTURE DIRECTIONS

TTM uses warped reference animations as motion guidance, image conditioning to preserve fidelity, and region-dependent dual-clock denoising to balance adherence with natural adaptation. Its limitations include clock tuning, first-frame identity constraints, and a requirement for full object masks.

5 CONCLUSIONS, LIMITATIONS AND FUTURE DIRECTIONS

  • TTM extends SDEdit to video by using warped reference animations for motion guidance and image conditioning for fidelity, with dual-clock denoising balancing adherence and adaptation.The dual-clock scheme still requires tuning of (tweak, tstrong).Source: §5 · p. 9
  • Identity preservation is limited to content visible in the first frame, and specified motion requires full object masks.The method remains robust to imperfect masks, but objects entering later cannot be anchored beyond what denoising implicitly recovers.Source: §5 · p. 9
  • Future directions include multiple regions, soft masks, smoother noise schedules, richer appearance edits, articulated motion, and long-horizon generation.Source: §5 · p. 9

A ABLATION STUDY: DUAL-CLOCK DENOISING

The ablation contrasts dual-clock denoising with single-clock, RePaint-style, and unconstrained-background settings. The dual-clock setting achieves the best overall trade-off between motion adherence, dynamics, and visual quality.

A ABLATION STUDY: DUAL-CLOCK DENOISING

B.1 OBJECT MOTION CONTROL

This appendix details object- and camera-control evaluation protocols, implementation choices, and supplementary appearance and qualitative examples.

B.1 OBJECT MOTION CONTROL

  • Object-control evaluation uses single-trajectory cases, covering over 91% of MC-Bench, and standardizes clip lengths to 16 frames for SVD-based methods and 49 for CogVideoX-based methods.SG-I2V is evaluated on its native 14-frame output, which may be slightly favorable to its metrics.Source: §B.1 · pp. 13–14
  • Inputs and trajectories are resized and padded consistently, while masks are projected to each model’s latent resolution and denoising settings are specified for TTMSVD and TTMCog.Exceptions include MotionPro’s original pipeline, DragAnything’s default input handling, and SG-I2V’s bounding-box conditioning.Source: §B.1 · pp. 13–14
  • Dynamic Degree uses a top-5% RAFT-flow threshold, with α reduced from 6.0 to 3.5 for MC-Bench’s small localized motions.The 25% sampled-frame rule is unchanged.Source: §B.1 · p. 14
  • CoTracker Distance measures mean per-frame pixel distance to the input trajectory; BG–Obj CTD averages background-grid versus object displacement distances, with higher values indicating less co-motion.Source: §B.1 · p. 14

B.2 CAMERA CONTROL ON DL3DV

  • Camera-control experiments use a DL3DV-10K subset and 960p PyTorch3D reference warps, with coordinate conventions adjusted and transition scale selected by binary search.Source: §B.2 · p. 14
  • The camera benchmark filters out scenes with estimated scale below 0.3 and evaluates the 150 scenes with lowest warp-to-ground-truth MSE, using cleaned masks for strong guidance.Scene prompts are automatically generated with GPT-4o.Source: §B.2 · pp. 14–15

B.3 APPEARANCE CONTROL

  • The chameleon appearance-control example pairs left-to-right walking with a prompt requesting gradual color change from green to purple.Source: §B.3 · p. 15

C EXTRA QUALITATIVE COMPARISONS

C.1 QUALITATIVE COMPARISONS FROM MC-BENCH

Additional MC-Bench results compare approaches using SVD as their common image-to-video backbone.

C.1 QUALITATIVE COMPARISONS FROM MC-BENCH

  • The supplementary results extend beyond those shown in Fig. 4 on the MC-Bench dataset.Source: §C.1 · p. 15
  • The comparisons use SVD as the common image-to-video backbone.Source: §C.1 · p. 15
  • The additional results compare TTM with leading approaches on MC-Bench.Source: §C.1 · p. 15

C.2 QUALITATIVE COMPARISONS FROM DL3DV

The DL3DV examples qualitatively compare camera-motion control by TTM and GWTF from an image, monocular depth estimate, and depth-warped video.

C.2 QUALITATIVE COMPARISONS FROM DL3DV

  • The qualitative camera-control examples use DL3DV inputs comprising an image, monocular depth estimate, and depth-warped video.Source: §C.2 · p. 17
  • The examples compare TTM with GWTF on camera-motion control.Source: §C.2 · p. 17
  • The authors report that TTM better maintains intended camera motion and overall visual fidelity in these examples.Source: §C.2 · p. 17

D CHALLENGING USER-CREATED EXAMPLES

The examples include user-crafted object- and camera-motion cases, with additional cut-and-drag scenarios illustrating where GWTF often struggles.

D CHALLENGING USER-CREATED EXAMPLES

Loading 2511.08633v1…