PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionClone: Training-Free Motion Cloning for Controllable Video Generation

Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, Yi Jin

arXiv:2406.05338v6cs.CV

TL;DR

Motion-based video control is challenging, and existing motion-cue methods commonly require training or fine-tuning. MotionClone uses sparse temporal-attention weights from reference videos for training-free motion guidance; experiments report proficiency in global and local motion, with superiority in motion fidelity, text alignment, and temporal consistency.

  • Problem

    Motion synthesis complicates video generation, while existing motion-control approaches commonly train or fine-tune cues and may transfer poorly beyond trained domains.

    Source: §1 · p. 1

  • Method

    MotionClone uses dominant temporal-attention weights from a reference video as sparse motion representations, extracting guidance in one denoising step without cumbersome inversion.

    Source: §1 · p. 2

  • Results

    MotionClone demonstrates proficiency in global camera and local object motion, with reported superiority in motion fidelity, text alignment, and temporal consistency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MotionClone supports motion transfer across scenarios and both text-to-video and image-to-video generation while preserving reference motion and incorporating novel textual semantics.

    Source: §1 · p. 2, §5 · p. 10

    Low spatial resolution in latent temporal-attention features limits subtle local motion, and overlapping moving objects risk quality drops.

    Source: §4.7 · p. 10

Abstract

from arXiv · show

Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, we propose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.

1 INTRODUCTION

Motion control can address motion-synthesis challenges in video generation, while existing motion-cue methods commonly train or fine-tune and can limit transferability. MotionClone extracts sparse temporal-attention motion representations from reference videos without training, and experiments report proficiency across global and local motion with superiority in motion fidelity, text alignment, and temporal consistency.

1 INTRODUCTION

  • Dense depth or sketch cues can entangle motion with reference structure, while trajectories miss fine localized actions; both approaches commonly require training or fine-tuning.These constraints can impede transfer to novel scenarios or lead to suboptimal generation outside the trained domain.Source: §1 · p. 1
  • MotionClone uses dominant temporal-attention weights as sparse motion representations, filtering noisy or subtle motions to guide primary motion without motion-specific fine-tuning.A representation extracted at one denoising step can guide all time steps, avoiding cumbersome video inversion; the framework supports text-to-video and image-to-video.Source: §1 · p. 2
  • Experiments report proficiency in global camera motion and local object action, with superiority in motion fidelity, text alignment, and temporal consistency.MotionClone is introduced as training-free motion cloning from reference videos for controllable video generation.Source: §1 · p. 2

2 RELATED WORK

Prior work advances video generation with motion modules and diverse control signals, but the approaches surveyed have limitations in content quality, transferability, or training complexity. MotionClone draws on temporal-attention features as motion guidance, extending attention-based control beyond its prior focus on spatial layers.

2.1 TEXT-TO-VIDEO DIFFUSION MODELS

  • VideoLDM uses 3D convolutions and temporal attention, AnimateDiff fine-tunes temporal layers, and VideoCraft2 learns motion from low-quality videos; diverse motion and limited high-quality video data remain challenges.The surveyed text-to-video models still show a quality disparity from text-to-image counterparts.Source: §2.1 · pp. 2–3

2.2 CONTROLLABLE VIDEO GENERATION

  • Video controls include first frames, trajectories, regions, and objects; dense depth or edge cues can introduce static structure, while motion-specific fine-tuning adds training complexity and potential model degradation.MotionClone instead extracts temporal correlations as explicit motion clues for plug-and-play customization.Source: §2.2 · p. 3

2.3 ATTENTION FEATURE CONTROL

  • Prior attention-control work mainly targets spatial layers, whereas MotionClone identifies temporal attention as a source of motion guidance for flexible motion cloning.Earlier work uses cross-attention for image layout and self-attention features for structural image information.Source: §2.3 · p. 3

3 MOTIONCLONE

MotionClone guides video synthesis by manipulating temporal-attention weights, extracting sparse motion representations from reference videos and applying them during denoising.

3 MOTIONCLONE

  • MotionClone uses temporal-attention manipulation for motion cloning, and its pipeline extracts reference motion representations to guide novel video synthesis.The framework is described as covering video-generation tasks and motion guidance by manipulating temporal-attention weights.Source: §3 · p. 3, Figure 4 · p. 6

3.1 PRELIMINARIES

  • Video diffusion encodes input videos into latents, learns to estimate noise, and begins inference from Gaussian noise, with guidance able to adjust the sampling trajectory.The condition signal can be text or an image.Source: §3.1 · pp. 3–4

3.2 OBSERVATION

  • Aligning only sparse, motion-relevant temporal-attention components significantly improves motion alignment over uniform full-map control.The authors attribute this to emphasizing motion-related cues while disregarding motion-irrelevant factors.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 MOTION REPRESENTATION

  • Top-k masking focuses guidance on primary attention components while maintaining spatially even constraints; k=1 selects each location’s highest activation.The temporal mask is based on the rank of reference attention values along the temporal axis.Source: §3.3 · p. 5
  • Preparing reference-video attention maps through inversion is laborious and time-consuming, creating challenges for large-scale preparation and efficient deployment.The method instead derives a motion representation through a single noise-adding and denoising step.Source: §3.3 · p. 5
  • For reference videos, H_tα comprises two highly temporally sparse elements and can be extracted by adding noise followed by a single denoising step.Motion guidance works over tα values from 200 to 600, while tα=800 shows discrepancies for complex head-turning motion; the authors recommend a later denoising stage.Source: §3.3 · pp. 5–6

3.4 MOTION GUIDANCE

  • Motion guidance is applied only during early denoising steps, leaving flexibility for semantic adjustment while supporting motion fidelity and textual alignment.The pipeline combines motion guidance with classifier-free guidance during iterative denoising.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments assess MotionClone across global camera motion, local object motion, and conditioned generation, with qualitative, quantitative, and ablation analyses. Preference tests favor MotionClone, while the paper identifies subtle and overlapping motion as limitations.

4.1 IMPLEMENTATION DETAILS

  • The pipeline extracts motion representations with one denoising step at tα=400, k=1, null-text prompts, and temporal-attention guidance in “up block.1”.For camera cloning, generation uses 100 denoising steps with guidance in 50; for object cloning, it uses 300 steps with guidance in the early 180.Source: §4.1 · p. 6

4.3 QUALITATIVE COMPARISON

  • Qualitative comparisons show stronger text alignment and motion consistency for camera cloning, and improved motion accuracy and text alignment for localized object motion.The comparisons are shown in Figures 5 and 6.Source: Figure 5 · p. 7, §4.3 · p. 8, Figure 6 · p. 8

4.2 EXPERIMENTAL SETUP

  • Evaluation uses 40 real videos (15 camera-motion, 25 object-motion), CLIP-based alignment and smoothness metrics, and 20 volunteers’ ratings on a 1–5 scale.Comparators include VideoComposer, Tune-A-video, Control-A-Video, VMC, and Gen-1.Source: §4.2 · p. 7

4.4 QUANTITATIVE COMPARISON

  • MotionClone outperformed rivals in human preference tests for motion preservation, appearance diversity, temporal consistency, and textual alignment across 40 real videos.The quantitative comparison covers 40 real videos with varied motion patterns.Source: Table 1 · p. 8, §4.4 · p. 9

4.5 VERSATILE APPLICATION

  • Figure 7 shows MotionClone extending beyond T2V to I2V and sketch-to-video, with generated outputs aligned to first-frame or sketch conditions.Red arrows indicate motion direction.Source: Figure 7 · p. 9, §4.5 · p. 9

4.6 ABLATION AND ANALYSIS

  • Lower k improves primary-motion alignment by suppressing scene-specific noise and subtle motion; tα=400 is the default, while tα=800 loses motion information.Values tα∈{200, 400, 600} can all achieve some motion alignment.Source: §4.6 · p. 9
  • “up block.1” best balances motion manipulation and visual quality; precise prompts matter little, and DDIM inversion yields no obvious quality gain despite its time cost.The authors leave improved diffusion inversion for enhanced motion cloning to future work.Source: §4.6 · pp. 9–10

4.7 LIMITATION

  • Latent-space attention’s lower spatial resolution limits subtle local motion, while overlapping moving objects can reduce quality.Figure 10 illustrates these limitations.Source: §4.7 · p. 10, Figure 10 · p. 10

5 CONCLUSION

MotionClone uses sparse temporal-attention weights as motion representations for training-free motion cloning across scenarios. It preserves motion fidelity while incorporating new textual semantics and avoids inversion processes.

5 CONCLUSION

  • MotionClone uses sparse temporal-attention weights as motion representations to guide primary motion alignment and transfer motion across scenarios.The framework is training-free.Source: §5 · p. 10
  • With a real reference video, MotionClone preserves motion fidelity while assimilating novel textual semantics.Source: §5 · p. 10
  • MotionClone avoids cumbersome inversion processes, supporting efficient motion customization across video-generation tasks.Source: §5 · p. 10

A.1 BASELINE DESCRIPTION

The paper compares MotionClone with methods that use extracted video features, reference-video structure, attention adaptation, or a first-frame motion cue. Some comparison methods fine-tune models for motion generation.

A.1 BASELINE DESCRIPTION

  • VideoComposer extracts frame-wise depth or canny maps, while Gen-1 uses reference-video structure in a video-to-video translation approach.Source: §A.1 · p. 14
  • Tune-A-Video expands spatial self-attention into spatiotemporal attention and fine-tunes it for motion-specific generation.Source: §A.1 · p. 14
  • Control-A-Video uses the first video frame as an additional motion cue for customized video generation.Source: §A.1 · p. 14
  • VMC distills motion patterns by fine-tuning temporal-attention layers in a pretrained text-to-video diffusion model.Source: §A.1 · p. 14

A.2 MORE GENERATED RESULTS

Additional examples show MotionClone extracting motion cues from varied existing videos. The generations are described as both prompt-aligned and motion-preserved.

A.2 MORE GENERATED RESULTS

  • Across diverse existing videos, MotionClone extracts motion cues and produces content described as prompt-aligned and motion-preserved.Source: §A.2 · p. 14
  • Figures 11–14 show additional camera-motion and object-motion cloning results.Source: §A.2 · p. 14
  • The paper recommends viewing its video file for a better demonstration of MotionClone.Source: §A.2 · p. 14

A.3 BROADER IMPACT

The broader-impact discussion describes potential benefits in content creation and education, alongside risks from realistic video misuse. It calls for transparency, usage guidelines, and collaborative oversight.

A.3 BROADER IMPACT

  • MotionClone may help film and media professionals streamline production and create more engaging visual experiences without extensive resource commitments.Source: §A.3 · p. 14
  • In education, customized instructional videos with precise text-aligned motions may support engagement and comprehension, particularly for demonstrations.Source: §A.3 · p. 14
  • Realistic videos based on text and existing motion cues raise concerns about deepfakes, misleading media, privacy violations, propaganda, and harmful content.Source: §A.3 · p. 14
  • The conclusion calls for transparency, clear usage guidelines, and collaboration to realize benefits while mitigating risks.Source: §A.3 · p. 14
  • Figures 11–14 contain additional camera-motion and object-motion cloning examples.Source: Figure 11 · p. 15, Figure 12 · p. 16, Figure 13 · p. 17

Related papers

Loading 2406.05338v6…