PaperlayerView exact revision on arXiv

Source-linked AI summary

Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer

Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, Tali Dekel

arXiv:2311.17009v2cs.CV

TL;DR

Text-driven motion transfer across different object categories is difficult because prior methods are mostly confined to similar subjects, while edits must preserve source motion and scene layout. The paper guides generation with a fixed text-to-video diffusion model’s space-time features and a zero-shot optimization framework. It reports state-of-the-art motion preservation and edit fidelity, while noting dependence on the model’s generative priors.

  • Problem

    Prior motion-transfer methods are mostly confined to similar or closely related object categories, while text-driven transfer must preserve input motion and layout despite changes in target shape and fine-grained motion.

    Source: §1 · pp. 1–2

  • Method

    A zero-shot optimization framework uses a fixed pre-trained text-to-video diffusion model and a space-time feature loss to guide generation toward the input video’s motion and scene layout.

    Source: §1 · p. 2, §4 · p. 4, §6 · p. 8

  • Results

    The method reports state-of-the-art performance in motion preservation and edit fidelity compared with existing text-to-video methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate the potential of leveraging text-to-video model priors and space-time features for motion transfer across objects that differ in shape and fine-grained motion traits.

    Source: §6 · pp. 8–9

    Performance depends on the text-to-video model’s learned generative priors, so out-of-distribution combinations of target object and input motion can reduce motion fidelity or produce visual artifacts.

    Source: §6 · p. 8

Abstract

from arXiv · show

We present a new method for text-driven motion transfer - synthesizing a video that complies with an input text prompt describing the target objects and scene while maintaining an input video's motion and scene layout. Prior methods are confined to transferring motion across two subjects within the same or closely related object categories and are applicable for limited domains (e.g., humans). In this work, we consider a significantly more challenging setting in which the target and source objects differ drastically in shape and fine-grained motion characteristics (e.g., translating a jumping dog into a dolphin). To this end, we leverage a pre-trained and fixed text-to-video diffusion model, which provides us with generative and motion priors. The pillar of our method is a new space-time feature loss derived directly from the model. This loss guides the generation process to preserve the overall motion of the input video while complying with the target object in terms of shape and fine-grained motion traits.

1. Introduction

The paper targets text-driven motion transfer across different object categories, where the source motion must be preserved despite changes in shape and scene. It uses a fixed text-to-video model’s space-time features to guide zero-shot editing and reports state-of-the-art motion preservation and edit fidelity.

1. Introduction

  • The task must preserve motion characteristics such as direction, speed, and pose while changing object structure and accounting for scene interactions.For example, a car-to-bicycle edit may require revealing background content and synthesizing plausible shadows.Source: §1 · p. 1
  • Prior motion-transfer methods primarily address similar-looking subjects within the same category, often relying on geometric priors such as human pose models.The paper frames cross-category transfer as a more general setting involving substantial shape changes and deformation over time.Source: §1 · pp. 1–2
  • The approach avoids explicit pose and shape models, using a fixed text-to-video model’s intermediate space-time features to guide generation toward the source scene layout and motion.The authors also report empirically analyzing and harnessing these features, and contrast them with spatial-only image priors that lack consistent structural editing across frames.Source: §1 · p. 2
  • The method reports state-of-the-art performance in motion preservation and edit fidelity against existing text-to-video methods.The framework is zero-shot, requiring no training data or fine-tuning.Source: §1 · p. 2
  • The stated contributions include a zero-shot framework, insights into text-to-video diffusion features, a motion-fidelity metric under structural deviation, and a stronger balance between motion preservation and prompt fidelity.The paper presents this balance as a result relative to competing methods.Source: §1 · p. 2

2. Related works

Related work generally addresses motion transfer between similar subjects or video edits that preserve appearance, layout, or frame consistency. This paper instead uses text-to-video motion priors and space-time features for text-driven transfer across distinct categories and substantial structural changes.

Motion transfer

  • Earlier motion-transfer methods typically use source motion and a target image or video, and are largely limited to related categories or domains with robust parametric models.Examples include human or face domains; cited approaches include and Open-Pose.Source: §2 · p. 2
  • The paper’s target is prompt-specified motion transfer across distinct categories, including differences in shape, appearance, and natural fine-grained motion.The figure illustrates full space-time feature inversion and a spatial marginal mean variant that retains pose, position, and scene layout while allowing pixel-level variation.Source: §2 · p. 2, Figure 2 · p. 3

Text-to-video models

  • Text-to-video models extend video generation beyond earlier small-domain VAE or GAN systems, with recent approaches using Transformers or diffusion models and temporal layers added to image models.The cited examples include, [24] [63], and [46].Source: §2 · p. 2
  • Video-to-video editing methods use structure or appearance representations, but the cited approaches do not preserve reference motion while substantially departing from its frame structure.The paper instead leverages a pre-trained text-to-video diffusion model for zero-shot motion transfer [64].Source: §2 · p. 2

Image & video editing via feature manipulation

  • Text-to-image feature editing and video editing methods offer spatial control or frame consistency, but lack a text-to-video motion prior for substantial structural deviation.Examples include Prompt-to-Prompt [20], Plug-and-Play Diffusion, Tune-A-Video [69], and TokenFlow [19]; the paper derives its editing loss from text-to-video feature space.Source: §2 · pp. 2–3

Consistent video editing

  • Layered-representation editing methods are often restricted to appearance changes or local structure edits, and use a 2D generative prior that does not support large structure deviation and motion adaptation.The cited work includes Text2LIVE and a later local-structure extension [31].Source: §2 · p. 3

3. Preliminary

The preliminary section describes diffusion as iterative denoising from noise to a data sample, and notes deterministic DDIM sampling and inversion. It then outlines latent image and video diffusion architectures used to generate RGB videos.

Diffusion models

  • Diffusion models map noise x_T to a clean sample x_0 through iterative denoising, and DDIM sampling can make denoising deterministic.DDIM inversion maps a clean sample back to its sequence of noisy samples.Source: §3 · p. 3
  • Latent text-to-image diffusion encodes an RGB image into a lower-resolution latent representation, which a UNet processes using convolution and attention modules.The cited example is StableDiffusion.Source: §3 · p. 3
  • Latent video diffusion extends latent image models with temporal convolutions and attention, generating a latent video that is decoded into RGB frames.The paper uses ZeroScope, which inflates StableDiffusion.Source: §3 · p. 3

4. Method

The method uses a pre-trained text-to-video model and space-time features to guide video generation toward preserving the input’s scene layout and motion while matching a target prompt. It analyzes spatial marginal means and guides editing with pairwise differences between them.

4. Method

  • The framework takes an input video and target prompt and uses a pre-trained latent text-to-video model to generate a video preserving the input’s overall motion and semantic layout.The pipeline extracts space-time features after DDIM inversion and computes pairwise differences between SMM features for generation guidance.Source: §4 · p. 3, Figure 3 · p. 4

4.1. Space-time analysis of diffusion features

  • Spatial Marginal Mean features summarize spatial dimensions into per-frame descriptors that retain pose and scene layout while tolerating pixel-level appearance and shape variation.The analysis also finds that nearest-neighbor frames retrieved using these descriptors show the same pose despite noticeable appearance and viewpoint changes.Source: §4 · p. 4, §4.1 · p. 4
  • SMM inversion retains object pose and position despite larger structural and appearance variations than inversion using full space-time features.Inversions using full space-time features nearly reconstruct the original frames across different random initializations.Source: §4.1 · p. 4

4.2. Motion-guided video generation

  • Pairwise SMM differences guide generation toward the target prompt while preserving relative feature changes over time and discarding exact source-appearance information.The method guides generation at each denoising step; directly optimizing SMM features can retain too much source appearance or converge to a local minimum.Source: §4.2 · p. 5
  • The method starts from a filtered latent that preserves low-frequency DDIM noise while random noise determines its higher frequencies.The authors report that initializing directly from the DDIM noise may restrict editability.Source: §4.2 · p. 6

5. Results

The results evaluate motion transfer across varied videos, compare the method with editing baselines, and test key design choices. The reported comparisons address text-edit fidelity and preservation of input motion, including under structural changes.

5. Results

  • The evaluation covers varied scenes and object categories, with most videos involving both camera and object motion.Driving videos come from DAVIS and the Web.Source: §5 · p. 6

5.1. Qualitative evaluation

  • Against SA-NLA, both methods preserve original motion, but ours allows greater structural deviation and adaptation of fine-grained target motion traits.Examples include matching a duck’s structure in the swan example and adapting a dog’s tail shape and movement to resemble a cat’s.Source: Figure 5 · p. 6, §5.1 · p. 6
  • Compared with TokenFlow, Gen-1, Tune-A-Video, and Control-A-Video, our method handles substantial structural edits and can synthesize dynamic scene elements.The comparisons describe TokenFlow as structure-preserving, Gen-1 and Control-A-Video as struggling to depart from input shapes, and Tune-A-Video as satisfying prompts without preserving pose and motion alignment.Source: §5.1 · pp. 6–7, Figure 6 · p. 7

5.2. Ablations

  • Ablations show that full space-time feature reconstruction restricts shape and appearance variation, while SMM feature reconstruction is more flexible but insufficient to match the edit.The alternatives replace the pairwise SMM differences loss.Source: Figure 7 · p. 7, §5.2 · p. 8
  • Without optimization, sampling from the initial latent yields unrelated object poses; starting optimization from random noise fails to retain input motion characteristics.The figure comparison reports lower motion fidelity for the random-noise initialization than for the full method.Source: Figure 7 · p. 7, §5.2 · p. 8

5.1. Qualitative evaluation

  • The new motion-fidelity metric compares unaligned long trajectories because structural deviations break pixel alignment and make optical-flow comparisons unsuitable.It uses tracked input and output tracklets and measures each tracklet’s similarity to its nearest neighbor in the other set, in both directions.Source: §5.1 · pp. 7–8
  • Across 54 video-edit text pairs, the method achieves high text-edit and original-motion fidelity; 7,000 judgments from 150 users consistently prefer it to every baseline.TokenFlow and Control-A-Video score high on motion fidelity but low on edit fidelity, while Tune-A-Video shows the inverse trend; no tested SDEdit noise level resolves the trade-off, and Gen-1 could not be evaluated quantitatively because its API returns a different number of frames.Source: §5.1 · p. 8

6. Discussion and conclusions

The conclusion presents text-driven motion transfer across objects differing in shape and fine-grained motion traits, using a pre-trained text-to-video diffusion model. It reports a better balance between prompt fidelity and motion preservation, while noting limitations tied to model priors and maturity.

6. Discussion and conclusions

  • The work introduces zero-shot motion transfer for source and target objects that differ in shape and fine-grained motion traits, using a pre-trained text-to-video diffusion model.The authors describe the method as a simple optimization framework and report new analysis of space-time T2V features.Source: §6 · p. 8
  • The method exhibits a better balance between CLIP text similarity and motion fidelity than the baselines in the quantitative evaluation.Figure 8 reports these two metrics, both with higher values better.Source: Figure 8 · p. 8
  • The user study reports the percentage of judgments in the method’s favor relative to each baseline.Source: Table 1 · p. 8
  • When the target object and input motion combination is out of distribution for the T2V model, motion fidelity can degrade or visual artifacts can occur.Source: §6 · p. 8
  • The method’s performance relies on T2V generative priors, and public models remain limited in quality, resolution, video length, and training-data scale.The authors nevertheless report improvement over prior state-of-the-art methods.Source: §6 · pp. 8–9

A. Text-to-Video Model Architecture and Feature Selection

The implementation uses the ZeroScope text-to-video model and selects coarse features from its second UNet decoder block. Baseline comparisons use differing protocols depending on access and training cost.

A. Text-to-Video Model Architecture and Feature Selection

  • Gen-1 was excluded from numerical comparisons because its web platform produced videos of varying lengths with duplicated frames, while SA-NLA was compared qualitatively due to its 10-hour training time.Source: A. Text-to-Video Model Architecture… · p. 12
  • ZeroScope is the text-to-video model used; its videos are 24 frames at 576x320 resolution, and temporal layers were added to StableDiffusion’s UNet.The model is described as fine-tuned from a Modelscope model.Source: A. Text-to-Video Model Architecture… · p. 12
  • Feature analysis uses coarse features from ZeroScope’s second UNet decoder block, with features after upsampling producing more visually appealing edits.The tested alternatives included spatial and temporal convolution features, attention tokens, and semantic DIFT features.Source: A. Text-to-Video Model Architecture… · p. 12

B. Implementation Details

Implementation extracts diffusion features from DDIM-inverted latents, then initializes and optimizes sampling with specified schedules and settings. Runtime depends on the number of optimization steps per denoising step.

B. Implementation Details

  • Intermediate latents are obtained by reverse DDIM sampling with classifier-free guidance scale 1, 1000 forward steps, and a video-specific inversion prompt.These latents initialize generation and provide diffusion features.Source: B. Implementation Details · p. 12
  • Sampling uses 50 denoising steps with Restart Sampling and DDIM, classifier-free guidance scale 10, and downsample/upsample initialization at factor ξ = 4.Source: B. Implementation Details · p. 12
  • Optimization runs during the first 20 denoising steps, usually with Adam at learning rate 0.01 for 30 steps, or with decayed learning rates for larger structural deviations.The alternate schedule decays linearly from 0.005 to 0.002 over 10 optimization steps.Source: B. Implementation Details · p. 12
  • DDIM inversion takes about 10 minutes, while sampling with optimization takes about 7 minutes for 10 optimization steps or 15 minutes for 30 steps per denoising step.Source: B. Implementation Details · p. 12

C. Baseline Comparison Details

The baseline comparisons with Tune-A-Video, TokenFlow, and Control-A-Video used their official repositories.

C. Baseline Comparison Details

Related papers

Loading 2311.17009v2…