PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Creation by Demonstration

Yihong Sun, Hao Zhou, Liangzhe Yuan, Jennifer J. Sun, Yandong Li, Xuhui Jia, Hartwig Adam, Bharath Hariharan, Long Zhao, Ting Liu

arXiv:2412.09551v1cs.CV

TL;DR

Video generation from demonstrations aims to transfer action concepts into a different scene, while existing explicit controls can limit expressiveness or be difficult to obtain. 𝛿-Diffusion uses implicit action latents and self-supervised training from video; experiments report improved controllability and concept transfer over text control, with a remaining physical-plausibility limitation in complex scenes.

  • Problem

    Existing video-generation controls often rely on explicit signals that can limit expressiveness or be difficult to acquire, while transferring actions across misaligned contexts is challenging.

    Source: §1 · p. 2

  • Method

    𝛿-Diffusion combines video foundation-model representations and an appearance bottleneck to extract implicit action latents, then trains video generation from a context image and demonstration-derived control.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Visual demonstration yields better controllability and concept transferability than text control, and the system generates high-fidelity videos across a wide range of action concepts.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Concatenating different demonstration videos can drive a coherent generated sequence, indicating potential as an alternative to generative interactive environments.

    Source: §1 · p. 3

    Generated results might not fully follow physical laws under complex scenes.

    Source: §1 · p. 3

Abstract

from arXiv · show

We explore a novel video creation experience, namely Video Creation by Demonstration. Given a demonstration video and a context image from a different scene, we generate a physically plausible video that continues naturally from the context image and carries out the action concepts from the demonstration. To enable this capability, we present $\delta$-Diffusion, a self-supervised training approach that learns from unlabeled videos by conditional future frame prediction. Unlike most existing video generation controls that are based on explicit signals, we adopts the form of implicit latent control for maximal flexibility and expressiveness required by general videos. By leveraging a video foundation model with an appearance bottleneck design on top, we extract action latents from demonstration videos for conditioning the generation process with minimal appearance leakage. Empirically, $\delta$-Diffusion outperforms related baselines in terms of both human preference and large-scale machine evaluations, and demonstrates potentials towards interactive world simulation. Sampled video generation results are available at https://delta-diffusion.github.io/.

1. Introduction

Video Creation by Demonstration transfers action concepts from a demonstration video into a different scene, addressing the limits of explicit controls and the difficulty of training across misaligned contexts. The paper combines video foundation-model representations, implicit action latents, and self-supervised training; experiments report high-fidelity generation and stronger controllability and concept transfer than text control.

1. Introduction

  • The proposed experience generates a video from a context image and demonstration, carrying out demonstrated action concepts with temporal continuity and physical plausibility.The demonstration and context can come from different scenes.Source: §1 · p. 2
  • Existing controls such as prompts, keypoints, depth, and segmentation can limit expressiveness or require signals that are difficult to acquire.General videos add transfer challenges because actions vary with the subject, object, and environment, and may involve camera changes or motion blur.Source: §1 · p. 2
  • The approach uses video foundation-model representations and implicit action latents, then trains on a context frame and following clip sampled from one video.The appearance bottleneck extracts action latents with minimal appearance and contextual information.Source: §1 · pp. 2–3
  • Visual demonstration yields better controllability and concept transferability than text control, while supporting high-fidelity videos across varied action concepts.The reported range includes everyday activities, egocentric perspectives, and complex robotic movements.Source: §1 · p. 3
  • The authors identify a remaining limitation: generated results might not fully follow physical laws under complex scenes.They present the paradigm as a potential direction toward interactive world simulation.Source: §1 · p. 3

2. Related Works

The related work situates Video Creation by Demonstration among image-and-video-conditioned generation, action transfer, and video foundation-model research. It distinguishes the approach through implicit action concepts intended for misaligned contexts and general real-world videos.

2.1. Video Generation

  • Video Creation by Demonstration is an image-and-video-to-video task that continues from a context image while integrating action concepts from a demonstration.Prior approaches commonly use explicit extracted signals such as text, depth, tracks, keypoints, or segmentation masks.Source: §2.1 · p. 3

2.2. Modeling Motion and Action

  • Action-transfer methods often assume alignment between source and target videos, whereas context misalignment can make low-level motion transfer fail to convey the intended action concept.For example, transferring a robot arm closing a top drawer to a bottom drawer involves different scene context.Source: §2.2 · p. 4

2.3. Video Foundation Models

  • Video foundation models provide representations that capture both action and context, and the paper studies their use for controlling image-and-video-to-video generation.The authors state that, to their knowledge, this is the first such study.Source: §2.3 · p. 4

3. Methodology

The method conditions video generation on a context image and action-rich latents extracted from a demonstration, using a self-supervised future-frame prediction setup. Its appearance bottleneck separates per-frame appearance information from temporally aggregated representations.

3.1. Task Formulation

  • The task takes a context image and demonstration video as inputs and generates a video that continues from the context while carrying out compatible action concepts.The task excludes actions incompatible with the context, such as simulating cutting when no hands appear in the context image.Source: §3.1 · p. 4

3.2. Method Overview

  • The generation model receives the context image and demonstration-derived control latents, which the appearance bottleneck is designed to make action-rich with minimal appearance information.Training samples the context and demonstration from the same video, with the context as the starting frame and the following clip as the reconstruction target.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3. Extracting Action Latents

  • The appearance bottleneck predicts per-frame features aligned with temporally aggregated features, then subtracts them to obtain action-rich control latents.The per-frame predictor is intended to capture appearance rather than information collectively defined by multiple consecutive frames; the remaining difference represents temporal surprisal associated with action concepts.Source: §3.3 · pp. 5–6

3.4. Training 𝛿-Diffusion

  • Training samples a context frame and following clip from one video, uses an augmented version of the clip to extract control latents, and trains the model to predict the target frames.Action-preserving augmentations such as random cropping create a mismatch that makes direct appearance copying from the demonstration harder.Source: §3.4 · p. 6

3.5. Implementation Details

  • The implementation uses VideoPrism as the vision encoder and WALT as the generation model, generating 16-frame videos at 128×128 resolution.The context image is passed for conditional generation, and projected control latents replace the original text embeddings.Source: §3.5 · pp. 6–7

4. Experiment

Experiments evaluate δ-Diffusion on three video datasets using machine and human assessments. Human raters favor it across datasets and evaluation criteria, while ablations and qualitative comparisons examine bottleneck choices and baseline behavior.

4.1. Datasets

  • Epic Kitchens 100, Something-Something v2, and Fractal supply evaluation data; machine-test pairs share frequent action labels, while human-test pairs are manually vetted for transferability.Source: §4.1 · p. 7

4.2. Evaluation Setup

  • Machine evaluation uses FVD, embedding cosine similarity, and retrieval Hit@100/500/1k; five raters compare paired outputs on visual quality, action transferability, and context consistency.Source: §4.2 · pp. 7–8

4.4. Main Results

  • Human raters favored δ-Diffusion over both baselines across datasets for visual quality, action transferability, and context consistency.Source: Table 2 · p. 8, §4.4 · p. 9

4.3. Ablation Study

  • The appearance bottleneck consistently outperforms no bottleneck and temporal normalization on Hit@k and ES, but not consistently on visual quality; alternatives show appearance leakage.Source: Table 1 · p. 8, §4.3 · p. 9, Figure 3 · p. 9

4.4. Main Results

  • Sequentially applying action latents from demonstrations in different scenes simulates complex task execution in one consistent environment, using the prior step’s last frames as context.Source: §4.4 · p. 9
  • Against MotionDirector and WALT, δ-Diffusion shows faithful action execution and content preservation; MotionDirector often fails, while WALT may execute inauthentic actions.Source: §4.4 · p. 9, Figure 4 · p. 10

5. Conclusion

The paper presents Video Creation by Demonstration, using a context image and demonstration video to generate a continuation that carries out the demonstrated action. Its δ-Diffusion approach addresses appearance and context misalignment with action-focused latent controls, and the paper notes that physical realism is not always preserved in complex scenes.

5. Conclusion

  • δ-Diffusion combines video-foundation-model representations with an appearance bottleneck to extract action-rich control signals with minimal appearance information.A generation model then uses those signals and the context image to continue the scene while carrying out the demonstrated action.Source: §5 · p. 10
  • The task transfers action concepts between a demonstration video and an initial context frame despite their appearance and context misalignment.The generated video is intended to continue from the context image and perform the demonstrated action in a similar manner.Source: §5 · p. 10
  • Experiments on three datasets using machine and human evaluations demonstrate the effectiveness of δ-Diffusion.Source: §5 · p. 10
  • The current system does not always preserve physical realism under complex scenes, a limitation the authors say needs further exploration.They note that scaling up the generation model could potentially alleviate it.Source: §5 · p. 10

Broader Impact

The paper describes Video Creation by Demonstration as a way to ease interactive video generation and simulate possible scene-wide effects of actions. It also identifies potential negative impacts and lists contributors, advisors, and a sponsor.

Broader Impact

  • The system may simulate effects of actions on the whole scene, including interactions absent from the demonstration video.Source: Broader Impact · p. 10
  • The authors identify possible risks including dataset bias, misleading or inappropriate generated content, and privacy concerns involving people without explicit consent.They call for vigilance and responsible implementation.Source: Broader Impact · p. 10

Core Contributors

  • Core contributors covered project leadership, dataset curation, evaluation, action-model and generation-model research, infrastructure, and demos.Ting Liu and Long Zhao provided project leadership; the passage assigns other listed responsibilities across the core team.Source: Core Contributors · p. 16

Partial Contributors and Advisors

Sponsors

B. Improving Generation Quality

For longer demonstrations, the generation setup supports autoregressive segments and trains the model to accept multiple consecutive frames as context. Training also uses self-conditioning and control-signal masking, with classifier-free guidance at inference.

B. Improving Generation Quality

  • During training, the context image is replaced with multiple consecutive frames with probability 0.5 to encourage smoother continual generation.Source: B. Improving Generation Quality · p. 16
  • At inference, long demonstrations are split into segments of length T=16, with later segments conditioned on the previous segment’s last four generated frames.The first segment uses the input context image; segments are aligned with their corresponding demonstration segments.Source: B. Improving Generation Quality · p. 16
  • Training uses self-conditioning with probability 0.9 and randomly masks the control signal with probability 0.2; inference uses classifier-free guidance weight 1.25.For unconditional generation, both self-conditioning and the control signal are dropped.Source: B. Improving Generation Quality · p. 16

C.1. Prompt Selection

The appendix describes how human-evaluation prompts were selected and assembled across three datasets, and gives evaluation instructions, study details, and baseline configurations.

C.1. Prompt Selection

  • Demonstrations were 16–24 frames at 12 FPS for Something-Something v2 and Epic Kitchens, or 10 FPS for Fractal.Source: §C.1 · p. 16
  • For Something-Something v2, 20 pairs per selected action class were manually chosen from 500 sampled pairs; Epic Kitchens and Fractal used different filtering and sampling procedures.Epic Kitchens pairs required CLIP similarity above 0.9 before manual selection; Fractal pairs were sampled across eight action classes.Source: §C.1 · p. 17
  • Prompt selection checked that a demonstrated action could reasonably apply to the context, without considering which methods would be evaluated.Source: §C.1 · p. 17
  • Each prompt pairs one selected video as the demonstration with the first frame of another video as the context image.Source: §C.1 · p. 17

C.2. Rating Instructions and Rubrics

  • Table 3 provides the instructions and rubrics shown to raters before they compare generated videos side by side.Source: §C.2 · p. 17, Table 3 · p. 17

C.3. Additional Details

  • The study used five raters, randomized which method appeared as Video A, and collected 1,800 preferences across two baselines, three datasets, and three metrics.Source: §C.3 · p. 18

D. Details on Baselines

  • MotionDirector was run using its official code, with a context-image spatial path trained for 300 epochs before 16-frame temporal-path training.Inference used a 0.0 noise prior and ground-truth caption labels for training and inference.Source: D. Details on Baselines · p. 18
  • WALT was fine-tuned for 200k steps per dataset; its generations preserved context consistency but could perform ungenuine actions due to limited expressiveness in text descriptions.At inference, the context image was paired with the demonstration video’s ground-truth caption.Source: D. Details on Baselines · p. 18

E. Qualitative Results

The paper presents qualitative demonstrations of generated video and of controlling alternative generations from the same context image with different demonstration videos.

E. Qualitative Results

F. Controllability by Demonstration

  • Figures 6–9 showcase δ-Diffusion’s controllability: video samples are generated for the same context image while changing the demonstration video, illustrating alternatives driven by different demonstrations.Source: F. Controllability by Demonstration · p. 18

G. Failure Cases

Figure 10 identifies three failure modes: action semantics may be incompletely carried out, object permanence may fail, and significant demonstration-context mismatches can produce inconsistent generations.

G. Failure Cases

  • Figure 10 illustrates three primary failure modes of 𝛿-Diffusion.One example places an object between existing objects instead of next to them.Source: G. Failure Cases · p. 18, Figure 10 · p. 23
  • Fast appearance changes during object motion can cause appearance leakage and failures of permanence.The paper gives fast object rotation as an example.Source: G. Failure Cases · p. 18
  • When demonstrations and context images are significantly mismatched, generations can be inconsistent, including an extra hand under a perspective mismatch.Source: G. Failure Cases · p. 18
Loading 2412.09551v1…