PaperlayerView exact revision on arXiv

Source-linked AI summary

DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory

Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, Nan Duan

arXiv:2308.08089v1cs.CV

TL;DR

Existing controllable video methods do not comprehensively combine semantic, spatial, and temporal controls, while trajectory research has limited open-domain and complex-motion capability. DragNUWA integrates text, image, and trajectory inputs with TS, MF, and AT, and experiments validate superior fine-grained video control.

  • Problem

    Existing methods incompletely combine text, image, and trajectory controls, while trajectory research has limited open-domain and complex curved-trajectory capability.

  • Method

    DragNUWA integrates text, image, and trajectory inputs with Trajectory Sampler, Multiscale Fusion, and Adaptive Training.

  • Results

    Experiments validate DragNUWA’s superior performance in fine-grained video control and synthesis.

  • Takeaways & Limitations

    DragNUWA provides fine-grained, user-friendly control from semantic, spatial, and temporal perspectives while generating videos consistent with complex trajectories.

  • Takeaways & Limitations

    The study excludes video conditioning, which requires users to provide specific video materials and can significantly constrain video creation.

Abstract

from arXiv · show

Controllable video generation has gained significant attention in recent years. However, two main limitations persist: Firstly, most existing works focus on either text, image, or trajectory-based control, leading to an inability to achieve fine-grained control in videos. Secondly, trajectory control research is still in its early stages, with most experiments being conducted on simple datasets like Human3.6M. This constraint limits the models' capability to process open-domain images and effectively handle complex curved trajectories. In this paper, we propose DragNUWA, an open-domain diffusion-based video generation model. To tackle the issue of insufficient control granularity in existing works, we simultaneously introduce text, image, and trajectory information to provide fine-grained control over video content from semantic, spatial, and temporal perspectives. To resolve the problem of limited open-domain trajectory control in current research, We propose trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to control trajectories in different granularities, and an Adaptive Training (AT) strategy to generate consistent videos following trajectories. Our experiments validate the effectiveness of DragNUWA, demonstrating its superior performance in fine-grained control in video generation. The homepage link is \url{https://www.microsoft.com/en-us/research/project/dragnuwa/}

1 INTRODUCTION

DragNUWA addresses the limited granularity of controllable video generation by jointly using text, image, and trajectory controls. Its trajectory framework targets open-domain, multiscale, and consistent control, with experiments reporting superior fine-grained video synthesis.

  • Existing language and image controls remain limited in representing temporal information such as camera movements and complex object trajectories.
  • Text, image, and trajectory controls regulate video content from semantic, spatial, and temporal perspectives, respectively.
  • DragNUWA is an end-to-end model that integrates Text, Image, and Trajectory controls for strong, user-friendly controllability.
  • Trajectory Sampler, Multiscale Fusion, and Adaptive Training support arbitrary trajectories, different control granularities, and consistent trajectory-following videos.
  • DragNUWA’s extensive experiments demonstrate superior performance in fine-grained control for video synthesis.

2 RELATED WORKS

Related work progressively added image, text, and video conditions, but these approaches remain limited in representing complex motion or require substantial video input. DragNUWA adds trajectory control to enable fine-grained open-domain manipulation of multiple objects and camera movements.

  • Text and images convey semantics and appearance but struggle to represent complex motion information and camera movements.
  • DragNUWA adds trajectory control to text and image control for fine-grained control over semantics, appearance, and motion.
  • Video-to-video methods provide rich control information but require users to supply video input, restricting practical fine-grained control.
  • DragNUWA enables dragging arbitrary image objects, controlling multiple objects, and accommodating complex trajectories and camera movements.

3 METHOD

DragNUWA integrates text, image, and trajectory controls while addressing open-domain trajectory modeling through sampling, multiscale fusion, and adaptive training. Its pipeline extracts and enhances trajectories, fuses controls in the UNet, and generates consistent videos from varied condition combinations.

  • DragNUWA incorporates text, image, and trajectory controls for semantic, spatial, and temporal video control.
  • 3.1 TRAJECTORY SAMPLER (TS): Trajectory Sampler samples arbitrary trajectories from open-domain optical flow and supports multiple trajectories.It sparsifies and tracks flow-selected points, then applies Gaussian filtering to obtain an enhanced trajectory map.
  • 3.2 MULTISCALE FUSION (MF): Multiscale Fusion downsamples image and trajectory conditions and integrates them with text within UNet blocks at different resolutions.Image and trajectory features are projected into hidden states, while text is injected through prompt cross-attention.
  • 3.2 MULTISCALE FUSION (MF): Mixed-condition training randomly omits text, images, and trajectories so inference can support different condition combinations.Dropped text is replaced with empty strings, while dropped images and trajectories use all-zero frames.
  • 3.3 ADAPTIVE TRAINING (AT): Adaptive Training first uses dense optical flow for consistency, then adapts the model to sparse user-friendly trajectories.Despite trajectory sparsity, the model generates dynamics consistent with trajectories while retaining stability and consistency.
  • 3.4 INFERENCE: During inference, DragNUWA processes text, image, and trajectory inputs to generate realistic and contextually consistent videos.

4 EXPERIMENTS

DragNUWA is evaluated for open-domain trajectory controllability and fine-grained control using text, image, and trajectory conditions. The experiments demonstrate camera movements, complex multi-object trajectories, and the distinct contributions of the three controls.

  • 4.1 DATASETS: DragNUWA is trained with WebVid’s 10 million web videos and a manually filtered VideoHD dataset of 75K high-resolution clips.WebVid provides diverse real-world motion patterns, while VideoHD is annotated with BLIP2 and manually filtered.
  • 4.3 TRAJECTORY CONTROLLABILITY: The experiments test trajectory control through camera movements and complex trajectories rather than focusing only on text or image control.Camera movement and complex-trajectory evaluations address distinct aspects of trajectory controllability.
  • 4.3 TRAJECTORY CONTROLLABILITY: DragNUWA learns horizontal, vertical, and zoom camera movements from open-domain trajectories despite not explicitly modeling camera motion.Changing dragging trajectories while keeping text and images fixed produces different camera movement effects, including zooming.
  • 4.3 TRAJECTORY CONTROLLABILITY: DragNUWA supports curved trajectories, variable trajectory lengths, and simultaneous trajectory control for multiple objects.Longer trajectories produce larger motion amplitudes, and the tests use identical text and images while varying the drag trajectories.
  • 4.4 ESSENTIAL OF THREE CONTROLS: Text, image, and trajectory controls regulate semantic, spatial, and temporal aspects, while their combination enables specifying both surfing on snow and rightward motion.Image-only control leaves background and movement imprecise, text-only control leaves appearance and dynamics uncontrollable, and trajectory supplies motion direction.
  • 4.4 ESSENTIAL OF THREE CONTROLS: Video conditioning is outside this study’s scope because it provides excessive information, constrains video creation, and requires users to supply specific video materials.The paper frames video conditioning primarily as useful for style transfer rather than its fundamental conditions.

5 CONCLUSION

The paper presents DragNUWA as an end-to-end model combining text, image, and trajectory inputs for fine-grained control. Its TS, MF, and AT trajectory framework addresses open-domain trajectory control, and experiments validate effective coherent video generation along complex trajectories.

  • 5 CONCLUSION: DragNUWA integrates text, image, and trajectory inputs for fine-grained, user-friendly control from semantic, spatial, and temporal perspectives.The model is presented as an end-to-end video generation system.
  • 5 CONCLUSION: Its trajectory framework combines Trajectory Sampler, Multiscale Fusion, and Adaptive Training to address open-domain control and generate coherent videos following complex trajectories.The three components respectively support arbitrary trajectories, different trajectory granularities, and consistency during trajectory following.
  • 5 CONCLUSION: Experiments validate DragNUWA’s ability to generate fine-grained videos effectively.
Loading 2308.08089v1…