Source-linked AI summary
DragAnything: Motion Control for Anything using Entity Representation
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, Di Zhang
TL;DR
Controllable video generation needs precise temporal motion control, while trajectory points may not adequately represent the intended entity. DragAnything uses latent diffusion features as open-domain entity representations for motion control, achieving state-of-the-art results and 26% higher human voting than the previous method for motion control.
Problem
Controllable video generation requires precise temporal motion control, but trajectory points may fail to represent the entity intended for manipulation.
Method
DragAnything extracts latent diffusion features indexed by entity masks to represent entities and control motion by manipulating their spatial positions.
Results
26% higher human voting for motion control than the previous method is reported, alongside state-of-the-art FVD, FID, and User Study performance.
Takeaways & Limitations
The open-domain entity representation supports motion control for diverse entities, including backgrounds, and distinct control of multiple objects.
Takeaways & Limitations
The method remains limited to 2D motion control and depends on the foundation model’s ability to generate large motions.
Abstract
from arXiv · showhide
We introduce DragAnything, which utilizes a entity representation to achieve motion control for any object in controllable video generation. Comparison to existing motion control methods, DragAnything offers several advantages. Firstly, trajectory-based is more userfriendly for interaction, when acquiring other guidance signals (e.g., masks, depth maps) is labor-intensive. Users only need to draw a line (trajectory) during interaction. Secondly, our entity representation serves as an open-domain embedding capable of representing any object, enabling the control of motion for diverse entities, including background. Lastly, our entity representation allows simultaneous and distinct motion control for multiple objects. Extensive experiments demonstrate that our DragAnything achieves state-of-the-art performance for FVD, FID, and User Study, particularly in terms of object motion control, where our method surpasses the previous methods (e.g., DragNUWA) by 26% in human voting.
1 Introduction
Controllable video generation requires both spatial manipulation and precise temporal motion control, but trajectory-based methods can mistake a pixel or pixel region for the intended entity. DragAnything addresses this with latent entity representations for precise, flexible motion control.
- Controllable video generation must coordinate spatial content manipulation with precise temporal motion control.
- Trajectory drawing is a user-friendly alternative to labor-intensive guidance signals such as masks or depth maps.
- Existing methods encode dragged pixels, pixel regions, sparse strokes, or trajectory coordinates as motion guidance.
- A single trajectory point may not represent the intended object, making pixel-level dragging imprecise and potentially confusing foreground motion with background motion.
- DragAnything extracts latent diffusion features from an entity mask to represent entities and controls motion by manipulating the corresponding latent feature positions.
- DragAnything supports training with videos, trajectory points, and first-frame entity masks, using CoTrack to predict entity motion trajectories.
- DragAnything introduces entity-level motion control distinct from pixel-level motion control.
- 26% higher human voting for motion control is reported against the previous method, alongside state-of-the-art FVD, FID, and User Study performance.
2 Related Works
Related work spans image and video generation, controllable video methods using varied control signals, and trajectory-based motion control. Existing trajectory representations remain limited in semantic entity modeling, motivating DragAnything’s entity-level approach.
- Controllable image generation uses guidance such as edges, lines, scribbles, pose points, and segmentation maps.
- Video generation has progressed more slowly than image generation, with video diffusion models extending diffusion architectures to generate video sequences.
- Controllable video methods condition generation on signals including edge maps, depth maps, and human pose.
- Trajectory-based methods use optical flow, recurrent networks, bounding boxes, dense flow, or vector maps to guide object motion.
- Box representations handle instance-level objects but not backgrounds, while trajectory maps overlook entity semantics because a single point cannot adequately represent an entity.
3 Methodology
DragAnything addresses the difficulty of controlling whole entities with trajectory-based video generation by replacing pixel-level guidance with entity representations. Its framework extracts entity features, associates them with trajectories and Gaussian representations, and uses masked training for targeted motion control.
- Motivation: Trajectory points may fail to represent the intended entity, causing imprecise or unintended motion.The toy experiment reports that dragging a cloud pixel can move the camera rather than the cloud.
- Motivation: Pixel-level trajectory control can produce larger motion near the drag point instead of moving the object as a whole.DragNUWA is described as dragging corresponding pixel regions, which can cause abnormal appearance deformation.
- Entity Representation: DragAnything represents each target entity with latent diffusion features extracted using the entity mask, enabling entity-level motion control.The extracted features are indexed from the first-frame mask and averaged to form entity embeddings.
- Representation and Control: The framework inserts entity embeddings along trajectory points and combines them with 2D Gaussian representations to emphasize central regions.The Gaussian representation assigns greater weight to pixels closer to an entity’s center.
- Training and Inference: Training uses masked loss so optimization targets the selected object while preserving generation quality for other objects and the background.The mask restricts MSE backpropagation to the areas being optimized.
- Training and Inference: During inference, users select a region with SAM and drag any pixel inside it to specify the desired trajectory.The interaction requires selecting the controllable region and drawing a trajectory.
4 Experiments
Experiments evaluate DragAnything on video quality, temporal coherence, object motion, and human preference, alongside ablations and diverse motion-control scenarios. DragAnything generally outperforms DragNUWA and supports foreground, background, simultaneous, and camera motion control.
- Evaluation Setup: Evaluation uses FID, FVD, ObjMC, and human voting to assess video quality, temporal coherence, object motion, and user preference.
- State-of-the-Art Comparisons: 33.5 FID outperforms DragNUWA’s 39.8 on the VIPSeg validation set under the same base architecture.
- State-of-the-Art Comparisons: 494.8 FVD improves on DragNUWA’s 519.3, indicating stronger temporal coherence on VIPSeg validation videos.
- State-of-the-Art Comparisons: 305.7 ObjMC improves on DragNUWA by 18.9, establishing new state-of-the-art object-motion performance.
- Ablation Studies: Entity Representation improves ObjMC by 92.3 to 318.4, while 2D Gaussian Representation improves it by 71.4 to 339.3; using both reaches 305.7.
- Various Motion Control: DragAnything supports precise foreground motion, background-layer motion, simultaneous foreground-background control, and basic camera zoom and translation.
5 Conclusion
DragAnything addresses limitations of trajectory-point representations by using latent diffusion features to represent entities. Its open-domain representation supports diverse entity motion, including background control, while experiments report state-of-the-art user-study performance.
- Trajectory points may inadequately represent entities, and pixels closer to the drag point can exert stronger influence and produce larger motions.
- DragAnything uses latent diffusion features as an open-domain entity representation for motion control of diverse objects, including backgrounds.
- 26% higher human voting for motion control than DragNUWA is reported as state-of-the-art User Study performance.
- DragAnything still has bad cases, especially when controlling larger motions.
6 Appendix
The appendix discusses ethical risks and technical limitations of DragAnything, including societal bias, misuse, privacy concerns, restricted 2D control, and failures under large motion.
- 6.1 Discussion of Potential Negative Impact.: Training-data biases may be reinforced, while generated videos may be misused for misleading or inappropriate visual materials.The paper also raises privacy concerns when videos involve individuals without explicit consent.
- 6.2 Limitation and Bad Case Analysis: Current trajectory-based motion control is limited to 2D scenes and cannot precisely control actions such as turning around or body rotations.
- 6.2 Limitation and Bad Case Analysis: Large motions can exceed the foundation model’s capabilities, producing implausible structures and degraded video quality.The examples include dinosaurs with extra legs and blurred eagle wings; proposed remedies include 3D trajectory information and stronger foundation models.