Source-linked AI summary
Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representations
Zhixue Fang, Zhimin Zhang, Bi'an Du, Zijie Meng, Yan Zhou, Wei Hu, Guoxin Zhang, Pengfei Wan, Kun Gai
TL;DR
Open-category motion transfer is difficult because fixed structural correspondence breaks down across differing morphologies. Motion Beyond Morphology bootstraps cross-category supervision from complementary motion abstractions and internalizes it for direct source-video-conditioned generation, achieving state-of-the-art motion fidelity and target preservation on OpenVMT-Bench.
Problem
Existing motion-transfer methods rely on compatible structural templates, limiting formulation when source and target differ substantially in morphology, semantics, or motion topology.
Method
A two-stage framework uses complementary motion abstractions to synthesize cross-category supervision, then internalizes it for direct source-video-conditioned motion transfer without explicit motion extraction at inference.
Results
The method achieves state-of-the-art performance on OpenVMT-Bench, with robust motion fidelity and target preservation across diverse source–target relations.
Takeaways & Limitations
OpenVMT-Dataset and OpenVMT-Bench provide cross-category training supervision and evaluation across progressively increasing Same, Near, and Far category gaps.
Takeaways & Limitations
The individual motion abstractions used for explicit supervision have inherently limited coverage and information capacity.
Abstract
from arXiv · showhide
Video motion transfer aims to animate a target object using dynamics from a reference video. Existing formulations largely rely on fixed structural correspondence, which becomes ill-defined when reference and target objects differ substantially in morphology, articulation, or deformation mechanisms. We introduce Motion Beyond Morphology, a perspective that seeks to transfer motion beyond fixed structural correspondence, by preserving dynamics that remain meaningful across different target morphologies. To realize this, we propose a two-stage framework. Stage~I learns complementary multi-granularity abstract motion views and uses them to bootstrap cross-category video pairs that preserve transferable dynamics across diverse morphologies. Stage~II internalizes this supervision into direct reference-video-conditioned generation, removing the need for explicit motion extraction at inference. We further introduce OpenVMT-Dataset and OpenVMT-Bench for training and evaluating image- and text-conditioned motion transfer across Same, Near, and Far category gaps. Extensive experiments demonstrate state-of-the-art motion fidelity and target preservation. Project page: https://miniz233.github.io/MotionBeyondMorphology/
Introduction
The introduction reframes motion transfer as preserving dynamics beyond fixed structural correspondence across open-category morphology gaps. It proposes bootstrapping cross-category supervision with complementary motion abstractions, then internalizing that supervision for direct source-video-conditioned generation, supported by new datasets and benchmarks.
- Problem: Existing motion transfer methods predominantly assume compatible structural templates, such as shared articulated structures and kinematic layouts in human or character animation.These assumptions become problematic when source and target objects differ substantially in morphology, articulation, or deformation mechanisms.
- Method: Complementary motion abstractions address the tradeoff between broad applicability and incomplete motion coverage, including semantic kinematics, trajectories, dense point tracks, structural edges, and 6-DoF axes.Explicit views are used only when their extraction is reliable and serve as mechanisms for constructing cross-morphology supervision rather than a fixed final inference representation.
- Motivation: Open-category motion transfer should adaptively preserve motion factors that remain meaningful across source-target structures while disentangling incompatible factors.The formulation addresses differences in semantics, morphology, and motion topology rather than relying on fixed correspondence.
- Method: The two-stage framework first bootstraps cross-category pairs from abstract motion conditions, then internalizes transferable dynamics into direct source-video-conditioned motion transfer.Stage II replaces the explicit motion condition with the source video, making source-specific appearance and morphology unreliable predictors across category changes.
- Contributions: OpenVMT-Dataset contains 10K motion-equivalent pairs, while OpenVMT-Bench evaluates transfer across progressively increasing category gaps.The dataset provides direct supervision for cross-category motion correspondence and the benchmark supports systematic evaluation.
Related Work
Prior motion-transfer work relies on structural cues or reference-derived motion, but these approaches either assume category-specific correspondence or may retain source-specific shape, layout, or appearance. This work instead uses complementary motion abstractions for diverse objects and introduces a benchmark that explicitly controls structural gaps and target-image availability.
- Structure-conditioned animation: Pose-guided animation uses keypoints, skeletons, or dense pose for precise control, but its structural assumptions do not naturally extend to arbitrary objects.The limitation also applies to landmark-based face and body animation.
- Reference-based motion transfer: Diffusion-based reference-transfer methods extract motion directly from videos using temporal attention, motion flow, diffusion guidance, or motion–appearance decoupling.Examples include MotionClone, DiTFlow, MotionShop, DeT, and DisMo.
- Motion representations and evaluation: Trajectory-, track-, pose-, and hybrid representations capture coarse motion, local dynamics, semantic structure, or multiple complementary cues, respectively.This work selects complementary motion abstractions according to their applicability across diverse objects.
- Motion representations and evaluation: OpenVMT-Bench supports T2V and I2V evaluation across Same, Near, and Far splits for motion fidelity, target preservation, and source leakage.Existing general and motion-transfer benchmarks provide limited control over structural gaps and rarely include curated target images.
Method
The method bootstraps cross-category motion supervision from complementary abstract motion views, then internalizes it so motion transfer can use raw reference videos directly. Its unified pipeline supports text- and image-conditioned targets without explicit motion extraction or test-time optimization.
- Stage I: Abstract motion views: The framework uses complementary abstract motion views to capture transferable dynamics across diverse objects and motion patterns.Views are sampled according to applicability over 100+ object categories and motion patterns, then unified into a video-like form.
- Stage I: Abstract motion views: The views include semantic keypoints, depth-aware global trajectories, dense point tracks, 6-DoF axes, and foreground edges.These representations respectively target articulated motion, object displacement and depth changes, local deformation, rigid orientation changes, and coarse shape evolution.
- Stage I: Abstract motion grounding: Stage I grounds heterogeneous motion abstractions through a shared conditioning interface and generates cross-category candidate pairs with shared motion attributes.The model combines an abstract motion view, first-frame image, and caption to reconstruct videos, then pairs reference videos with target conditions from different categories.
- Stage I: Cross-category pair generation: Stage I converts readily extractable motion abstractions into cross-category supervision that is difficult to obtain directly from real videos.Generated pairs are filtered using motion fidelity, target fidelity, reference leakage, and video quality criteria.
- Stage II: Direct reference-video conditioning: Stage II initializes from Stage I and replaces abstract motion conditions with raw reference videos, enabling direct inference conditioned on the reference video and target condition.The target condition contains text and an optional reference image, covering both T2V and I2V settings; inference eliminates explicit motion extraction and per-video optimization.
Experiments
Experiments evaluate motion transfer across Same, Near, and Far morphology gaps using newly curated datasets and benchmarks. The method achieves leading quantitative and perceptual performance, while ablations support morphology-diverse supervision and complementary motion views.
- Dataset and benchmark: OpenVMT-Dataset provides 10K cross-content motion-equivalent pairs, while OpenVMT-Bench evaluates image- and text-conditioned transfer across Same, Near, and Far gaps.The dataset contains 4K Same, 4K Near, and 2K Far pairs; the benchmark includes 123 I2V and 166 T2V test cases.
- Image-to-video motion transfer: On OpenVMT-I2V, our method leads all reported metrics, combining highest motion fidelity with stronger target preservation, lower source leakage, and better temporal smoothness.It achieves the highest HMF and G-Mot. across Same-to-Far transfers, alongside the best target fidelity, lowest G-Leak, and highest smoothness score.
- Text-to-video motion transfer: On OpenVMT-T2V, our method leads overall HMF, G-Mot., CLIP-T, and G-Leak, although DisMo-T2V slightly leads HMF-F and temporal smoothness.DisMo-T2V’s stronger source-motion adherence is accompanied by lower target alignment and higher source leakage.
- Perceptual evaluation: Human preference rates reach 93.0% on I2V and 97.3% on T2V, with consistent preference for the method in motion fidelity, target preservation, and overall quality.The study used pairwise G/S/B comparisons against DisMo for I2V and FlowMotion for T2V, with 12 evaluators.
- Ablation study: Replacing Stage-II cross-category pairs with in-category pairs reduces HMF-All from 0.7707 to 0.7588 and G-Mot. from 3.976 to 3.436.This ablation confirms the benefit of morphology-diverse supervision.
- Ablation study: The motion views play complementary roles: trajectories favor global displacement, tracks capture local dynamics, and edges provide stronger structural cues.The full model better balances global trajectory, local motion details, pose, and target preservation.
Conclusion
The paper presents Motion Beyond Morphology, a two-stage framework for open-category video motion transfer that constructs cross-category supervision from heterogeneous motion abstractions instead of imposing a fixed inference representation.
- Framework: Motion Beyond Morphology frames open-category video motion transfer around heterogeneous motion abstractions rather than fixed structural representations.The framework uses these abstractions to construct cross-category supervision.
- Stage I: Stage I combines semantic kinematics, depth-aware global trajectories, dense point tracks, 6-DoF axes, and structural edges through a shared conditioning interface.These complementary motion views provide multiple granularities for representing transferable dynamics.
- Stage I: Stage I bootstraps motion-equivalent videos across diverse morphologies for cross-category supervision.The supervision is constructed across varied object morphologies rather than relying on fixed correspondence.