Source-linked AI summary
DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data
Wonjoon Jin, Jiyun Won, Janghyeok Han, Qi Dai, Chong Luo, Seung-Hwan Baek, Sunghyun Cho
TL;DR
Video diffusion models lack sufficient examples of highly dynamic motion and face difficult fine-grained control, especially for extreme camera movement. DynaVid trains a two-stage optical-flow-to-video framework with synthetic motion and real-world appearance data, improving dynamic-motion realism and camera controllability across challenging scenarios.
Problem
Highly dynamic motions and precise camera control remain difficult because common training data underrepresent such motions and accurate extreme-motion camera poses are unreliable.
Method
DynaVid renders synthetic optical flow, then uses a motion generator followed by a motion-guided video generator, combining synthetic motion supervision with real-world video training.
Results
DynaVid outperforms existing video diffusion models in dynamic motion generation and camera-controlled synthesis while maintaining visual realism and controllability.
Takeaways & Limitations
Separating motion from appearance lets DynaVid learn highly dynamic patterns from synthetic data while preserving realistic video generation from real-world data.
Takeaways & Limitations
The predominantly single-person synthetic dataset limits performance on videos containing multiple people with highly dynamic motions.
Abstract
from arXiv · showhide
Despite recent progress, video diffusion models still struggle to synthesize realistic videos involving highly dynamic motions or requiring fine-grained motion controllability. A central limitation lies in the scarcity of such examples in commonly used training datasets. To address this, we introduce DynaVid, a video synthesis framework that leverages synthetic motion data in training, which is represented as optical flow and rendered using computer graphics pipelines. This approach offers two key advantages. First, synthetic motion offers diverse motion patterns and precise control signals that are difficult to obtain from real data. Second, unlike rendered videos with artificial appearances, rendered optical flow encodes only motion and is decoupled from appearance, thereby preventing models from reproducing the unnatural look of synthetic videos. Building on this idea, DynaVid adopts a two-stage generation framework: a motion generator first synthesizes motion, and then a motion-guided video generator produces video frames conditioned on that motion. This decoupled formulation enables the model to learn dynamic motion patterns from synthetic data while preserving visual realism from real-world videos. We validate our framework on two challenging scenarios, vigorous human motion generation and extreme camera motion control, where existing datasets are particularly limited. Extensive experiments demonstrate that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.
1. Introduction
DynaVid addresses scarce training examples and unreliable control data for highly dynamic motion by combining synthetic optical-flow supervision with real-world appearance learning. Its two-stage framework targets vigorous human motion and extreme camera control while preserving realism.
- Highly dynamic motions are underrepresented in large video datasets, while manually collecting balanced examples is labor-intensive and difficult to scale.
- Extreme camera-motion control is difficult because accurate 3D camera poses are unreliable when rapid viewpoint changes produce minimal frame overlap.
- Rendered synthetic videos provide dynamic motion and precise controls but can cause models to reproduce artificial textures, lighting, and shadows.
- DynaVid uses rendered optical flow instead of synthetic videos, decoupling motion from appearance to reduce domain gaps and preserve visual realism.
- The two-stage framework synthesizes optical flow first and then generates RGB frames conditioned on that motion, training the stages with synthetic and real data.
- A control branch conditions motion generation on camera signals, enabling flow maps that represent rapidly changing viewpoints for camera-controlled synthesis.
- Experiments cover vigorous human motion and rapid camera motion, with reported improvements in dynamic generation and fine-grained camera control.
2. Related Work
Related work spans video diffusion, explicit object-motion control, camera-controlled synthesis, synthetic-data training, and multimodal flow-to-video pipelines. DynaVid combines synthetic motion supervision with enhanced controllability, addressing limitations in prior approaches.
- Video diffusion models learn spatio-temporal structure from large datasets but still struggle with highly dynamic scenes because available motion diversity is limited.
- Explicit motion-control methods condition video generation on signals such as 2D poses or motion trajectories.
- Camera-controlled models commonly train on paired video frames and camera parameters, using representations such as Pl¨ucker embeddings or depth estimates.
- Synthetic rendering has been used for dense prediction and generative modeling, including controllable material and object-motion synthesis.
- Prior two-stage optical-flow/RGB pipelines mainly rely on real data and provide limited motion controllability, whereas DynaVid adds synthetic motion and a control branch.
3. Method
DynaVid constructs synthetic motion datasets and uses a two-stage flow-to-video architecture trained with complementary synthetic and real data. Optical flow is normalized, encoded through HSV and a pretrained VAE, and used to guide realistic video generation and camera control.
- DynaVid Datasets: The dataset pipeline constructs 3D scenes, defines camera trajectories, and renders synthetic RGB videos with corresponding optical-flow maps.
- DynaVid Datasets: DynaVid-Human combines animatable human characters with Mixamo motion sequences, while DynaVid-Camera uses complex environments and NURBS-interpolated trajectories.
- DynaVid Datasets: Rendered optical flow is computed from visible surface-point displacement and projected onto the image plane as pixel-wise motion vectors.
- Video and Motion Representations: Flow vectors are normalized using a dataset-level scale factor equal to the 99th percentile of flow magnitudes, preserving direction while equalizing magnitudes.
- DynaVid Framework: The framework decouples motion synthesis from RGB video synthesis: a motion generator produces optical flow, and a video generator conditions on that flow.
- DynaVid Framework: Pl¨ucker embeddings condition camera-controlled motion generation, while optical flow supplies control to the motion-guided video generator.
- Training DynaVid: Training combines synthetic and real data, using synthetic flows for dynamic motion supervision and real paired videos and flows for natural appearance learning.
- Training DynaVid: A flow-cycle-consistency filter removes real motion-video pairs with large estimation errors before training the motion-guided video generator.
4. Experiments
Experiments evaluate DynaVid on dynamic object motion and extreme camera control, using real and synthetic datasets with visual, motion, and controllability metrics. Results show improved dynamic-motion realism and camera-path fidelity, while ablations clarify the roles of synthetic motion, real flow, and filtering.
- Evaluation setup: Experiments evaluate object motion synthesis on Pexels and DynaVid-Human, and camera-controlled synthesis on RE10K and DynaVid-Camera.The datasets represent common scenes and highly dynamic motions or viewpoint changes.
- Evaluation setup: The evaluation measures visual quality, motion quality, camera controllability, and motion fidelity using FVD, VBench metrics, rotation error, and flow-based mean squared error.Visual metrics include FVD, aesthetic quality, and imaging quality; motion metrics include smoothness and temporal flickering.
- Dynamic object motion: DynaVid captures natural, highly dynamic object motions more successfully than CogVideoX, Wan2.2, and HyperMotion in qualitative comparisons.CogVideoX and Wan2.2 often produce unrealistic movements or artifacts, while HyperMotion can produce artificial-looking frames because it uses the first frame as a visual prior.
- Dynamic object motion: DynaVid achieves comparable or superior visual and motion quality across common and highly dynamic object-motion datasets, although HyperMotion performs comparably in dynamic scenarios.HyperMotion exhibits degraded visual quality on common scenes and lower FVD partly because its first-frame input increases appearance similarity to evaluation data.
- Extreme camera control: DynaVid significantly outperforms both camera-control baselines on rapidly changing trajectories while remaining comparable on RE10K's moderate viewpoint changes.Qualitatively, DynaVid follows challenging camera paths, whereas AC3D fails on extreme rotations and GEN3C produces artifacts in unseen regions.
- Analysis: Removing synthetic motion data substantially worsens DynaVid-Human quality, whereas synthetic-only training overfits motion patterns and real flow preserves general motion priors.The ablation reports much higher FVD on highly dynamic generation without synthetic motion, with only a minor FVD change on Pexels.
- Analysis: Flow-cycle-consistency filtering improves visual alignment in highly dynamic videos despite similar quantitative results, while rendered-video training produces artificial appearances.Without filtering, generated body parts become misaligned with input motion; rendered synthetic videos can obtain lower FVD by resembling synthetic evaluation data.
- Analysis: The method can extend beyond single-person human training scenes, but it often struggles with multiple people performing highly dynamic motions.The authors identify greater diversity and scale in the synthetic dataset as a possible mitigation.
5. Conclusion
DynaVid generates realistic videos with highly dynamic motions and controllable camera movements by separately modeling motion generation and motion-guided video synthesis. Experiments show improvements over existing video diffusion models while retaining robustness to motion errors and moderate out-of-domain scenarios.
- DynaVid targets realistic video synthesis with highly dynamic motions and controllable camera movements.
- Its two-stage design separately models motion generation and motion-guided video synthesis using synthetic and real-world data.
- DynaVid outperforms existing video diffusion models in dynamic motion generation and camera-controlled synthesis.
- The framework maintains robustness to motion generation errors and moderate out-of-domain scenarios.