Source-linked AI summary
Motion Attribution for Video Generation
Xindi Wu, Despoina Paschalidou, Jun Gao, Antonio Torralba, Laura Leal-Taixé, Olga Russakovsky, Sanja Fidler, Jonathan Lorraine
TL;DR
Video-generation data influence on motion remains poorly understood. Motive provides motion-centric gradient attribution using motion-weighted loss masks and scalable computation, then uses influential clips for fine-tuning. The method improves motion metrics with selected data and supports targeted motion-focused curation.
Problem
Which training clips influence motion in generated videos remains underexplored, although motion encompasses temporal dynamics distinct from static appearance.
Method
Motive uses motion-weighted loss masks and efficient gradient-based attribution to isolate temporal dynamics and rank influential training clips.
Results
Motive-selected data improves dynamic degree and motion smoothness, with 10% of training data surpassing full fine-tuning on dynamic degree for both evaluated models.
Takeaways & Limitations
The attribution scores enable targeted data curation for improving motion quality with a fraction of the data.
Takeaways & Limitations
Whole-video attribution can dilute informative motion intervals, and motion masks may overemphasize camera-only motion.
Abstract
from arXiv · showhide
Despite the rapid progress of video generation models, the role of data in influencing motion is poorly understood. We present Motive (MOTIon attribution for Video gEneration), a motion-centric, gradient-based data attribution framework that scales to modern, large, high-quality video datasets and models. We use this to study which fine-tuning clips improve or degrade temporal dynamics. Motive isolates temporal dynamics from static appearance via motion-weighted loss masks, yielding efficient and scalable motion-specific influence computation. On text-to-video models, Motive identifies clips that strongly affect motion and guides data curation that improves temporal consistency and physical plausibility. With Motive-selected high-influence data, our method improves both motion smoothness and dynamic degree on VBench, achieving a 74.1% human preference win rate compared with the pretrained base model. To our knowledge, this is the first framework to attribute motion rather than visual appearance in video generative models and to use it to curate fine-tuning data.
1. Introduction
The paper asks which training clips influence motion in generated videos and introduces Motive to attribute temporal dynamics separately from static appearance. Motion-aware attribution supports scalable data curation, improving motion quality with selected fine-tuning data.
- Motion distinguishes video generation by capturing object movement, interactions, and physical constraints beyond single-frame appearance.
- The central question is which training data influence motion in generated videos.
- Motion-specific attribution is especially valuable during fine-tuning, where carefully selected clips can have an outsized impact.
- Naïve video attribution risks collapsing temporal dynamics into appearance and must localize motion, scale across sequences, and capture temporal relations.
- Motive uses motion-aware gradient masking to emphasize dynamic regions, with efficient approximations for large datasets and video models.
- 10% of the data matches or surpasses full-dataset fine-tuning performance, while improving motion smoothness and dynamic degree and outperforming motion-unaware baselines.
2. Background
The background reviews latent diffusion and flow matching, video-specific temporal representations, and gradient-based attribution. It explains why naïve attribution is costly and confounds motion with appearance.
- Latent video generators encode clips into VAE latents and train time-indexed denoisers or velocity fields under diffusion or flow matching.
- Video generation must model spatial appearance alongside object and camera motion, deformations, and interactions using temporal-capable architectures.
- Optical flow represents motion through per-pixel displacement vectors, whose magnitudes summarize motion amount and spatial layout for loss weighting.
- Data attribution estimates how individual training samples affect model predictions, while practical methods approximate influence with gradient inner products or projections.
- Diffusion attribution must address timestep and noise variation because gradient norms can create spurious influence rankings.
- Naïve video attribution overemphasizes appearance, overlooks dynamics, and becomes impractical as clip length, sampling, and gradient dimensionality grow.
3. Method
Motive attributes training clips according to their influence on video motion by combining scalable gradient estimation with motion-aware loss weighting. It corrects video-specific biases, projects gradients for tractable storage, and ranks clips for motion-centric fine-tuning.
- Problem formulation: Motive assigns motion-aware influence scores to fine-tuning clips relative to query videos and selects influential subsets for motion-centric fine-tuning.The framework targets predictivity, efficiency, and motion-centric attribution in modern video generators.
- Scalable gradient-based attribution: The scalable estimator uses gradient similarity, common randomness, a single sampled timestep-noise pair, and Fastfood projection instead of exact inverse-Hessian computation.These choices reduce variance, compute, and storage while preserving ranking behavior close to full-gradient cosine similarity.
- Frame-length bias correction: Frame-length correction divides gradients by the number of frames before projection and normalization, preventing longer clips from receiving biased influence scores.The correction is combined with ℓ2 normalization to stabilize scales across examples.
- Motion attribution: Motion attribution reweights per-location gradients with motion masks, emphasizing temporal dynamics and de-emphasizing static appearance without changing forward noising or generation.Loss-space masking focuses attribution on dynamic regions while leaving the generation process unchanged.
4. Experiment
Experiments evaluate Motive across diverse motion queries, video models, qualitative comparisons, VBench metrics, human judgments, and efficiency ablations. Motive-selected data produces realistic, temporally consistent motion while frame normalization and projected gradients improve attribution reliability and scalability.
- 4.1. Setup: Experiments use VIDGEN-1M and 4DNeX-10M data, 10 motion categories, and Wan2.1-T2V-1.3B and Wan2.2-TI2V-5B models.The evaluation uses 10k videos from each dataset and five query videos per motion category.
- 4.2. Main Results: Motive ranks clips with continuous, physically grounded dynamics while downranking static footage, camera-only motion, and non-transferable cartoon kinematics.This selection pattern holds across motion categories and supports negative filtering.
- 4.2. Main Results: Motive yields higher motion fidelity and temporal consistency than base and random-selection models across four motion scenarios, especially for deformation and physics-driven motion.The comparisons use equal data budgets for the fine-tuned models.
- 4.2. Main Results: 47.6% and 48.3% dynamic degree on Wan2.1-T2V-1.3B and Wan2.2-TI2V-5B exceed random selection at 41.3% and 41.6%, respectively.Motive also reports aesthetic quality of 46.0% and 45.6%, with motion smoothness of 96.3% and 97.6%.
- 4.3. Human Evaluation: 74.1% win rate against the base model and 53.1% against the full fine-tuned model show human preference for Motive-guided motion.The pairwise evaluation uses 17 annotators, 10 motion categories, and 50 test cases.
- 4.4. Ablations: Frame-length normalization reduces spurious duration effects by 54.0%, while projected gradients reach 74.7% Spearman correlation at dimension 512.Without normalization, length correlates with attribution scores at 78.0%; dimension 512 offers the stated accuracy-efficiency trade-off.
5. Conclusion
Motive addresses which training clips influence motion in video diffusion by isolating temporal dynamics from static appearance. It enables targeted data curation and supports more controllable generative systems.
- Motive is a motion-aware attribution framework that isolates temporal dynamics from static appearance in video diffusion.
- Tracing generated motion to influential training clips enables targeted curation that improves motion quality with a fraction of the data.
- Data-level understanding can help diagnose failure modes and build more controllable generative systems as video models scale.
B.1. Data Attribution
Data attribution estimates how training examples influence model behavior, but video generation introduces temporal structure that image-focused methods do not capture. Motive addresses this gap with motion-aware attribution and evaluates its selection against baselines.
- Existing attribution methods include retraining-based, Shapley-value, and gradient-based approaches such as Influence Functions, TracIn, and TRAK.
- Data attribution estimates how individual training examples contribute positively or negatively to model predictions or behavior on a query.
- Video generation requires coherent motion across frames, while image-oriented attribution methods primarily explain static content.
- Motive is evaluated alongside selection methods on VBench, using 10% of training data and including motion masking as a condition.
- Gradient-based influence scores identify clips that directly improve target motion dynamics rather than merely selecting videos with more overall motion.
C.1. Results on Additional Video Generation Models
Motive was evaluated on LTX-2B, an architecture and training paradigm distinct from the Wan models. The results show effectiveness across different architectures and scales.
- Motive works effectively on LTX-2B, demonstrating applicability beyond the Wan models used in the main evaluation.
- The additional evaluation covers a different architectural design and training paradigm, supporting cross-architecture use of the framework.
D.1. Motion Distribution Analysis
Motive’s influential clips span the full motion spectrum rather than simply concentrating on motion-rich videos. Cross-motion overlaps reveal both shared and specialized influential data patterns across datasets.
- Motion Distribution Analysis: Motive’s influence scores are gradient-based and select clips that help lower the loss for target motion dynamics, not clips with the most motion overall.
- Motion Distribution Analysis: 3.85 is the mean motion magnitude for the top 10% influential videos, only 4.3% higher than 3.69 for the bottom 10%.
- Motion Distribution Analysis: High- and low-influence videos both occur in low, moderate, and high motion bins, showing that influential clips span the entire motion spectrum.
- Cross-Motion Influence: Top-100 influential samples show mean cross-motion overlaps of 24.0% in 4DNEX and 24.3% in VIDGEN.
- Cross-Motion Influence: Bounce-float, compress-float, and compress-spin have high overlap, while free fall-stretch and float-slide have lower overlap across the two datasets.
- Cross-Motion Influence: Similar cross-motion patterns across 4DNEX and VIDGEN suggest relationships that generalize across datasets and reflect dynamic similarity.
- Attribution Procedure: Motive extracts motion masks, computes motion-weighted gradients, normalizes them by frame length, projects them, and ranks clips by gradient similarity.
E. Additional Method Details
Motive supports flexible attribution across motion estimators and video-generation objectives. Its pipeline summarizes motion-weighted gradients, projects them into a compact space, and ranks training clips by influence.
- Motive treats the motion estimator as a pluggable saliency source, allowing dense optical flow or point tracking without changing attribution code.Displacement magnitudes are mapped and normalized into latent-space weights.
- The attribution applies to diffusion and flow-matching objectives because it requires only per-example gradients under matched fixed timestep and noise conditions.The influence score is a gradient inner product under a fixed preconditioner.
- The full pipeline computes motion-weighted gradients, projects them into a low-dimensional space, then ranks and selects influential training clips.
F.1. Hyperparameter Settings
The experiments document attribution, selection, fine-tuning, and evaluation settings, including fixed optimization choices and motion-query prompts spanning controlled categories.
- Attribution uses a fixed midpoint timestep, shared Gaussian noise, and Fastfood projection from D=1 418 996 800 to D′=512.The projection balances performance and storage, while shared noise reduces stochastic variance.
- Fine-tuning selects the top-10% of data, freezes the text encoder and VAE, and uses 480×832 inputs with a 1×10−5 AdamW learning rate.
- Evaluation uses the same 10 motion categories as the query set but different visual appearances.
- The documented query prompts cover controlled motion primitives including compression, bouncing, rolling, floating, and explosions.Examples specify camera, lighting, background, and the intended motion emphasis.
F.2. Details on Motion Query Data
The motion-query data are synthetic, standardized stimuli designed to isolate motion primitives while supporting scalable attribution and repeated query evaluation. The main computational cost is paid once for training gradients and amortized across queries.
- Synthetic queries isolate motion primitives by minimizing textured backgrounds and uncontrolled camera motion, then screening outputs for realism and physical plausibility.
- The query set is used for attribution targets and multi-query aggregation, not as training data.Synthetic generation provides controllable, standardized stimuli that are difficult to obtain from web videos.
- The query set contains near-realistic Veo-3 videos across ten motion categories, with five manually screened videos per category.Controlled prompts target clarity and physical plausibility.
- For 10k samples, one-time training-gradient computation dominates runtime, while new queries require a single gradient pass and lightweight similarity computation.The training gradients are projected into 512-dimensional vectors and reused across queries.
- Motive has higher upfront cost than baseline selection methods, but amortization and gradient reuse support large-scale multi-query curation.
G.2. Limitations
The limitations concern computational cost, clip-level granularity, motion disentanglement, classifier-free guidance, and possible trade-offs from targeted fine-tuning. The paper proposes several directions for improving robustness, curation, scope, and governance.
- Limitations: Clip-level attribution can dilute informative motion segments when static or redundant portions dominate the same video.The paper identifies motion-segment or motion-event attribution as an open direction.
- Limitations: Motion masks may overemphasize camera-only motion, while full disentanglement of ego and object motion remains future work.The method detects spatially uniform weights and down-weights such clips.
- Limitations: The framework does not explicitly model classifier-free guidance, so influence estimates may not fully capture guidance-altered motion behavior.
- Future Directions: Future work includes tracker-robust saliency, self-generated queries, iterative curation, broader modalities, and safety-oriented negative-influence filtering.
- Motion Visualization: Motion overlays emphasize dynamic content while attenuating static scene elements, providing intuition for the spatial weighting used by the motion loss.