Source-linked AI summary
MotionInsight: Diagnosing Object Motion Deficiencies in Generated Videos
Jiahao Zhan, Yongrui Ma, Qunliang Xing, Xuanyu Zhang, Jingqi Tong, Junlin Li, Li zhang, Shijie Zhao, Tianfan Xue
TL;DR
Video evaluation emphasizes aesthetics and text-video alignment, leaving object-motion fidelity and subtle motion deficiencies less attended to. MotionInsight combines VidMotion’s fine-grained annotations with explicit motion-space diagnosis and motion-specific GRPO rewards. It produces human-aligned dimension-wise scores and grounded explanations, with reported sensitivity to localized motion failures.
Problem
Video evaluation emphasizes aesthetics and text-video alignment, while giving limited attention to stable, continuous, physically reasonable object motion and often missing subtle deficiencies in RGB-frame observations.
Source: §1 · pp. 1–2Method
VidMotion annotates designated moving objects with three-dimensional scores and failure causes; MotionInsight uses motion-aware representations, semantic alignment, and motion-specific GRPO rewards for diagnosis.
Source: §1 · p. 2Results
Takeaways & Limitations
Object consistency, motion continuity, and physical plausibility provide dimensions for diagnosing object motion, and MotionInsight supports scoring and interpretable failure diagnosis.
Source: §6 · pp. 8–9VidMotion and MotionInsight may be less suitable for phenomena without clear boundaries, persistent identities, and trackable trajectories; reward-model use also risks generator overfitting.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Despite rapid progress in video generation models, they still exhibit obvious motion deficiencies, often manifested as incorrect object motion. However, most existing video quality evaluations focus on aesthetic quality or text-video alignment. To address this gap, we study object-centric motion fidelity assessment, evaluating target objects along object consistency, motion continuity, and physical plausibility. To achieve this, we first introduce VidMotion, a diagnostic dataset of 6,879 videos with designated moving objects and fine-grained annotations including dimension-wise scores and failure causes. We further propose MotionInsight, a diagnostic evaluator that shifts assessment from implicit RGB-frame observation to explicit motion-space diagnosis. By constructing motion-aware representations, MotionInsight makes subtle motion deficiencies more observable. We also introduce motion-specific rewards during GRPO to transform observed motion into a diagnostic assessment. Experiments demonstrate that MotionInsight provides an effective basis for diagnosing object motion deficiencies, producing human-aligned scores along three dimensions and grounded explanations.
1 Introduction
Video evaluators give limited attention to whether objects move stably, continuously, and physically plausibly, even when generated videos look compelling. The paper introduces object-centric diagnosis and MotionInsight, which represents motion explicitly and is reported to align with human judgments.
1 Introduction
- Current video evaluation emphasizes aesthetic quality and text-video alignment, while motion deficiencies can persist even in visually compelling videos.The paper cites film production and world modeling as applications affected by such deficiencies.Source: §1 · p. 1
- VidMotion contains generated and real videos with designated moving objects, dimension-wise scores, and failure causes, supporting fine-grained diagnosis.It includes 5,713 generated videos from nine models and 1,166 real videos.Source: §1 · p. 2
- Existing evaluators often miss subtle motion deficiencies because sampled RGB frames provide partial temporal observations and substantial background redundancy.Missed local evidence can also limit grounded explanations.Source: §1 · p. 2
- MotionInsight builds motion-aware representations from object tracking features and global camera motion, then uses motion-specific GRPO rewards derived from multi-dimensional scores and failure-cause supervision.The representations are aligned with the VLM’s semantic space using automatically generated motion descriptions.Source: §1 · p. 2
- MotionInsight aligns well with human judgments, producing accurate dimension-wise scores, grounded diagnostic reasoning, and sensitivity to localized motion failures.The evaluation focuses on object consistency, motion continuity, and physical plausibility.Source: §1 · pp. 1–2
2 Related Work
Prior video-generation evaluation emphasizes overall visual quality and text-to-video alignment, while motion-focused approaches often rely on task-specific signals or sampled RGB frames. The paper positions explicit motion-space diagnosis as an alternative for broader object-motion assessment.
2 Related Work
- Optical-flow metrics may lose temporal correspondence when objects disappear, while track filtering, 3D human models, and camera-constraint checks use task-specific priors.The cited approaches include VBench, VMBench, HumanScore, and WorldScore.Source: §2 · p. 2
- These task-specific priors constrain motion methods to narrow domains, while rule-based or constrained-scenario approaches do not serve as general evaluators for object-motion deficiencies.The passage cites VBench, VMBench, HumanScore, and WorldScore, alongside work on physical plausibility and world modeling.Source: §2 · pp. 2–3
- VLM-based video evaluators still rely on sampled RGB frames, where subtle pixel-level motion changes can be diluted during visual-token aggregation.MotionInsight instead models object motion in motion space to increase sensitivity to local artifacts.Source: §2 · p. 3
3 VidMotion Dataset
VidMotion is an object-centric motion-fidelity dataset that pairs designated moving objects with dimension-wise scores and failure causes. Its construction pairs real videos with generated counterparts, and its annotations aggregate judgments from multiple annotators.
3 VidMotion Dataset
- VidMotion contains 6,879 annotated videos with designated objects, three-dimensional motion-fidelity scores, and fine-grained failure causes.Real videos use the same human annotation protocol as generated videos, enabling direct comparison rather than assigning real videos perfect scores by default.Source: §3 · p. 3
3.1 Dataset Construction
- 1,166 screened real videos anchor prompts paired with generated videos; proprietary-model outputs are reserved for challenging prompts to test generalization to unseen generators.The benchmark split contains 1,386 videos associated with 166 challenging prompts; the remaining 5,493 videos form VidMotion-Train.Source: §3.1 · pp. 3–4
3.2 Human Annotation
- Three annotators independently score each video on three dimensions and select failure causes; aggregated scores use MOS and final failure labels use cause intersections.Twenty-one annotators participated, and average Krippendorff’s α across the dimensions was 0.7420.Source: §3.2 · p. 4
4 MotionInsight
MotionInsight assesses a designated object in a video with scores for object consistency, motion continuity, and physical plausibility, plus diagnostic reasoning. It makes motion evidence explicit and aligns VLM-based diagnosis with semantic descriptions and human judgments.
4 MotionInsight
- Given a video and target-object prompt, MotionInsight returns scores for three motion dimensions and diagnostic reasoning; without a target, it identifies salient moving objects and aggregates their scores.The three dimensions are object consistency, motion continuity, and physical plausibility.Source: §4 · p. 4
4.1 Motion-Aware Representations
- MotionInsight combines tracked object motion with camera poses to make motion deficiencies explicit and separate object dynamics from viewpoint changes.It pools variable-count tracked points into a fixed number of object-motion tokens, then interleaves motion embeddings with sampled frames for joint VLM reasoning.Source: §4.1 · pp. 4–5
4.2 Motion Description Alignment
- Automatically generated clip-level motion descriptions, summarized per video, supervise semantic alignment of the motion representations with the VLM.The descriptions are generated from clips of 16 consecutive frames; the collected 80,721 question-answer pairs come from OpenVid.Source: §4.2 · p. 5
4.3 GRPO for Human Preference Alignment
- Two GRPO rewards align scores with human judgments and train the evaluator to connect observed motion deficiencies to interpretable failure causes.The scoring reward uses annotated three-dimensional scores, while the failure-cause reward gives a reward of 1 for selecting an annotated cause and 0 otherwise.Source: §4.3 · pp. 5–6
5 Experiments
Experiments show that MotionInsight aligns with human scoring and produces more grounded failure diagnoses. Its motion-aware representations and motion-specific rewards also improve evaluation across dimensions and sensitivity to localized failures.
5 Experiments
- Evaluation uses VidMotion-Train and VidMotion-Test, score-correlation and failure-cause metrics, general and specialized evaluators, and a volunteer reference.Source: §5 · p. 6
5.1 Results
- MotionInsight achieves scoring performance comparable to a human evaluator across object consistency, motion continuity, and physical plausibility.Source: §5.1 · pp. 6–7, Table 3 · p. 7
- MotionInsight achieves higher Jaccard similarity, precision, and recall for failure-cause grounding than existing VLMs.Source: Table 4 · p. 7, §5.1 · pp. 7–8
- MotionInsight maintains a consistently small score gap across K, while GPT-5.4 and Gemini 3.1 Pro show large gaps that decrease as clips increase.Source: Figure 4 · p. 7, §5.1 · p. 8
5.2 Ablation Study
- Motion-aware representations achieve the best performance across all three dimensions; denser RGB sampling and trajectory overlays yield no clear gains.Denser sampling costs more computation and can dilute localized artifacts with redundant visual tokens.Source: Table 5 · p. 8, §5.2 · p. 8
- Adding failure-cause rewards after scoring rewards strengthens diagnostic reasoning and yields further gains in human-aligned scoring.Source: Table 6 · p. 8, §5.2 · p. 8
6 Conclusion
The conclusion frames object-centric motion fidelity as assessment of designated objects across three dimensions. It presents VidMotion and MotionInsight as a dataset and evaluator for scoring and interpretable failure diagnosis.
6 Conclusion
- Object-centric motion fidelity assessment evaluates object consistency, motion continuity, and physical plausibility.Source: §6 · p. 8
- VidMotion provides a diagnostic dataset with multidimensional scores and failure causes for object motion.Source: §6 · p. 8
- MotionInsight shifts evaluation from RGB-frame observations to explicit motion-space diagnosis and demonstrates strong scoring and interpretable failure diagnosis.Source: §6 · p. 9
Limitations
The paper identifies two boundaries: VidMotion and MotionInsight are less suited to ambiguous, hard-to-track phenomena, and using MotionInsight as a fixed reward model may invite generator overfitting.
Limitations
- VidMotion and MotionInsight may be less suitable for fluids, smoke, fire, splashes, and highly deformable materials with ambiguous object boundaries or trajectories.The stated scope centers on entities with clear spatial boundaries, persistent identities, and trackable motion trajectories.Source: Limitations · p. 9
- A fixed MotionInsight reward model may let generators raise scores by overfitting its scoring patterns without improving real object-motion fidelity.The evaluator does not automatically evolve as generators advance.Source: Limitations · p. 9
Ethical Considerations
The work describes its research-use and privacy safeguards, its annotator recruitment and consent process, and agreement across the motion annotations.
Ethical Considerations
- VidMotion uses open-source and curated video sources for academic research, screens videos for sensitive content, and releases annotations without annotator identities or personal information.The work states that external artifacts are used according to their licenses and terms, and that released artifacts are intended solely for research.Source: Ethical Considerations · p. 9
A Annotation Details
- All 21 annotators had at least a bachelor’s degree, were fairly compensated, and consented after being informed of the academic research purpose.They were students and researchers experienced in video understanding.Source: A Annotation Details · p. 12
- Krippendorff’s α was 0.7981 for object consistency, 0.6846 for motion continuity, and 0.7432 for physical plausibility; failure-cause Jaccard similarity was 0.61.The paper characterizes the scalar-dimension agreement as reliable and the failure-cause agreement as reasonable despite fine-grained ambiguity.Source: A Annotation Details · p. 12
B VidMotion Analysis
VidMotion analysis reports diverse target objects and recurring motion failures involving deformation, unnatural movement, and physical causality.
B VidMotion Analysis
- VidMotion covers diverse everyday dynamic objects; common failures include fast-motion deformation, stiff or unnatural continuity, and motion misaligned with observed force.The analysis attributes these patterns respectively to object consistency, motion continuity, and physical plausibility.Source: B VidMotion Analysis · p. 12
D More Qualitative Results
MotionInsight’s examples illustrate diagnosis across appearance stability, temporal continuity, and physical fidelity. Preference optimization using its scores is strongly preferred in a human study and reduces reported motion artifacts.
D More Qualitative Results
- A hockey ball flashing backward after impact illustrates a continuity failure, while a skateboard moving without clear applied force illustrates a physical-plausibility failure.The examples support evaluation progressing from appearance stability to temporal continuity and physical fidelity.Source: D More Qualitative Results · p. 12
E Implementation Details
- The implementation uses Qwen-3-VL-8B-Instruct, eight learnable motion queries, and RGB-frame sampling at a stride of 16 frames.Motion encoding and GRPO training each run for five and three? The passage specifies three epochs for modality alignment and five epochs for GRPO.Source: E Implementation Details · p. 12
F MotionInsight for Generation
- For each prompt, eight candidate videos are sampled and salient objects are scored across three dimensions; their normalized scores are multiplied, then the minimum object reward gives the video-level reward.The highest- and lowest-reward candidates form positive and negative DPO samples, respectively.Source: F MotionInsight for Generation · p. 12
- A VideoPhy2-scored preference set is built from the same candidate pool as a comparison DPO baseline.Both preference sets are used to fine-tune the base generator.Source: F MotionInsight for Generation · pp. 12–13
- MotionInsight-derived DPO preferences were strongly preferred over both Wan 2.1 and VideoPhy2-DPO, while reducing reported deformation and foot-penetration artifacts.The 2AFC study involved 20 volunteers, and the paper reports preservation of overall video quality.Source: F MotionInsight for Generation · p. 13
G Ablation on Motion-Aware Representations
The ablation tests whether MotionInsight benefits from temporally structured motion and camera-pose information, finding that both contribute to performance across the three dimensions.
G Ablation on Motion-Aware Representations
- Shuffling motion-embedding order substantially reduces performance across all three dimensions.The ablation indicates the model relies on temporally structured motion, not merely additional features.Source: G Ablation on Motion-Aware… · p. 13
- Keeping only object-motion tokens underperforms the full model.The comparison indicates camera poses also matter for disentangling object dynamics from camera-induced motion.Source: G Ablation on Motion-Aware… · p. 13
- Together, the ablations support using both temporal structure and camera poses in motion-aware representations.The reported findings cover temporal ordering and the combination of object motion with camera poses.Source: G Ablation on Motion-Aware… · p. 13