PaperlayerView exact revision on arXiv

Source-linked AI summary

VMBench: A Benchmark for Perception-Aligned Video Motion Generation

Xinran Ling, Chen Zhu, Meiqi Wu, Hangyu Li, Xiaokun Feng, Cundian Yang, Aiming Hao, Jiashu Zhu, Jiahong Wu, Xiangxiang Chu

arXiv:2503.10076v2cs.CV

TL;DR

Video motion evaluation remains challenging because current metrics do not fully align with human perception. VMBench combines perception-driven motion metrics with a broad motion-assessment benchmark; its metrics achieve a 35.3% average improvement in Spearman’s correlation over current methods.

  • Problem

    Existing motion metrics do not fully align with human perception, and prior evaluation approaches often emphasize static content or motion smoothness over broader motion deficiencies.

    Source: §1 · p. 1

  • Method

    VMBench combines a comprehensive motion benchmark with five perception-driven metrics for assessing video motion quality.

    Source: §1 · p. 1

  • Results

    35.3% average improvement in Spearman’s correlation over current methods is reported for the metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VMBench provides a standardized framework, diagnostic tools, and quantitative standards for evaluating motion artifacts against human perceptual expectations.

    Source: §5 · p. 8

Abstract

from arXiv · show

Video generation has advanced rapidly, improving evaluation methods, yet assessing video's motion remains a major challenge. Specifically, there are two key issues: 1) current motion metrics do not fully align with human perceptions; 2) the existing motion prompts are limited. Based on these findings, we introduce VMBench--a comprehensive Video Motion Benchmark that has perception-aligned motion metrics and features the most diverse types of motion. VMBench has several appealing properties: 1) Perception-Driven Motion Evaluation Metrics, we identify five dimensions based on human perception in motion video assessment and develop fine-grained evaluation metrics, providing deeper insights into models' strengths and weaknesses in motion quality. 2) Meta-Guided Motion Prompt Generation, a structured method that extracts meta-information, generates diverse motion prompts with LLMs, and refines them through human-AI validation, resulting in a multi-level prompt library covering six key dynamic scene dimensions. 3) Human-Aligned Validation Mechanism, we provide human preference annotations to validate our benchmarks, with our metrics achieving an average 35.3% improvement in Spearman's correlation over baseline methods. This is the first time that the quality of motion in videos has been evaluated from the perspective of human perception alignment. Additionally, we will soon release VMBench at https://github.com/GD-AIGC/VMBench, setting a new standard for evaluating and advancing motion generation models.

1 Introduction

VMBench addresses motion evaluation gaps with perception-aligned metrics and a broad motion-prompt benchmark. Human preference annotations validate the metrics, which outperform current methods in Spearman’s correlation.

1 Introduction

  • VMBench covers 969 motion categories and integrates perception-aligned metrics for systematically evaluating video motion generation.The paper reports that its category range exceeds other benchmarks –, [22], [23].Source: §1 · p. 1
  • PMM evaluates motion quality across five dimensions, including object integrity, perceptible amplitude, temporal coherence, smoothness, and commonsense adherence.The framework is intended to assess spatiotemporal inconsistencies and violations of physical laws beyond smoothness alone.Source: §1 · p. 1
  • MMPG extracts subjects, places, and actions from datasets, uses LLMs to enrich metadata and generate motion descriptions, then refines prompts through human–AI collaboration.Its six movement modes are fluid dynamics, biological motion, mechanical motion, weather phenomena, collective behavior, and energy transfer.Source: §1 · p. 1
  • VMBench’s prompts structure motion around subject, place, and action, while its evaluation comprises five human-centric quality metrics.Figure 1 presents the benchmark overview and its motion-prompt and evaluation components.Source: §1 · p. 1, Figure 1 · p. 2
  • 35.3% average improvement in Spearman’s correlation over current methods validates VMBench’s metrics against human preference annotations.The comparison is reported against current methods,,, –[43].Source: §1 · p. 2

2 Related work

Related work describes progress in video generation alongside persistent motion-quality deficiencies. Existing evaluation approaches have distinct limitations, motivating five human-aligned metrics for systematic motion assessment.

2.1 Video Generative Models

  • Current T2V models excel at static visual content, but their dynamic performance still deviates from human perception.The related work traces progress from T2I-based models to latent-space diffusion, large-scale data filtering, and open-source efforts.Source: §2.1 · p. 2
  • Recent models and motion-guidance methods improve video generation, yet generated videos still show disappearing subjects, body distortions, and physically implausible actions.Mochi 1 and HunyuanVideo improve dynamic quality through architectural approaches, while other work uses motion information during data selection or sampling.Source: §2.1 · pp. 2–3

2.2 Evaluations Metrics for Video Generation

  • Feature-based metrics can ignore temporal coherence or reduce motion dynamics to distributions, while rule-based metrics risk subjective, fragmentary assessment.These approaches may overlook implausible movement, inertia preservation, and multiobject interaction coherence.Source: §2.2 · p. 3
  • MLLM-based methods align scoring with human judgments but can obscure specific defects, amplify annotation biases, and generalize poorly across domains.The paper introduces five human-aligned metrics to systematically quantify motion quality in response to limitations across evaluation paradigms.Source: §2.2 · p. 3

3 VMBench

VMBench addresses gaps in human alignment of motion metrics and breadth of motion prompts with a benchmark combining perception-aligned evaluation and diverse motion types.

3 VMBench

  • VMBench combines perception-aligned motion metrics with a broad range of motion types to address limitations in existing motion evaluation and prompts.The paper describes existing motion prompts as limited and current motion metrics as insufficiently aligned with human perception.Source: §3 · pp. 3–4

3.1 Perception-Driven Motion Evaluation Metrics

  • PMM assesses motion through commonsense and visual-quality scores, alongside checks for anatomical integrity, subject-motion amplitude, and temporal continuity.Its components are CAS, MSS, OIS, PAS, and TCS; TCS filters legitimate object transitions such as occlusion or frame-boundary exits.Source: §3.1 · pp. 4–5

3.2 Meta-Guided Motion Prompt Generation

  • MMPG extracts subject, place, and action metadata, iteratively generates and filters plausible prompts, then applies human–LLM review to curate 1,050 prompts.The generation process produces approximately 50,000 candidates before filtering and joint validation.Source: §3.2 · pp. 5–6, Figure 3 · p. 5

4 Experiments

The experiments compare motion metrics with human judgments, test PMM’s component contributions, analyze metric correlations, and assess six video-generation models. PMM aligns more closely with human judgments than alternative metrics, while Wan2.1 performs best on motion-video generation.

4.1 Experimental Setup

  • The study evaluates six text-to-video models on 1,050 prompts each and samples 1,200 videos for human-aligned validation.Three domain experts independently rate each sample across five dimensions, yielding 6,000 ratings; metric alignment is assessed with Spearman correlation.Source: §4.1 · p. 6, §4.2 · pp. 6–7

4.2 Comparison with State-of-the-Art

  • PMM reaches 65.2% alignment efficacy for PAS and 54.5% for TCS, while tested alternatives show lower correlations with human perception.The comparison also reports MLLMs averaging 10.0%–30.0% correlation across dimensions.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Ablation Study

  • Removing any single metric lowers accuracy, and removing CAS causes the largest drop, to 64.1%.Progressively adding metrics to performance-oriented variants also significantly improves accuracy.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Qualitative Analysis

  • Human ratings and PMM show matching correlation patterns: OIS correlates positively with MSS (ρ = 0.59) and CAS (ρ = 0.50), while PAS correlates negatively with OIS (ρ = −0.18).TCS has low correlation with other dimensions; the reported matrices compare human annotations and PMM metrics.Source: §4.4 · p. 7, Figure 4 · p. 8
  • Wan2.1 delivers the best motion-video generation performance among six evaluated models, with higher scores indicating better category performance.The six models are Mochi 1, OpenSora, CogVideoX, OpenSora-Plan, HunyuanVideo, and Wan2.1.Source: Table 3 · p. 7, §4.4 · p. 8

5 Conclusion

VMBench provides a benchmark for assessing motion quality through human-aligned metrics and a broad motion evaluation framework. It decomposes motion perception into global motion-field parsing and fine-grained detail capture, while noting that individual viewer preferences may not be fully represented.

5 Conclusion

  • VMBench combines human-aligned motion metrics with a benchmark intended to reveal deficiencies in physically plausible movement, but may miss subtle differences in individual viewer preferences.The conclusion describes the benchmark as an open-source resource with diagnostic tools and quantitative standards for evaluating motion quality.Source: §5 · p. 8

A Human Perception Flow

  • The framework decomposes motion perception into global motion-field parsing and fine-grained detail capture, addressing scene plausibility and physically implausible movement patterns.Examples include jitter artifacts, acceleration profiles that violate Newton’s laws, and trajectories with positional discontinuities.Source: A Human Perception Flow · p. 12

B.1 Commonsense Adherence Score (CAS)

CAS assesses whether generated video aligns with human commonsense using five quality levels. Its classifier is trained on preference-derived labels and maps predicted category probabilities to a final score.

B.1 Commonsense Adherence Score (CAS)

  • CAS evaluates whether generated videos align with human commonsense, assigning quality levels from Bad through Perfect.The five levels are Bad, Poor, Fair, Good, and Perfect.Source: §B.1 · p. 12
  • The CAS training labels come from pairwise VideoReward comparisons of 10k videos, ranked by win rate and divided into five equal groups.The videos combine existing web-dataset examples and outputs from legacy approaches and popular models.Source: §B.1 · p. 12
  • A VideoMAEv2 classifier predicts five quality categories, and CAS weights their probabilities from 0 for Bad to 1 for Perfect.The intermediate category weights are 0.25, 0.5, and 0.75 for Poor, Fair, and Good, respectively.Source: §B.1 · p. 12

B.2 Motion Smoothness Score (MSS)

MSS evaluates motion smoothness through frame-to-frame visual-quality changes, adapting its tolerance to motion intensity. OIS measures anatomical distortion in moving subjects using keypoint-based shape analysis.

B.2 Motion Smoothness Score (MSS)

  • Complex real-world movements can produce blur and artifacts, reflecting the challenge of preserving fine details while representing high-motion changes.Source: §B.2 · p. 12
  • MSS uses Q-Align aesthetic scores to measure frame-to-frame visual-quality drops, with higher motion allowing greater degradation under its adaptive threshold.Its 0-to-1 score is lower when more frames contain noticeable artifacts or blur.Source: §B.2 · pp. 12–13

B.3 Object Integrity Score (OIS)

  • OIS uses MMPose-detected subject keypoints to estimate shapes and identify perceptually noticeable distortions in moving objects.The analysis checks component lengths and joint angles against tolerances derived from natural-motion samples.Source: §B.3 · p. 13
  • OIS combines length and angle deviations to assess anatomical integrity across frames.It normalizes checks against component-specific tolerances established through statistical analysis of natural motion.Source: §B.3 · p. 13

B.4 Perceptible Amplitude Score (PAS)

PAS isolates subject movement from camera motion and scores its perceptual magnitude in context; TCS discounts legitimate object appearances and disappearances. The prompt analyses describe broad motion coverage and a human-LLM plausibility review.

B.4 Perceptible Amplitude Score (PAS)

  • PAS isolates primary-subject motion from camera movement by tracking subject keypoints and measuring their average displacement against scenario-specific perceptual thresholds.GroundingDINO detects the subject, GroundedSAM generates masks, and CoTracker tracks keypoints.Source: §B.4 · pp. 14–15

B.5 Temporal Coherence Score (TCS)

  • TCS uses segmentation and tracking to detect discontinuous object presence, then filters legitimate occlusions, frame exits or entries, and apparent size changes due to depth.The score penalizes discontinuities only when the filtering rules find no legitimate transition.Source: §B.5 · pp. 15–16

C.1 Prompts Statistic

  • VMBench prompts cover six motion patterns and varied subjects, places, actions, and lengths, with statistics indicating high dynamism and especially rich mechanical and biological motion.The authors report the most comprehensive coverage of action types and the most detailed prompt descriptions compared with previous works.Source: §C.1 · pp. 16–17

C.2 Human-LLM Reasoning Validation

  • DeepSeek-R1 plausibility scoring followed by human review retains 1,050 prompts describing reasonably realistic motion.Source: §C.2 · p. 17

D.1 Inference Details of Video Generation Models

The benchmark uses each model's available architecture, weights, and demo settings for video generation, while recording inference time. Human annotators separately score five motion dimensions using a focused, five-level rating process.

D.1 Inference Details of Video Generation Models

  • Model comparisons use available open-source architectures and weights with each model's demo hyperparameters, generate videos of about five seconds, and record inference time excluding loading.Configurations vary; for example, HunyuanVideo uses 129 frames at 624 × 832 with four-GPU inference, while OpenSora v1.2 uses 102 frames at 720 × 1280 and 30 steps.Source: §D.1 · pp. 17, 19

D.2 Human Annotation

  • Three annotators rate five motion dimensions on five-level Likert scales, with separate annotation packages focusing each annotator's assessment on one dimension at a time.The dimensions are Commonsense Adherence, Motion Smoothness, Object Integrity, Perceptible Amplitude, and Temporal Coherence.Source: §D.2 · p. 19, Figure 15 · p. 19

E Qualitative Analysis

Among evaluated models, Wan2.1 shows the most promising qualitative performance, while other models exhibit blurring or structural distortions. Across models, the analysis identifies failures involving persistence, structure, temporal continuity, and physical plausibility.

E Qualitative Analysis

  • The qualitative analysis identifies four failure modes: object-persistence errors, structural degeneration, temporal artifacts, and violations of Newtonian physics.Source: E Qualitative Analysis · p. 19
  • Mochi 1, OpenSora, and OpenSora-Plan show severe blurring, while CogVideoX and HunyuanVideo have smoother motion but often distort object shapes.Source: E Qualitative Analysis · p. 19
  • Wan2.1 is the most promising evaluated model, with smooth, physically plausible motion and comparatively natural shapes, limb movements, and movement amplitude.Source: E Qualitative Analysis · pp. 19–20, Figure 16 · p. 20
  • Consistency, physical plausibility, and natural motion remain challenges for text-to-video generation across diverse scenarios.Source: E Qualitative Analysis · p. 20

Related papers

Loading 2503.10076v2…