Source-linked AI summary
MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models
Wenyi Hong, Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, Jie Tang
TL;DR
Fine-grained motion comprehension is under-evaluated despite its foundational role in video understanding. The paper introduces MotionBench and the Through-Encoder Fusion method; current models remain below 60% accuracy on MotionBench, while higher frame rates and TE Fusion improve performance but leave substantial room for enhancement.
Problem
Most video benchmarks lack dedicated, diverse evaluation of fine-grained motion-level understanding, despite its foundational role in video analysis.
Method
MotionBench evaluates motion-level perception across six motion-oriented categories using diverse video sources, while TE Fusion deeply integrates temporal fusion within the visual encoder.
Results
Current state-of-the-art models achieve below 60% accuracy on MotionBench, while higher frame rates and TE Fusion improve motion understanding.
Takeaways & Limitations
MotionBench and its annotated motion descriptions provide resources for developing and training video models with stronger fine-grained motion comprehension.
Takeaways & Limitations
The dataset may contain geographical, cultural, and contextual biases that limit generalizability, and annotations may include occasional inaccuracies or inconsistencies.
Abstract
from arXiv · showhide
In recent years, vision language models (VLMs) have made significant advancements in video understanding. However, a crucial capability - fine-grained motion comprehension - remains under-explored in current benchmarks. To address this gap, we propose MotionBench, a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. MotionBench evaluates models' motion-level perception through six primary categories of motion-oriented question types and includes data collected from diverse sources, ensuring a broad representation of real-world video content. Experimental results reveal that existing VLMs perform poorly in understanding fine-grained motions. To enhance VLM's ability to perceive fine-grained motion within a limited sequence length of LLM, we conduct extensive experiments reviewing VLM architectures optimized for video feature compression and propose a novel and efficient Through-Encoder (TE) Fusion method. Experiments show that higher frame rate inputs and TE Fusion yield improvements in motion understanding, yet there is still substantial room for enhancement. Our benchmark aims to guide and motivate the development of more capable video understanding models, emphasizing the importance of fine-grained motion comprehension. Project page: https://motion-bench.github.io .
1. Introduction
Existing video benchmarks largely emphasize general, event-level, or story-level understanding, leaving fine-grained motion perception insufficiently evaluated. MotionBench addresses this gap with diverse motion-focused evaluation, while current models remain weak under fast and detailed motion demands.
- Motion-level understanding captures fine-grained motion and supports applications including anomaly detection, open-domain action analysis, and detailed video captioning.
- 8,052 questions comprise MotionBench, and most state-of-the-art models achieve below 60% accuracy, below the stated threshold for practical applications.
- Most existing benchmarks lack a dedicated, diverse evaluation set for motion-level understanding, often focusing instead on event- or story-level comprehension.
- MotionBench provides six motion-oriented task categories using data from seven distinct sources, including web, public-dataset, and synthetic videos.
- Video VLMs face computational and memory demands when processing high-frame-rate, long-duration videos, motivating compression methods in pixel and feature spaces.
- Existing compression approaches lack a comprehensive fair comparison as compression ratios increase and generally use shallow fusion confined to the compression operator.
3. MotionBench: Motion-Level Benchmarking
MotionBench is a manually curated benchmark for fine-grained motion perception, built from diverse video sources and six motion-oriented question categories. Its curation filters out questions answerable from common knowledge or a single frame, producing a dense dataset with 5,385 videos and 8,052 question-answer pairs.
- Benchmark Scope: MotionBench evaluates motion-level perception using six question categories across videos from intricate interactions, specific fields, and virtual scenes.The benchmark combines publicly available and self-collected videos, including synthetic Unity renderings.
- Curation Pipeline: The benchmark uses manually annotated questions for real-world and field-specific videos, while virtual-scene questions are automatically constructed from ground-truth annotations.For intricate-interaction videos, dynamic captions are first created and GPT-4o generates question-answer pairs before manual filtering.
- Curation Pipeline: Questions answerable from common knowledge or a single frame are removed, and ambiguous or incorrectly phrased items are manually filtered.Image VLMs are used on first-frame inputs to identify questions that do not require motion information.
- Dataset Statistics: Annotation Density reaches 68.4, twice that of existing benchmarks, supporting evaluation of fine-grained motions through denser annotations.MotionBench video lengths are primarily concentrated below 10 seconds.
- Video Compression Context: Figure 5 summarizes existing video-compression paradigms and TE Fusion before the VLM decoder, where temporal compression occurs.The benchmark section motivates efficient video representation under sequence-length constraints.
4. Model Design: Motion-Level Perception
Motion-level perception requires dense frame input, but sequence-length limits make efficient temporal compression necessary. The proposed Through-Encoder Fusion performs deep inter-frame fusion throughout visual encoding before spatial-temporal compression.
- Motivation: High-frame-rate input is important for motion perception but constrained by VLM sequence-length and computational limits.These constraints motivate dense video representations and efficient compression architectures.
- Limitations of Existing Compression: Existing compression methods often fuse neighboring frames only within a small compression operator, treating frames independently beforehand.This shallow fusion makes higher-level temporal redundancies difficult to capture.
- Through-Encoder Fusion: Deep fusion is intended to unify changing positions, postures, and camera angles across frames while retaining critical motion features.The paper motivates this design through the example of a running person whose visual attributes vary continuously.
- Through-Encoder Fusion: TE Fusion groups adjacent frames and applies group-level self-attention during visual encoding to compute temporal dependencies across the encoder.Spatial-temporal compression is then applied to each frame group.
- Scope: This work evaluates one straightforward TE Fusion implementation, while leaving other variations of the broader paradigm for future exploration.The limitation concerns the range of architectures tested, not the definition of the paradigm.
5. Experiments
Experiments show that MotionBench remains difficult for current video VLMs, while TE Fusion improves performance across compression settings and is especially effective at higher compression ratios. The study compares compression paradigms under controlled backbones, frame counts, and decoder sequence lengths.
- Evaluation on MotionBench: Current video VLMs improve over text-only performance by only 0.05 to 0.2 accuracy, indicating persistent difficulty recognizing short, simple motions.GPT-4 with text input reaches 0.3 to 0.4 accuracy, above the 0.25 random baseline.
- Evaluation on MotionBench: 11% to 14% of short 0-4-second videos contain questions that all models fail, rising to 18% for videos longer than 18 seconds.The largest share of universally failed questions involves fine-grained motion.
- Experimental Setup: The comparison re-implements pre-encoder, post-encoder, through-encoder, and no-temporal-fusion architectures on the GLM-4V-9B backbone.Models are trained with the same input resolution, iteration count, batch size, and open-source dataset collection.
- Compression Experiments: TE Fusion achieves the highest scores in 9 out of 10 metrics, improving over baseline by 11.8% on MVBench and 18.7% on VideoMME-short with k = 4.These results are reported under a fixed VLM decoder sequence length.
- Compression Experiments: QFormer occasionally underperforms the baseline, potentially because its additional module is difficult to optimize during video-compression training.This contrasts with the stronger results of PLLaVA, Kangaroo, and TE Fusion in the controlled comparison.
- Compression Experiments: With Ninput = 16, TE Fusion shows almost no performance drop for k ≤4 and less than 4% average reduction at k = 16 versus the uncompressed baseline.Compression hurts shorter-duration videos less than longer-duration videos, and TE Fusion peaks at k = 4 on 3 of 4 datasets.
6. Conclusion
MotionBench evaluates fine-grained motion understanding, and the experiments show that current models struggle with motion-level comprehension. TE Fusion improves video feature representation and achieves state-of-the-art results, particularly under high compression.
- Conclusion: MotionBench is a benchmark for assessing fine-grained motion understanding in video models.It targets motion-level perception as a dedicated evaluation capability.
- Conclusion: Current state-of-the-art models struggle with motion-level comprehension, motivating specialized evaluation.The conclusion frames this difficulty as a central experimental finding.
- Conclusion: TE Fusion deeply integrates fusion within the visual encoder to improve video feature representation.The method is presented as the paper’s response to the motion-perception challenge.
- Conclusion: TE Fusion achieves state-of-the-art results, especially under high compression.The conclusion identifies high-compression performance as a particular strength.
Supplementary Material
The supplementary material details TE Fusion’s architecture, training settings, and controlled ablations. Its implementation groups adjacent frames for deep attention, then reshapes and downsamples the encoded features before sending them to the LLM.
- Training Settings: TE Fusion is trained alongside ablated architectures using datasets including VideoChat, NExT-QA, Kinetics-710, Sth-SthV2, Ego4D, and TGIF-QA.The training collection also includes VideoChatGPT, CLEVRER, WebVidQA, and an in-house video QA dataset.
- TE Fusion Architecture: The input video is patchified, grouped every kt frames, and processed by a ViT encoder whose self-attention operates within each group.This group-wise attention is the concrete implementation of deep fusion across adjacent frames.
- TE Fusion Architecture: The encoder output is reshaped and passed through a 3D convolution with kernel and stride (kt, ks, ks) to produce downsampled features for the LLM.The resulting features have the LLM-compatible output dimension described in the supplementary formulation.
- Architecture Ablations: TE Fusion concatenates neighboring frames before visual encoding, applies cross-frame self-attention, and projects temporally downsampled tokens after encoding.The comparison also distinguishes Qwen2-VL, Kangaroo, and QFormer by where and how they perform temporal fusion.
- Architecture Ablations: All ablated architectures use the same GLM-4V backbone to maintain a fair comparison.The supplementary material provides the corresponding model configurations and training hyperparameters.
9. QA Construction Process for Videos with Intricate Interactions
For intricate-interaction videos, MotionBench constructs motion-focused questions by combining manual dynamic captions, GPT-4o generation, model-based filtering, and human quality checks.
- Annotation and generation: Annotators first describe dynamic changes in intricate-interaction videos rather than directly annotating the full clips.The process used 15 adult annotators with at least bachelor’s degrees over 20 working days.
- Annotation and generation: GPT-4o generates six question-answer pairs per video description, emphasizing diversity and accuracy.
- Question design: Questions combine a content dimension with a question-logic dimension while focusing on prominent actions and clear, unique options.The content system covers human, object, animal, camera, and appearance dynamics; logic includes movement occurrence, count, sequence, and appearance judgment.
- Question design: Each generated question uses four answer options and excludes logical relationships when the dynamic description does not encode them.
- Difficulty filtering: Image VLMs filter out questions that all tested models answer correctly from the first frame, retaining questions requiring motion comprehension.The filtering models are GPT-4o, Qwen2-VL, and GLM-4V-plus.
- Quality control: Ten annotators manually check the generated questions for reasonableness, category correctness, video relevance, option accuracy, and answer uniqueness.The checks lasted five days, with each annotator paid approximately 250 RMB per day.
10. Copyrights
MotionBench is distributed as a non-commercial research preview, with source-specific restrictions governing video access and redistribution.
- Usage restrictions: MotionBench is intended only for non-commercial use.Open-source videos follow their licenses; Pexels videos require user agreements, while self-collected movie clips are provided through links and processing scripts rather than raw files.
11. The originality of MotionBench.
MotionBench distinguishes itself by targeting general-domain motion-level understanding with diverse sources and question designs, while evaluating compression under controlled settings.
- Benchmark originality: MotionBench targets general-domain motion understanding rather than fixed action labels, limited gestures, or predefined video and action fields.
- Compression evaluation: Compression methods are compared at rates k = 2, 4, 6, and 8 with the same decoder sequence length.The experiments include k = 4 and 8 settings, and Table 7 reports the complete results.
13. Case Study on Model Performance
The case studies show that current models struggle with motion-level questions, especially repetitive-action counting, and can misinterpret temporal motion from a single frame.
- Overall performance: Current video understanding models achieve less than 60% accuracy on MotionBench, and some questions are answered incorrectly by every model.
- Task difficulty: The highest proportion of universally incorrect questions occurs in the Fast action count task type.The paper attributes this difficulty to counting repetitive actions at the motion level.
- Temporal case study: A hand moving from the top to the lower left of a car is often misclassified as tapping because the single frame supports a misleading static interpretation.The example motivates evaluating temporal sequences rather than relying only on single-frame perception.
14. Limitations and Broader impact
MotionBench’s limitations concern dataset representativeness and annotation reliability, while its broader impact is to support evaluation and improvement of motion-level video understanding.
- Limitations: The dataset may retain geographical, cultural, and contextual biases that limit generalizability to different settings.
- Limitations: Human and automatic tool errors may introduce occasional annotation inaccuracies or inconsistencies.
- Broader impact: Motion-level perception is presented as pivotal across video understanding applications, and MotionBench provides comprehensive benchmarking for this capability.
- Broader impact: Public release of the dataset is intended to support further enhancement of video understanding models and their applicability in real-world scenarios.
15. More Dataset Samples
The samples illustrate MotionBench’s fine-grained questions across object actions, camera and location changes, trajectories, repetition counts, and ordered interactions.
- Motion Recognition: Motion-recognition samples evaluate ordered actions between people and subtle facial movements.
- Motion-related Objects: Motion-related object questions ask which object is taken, how a hand acts, or how balls and people move in sequence.
- Repetition Count: Trajectory and repetition questions ask for an object’s movement path or the number of repeated actions.
- Location-related Motion: Camera and location questions test whether the camera moves, how its movements are sequenced, and how objects traverse the frame.