PaperlayerView exact revision on arXiv

Source-linked AI summary

FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos

Yulu Gan, Ligeng Zhu, Dandan Shan, Baifeng Shi, Hongxu Yin, Boris Ivanovic, Song Han, Trevor Darrell, Jitendra Malik, Marco Pavone, Boyi Li

arXiv:2512.10927v1cs.CV

TL;DR

VLMs still struggle with fine-grained motion understanding, while creating motion annotations manually is costly. FoundationMotion automates object tracking and motion-data generation; fine-tuned models outperform larger open-source models and some closed-source baselines on motion tasks.

  • Problem

    VLMs can interpret objects and events but struggle with nuanced motion, while fine-grained motion annotation is costly to scale.

    Source: §1 · p. 2

  • Method

    FoundationMotion automatically detects and tracks objects, then uses motion annotations and video frames with LLMs to generate captions and question-answer pairs.

    Source: §1 · p. 3

  • Results

    Fine-tuned models outperform larger open-source and some closed-source models; NVILA-Video-15B scores 91.5% on AV-Car versus Gemini-2.5-Flash (84.1%) and Qwen-2.5-VL-72B (83.3%).

    Source: §1 · p. 3, §4.2 · p. 9, §6 · p. 11

  • Takeaways & Limitations

    The automated pipeline provides a scalable way to curate motion data for fine-tuning open-source models on motion understanding.

    Source: §1 · p. 3, §6 · p. 11

    Current spatial understanding is primarily limited to 2D; understanding object motion in 3D remains a challenge.

    Source: §6 · p. 11

Abstract

from arXiv · show

Motion understanding is fundamental to physical reasoning, enabling models to infer dynamics and predict future states. However, state-of-the-art models still struggle on recent motion benchmarks, primarily due to the scarcity of large-scale, fine-grained motion datasets. Existing motion datasets are often constructed from costly manual annotation, severely limiting scalability. To address this challenge, we introduce FoundationMotion, a fully automated data curation pipeline that constructs large-scale motion datasets. Our approach first detects and tracks objects in videos to extract their trajectories, then leverages these trajectories and video frames with Large Language Models (LLMs) to generate fine-grained captions and diverse question-answer pairs about motion and spatial reasoning. Using datasets produced by this pipeline, we fine-tune open-source models including NVILA-Video-15B and Qwen2.5-7B, achieving substantial improvements in motion understanding without compromising performance on other tasks. Notably, our models outperform strong closed-source baselines like Gemini-2.5 Flash and large open-source models such as Qwen2.5-VL-72B across diverse motion understanding datasets and benchmarks. FoundationMotion thus provides a scalable solution for curating fine-grained motion datasets that enable effective fine-tuning of diverse models to enhance motion understanding and spatial reasoning capabilities.

1 Introduction

FoundationMotion targets VLMs’ difficulty reasoning about how motion unfolds, a gap linked to scarce, costly fine-grained motion data. It automates motion-data curation and reports that fine-tuned models outperform larger open-source and some closed-source models.

1 Introduction

  • Current VLMs can interpret objects, scenes, and events but still struggle with nuanced spatial dynamics and “how” an action unfolds.The introduction contrasts recognizing an action such as pouring water with understanding how it happens, such as pouring from a bottle into a glass.Source: §1 · p. 2
  • Fine-grained motion annotation requires accurate detection, tracking, and linking behaviors to motions, making manual dataset construction costly and time-consuming.The paper estimates that annotating 100,000 videos would take a team of 10 about 100 days.Source: §1 · p. 2
  • The paper introduces a fully automated pipeline that detects and tracks objects, then turns their motion into structured annotations, captions, and question-answer pairs.The pipeline is used to produce approximately 500K question-answer pairs and captions, collectively called the FoundationMotion Dataset.Source: §1 · p. 3
  • The authors manually collect videos and annotate “how” motion questions across human daily activities, driving, and robot manipulation.They plan to release the code, data, and benchmarks.Source: §1 · p. 3
  • Fine-tuned models outperform larger open-source models and closed-source baselines such as Gemini-2.5-Flash on motion benchmarks.The paper evaluates public benchmarks and a manually annotated “how” motion benchmark.Source: §1 · p. 3

2 Related Work

The related work identifies gaps in motion benchmarks, automated video annotation, and video-language models. FoundationMotion addresses them with tracking-based motion captions and QA data, and motion-focused model training.

2.1 Motion-Focused Video Understanding Benchmarks

  • Existing motion benchmarks emphasize fine-grained recognition but overlook spatial reasoning, including motion interactions, relative trajectories, and geometric constraints.The authors present automated large-scale data construction as a response to data scarcity and these benchmark gaps.Source: §2.1 · p. 3

2.2 Automated Video Dataset Construction and Annotation

  • FoundationMotion extends automated video annotation with multi-object tracking and detailed captions and QA pairs about object trajectories.The resulting data is tailored to spatial object behavior at a scale and granularity described as infeasible with manual labeling.Source: §2.2 · p. 4

2.3 Vision-Language Video Foundation Models

  • The paper introduces motion-aware training to address video models’ difficulty with fine-grained motion and spatiotemporal reasoning.The authors report gains in motion recognition, localization, and reasoning while preserving broad video-language capabilities.Source: §2.3 · p. 4

3 FoundationMotion Data Curation Pipeline

FoundationMotion combines video preprocessing, object detection and tracking, captioning, and question generation to curate motion-focused training data. Its pipeline uses vision models for spatial annotations and language models to summarize motion and generate QA pairs.

3 FoundationMotion Data Curation Pipeline

  • The pipeline combines video preprocessing, object detection and tracking, caption generation, and QA generation to produce motion annotations and question-answer pairs.It builds on complementary strengths of vision models in detection, tracking, and summarization, and LLMs in automated pipelines.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Video Preprocessing

  • Videos are cropped to 5–10-second segments, while videos of five seconds or less are retained in full.The sampled segment is centered approximately at the video midpoint with temporal variation; videos exceeding a camera-motion threshold of 0.3 are excluded.Source: §3.1 · p. 5

3.2 Object Detection and Tracking

  • Object detection combines open-vocabulary detection with specialized human-centric detection for people, left and right hands, and held objects.The human-centric detector addresses the difficulty standard detectors have distinguishing left from right hands.Source: §3.2 · p. 5

3.2.1 Open-Vocabulary Object Detection

  • The method generates salient object categories from the video, then queries Grounded-DINO on the first frame with each class separately rather than concatenating classes, aligning boxes and labels and improving detection quality.Source: §3.2.1 · p. 6

3.2.2 Human-Centric Detection

  • Human-centric detection refines person detections to hands, using pose keypoints to initialize hand regions and Hands23 to detect hands and assess contact.Hand-person associations require IoU > 0.3; the pipeline also detects hand-object interactions.Source: §3.2.2 · p. 6

3.2.3 Temporal Tracking

  • SAM2 propagates initialized detections through video frames, with hand and hand-object detections added at every fifth frame to refine tracking.A hierarchical ID scheme links people with their body parts, while separate IDs are assigned to objects.Source: §3.2.3 · p. 6

3.3 Caption Generation

  • GPT-4o-mini uses sampled frames, JSON trajectories, and color-coded overlays to generate captions spanning seven dimensions of motion and spatial relations.The dimensions include action recognition, temporal sequencing, repetition, motion dynamics, and evolving spatial relationships.Source: §3.3 · pp. 6–7

3.4 Question-Answer Generation

  • GPT-4o-mini generates four-option multiple-choice questions from captions and video frames across five motion and spatial-understanding categories.Distractors come from video content, and correct-answer positions are randomized to avoid position bias.Source: §3.4 · p. 7

4 Fine-tuning with FoundationMotion for State-of-the-Art Motion Understanding

Fine-tuning with FoundationMotion improves performance across motion benchmarks, with mid-sized models surpassing larger baselines on several tasks. Evaluation combines public benchmarks with four manually labeled, zero-shot motion benchmarks.

4.1 Experimental Setup

  • Training uses randomly extracted 5-second InternVid clips, producing 467K caption/QA-video pairs.Source: §4.1 · p. 7
  • Evaluation includes MotionBench and VLM4D, plus four zero-shot benchmarks covering daily scenes, autonomous vehicles, and robotics.The four are AV-Car, AV-Hand, Daily, and Robotics; the authors report no overlap with FoundationMotion training data.Source: §4.1 · pp. 7–8

4.2 Main Results

  • Qwen-2.5-VL-7B gains +2.1% on MotionBench, +3.2% on VLM4D, and +11.7% on Daily after fine-tuning.It also gains +1.8% on AV-Car, +5.6% on AV-Hand, and +4.2% on Robotics.Source: §4.2 · pp. 8–9
  • FoundationMotion outperforms equal-example PLM fine-tuning on most reported NVILA-Video-15B and NVILA-Video-8B benchmarks, with exceptions on MotionBench and VLM4D for 15B and MotionBench for 8B.For NVILA-Video-15B, AV-Car gains are +7.1% vs. -5.0%; for NVILA-Video-8B, Robotics gains are +17.8% vs. +6.1%.Source: §4.2 · p. 9
  • NVILA-Video-15B + FoundationMotion reaches 91.5% on AV-Car, above Gemini-2.5-Flash (84.1%) and Qwen-2.5-VL-72B (83.3%).Source: §4.2 · p. 9

5 Analysis

The analysis tests how structured object annotations and motion-focused question types affect QA generation and model performance, then describes the FoundationMotion dataset’s composition.

5.1 Data Curation Related Analysis

  • Adding bounding-box JSONs improves QA scores across all dimensions, with gains of +2.6 in fine-grained action accuracy and motion detail, and +2.4 in temporal coherence.The comparison uses GPT-4 evaluation of video-only versus video-plus-JSON inputs.Source: Table 2 · p. 9, §5.1 · p. 10
  • Motion Recognition and Action Order each score 52% (+8.3% over Base); Motion-related Objects and Location-related Motion each score 53% (+10.4%), while Repetition Count reaches approximately 55% (+14.6%).The combined ALL setting also reaches 55%, matching the best single type; the passage attributes complementary coverage to the different QA types.Source: Figure 4 · p. 10, §5.1 · p. 10

5.2 Data Distribution of the FoundationMotion Dataset

  • The dataset contains 46.7k videos and 467k QAs; correct answers are evenly distributed, most questions are 30–80 characters, and most videos last 3–7 seconds.Each QA has a question, four options, an answer, and a category.Source: Figure 5 · p. 10, §5.2 · p. 11

6 Conclusion

FoundationMotion automates motion labeling for object detection, tracking, and behavior understanding. Fine-tuning with its dataset improves open-source VLM performance on “how” motion benchmarks, while current spatial understanding remains primarily 2D.

6 Conclusion

  • FoundationMotion is an automated motion-labeling pipeline for generalized spatial detection, tracking, and understanding of object behaviors.Fine-tuning with its dataset on various “how” motion benchmarks lets existing open-source VLMs outperform larger models and compete with or surpass some closed-source models, including Gemini-2.5-Flash.Source: §6 · p. 11
  • Current spatial understanding is primarily limited to 2D; understanding 3D object motion remains a challenging next step.The paper identifies joint-level 3D hand motion as an example relevant to robotics and related applications.Source: §6 · p. 11

A.1 Basic Statistics of FoundationMotion Dataset

The appendix reports dataset-level averages for video duration, question count, annotation density, and question length.

A.1 Basic Statistics of FoundationMotion Dataset

  • Videos average 17.51 seconds and about 10 questions, yielding 1.671 questions per second; questions average 55.9 characters.Table 3 summarizes the dataset’s overall statistics.Source: §A.1 · p. 15, Table 3 · p. 15

A.2 Prompts used for Caption Generation

The caption-generation prompt combines video frames with JSON motion information and visualized object boxes to guide motion description.

A.2 Prompts used for Caption Generation

  • The captioning tool uses the video and supplied object-motion information, both drawn on the video and represented in JSON.The JSON records object IDs and, for each object, bounding boxes, object type, and interactions.Source: §A.2 · p. 15
  • Bounding boxes use normalized [left, top, right, bottom] coordinates; missing detections and absent interactions are represented as None for the corresponding frame.The coordinates are normalized by video width and height.Source: §A.2 · pp. 15–16
  • Frames display labeled, colored bounding boxes to make object identities easy to distinguish.Source: §A.2 · p. 16

A.3 Prompts used for QA Generation

The prompt asks for concise video-and-caption-based questions and answers that test motion analysis and spatial relationships. It specifies question coverage, answer style, and multiple-choice output requirements.

Question and answer setup

  • Each video is paired with a caption describing object motion and spatial relations, and questions should assess a viewer’s understanding of them.Answers should be concise and directly address each question, using directional terms, distance measurements, and spatial descriptors when available.Source: §A.3 · p. 16

Question coverage

  • Questions cover action recognition, temporal order, and which actors or objects perform particular motions.The prompt also asks about action timing and object-action associations.Source: §A.3 · p. 16
  • Questions also address motion location, repetition counts, direction, distance, velocity, trajectories, and changing spatial relationships.Examples include left/right positions and how the relative positions of two objects change.Source: §A.3 · p. 16

Multiple-choice generation

  • The prompt first requests question-answer pairs, then four distinctive, unambiguous choices per question, with the correct answer first and video content checked.The choices are generated from the caption.Source: §A.3 · p. 17

Output format

  • The final output is a list of strings, each containing one question and its four A–D choices; the number of questions matches the list length.The prompt says not to include other text.Source: §A.3 · p. 17

A.4 Prompts used for evaluate QA quality

The appendix gives QA-generation evaluation criteria and example multiple-choice items for motion recognition, object-action links, action order, repetition, and location.

A.4 Prompts used for evaluate QA quality

  • Generated QAs made from video alone and video plus bounding-box JSONs are rated separately on accuracy, motion detail, temporal coherence, and relevance.Each dimension is scored from 0 to 10.Source: §A.4 · p. 17

A.5 Question-Answer Examples

Loading 2512.10927v1…