Source-linked AI summary
ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos
Peiran Wu, Yunze Liu, Miao Liu, Junxiao Shen
TL;DR
The paper asks whether MLLMs can reason about complex spatial-temporal events from egocentric videos. It introduces Ego-ST Bench and trains ST-R1 with reverse thinking, long-CoT supervision, and GRPO. Existing models still have substantial room for improvement, while ST-R1 outperforms traditional SFT and generalizes strongly from limited training data.
Problem
Whether MLLMs possess sophisticated spatial-temporal reasoning abilities for understanding dynamic 4D environments from egocentric videos remains unresolved.
Method
The paper introduces Ego-ST Bench for bidirectional egocentric-video evaluation and ST-R1, which uses reverse thinking, long-CoT SFT, and GRPO.
Results
Existing models still have substantial room for improvement, while ST-R1 significantly outperforms traditional SFT and improves OOD performance by over 32% in two task types.
Takeaways & Limitations
Ego-ST Bench and ST-R1 provide evaluation and training resources for advancing video-based spatial-temporal reasoning research.
Takeaways & Limitations
Future work includes adding modalities and diverse data, encouraging implicit reconstruction, improving generalized video reasoning, and reducing inference cost.
Abstract
from arXiv · showhide
Humans excel at spatial-temporal reasoning, effortlessly interpreting dynamic visual events from an egocentric viewpoint. However, whether multimodal large language models (MLLMs) can similarly understand the 4D world remains uncertain. This paper explores multimodal spatial-temporal reasoning from an egocentric perspective, aiming to equip MLLMs with human-like reasoning capabilities. To support this objective, we introduce \textbf{Ego-ST Bench}, a novel benchmark containing over 5,000 question-answer pairs across four categories, systematically evaluating spatial, temporal, and integrated spatial-temporal reasoning. Additionally, we propose \textbf{ST-R1} training paradigm, a video-based reasoning model that incorporates reverse thinking into its reinforcement learning process, significantly enhancing performance. We combine long-chain-of-thought (long-CoT) supervised fine-tuning with Group Relative Policy Optimization (GRPO) reinforcement learning, achieving notable improvements with limited high-quality data. Ego-ST Bench and ST-R1 provide valuable insights and resources for advancing video-based spatial-temporal reasoning research.
1 Introduction
The paper addresses whether MLLMs can perform human-like spatial-temporal reasoning from egocentric videos. It introduces Ego-ST Bench and ST-R1, combining reverse thinking, long-CoT supervision, and GRPO to improve this capability.
- Motivation: Current benchmarks mainly assess spatial reasoning, leaving trajectory, path description, action prediction, and reverse reasoning insufficiently evaluated.The paper positions Ego-ST Bench as addressing these missing spatial-temporal and bidirectional capabilities.
- Contributions: ST-R1 uses reverse thinking as a reasoning process, with spatial-temporal CoT data for SFT followed by GRPO to enhance reasoning performance.The model restructures forward and reverse route descriptions so reverse descriptions serve as an intermediate reasoning process.
- Findings: Existing models still have substantial room for improvement, while open-source and closed-source models show no significant performance gap on Ego-ST Bench.Some open-source models with longer context windows even achieve superior performance.
- Findings: ST-R1 significantly outperforms traditional SFT after post-training with a small amount of high-quality long-CoT data.The result supports the effectiveness of the proposed multi-stage post-training strategy.
- Contributions: Ego-ST Bench contains over 5,000 annotated instances and evaluates forward and reverse spatial-temporal reasoning from an egocentric perspective.The benchmark is intended to support comprehensive bidirectional evaluation of MLLMs.
2 Related Work
Prior work studies egocentric video understanding, spatial-temporal benchmarks, and multimodal reasoning, but does not specifically evaluate higher-level spatial-temporal reasoning in egocentric MLLM videos. The paper addresses this gap with Ego-ST Bench and ST-R1.
- Egocentric Video Understanding: Existing egocentric-video benchmarks do not specifically target higher-level spatial-temporal reasoning capabilities in MLLMs.Earlier resources include Ego4D, EPIC-KITCHEN, HD-EPIC, HOI4D, HourVideo, EgoPlan-Bench2, and X-LeBench.
- Multimodal Spatial Benchmarks: Prior multimodal spatial benchmarks cover image-level spatial understanding or broad spatial-temporal understanding, but in-depth reasoning evaluation remains lacking.The paper contrasts its focus with SpatialVLM, SpatialRGPT, HourVideo, OVO-Bench, and X-LeBench.
- Multimodal Reasoning: Multimodal reasoning frameworks seek to bridge textual and visual modalities, while this work focuses on the distinct challenges of spatial-temporal reasoning in videos.ST-R1 is presented as a model designed for complex spatial-temporal reasoning tasks.
- Benchmark Positioning: The benchmark pipeline standardizes filtered video segments, manually creates forward and reverse route-description QA pairs, and uses GPT-4o to generate three multiple-choice QA types.The generated questions concern direction, landmark, and action descriptions.
3 Ego-ST Bench
Ego-ST Bench evaluates higher-level spatial-temporal reasoning in egocentric videos using 789 clips and over 5,000 QA pairs across forward and reverse tasks. Its construction combines filtered video sources, manual route annotations, and automated multiple-choice generation.
- 3.1 Overview: Ego-ST Bench contains over 5,000 QA pairs from 789 videos spanning indoor and complex outdoor scenarios.The videos come from five listed source groups, including SUN3D, HUJI, DoMSEV, Aria Project, and self-collected data.
- 3.1 Overview: The benchmark defines eight tasks across route, direction-change, landmark, and action-change descriptions, each supporting forward and reverse reasoning.This design is intended to assess whether models can reason inversely about forward video information.
- 3.2 Benchmark Construction: Benchmark construction uses a pipeline that filters and standardizes video segments before manually annotating forward and reverse route-description QA pairs.The pipeline is designed to produce high-quality QA pairs at scale.
- 3.2 Benchmark Construction: GPT-4o generates direction, landmark, and action multiple-choice questions from the annotated route information.These questions evaluate chronological order and changes in direction, landmarks, or actions.
- 3.2 Benchmark Construction: The benchmark statistics report task distributions across four categories and video-length statistics.Figure 4 summarizes these two aspects of the dataset.
4 Spatial-Temporal R1
ST-R1 reasons about 4D spatial-temporal video information through two stages: supervised chain-of-thought training followed by GRPO reinforcement learning. The pipeline combines forward and reverse reasoning with reward-guided policy refinement for video question answering.
- 4 Spatial-Temporal R1: ST-R1 analyzes objects, actions, and interactions across space and time to answer video questions about complex events and scene changes.Its input is a video sequence and a query, and its output is a reasoned answer based on visual cues over time.
- 4.1 Chain-of-Thought Supervised Fine-Tuning: The first stage fine-tunes the model on video-question inputs paired with human-crafted chain-of-thought traces and correct answers.The model learns to generate the reasoning trace followed by the answer, rather than only producing a final response.
- 4.1 Chain-of-Thought Supervised Fine-Tuning: Forward reasoning serves as the thinking process for reverse problems, while reverse reasoning serves as the thinking process for forward problems.The paper states that integrating both directions improves reasoning accuracy by helping the model derive and validate answers [10].
- 4.2 Post-training using GRPO: The second stage applies GRPO to the CoT-trained model, sampling multiple answers and updating the policy toward higher-reward outputs while maintaining closeness to its initial policy.GRPO uses group-based reward statistics instead of a learned critic, and the KL term penalizes divergence from a reference policy.
- 4.2 Post-training using GRPO: Multiple-choice reward design supplies clear correctness signals, allowing iterative GRPO updates to favor answers that are correct and well-reasoned.The evaluation covers four question categories, each split into forward and reverse reasoning tasks, with strong-temporal and overall average scores reported.
5 Evaluation on Ego-ST Bench
Evaluation on Ego-ST Bench shows that current MLLMs handle isolated spatial and temporal cues better than integrated spatial-temporal reasoning, while ST-R1 improves generalization through two-stage training.
- Overall model performance: Open-source models perform poorly on direction-change tasks but relatively well on landmark and action tasks, revealing a gap in integrated spatial-temporal reasoning.The benchmark evaluates 10 MLLMs in zero-shot settings, including proprietary and open-source models.
- Evaluation Setup: The evaluation separates direction, landmark, and action changes to measure integrated spatial-temporal, spatial-semantic, and temporal understanding.Multiple-choice questions require selecting the correct chronological order and change among options.
- Open-ended Route Description: Complete route descriptions are substantially harder than single-dimensional multiple-choice tasks because they require integrated spatial-temporal information and temporal-causal reasoning.Route descriptions are scored across direction, landmark changes, and logical semantic consistency, with forward and reverse reasoning scores.
- Open-ended Route Description: Open-ended results align with multiple-choice results: landmark understanding is strongest, whereas direction-changing environments remain difficult for both model types.The evaluation uses LLM judging for open-ended answers and exact-match accuracy for multiple-choice questions.
- Comparison of different training paradigms: ST-R1 achieves superior generalization with limited training data by combining spatial-temporal CoT SFT with subsequent GRPO training.CoT SFT provides a cold start, while GRPO further enhances reasoning; regular SFT+GRPO is unstable and markedly inferior.
- Out of Distribution (OOD) Generalization: ST-R1 improves performance by over 32% on each of two OOD task types using only 630 training samples.The experiments include indoor OOD data and indoor-to-outdoor transfer.
6 Conclusion And Future Work
The paper concludes that current MLLMs struggle to integrate complex spatial-temporal information despite stronger performance on isolated tasks, while two-stage post-training substantially improves video reasoning. Future work targets broader modalities, implicit reconstruction, generalized models, and efficient inference.
- Conclusion: Current MLLMs show a minimal open-source versus closed-source gap but struggle with integrated open-ended route descriptions despite stronger isolated-task performance.The proposed two-stage post-training approach substantially improves video-based spatial-temporal reasoning.
- Future Work: Future work should explore additional modalities, implicit spatial-temporal reconstruction, generalized video reasoning, and computationally efficient inference.These directions are presented as priorities for advancing multimodal spatial-temporal reasoning.