Source-linked AI summary
STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?
Yun Li, Yiming Zhang, Tao Lin, Xiangrui Liu, Wenxiao Cai, Zheng Liu, Bo Zhao
TL;DR
MLLMs are increasingly used for embodied AI and autonomous driving, but precise quantitative spatial-temporal understanding in real-world settings remains insufficiently examined. STI-Bench evaluates this capability with real-world videos and QA pairs spanning multiple scenarios and task types, finding substantial limitations in current models, especially for quantitative measurement and motion reasoning.
Problem
Precise and quantitative spatial-temporal understanding by MLLMs in real-world applications remains largely unexamined despite their use in embodied AI and autonomous driving.
Method
STI-Bench evaluates MLLMs with over 300 real-world videos and 2,000 QA pairs covering static spatial measurement, physical motion, and desktop, indoor, and outdoor scenarios.
Results
Current MLLMs show significant spatial-temporal limitations, with top-performing models such as Gemini-2.5-Pro achieving around 41.4% average accuracy and particular difficulty in dimensional measurement.
Takeaways & Limitations
The findings identify inaccurate spatial quantification, flawed temporal dynamics understanding, and weak cross-modal integration as central challenges for reliable embodied applications.
Takeaways & Limitations
MLLMs remain far from the reliability required for embodied AI or autonomous driving, with difficulties including sparse-temporal reasoning and separating ego-motion from object motion.
Abstract
from arXiv · showhide
The use of Multimodal Large Language Models (MLLMs) as an end-to-end solution for Embodied AI and Autonomous Driving has become a prevailing trend. While MLLMs have been extensively studied for visual semantic understanding tasks, their ability to perform precise and quantitative spatial-temporal understanding in real-world applications remains largely unexamined, leading to uncertain prospects. To evaluate models' Spatial-Temporal Intelligence, we introduce STI-Bench, a benchmark designed to evaluate MLLMs' spatial-temporal understanding through challenging tasks such as estimating and predicting the appearance, pose, displacement, and motion of objects. Our benchmark encompasses a wide range of robot and vehicle operations across desktop, indoor, and outdoor scenarios. The extensive experiments reveals that the state-of-the-art MLLMs still struggle in real-world spatial-temporal understanding, especially in tasks requiring precise distance estimation and motion analysis.
1. Introduction
MLLMs have advanced mainly on semantic visual tasks, but embodied and autonomous systems also require precise spatial-temporal understanding. STI-Bench evaluates this capability with video-based tasks across desktop, indoor, and outdoor environments, revealing substantial difficulty for current models.
- MLLMs excel at general visual question answering but must additionally understand 3D space and time for embodied AI and autonomous driving.
- STI-Bench evaluates spatial-temporal understanding from single videos or multiple images rather than 3D point clouds.The benchmark uses images or videos because leading models commonly accept them and videos contain information for inferring spatial-temporal environments.
- STI-Bench comprises 300 videos and over 2,000 tailored questions for systematic quantitative assessment of MLLMs’ spatial-temporal understanding.
- The benchmark covers desktop, indoor, and outdoor scenarios sourced from Omni6DPose, ScanNet, and Waymo, respectively.
- STI-Bench includes eight tasks spanning static spatial measurement and grounding alongside dynamic speed, acceleration, and trajectory estimation.
- Experiments show that even advanced MLLMs struggle across tasks, particularly with precise distance estimation and motion analysis.The study identifies inaccurate spatial quantification, flawed temporal dynamics understanding, and weak cross-modal grounding and integration as fundamental limitations.
2. Related Work
Prior MLLM benchmarks emphasize semantic video understanding and high-level visual reasoning, while spatial-temporal evaluation remains limited. STI-Bench addresses this gap by testing precise spatial measurements and physical motion across real-world scenarios and viewpoints.
- Spatial Understanding with MLLMs: Video MLLM research has concentrated on semantic understanding, while spatial understanding remains a significant challenge.
- Benchmark Comparison: The benchmark comparison distinguishes video or image sources, simulated or real environments, ego-centric or allocentric views, numerical or textual ground truth, and spatial-temporal capabilities.
- Embodied Intelligence: STI-Bench connects spatial-temporal evaluation with embodied-agent progress, where MLLMs are integrated into robotic control for generalization and semantic reasoning.
- Video Benchmarks for MLLMs: Existing video benchmarks primarily evaluate high-level semantic abilities such as entity recognition and event understanding as temporal extensions of 2D image understanding.
- Spatial Understanding with MLLMs: STI-Bench evaluates precise spatial-temporal understanding through static spatial measurement and physical motion tasks in desktop, indoor, and outdoor scenarios.
3. STI-Bench
STI-Bench is constructed from multi-scene datasets and human-verified QA generation to evaluate eight static and dynamic spatial-temporal understanding tasks across desktop, indoor, and outdoor scenarios. Its tasks require models to combine 3D spatial reasoning, physical motion understanding, and cross-modal integration over time.
- Task design: The benchmark proposes eight tasks divided into Static Understanding and Dynamic Understanding.The task suite spans measurement, spatial relations, grounding, displacement, speed, acceleration, orientation, trajectory description, and pose estimation.
- Static Understanding: Static tasks measure geometry, spatial relations, and 3D grounding from visual observations and semantic descriptions.Dimensional Measurement transforms 2D pixels into physical measurements, while Spatial Relation and 3D Video Grounding test relative positioning and language-visual alignment.
- Dynamic Understanding: Dynamic tasks evaluate displacement, path length, speed, acceleration, ego-centric orientation, trajectory description, and pose estimation across video frames.These tasks require tracking motion, integrating displacement with time, interpreting camera rotations, describing paths, and estimating poses from RGB data.
- Benchmark scope and construction: STI-Bench covers Desktop, Indoor, and Outdoor scenarios using Waymo, ScanNet, and Omni6DPose data with camera parameters and object point clouds.These sources support mapping scene information to two-dimensional visual inputs and computing benchmark ground truth.
- Fine-grained adjustment: Scenario-specific distractor adjustment samples logarithmic errors and equalizes the nearest distractor distance to the correct answer.The stated error ranges are [0.5 cm, 5 cm] for Desktop, [5 cm, 50 cm] for Indoor, and [0.5 m, 5 m] for Outdoor.
- QA generation and quality control: The curation pipeline computes numerical and textual ground truth, uses MLLMs to generate QA pairs, and applies multiple rounds of manual filtering.Human quality control addresses inaccurate object descriptions, unreasonable questions or options, and cases where video lacks information available from lidar.
4. Experiments
STI-Bench evaluates proprietary and open-source MLLMs across scenes and tasks, finding that even leading models remain weak at precise spatial-temporal reasoning. Performance varies by scene and task, with quantitative estimation and motion analysis especially challenging.
- Overall Results: Gemini-2.5-Pro achieves the highest average accuracy at 41.4%, followed by Qwen2.5-VL-72B at 40.7% among open-source models.Both results exceed the 20% random-guess baseline, but overall accuracy remains limited.
- Scene-Specific Results: Qwen2.5-VL-72B performs best outdoors at 50.6%, while Gemini-2.5-Pro leads outdoors at 48.7% and desktop scenes at 35.8%.In indoor environments, Claude-3.7-Sonnet leads at 38.2%, followed by Gemini-2.5-Pro at 37.1%.
- Task-Specific Results: Gemini-2.5-Pro reaches 38.7% on Dimensional Measurement and 33.9% on Displacement & Path Length, exposing difficulty with precise quantitative estimation.The strongest task results are higher for Pose Estimation at 62.7% and Spatial Relation at 53.8%.
- Open-Source Models: Among open-source models, Qwen2.5-VL-72B reaches 40.7% average accuracy, InternVL2.5-78B reaches 38.5%, and MiniCPM-V-2.6 reaches 26.9%.The reported results show stronger performance for the larger open-source models in this evaluation.
- Limitations: Despite leading the benchmark, Gemini-2.5-Pro achieves only 41.4% average accuracy, indicating substantial room for improvement in precise spatial-temporal intelligence.The reported limitations leave current MLLMs short of the reliability required for embodied AI or autonomous driving applications.
- Error Analysis: Gemini-2.5-Pro’s error patterns include inaccurate spatial quantification, flawed temporal dynamics understanding, and weak cross-modal grounding and integration.These errors affect estimation of dimensions, distances, coordinates, displacement, path length, speed, acceleration, trajectories, and the use of structured inputs.
5. Conclusion
STI-Bench evaluates MLLMs’ spatial-temporal understanding using real-world videos and QA pairs, revealing substantial limitations in precise physical-world reasoning.
- STI-Bench contains over 300 real-world videos and 2,000 QA pairs spanning robot outdoor, indoor, and desktop scenarios.
- Top-performing models such as Gemini-2.5-Pro achieve around 41.4% average accuracy on the benchmark.
- Models particularly struggle with precise quantitative tasks, including dimensional measurement.
- Error analysis identifies inaccurate spatial quantification, flawed temporal dynamics understanding, and weak cross-modal integration as three key weaknesses.
- The benchmark exposes a gap between current MLLM capabilities and the reliability needed for embodied AI and autonomous driving applications.
A.1. Dataset Interface
The dataset interface addresses inconsistent source-data formats by aligning multimodal information into a unified frame-by-frame representation.
- Some source datasets lack 3D bounding-box information, limiting their use to quantities such as camera displacement and velocity.
- Waymo provides 3D bounding-box information, unlike some other source datasets.
- STI-Bench aims to align different datasets into a unified format by converting their data into frame-by-frame instance-level representations.
A.2. Ground Truth Computation
STI-Bench computes spatial-temporal ground truth from multimodal scene data using unified geometric, pose, motion, orientation, and trajectory representations.
- Ground Truth Sources: Ground-truth annotations combine frame-by-frame camera intrinsics, extrinsics, and precise 3D point-cloud annotations from Waymo, ScanNet, and Omni6DPose.
- Dimensional Measurement: Object dimensions are computed as coordinate-axis ranges, with l_x = x_max - x_min, l_y = y_max - y_min, and l_z = z_max - z_min.
- Dimensional Measurement: Distance between object centers, or between a camera and an object, is computed using Euclidean distance in 3D coordinates.
- Spatial Relation: Spatial relations are determined from coordinate differences by selecting the axis with the greatest difference and using its sign.
- 3D Video Grounding: A 3D video-grounding bounding box represents object dimensions, center position, and optional yaw, pitch, and roll rotations at frame t.
- Pose Estimation: Pose at time t combines the initial position and orientation with extrinsic-derived rotation, translation, and orientation-change terms.
- Displacement and Path Length: Displacement is the straight-line distance between initial and final positions, whereas path length sums consecutive segment distances.
- Speed and Acceleration: Speed divides adjacent-frame displacement by the time interval, and acceleration divides the speed change by that interval.
A.3. Human Involved Quality Control
Automated question-answer generation is supplemented by human quality control to address errors in descriptions, questions, answer options, and video evidence.
- Human intervention corrects inaccuracies or ambiguities in automatically generated object descriptions.
- Quality control addresses logical inconsistencies and irrelevant content in generated questions and answer options.
- Reviewers account for insufficient visual information caused by video occlusions or limited resolution.
B. Adjustment Details
STI-Bench refines distractor options by sampling scene-dependent target errors and jointly moving distractors toward the correct answer while preserving their dispersion.
- Scene-dependent error interval: The benchmark defines scene-specific error intervals for desktop, indoor, and outdoor scenes.The intervals are [0.5 cm, 5 cm], [5 cm, 50 cm], and [0.5 m, 5 m], respectively.
- Log-uniform sampling of target error: Target errors are sampled log-uniformly between each scene’s minimum and maximum error.The method samples u from U(0, 1) and computes e from the interval endpoints.
- Joint adjustment of distractors: All four distractors are jointly shifted toward the correct answer using a common weight.Each adjusted distractor is computed as (1-w)A_d,i + wA_c.
- Joint adjustment of distractors: The weight is selected so the closest adjusted distractor reaches the sampled target error.The remaining distractors stay farther from the correct answer, preserving option dispersion.
C. Detailed Results
The detailed-results section illustrates STI-Bench across outdoor, indoor, and desktop scenarios, including visual examples, fine-grained tables, and model reasoning cases. It also presents a camera-speed example whose visual-estimation procedure selects the closest answer option.
- Scenario examples: Visualization examples cover representative outdoor, indoor, and desktop STI-Bench question-answer pairs.These examples correspond to Figures 8, 9, and 10.
- Reasoning analysis: Over 200 erroneous Gemini 2.5 Pro reasoning processes were sampled, with one correct and one incorrect example presented.The examples are shown in Figures 11 and 12.
- Fine-grained results: Tables 4–6 provide fine-grained performance breakdowns for outdoor, indoor, and desktop categories.The tables organize results by scenario-specific fine-grained categories.
- Motion estimation example: The computed camera speed is approximately 0.2355 m/s, and the closest listed option is C at 0.24 m/s.The options are compared by absolute difference from the estimated speed.