Source-linked AI summary
Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes
Mohsen Gholami, Ahmad Rezaei, Zhou Weimin, Sitong Mao, Shunbo Zhou, Yong Zhang, Mohammad Akbari
TL;DR
Current VLMs remain limited in 3D spatial reasoning for embodied agents because existing benchmarks emphasize single images or static indoor videos rather than ego-centric multi-view observations. The paper introduces Ego3D-Bench and the Ego3D-VLM post-training framework, showing a persistent human–VLM gap while improving spatial reasoning through textual cognitive maps built from global 3D coordinates.
Problem
Existing spatial benchmarks mainly use single images or static indoor videos, leaving ego-centric multi-view spatial reasoning for embodied agents insufficiently evaluated.
Method
The paper introduces Ego3D-Bench and Ego3D-VLM, which constructs textual cognitive maps from estimated global 3D coordinates for referred objects.
Results
Ego3D-Bench reveals a significant gap between human performance and current VLMs, while Ego3D-VLM significantly improves both generalist and 3D spatial VLMs.
Takeaways & Limitations
Together, the benchmark and post-training framework provide tools for evaluating and enhancing VLM 3D spatial understanding in ego-centric multi-view scenarios.
Takeaways & Limitations
Ego3D-VLM relies on the underlying VLM’s reasoning ability, producing little to no improvement for models with limited reasoning capability.
Abstract
from arXiv · showhide
Understanding 3D spatial relationships remains a major limitation of current Vision-Language Models (VLMs). Prior work has addressed this issue by creating spatial question-answering (QA) datasets based on single images or indoor videos. However, real-world embodied AI agents such as robots and self-driving cars typically rely on ego-centric, multi-view observations. To this end, we introduce Ego3D-Bench, a new benchmark designed to evaluate the spatial reasoning abilities of VLMs using ego-centric, multi-view outdoor data. Ego3D-Bench comprises over 8,600 QA pairs, created with significant involvement from human annotators to ensure quality and diversity. We benchmark 16 SOTA VLMs, including GPT-4o, Gemini1.5-Pro, InternVL3, and Qwen2.5-VL. Our results reveal a notable performance gap between human level scores and VLM performance, highlighting that current VLMs still fall short of human level spatial understanding. To bridge this gap, we propose Ego3D-VLM, a post-training framework that enhances 3D spatial reasoning of VLMs. Ego3D-VLM generates cognitive map based on estimated global 3D coordinates, resulting in 12% average improvement on multi-choice QA and 56% average improvement on absolute distance estimation. Ego3D-VLM is modular and can be integrated with any existing VLM. Together, Ego3D-Bench and Ego3D-VLM offer valuable tools for advancing toward human level spatial understanding in real-world, multi-view environments.
1 Huawei Technologies Canada, 2 Huawei Cloud
Figure 1 presents Ego3D-Bench as a five-category 3D spatial benchmark using ego-centric multi-view images, while showing a human–VLM performance gap that Ego3D-VLM narrows.
- Ego3D-Bench spans ego-centric and object-centric perspectives across five spatial categories.
- Ego3D-VLM consistently narrows the performance gap between humans and VLMs.
1 Introduction
Existing spatial-understanding benchmarks mainly use single images or static indoor videos, whereas embodied agents operate with ego-centric, multi-view observations. This motivates Ego3D-Bench and Ego3D-VLM, which respectively evaluate and improve VLM 3D spatial reasoning in this setting.
- 3D spatial understanding supports embodied agents by requiring object localization, distance estimation, and motion reasoning.
- Prior benchmarks mainly evaluate spatial reasoning from single images or static indoor videos, unlike real-world ego-centric multi-view perception.
- Ego3D-Bench contains 8.6K QA pairs curated from NuScenes, Waymo Open Dataset, and Argoverse 1, with human-led construction and quality review.
- Evaluation of 16 SOTA VLMs reveals a significant gap between human performance and current VLMs, with coherent world-model construction hypothesized as a key limitation.
- Ego3D-VLM creates a textual cognitive map centered on the ego that locates prompt-referred objects in 3D coordinate space.
- The framework estimates 2D referred-object locations and metric depth, converts them into camera-coordinate 3D points, and transforms points across views into global coordinates.
2 Related Work
Prior VLM spatial benchmarks and models largely rely on single images, indoor static scenes, or point-cloud representations, leaving ego-centric outdoor multi-view reasoning underrepresented.
- 3D Spatial Benchmarks/Datasets for VLMs: Existing spatial datasets primarily use single-view images or videos from static indoor scenes, unlike embodied agents’ ego-centric multi-view observations.These settings emphasize passive observation in relatively static environments.
- 3D Spatial Benchmarks/Datasets for VLMs: All-Angle introduced multi-view evaluation, but its cameras observe scenes from different directions like surveillance systems rather than an agent-centered viewpoint.Ego3D-Bench is positioned as the first ego-centric multi-view benchmark using dynamic outdoor scenes.
- 3D Spatial VLMs: 3D spatial VLMs target tasks including 3D grounding, spatial reasoning, depth estimation, and distance measurement through point-cloud or image-based inputs.The literature is organized into models using point clouds or models operating directly on images.
- 3D Spatial VLMs: Point-cloud approaches provide rich spatial information but are difficult to reconstruct dynamically, struggle with sparse views, and increase inference time by over 10×.Image-based approaches avoid this representation while addressing related spatial tasks.
- 3D Spatial VLMs: Prior image-based models use depth, camera poses, BEV, synthetic data, region proposals, or tool-assisted spatial reasoning, but many target indoor static scenes or limited quantitative relationships.Ego3D-VLM differs as a post-training method applicable to existing VLMs for ego-centric multi-view reasoning.
3 Ego3D-Bench
Ego3D-Bench evaluates VLM 3D spatial understanding with over 8.6K QA pairs from ego-centric multi-view outdoor datasets, emphasizing cross-view reasoning and multiple spatial tasks.
- Benchmark Overview: Ego3D-Bench focuses on ego-centric multi-view outdoor data relevant to self-driving and robotics.It is designed to quantitatively evaluate 3D spatial understanding from multi-view images.
- Benchmark Overview: Over 8.6K QA pairs span five categories and are constructed from nuScenes, Waymo Open Dataset, and Argoverse 1.The source datasets provide 6, 7, and 5 camera views respectively and cover urban, highway, and rural environments.
- Benchmark Construction: Human annotators support source-scene review, object identification, dataset construction, and rigorous quality review.Outdoor scenes contain multiple similar objects, making unique target references challenging.
- Benchmark Construction: Questions use predefined templates and rule-based answer functions leveraging ground-truth 3D annotations.Challenging categories receive additional handling during answer generation.
- Benchmark Tasks: The benchmark excludes questions answerable from one view and includes ego-centric and object-centric perspectives across five multi-view tasks.The tasks include absolute and relative distance, localization, motion reasoning, and travel time.
- Evaluation: Most questions use accuracy, while two absolute distance-estimation tasks use RMSE in meters.The benchmark therefore evaluates both categorical QA and metric distance estimation.
4 Post-Training 3D spatial understanding : Ego3D-VLM
Ego3D-VLM converts multi-view detections and depth estimates into globally aligned 3D coordinates, then generates a compact cognitive map that grounds VLM query answering.
- Camera Coordinate Transformation: The framework detects prompt-referred objects in each camera view using referring expression comprehension models.Detected objects are represented through 2D bounding-box centers and matched referring expressions.
- Camera Coordinate Transformation: A metric depth estimator supplies depth at each object center, which is projected into 3D camera coordinates using camera intrinsics.The resulting points provide spatial information for each detected object.
- Camera Coordinate Transformation: 3D points from all views are transformed into a global coordinate system defined by the front-camera viewpoint.This creates a unified 3D scene representation from multi-view images.
- Relational Scaling: 3D points are relationally scaled using canonical object heights, such as 1.7 m for humans, to produce physically plausible scales without ground-truth depth.The scale is computed as s = hcs/hest from observed and canonical heights.
- Creating a Cognitive Map: The cognitive map generator organizes referred objects by global 3D position and referring expression into a textual, ego-centric world model.The map links each object to its spatial position and originating viewpoint in a compact, human-interpretable form.
- VLM Reasoning: The VLM answers queries using the cognitive map together with multi-view images, combining structured spatial grounding with appearance and fine-grained visual cues.The cognitive map supplies spatial structure that the images alone may not organize explicitly.
5 Evaluation on Ego3D-Bench
The evaluation compares generalist VLMs, 3D-VLMs, tool-augmented systems, and ablations on Ego3D-Bench. Results show substantial human–VLM gaps, while Ego3D-VLM improves spatial reasoning, especially when structured 3D information is provided.
- Evaluation setup: The evaluation covers generalist VLMs, 3D-VLMs, REC-plus-depth systems, and ablation studies on Ego3D-Bench.The benchmark includes chance and human comparisons, blind-model analyses, and evaluations on alternative multi-view settings.
- Generalist VLMs: Smaller 3B and 8B VLMs operate near chance, whereas larger models improve but remain below human performance.Ego3D-VLM yields 12% average relative improvement in Accuracy and 56% in RMSE across model sizes and tasks.
- Performance analysis: Travel time, localization, and object-centric absolute distance are hardest for VLMs, with 40%-45% average accuracy.Cognitive maps narrow the human gap on absolute-distance tasks, and augmented models surpass humans on object-centric absolute distance, but localization remains below human proficiency.
- Blind VLMs: Blind VLMs score 5% below vision-enabled VLMs but 16.4% above chance, showing that text-only world knowledge contributes to spatial answers.Vision-enabled models score 58.8% versus 53.8% for blind models.
- 3D-VLMs: SpaceThinker-Qwen2.5-3B achieves the highest overall 3D-VLM performance, while adding Ego3D-VLM improves multiple-choice results by 3% and reduces absolute-distance RMSE by more than 4 meters.The result supports the importance of dedicated 3D spatial pretraining and an end-to-end architecture.
- Perception and reasoning: Adding REC and depth estimation improves baselines but remains inferior to Ego3D-VLM, which integrates depth information into a unified map representation.Ground-truth 2D boxes indicate that baseline limitations concern 3D perception and reasoning more than 2D perception.
- Ablation studies: Ablations show that estimated camera parameters can substantially improve performance, while relational scaling reduces RMSE by 2.5 meters.The full configuration is v4, which includes all components.
6 Conclusion and Future Work
The paper introduces Ego3D-Bench for ego-centric multi-view spatial understanding and Ego3D-VLM as a post-training solution. It reports a significant human–VLM performance gap and identifies ego-centric multi-view fine-tuning and 3D projection modules as future directions.
- Conclusion: Ego3D-Bench evaluates VLM spatial understanding on ego-centric multi-view images, while Ego3D-VLM enhances their performance through post-training.The paper reports a significant gap between human scores and VLMs.
- Future work: Future work should explore fine-tuning VLMs with ego-centric multi-view QA and incorporating Ego3D-VLM’s 3D projection modules during fine-tuning.The paper states that its limitations are provided in Appendix A.9.
A Appendix
The appendix collects supplementary discussions and experiments covering benchmark construction, additional evaluations, efficiency, robustness, and limitations.
- Supplementary material: The appendix includes discussions of limitations and future work, benchmark comparisons, dataset creation, and open-QA question templates.It also covers results for the three source datasets and more generalist VLMs.
- Supplementary analyses: Additional analyses examine inference time, robustness in challenging conditions, qualitative results, and prompt structure.These analyses supplement the main benchmark and model evaluations.
A.1 Ego3D-Bench vs. Other Benchmarks
Ego3D-Bench targets ego-centric multi-view spatial reasoning, unlike benchmarks built around stationary multi-view cameras or other alternative configurations.
- Benchmark settings: Ego3D-Bench is designed for ego-centric multi-view scenarios relevant to AI agents, whereas All-Angle Bench and VSI-Bench use alternative multi-view settings.Figure 7 compares the configurations of these benchmarks.
A.2 Ego3D-Bench: Further Details
Ego3D-Bench questions are generated from structured templates that substitute object descriptions, camera views, and motion directions. Distance options are randomly generated with a minimum separation constraint.
- Templates replace object placeholders with descriptions and view placeholders with camera-view labels such as Front-Right or Left.
- Motion-reasoning templates substitute the direction placeholder with one of four cardinal directions.
- Distance-answer options are randomly generated while maintaining at least 8 meters between any two options.
A.3 Benchmarking the Thinking of Ego3D-Bench as an Open-QA
The open-ended evaluation examines VLM thinking for numerical distance questions and compares Ego3D-VLM with baselines, efficiency, robustness, and limitations. Results indicate improved reasoning and benchmark consistency, but gains depend on model reasoning capacity and imperfect spatial tools.
- Open-ended thinking evaluation: GT thinking is generated with GPT-4o and ground-truth cognitive maps for ego-centric and object-centric absolute-distance questions.
- Open-ended thinking evaluation: Ego3D-VLM thinking is reported as better than the baselines on the open-ended evaluation.
- Inference analysis: 0.6% memory and 31% latency overhead are reported for Ego3D-VLM over InternVL3-78B, with latency attributed to additional reasoning from the cognitive map.
- Ablations and robustness: 60.1% to 79.4% accuracy is achieved with ground-truth cognitive maps, while the proposed solution remains more robust than the baseline under low brightness, motion blur, and occlusion.
- Benchmark consistency: Performance deviations across source-dataset splits are minimal, supporting benchmark consistency despite differing numbers of camera viewpoints.
- Limitations: Ego3D-VLM provides little to no improvement for models with limited reasoning ability, and redundant referenced-object information can confuse or mislead VLMs.
- Limitations: Metric depth estimation outdoors is an additional limitation because the proposed relational scaling method is approximate and not fully reliable.