Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes

• Ego3D-Bench spans ego-centric and object-centric perspectives across five spatial categories.Source: Figure 1 · p. 1
• Ego3D-VLM consistently narrows the performance gap between humans and VLMs.Source: Figure 1 · p. 1

• 3D spatial understanding supports embodied agents by requiring object localization, distance estimation, and motion reasoning.Source: §1 · p. 2
• Prior benchmarks mainly evaluate spatial reasoning from single images or static indoor videos, unlike real-world ego-centric multi-view perception.Source: §1 · p. 2
• Ego3D-Bench contains 8.6K QA pairs curated from NuScenes, Waymo Open Dataset, and Argoverse 1, with human-led construction and quality review.Source: §1 · p. 2
• Evaluation of 16 SOTA VLMs reveals a significant gap between human performance and current VLMs, with coherent world-model construction hypothesized as a key limitation.Source: §1 · p. 2
• Ego3D-VLM creates a textual cognitive map centered on the ego that locates prompt-referred objects in 3D coordinate space.Source: §1 · p. 2
• The framework estimates 2D referred-object locations and metric depth, converts them into camera-coordinate 3D points, and transforms points across views into global coordinates.Source: §1 · p. 2

3D Spatial Benchmarks/Datasets for VLMs

• Existing spatial datasets primarily use single-view images or videos from static indoor scenes, unlike embodied agents’ ego-centric multi-view observations.These settings emphasize passive observation in relatively static environments.Source: §2 · p. 3
• All-Angle introduced multi-view evaluation, but its cameras observe scenes from different directions like surveillance systems rather than an agent-centered viewpoint.Ego3D-Bench is positioned as the first ego-centric multi-view benchmark using dynamic outdoor scenes.Source: §2 · p. 3

3D Spatial VLMs

• 3D spatial VLMs target tasks including 3D grounding, spatial reasoning, depth estimation, and distance measurement through point-cloud or image-based inputs.The literature is organized into models using point clouds or models operating directly on images.Source: §2 · p. 3
• Point-cloud approaches provide rich spatial information but are difficult to reconstruct dynamically, struggle with sparse views, and increase inference time by over 10×.Image-based approaches avoid this representation while addressing related spatial tasks.Source: §2 · p. 3
• Prior image-based models use depth, camera poses, BEV, synthetic data, region proposals, or tool-assisted spatial reasoning, but many target indoor static scenes or limited quantitative relationships.Ego3D-VLM differs as a post-training method applicable to existing VLMs for ego-centric multi-view reasoning.Source: §2 · p. 3

Benchmark Overview

• Ego3D-Bench focuses on ego-centric multi-view outdoor data relevant to self-driving and robotics.It is designed to quantitatively evaluate 3D spatial understanding from multi-view images.Source: §3 · p. 3
• Over 8.6K QA pairs span five categories and are constructed from nuScenes, Waymo Open Dataset, and Argoverse 1.The source datasets provide 6, 7, and 5 camera views respectively and cover urban, highway, and rural environments.Source: §3 · p. 4

Benchmark Construction

• Human annotators support source-scene review, object identification, dataset construction, and rigorous quality review.Outdoor scenes contain multiple similar objects, making unique target references challenging.Source: §3.1 · p. 4
• Questions use predefined templates and rule-based answer functions leveraging ground-truth 3D annotations.Challenging categories receive additional handling during answer generation.Source: §3.1 · p. 4

Benchmark Tasks

• The benchmark excludes questions answerable from one view and includes ego-centric and object-centric perspectives across five multi-view tasks.The tasks include absolute and relative distance, localization, motion reasoning, and travel time.Source: §3.2 · pp. 4–5

Evaluation

• Most questions use accuracy, while two absolute distance-estimation tasks use RMSE in meters.The benchmark therefore evaluates both categorical QA and metric distance estimation.Source: §3.2 · p. 5

Camera Coordinate Transformation

• The framework detects prompt-referred objects in each camera view using referring expression comprehension models.Detected objects are represented through 2D bounding-box centers and matched referring expressions.Source: §4 · p. 5
• A metric depth estimator supplies depth at each object center, which is projected into 3D camera coordinates using camera intrinsics.The resulting points provide spatial information for each detected object.Source: §4 · p. 5
• 3D points from all views are transformed into a global coordinate system defined by the front-camera viewpoint.This creates a unified 3D scene representation from multi-view images.Source: §4 · p. 5

Relational Scaling

• 3D points are relationally scaled using canonical object heights, such as 1.7 m for humans, to produce physically plausible scales without ground-truth depth.The scale is computed as s = hcs/hest from observed and canonical heights.Source: §4 · pp. 5–6

Creating a Cognitive Map

• The cognitive map generator organizes referred objects by global 3D position and referring expression into a textual, ego-centric world model.The map links each object to its spatial position and originating viewpoint in a compact, human-interpretable form.Source: §4 · p. 6

VLM Reasoning

• The VLM answers queries using the cognitive map together with multi-view images, combining structured spatial grounding with appearance and fine-grained visual cues.The cognitive map supplies spatial structure that the images alone may not organize explicitly.Source: §4 · p. 6

Evaluation setup

• The evaluation covers generalist VLMs, 3D-VLMs, REC-plus-depth systems, and ablation studies on Ego3D-Bench.The benchmark includes chance and human comparisons, blind-model analyses, and evaluations on alternative multi-view settings.Source: §5 · p. 6, §5.1 · p. 6

Generalist VLMs

• Smaller 3B and 8B VLMs operate near chance, whereas larger models improve but remain below human performance.Ego3D-VLM yields 12% average relative improvement in Accuracy and 56% in RMSE across model sizes and tasks.Source: §5.1 · p. 7

Performance analysis

• Travel time, localization, and object-centric absolute distance are hardest for VLMs, with 40%-45% average accuracy.Cognitive maps narrow the human gap on absolute-distance tasks, and augmented models surpass humans on object-centric absolute distance, but localization remains below human proficiency.Source: §5.1 · pp. 7–8, Figure 6 · p. 7

Blind VLMs

• Blind VLMs score 5% below vision-enabled VLMs but 16.4% above chance, showing that text-only world knowledge contributes to spatial answers.Vision-enabled models score 58.8% versus 53.8% for blind models.Source: §5.1 · p. 8

3D-VLMs

• SpaceThinker-Qwen2.5-3B achieves the highest overall 3D-VLM performance, while adding Ego3D-VLM improves multiple-choice results by 3% and reduces absolute-distance RMSE by more than 4 meters.The result supports the importance of dedicated 3D spatial pretraining and an end-to-end architecture.Source: §5.2 · p. 8, Table 3 · p. 8

Perception and reasoning

• Adding REC and depth estimation improves baselines but remains inferior to Ego3D-VLM, which integrates depth information into a unified map representation.Ground-truth 2D boxes indicate that baseline limitations concern 3D perception and reasoning more than 2D perception.Source: §5.3 · pp. 8–9, Table 4 · p. 8, §5.5 · pp. 9–10

Ablation studies

• Ablations show that estimated camera parameters can substantially improve performance, while relational scaling reduces RMSE by 2.5 meters.The full configuration is v4, which includes all components.Source: §5.5 · p. 9, Table 6 · p. 9

Conclusion

• Ego3D-Bench evaluates VLM spatial understanding on ego-centric multi-view images, while Ego3D-VLM enhances their performance through post-training.The paper reports a significant gap between human scores and VLMs.Source: §6 · p. 10

Future work

• Future work should explore fine-tuning VLMs with ego-centric multi-view QA and incorporating Ego3D-VLM’s 3D projection modules during fine-tuning.The paper states that its limitations are provided in Appendix A.9.Source: §6 · p. 10

Supplementary material

• The appendix includes discussions of limitations and future work, benchmark comparisons, dataset creation, and open-QA question templates.It also covers results for the three source datasets and more generalist VLMs.Source: A Appendix · p. 14

Supplementary analyses

• Additional analyses examine inference time, robustness in challenging conditions, qualitative results, and prompt structure.These analyses supplement the main benchmark and model evaluations.Source: A Appendix · p. 14

Benchmark settings

• Ego3D-Bench is designed for ego-centric multi-view scenarios relevant to AI agents, whereas All-Angle Bench and VSI-Bench use alternative multi-view settings.Figure 7 compares the configurations of these benchmarks.Source: §A.1 · p. 14, Figure 7 · p. 14

• Templates replace object placeholders with descriptions and view placeholders with camera-view labels such as Front-Right or Left.Source: §A.2 · p. 14
• Motion-reasoning templates substitute the direction placeholder with one of four cardinal directions.Source: §A.2 · p. 14
• Distance-answer options are randomly generated while maintaining at least 8 meters between any two options.Source: §A.2 · p. 15

Open-ended thinking evaluation

• GT thinking is generated with GPT-4o and ground-truth cognitive maps for ego-centric and object-centric absolute-distance questions.Source: §A.3 · pp. 15–16, Table 9 · p. 16
• Ego3D-VLM thinking is reported as better than the baselines on the open-ended evaluation.Source: §A.3 · p. 16

Inference analysis

• 0.6% memory and 31% latency overhead are reported for Ego3D-VLM over InternVL3-78B, with latency attributed to additional reasoning from the cognitive map.Source: §A.5 · p. 16, Table 11 · p. 17

Ablations and robustness

• 60.1% to 79.4% accuracy is achieved with ground-truth cognitive maps, while the proposed solution remains more robust than the baseline under low brightness, motion blur, and occlusion.Source: §A.6 · p. 17, Table 12 · p. 17

Benchmark consistency

• Performance deviations across source-dataset splits are minimal, supporting benchmark consistency despite differing numbers of camera viewpoints.Source: §A.7 · p. 17, Table 13 · p. 18

Limitations

• Ego3D-VLM provides little to no improvement for models with limited reasoning ability, and redundant referenced-object information can confuse or mislead VLMs.Source: §A.9 · p. 17
• Metric depth estimation outdoors is an additional limitation because the proposed relational scaling method is approximate and not fully reliable.Source: §A.9 · pp. 17–18