Source-linked AI summary
Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?
Zihao Dongfang, Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Danda Pani Paudel, Luc Van Gool, Kailun Yang, Xuming Hu
TL;DR
Most MLLM research centers on pinhole images, leaving spatial reasoning in 360° panoramic scenes underexplored. OSR-Bench combines large-scale, geometry-grounded QA, distractor-based robustness tests, and two-stage evaluation; results show task-dependent cognitive-map effects and vulnerability to distractors.
Problem
Method
OSR-Bench pairs over 153,000 geometry-grounded questions across three spatial tasks with negative distractors and a two-stage evaluation of cognitive maps and QA across eight zero-shot MLLMs.
Source: §1 · pp. 2–3Results
Cognitive maps help all tested models on counting but usually not direction, and negative distractors degrade spatial reasoning while all models show hallucination tendencies.
Source: §3.3.1 · pp. 7–8Takeaways & Limitations
OSR-Bench provides fine-grained, interpretable metrics for evaluating panoramic MLLM spatial reasoning and robustness.
Source: §4 · p. 9The benchmark primarily focuses on indoor environments, which may not capture outdoor complexities such as dynamic objects, weather, and large-scale spatial structures.
Source: §4 · p. 9
Abstract
from arXiv · showhide
The 180x360 omnidirectional field of view captured by 360-degree cameras enables their use in a wide range of applications such as embodied AI and virtual reality. Although recent advances in multimodal large language models (MLLMs) have shown promise in visual-spatial reasoning, most studies focus on standard pinhole-view images, leaving omnidirectional perception largely unexplored. In this paper, we ask: Are MLLMs ready for omnidirectional spatial reasoning? To investigate this, we introduce OSR-Bench, the first benchmark specifically designed for this setting. OSR-Bench includes over 153,000 diverse question-answer pairs grounded in high-fidelity panoramic indoor scene maps. It covers key reasoning types including object counting, relative distance, and direction. We also propose a negative sampling strategy that inserts non-existent objects into prompts to evaluate hallucination and grounding robustness. For fine-grained analysis, we design a two-stage evaluation framework assessing both cognitive map generation and QA accuracy using rotation-invariant matching and a combination of rule-based and LLM-based metrics. We evaluate eight state-of-the-art MLLMs, including GPT-4o, Gemini 1.5 Pro, and leading open-source models under zero-shot settings. Results show that current models struggle with spatial reasoning in panoramic contexts, highlighting the need for more perceptually grounded MLLMs. OSR-Bench and code will be released at: https://huggingface.co/datasets/UUUserna/OSR-Bench
1 Introduction
Most MLLM spatial-reasoning benchmarks use pinhole-view images, leaving 360° panoramic reasoning underexplored. OSR-Bench addresses this gap with geometry-grounded spatial questions, negative distractors, and a two-stage evaluation of eight MLLMs.
1 Introduction
- Most MLLM approaches use 2D pinhole images, while omnidirectional views contain spatial information relevant to embodied AI and spatial reasoning.The paper frames panoramic spatial reasoning as an underexplored setting.Source: §1 · p. 2
- OSR-Bench is introduced as the first benchmark specifically designed to evaluate MLLM spatial reasoning in 360° panoramic environments.It uses high-fidelity 3D layout annotations from ReplicaPano and DeepPanoContext to construct top-down omni-cognitive maps as references and structured priors for question generation.Source: §1 · pp. 2–3
- The scalable templated pipeline creates geometry-grounded questions about object counting, relative distance, and relative direction, and adds non-existent distractors to probe hallucination and visual grounding.The benchmark contains over 153,000 QA pairs across 4,100 panoramic images, 1,526 indoor scenes, and 65 object categories.Source: §1 · p. 2
- The two-stage protocol evaluates cognitive-map construction and QA performance, using rotation-invariant Hungarian matching plus rule-based and LLM-based evaluators.Eight proprietary and open-source MLLMs are tested in strict zero-shot settings with greedy decoding and no task-specific fine-tuning.Source: §1 · p. 3
2 Related Work and Motivation
Prior visual and panoramic benchmarks cover multimodal tasks, hallucination, reconstruction, and scene understanding, but leave gaps in unified spatial QA across 360° scenes. OSR-Bench is proposed to address those gaps with systematic panoramic spatial-reasoning assessment.
2 Related Work and Motivation
- Existing visual-reasoning benchmarks span VQA [31] [32] [33], broader MLLM evaluation [34] [36], multi-turn dialogue [38], hallucination [24] [39], and spatial reasoning [40] [41].The review also notes that panoramic scene work covers reconstruction [23] [22], segmentation [7] [9] [10] [11] [12] [13] [14] [15] [17], and specialized datasets such as Pandora, 360+x [44], and R3DS.Source: §2 · p. 3
- Most existing benchmarks focus on standard field-of-view images and overlook object relationships, counting, and relative direction in panoramic scenes.The paper proposes OSR-Bench as a benchmark specifically designed to assess spatial reasoning in 360° environments.Source: §2 · p. 3
- Panoramic VQA datasets lack comprehensive spatial reasoning, especially object positions across the full 360° field, while handcrafted QA may miss complexity and increase hallucination risk.Other limitations include task mismatch and translation overhead when converting geometric reconstruction or dense-prediction tasks into natural-language queries.Source: §2 · p. 4
3 OSR-Dataset & Benchmark
OSR-Bench builds a large panoramic spatial-reasoning dataset and evaluates models through cognitive-map construction and map-grounded question answering. Results show that cognitive maps help counting but often hinder direction, while negative distractors expose hallucination and reasoning weaknesses.
3.1 Dataset Construction
- Ground-truth omni-cognitive maps and object pools from panoramic images and 3D layouts feed predefined templates that generate spatial-reasoning questions.The maps also serve as references for evaluating models prompted to construct cognitive maps.Source: §3.1 · pp. 4–5
- Approximately 153k QA pairs cover 4,100 panoramic images, three spatial-reasoning categories, and equal numbers of regular and negative-sampling pairs.The images span 65 object categories and 1,526 photorealistic synthetic indoor scenes.Source: §3.1 · p. 5
3.2 Our Evaluation Method
- Evaluation first assesses cognitive-map construction, then map-grounded QA, using rotation-invariant matching and rule-based or LLM-based scoring.The framework selects the best F1 across rotations; counting uses percentage-based accuracy, while distance and direction use binary scoring.Source: §3.2 · pp. 5–6
3.3 Benchmark Experiments
- Eight MLLMs, including proprietary and open-source models, were evaluated zero-shot without task-specific fine-tuning using greedy decoding.The evaluated set includes Gemini-1.5-Pro, GPT-4o, InternVL2_5-78B, LLaVA-v1.5-13B, Llama-3.2-90B-Vision-Instruct, Qwen2.5-VL-72B-Instruct, deepseek-vl2, and Janus-Pro-7B.Source: §3.3 · p. 6
3.3.1 Results
- 2% to 15% gains in object counting with cognitive maps contrast with seven of eight models performing better without maps on relative direction.Only Llama-3.2-90B improved on relative direction with a cognitive map; imperfect maps may interfere with directional reasoning.Source: §3.3.1 · pp. 7–8
- 0.767 to 0.884 CHAIR_S scores (lower is better) indicate substantial hallucination under negative sampling across all models.GPT-4o had the lowest CHAIR_I (0.198) and CHAIR_instance (0.180); Qwen2.5-VL-72B had the lowest CHAIR_S (0.767).Source: §3.3.1 · p. 8
- 99.50% for Gemini-1.5-Pro and 97.50% for GPT-4o were the highest reported properly formatted cognitive-map success rates.Rates were 10.84% for LLaVA-v1.5-13B and 4.29% for Janus-Pro-7B; low formatting success was linked to poor F1 scores.Source: Table 5 · p. 8, §3.3.1 · p. 9
- +0.037 for Gemini-1.5-Pro object counting without cognitive maps and +0.061 for relative direction were reported with thinking-mode prompts.Structured reasoning improved performance across most tasks, with generally smaller gains when cognitive maps were provided.Source: Table 6 · p. 9, §3.3.1 · p. 9
4 Conclusion
OSR-Dataset and OSR-Bench evaluate MLLM spatial reasoning in 360° indoor panoramas, and tests of eight models reveal substantial performance gaps. The authors note the benchmark’s indoor scope and identify future extensions and broader-impact concerns.
4 Conclusion
- Over 153,000 QA pairs across three spatial tasks ground OSR-Dataset and OSR-Bench in high-fidelity 3D scene layouts, with negative sampling and two-stage evaluation.The evaluation combines rotation-invariant cognitive-map alignment with rule-based and LLM-based QA metrics.Source: §4 · p. 9
- Evaluated MLLMs show significant spatial-reasoning gaps in panoramic settings, motivating more perceptually grounded architectures.The benchmark evaluates eight state-of-the-art MLLMs.Source: §4 · p. 9
- The benchmark focuses primarily on indoor environments and may not capture outdoor dynamics, weather, or large-scale spatial structures.The paper also flags privacy, surveillance, and military-misuse concerns associated with deployment.Source: §4 · p. 9
- Future work includes real-world scenes, out-of-distribution generalization, dynamic-scene tasks, and post-training strategies for panoramic reasoning.The authors also identify possible privacy and surveillance concerns, including military or surveillance misuse.Source: §4 · p. 9
A Algorithm Detail
This appendix section provides further details about the OSR-Bench construction pipeline.
A Algorithm Detail
- The appendix section introduces further details on the OSR-Bench construction pipeline.The passage does not specify additional pipeline steps.Source: A Algorithm Detail · p. 14
A.1 QA Pair Generation
QA pairs are generated from omni-cognitive maps with structured templates for counting, relative distance, and relative direction. Scene source determines the question cap, while sampling rules constrain which spatial questions can be generated.
A.1 QA Pair Generation
- For each scene, templates generate counting, distance, and direction questions from an omni-cognitive map; the per-type cap is 10 for DeepPanoContext images and 5 for ReplicaPano images.The maximum is determined based on similarity of the scenes corresponding to the images.Source: §A.1 · p. 14
- Counting questions sample objects without replacement; distance and direction questions use unique positioning objects, and those spatial questions are omitted when fewer than five object categories are present.Direction questions also require the three object types to occupy different cognitive-map grid units.Source: §A.1 · p. 14
- Distance answers select the nearest candidate by Euclidean distance in grid coordinates, while direction answers derive from vector relationships among positioned objects.The algorithm samples distinct candidates and assigns counts from the class-count dictionary.Source: §A.1 · p. 14
A.2 Negative Sampling for Hallucination Testing
Negative sampling adds absent objects to map-generation prompts and QA generation to test hallucination robustness. Distractors are selected from global frequency and co-occurrence statistics, then handled with absent-object answers in the generated QA pairs.
A.2 Negative Sampling for Hallucination Testing
- Negative sampling adds two absent-object distractor types: globally frequent objects missing from a scene and objects commonly co-occurring with present objects but absent there.The method derives these sets from global object-frequency and co-occurrence statistics.Source: §A.2 · p. 15
- Each image receives five frequent-absent and five co-occurring-absent distractors for negative sampling.The passage sets the number of each distractor type to five.Source: §A.2 · p. 15
- For augmented QA, absent distractors receive count answers of zero or appropriate “not found” responses in spatial questions.The augmented object set combines present objects with both types of negative distractors.Source: §A.2 · p. 15
B Dataset Detail
OSR-Bench reports separate question totals for standard and negative-sampling datasets across object counting, relative distance, and relative direction.
B Dataset Detail
- The standard dataset contains 74,326 questions: 26,240 on relative distance, 24,988 on object counting, and 23,098 on relative direction.Source: B Dataset Detail · p. 15
- The negative-sampling dataset contains 78,986 questions: 27,125 on object counting, 26,250 on relative distance, and 25,611 on relative direction.Source: B Dataset Detail · p. 15
- Figure 6 summarizes object frequency across the dataset, while Figure 7 shows object-counting answer distributions.Source: Figure 6 · p. 16, Figure 7 · p. 16
C Prompt Template
The prompt templates guide models to map objects in panoramas or answer questions directly, with different post-prompts selecting Vanilla or Think Mode and LLM prompts evaluating responses.
C.1 Omni-Cognitive Map Generate Prompt
- The cognitive-map prompt asks models to analyze a panorama and construct a structured spatial representation, with the object pool drawn from ground truth or augmented with distractors.Source: §C.1 · p. 16
Cognitive Map Prompt
- For each listed category, models estimate every instance’s center on a 10x10 scene grid while preserving relative positions and accounting for distance and occlusion.Source: Cognitive Map Prompt · p. 16
- Figures 8 and 9 show answer distributions for relative-distance and relative-direction questions, respectively.Source: Figure 8 · p. 17, Figure 9 · p. 17
- The map output is a dictionary of category names and coordinate lists, using integer coordinates within 10.Source: Cognitive Map Prompt · p. 18
C.2 Pre and Post Prompt
- Direct-question inputs place the image token, pre-prompt, question, and post-prompt in that order.Source: §C.2 · p. 18
- Vanilla Mode and Think Mode are selected by changing the post-prompt in the reasoning pipeline.Source: §C.2 · p. 18, Figure 10 · p. 18
Pre-Prompt
- The pre-prompt asks models to answer a question using the panorama and predicted object-center locations.Source: Pre-Prompt · p. 18
C.3 LLM Evaluator Prompt
- The LLM evaluator uses question-specific prompts with binary scores and confidence levels, and DeepSeek-V3 evaluates model responses.Source: §C.3 · p. 18
D Evaluation Details
Evaluation details specify deterministic decoding, approximate task duration, and hardware requirements by model size.
D Evaluation Details
- All models use greedy decoding: temperature 0, top-p and top-k 1 for proprietary models, and do_sample=False for open-source models.Source: D Evaluation Details · p. 20
- A complete evaluation task takes approximately 4–5 days; models below 13B use one A40 GPU, 13B–30B models use three A40 GPUs, and models above 70B use three A800 GPUs.Source: D Evaluation Details · p. 20
- Figure 11 is labeled as an overview of the OSR benchmark.Source: Figure 11 · p. 20
E More Cases
The included examples comprise a visualization of cognitive-map generation and a case labeled as good in Think Mode.
E More Cases
- Figure 12 presents a visualization of Omni-Cognitive map generation.Source: Figure 12 · p. 21
- Figure 13 presents a case identified as good in Think Mode.Source: Figure 13 · p. 21