Communicating about Space: Language-Mediated Spatial Integration Across Partial Views

Ankur Sikarwar; Debangan Mishra; Sudarshan Nikhil; Ponnurangam Kumaraguru; Aishwarya Agrawal

Source-linked AI summary

Communicating about Space: Language-Mediated Spatial Integration Across Partial Views

Ankur Sikarwar, Debangan Mishra, Sudarshan Nikhil, Ponnurangam Kumaraguru, Aishwarya Agrawal

arXiv:2603.27183v2Mar 28, 2026cs.CV

TL;DR

The paper asks whether MLLMs can communicate across partial, viewpoint-dependent observations to build shared spatial models. COSMIC evaluates this ability and finds partial anchor-grounding success but near-chance cognitive mapping, while humans achieve 95.22% accuracy.

Problem
Existing spatial benchmarks provide limited evidence about whether MLLMs can communicate across partial views to build consistent shared spatial models.
Method
COSMIC evaluates two agents integrating distinct egocentric views through multi-turn dialogue across 899 indoor scenes and 1,250 questions spanning five spatial tasks.
Results
MLLMs show a hierarchy from partial anchor grounding to deteriorating relational reasoning and near-chance cognitive mapping; Gemini-3-Pro-Thinking reaches 71.82% versus humans’ 95.22%.
Takeaways & Limitations
Current MLLMs remain far from forming shared spatial models, producing verbose dialogue that fails to converge across turns.
Takeaways & Limitations
COSMIC uses static, controlled indoor scenes, fixed viewpoints, and limited object categories, omitting dynamic environments, viewpoint changes, richer vocabularies, and action-based information gathering.

Abstract

from arXiv · show

Humans build shared spatial understanding by communicating partial, viewpoint-dependent observations. We ask whether Multimodal Large Language Models (MLLMs) can do the same, aligning distinct egocentric views through dialogue to form a coherent, allocentric mental model of a shared environment. To study this systematically, we introduce COSMIC, a benchmark for Collaborative Spatial Communication. In this setting, two static MLLM agents observe a 3D indoor environment from different viewpoints and exchange natural-language messages to solve spatial queries. COSMIC contains 899 diverse scenes and 1250 question-answer pairs spanning five tasks. We find a capability hierarchy, MLLMs are most reliable at identifying shared anchor objects across views, perform worse on relational reasoning, and largely fail at building globally consistent maps, performing near chance, even for frontier models. Moreover, we find thinking capability yields gains in anchor grounding, but is insufficient for higher-level spatial communication. To contextualize model behavior, we collect 250 human-human dialogues. Humans achieve 95% aggregate accuracy, while the best model, Gemini-3-Pro-Thinking, reaches 72%, leaving substantial room for improvement. Moreover, human conversations grow more precise as partners align on a shared spatial understanding, whereas MLLMs keep exploring without converging, suggesting limited capacity to form and sustain a robust shared mental model throughout the dialogue. Our code and data is available at https://github.com/ankursikarwar/Cosmic.

1. Introduction · 2. Related Work · 3. COSMIC Benchmark

COSMIC benchmarks whether MLLM agents can integrate complementary egocentric views through dialogue into shared spatial understanding. It combines a five-task diagnostic hierarchy with diverse procedurally generated indoor scenes and evaluates object-, relation-, and map-level reasoning.

1. Introduction: COSMIC places two static MLLM agents with complementary egocentric views in one indoor scene and requires multi-turn dialogue to answer spatial questions.The Answerer receives the question, while the Helper contributes observations from a different viewpoint.
1. Introduction: 899 scenes and 1,250 question-answer pairs span five tasks covering object-level, relation-level, and map-level collaborative spatial reasoning.The benchmark is designed as a diagnostic evaluation of cross-view grounding, relational reasoning, and allocentric integration.
2. Related Work: Prior benchmarks evaluate single-view spatial understanding, including positional reasoning, mental rotation, folding, and navigation, but do not address spatial reasoning under egocentric partial observability.Related work also studies multimodal agents cooperating with complementary observations in games and navigation.
3.1. Overview: COSMIC decomposes collaboration into OBJECT-LEVEL, RELATION-LEVEL, and MAP-LEVEL skills to isolate failures in cross-view grounding, relational reasoning, and allocentric integration.The five tasks are ANCHOR RECOGNITION, GLOBAL COUNTING, RELATIVE DISTANCE, RELATIVE DIRECTION, and COGNITIVE MAPPING.
3.1. Overview: ANCHOR RECOGNITION identifies shared objects, GLOBAL COUNTING aggregates instances without double-counting, and relation tasks require cross-view proximity or directional inference.The tasks progressively test common reference objects, instance disambiguation, relative distance, and directions absent from the Answerer’s view.
3.1. Overview: COGNITIVE MAPPING tests whether agents can combine partial observations into an allocentric map by judging candidate top-down layouts as correct or incorrect.All tasks are multiple-choice with one correct answer and three distractors, except COGNITIVE MAPPING’s binary judgment.
3.2. Benchmark Curation: Viewpoints are sampled for controlled partial overlap, sharing anchor objects while retaining perspective-exclusive objects; ambiguous, small, indistinct, or occluded objects are filtered from question generation.Objects receive unique descriptors based on color, size, or neighboring objects, while scenes span more than 23 categories and average 17.71 object instances.

4. Evaluation on the COSMIC Benchmark

The evaluation tests MLLMs in a natural-language, two-agent dialogue protocol with up to 10 rounds, and compares model behavior against human dialogues collected on a 250-question subset. Results are interpreted relative to task-specific chance levels, including 25% for four-choice questions, 50% for binary map tasks, and 30% overall.

Evaluation Protocol: MLLM agents communicate exclusively through natural language for up to 10 rounds, with shared dialogue history but no parameter sharing or hidden-state exchange.Thinking models may perform explicit intermediate reasoning before each dialogue message.
Evaluation Protocol: Both roles use the same underlying MLLM, with role-conditioning prompts specializing Answerer and Helper behavior.The Answerer receives its image, question, answer options, task instruction, and dialogue history; cognitive mapping additionally includes a candidate top-down map.
Human Study: 250 human–human dialogues, comprising 50 questions per task, establish a human performance baseline and enable comparison of model and human communication patterns.The sessions use the same two-agent protocol and interface as COSMIC.
Evaluation Metrics: 25% for 4-choice MCQ, 50% for binary map tasks, and 30% overall define the chance levels used to interpret COSMIC evaluation results.Figure error bars denote 90% confidence intervals computed via bootstrap resampling.

5. Results on COSMIC

MLLMs substantially underperform humans on COSMIC, with accuracy declining from anchor identification to higher-level spatial reasoning and map construction. Communication itself compounds these difficulties: models are verbose, spatially shallow, slow to converge, and rarely repair incorrect reasoning trajectories.

Overall performance: 71.82%: Gemini-3-Pro-Thinking’s average COSMIC-HUMAN accuracy trails the 95.22% human baseline by over 23 percentage points.The result indicates substantial room for improving language-mediated spatial communication.
Capability hierarchy: MLLM performance degrades from Anchors to Maps, with RELATIVE DIRECTION especially difficult at 46.21% for Gemini-3-Pro-Thinking and 51.61% for GPT-5.2-Thinking.RELATIVE DIRECTION requires transforming a partner’s allocentric descriptions into the answering agent’s egocentric frame.
Communication effects: 64.62% to 52.62%: Qwen3-VL-32B-Instruct declines when communicating, while Gemini-3-Flash-Thinking drops from 78.38% to 68.07%.The communication gap shows that coordinating through natural language and maintaining a shared spatial model add difficulty beyond spatial reasoning alone.
Failure analysis: 46.09%: Cross-View Grounding Failures are the dominant overall error category, especially in ANCHOR RECOGNITION at 67.85% and GLOBAL COUNTING at 64%.Perceptual Failures account for the smallest overall share at 19.70%, while unresolved grounding errors can cascade into later geometric and relational failures.
Dialogue behavior: 95.22% accuracy with 199.65 words: humans outperform MLLMs, which average 438.48 words and 50.48% accuracy, while verbosity and accuracy are largely uncorrelated.Human dialogues quickly narrow to targeted spatial updates, whereas MLLMs decline more slowly and repair errors less often: 79.31% for humans versus 28.04% for Gemini-3-Pro-Thinking.

6. Conclusion and Future Work · 1. COSMIC Benchmark Design (Sec. 7)

COSMIC reveals that MLLMs can ground shared anchors but deteriorate on relational reasoning and cognitive mapping, while humans converge through targeted exchanges and models fail to sustain shared spatial understanding. Future progress may require explicit visual-geometric reasoning and structured communication protocols, alongside the benchmark’s documented evaluation, analysis, dialogue, reproducibility, and impact materials.

6. Conclusion and Future Work: MLLMs show partial success at anchor grounding, deteriorate on relational reasoning, and perform near chance on cognitive mapping.This establishes a consistent capability hierarchy across the evaluated frontier models.
6. Conclusion and Future Work: Human pairs achieve rapid convergence through targeted, information-dense exchanges, whereas MLLM agents use verbose dialogue that fails to produce a shared spatial model across turns.The contrast indicates that dialogue length alone does not ensure shared spatial understanding.
6. Conclusion and Future Work: Higher-level spatial tasks may require explicit visual reasoning mechanisms supporting internal geometric verification and mental rotation beyond linguistic chain-of-thought.The proposed direction targets the gap between language-based reasoning and spatially grounded computation.
6. Conclusion and Future Work: Structured protocols such as anchor-first grounding, explicit reference-frame agreement, and sketch-based descriptions could mitigate referential ambiguity and improve cross-view communication.These mechanisms are proposed as ways to make spatial references more precise across viewpoints.
1. COSMIC Benchmark Design (Sec. 7): The benchmark documentation includes an evaluation and reproducibility protocol, failure-mode analysis, and a human data-collection interface.These materials are listed as dedicated sections accompanying COSMIC.
1. COSMIC Benchmark Design (Sec. 7): Additional documentation covers multi-turn dialogue, dialogue repair, model and human conversations, a case study, compute resources, broader impact, and limitations.The listed sections provide coverage of interaction, analysis, resources, societal considerations, and constraints.

7. COSMIC Benchmark Design

COSMIC constructs dual-view indoor environments whose partial observations cover most objects while sharing anchor objects for cross-view alignment. Its templated, paraphrased questions and task-specific distractors test instance-level communication, relational reasoning, and globally consistent spatial mapping.

Environment and views: Each environment places Answerer and Helper at distinct viewpoints whose visible-object union covers most of the environment and intersection supplies shared anchors.The views are denoted I_A and I_H, with visible sets O_A and O_H.
Environment and views: Cameras are fixed 1.5m above the floor with fixed horizontal pitch, independently sample yaw uniformly over [−180◦, 180◦], and use ray casting to determine visibility.Camera poses are resampled until O_A ∩ O_H ≠ ∅ and O_A ∪ O_H ≈ O.
Question generation: Questions begin from predefined task templates and are paraphrased by gpt-4o-mini to increase linguistic diversity and naturalness while preserving semantic content.The paraphrasing stage produces the natural-language variants illustrated in Table 2.
Task construction: Anchor Recognition requires identifying the shared physical object, using attributes and relations because same-category instances can appear across views.Its distractors include an Answerer-exclusive object, a Helper-exclusive object, and a same-category object differing in a discriminative attribute or relation.
Task construction: Cognitive Mapping asks agents to integrate partial observations into a globally consistent top-down map, with distractor maps created by swapping Helper-exclusive object positions.The correct map projects scene positions onto a 2D grid, and answer options are binary Yes or No.

8. Evaluation Protocol and Reproducibility · 9. Failure Mode Analysis

COSMIC’s multi-turn evaluation has higher performance variance, motivating averaged results with bootstrap confidence intervals. Failure analysis identifies perceptual, cross-view grounding, and geometric-relational breakdowns that produce inconsistent spatial models and incorrect answers.

8. Evaluation Protocol and Reproducibility: Evaluation Protocol and Reproducibility: COSMIC’s up-to-10-round dialogues create higher performance variance than typical single-turn evaluations.The final answer depends on the full reasoning trajectory accumulated across turns.
8. Evaluation Protocol and Reproducibility: Evaluation Protocol and Reproducibility: Future work should average results across independent runs and report 90% confidence intervals from 10,000 bootstrap iterations.This recommendation is intended to address variance in multi-turn dialogue evaluation.
9. Failure Mode Analysis: Failure Mode Analysis: Representative Gemini-3-Pro-Thinking failures cover perceptual, cross-view grounding, and geometric-relational error categories.Figure 10 summarizes these categories, with green ticks denoting ground-truth answers.
9. Failure Mode Analysis: Failure Mode Analysis: Attribute mislabelling creates perceptual uncertainty that propagates through the Answerer’s reasoning to an incorrect prediction.In the example, the Helper mislabels the window blinds’ color as possibly orange curtains.
9. Failure Mode Analysis: Failure Mode Analysis: Referential ambiguity arises when descriptions are insufficiently discriminative to identify one object instance, preventing a reliable shared referent.The Answerer cannot establish which object is intended across views.
9. Failure Mode Analysis: Failure Mode Analysis: Instance duplication occurs when agents treat descriptions of the same physical white cabinet as two distinct objects.The agents fail to recognize that the cabinet in each view is the same instance.
9. Failure Mode Analysis: Failure Mode Analysis: Perspective-taking failure produces mutually inconsistent viewpoint models that compound into an erroneous directional judgment.The Helper and Answerer disagree about which wall lies behind the other agent.

10. Human Data Collection Interface · 11. Multi-turn Dialogue

The study collects human multi-turn dialogues through role-specific interfaces and evaluates collaborative agents that combine different views of the same room under constrained turn-taking, termination, and answer-format rules. The dialogue tasks require agents to integrate partial views for shared-object identification, counting, spatial relations, direction inference, or cognitive-map evaluation.

10. Human Data Collection Interface: Both human interfaces pair each participant’s egocentric image with a chat box for multi-turn dialogue.The Answerer interface also displays the question and multiple-choice options.
10. Human Data Collection Interface: The Answerer submits after 10 exchanged messages or can end the dialogue early.The submit button becomes accessible once the dialogue concludes.
11.1. System Prompts for Answerer and Helper: The Answerer and Helper are assigned complementary roles while each receives a different view of the same room.The Answerer must answer the question, whereas the Helper assists through conversation.
11.2. Prompts for Multi-turn Dialogue: Agents alternate messages, cannot send consecutively, and must wait for the partner’s response before continuing.The Answerer and Helper prompts impose a maximum number of messages and require role-specific response formats.
11.2. Prompts for Multi-turn Dialogue: The Answerer may terminate the conversation early by responding with the exact word ‘TERMINATE’.This termination instruction appears in the Answerer’s multi-turn prompt.
11.3. QA Prompt after Multi-turn Dialogue: After dialogue, the Answerer selects one multiple-choice option and formats the response as an XML-style answer tag.Standard questions require one choice from A, B, C, or D.
11.4. Task Description Prompts: The task prompts cover shared-object matching, unique object counting, cross-view distance comparison, egocentric direction inference, and top-down map evaluation.These tasks require combining complementary views rather than relying solely on one participant’s perspective.

12. Dialogue Repair

Dialogue repair is evaluated at the full-conversation level using an automated MLLM judge that determines whether flawed reasoning trajectories are absent, persist unresolved, or are successfully repaired. The evaluation focuses on whether agents recover from cascading reasoning errors before finalizing the answer.

Evaluation Method: The automated Gemini-3-Flash-Thinking judge receives both views, the question, answer options, ground truth, and complete dialogue transcript.It assigns one of three labels for the presence and resolution of flawed reasoning trajectories.
Evaluation Method: Dialogue repair is scored at the conversation level rather than from isolated utterances or expressions of uncertainty.Agents may make and later correct errors, but the evaluation considers overall recovery toward the correct answer.

13. Model and Human Conversations

Figures 17 and 18 present example dialogues from Gemini-3-Pro-Thinking and human participants across all five COSMIC tasks. The examples expose qualitative differences in their communication strategies by pairing dialogue transcripts with views, answer options, predictions, and ground truth.

Dialogue Examples: Examples cover one conversation per task for Gemini-3-Pro-Thinking and human participants across all five COSMIC tasks.The dialogues appear in Fig. 17 for MLLM agents and Fig. 18 for human participants.
Communication Strategies: Together, the examples illustrate qualitative differences in communication strategy between human pairs and MLLM agents.
Dialogue Examples: Each figure shows agents’ egocentric views, multiple-choice options, full multi-turn dialogue, prediction, and ground truth.

14. Case Study

The case study shows that Gemini-3-Pro-Thinking can perform partial cross-view spatial reasoning, but an early object-grounding error propagates through its reasoning and produces an incorrect answer. Positive utterance-level behaviors are ultimately negated because the agents cannot detect or recover from compounding failures.

Perception and grounding: The Answerer recognizes only one black desk and misses a second relevant desk near the window with brown curtains.This is identified as an object recognition failure.
Cross-view reasoning: The agents correctly infer that their viewpoints correspond to opposite walls, demonstrating basic cross-view layout understanding.This indicates partial spatial reasoning despite the initial recognition error.
Cross-view reasoning: The Helper correctly determines that options B and D are farthest from the anchor object, reflecting cross-view distance reasoning.Option D lies outside the Answerer’s view, requiring distance inference across viewpoints.
Error propagation: The final prediction is incorrect because reasoning remains grounded to the wrong desk after the initial object-grounding error.The agents fail to recover from the early perception error, and compounding failures negate their positive utterance-level behaviors.

15. Compute Resources · 16. Broader Impact

Experiments used four NVIDIA A100 GPUs, while COSMIC identifies deployment-relevant bottlenecks in collaborative spatial communication and motivates structured protocols for multi-agent systems. The authors report no anticipated direct negative societal impacts.

15. Compute Resources: Experiments ran on 4 NVIDIA A100 80GB GPUs, with open-source models evaluated using vLLM tensor parallelism across all GPUs.Each open-source model took approximately 2.5 hours to evaluate on the full benchmark; closed-source models were evaluated through their respective APIs.
16. Broader Impact: COSMIC surfaces concrete bottlenecks—including cross-view grounding, perspective transformation, and globally consistent spatial-map construction—that impede reliable real-world deployment.These failure modes have implications for assistive home robotics, multi-agent warehouse coordination, and AR-based remote guidance.
16. Broader Impact: COSMIC’s distributed communication setting more closely mirrors spatial reasoning in human-AI teaming than existing single-agent benchmarks.Shared spatial mental models are foundational for human activities and increasingly important in shared environments such as warehouses, hospitals, and construction sites.
16. Broader Impact: The benchmark informs spatial communication protocols for multi-agent AI systems by contrasting humans’ rapid anchor-first convergence with models’ redundant exploration.The passage suggests that reference-frame agreement and early anchor establishment could meaningfully improve model performance.
16. Broader Impact: Explicitly structured conventions, including reference-frame agreement and early anchor establishment, could guide the design of more effective spatial communication protocols.These conventions are proposed in response to the dialogue-strategy contrast between humans and models.
16. Broader Impact: The authors do not foresee any direct negative societal impacts from this work.This statement concerns the authors’ assessment of direct societal effects.

17. Limitations

COSMIC’s controlled static environments and constrained evaluation simplify benchmarking but omit important aspects of real-world spatial communication. Its multiple-choice, accuracy-based protocol and narrowly sampled human study also limit what the benchmark and baseline can reveal.

Benchmark scope: COSMIC uses static, controlled indoor environments with limited object categories, fixed viewpoints, and no active information gathering.These choices support systematic diagnosis but exclude dynamic scenes, continuous viewpoint changes, and richer object vocabularies.
Evaluation design: Multiple-choice evaluation constrains responses, while binary Cognitive Mapping avoids free-form map generation and evaluation.The passage identifies free-form map generation and evaluation as an open challenge.
Evaluation design: Final-answer accuracy does not assess dialogue quality, making sub-optimal reasoning indistinguishable from perfect reasoning when both produce the correct answer.The evaluation rewards outcomes without directly rewarding the quality of the dialogue itself.
Human baseline: The human study samples university students in a controlled laboratory setting, which may not represent broader demographic groups or levels of spatial expertise.Thus, the human baseline provides a valuable reference but may underrepresent the diversity of spatial communication strategies.