PaperlayerView exact revision on arXiv

Source-linked AI summary

Humans in Space: Can MLLMs Construct Human-Centric Spatial Understanding via Active Exocentric Exploration?

Tianchen Guo, Tengyue Zhang, Chen Liu, Ling Chen, Xin Yu

arXiv:2610.34149v1cs.CVcs.CL

TL;DR

MLLM spatial reasoning is mainly evaluated on static images, despite occlusion hiding important geometry and leaving visual self-awareness largely untested. SATE-Bench evaluates interactive exocentric exploration with paired Active and Passive settings. Models often stop early under occlusion, and increasing view-seeking through GRPO can reduce accuracy.

  • Problem

    Spatial evaluations mainly use predetermined images, leaving visual self-awareness—the ability to recognize observational blind spots—largely unexamined in occluded scenes.

    Source: §1 · p. 2

  • Method

    SATE-Bench pairs occluded-view Active exploration with comprehensive-view Passive observation to evaluate evidence seeking and stopping across real-world human-centric scenes.

    Source: §1 · p. 2

  • Results

    MLLMs show an Active-setting accuracy drop of 5–26%, and GRPO raises Qwen2.5-VL-3B’s active exploration rate from 0.2% to 72.8% while reducing accuracy from 30.4% to 17.9%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that scaling passive perception does not yield active spatial reasoning, and that active view-seeking can coexist with over-exploration and multi-view fusion bottlenecks.

    Source: §6 · p. 8

    The interface is limited to discrete left/right camera shifts over synchronized static frames, and iterative querying costs more computation than single-pass evaluation.

    Source: §I · p. 21

Abstract

from arXiv · show

While Multimodal Large Language Models (MLLMs) excel at zero-shot reasoning, their spatial capabilities are primarily evaluated through static, passive observation. In complex, human-centric scenes, occlusion often hides crucial geometric evidence, and passive benchmarks force models to guess rather than recognize their own visual uncertainty. To bridge this gap, we introduce SATE-Bench, a novel interactive benchmark comprising 316 real-world scenes and 2,525 multi-turn QA pairs to evaluate visual self-awareness and active exocentric view-exploration. It combines a Passive multi-view control with an Active exploration setting, where models start from a severely occluded view and must iteratively query adjacent viewpoints until they gather enough evidence to fully ``sate'' the question, scored by a step-penalized efficiency reward. Evaluating state-of-the-art MLLMs reveals that they terminate exploration prematurely while reporting high confidence, and struggle with fine-grained 3D geometric tasks; matched-view controls further attribute the Active--Passive gap primarily to insufficient evidence acquisition. Moreover, fine-tuning with Group Relative Policy Optimization (GRPO) shows that rewarding view-seeking increases active exploration but lowers overall accuracy, as models over-explore and fail to fuse expanding multi-view contexts across turns. These findings indicate that scaling passive multi-view perception does not intrinsically yield active spatial reasoning, highlighting the need for models that know where to look, how to integrate what they see, and when to stop. Code and data are available at https://gtc-gh.github.io/SATE-Bench/

1. Introduction

SATE-Bench addresses the limited evaluation of MLLM spatial reasoning under passive observation, where occlusion can hide geometric evidence and models may rely on linguistic priors. It introduces an Active/Passive benchmark and finds that models often stop too early, while GRPO-driven view-seeking can come at an accuracy cost.

1. Introduction

  • Passive spatial evaluations supply predetermined images, leaving visual self-awareness—the recognition of observational blind spots—largely unexamined despite occlusion hiding geometric cues.In these settings, models may rely on linguistic priors and hallucinate missing details rather than seek new information.Source: §1 · p. 2
  • SATE-Bench comprises 316 real-world scenes sampled from 10 datasets and 2,525 high-quality QA pairs, averaging 6.04 camera views per scene.Its focus is real-world, human-centric exocentric exploration, underexamined by interactive benchmarks that predominantly use synthetic environments and egocentric trajectory planning.Source: §1 · p. 2
  • Active evaluation starts from one severely occluded view and lets models query adjacent exocentric viewpoints, while Passive evaluation supplies comprehensive static views upfront.The paired settings test when a model seeks more evidence and when it stops; an exponentially decaying reward penalizes redundant queries.Source: §1 · p. 2
  • All evaluated MLLMs lose 5–26% in Active settings, with over 70% of Active errors for most models occurring at the initial view without a move.For Qwen2.5-VL-3B, GRPO raises active exploration from 0.2% to 72.8% but lowers accuracy from 30.4% to 17.9%.Source: §1 · p. 2
  • The benchmark combines an Active/Passive diagnostic protocol and step-penalized efficiency reward with MLLM evaluation and GRPO-based active exploration.The authors report that matched-view controls attribute the Active–Passive gap to insufficient evidence acquisition, alongside a trade-off between view-seeking and reasoning accuracy.Source: §1 · p. 2

2. Related Work

Prior work examines static multi-image reasoning, active perception, and multi-view human-centric datasets, but leaves real-world exocentric exploration in dense human scenes largely untested. SATE-Bench repurposes static captures as interactive exploration environments.

2.1. Multi-Image MLLM Benchmarks

  • Static multi-image benchmarks provide preconfigured viewpoints, while interactive ActiView permits 2D cropping but lacks real-world 3D spatial complexity; SATE-Bench instead requires active exocentric exploration in 360-degree scenes.The static multi-view benchmarks AllAnglesBench and CVBench average three and four views per scene, respectively.Source: §2.1 · pp. 2–3

2.2. Active Visual Perception

  • Existing active-perception benchmarks focus mainly on egocentric navigation in synthetic simulations or general proactive querying, unlike SATE-Bench’s real-world exocentric exploration under severe mutual occlusion.SATE-Bench evaluates recognition of observational blind spots and cross-view geometry through a step-penalized Active/Passive framework.Source: §2.2 · p. 3

2.3. Multi-View Human-Centric Datasets

  • SATE-Bench turns existing dense multi-view human-centric datasets, previously used for passive spatial tasks, into interactive arenas where MLLMs step through discrete camera views to uncover occluded information.The datasets capture dense activities, fine-grained interactions, and occluded workflows.Source: §2.3 · pp. 3–4

3. SATE-Bench

SATE-Bench places MLLMs in an interactive 360-degree environment to test whether they recognize visual uncertainty, seek missing evidence, and know when to answer. Its ten tasks, annotated multi-view data, quality checks, and step-penalized reward operationalize that evaluation.

3.1. Benchmark Overview

  • The benchmark evaluates whether an MLLM recognizes uncertainty in an occluded view, queries new exocentric perspectives, and knows when it has enough evidence to answer.It pairs Active Exploration with Passive Observation and uses controlled exocentric view selection rather than unconstrained 3D navigation.Source: §3.1 · p. 4

3.2.1. Task Definitions

  • Ten human-centric tasks span five cognitive dimensions, covering entity counting and existence, pose and interaction, relative proximity, and view-conditioned spatial geometry.The tasks probe capabilities from cross-view re-identification and blind-spot recognition to geometric alignment and coordinate-frame reasoning.Source: §3.2.1 · p. 4

3.2.2. Data Collection

  • The benchmark sources multi-view sequences from ten real-world human-centric datasets selected for dense physical overlap, multi-person interactions, and severe occlusion.Table 2 reports dataset statistics and view distributions by task; “Avail. Views” and “Passive Views” are average available and selected passive views.Source: §3.2.2 · p. 4, Table 2 · p. 4

3.2.3. Question-Answer Annotations

  • Annotators create challenging multiple-choice QA pairs from synchronized frames and select occluded initial views for Active tasks and fully answerable view subsets for Passive tasks.Active and Passive views are randomly sampled from valid camera configurations to limit viewpoint or human-preference bias.Source: §3.2.3 · p. 4

3.2.4. Quality Control

  • Two independent reviewers require at least 85% agreement on the correct answer and initial Active view, with a third expert resolving disputes.Reviewers remove ambiguous or text-answerable items; 9.2% of candidates are discarded and 38.7% revised for clarity.Source: §3.2.4 · p. 4

3.3. Benchmark Statistics

  • The finalized benchmark contains 2,525 multi-turn QA pairs across ten human-centric tasks, with text-length and answer-distribution statistics reported in Table 3.The benchmark’s detailed license, release plan, and additional quantitative results are provided in Appendix A.Source: Table 3 · p. 4, §3.3 · p. 5

3.4. Evaluation Metrics

  • The exploration reward averages correctness-weighted scores that decay exponentially with the number of steps, rewarding correct answers reached through efficient exploration over random, exhaustive querying.Accuracy is exact-match; the reward uses an indicator for correct final answers and a predefined penalty coefficient λ.Source: §3.4 · pp. 5–6

4. Experiments and Main Results

Across ten tasks, MLLMs perform worse when they must actively gather views than when static views are provided, often stopping while uncertainty remains. Broader viewpoint choices and video inputs do not resolve the central difficulties, which include fine-grained geometry and multi-view context handling.

4.1. Experimental Setup

  • The evaluation reports passive exact-match accuracy and inference time, and Active accuracy, steps, reward, active rate, and inference time.The nine baselines include reference points and six Gemini-3.1-flash-lite settings; interaction histories are visible and generation temperature is 0.0.Source: §4.1 · p. 6

4.2. Main Results and Discussion

  • Gemini-3.5-flash accuracy falls from 69.2% in Passive to 56.8% in Active; models also stop prematurely, with over 70% of most models’ Active errors occurring at the initial view.Human evaluators shift viewpoints in 88.1% of cases, taking 4.7 steps to reach 82.7% accuracy; Gemini-3.5-flash averages 2.5 steps.Source: §4.2 · p. 6

4.3. Single View Evaluation

  • With only the initial occluded view, GPT-5-mini scores 39.6%, slightly above GPT-5.2 at 37.3%, while model scaling does not overcome the missing physical evidence.Table 5 evaluates models passively on the Active set’s initial view, without spatial exploration.Source: Table 5 · p. 6, §4.3 · p. 7

4.4. Evaluation on Expanded View Selection

  • Gemini-2.5-flash accuracy changes only from 45.4% to 45.7% with expanded ±1 or ±2 view choices, while average reward rises from 35.3 to 38.0.The expanded setting retains eight total views and permits broader choices at each step.Source: §4.4 · p. 7, Table 6 · p. 7

4.5. Matched-View Evaluation

  • Active and matched-view Passive achieve similar accuracy when given identical views, locating the performance gap primarily in insufficient evidence acquisition.Table 7 compares Active with matched-view Passive, which supplies the views in the corresponding Active trajectory’s order.Source: §4.5 · p. 7, Table 7 · p. 7

4.6. Long-Context Video Modeling Evaluation

  • Video inputs lower accuracy versus multi-image inputs: Gemini-3.1-flash-lite drops from 62.6% to 57.6%, and GPT-5-mini from 64.6% to 63.7%.The authors attribute the pattern to video encoding’s weaker integration of discontinuous cross-camera views.Source: §4.6 · p. 7

4.7. Direct Measurement of Visual Self-Awareness

  • Models average about 1.4 exploration steps before stopping, yet report high terminal confidence despite low overall accuracy.The authors interpret the confidence results as evidence that premature stopping reflects false certainty rather than arbitrary policy limits.Source: §4.7 · p. 7, Table 8 · p. 8

4.8. Error Analysis

  • Closed-source errors include multi-view fusion bottlenecks and hallucinations, while open-source models often lack proactivity or guess despite occlusion.Reasoning-focused open-source variants may exhaust token limits without taking spatial actions; Figure 3 reports that increasing GRPO action reward raises exploration but lowers active accuracy and reward, while passive accuracy remains stable.Source: §4.8 · pp. 7–8, Figure 3 · p. 8

5. Can MLLMs Learn Active Exploration via Reinforcement Learning?

The authors fine-tune Qwen2.5-VL-3B-Instruct with GRPO to encourage active view-seeking. Tuning increases exploration, but stronger action rewards bring over-exploration and lower active accuracy and reward.

Training Setup

  • GRPO fine-tunes Qwen2.5-VL-3B-Instruct on 19.3K synthesized samples using a single-turn choice between answering and taking a movement action.The balanced training mixture combines active and passive tasks; the reward includes accuracy, format, and action components.Source: §5 · p. 8

Results

  • For base Qwen2.5-VL-3B in the Active setting, tuning with action reward weight 1.2 raises active exploration from 0.2% to 72.8% and average steps per query from 1.0 to 6.0.In the Passive setting, base and GRPO-tuned models both maintain approximately 51.0% accuracy; higher action rewards increase exploration but lower overall accuracy and final rewards.Source: §5 · p. 8

6. Conclusion

The conclusion presents SATE-Bench as an evaluation of visual self-awareness and active view-exploration. It reports premature stopping under occlusion and lower accuracy when GRPO encourages over-exploration amid multi-view fusion bottlenecks.

Conclusion

  • SATE-Bench evaluates MLLM visual self-awareness and active view-exploration.The conclusion identifies premature stopping under occlusion as a recurring finding.Source: §6 · p. 8
  • GRPO incentivizes active view-seeking, but over-exploration and multi-view fusion bottlenecks lower accuracy.Source: §6 · p. 8
  • The authors conclude that scaling passive perception does not intrinsically yield active spatial reasoning.Source: §6 · p. 8

Supplementary Material

The supplementary material points readers to qualitative examples in Figure 4.

A.1. More Qualitative Examples

  • Figure 4 provides additional qualitative examples.Source: §A.1 · p. 14

A.2. Details of Source Datasets

The appendix details SATE-Bench’s human-centered source data, task dimensions, evaluation controls, and failure analyses. Across the evaluated models, active errors often reflect stopping before decisive evidence is seen, while passive errors include counting, existence, and perspective mistakes.

A.2. Details of Source Datasets

  • SATE-Bench draws on ten multi-view human-centered datasets covering varied scene structures, interactions, and camera configurations, including dense collaboration and severe occlusion.Core4D captures shared-object manipulation; M3GYM captures high-intensity gym activity; Ego-Exo4D pairs egocentric and exocentric views.Source: §A.2 · p. 14

A.3. Detailed Task Definitions

  • The benchmark defines ten tasks across five dimensions: counting, existence, pose and interaction, global relative proximity, and view-conditioned geometry.Tasks test cross-view entity tracking, target search, pose and contact reasoning, distance estimation, and perspective-dependent spatial relations.Source: §A.3 · pp. 15–16

A.4. Pre-processing

  • Scenes use synchronized, calibrated views arranged in clockwise order, while five-option questions include an abstention choice and distractors based on misleading visual artifacts or semantic and geometric proximity.The view arrangement supports standardized left/right exploration; Active and Passive pairs share questions and answers but differ in visual initialization.Source: §A.4 · p. 16, §A.5 · p. 16, §A.6 · p. 16

A.7. Data Leakage Analysis

  • Evaluation imagery comes from official test splits or manually held-out subsets, and benchmark questions and spatial annotations are newly authored rather than taken from prior training corpora.GRPO training data is isolated from evaluation scenes and uses newly synthesized QA pairs.Source: §A.7 · p. 16

A.8. Human Baseline Evaluation

  • Five postgraduate evaluators completed tasks without time limits under the same step-wise exploration constraints, prompts, and scene context as the MLLMs.A guided 20-question trial preceded the formal assessment.Source: §A.8 · p. 16

B.1. Active Failure Modes

  • For seven of ten models, 71–100% of Active errors occurred at the start view; other errors included stopping before the decisive view and difficulty combining views.LLaVA-OneVision-2-8B never moved; 13.5% of Gemini-3.5-flash’s errors occurred after it had seen every view.Source: §B.1 · pp. 17–18

B.2. Passive Failure Modes

  • In Passive evaluation, 66% of numeric counting errors were under-counts, 76% of existence errors were false positives, and many direction errors flipped left and right.Physical-contact questions also often elicited reports of contact that did not exist.Source: §B.2 · p. 18

C.1. Details of the baselines

  • Baselines include human and oracle conditions, fixed-direction and random exploration, exhaustive search, and text-only input; their comparisons identify premature stopping and ineffective viewpoint selection as major Active bottlenecks.The text-only baseline measures exploitable linguistic priors, while oracle views and trajectories separate spatial reasoning from exploration decisions.Source: §C.1 · p. 18

C.4. Full table for Long-Context Video Modeling Evaluation

These appendices describe supplemental evaluation controls and exploration settings. Removing previously seen images reduces accuracy, confidence intervals qualify a small measured shift, and inference costs constrain which models could be evaluated.

C.6. Current View Only Evaluation

  • Removing previously observed images at each turn degrades overall accuracy, indicating that visual history matters in the Current-View-Only evaluation.The question, answer options, and textual interaction history remain available in this condition.Source: §C.6 · p. 19

C.7. Confidence Interval Analysis

  • The 0.3% accuracy shift in expanded view selection falls within the 95% confidence interval and is treated as statistical noise.The analysis covers overall accuracy across Active and Passive settings, including evaluated open-source models.Source: §C.7 · p. 19

C.8. Extended Open-Source Model Evaluation

  • Explicit-reasoning models can require more than 20 hours for multi-turn Active inference, and larger open-source variants were excluded when exploration took several days.Reported hardware included two RTX 3090 GPUs for standard models and one H100 for larger models.Source: §C.8 · p. 19

C.9. Exploration Details

  • Exploration uses closed-loop camera layouts, caps steps at the number of scene views, permits revisiting views, and retains previously queried images by default.Scenes typically provide eight views; the appendix also identifies a comparison of multi-image and video modalities without reporting its outcomes here.Source: §C.9 · pp. 19–20, Table 13 · p. 20

D. Abstention Option Analysis

The appendix examines abstention behavior, the reward step penalty, and the optimization background for GRPO. Abstention is rare in the ground truth, while the selected penalty balances reward scale and trajectory length.

D. Abstention Option Analysis

  • The Abstention Option is correct for 2.48% of Active samples and 1.84% of Passive samples; weaker models over-select it, while stronger models tend to guess when evidence is insufficient.In Passive evaluation, abstention selections decrease as visual ambiguity is reduced.Source: D. Abstention Option Analysis · p. 20

E. Ablation Study of Average Reward Coefficient

  • With λ = 0.5, the reward penalty overemphasizes short trajectories and changes model rankings; λ = 0.1 is retained as a moderate trade-off.The selected value applies an approximately 9.5% penalty per additional exploration step; conclusions remain stable for moderate penalties from 0.05 to 0.2.Source: E. Ablation Study of Average Reward… · p. 20

F.1. Preliminaries

  • GRPO removes the critic model and compares groups of candidate outputs using rewards that can assess answer accuracy and format compliance against ground truth.The appendix describes a token-level objective with a KL penalty scaled by λ and advantages normalized across sampled sequences.Source: §F.1 · p. 20

F.2. Training Data Details

The remaining appendices document GRPO training data and resources, prompt materials, benchmark limitations, artifact use, and safety practices. They describe a narrowly scoped research artifact built from existing datasets and note practical limits in exploration and computational cost.

F.2. Training Data Details

  • GRPO training uses 19,370 QA pairs sampled from 86 Ego-Exo4D and M3GYM scenes, split randomly into 90% training and 10% validation data.Source: §F.2 · p. 21

F.3. Training Hyperparameters

G. Computational Resources Analysis

  • GRPO runs use one node with four H100 GPUs and typically finish within hours; Active inference raises API costs above $150 for the largest proprietary models.Smaller open-source models use two RTX 3090 GPUs, while larger models use one H100; smaller proprietary models and standard Passive evaluation generally cost under $50.Source: G. Computational Resources Analysis · p. 21

H. Prompt Details

  • The appendix supplies prompt templates for Passive, Active, expanded-view, confidence-score, and exhaustive-search evaluations.Source: H. Prompt Details · p. 21

I. Limitations

  • The interface is limited to discrete left/right shifts on a camera ring, uses synchronized static frames rather than active temporal exploration, and incurs high multi-turn computational costs.The authors identify memory-efficient caching as a need for processing expanding visual contexts.Source: §I · p. 21

J. Consistency with Intended Artifact Use

  • SATE-Bench is intended for non-commercial academic research, with release constraints following the most restrictive source-dataset licensing terms.Raw imagery requiring signed agreements is withheld, with metadata extraction scripts provided instead.Source: J. Consistency with Intended Artifact… · pp. 22–23

K. Personally Identifying Information and Offensive Content

M. AI Usage

  • Authors report using language models for editing, parsing irregular outputs, and initial distractor generation, with generated material reviewed and validated by humans.Source: M. AI Usage · pp. 23–24

N. Answer Extraction and Fallback LLM Usage

Evaluated proprietary MLLMs followed answer-format instructions in both Active and Passive settings without fallback answer extraction.

N. Answer Extraction and Fallback LLM Usage

  • 0.00% fallback usage across Active and Passive settings shows that all evaluated proprietary MLLMs adhered to the required answer format.Table 23 reports fallback usage for answer extraction.Source: N. Answer Extraction and Fallback LLM… · p. 24, Table 23 · p. 24
  • A dataset prior led Gemini-3.5-flash to stop at the initial view, although one move would reveal a second person visible from V3 and V4.The case illustrates a premature decision without visual verification.Source: N. Answer Extraction and Fallback LLM… · p. 25
  • From V4, Gemini-3.5-flash judged an occluded region empty, while views V1 and V3 revealed the red fuel container.The model treated absence from its current view as evidence of absence.Source: N. Answer Extraction and Fallback LLM… · p. 25
  • Partial exploration accounts for 65% of Gemini-3.5-flash’s and 55% of GPT-5.2’s errors in the equipment-case example.The case was visible in V1, but appeared with the glass-door landmark only in V3, one step in the opposite direction.Source: N. Answer Extraction and Fallback LLM… · p. 26
  • In a passive interaction example, several models asserted foot-ball contact despite close-ups showing the player’s feet on the turf beside the ball.The analysis describes a stereotyped pose being treated as confirmed across views, including distant views where the foot covered few pixels.Source: N. Answer Extraction and Fallback LLM… · p. 27
  • In a passive existence question, Gemini-3.5-flash accepted a partial match between two people’s attributes, a typical form of 76% of false-positive existence errors.The scene included one person with a black cap and dark navy sweatshirt and another wearing grey without a cap.Source: N. Answer Extraction and Fallback LLM… · p. 28

Related papers

Loading 2610.34149v1…