Source-linked AI summary
OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs
Yifei Li, Pengyiang Liu, Yuhang Zang, Zhongyue Shi, Qi Fu, Hongye Hao, Jiwen Lu
TL;DR
Continuous egocentric spatial reasoning remains under-tested because existing benchmarks provide offline context or focus on events rather than spatial structure. OVO-S-Bench introduces a human-annotated, prefix-only benchmark spanning four abstraction levels, and finds substantial model weaknesses, especially in allocentric mapping. Its evaluation is bounded by passive, multiple-choice testing rather than a closed-loop perception-action setting.
Problem
Existing benchmarks do not test spatial-structure reasoning from continuous causal egocentric streams, although embodied agents often need evidence outside the current frame.
Method
OVO-S-Bench uses 1,680 human-annotated questions over 348 videos, exposing models only to the prefix before each query across four spatial-abstraction levels.
Results
Across 38 systems, allocentric mapping is the dominant bottleneck, and Gemini-3.1-Pro scores 59.2 versus 86.6 for human experts.
Takeaways & Limitations
OVO-S-Bench provides a demanding testbed for evaluating whether streaming spatial MLLMs can perceive, remember, and reason about space as it unfolds.
Takeaways & Limitations
The benchmark evaluates passive observation with multiple-choice scoring under one streaming protocol and does not close the perception-action loop.
Abstract
from arXiv · showhide
Multimodal agents in robotics, AR, and autonomous driving must reason about places and layouts from continuous egocentric streams, often using evidence outside the current view. Existing benchmarks either evaluate offline over full videos or target events rather than spatial structure. We introduce OVO-S-Bench, a fully human-annotated benchmark for streaming spatial intelligence, comprising 1,680 questions over 348 source videos. Annotation involves 12 trained annotators, each also serving as a blind cross-reviewer, across roughly 804 person-hours of multi-round quality assurance. Each question carries a query timestamp and an evidence interval, and at evaluation, the model sees only the prefix preceding the query. Questions span four levels of increasing abstraction: instantaneous egocentric perception, spatiotemporal context tracking, spatial simulation and reasoning, and allocentric mapping. Across 38 proprietary and open-source MLLMs, Gemini-3.1-Pro trails human experts by 27 points, 59.2 vs. 86.6, with allocentric mapping as the dominant bottleneck. Notably, streaming and spatially fine-tuned MLLMs underperform their own backbones. We further find that chain-of-thought reasoning amplifies spatial errors when ungrounded in the stream. By exposing these limitations, OVO-S-Bench establishes a demanding testbed for next-generation streaming spatial MLLMs.
1. Introduction
OVO-S-Bench targets the untested problem of reasoning about spatial structure from continuous egocentric streams, where relevant evidence may lie outside the current frame. It evaluates this capability under a prefix-only protocol across four increasing levels of spatial abstraction.
- Existing benchmarks test spatial reasoning offline or streaming event understanding, but not allocentric map construction from causal egocentric streams.
- OVO-S-Bench exposes only the video prefix before each query timestamp and records an evidence interval for questions spanning four spatial-abstraction levels.The levels range from instantaneous egocentric perception to allocentric spatial mapping.
- Every item is human-written and independently cross-reviewed for answerability and evidence sufficiency, with text-only probing used to detect leakage from wording or world knowledge.
- 38 systems are evaluated, revealing allocentric mapping as the dominant bottleneck and a 27-point gap between Gemini-3.1-Pro and human experts.Gemini-3.1-Pro scores 59.2 overall versus 86.6 for human experts.
- Chain-of-thought helps L2 cross-frame integration but harms L1 perception and amplifies hallucinated spatial relations when ungrounded in the stream.The reported mean changes are +3.9 on L2 and −1.0 on L1.
2. Related Work
Prior spatial benchmarks generally provide offline or curated visual context, while streaming benchmarks emphasize events, interaction, or counting. OVO-S-Bench addresses the remaining gap by making persistent spatial evidence in continuous egocentric streams an empirical target.
- Image, multi-image, and video spatial benchmarks evaluate geometry, relations, recall, or motion while assuming offline access to visual context.
- Embodied and egocentric benchmarks study episodic memory, exploration, or disjoint-frame relations, but use curated context windows rather than continuous causal streams.
- Spatial-reasoning MLLMs mainly use data scaling, architectural 3D augmentation, or specialized spatial representations.
- Streaming video benchmarks enforce causal input or bounded memory but target event understanding, interaction, or counting rather than persistent spatial structure.
3. OVO-S-Bench
OVO-S-Bench organizes streaming spatial questions by the spatial state needed at query time, progressing from current-view perception to global allocentric mapping. Its human-authored, quality-controlled construction spans diverse video sources and evidence demands.
- 3.1. Four-Level Streaming Spatial Taxonomy: The four levels progress from evidence directly visible at query time to allocentric map queries requiring cross-viewpoint integration.
- 3.1. Four-Level Streaming Spatial Taxonomy: L1 tests current-view metric, local-relation, and dynamic spatial perception without requiring past observations.
- 3.1. Four-Level Streaming Spatial Taxonomy: L2 requires binding previously observed spatial facts to places, objects, and timestamps after their visual support disappears.
- 3.1. Four-Level Streaming Spatial Taxonomy: L3 tests mental rotation, state-change inference, path feasibility, and spatiotemporal consistency rather than simple observation retrieval.
- 3.1. Four-Level Streaming Spatial Taxonomy: L4 integrates multiple viewpoints into an allocentric representation for direction, topology, and trajectory-map alignment queries.
- 3.2. Benchmark Construction: Human annotators select spatially informative clips, record query timestamps and shortest sufficient evidence intervals, and write plausible distractors under the streaming protocol.
- 3.2. Benchmark Construction: Quality control combines text-only leakage checks, intended-stream probing, and blind second-annotator review of answers and evidence sufficiency.
- 3.2. Benchmark Construction: 1,680 questions cover 348 source videos, 30 canonical task types, nine datasets, and four levels; mean query prefixes last 8.8 minutes.Median evidence spans are 2.0 s for L1, 36.8 s for L2, 2.0 s for L3, and 278.7 s for L4.
4. Experiments
OVO-S-Bench evaluates 38 multimodal systems under a prefix-only streaming protocol, revealing substantial human–model gaps and an allocentric-mapping bottleneck. Specialized variants, frame-selection strategies, and explicit reasoning do not consistently solve persistent spatial-reasoning challenges.
- Evaluation setup: 38 systems are evaluated with only the video prefix before each query timestamp, using 128 uniformly sampled frames for most models.Native streaming architectures instead use their published sequential ingestion rates and compressed states.
- Benchmark results: 59.2 is Gemini-3.1-Pro’s overall accuracy, trailing streaming human experts at 86.6 by 27 points.Random and Text-Only baselines score 31.3 and 37.1, respectively.
- Benchmark results: L4 is the lowest-scoring level for 28 of 34 systems, with an average 9.3% gap from L1–L3 and sharper drops when models must abstract an allocentric map.The six exceptions generally have degraded L1 perception rather than strong allocentric mapping.
- Effect of thinking mode: Thinking mode helps L2 by a mean delta of +3.9 across 8 of 9 pairs but slightly reduces L1 by −1.0 across 6 of 9 pairs.Failure analysis finds non-visual and visual-content errors comprise 60–80% of wrong traces in three of eight thinking-mode models.
- Specialized methods versus backbones: 13 of 15 specialized methods lose overall accuracy relative to their base backbones, with a median delta of −2.0 and L4 showing the most uniform damage.Only HERMES and FluxMem improve overall, while current-view L1 sometimes benefits from perception-oriented specialization.
5. Conclusion
OVO-S-Bench evaluates streaming spatial intelligence under a prefix-only protocol and reveals substantial weaknesses in current multimodal models. Its analyses also show how frame sampling and specialization affect performance, while motivating more interactive evaluation.
- Benchmark and protocol: 1,680 questions over 348 source videos evaluate 38 systems under a prefix-only streaming protocol with annotated evidence intervals.The benchmark includes streaming and oracle frame-sampling policies, with the oracle unavailable at deployment.
- Limitations and future work: The benchmark’s multiple-choice passive-observer setting does not close the perception-action loop and may obscure partial spatial knowledge.The specialization analysis also does not control for domain shift, so controlled fine-tuning is needed to isolate architectural limitations.
- Frame sampling: Frame selection trades evidence concentration against global context: oracle sampling can discard useful framing, whereas log-128 retains 10% of its budget on the far past.This distinction matters especially for L3 counterfactuals and L4 trajectory queries.
A.4. Additional Analysis
Additional analyses show that longer context and larger backbones do not reliably solve the benchmark’s cross-frame and allocentric challenges. Sampling strategy, reasoning mode, and failure patterns expose distinct perception–memory and grounding trade-offs.
- Context Sampling: 0.5–3.3 points: log-128 outperforms oracle-evidence on Overall for five of eight models.Oracle-evidence concentrates all frames within the annotated interval, whereas log-128 preserves 10% of its budget for the far past.
- Context Sampling: −2.6 to +0.3: doubling uniform context from 128 to 256 frames yields negligible or negative Overall deltas across seven models.No model improves by more than 0.3 points, consistent with early L4 saturation across families.
- Perception–Memory Trade-off: 5.4–7.8 points: one-frame query sampling improves L1 over uniform-128, but loses L2 by 15.1–21.1 points across large general backbones.A clean current view benefits egocentric perception, while L2 requires integrating evidence across the prefix.
- Length Effects: ρ=0.06: level-controlled accuracy correlations with prefix length and evidence-span length are small, while full-video duration has ρ=0.03.With source controls, correlations shrink further, and the evidence-span association is slightly positive rather than supporting a longer-is-harder explanation.
- Per-Family Scaling Curves: +11.8/+9.5 to +20.3/+16.2: L1 and L2 scale log-linearly, whereas L3 scales unevenly and L4 plateaus in the 40–46 range across open-source families.InternVL-3.5 L4 rises only +0.8 across a 60× parameter increase and remains roughly 39 points below human-streaming performance.
- Failure Analysis: 75.9% exact agreement and κ=0.65: re-judging wrong-answer traces with Gemini-3.1-Flash-Lite provides substantial cross-judge agreement.The failure taxonomy uses five labels and excludes records without a majority among three judgments.
C.3. Per-Level Failure Distribution
Per-level failure analysis separates process failures from reasoning failures and reveals distinct bottlenecks. L2 is dominated by temporal binding, while L4 divides into family-specific non-visual and directional errors.
- Per-Level Patterns: L2: all eight models exhibit 17–36% temporal-binding error, compared with ≤5.6% on the other levels.The mean L2 temporal-binding error is 28.5%, even though thinking mode provides a +3.9 average L2 gain.
- Per-Level Patterns: L4: GLM and Qwen3-VL show 69–85% non-visual error, whereas InternVL-3.5 shows 27–30% direction error.The family split contrasts reliance on world priors with attempted cardinal-frame reasoning that inverts directions.
- Failure Tiers: Tier 1 failures occur when models fail to conclude or bypass visual evidence, while Tier 2 failures engage frames but mis-ground spatial content.Tier 2 includes visual-content, direction, and temporal-binding errors; examples show traces alongside evidence frames and analyses.
- Failure Taxonomy: The five-class judge taxonomy distinguishes no-conclusion, non-visual, visual-content, direction, and temporal-binding errors.The taxonomy is applied to wrong-answer traces using sampled frames from annotated evidence intervals.
D.1.1. Egocentric Metric Perception
The benchmark’s first level tests instantaneous egocentric perception from cues visible near the query, while later levels require memory or operations over spatial state. Its task definitions distinguish metric, relational, motion, and temporal capabilities by the evidence and operation required.
- Metric and Depth: Absolute-distance tasks estimate a numeric separation, whereas depth-ordering tasks compare which visible object is closer or farther.The two task families use the same current-view setting but differ in metric versus ordering output.
- Spatial Relations: Relative-scale tasks compare physical object size, while containment, support, occlusion, and local-topology tasks classify visible spatial arrangements.These questions require simultaneous visibility of the relevant objects, containers, surfaces, or layouts.
- Motion Perception: Ego-motion translation measures position change, whereas ego-motion rotation measures heading or orientation change.Object-motion separation additionally isolates independently moving entities from camera ego-motion.
- Temporal Scope: Motion-state and speed tasks classify current movement or compare visible speeds, while revisit and chronological tasks require tracking observations across the prefix.Examples explicitly reject questions that require recalling earlier motion, locations, or visits as L1 items.
- Level Boundary: L1 assigns items by query-time visibility: cues visible near the query belong to L1, while cues requiring recall after leaving view belong to L2.This boundary applies across spatial relations, object locations, motion, and camera movement.
E.2. Source Datasets and Licensing
OVO-S-Bench uses public or accessible video sources and human-written annotations under an online-prefix protocol. Its construction emphasizes unique visual answerability, controlled distractors, blind review, and explicit licensing boundaries for raw videos.
- Licensing: Users must obtain the raw videos under the original dataset or platform terms because the benchmark does not redefine their licenses.The benchmark contributes human-written QA annotations rather than new licensing terms for source footage.
- Annotation Protocol: Every item must be uniquely answerable from visual evidence in the prefix before its query timestamp, without world knowledge or option-elimination heuristics.Annotators record the source video, task label, question, options, answer, query timestamp, and evidence interval.
- Quality Control: Distractors share the correct option’s answer type, scale, and axis, while reviewers screen for wording, world-knowledge, plausibility, and lexical shortcuts.A text-only probe and blind review are used to identify and revise leakage or ambiguity.
- Quality Control: κ=0.87: blind review of 150 stratified items indicates almost perfect annotation agreement.The main disagreement concerned the L2/L3 boundary for spatial-change detection and prompted a guideline update.
- Release Format: 1,680 items are released in one JSONL file covering L1–L4, with evidence intervals strictly before their corresponding query times.Revisit Counting can contain multiple query timestamps, with aligned query-time, evidence-time, and answer lists.
F.1. Image, Multi-image, and Top-view Benchmarks
Prior image, multi-image, top-view, and offline-video benchmarks expose substantial spatial weaknesses but do not test the full streaming hierarchy. Their supplied views or offline access leave continuous temporal memory and allocentric reconstruction largely unmeasured.
- Image and multi-image benchmarks: 36% and 49% trail human accuracy of 90% on basic egocentric relations in EmbSpatial-Bench.Its static-image format cannot evaluate L2 memory, L3 simulation, or L4 mapping.
- Top-view benchmarks: >50% human–model gaps widen from recognition to spatial reasoning in TopViewRS.Because the bird’s-eye map is directly provided, models need not reconstruct an allocentric layout.
- Image and multi-image benchmarks: ∼55-point human–model gaps appear on MMSI-Bench’s multi-image spatial reasoning split.Its reported failure modes include grounding, overlap matching, situation transformation, and spatial logic.
- Offline video benchmarks: VSI-Bench finds that linguistic CoT hurts spatial-video QA, with spatial reasoning—not perception or language understanding—as the dominant bottleneck.Its offline full-video protocol does not test causal prefix-only access or L4 allocentric mapping.
- Offline video benchmarks: Existing offline benchmarks cover only parts of the hierarchy: DISJOINT-3DQA targets L1+L2, while MMSI-Video-Bench lacks L4 mapping.MMSI-Video-Bench also reports a ∼58-point human–AI gap and limited gains from 3D cues or CoT.
F.3. Streaming Video Benchmarks
Streaming benchmarks enforce causal or prefix-only input, but most focus on events, narrative context, counting, or narrow spatial tasks rather than a complete spatial hierarchy. OVO-S-Bench combines causal evaluation with four spatial abstraction levels and diverse source domains.
- Prior streaming benchmarks: ∼25 points separate even the strongest model from humans on StreamingBench’s prefix-only protocol.Spatial understanding is only one of its 18 tasks.
- Prior streaming benchmarks: Streaming benchmarks span event, temporal, hallucination, driving, and embodied tasks, but spatial structure remains narrow or domain-specific.OVBench uses event-level subtasks, OVO-Bench has one spatial task, ODV-Bench is driving-only, and StreamEQA uses one kitchen domain.
- Prior streaming benchmarks: OST-Bench is closest in spirit but covers only L1+L2 agent-state and agent–object relations in indoor scenes.It lacks L3 simulation, L4 mapping, and outdoor, driving, and 3D-rendered environments.
- Synthesis: Three gaps recur: spatial benchmarks lack prefix-only evaluation, streaming benchmarks underrepresent spatial structure, and prior video benchmarks omit systematic four-level coverage.L4 allocentric mapping is absent from every prior video benchmark.
- OVO-S-Bench: OVO-S-Bench enforces prefix-only access per item, spans indoor, outdoor, driving, egocentric, and 3D-rendered sources, and stratifies tasks across L1–L4.Its L4 supervision includes named entities, bird’s-eye views, and topology.
F.5. Text-Only Shortcut Analysis
The text-only shortcut analysis measures how much OVO-S-Bench can be answered from question wording and options alone. Its residual advantage is modest and falls within the range observed on similarly filtered spatial benchmarks.
- Text-only probe: +5.8 pp is OVO-S-Bench’s text-only-minus-random delta, near the median across 12 surveyed spatial benchmarks.The probe uses question and options only, without visual input, on a 20% stratified sample.
- Text-only probe: Text-only deltas range from +42 to −2 percentage points across the surveyed benchmarks.OVO-S-Bench applies text-only probing and blind cross-review to suppress language shortcuts.
F.6. Source-Robustness Analysis
The main ranking is robust to RoomTour3D’s large contribution, although per-source accuracy varies widely across domains. Removing that source preserves the top-two ordering across the reported systems.
- Ranking stability: 48% of questions come from RoomTour3D, yet removing them preserves Gemini-3.1-Pro and Qwen3-VL-235B-A22B in the top two positions.Across all 38 systems, the ranking correlation is ρ=0.92 with p<10^-4.
- Source variation: Per-source accuracy varies widely, indicating cross-domain difficulty differences beyond RoomTour3D concentration.The per-source comparison is shown in Figure 12 for the top-8 reported models.
G. Evaluation Methods Details
The evaluation fixes model-specific inference recipes while standardizing prompts, frame inputs, and deterministic scoring where applicable. Streaming architectures instead use their published ingestion and compression protocols, and future evaluation directions include richer answer formats and embodied tasks.
- Evaluation configuration: Published model recipes determine resolution, frame count, decoding, reasoning toggles, and streaming settings without tuning on OVO-S.The appendix states that per-model choices follow authors’ recipes, while streaming ingest and compression values come from released sources.
- Decoding parameters: Non-reasoning models use greedy decoding at T=0, no nucleus filtering, and a 1,024-token output cap.The cap accommodates short justifications before the required answer letter.
- Decoding parameters: Reasoning variants follow each model author’s published recipe, including model-specific temperature, top-p, prompt, and thinking-token limits.InternVL-3.5-thinking uniquely uses an explicit chain-of-thought prompt, while other thinking rows rely on model-side flags.
- Visual input and ingestion: The default visual input uses 128 uniformly sampled frames from the prefix, whereas streaming architectures consume frames sequentially at their published rates.Native streaming models generate answers from their resulting compressed state rather than the uniform 128-frame default.
- Prompting and scoring: All frame-sampling policies retain the same multiple-choice prompt, and evaluation uses deterministic multiple-choice scoring instead of LLM-as-judge.This keeps textual inputs fixed while varying only the visual input and avoids additional spatial-grounding errors from open-ended judging.
- Answer extraction: Unmatched model responses are scored as wrong and remain in every per-task and overall accuracy denominator.The extractor applies tail-first matching patterns to de-thinked responses, with one full-response rule requiring an answer-related trigger and separator.