PaperlayerView exact revision on arXiv

Source-linked AI summary

OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding

Jingli Lin, Chenming Zhu, Runsen Xu, Xiaohan Mao, Xihui Liu, Tai Wang, Jiangmiao Pang

arXiv:2507.07984v2cs.CV

TL;DR

Most MLLM benchmarks use fixed, pre-recorded inputs, leaving online embodied scene understanding under-evaluated. OST-Bench tests agents’ incremental observations and spatial reasoning, and leading models lag human performance by over 30%.

  • Problem

    Most existing benchmarks evaluate MLLMs on fixed, pre-recorded inputs rather than the online observations of embodied tasks.

    Source: §1 · p. 2

  • Method

    OST-Bench evaluates online spatio-temporal reasoning through agent exploration, with 1.4k scenes and 10k question-answer pairs from three real-world datasets.

    Source: §1 · p. 2

  • Results

    Leading MLLMs lag human performance by over 30%, and their accuracy declines as exploration steps and memory grow, with weaknesses in complex spatial reasoning and long-term retrieval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Complex clue-based spatial reasoning and long-term memory retrieval are distinct challenges for online embodied reasoning, while fine-tuning alone yields limited improvement.

    Source: §1 · p. 2

    The benchmark assumes object positions and states remain unchanged during exploration.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Recent advances in multimodal large language models (MLLMs) have shown remarkable capabilities in integrating vision and language for complex reasoning. While most existing benchmarks evaluate models under offline settings with a fixed set of pre-recorded inputs, we introduce OST-Bench, a benchmark designed to evaluate Online Spatio-Temporal understanding from the perspective of an agent actively exploring a scene. The Online aspect emphasizes the need to process and reason over incrementally acquired observations, while the Spatio-Temporal component requires integrating current visual inputs with historical memory to support dynamic spatial reasoning. OST-Bench better reflects the challenges of real-world embodied perception. Built on an efficient data collection pipeline, OST-Bench consists of 1.4k scenes and 10k question-answer pairs collected from ScanNet, Matterport3D, and ARKitScenes. We evaluate several leading MLLMs on OST-Bench and observe that they fall short on tasks requiring complex spatio-temporal reasoning. Under the online setting, their accuracy declines as the exploration horizon extends and the memory grows. Through further experimental analysis, we identify common error patterns across models and find that both complex clue-based spatial reasoning demands and long-term memory retrieval requirements significantly drop model performance along two separate axes, highlighting the core challenges that must be addressed to improve online embodied reasoning. To foster further research and development in the field, our codes, dataset, and benchmark are available. Our project page is: https://rbler1234.github.io/OSTBench.github.io/

1 Introduction

Most scene-understanding benchmarks test models on fixed, pre-recorded inputs, unlike embodied agents that must update their understanding as they explore. OST-Bench evaluates this online spatio-temporal reasoning and finds persistent model weaknesses.

1 Introduction

  • Embodied agents need to track their changing position and relationships to objects so they can respond to commands and act correctly.The introduction illustrates this with an agent locating a brown pillow relative to its current viewpoint.Source: §1 · p. 2
  • Existing benchmarks commonly evaluate MLLMs on fixed, pre-recorded inputs, which do not capture the online nature of embodied tasks..Source: §1 · p. 2
  • OST-Bench evaluates online spatio-temporal understanding as an agent actively explores, integrating incrementally received observations with historical memory.It contains 1.4k real-world scenes and 10k question-answer pairs from ScanNet, Matterport3D, and ARKitScenes, organized around agent state, visible information, and agent-object spatial relationships.Source: §1 · p. 2
  • Models lag human performance by over 30%, and accuracy declines as exploration steps and memory grow; complex spatial reasoning and long-term retrieval are distinct weaknesses.The analysis also identifies a shortcut pattern: models may avoid retrieving key information and rely on shallow, unsupported inferences; fine-tuning alone yields limited improvement.Source: §1 · p. 2

2 Related Work

Prior spatial benchmarks cover semantic and spatial reasoning, while temporal video benchmarks add egocentric or online settings. OST-Bench emphasizes online, agent-centric evaluation that combines temporal understanding with complex 3D spatial reasoning.

Spatial Reasoning Benchmarks

  • Early scene-understanding benchmarks introduced diverse task taxonomies and treated spatial understanding as a fundamental component.Source: §2 · p. 2
  • Earlier spatial benchmarks often treat spatial relations as semantic attributes, while newer ones assess 3D reasoning from images or video but remain predominantly offline.OST-Bench instead evaluates online, agent-centric spatial reasoning; the comparison with other datasets highlights its online and spatio-temporal focus.Source: §2 · p. 3, Table 1 · p. 3

Video Benchmarks for Temporal Understanding

  • Temporal video benchmarks have added egocentric views, online inference, and spatial tasks, but those tasks often remain limited to 2D relations or short-term motion cues.OST-Bench emphasizes complex 3D spatial reasoning from multi-view observations.Source: §2 · p. 3

3 OST-Bench

OST-Bench is built through task formulation, scene processing, and question generation. Its design specifies a dynamic agent in a static environment, operationalizes visibility, and generates temporally grounded questions along exploration trajectories.

3 OST-Bench

  • OST-Bench’s methodology comprises task formulation, a data collection and processing pipeline, and benchmark sample generation.Source: §3 · p. 3

3.1 Task Formulation

  • The benchmark assumes object positions and states remain unchanged during exploration, while spatial references are relative to an anchor rather than an absolute scene coordinate system.Its evaluation covers agent state, agent-visible information, and changing 3D relationships between the agent and objects.Source: §3.1 · pp. 3–4

3.2 Meta-dataset Collection and Processing

  • The dataset selects 1.4k scenes from ScanNet, ARKitScenes, and Matterport3D, using existing object and region annotations.Because Matterport3D lacks continuous exploration paths, the pipeline generates trajectories over camera viewpoints using a minimum-spanning tree and an image-overlap threshold.Source: §3.2 · p. 4
  • Attribute visibility means an object can be identified from a frame; spatial visibility additionally requires its 3D information to be inferable.The pipeline determines visibility using projected point-cloud area and visibility of 9-DoF bounding-box vertices.Source: §3.2 · p. 4

3.3 Benchmark Samples Generation

  • Questions are generated in multi-round dialogue: each round adds temporally ordered observations before asking about the agent’s current situation.Rule-based templates produce samples across the three main categories; the benchmark contains 15 fine-grained subtypes and four formats: Judgment, Counting, Temporal Localization, and Estimation.Source: §3.3 · pp. 4–5, Figure 2 · p. 5
  • Manual review marks questions with insufficient information or incorrect answers invalid, and reported human evaluation found an error rate below 5%.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate MLLMs on OST-Bench using online multi-turn inputs and find substantial gaps from human performance, especially on complex spatio-temporal reasoning. Accuracy also declines with longer exploration, and analyses identify reasoning errors, cross-view difficulty, and limits of fine-tuning.

4.1 Benchmark Models & Evaluation Metrics

  • Models were evaluated zero-shot in multi-turn dialogue against human and chance baselines, using format-specific exact-match rules or Mean Relative Accuracy.Tables 2 and 3 report fine-grained subtype results and performance across main categories and question formats.Source: §4.1 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6

4.2 Main Results

  • Even the strongest MLLMs lag human performance by nearly 30% overall; models score near 70% on Agent Visible Info but near chance on Agent State and Agent–Object Spatial Relationships.The reported results are summarized across model categories and question formats in Table 3.Source: Table 3 · p. 6, §4.2 · p. 7
  • Accuracy declines as exploration continues; Agent State and Agent–Object Spatial Relationship performance typically falls to near chance within 2–4 turns and stays low.Agent Visible Info accuracy instead declines gradually over turns.Source: §4.2 · p. 7, Figure 3 · p. 7
  • Proprietary models outperform open-source models, while scaling open-source models and enabling Gemini-2.0-Flash thinking mode improve performance in the reported settings.The thinking-mode gains are especially noted for Temporal Localization and Agent–Object Spatial Relationship questions.Source: §4.2 · p. 7

4.3.1 Insights from Model Explanations

  • Reasoning errors account for an estimated over 60% of failures; models often avoid retrieving useful long-term spatial anchors and make unsupported inferences.The manual analysis examined 270 errors across three task categories and three representative models; the named shortcut includes inferring a TV is behind the agent merely because it is no longer visible.Source: §4.3.1 · p. 8

4.3.2 Cross-View Analysis

  • Cross-view tests vary single- versus multi-step spatial links and keyframe- versus sequence-based context; all models drop substantially with either added reasoning complexity or long-memory retrieval.In the most challenging multi-step, sequence-based setting, all models fall to around 10% accuracy.Source: §4.3.2 · p. 9, Figure 7 · p. 9

4.3.3 Fine-tuning Analysis

  • Fine-tuning improves all evaluated models by over 10%, but complex-reasoning categories remain at or below 50% accuracy, so simple supervised tuning is insufficient.The experiment used 50k annotated samples generated from 7k training scenes and fine-tuned each model for one epoch; the comparison is summarized in Table 4.Source: §4.3.3 · p. 10, Table 4 · p. 10
  • Fine-tuning raises scores across all question formats, but Estimation and Judgement gains can reflect shortcuts, while instruction-following degrades; the templated benchmark resists superficial shortcut learning.Models often repeat nearly identical estimates or default to one option, and some fail to provide both an answer and required reasoning after fine-tuning.Source: §4.3.3 · p. 10

5 Limitations and Conclusion

OST-Bench evaluates MLLMs’ online spatio-temporal reasoning and finds particular difficulty with complex reasoning and maintaining accuracy as inputs accumulate. Its static-scene and perception-only scope leaves object changes, interaction, and active manipulation outside the benchmark.

5 Limitations and Conclusion

  • The benchmark assumes static object states and positions, and evaluates online perception and reasoning without interactive behavior or active manipulation.The authors identify changing objects and positions, interaction, and manipulation as directions for future benchmark development.Source: §5 · p. 10

A Benchmark Details

The appendix details OST-Bench’s route construction, visibility processing, rule-based question generation, and dataset statistics. Together, these procedures define how exploration sequences and questions are constructed from scene data.

A.1 Exploration Route Generation

  • Routes use camera-position transitions along a minimum spanning tree and require overlap between adjacent images; random walks produce sequences of position, view-angle, and image tuples.Movement and observation continuity are enforced, but generated frame ordering does not encode time intervals; the benchmark treats rounds as discrete timestamps.Source: §A.1 · pp. 16–17, Figure 8 · p. 17

A.2 Visible Information Processing

  • Attribute visibility requires sufficient visible projected area, while spatial visibility requires at least five visible vertices of an object’s 9-DoF bounding box.Attribute visibility uses either a visible-to-total projected-area ratio threshold or an absolute visible-area threshold.Source: §A.2 · p. 17

A.3 Rule-based Generation

  • Questions arrive after each round’s 4–5 new frames, are sampled from subtype-specific pools, use category- or instance-level references, and require information from earlier rounds.Generation rules also exclude ambiguous cases using thresholds, including for spatial comparisons and directional judgments.Source: §A.3 · p. 18
  • Agent State tasks assess position and orientation relative to a historical time point, including displacement judgments and estimates of movement or rotation.Position judgments use a 1-meter threshold; orientation judgments exclude borderline angles.Source: §A.3 · pp. 18–19
  • Agent Visible Info tasks cover object existence and timing, category counts, newly observed objects, and the order in which categories first appear.Counting excludes answers of one and includes absent-category negatives in 25% of samples.Source: §A.3 · pp. 19–20
  • Agent-object relationship tasks assess distances and directions, including comparisons across rounds and judgments that require recalling objects absent from the current view.Spatial relationships require objects whose centers and dimensions can be obtained reliably from previous observations.Source: §A.3 · pp. 20–21

A.4 Statistics

A.5 Benchmark Examples

The appendix provides examples of benchmark samples from two exploration scenes.

A.5 Benchmark Examples

  • Twelve benchmark samples illustrate two scenes and their exploration trajectories.The examples are presented in Figures 16 and 17.Source: §A.5 · p. 21

B Implementation Details

The implementation presents sequential image observations and questions through multi-round prompts, using different context construction approaches for proprietary and open-source models. The appendix also specifies inference hardware for open-source models.

B Implementation Details

  • Each round supplies new video-frame images and a question; judgment prompts include options, while other formats request a numerical answer.The prompting instruction was added after models sometimes refused to answer on grounds of insufficient information.Source: B Implementation Details · p. 21, Figure 12 · p. 22
  • Proprietary models receive multi-round dialogue through APIs, whereas open-source model inputs concatenate dialogue history, new images, and the current prompt.Input images are resized to avoid out-of-memory errors.Source: B Implementation Details · p. 21
  • Open-source models up to 8 billion parameters run on one NVIDIA A100; models with 32 billion parameters or more use eight A100 GPUs.The authors also use multithreaded processing to accelerate open-source inference.Source: B Implementation Details · p. 22

C.1 Spatially-Grounded Model Evaluation

Spatially grounded models do not consistently improve performance on OST-Bench, and their gains can come with weaker instruction following and generalization. Additional findings identify estimation, cross-frame counting, and distance reasoning as persistent challenges.

C.1 Spatially-Grounded Model Evaluation

  • Only VLM-3R shows consistent gains over its base model; Spatial-MLLM and LLaVA-3D drop, while VLM-3R improves particularly on agent-state, agent-object spatial-relationship, and estimation tasks.Table 5 compares each specially designed model with its corresponding base model.Source: §C.1 · p. 22, Table 5 · p. 23
  • All three grounded models struggle with required response formats, often omitting reasoning or producing incoherent explanations; Spatial-MLLM is limited to floating-point or multiple-choice outputs.The comparison is with base models that reliably follow prompts and provide answers and reasoning.Source: §C.1 · p. 22
  • Spatial-MLLM and LLaVA-3D generalize poorly from spatial-reasoning datasets to OST-Bench’s diverse, temporally grounded prompts, while VLM-3R transfers only partially with modest gains.The authors note that memory-enhanced spatial grounding can come at the expense of broader generalization and base-model robustness.Source: §C.1 · pp. 22–23

C.2 More Findings in Tables

  • Estimation scores fall well below chance for models, and humans also score significantly lower on estimation than on other task categories.The authors attribute the difficulty to precise measurement requiring extensive empirical knowledge beyond approximate spatial perception.Source: §C.2 · p. 23
  • Models perform notably better on object-existence questions than on counting, where they confuse distinct objects across frames or fail to track the same object.The authors identify cross-frame reasoning as necessary for counting.Source: §C.2 · p. 23
  • The apparent advantage on agent-object distance over direction is superficial: models often use visibility as a near/far shortcut and perform poorly on distance overall.That heuristic sometimes works in a small portion of the benchmark but does not generalize.Source: §C.2 · p. 23

C.3 Cases of Three Error Types

The examples distinguish prompt-analysis, perception, and reasoning errors, and illustrate shortcuts in spatio-temporal reasoning. The Cross-View subset uses rule-based single- and multi-step object connections before manual filtering.

C.3 Cases of Three Error Types

  • Examples classify errors as prompt analysis, perception, or reasoning, including misreading a requested quantity or “discover,” missing objects, and misjudging spatial relations.The reasoning examples involve correctly understood prompts and perceived objects but incorrect conclusions about the agent’s spatial relation to them.Source: §C.3 · p. 23, Figure 13 · p. 24

C.4 Cases of Spatio-temporal Reasoning Shortcut

  • Shortcut examples include inferring movement from an object becoming visible, ignoring contradictory intermediate frames, assuming an invisible object stayed put, and skipping frames that show a turn.Other examples involve errors when the target object is invisible in the specific frames considered.Source: §C.4 · p. 25

C.5 Subset Construction Process for Cross-View Analysis

  • Single-step Cross-View samples pair frames where different objects appear, with an overlapping object linking the frames; sequence-based samples embed them among other frames.This construction is intended to make the spatial relation inferable through one reasoning step.Source: §C.5 · p. 25
  • Multi-step Cross-View samples use three frames, with each object appearing in a different frame and overlap links that require multiple reasoning steps; 200 rule-generated samples are manually filtered.The sequence-based context embeds the selected frames in a video that also includes frames without either target object.Source: §C.5 · p. 25

D Inference Time of the Models

For open-source models, inference latency rises rapidly with exploration history and becomes prohibitively high beyond about 10 dialogue rounds, while human response times remain stable. The discussion connects this contrast to knowledge compression and notes the benchmark’s static-scene and perception/reasoning scope.

D Inference Time of the Models

  • Beyond about 10 dialogue rounds, model inference time becomes prohibitively high, especially for large models, as latency grows with accumulated historical context.The study excludes proprietary models because network latency also affects their inference time; inference time scales approximately linearly with history length.Source: D Inference Time of the Models · pp. 25, 27, Figure 15 · p. 27
  • Human response time remains stable as exploration length increases, unlike model latency; the authors attribute this to humans forming and recalling compressed internal knowledge rather than reprocessing all history.The human comparison is not directly comparable in average latency because of individual variation.Source: D Inference Time of the Models · p. 27
  • The authors identify dynamic distillation and retention of essential, retrievable knowledge during exploration as a need for scalable, real-time embodied reasoning.They contrast this with passively accumulating history or answering each question in isolation.Source: D Inference Time of the Models · p. 27

E Social Impact

  • Because OST-Bench assumes static environments and evaluates perception and reasoning only, it may overestimate deployment readiness without interaction or manipulation capabilities.The authors caution against overreliance on models lacking those broader capabilities.Source: E Social Impact · p. 27

F.1 License and Acess for Existing Assets

  • Scene data come from ScanNet, Matterport3D, and ARKitScenes, while bounding-box and text annotations come from EmbodiedScan and MMScan; users must respect the datasets’ licenses and terms.Access to the source datasets is subject to their original licenses and, for EmbodiedScan and MMScan, a request process.Source: §F.1 · p. 28

F.2 License and Acess for OST-Bench

OST-Bench is distributed under CC BY 4.0 with stated terms of use. Its sample figures show observations added round by round, with images ordered within each round.

F.2 License and Acess for OST-Bench

  • OST-Bench is available under CC BY 4.0, and evaluation use must be properly disclosed under the benchmark’s terms.The benchmark’s terms do not guarantee absolute completeness or correctness of samples.Source: §F.2 · p. 28
  • The terms supplement, but do not override, source-material licenses; users remain responsible for applicable legal requirements concerning data subjects.The authors disclaim responsibility for legal issues arising from benchmark use.Source: §F.2 · p. 28
  • In both sample figures, each row represents newly added observations for a round, images are ordered left to right, and examples cover the first six rounds.The figures illustrate question-answer pairs across those rounds.Source: Figure 16 · p. 29, Figure 17 · p. 30
Loading 2507.07984v2…