PaperlayerView exact revision on arXiv

Source-linked AI summary

Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

Hanxun Yu, Xuan Qu, Lei Ke, Boqiang Zhang, Yuxin Wang, Jianke Zhu, Dong Yu

arXiv:2606.06891v1cs.CV

TL;DR

Existing 3D LMMs are largely offline, requiring complete scene observations or predefined clips, leaving online spatial interaction from streaming video insufficiently addressed. Stream3D-VLM learns response timing, incrementally injects geometry priors, compresses visual tokens, and uses scalable timestamped data and benchmarking; the paper reports leading performance across diverse online and offline 3D-language tasks.

  • Problem

    Existing 3D LMMs operate offline and require complete scene observations or predefined video clips, while streaming 3D-language data is scarce.

  • Method

    Stream3D-VLM uses autoregressive streaming control, incremental geometry-prior integration, geometry-adaptive voxel compression, and scalable timestamped 3D-language data generation.

  • Results

    The model achieves leading performance across diverse online and offline 3D-language tasks.

  • Takeaways & Limitations

    The work provides an online 3D spatial understanding model and benchmark for streaming-video perception, reasoning, interaction, and temporal response evaluation.

Abstract

from arXiv · show

Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction objective to learn when to respond, and employs a lightweight Visual-Spatial Feature Integration (VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M online spatio-temporal 3D QA pairs and establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/

1 Introduction

Stream3D-VLM addresses the offline nature of existing 3D multimodal models by enabling online spatial understanding and interaction from streaming video. It combines incremental geometry integration, learned response timing, scalable data generation, and a broad benchmark for evaluation.

  • Existing 3D vision-language models depend on complete 3D observations or predefined video clips, whereas real-world embodied applications require interaction during continuous streaming.
  • The model learns when to respond through autoregressive streaming control and incrementally integrates temporally aligned geometry priors for continuous 3D scene comprehension.
  • Over 1 million timestamped online 3D spatio-temporal QA pairs across 5.2k videos address the scarcity of large-scale streaming 3D-language data.
  • Experiments report state-of-the-art online 3D spatial understanding and reasoning while maintaining leading performance on offline visual grounding and dense captioning.
  • Stream3D-VLM is presented as an online 3D spatial understanding model operating solely on streaming video.

2 Related Work

Prior 3D LMMs use explicit 3D inputs or RGB-video reconstruction features, while Stream3D-VLM incrementally integrates geometry priors for continuous, real-time scene comprehension and supports systematic online evaluation.

  • Earlier 3D LMMs align point-cloud, mesh, or depth features with LLMs, but their reliance on sparse and costly 3D data limits scalability and model capacity.
  • Stream3D-VLM incrementally integrates geometry priors into the visual stream to enable continuous and real-time 3D scene comprehension.

3 Streaming 3D-Language Data Generation

The paper constructs scalable streaming 3D-language data and a benchmark organized around spatial competencies and temporal interaction modes.

  • Dataset and benchmark: Over 1 million QA pairs across 29 tasks and 5.2k 3D scans form the paper’s large-scale online spatio-temporal dataset.The pipeline uses explicit timestamps for instruction tuning.
  • Task taxonomy: The task taxonomy spans ego-motion, environment measurement, object–camera relationships, object chronology, and object attributes.These competencies range from agent motion and scene structure to fine-grained object analysis.
  • Temporal interaction modes: Three temporal interaction modes cover backward tracing of past frames, realtime perception of current evidence, and forward monitoring for future conditions.The modes probe memory, immediate spatial perception, and asynchronous response timing.
  • Data construction: The data pipeline combines RGB streams, depth maps, camera parameters, and 3D instance annotations from ScanNet, ScanNet++, and ARKitScenes.The source data retain real-world effects including motion blur, illumination changes, and sensor noise.
  • Question-answer generation: Rule-based generation uses computed metadata for ego-motion, object–camera relationships, environment measurement, and object chronology tasks.The paper also reports Stream3D-Bench with 10,037 manually curated samples from 518 real videos, balanced across content categories and interaction modes.
  • Evaluation: Stream3D-Bench evaluates numerical, multiple-choice, and open-ended answers with task-specific metrics, additionally reporting Answer-Timing Accuracy, TTFT, latency, and memory usage.Answer-Timing Accuracy measures alignment between predicted response time and the ground-truth timestamp.

4 Stream3D-VLM Architecture

Stream3D-VLM processes video incrementally, learns when to respond through autoregressive decision tokens, injects geometry priors into visual features, and compresses tokens using 3D structure.

  • Architecture overview: The architecture processes streaming video as a temporally ordered sequence and jointly trains streaming control with standard language modeling.The model learns whether to continue ingesting frames or initiate response generation.
  • Autoregressive streaming mechanism: Two decision tokens, <SEP> and <END>, represent streaming continuation and response triggering during autoregressive inference.At each frame, next-token prediction determines whether the model continues consuming visual input or responds.
  • Autoregressive streaming mechanism: The training objective combines streaming decision loss and language-modeling loss, with λ balancing the two cross-entropy objectives.The decision-token set contains <SEP> and <END>, while the language-modeling set contains the remaining tokens.
  • Visual-Spatial Feature Integration: VSFI combines 2D visual tokens with geometry and camera tokens extracted incrementally by StreamVGGT, projecting them into the LLM embedding space.A two-layer MLP produces geometry features, and cross-attention uses 2D tokens as queries and geometry tokens as keys and values.
  • Visual-Spatial Feature Integration: Cross-attention produces geometry-enhanced visual tokens while a residual connection preserves the original visual semantics.The attention uses learnable W_Q, W_K, and W_V projections with key dimension d_k.
  • Geometry-Adaptive Voxel Compression: GAVC back-projects image patches into 3D using predicted depth and camera parameters, then adds positional encoding to form spatially aware voxels.The back-projection uses depth, camera intrinsics, and camera extrinsics.
  • Geometry-Adaptive Voxel Compression: GAVC applies GPU-parallel spatial K-Means to group nearby voxels and aggregates each cluster with feature-similarity and spatial-proximity weights.The resulting compressed features encode semantic and spatial cues before being passed to the LLM.

5 Experiments

Experiments evaluate Stream3D-VLM on online, offline, and downstream 3D-language benchmarks, plus streaming-loss, geometry-prior, and token-compression ablations. The model achieves strong cross-task performance while balancing response timing, accuracy, and latency.

  • Experimental Setup: The experiments train on 1M+ streaming 3D spatio-temporal QA pairs and evaluate online, offline, grounding, question-answering, and dense-captioning tasks.Stream3D-Bench, VSI-Bench, ScanRefer, ScanQA, and Scan2Cap cover these evaluation settings.
  • Online Evaluation: Stream3D-VLM significantly outperforms proprietary and open-source models across three Stream3D-Bench categories for spatial understanding and temporal reasoning.The benchmark jointly measures answer quality and response timing under streaming input.
  • Offline Evaluation: 65.9% average accuracy is achieved by the 8B model on VSI-Bench, while the 4B variant reaches 55.2% and surpasses much larger 72B models and Gemini-2.5 Pro.These results indicate strong offline spatial understanding despite the model being designed for streaming scenarios.
  • Downstream Tasks: The model outperforms other methods across all ScanQA, ScanRefer, and Scan2Cap tasks and metrics without explicit 3D data.The 4B variant remains competitive with many larger methods.
  • Ablation Studies: A streaming-to-generation loss weight ratio of 2.0 provides the optimal balance, whereas over-weighting harms overall performance and underweighting reduces response-time accuracy.Standard cross-entropy and focal loss perform similarly in the reported ablation.
  • Ablation Studies: 25% token retention preserves competitive accuracy with substantially reduced latency in GAVC, whose spatially guided updates handle irregular voxel distributions.The comparison concerns response accuracy, answer-timing accuracy, TTFT, and end-to-end latency on Stream3D-Bench.

6 Conclusion

Stream3D-VLM is presented as an online 3D spatial-understanding model operating solely on streaming video. It combines next-token streaming control, incremental geometric-prior injection, and geometry-guided visual-token compression, with leading performance across online and offline tasks.

  • Conclusion: Stream3D-VLM reformulates streaming control as next-token prediction so the model learns when to respond or remain silent.This preserves the LLM’s autoregressive training framework for online interaction.
  • Conclusion: VSFI incrementally injects temporally aligned geometric priors into the visual stream for continuous 3D scene comprehension.The module is part of the model’s streaming-video architecture.
  • Conclusion: GAVC dynamically compresses visual tokens using 3D structure to reduce long-context redundancy during online inference.It is described as a geometry-guided token-compression module.
  • Conclusion: Extensive experiments show leading performance across diverse online and offline 3D-language tasks.

Supplementary Material

The supplementary material provides metadata-computation details and additional experimental, visualization, dataset, benchmark, and evaluation-prompt information. Its methods include visibility reasoning, camera-motion characterization, and geometric measurement.

  • Supplementary Organization: The supplement contains metadata-computing details, data-generation procedures, Stream3D-1M statistics, Stream3D-Bench details, visualizations, and offline-model evaluation prompts.
  • Visibility and Occlusion Reasoning: Object visibility is determined by projecting 3D mesh vertices into the image using camera intrinsics and extrinsics, then checking projected depth against the sensor depth map.The visibility rule also requires projected coordinates to lie within image bounds.
  • Visibility and Occlusion Reasoning: Frame-level object visibility is normalized by the maximum visible-vertex count across the video and thresholded into task-specific binary labels.
  • Camera Motion: Camera trajectories are characterized through path length, displacement, direction, and horizontal rotation derived from camera centers and orientations.
  • Geometric Measurement: Scene and object geometry metrics include object-camera direction and distance, inter-object distance, oriented-bounding-box size, and room area.Inter-object distance uses nearest-neighbor queries, while room area comes from the scene bounding box projected onto the horizontal plane.

B Detailed Data Generation Pipeline

The data-generation pipeline organizes streaming 3D tasks by cognitive competency and temporal interaction mode, then combines geometric metadata, templates, and verification to construct benchmark data. It applies a global object whitelist to improve consistency and spatial localizability.

  • Rule-based Generation Templates: Rule-based generation uses computed metadata and linguistic templates for ego-motion, object-camera relations, environment measurement, and object chronology.These tasks include motion, distance, direction, object size, room area, inter-object distance, counting, appearance order, and timestamps.
  • Task Taxonomy: Stream3D-Bench contains 29 tasks organized by three temporal interaction modes and five cognitive categories.The taxonomy spans perception and reasoning capabilities across online interaction modes.
  • Semantic Task Construction: Semantic tasks such as Attribute, Position, and Recognition adapt static ScanQA data by temporally grounding questions to the clearest frame.GPT-5 verification then checks whether the frame provides sufficient visual evidence and discards unsupported samples.
  • Target Object Curation: A Target Object Whitelist retains concrete, standalone, spatially localizable object classes while excluding broad categories, dependent parts, and continuous structural elements.Examples retain chairs, monitors, doors, and windows while excluding furniture, door frames, floors, ceilings, and walls.
  • Target Object Curation: The whitelist is applied globally to reduce referential ambiguity and focus evaluation on distinct 3D instances across object-centric tasks.

C Stream3D-1M Dataset Statistics

Stream3D-1M contains approximately 1 million QA pairs designed for streaming 3D capabilities, with distributions emphasizing spatial tracking, interaction, and temporal memory. Its task mix combines broad motion coverage with more specialized geometric and semantic questions.

  • 1,003,203 QA pairs from 5,154 unique scans comprise the Stream3D-1M dataset.The dataset is designed to train MLLMs for streaming 3D capabilities.
  • 43.5% of QA pairs come from ScanNet++, while ARKitScenes contributes 28.6%.ScanNet++ provides dense annotations, whereas ARKitScenes emphasizes camera motion and trajectories.
  • Ego-Motion Estimation dominates the task distribution at 50.1%, followed by Object-Camera Relation at 25.0% and Object Chronology at 14.3%.Environment Measurement accounts for 8.2%, and Object Attributes for 2.4%.
  • Backward Tracing accounts for 46.3% of temporal modes, with Realtime Perception at 19.8% and Forward Response at 33.9%.The distribution emphasizes memory and prediction capabilities alongside current perception.

D Stream3D-Bench Details

Stream3D-Bench is a 10,037-sample benchmark spanning diverse 3D sources, five task categories, three temporal modes, and balanced answer formats. Its composition is intended to support broad and fair evaluation of online spatial understanding.

  • 10,037 high-quality samples from 518 unique videos form Stream3D-Bench.The benchmark integrates ScanNet, ScanNet++, and ARKitScenes and is designed for systematic online 3D evaluation.
  • The benchmark covers Ego-Motion Estimation (35.5%), Object-Camera Relationship (31.9%), Environment Measurement (11.3%), Object Attributes (10.6%), and Object Chronology (10.6%).
  • Backward Tracing represents 41.3% of temporal modes, Realtime Perception 24.8%, and Forward Response 33.9%.These modes assess memory, current perception, and future monitoring or asynchronous response.
  • Open-ended, Numerical, and Multiple-choice answers comprise 36.2%, 31.9%, and 31.9%, respectively.The near-perfect balance prevents bias toward a specific answering style.

E More Visualization Results

The visualization examples span Stream3D-Bench’s temporal interaction patterns and cognitive categories, illustrating questions and responses about motion, object relations, chronology, and environment measurements. Examples include both immediate acknowledgments and later quantitative answers.

  • The examples cover all 29 Stream3D-Bench tasks across three temporal interaction patterns and five cognitive categories.
  • Backward interaction: 2.7m of camera travel is reported in one backward path example, while another reports 0.4m displacement at the 3 o'clock position after rotation.
  • Backward interaction: Backward examples include camera path, direction, object chronology, object-camera distance, object position, counting, and room-area questions.
  • Forward interaction: Forward examples include delayed object finding, predicted camera motion, object direction, object distance, and room measurement.
  • Forward interaction: A delayed response locates a backpack 2.1m away at the 1 o'clock position, while another reports a pillow at 5 o'clock and 1.5m distance.

F Evaluation Prompts for Offline Models

Offline MLLMs are adapted to the benchmark by receiving complete videos and explicit query times while being prompted to simulate streaming behavior. This gives them more visual information than true streaming, yet Stream3D-VLM still significantly outperforms offline baselines.

  • Offline models receive the full video and an explicit query timestamp to jointly predict response timing and answer content.The protocol is intended to simulate an online scenario despite offline input access.
  • Despite receiving the entire video upfront, offline baselines are consistently outperformed by the real-time streaming 3D model.The authors characterize the full-video protocol as favorable and unrealistic for offline models.
  • The evaluation prompt requires answers based only on visual information available up to the predicted response time, without future frames.
  • Response times are reported in seconds with 0.1s precision, using a JSON object containing prediction_answer_time and prediction_answer.
Loading 2606.06891v1…