PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams

Yibin Yan, Jilan Xu, Shangzhe Di, Haoning Wu, Weidi Xie

arXiv:2603.12265v1cs.CV

TL;DR

Visual agents need general, causal, physically structured representations, whereas existing vision foundation models remain fragmented across semantics, temporal modeling, and geometry. OmniStream combines causal spatiotemporal attention, 3D-RoPE, and synergistic multi-task pre-training into a frozen streaming backbone. Across diverse perception, reasoning, reconstruction, and manipulation evaluations, it is consistently competitive with specialized experts, supporting the viability of a versatile unified vision backbone while leaving scaling for future work.

  • Problem

    Current vision foundation models specialize narrowly in image semantics, offline temporal modeling, or spatial geometry, leaving limited evidence for one causal backbone spanning these capabilities in streaming settings.

  • Method

    OmniStream extends a pre-trained image encoder with causal spatiotemporal attention and 3D-RoPE, then couples static and temporal learning, geometric reconstruction, and vision-language alignment across 29 datasets.

  • Results

    OmniStream remains consistently competitive with specialized experts across image and video probing, streaming geometric reconstruction, video and spatial reasoning, and robotic manipulation with a strictly frozen backbone.

  • Takeaways & Limitations

    The results support training a single versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning for interactive and embodied agents.

  • Takeaways & Limitations

    OmniStream does not uniformly surpass specialized state-of-the-art methods, and model scaling is left as future work to close remaining gaps.

Abstract

from arXiv · show

Modern visual agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and video probing, streaming geometric reconstruction, complex video and spatial reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a single, versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and embodied agents.

1 Introduction

OmniStream addresses the fragmentation of vision models by targeting general, causal, and physically structured representations for continuous streams. It combines causal streaming architecture with multi-task training so one backbone can support diverse perception, reasoning, and embodied tasks.

  • Motivation: Streaming agents need representations that update beliefs online from continuous visual input under tight latency and memory budgets.The desired representation should support recognition, reasoning, and interaction while capturing appearance, geometry, and motion.
  • Research gap: Existing vision systems specialize in static semantics, temporal dynamics, or spatial geometry rather than transferring across all three in an online causal regime.Vision tasks also differ substantially in supervision and output types, encouraging fragmented foundation models.
  • Research question: The paper asks whether a single streaming backbone can provide sufficiently universal representations for downstream tasks without modifying or fine-tuning the backbone.This representation-centric goal differs from unifying tasks only through text-like output interfaces.
  • Approach: OmniStream turns a pre-trained image model into a causal streaming backbone using causal spatiotemporal attention, persistent KV-caching, and 3D-RoPE.These choices support frame-by-frame inference, avoid recomputing past frames, and encode where-and-when relationships across long streams.
  • Contribution: OmniStream is pre-trained across 29 datasets with complementary semantic, dynamic, geometric, and language objectives, then evaluated as a frozen backbone across perception, reasoning, and robotic manipulation.The authors report competitive or superior performance against specialized experts across the supported task spectrum.

2 Related Work

Related work spans unified vision-language interfaces, spatiotemporal visual representation learning, feed-forward 3D reconstruction, and visual encoders for VLMs and VLAs. These lines of work motivate the need for representations that combine semantics, temporal structure, geometry, and language grounding.

  • Video and 3D reconstruction: Video representation research focuses on complex spatiotemporal dependencies, while feed-forward 3D reconstruction has expanded from image pairs toward multi-view and video inputs.The related work reflects a shift from offline sequence reconstruction toward more temporally capable visual modeling.
  • VLMs and VLAs: VLMs and VLAs depend on visual encoders for textual reasoning and grounding, but standard contrastive encoders can remain weak on fine-grained visual perception.Recent approaches therefore explore stronger backbones and combinations of specialized encoders.

3 Method

OmniStream converts a pre-trained image encoder into a unified streaming backbone for causal perception, geometric reconstruction, reasoning, and action. It combines causal spatiotemporal processing with multi-task training across representation learning, geometry, and vision-language alignment.

  • 3.1 Problem Formulation: OmniStream processes each frame with historical context while enforcing strict causality, so outputs cannot depend on future frames.The formulation produces Ot = fθ(It | I1, . . . , It−1) and supports static, dynamic, geometric, and embodied regimes.
  • 3.2 Streaming Visual Backbone: Causal spatiotemporal attention lets tokens attend only to the current and earlier frames, enabling online inference without future-frame access.The causal mask permits every patch in frame t to attend to patches from frames ≤t, but never to future frames.
  • 3.2 Streaming Visual Backbone: Persistent KV-caching reuses keys and values from earlier frames, avoiding recomputation over the full past stream while matching full causal attention.When frame t arrives, only its queries are computed while cached keys and values from frames ≤t−1 are reused.
  • 3.2 Streaming Visual Backbone: 3D-RoPE extends 2D spatial positional encoding into the spatiotemporal domain, allocating per-head dimensions across time, height, and width.The allocation uses a 2:3:3 split across temporal, vertical, and horizontal components while preserving pre-trained spatial structure.
  • 3.3 Unified Multi-task Learning Objectives: The unified training framework couples static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment.Static and temporal learning uses DINOv3-style student-teacher distillation, geometric learning uses depth, ray, point, and camera supervision, and language alignment uses a projector and autoregressive decoder.
  • 3.3 Unified Multi-task Learning Objectives: Geometric heads reconstruct streaming scene structure by predicting dense depth and ray maps plus per-frame camera poses from selected backbone features.The camera pose contains rotation quaternion, translation vector, and field-of-view parameters, supervised alongside normalized depth, ray, point, and pose targets.
  • 3.3 Unified Multi-task Learning Objectives: Vision-language training back-propagates decoder gradients into the vision backbone, providing fine-grained semantic and spatial supervision for downstream integration.The branch uses an MLP projector and lightweight autoregressive language decoder on dense captioning, OCR, and object-grounding tasks.

4 Experiments

OmniStream is evaluated as a frozen visual backbone across static and dynamic perception, online geometric reconstruction, video-language reasoning, and robotic manipulation. Across these settings, the experiments test whether one representation can preserve spatial features while supporting temporal, geometric, and embodied capabilities.

  • Experimental Setup: OmniStream is evaluated with a strictly frozen backbone using image and video probing, geometric reconstruction, VLM reasoning, and robotic manipulation.The evaluation includes linear probing, online depth and pose estimation, video question answering, spatial reasoning, and VLA policies.
  • Image & Video Probing: 68.5% versus 54.0% on SSv2 shows stronger motion understanding than DINOv3, while dense image performance remains comparable to the image specialist.OmniStream also reaches a J &F mean of 71.6, compared with 73.2 for DINOv3, and avoids the poorer spatial alignment reported for V-JEPA 2.
  • Streaming Geometric Reconstruction: OmniStream achieves highly competitive or superior online depth and camera-pose reconstruction against specialized 3D models.Causal attention enables frame-by-frame KV-cached inference with O(T) temporal complexity per step; a T = 16 training window extrapolates to streams of up to 110 frames.
  • Visual Backbone for VLMs: With a frozen encoder, OmniStream outperforms LLaVA-Video on general Video QA benchmarks and achieves state-of-the-art VSI-Bench performance against geometry-aware baselines.It is marginally behind on VideoMME but superior on VideoMMMU, PerceptionTest, and EgoSchema.
  • Visual Backbone for VLA Policies: Frozen-backbone robotic manipulation reaches 3.885 success-rate units on CALVIN and 45.8% on SIMPLER-ENV.The model is adapted with a lightweight MLP action head for long-horizon instruction following and real-to-sim generalization.
  • Ablation Study: Ablations show that video modeling, 3D geometry, and captioning each support different downstream capabilities, making the multitask formulation synergistic.Removing video modeling harms motion and control; removing geometry harms spatial perception, spatial intelligence, and VLA performance; removing captioning harms VLM integration.

5 Conclusion

OmniStream is presented as a unified vision foundation model for semantic, spatial, and temporal reasoning across diverse tasks. The authors emphasize versatility across tasks rather than uniform dominance over specialized methods.

  • OmniStream integrates causal spatiotemporal attention and 3D-RoPE into a pre-trained Vision Transformer.The model is trained with multi-task objectives spanning diverse data sources.
  • The evaluation covers image and video probing, streaming geometric reconstruction, video question answering, spatial reasoning, and robotic manipulation.
  • The model remains consistently competitive across semantic, spatial, and temporal tasks without uniformly surpassing specialized state-of-the-art methods.
  • The SSL objective combines DINO, iBOT, KoLeo regularization, and Gram anchoring for static and temporal representation learning.DINO captures global semantic information, iBOT trains on masked local patches, KoLeo encourages uniform batch features, and Gram anchoring maintains patch-level consistency.

B Computational Efficiency and Scaling Analysis

The computational analysis compares causal KV-cache inference with full bi-directional recomputation as temporal context grows. OmniStream provides substantially better latency and memory scaling for long streaming contexts.

  • The evaluation reports per-frame latency and peak VRAM for the T-th frame given T −1 historical frames.
  • 0.998s recomputation latency is reached at T = 64, while the bi-directional baseline encounters OOM at T = 128.The baseline’s attention map grows quadratically with context length.
  • 0.067s latency at T = 64 makes OmniStream approximately 15× faster than the recomputation baseline.
  • OmniStream maintains linear, manageable memory growth and avoids OOM through T = 512.The persistent KV-cache leaves VRAM available for downstream VLM heads or real-time policy execution.

C Details of Downstream Tasks

Static representation quality is evaluated with linear probing on image classification, monocular depth estimation, and semantic segmentation using a strictly frozen visual backbone.

  • Linear probing evaluates ImageNet-1K classification, NYUv2 monocular depth estimation, and ADE20K semantic segmentation.
  • Each downstream task uses a task-specific linear head trained on top of the frozen visual backbone.
  • ImageNet classification uses a 13-configuration learning-rate sweep ranging from 1 × 10^-5 to 1 × 10^-1.The reported performance uses the optimal configuration from this sweep.

C.2 Video Probing Details

Video probing measures dynamic spatiotemporal representations through attentive action-recognition probes and dense video object segmentation. OmniStream uses causal KV-cache inference to process full sequences continuously.

  • Attentive probing evaluates Kinetics-400 and Something-Something V2 with a lightweight probe of 4 Transformer blocks and 16 attention heads.
  • Training samples 16 frames with a frame stride of 4, while inference uses multiple temporal segments and spatial crops.
  • Dense spatiotemporal tracking is evaluated on the DAVIS’17 Video Object Segmentation benchmark at 480p resolution.
  • OmniStream encodes the entire video sequence as one continuous stream using its causal KV-cache.This inference strategy is described as maintaining long-term temporal coherence.

C.3 VLM Probing Details

The VLM probing setup uses a frozen OmniStream vision model within the LLaVA-Video pipeline, while VLA experiments adapt OmniStream-7B for continuous action prediction on two robotic manipulation benchmarks.

  • VLM Probing: VLM experiments freeze the vision model throughout training and follow a three-stage fine-tuning procedure based on LLaVA-SI and LLaVA-Video.The training mixture includes LLaVA-Video-178K, ActivityNet-QA, NExT-QA, Perception Test, LLaVA-Hound-255K, and LLaVA-OV-1.
  • VLM Probing: Video inputs are average-pooled spatially, sampled at FPS=1, and limited to 32 frames during training.Image inputs use base resolution 512 with the AnyRes-Max strategy.
  • VLA Fine-tuning: The VLA model appends a learnable query token and uses a lightweight MLP action head to decode continuous 7-DoF actions.OmniStream-7B is built upon Qwen2.5-7B-Instruct and adapted using the VLM4VLA training pipeline.
  • VLA Fine-tuning: During VLA fine-tuning, optimization updates the multimodal projector, LLM, text embeddings, and newly initialized action head rather than the frozen visual backbone.The visual backbone remains frozen specifically to evaluate the quality of its pre-trained representations.
  • VLA Evaluation: VLA evaluations use the CALVIN Task ABC-D and SimplerEnv-BridgeV2 robotic manipulation benchmarks.The experiments use Adam, bfloat16 precision, and DeepSpeed ZeRO-2 optimization.

D.1 Video Depth Visualizations

OmniStream’s qualitative visualizations show streaming depth prediction on Sintel and mask propagation on DAVIS’17, emphasizing spatial detail, temporal consistency, and continuous object tracking.

  • Sintel Video Depth: On Sintel, OmniStream maintains high-fidelity spatial details and long-term temporal consistency across highly dynamic scenes.Figure 4 presents qualitative visualizations of streaming video depth prediction.
  • DAVIS’17 Mask Propagation: On DAVIS’17, causal KV-cache encoding helps OmniStream track intricate object shapes through fast appearance changes.Figure 5 presents qualitative mask-propagation results.
  • DAVIS’17 Mask Propagation: OmniStream is qualitatively advantaged over V-JEPA 2 because V-JEPA 2 frequently loses dense spatial alignment under sliding-window inference.The comparison concerns continuous encoding and tracking under fast appearance changes.
Loading 2603.12265v1…