PaperlayerView exact revision on arXiv

Source-linked AI summary

Cambrian-S: Towards Spatial Supersensing in Video

Shusheng Yang, Jihan Yang, Pinzhi Huang, Ellis Brown, Zihao Yang, Yue Yu, Shengbang Tong, Zihan Zheng, Yifan Xu, Muhan Wang, Daohan Lu, Rob Fergus, Yann LeCun, Li Fei-Fei, Saining Xie

arXiv:2511.04670v1cs.CV

TL;DR

The paper addresses the limited spatial and continual reasoning tested by current video MLLM benchmarks. It introduces VSI-SUPER, scales spatial data to train Cambrian-S, and prototypes predictive sensing; the results show strong standard-benchmark gains but continued VSI-SUPER limits, while prediction-driven sensing substantially outperforms leading long-context baselines. The authors conclude that spatial supersensing requires models that actively predict, select, and organize experience.

  • Problem

    Current video MLLM benchmarks and systems largely emphasize linguistic understanding and semantic perception while insufficiently testing continual spatial reasoning and predictive world modeling.

  • Method

    The paper introduces VSI-SUPER, trains Cambrian-S on the spatially focused VSI-590K corpus, and prototypes predictive sensing with self-supervised next-latent-frame prediction and surprise-driven memory and event segmentation.

  • Results

    Predictive sensing substantially outperforms strong long-context baselines on VSI-SUPER, while Cambrian-S achieves over 30% absolute gain on VSI-BENCH but still falls short on VSI-SUPER.

  • Takeaways & Limitations

    The results support predictive sensing as a promising direction beyond passive context expansion for long-horizon spatial reasoning.

  • Takeaways & Limitations

    The benchmark, dataset, and model design remain limited in quality, scale, and generalizability, and the predictive-sensing prototype is only a proof of concept.

Abstract

from arXiv · show

We argue that progress in true multimodal intelligence calls for a shift from reactive, task-driven systems and brute-force long context towards a broader paradigm of supersensing. We frame spatial supersensing as four stages beyond linguistic-only understanding: semantic perception (naming what is seen), streaming event cognition (maintaining memory across continuous experiences), implicit 3D spatial cognition (inferring the world behind pixels), and predictive world modeling (creating internal models that filter and organize information). Current benchmarks largely test only the early stages, offering narrow coverage of spatial cognition and rarely challenging models in ways that require true world modeling. To drive progress in spatial supersensing, we present VSI-SUPER, a two-part benchmark: VSR (long-horizon visual spatial recall) and VSC (continual visual spatial counting). These tasks require arbitrarily long video inputs yet are resistant to brute-force context expansion. We then test data scaling limits by curating VSI-590K and training Cambrian-S, achieving +30% absolute improvement on VSI-Bench without sacrificing general capabilities. Yet performance on VSI-SUPER remains limited, indicating that scale alone is insufficient for spatial supersensing. We propose predictive sensing as a path forward, presenting a proof-of-concept in which a self-supervised next-latent-frame predictor leverages surprise (prediction error) to drive memory and event segmentation. On VSI-SUPER, this approach substantially outperforms leading proprietary baselines, showing that spatial supersensing requires models that not only see but also anticipate, select, and organize experience.

1. Introduction

The paper argues that multimodal intelligence must progress from language-centric perception toward spatial supersensing, which constructs, updates, and predicts an implicit 3D world from continual experience. It introduces VSI-SUPER to expose limits of current video MLLMs, tests whether data scaling suffices with Cambrian-S, and proposes predictive sensing as an alternative direction.

  • Motivation: Most video MLLMs treat video as sparse frames, underrepresent spatial structure and dynamics, and rely heavily on textual recall.This overlooks video’s continual projection of a hidden, evolving 3D world.
  • Motivation: Spatial supersensing extends multimodal understanding toward constructing, updating, and predicting an implicit 3D world from continual sensory experience.The paper presents this as a developmental path rather than claiming to realize supersensing fully.
  • Benchmark: VSI-SUPER comprises VSR for long-horizon spatial recall and VSC for continual counting across changing viewpoints and scenes.Its arbitrarily long spatiotemporal videos require selective, structured perception rather than indiscriminate context accumulation.
  • Data scaling: Cambrian-S achieves over 30% absolute gain on VSI-BENCH through spatially focused data and training, without sacrificing general capabilities.Despite this progress, the model still falls short on VSI-SUPER, showing that scale alone is insufficient.
  • Predictive sensing: Predictive sensing uses self-supervised next-latent-frame prediction and surprise to allocate memory and segment unbounded streams.The approach significantly outperforms strong long-context baselines such as Gemini-2.5 on the two new tasks.

2. Benchmarking Spatial Supersensing

The benchmark audit finds that existing video benchmarks often emphasize linguistic understanding and semantic perception, while VSI-SUPER targets harder continual spatial capabilities through long-horizon recall and counting tasks.

  • Benchmark diagnostics: The diagnostic evaluates multiple frames, a single middle frame, frame captions, blind inputs, and chance accuracy to separate visual sensing from language priors.Multiple-frame input uses 32 uniformly sampled frames; captions correspond to those same frames.
  • Existing benchmarks overwhelmingly focus on linguistic understanding and semantic perception while neglecting advanced spatial and temporal reasoning.
  • VSI-SUPER: VSI-SUPER introduces VSR for long-horizon spatial observation and recall, and VSC for continual counting across changing viewpoints and scenes.VSR recalls unusual-object locations in order across arbitrarily long videos, while VSC accumulates target-object counts across concatenated room tours.
  • VSI-SUPER: VSC concatenates multiple room-tour clips and requires consistent cumulative counting despite viewpoint shifts, repeat sightings, and scene transitions.
  • VSI-SUPER: Gemini-2.5-Flash reaches its context limit on two-hour videos, while its VSI-SUPER performance remains limited even on 60-minute videos within the context window.The paper reports 41.5 and 10.9 on VSR and VSC, respectively, for 60-minute videos.

3. Spatial Sensing Under the Current Paradigm

The paper tests whether data and model scaling can improve spatial sensing under the current MLLM paradigm. Cambrian-S achieves strong VSI-Bench performance and generalization, but remains limited on continual, long-horizon spatial sensing.

  • Data and model development: VSI-590K is a spatial-focused instruction-tuning dataset built from diverse real, simulated, and pseudo-annotated visual sources.The dataset uses 3D-annotated videos and pseudo-annotated web-video frames, with automatically generated question–answer pairs and ten contributing sources.
  • Data and model development: The dataset covers 12 spatiotemporal question types spanning size, direction, count, distance, and appearance order.Questions include relative and absolute variants, varied viewpoints, wording, and measurement units.
  • Data and model development: Cambrian-S comprises 0.5B, 1.5B, 3B, and 7B models trained through a four-stage pipeline that first establishes semantic perception and then specializes spatial sensing.The training design combines general semantic capabilities with spatially focused skills.
  • Spatial cognition results: 67.5% on VSI-Bench: Cambrian-S-7B surpasses Gemini-2.5-Pro by over 16 absolute points while retaining competitive general video performance.The models also generalize to unseen spatial question types, including Route Planning, and the 0.5B model performs comparably to Gemini-1.5 Pro.
  • Spatial cognition results: About 8% lower performance on VSI-Bench-Debiased still leaves Cambrian-S ahead of proprietary counterparts, supporting visual rather than purely language-prior-based reasoning.The debiased benchmark is designed to remove language shortcuts from spatial reasoning tasks.
  • Continual spatial sensing limitations: 38.3% to 6.0%: VSI-SUPER scores decline as videos grow from 10 to 60 minutes, and the model fails on videos longer than 60 minutes.Performance also drops on short unseen scenarios that fit within the context window, indicating limitations beyond context length.

4. Predictive Sensing as a New Paradigm

The paper proposes predictive sensing, using self-supervised next-latent-frame prediction and surprise to manage memory and segment unbounded video streams. Across recall and counting tasks, surprise-based mechanisms improve long-horizon performance, while the prototype remains limited by its simple predictive head.

  • 4.1. Predictive Sensing via Latent Frame Prediction: Predictive sensing uses next-latent-frame prediction error as surprise to manage memory and segment unbounded visual streams.The prediction error selectively allocates resources to unexpected events and organizes continuous input into meaningful chunks.
  • 4.2. Surprise-driven Memory Management: Cambrian-S with surprise-driven memory maintains accuracy and stable GPU memory usage across video lengths on VSR.It outperforms both Gemini-1.5-Flash and Cambrian-S without memory at all tested durations.
  • 4.2. Surprise-driven Memory Management: Prediction error consistently outperforms adjacent-frame similarity as the surprise signal across VSR durations.The comparison uses otherwise identical experimental setups with separately tuned thresholds.
  • 4.3. Surprise-driven Event Segmentation: On VSC, surprise-driven segmentation gives Cambrian-S higher and more stable performance across video lengths than Gemini baselines.Ground-truth segmentation improves performance further, while Cambrian-S counts correlate more strongly with true object numbers than Gemini-2.5-Flash.
  • 4.3. Surprise-driven Event Segmentation: Streaming commercial models achieve under 15% MRA on 10-minute VSC videos, with performance declining toward zero on longer videos.The evaluated systems are Gemini-Live and GPT-Realtime.
  • Summary: The authors present predictive sensing as an early prototype for systems that anticipate, select, and organize experience beyond frame-level question answering.They characterize richer world models as a future direction rather than a realized capability.

5. Related Work

Related work spans video multimodal language models, streaming video understanding, visual spatial intelligence, and predictive modeling. These areas address visual representation, long-context processing, spatial reasoning, and internal models for anticipating environments.

  • Video Multimodal Large Language Models: Video MLLMs combine pretrained language models with vision foundation model features to extend multimodal understanding beyond text.This work has primarily advanced semantic perception, especially for images.
  • Streaming Video Understanding: Streaming video understanding addresses unbounded inputs through efficient architectures, context expansion, retrieval, and visual token compression.These approaches target rising computational and storage costs as video duration increases.
  • Visual Spatial Intelligence: Visual spatial intelligence research studies spatial relationships needed for perceiving and interacting with the physical world.Recent work has introduced benchmarks and methods for improving spatial reasoning.
  • Predictive Modeling: Predictive modeling uses learned internal models to represent and simulate environments for planning and decision-making.Model predictive control applies related forward-model principles to anticipate trajectories and choose actions in real time.

6. Conclusion

The conclusion frames spatial supersensing as requiring spatial cognition and predictive world models beyond text-based knowledge and semantic perception. It reports that predictive sensing improves VSI-SUPER performance while emphasizing that the benchmark, dataset, model, and prototype remain limited.

  • Conclusion: The paper proposes a hierarchy for spatial supersensing and introduces VSI-SUPER to measure capabilities beyond semantic perception.Current multimodal language models struggle with the benchmark.
  • Conclusion: Cambrian-S performs strongly on standard spatial reasoning benchmarks but does not directly solve VSI-SUPER’s continual sensing challenges.Its role is to provide a foundation motivating the predictive-sensing paradigm.
  • Conclusion: Predictive sensing uses latent frame prediction and surprise estimation to handle unbounded visual streams and improve Cambrian-S on VSI-SUPER.The conclusion characterizes this as an early step toward spatial supersensing.
  • Limitations: The benchmark, dataset, and model design remain limited in quality, scale, and generalizability, and the prototype is only a proof of concept.Future work should examine more diverse and embodied scenarios and connect with advances in vision, language, and world modeling.

A. Benchmark Diagnostic Test Results

The appendix reports detailed benchmark results for the improved Cambrian-1-7B under multiple evaluation setups. These materials support diagnostic comparisons of visual inputs and textual cues across video MLLM benchmarks.

  • Benchmark Diagnostic Test Results: Table 8 presents detailed results for improved Cambrian-1-7B across video MLLM benchmarks and evaluation setups.The appendix identifies the table as the detailed result source for the benchmark analysis.

B. VSI-SUPER Benchmark

VSI-SUPER is constructed to test continual spatial recall and counting over arbitrarily long visual streams. Its taxonomy organizes spatial questions across attributes, relationality, perspective, modality, and task group.

  • VSI-SUPER Recall: VSI-SUPER Recall inserts unusual objects into four video frames, records each object’s location and appearance order, and concatenates edited clips with unedited videos.The inserted objects are designed to appear in plausible locations within indoor walkthrough videos.
  • VSI-SUPER Count: VSI-SUPER Count concatenates VSI-Bench videos and sums their object counts into a new ground truth.Construction normalizes object-category labels and rebalances question-answer pairs toward a more uniform count distribution.
  • VSI-SUPER Count: In streaming evaluation, models are queried for total object counts at 10 timestamps, with ground truth derived from each unique object’s first appearance time.The appearance-time procedure follows VSI-Bench.
  • VSI-590K taxonomy: VSI-590K defines spatial-temporal questions across size, direction, count, distance, and appearance order, plus relative-versus-absolute distinctions.The taxonomy also includes perspective, modality, and task group.
  • VSI-590K taxonomy: The taxonomy groups questions into configuration, measurement, and spatiotemporal categories.These groups are defined according to differing spatiotemporal characteristics.
  • Dataset construction: VSI-590K combines spatial question-answer construction from 3D-annotated real and simulated data with processed unannotated web-crawled videos.The web-video pipeline samples and filters frames, detects objects, extracts instance masks, and reconstructs spatial information.

C.3. Additional Ablation Study

The ablation finds that configuration, measurement, and spatiotemporal task groups all contribute to spatial sensing, with configuration most influential on held-out route planning.

  • Task-group ablation: Removing any one of VSI-590K’s three task groups degrades spatial-sensing performance.The evaluated groups are configuration, measurement, and spatiotemporal.
  • Task-group ablation: On the held-out Route Plan subtask, configuration is the most influential task group and measurement is the least.The authors attribute configuration’s effect to route-planning requirements.
  • Dataset examples: VSI-590K is presented with qualitative visualizations intended to demonstrate diversity and quality in spatial question-answering supervision.The visualizations appear in Figures 19 to 25.

D.2. Training Data Mixture

Cambrian-S stage 3 uses a curated general video instruction mixture, while the preceding training stages rely on the model’s established image-training setup.

  • Training data mixture: Cambrian-S uses four training stages, with stages 1 and 2 following Cambrian-1’s vision-language alignment and image instruction-tuning data mixture.The supplied passage introduces the four-stage schedule but does not provide the full stage-3 mixture details.
  • QA generation pipeline: The QA-generation pipeline takes a video sequence and category and filtering parameters as input, then returns selected frames and question-answer pairs.Its listed inputs include sampling interval, blur threshold, minimum object count, minimum 3D-point count, and an erosion kernel.
  • Training data mixture: CambrianS-3M is a curated video instruction-tuning dataset containing around 3M video question-answer samples.It is built from open-sourced video datasets spanning general video understanding sources.

D.3. Training Recipe

Cambrian-S progresses from alignment and image instruction tuning to general and spatial video tuning, while experiments test how image and video data mixtures affect video benchmarks.

  • Training recipe: Stage 1 freezes most parameters and trains the vision-language connector on Cambrian-Alignment-2.5M.Images use a fixed 384 × 384 resolution and a maximum sequence length of 2048.
  • Training recipe: Stage 2 jointly trains the connector and language decoder on Cambrian-7M while keeping the vision encoder frozen.AnyRes divides aspect-ratio-preserving resized images into multiple 384 × 384 sub-images.
  • Training recipe: Stage 3 performs general video instruction tuning on CambrianS-3M mixed with sampled Cambrian-7M image data.The vision encoder remains frozen; videos are uniformly sampled at 64 frames and resized to 384 × 384.
  • Training recipe: Stage 4 fine-tunes on VSI-590K while mixing 590K CambrianS-3M video samples and 120K Cambrian-7M image samples.The recipe increases videos to 128 frames and extends sequence length to 16,384 for richer temporal modeling.
  • Systems implementation: GSPMD shards training states across devices, while FlashAttention with Pallas reduces TPU memory use for long-context inputs.This supports sequence lengths up to 16,384 tokens for the 7B model on a TPU v4-512 Pod.
  • Data-mixture experiments: Increasing video fine-tuning data does not guarantee consistent gains across benchmarks; only EgoSchema, MVBench, and Perception Test improved consistently.VideoMME, VSI-Bench, and Tomato did not show further gains with more video data.

E.4. On the Trade-off between Spatial Sensing and General Video Understanding

Spatial-sensing fine-tuning improves spatial tasks, while mixed-data fine-tuning better balances spatial sensing with general video understanding. This trade-off persists across model scales, but the VSI-Bench gap narrows for larger models.

  • VSI-590K-only fine-tuning consistently outperforms mixed-data fine-tuning on spatial sensing tasks.
  • Mixed-data fine-tuning provides a better balance between spatial sensing and general video understanding.
  • The spatial-sensing versus general-understanding trade-off holds across all evaluated model scales.
  • The performance gap on VSI-Bench narrows as model size increases.The authors attribute this trend to larger models’ greater capacity to learn and retain diverse capabilities.

F.1. Latent Frame Prediction Implementation Details

The implementation combines latent next-frame prediction with surprise-driven memory operations for long-horizon video processing. It uses sensory, long-term, and working memory, selectively compresses and consolidates information, and triggers responses at surprising events.

  • The latent frame prediction head is a two-layer GELU MLP running in parallel with the original language-model head.Its output dimension is 1152, matching the SigLIP2 vision encoder output.
  • Auxiliary cosine-distance and mean-squared-error losses optimize next-frame prediction alongside instruction tuning.A coefficient balances the latent frame prediction loss against the instruction-tuning loss.
  • Predictive memory comprises sensory, long-term, and working memory connected by streaming, compression, consolidation, and retrieval functions.
  • Incoming frames use a sliding window and reference selected previous-frame key-value pairs cached in sensory memory.
  • Non-surprising frames are spatially downsampled when transferred to long-term memory, reducing redundancy.The compression is applied when a frame’s surprise score falls below a predefined threshold.
  • When long-term memory exceeds its token budget, the lowest-surprise frame is removed; query-relevant frames are then retrieved into working memory.Retrieval uses similarity measured through the LLM’s attention mechanism.
  • For VSI-SUPER Count, prediction error above a threshold marks surprising frames that may indicate scene or spatial boundaries and trigger query responses.Intermediate answers are stored and aggregated into the final answer.

F.4. Comparisons with Existing Long-video Methods

The paper compares its surprise-driven memory and agentic framework with existing long-video methods on VSI-SUPER using LFP-finetuned Cambrian-S-7B. The supplied passages also show benchmark examples spanning recall, counting, and annotated video data.

  • The comparison evaluates both surprise-driven memory and the agentic framework against existing long-video methods on VSI-SUPER.
  • All comparison experiments use the LFP-finetuned Cambrian-S-7B with different strategies for handling expanding visual sensory input.
  • The supplied examples include VSI-SUPER Recall, VSI-590K annotated real and simulated video, and object-counting questions.
  • An example comparison asks whether chairs are fewer than, more than, or equal in number to tables.
Loading 2511.04670v1…