Source-linked AI summary
Pixel-level Scene Understanding in One Token: Visual States Need What-is-Where Composition
Seokmin Lee, Yunghee Lee, Byeonghyun Pak, Byeongju Woo
TL;DR
Visual state representations for robotic agents must preserve both semantic identities and precise spatial locations, but prior self-supervised methods do not explicitly specify this requirement. CroBo addresses it with global-to-local reconstruction from a compact bottleneck token and achieves state-of-the-art robot policy-learning performance while preserving pixel-level scene composition and what-moves-where information.
Problem
Visual state learning must compress raw observations for sequential decision making while preserving what-is-where scene composition, a requirement not explicitly addressed by prior self-supervised methods.
Method
CroBo compresses a global reference observation into a bottleneck token and uses it with sparse visible cues to reconstruct a heavily masked local crop.
Results
CroBo achieves state-of-the-art performance across vision-based robot policy-learning benchmarks and captures pixel-level what-is-where and what-moves-where information.
Takeaways & Limitations
Compact visual representations that preserve pixel-level scene information support dynamics-aware understanding for sequential decision making.
Takeaways & Limitations
CroBo constructs both views from a single frame and is trained on video datasets primarily for fair comparison with prior work.
Abstract
from arXiv · showhide
For robotic agents operating in dynamic environments, learning visual state representations from streaming video observations is essential for sequential decision making. Recent self-supervised learning methods have shown strong transferability across vision tasks, but they do not explicitly address what a good visual state should encode. We argue that effective visual states must capture what-is-where by jointly encoding the semantic identities of scene elements and their spatial locations, enabling reliable detection of subtle dynamics across observations. To this end, we propose CroBo, a visual state representation learning framework based on a global-to-local reconstruction objective. Given a reference observation compressed into a compact bottleneck token, CroBo learns to reconstruct heavily masked patches in a local target crop from sparse visible cues, using the global bottleneck token as context. This learning objective encourages the bottleneck token to encode a fine-grained representation of scene-wide semantic entities, including their identities, spatial locations, and configurations. As a result, the learned visual states reveal how scene elements move and interact over time, supporting sequential decision making. We evaluate CroBo on diverse vision-based robot policy learning benchmarks, where it achieves state-of-the-art performance. Reconstruction analyses and perceptual straightness experiments further show that the learned representations preserve pixel-level scene composition and encode what-moves-where across observations. Project page available at: https://seokminlee-chris.github.io/CroBo-ProjectPage.
1. Introduction
CroBo addresses the open question of what visual states should encode by learning compact representations that preserve scene-wide semantic identities and precise spatial locations. Its global-to-local reconstruction objective supports pixel-level scene understanding and achieves state-of-the-art robot policy-learning performance.
- Motivation: Dynamic environments require visual state representations that preserve task-relevant information from streaming observations for sequential decision making.This challenge applies to robot learning and world modeling.
- Motivation: A good visual state must encode what-is-where: which semantic entities are present and how they are spatially arranged.This enables detection of subtle spatial changes and interactions across observations.
- Method: CroBo uses a compact bottleneck from a global reference observation to reconstruct an arbitrarily and heavily masked local crop from sparse visible hints.The task requires inferring both the crop’s scene location and its expected semantic content.
- Method: The reconstruction objective encourages the bottleneck to encode fine-grained, pixel-level scene composition across the full observation.The representation is intended to retain object identities, locations, and spatial relationships.
- Results: CroBo achieves state-of-the-art performance on vision-based robot policy-learning benchmarks while capturing what-moves-where in qualitative and perceptual analyses.The reported evidence includes reconstruction analyses and perceptual straightness experiments.
2. Method
CroBo learns compact visual states that preserve what-is-where scene composition through global-to-local reconstruction of masked local crops. Its bottleneck token carries global scene context needed to reconstruct missing target content from sparse local cues.
- CroBo targets visual states that preserve both scene-element identities and their spatial arrangement for dynamics-aware sequential decision making.
- Input views: The model samples a global source view and a spatially contained local target crop, then patchifies both views into non-overlapping patches.
- Siamese encoder and Masking: A shared-weight Siamese encoder processes the source without masking and the target with heavy masking, such as 90%, to prevent reconstruction from visible target tokens alone.
- Decoder: The decoder restores the full target token sequence with mask tokens and positional embeddings, then reconstructs masked patches using sparse target tokens and the source [CLS] bottleneck token.
- Decoder: This reconstruction design encourages the bottleneck to preserve which semantic objects are present and where they are located, enabling faithful reconstruction of masked regions.
- Objective: Training minimizes mean squared error over masked target patches using normalized pixel targets, while video data are used for fair comparison despite single-frame view construction.
3. Experiments
CroBo is evaluated across robot policy benchmarks and representation analyses, showing strong control performance, scene-composition reconstruction, and temporally coherent video dynamics. Ablations support the global-to-local Crop design and aggressive target masking.
- Vision-based Robot Learning: CroBo is evaluated on Franka Kitchen and DeepMind Control Suite benchmarks spanning robotic manipulation and locomotion.The visual backbone is frozen and an MLP policy head is trained by behavior cloning.
- Vision-based Robot Learning: CroBo consistently outperforms existing approaches across most tasks, achieving the best performance on four of five Franka Kitchen tasks and several DMC tasks.Reported gains include +13.6% on Micro open, +7.2% on Knob on, +7.0% on Light on, +8.3% on reacher/easy, +5.0% on walker/stand, and +3.1% on walker/walk.
- Scaling Behavior: 65.0% average success rate is achieved by CroBo with ViT-S/16, outperforming baselines built on the larger ViT-L/16.ViT-B/16 and ViT-L/16 reach 70.5% and 71.1% five-task averages, surpassing prior state of the art by +9.4% and +7.8%.
- Qualitative Analysis: CroBo reconstructs object attributes, spatial locations, and fine-grained scene details from heavily masked crops across synthetic, natural-video, and robotic scenes.The cyan spheres remain accurately localized despite being fully occluded in the masked input.
- Perceptual Straightness in Video: 75.4° average curvature versus 103.28° for DINOv2 indicates that CroBo representations follow more locally linear trajectories on DAVIS videos.PCA visualizations also show smooth CroBo trajectories that track the video's back-and-forth motion, unlike jagged trajectories from DINOv2 and CropMAE.
- Ablation Studies: Crop consistently outperforms Time across all five Franka Kitchen tasks because spatially grounded targets provide less ambiguous reconstruction supervision.Combining Time and Crop performs worse than either formulation, while increasing masking from 75% to 90% substantially improves performance and 95% performs best across tasks.
4. Related Work
Self-supervised learning methods learn transferable visual representations through alignment or reconstruction, while dynamic-scene methods target correspondence across views or time. However, patch-wise correspondence can be poorly suited to tasks requiring a single compact representation.
- Self-supervised learning methods commonly use contrastive alignment, teacher-student self-distillation, or masked reconstruction to learn visual representations.
- Siamese Masked Autoencoder predicts masked patches in one video frame from another to learn patch-wise temporal correspondence.
- CropMAE learns similar patch-wise correspondence from static images by reconstructing one cropped view from another.
- Patch-wise correspondence is suboptimal for downstream video tasks such as robotic manipulation that require a single compact frame representation.
5. Conclusion
CroBo addresses dynamic scene understanding by encoding semantic identities and spatial locations together in a compact bottleneck token. Its reconstruction and temporal analyses link pixel-level scene composition with what-moves-where tracking and state-of-the-art robot policy performance.
- CroBo uses global-to-local reconstruction, with a compact bottleneck token reconstructing masked local crops to encode unified semantic and spatial scene information.
- Reconstruction analyses show that CroBo captures fine-grained what-is-where scene composition.
- Perceptual straightness evaluations indicate that spatial grounding helps representations track what-moves-where across dynamic observations.
- CroBo achieves state-of-the-art performance on vision-based robot policy learning benchmarks.
Appendix
The supplemental material contains additional implementation details, related work, reconstruction visualizations, and perceptual straightness results.
- The supplemental material provides additional implementation details and related works.
- It also includes additional reconstruction visualizations and perceptual straightness results.
A. Additional Implementation Details
The appendix specifies CroBo’s decoder, pre-training schedule, and augmentation procedure. These details include MAE-style decoding, repeated sampling during the main comparison, and nested global-local crops.
- Decoder: The decoder follows ToBo and uses self-attention and multilayer perceptron layers configured according to MAE.
- Pre-training: Main comparisons use 400 pre-training epochs, equivalent to 200 practical epochs with repeated sampling, while ablations use 100 epochs.
- Augmentation: The global crop is sampled first, and the local crop is sampled within it so the local view remains fully contained.
- Augmentation: Global and local crop scales are [0.5, 1.0] and [0.3, 0.6], respectively, following CropMAE.
B. Additional Related Works
The paper situates CroBo among benchmarks for fine-grained video understanding, compact world-model states, and locality-aware representation learning. It uses challenging VOS datasets for reconstruction and trajectory analyses rather than standard VOS evaluation.
- VOS benchmarks evaluate fine-grained spatiotemporal understanding using dense pixel-level annotations across increasingly complex scenes.
- CroBo instead learns pixel-level scene understanding in a compact visual state representation based on a [CLS] token.
- CroBo uses DAVIS and MOSEv2 for reconstruction and perceptual-straightness analyses because its compact-state objective is not directly aligned with standard VOS protocols.
- Latent world models depend on visual representations to encode high-dimensional observations into compact states for prediction and decision-making.
- Locality-aware representation learning motivates preserving object- and region-level information for fine-grained image understanding.
C. Additional Reconstruction Visualization
Additional visualizations show that CroBo reconstructs heavily masked target views across diverse scene types while preserving object identities and spatial locations.
- 90% masked target views are reconstructed across CLEVR, DAVIS, MOSEv2, and Franka Kitchen using the reference-view bottleneck token.
- The reconstructions faithfully restore object identities and spatial locations across diverse scene types.
D. Additional Perceptual Straightness Results
Additional trajectory analyses compare CroBo with DINOv2 and CropMAE across linear motion, periodic rotation, and sequential manipulation. CroBo’s trajectories are smoother and more interpretable, with patterns that correspond to scene evolution and interaction events.
- Analysis setup: PCA visualizations compare representation trajectories across near-linear motion, periodic rotation, and sequential manipulation scenarios.
- Straight taxiing airplane: CroBo produces a smooth, nearly linear trajectory for an airplane taxiing under steady camera motion.
- Rotating radar antenna: CroBo captures five-turn radar rotation as a coherent repeating C-shaped trajectory reflecting the motion’s cyclic structure.
- Opening microwave: During microwave opening, CroBo’s trajectory turning point corresponds to grasping the handle, whereas competitors produce irregular zigzagging trajectories.
- Reconstruction visualization: The reconstruction examples pair reference, masked, reconstructed, and target views to illustrate the visual inputs and outputs.