Source-linked AI summary
Semantic Slots for Video Object-Centric Learning
Khalil Sabri, Guillaume-Alexandre Bilodeau, Nicolas Saunier, Wassim Bouachir
TL;DR
Video OCL has emphasized temporal objectives and encoder consistency, but context-free decoders force slots to encode spatial coordinates, limiting motion adaptation. SemanticSlots uses a context-aware Transformer decoder so slots act as position-invariant semantic queries that can be reused across frames without complex temporal predictors or auxiliary losses. It achieves state-of-the-art YouTube-VIS results with 86.6% ARI and 62.8% mBO.
Problem
Context-free decoders force slots to encode semantic identity together with absolute spatial coordinates, creating spatial entanglement that hinders adaptation to motion.
Method
SemanticSlots uses context-aware cross-attention to retrieve matching objects from frame features, reusing initial slots as fixed semantic queries across video frames.
Results
86.6% ARI and 62.8% mBO on YouTube-VIS establish state-of-the-art performance through architectural simplification.
Takeaways & Limitations
Slots computed from a single frame can decompose subsequent video frames without complex temporal predictors or auxiliary temporal losses.
Takeaways & Limitations
The framework prioritizes semantic retrieval over instance-level temporal continuity, which can create ambiguity between identical entities in a scene.
Abstract
from arXiv · showhide
Video Object-Centric Learning (OCL) has traditionally focused on refining the encoder architecture to ensure temporal consistency. In this paper, we argue that the primary bottleneck lies in the decoder. We show that traditional decoders force slots to be spatially anchored, hindering their ability to adapt to motion. We propose SemanticSlots, which uses a Transformer-based decoder that leverages image context, relieving slots from encoding boundary precision and spatial location. This allows slots to function as semantic queries that are inherently object position invariant, retrieving matching features rather than memorizing coordinates. More importantly, this property allows slots computed from a single frame to decompose subsequent video frames, eliminating the need for complex temporal predictors or auxiliary temporal losses. Results on YouTube-VIS show that SemanticSlots improves upon VideoSAUR by 31 points in mBO and outperforms current state-of-the-art methods by 21 points, achieving 86.6% ARI and 62.8% mBO.
1 Introduction
SemanticSlots identifies context-free decoding as a structural bottleneck in video OCL and uses image-aware retrieval to make slots position-invariant across frames. This enables single-frame slots to decompose videos without complex temporal machinery while achieving strong YouTube-VIS results.
- Motivation: Traditional feature-agnostic decoders force low-capacity slots to encode absolute spatial coordinates and high-frequency details for reconstruction.This spatial burden limits adaptation to object motion.
- Motivation: Context-aware cross-attention uses updated image features as a search space, re-localizing objects instead of reconstructing them at memorized positions.The decoder shifts slots from spatial memorization toward contextual retrieval.
- Method: Position-invariant slots computed from one frame can be reused across an entire video as fixed semantic queries.Cross-attention re-binds these queries to objects in subsequent frames without heavy temporal prediction or recurrent sequence modeling.
- Method: SemanticSlots is trained entirely on single images without temporal supervision and eliminates complex temporal predictors and auxiliary losses.Refreshing queries can further sharpen attention, handle shape changes, and capture objects entering late.
- Results: 86.6% ARI and 62.8% mBO establish state-of-the-art performance on YouTube-VIS, surpassing the previous best by 39.8 and 21.3 points.The reported gains are achieved through architectural simplification rather than added temporal complexity.
2 Related Work
Prior OCL research established slot-based object discovery and feature reconstruction, while video extensions mainly added temporal machinery and encoder-side consistency. SemanticSlots instead makes cross-image transferability central, using position-invariant slots to reduce reliance on temporal modeling.
- Foundations and Feature-Based Object Discovery: OCL decomposes images into discrete latent entities called slots for downstream relational reasoning, control, and visual question answering.Slot Attention extracts these representations through an iterative competitive bottleneck.
- Foundations and Feature-Based Object Discovery: Feature reconstruction with DINO representations improved object discovery in real-world images by avoiding the capacity wasted on high-frequency pixel noise.DINOSAUR introduced this feature-based direction.
- Decoder Design: Traditional context-free decoders force slots to encode absolute spatial coordinates because they lack direct access to image context during reconstruction.Transformer decoders were also explored for complex visual dependencies, including continuous feature reconstruction.
- SemanticSlots: SemanticSlots makes cross-image slot transferability central by treating slots as position-invariant representations and using that property to eliminate video temporal machinery.The approach reframes the key challenge around decoder-induced spatial entanglement.
- Video Object-Centric Learning: Video OCL has primarily followed transitioner designs with recurrent slot updates, auxiliary motion or depth signals, and temporal similarity objectives.VideoSAUR combined feature reconstruction with a SlotMixer decoder and temporal similarity losses.
3 Methodology
SemanticSlots addresses spatial entanglement in standard video OCL decoders by using context-aware decoding to make slots position-invariant. It computes slots from images and reuses them across video frames without temporal predictors or temporal losses.
- 3 Methodology: Standard video OCL maintains temporal continuity with recurrent slot updates, predictors, and temporal losses.
- 3 Methodology: Context-free decoders force slots to encode both object identity and absolute spatial coordinates, causing reconstructions to remain at memorized locations when objects move.Frozen slots decoded against shifted images reconstruct objects at their original positions.
- 3 Methodology: SemanticSlots uses a Transformer-based decoder conditioned on slots and current-frame features, shifting precise spatial encoding from slots to the decoder.Cross-attention retrieves information from slots while causal self-attention uses spatial context within the current frame.
- 3 Methodology: Slots computed from the first frame can be reused as fixed semantic queries for later frames without predictors or temporal losses.SemanticSlots evaluates frozen slots, which remain fixed, and refined slots, which undergo slot-attention iterations at each frame.
- 3 Methodology: Segmentation is obtained from decoder cross-attention weights rather than reconstructed features, assigning patches to slots during inference.Reconstruction is used only during training to encourage useful slot representations.
Experiments
Experiments evaluate SemanticSlots on synthetic and real-world video benchmarks using decomposition, segmentation, efficiency, and stability measures. The results show strong performance, reduced temporal machinery, and a decoder advantage under object motion, while dynamic-scene handling and instance-level continuity remain bounded.
- Benchmark results: SemanticSlots achieves SOTA results across nearly all metrics on synthetic and real-world benchmarks, surpassing the previous best transition-based method on YouTube-VIS by 39.8 ARI points and 21.3 mBO points.On MOVi-C, it also exceeds the strongest baseline by 18.1 ARI points and 8.0 mBO points; on MOVi-D it reaches 62.7% ARI and 35.5% mBO.
- Qualitative results: The model produces more spatially consistent masks than prior methods, which sometimes show flickering or misassigned patches across frames.The authors attribute this stability to retrieval-based decoding without a temporal predictor, avoiding error propagation in transition-based frameworks.
- Slot utilization: SemanticSlots often uses fewer active slots because its Transformer decoder retrieves spatial details from image context instead of encoding them in slots.The paper links this reduced slot usage to avoiding background over-clustering and achieving higher ARI across benchmarks.
- Dynamic scenes: SemanticSlots-Frozen can fail when objects enter late or are initially occluded, while Adaptive refreshes slots selectively and Refined remains the most robust mode.Adaptive improves mBO from 60.8 to 61.3 while increasing inference time from 58.6 ms to 60.8 ms per video, but novelty triggering can miss poorly represented existing objects.
- Efficiency and ablations: Removing temporal machinery and switching to image-level training achieves the best results, indicating that retrieval can replace additional video components in this setting.The three inference regimes share the same model parameters and inherit the single-frame training efficiency; the model trains with T=1 rather than T=6 frames per iteration.
- Limitations: The retrieval-focused framework can create ambiguity between identical entities because it prioritizes semantic retrieval over instance-level temporal continuity.The paper identifies re-integrating temporal priors as future work.
5 Conclusion
SemanticSlots addresses spatial entanglement in context-free decoders through retrieval-based decoding, achieving state-of-the-art YouTube-VIS performance. Qualitative comparisons also report more stable identity assignments and sharper boundaries than prior methods.
- 5 Conclusion: 86.6% ARI and 62.8% mBO establish SemanticSlots as state of the art on YouTube-VIS.The paper attributes these results to architectural simplification rather than auxiliary temporal complexity.
- 5 Conclusion: Retrieval-based Transformer decoding decouples semantic identity from spatial location, allowing slots computed from one frame to decompose subsequent video frames.This addresses the spatial entanglement inherent in context-free decoders.
- 5 Conclusion: The paper reports no ethical issues.This statement appears in the ethical statement.
- 5 Conclusion: Qualitative comparisons show more stable identity assignments, better boundary precision, and less background over-clustering noise than transition-based recurrent baselines.Figure 6 compares input frames, ground-truth masks, SlotContrast, RandSF.Q, and SemanticSlots across diverse visual domains.