Source-linked AI summary
Conditional Object-Centric Learning from Video
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, Klaus Greff
TL;DR
Unsupervised object-centric methods struggle to scale from toy scenes to realistic data and offer limited control over task-dependent object granularity. The paper introduces SAVi, a sequential Slot Attention model trained with optical flow and conditioned on simple first-frame location cues. It reports improved multi-object segmentation and tracking, generalization to new objects, backgrounds, and longer sequences, and inference-time control over object granularity.
Problem
Unsupervised object-centric methods remain limited on realistic data, while their decompositions may not provide the task-dependent object granularity needed for training and inference.
Method
SAVi sequentially predicts and corrects slot representations for video, using optical-flow prediction and simple first-frame conditioning such as object center-of-mass cues.
Results
SAVi successfully segments and tracks complex synthetic videos, with benefits that generalize to novel objects, novel backgrounds, and longer sequences; conditional SAVi reaches 72.0% mIoU on MOVi.
Takeaways & Limitations
Conditioning the initial slot state can steer decomposition toward a requested granularity and provide an inference interface for tracking objects or object parts.
Takeaways & Limitations
The framework assumes optical-flow information during training and is evaluated on environments containing rigid objects with simple physics, including moving objects in the MOVi datasets.
Abstract
from arXiv · showhide
Object-centric representations are a promising path toward more systematic generalization by providing flexible abstractions upon which compositional world models can be built. Recent work on simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision. However, such fully-unsupervised methods still fail to scale to diverse realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene. In this paper, we instead take a weakly-supervised approach and focus on how 1) using the temporal dynamics of video data in the form of optical flow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in significantly more realistic synthetic data. We introduce a sequential extension to Slot Attention which we train to predict optical flow for realistic looking synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the first frame, is sufficient to significantly improve instance segmentation. These benefits generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences. We also find that such initial-state-conditioning can be used during inference as a flexible interface to query the model for specific objects or parts of objects, which could pave the way for a range of weakly-supervised approaches and allow more effective interaction with trained models.
1 INTRODUCTION
Object-centric representations offer compositional abstractions for generalization, but unsupervised methods struggle with realistic visual data and ambiguous object granularity. The paper addresses these limitations using video dynamics and simple location cues to guide segmentation and tracking.
- Object-centric representations provide compositional building blocks that may improve generalization, sample efficiency, robustness, and interpretability.
- Unsupervised object-centric methods learn object representations from statistical structure without supervision, but remain restricted mainly to toy data.
- Realistic scenes create a second challenge because learned segmentations may not match the task-dependent object granularity users need.
- SAVi combines optical-flow prediction with first-frame conditioning, such as object center-of-mass cues, to guide multi-object segmentation and tracking.The model maintains latent slots and can condition their initial state on simple object cues.
- The approach generalizes to novel objects, novel backgrounds, and longer sequences, while remaining robust to noise in optical flow and conditioning.
2 SLOT ATTENTION FOR VIDEO (SAVI)
SAVi extends Slot Attention sequentially for video by predicting temporal dynamics between frames and correcting slot representations using visual features. It decodes slot-wise optical flow or frame reconstructions while supporting conditional initialization and permutation-equivariant processing.
- SAVi performs prediction and correction at each video frame, using self-attention for dynamics and Slot Attention to update slots from visual features.The predictor initializes the corrector at the next time step, supporting consistent object tracking.
- Slot Initialization: SAVi maintains K permutation-equivariant slots that can represent objects or object parts, initialized conditionally or unconditionally.Bounding boxes and center-of-mass coordinates use an MLP conditioner, while segmentation masks use a CNN.
- Corrector: The corrector uses slot-normalized iterative attention to decompose encoder features across multiple slots before recurrent slot updates.Unlike standard cross-attention, normalization over slots encourages decomposition into multiple components.
- Predictor: The predictor models independent object dynamics and slot interactions with a permutation-equivariant Transformer encoder.
- Decoder and Training: A slot-wise Spatial Broadcast Decoder produces per-slot optical-flow or RGB predictions and alpha masks that combine into an output frame.The training target is optical flow, with pixel-wise squared reconstruction error also supported for image-shaped targets.
3 RELATED WORK
Related work spans unsupervised object-centric learning for images and videos, motion- and geometry-based approaches, and video object segmentation. SAVi differs by combining multi-object visual processing with optical flow as a training signal and conditional interaction.
- Prior object-centric methods learn representations from static scenes or videos without explicit supervision, including sequential models with internal motion representations.
- Recent approaches incorporate 3D scene-geometry inductive biases to address visually richer environments, whereas SAVi instead uses conditioning and optical flow.
- The conditional tasks relate to semi-supervised video object segmentation, but SAVi uses simpler cues than methods requiring first-frame segmentation masks.
- Motion-based computer-vision methods segment objects using test-time motion, while SAVi uses motion information for training and operates on textured visual inputs.This allows SAVi to support multi-object environments and static-scene segmentation when optical flow is unavailable at test time.
- MOVi++ provides a substantially more complex evaluation setting through real-world backgrounds and 3D-scanned objects.
4 EXPERIMENTS
Experiments evaluate SAVi on progressively more realistic video decomposition settings, testing temporal objectives, conditional hints, robustness, generalization, and limitations. Results show that optical-flow training with simple first-frame cues supports segmentation and tracking beyond the training distribution, while heavy occlusion and real-world complexity remain challenging.
- Experimental setup: SAVi is evaluated against unsupervised video decomposition methods using ARI and mIoU, with training on six-frame subsequences and conditioning supplied in the first frame.The evaluation includes CATER, MOVi, and MOVi++, with mean ± standard error reported across five seeds where specified.
- More realistic datasets: 22.7 ± 0.9% FG-ARI is achieved by SCALOR on MOVi++, compared with 81.2 ± 0.4% on MOVi, highlighting the difficulty of realistic visual complexity.SIMONe reaches around 33% FG-ARI on MOVi++, while the evaluated models can converge to fixed image regions rather than individual objects.
- Conditional video decomposition: 72.0% mIoU is achieved by SAVi on MOVi, while its 43.0% mIoU on MOVi++ is slightly below T-VOS and CRW under the reported comparison.The authors note that T-VOS and CRW use stronger or higher-resolution backbones, making the direct comparison not entirely fair.
- Conditioning ablations: Bounding boxes or center-of-mass points in the first frame preserve segmentation and tracking quality, whereas removing the conditioning signal causes a significant performance drop.These hints establish correspondence between the conditioning input and scene objects, and rough hints are sufficient for the model.
- Generalization: SAVi maintains largely stable FG-ARI on longer test sequences and shows no significant FG-ARI or mIoU difference for unseen objects or backgrounds, with less than 2% absolute drop when both are new.A model trained on MOVi++ also transfers well to MOVi at test time.
- Test-time generalization: Conditioning at whole-object or part-level granularity lets SAVi segment and track either a composite object or its individual parts, but heavy occlusion can cause tracking loss and identity swaps.The part-whole result is reported as anecdotal evidence, while the occlusion failure appears in challenging qualitative examples.
5 CONCLUSION
The paper shows that simple first-frame location information can steer SAVi toward the desired object granularity while tracking and segmenting multiple video entities. Its analysis remains focused on physically simulated household-object videos, with real-world applications left to future work.
- SAVi discovers, represents, and temporally tracks individual entities using attention over a latent set of slots.The architecture is presented as an object-centric model for learning representations and physical dynamics from video.
- Simple, possibly unreliable first-frame position information is sufficient to steer decomposition toward the right granularity for segmenting and tracking multiple objects.The same initialization can guide the model toward parts of objects or entire objects during inference.
- The analysis focuses on multi-object videos containing physically simulated common household objects.
A APPENDIX
The appendix supplies supplementary material covering related work, additional results, ablations, datasets, architecture, and training details.
- Section A.1 discusses additional related work.
- Sections A.2–A.4 present additional qualitative and quantitative results, including an SAVi ablation study in Section A.4.
- Sections A.5–A.8 report dataset, model-architecture, and training details, including hyperparameters, baselines, and metrics.
A.1 ADDITIONAL RELATED WORK
The related-work appendix situates SAVi among attention-based modular networks, supervised slot-based tracking and segmentation models, and object-centric approaches to dynamic visual reasoning.
- Attention-based networks for sets of latent variables: SAVi shares attention-based modular-network connections with Relational Recurrent Networks and RIM, which map inputs or slots to latent slots.
- Supervised slot-based models for visual tracking: Supervised slot-based models such as MOHART, TrackFormer, and TubeR address multi-object tracking and segmentation, while GPV-I and MDETR condition slot-based models on auxiliary information.
- Dynamic visual reasoning and action graphs: Object-centric models have also been used for supervised visual reasoning, activity recognition, and compositional video synthesis with action graphs.
A.2 REAL-WORLD ROBOTICS TASK
A qualitative Sketchy evaluation tests unsupervised SAVi on real-world robotic-grasping videos. The model decomposes scenes into meaningful components and tracks them over long horizons beyond the training duration.
- The Sketchy dataset contains videos of a real-world robotic grasper interacting with various objects.
- The evaluation uses 2930 training videos of 201 frames from the human-demonstration “rgb30 all” subset.
- SAVi qualitatively decomposes unseen real-world scenes into meaningful object components and consistently tracks individual components over long time horizons.The reported horizons extend far beyond what is observed during training.
- The model is trained with RGB reconstruction using an MLP predictor with one hidden layer of 256 units for 1M steps.
- Figure A.1 visualizes predicted scene segmentation and individual slot reconstructions multiplied by their predicted soft segmentation masks.Three discovered background slots are colored black for easier interpretation.
A.3 ADDITIONAL QUALITATIVE RESULTS
The additional qualitative results show that conditioning can steer SAVi toward whole-object or part-level segmentation and tracking, while evaluations examine longer sequences, moving cameras, and baseline behavior across MOVi variants.
- Part-whole segmentation: Bounding-box granularity steers SAVi between segmenting a laptop as two parts and segmenting it as one whole object.The model identifies and tracks separate screen and base parts despite not having seen separate screen objects during training.
- Part-whole segmentation: Corrector attention masks outline objects more sharply than decoder masks but show artifacts on large objects.The authors associate these artifacts with the encoder CNN’s limited receptive field.
- Extrapolation to long sequences: SAVi with bounding-box conditioning is evaluated on full-length 24-frame MOVi and MOVi++ videos after training on six-frame sequences.These figures present qualitative extrapolation to longer sequences than those used for training.
- Moving camera: With a moving camera, SAVi achieves approximately 65.5 ± 0.5% FG-ARI while still segmenting and tracking objects at lower fidelity.Camera motion introduces background motion into optical flow and weakens its usefulness for foreground-background segmentation.
- Baselines: The Segmentation Propagation baseline struggles to retain small objects and separate the background, while SCALOR merges nearby small objects and captures background texture as extra objects.These qualitative comparisons cover MOVi and MOVi++ examples.
A.4 ADDITIONAL QUANTITATIVE RESULTS
Additional experiments show that SAVi can segment static scenes, track selected object subsets, generalize across objects and backgrounds, and retain useful semantic information. Architectural ablations further show that slot interactions support hint–object correspondence, while frame reconstruction preserves more appearance information than flow prediction alone.
- Transfer to static scenes: 88.7 ± 0.2 FG-ARI demonstrates accurate segmentation of static MOVi++ scenes when the first frame is repeated.The evaluation uses SAVi + Bounding box models on repeated first frames, removing motion while retaining the scene.
- Tracking subsets of objects: SAVi can track selected object subsets, but mIoU decreases as more unconditioned objects must be represented by randomly initialized slots.The experiments evaluate bounding-box and segmentation-mask conditioning across varying subset sizes.
- Tracking subsets of objects: Conditioning on up to 6 objects provides a trade-off between weak hint–object correspondence and poor subset generalization from conditioning every training object.Training on subsets of sizes 1, 3, 6, and 10 exposes the tension between too few and too many conditioned objects.
- Generalization to unseen objects and backgrounds: SAVi generalizes without decreased FG-ARI or mIoU on unseen objects, remains comparable on unseen backgrounds, and suffers only a slight decrease with both novel.The reported splits contain approximately 100 unseen objects and 40 unseen backgrounds; a MOVi++ model also transfers to MOVi.
- Architecture ablations: Removing or weakening slot-interaction mechanisms harms hint–object correspondence, especially when attention is normalized over visual features rather than slots.The no-predictor and MLP-predictor variants negatively affect correspondence-related mIoU, while inverted corrector attention causes a sharp drop.
A.5 DATASET DETAILS
The appendix documents the licensing and content properties of the datasets and their source materials. MOVi++ uses publicly available CC-0 HDR backgrounds, while CATER materials are distributed under Apache 2.0 or supplied by the SIMONe authors.
- MOVi++: MOVi++ contains approximately 380 publicly available CC-0 HDR backgrounds and no personally identifiable or offensive content.The Kubric dataset-generation pipeline is publicly available under an Apache 2.0 license.
- CATER: The original CATER dataset is Apache 2.0 licensed, while its segmentation-mask variant was provided by the SIMONe authors.The original CATER release does not include segmentation-mask annotations.
A.6 ARCHITECTURE DETAILS AND HYPERPARAMETERS
SAVi shares most parameters across time and uses convolutional encoders, Slot Attention correction, self-attention prediction, and slot-specific decoders. Its initializer converts bounding boxes, center-of-mass coordinates, or masks into initial slot representations, with dataset-specific training schedules and resolutions.
- Parameter sharing: SAVi shares module parameters across time, except for the initializer, which is applied only at the first time step.This design makes the initial conditioning signal temporally localized.
- Encoder: The encoder uses four-direction linear position embeddings projected to the CNN feature-map size, with a larger stride and channel count for 128 × 128 MOVi++ inputs.The encoder specifications are summarized separately for 64 × 64 and 128 × 128 inputs.
- Corrector and predictor: The corrector uses Slot Attention followed by a GRU, while the predictor uses multi-head self-attention with a 128-dimensional projection and four heads.The corrector uses one attention iteration on MOVi and MOVi++ and two on CATER; the predictor MLP has 256 hidden units.
- Decoder: Each slot is spatially broadcast, augmented with position embeddings, and decoded by a CNN to the input-frame resolution.The decoder uses an 8 × 8 broadcast grid before producing full-resolution representations.
- Initializer: Bounding-box and center-of-mass coordinates are encoded independently through a shared MLP, while segmentation masks use independent CNN encoders for conditioned slots.Unconditioned slots receive zero-filled masks in the mask-conditioning pathway.
A.7 BASELINE DETAILS
The appendix describes the comparison baselines and the implementation choices used to make their evaluations comparable with SAVi. These include reconstruction-based, propagation-based, contrastive, and flow-pretrained segmentation systems.
- SCALOR: SCALOR proposes objects on image-grid cells, accepts or rejects them using overlap with propagated objects, and reconstructs frames with object and background decoders.Its background decoder includes an encoding bottleneck dimension.
- SIMONe: SIMONe encodes video frames with CNNs and a transformer, pools per-frame and per-object embeddings, and decodes pixels using sampled latent variables.Unlike the other considered video baselines, SIMONe processes all frames in parallel and cannot generalize beyond its fixed training clip length.
- CRW: CRW learns a contrastive loss between neighboring-frame patch embeddings and supports downstream tracking through label propagation.The authors use the released implementation and evaluate tracking with Davis 2017 utilities.
- T-VOS: T-VOS propagates first-frame segmentation labels by correlating target-frame pixels with features from previous reference frames.The simplified comparison uses up to 9 immediate past frames and omits sparse sampling.
- T-VOS implementation: For a fairer T-VOS comparison, the authors match SAVi’s optical-flow loss and visual backbone dimensionality, adding only a 5 × 5 projection convolution.Evaluation uses specified motion-prior parameters and L2-normalized per-pixel embeddings.
- Evaluation details: FG-ARI is evaluated on all frames for CATER, while most FG-ARI and mIoU evaluations omit the first conditioned frame.mIoU requires strict slot-to-object alignment and therefore is sensitive to whether conditioned slots track their intended objects.