PaperlayerView exact revision on arXiv

Source-linked AI summary

SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models

Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, Animesh Garg

arXiv:2210.05861v2cs.CVcs.AIcs.LG

TL;DR

Visual dynamics learning needs object-level representations and effective interaction modeling, but existing approaches remain limited for complex, long-horizon behavior. SlotFormer uses an unsupervised object-centric representation with an autoregressive Transformer to predict future object states and support downstream reasoning. It achieves strong video prediction, VQA, and planning results, while remaining limited by pretrained object-centric models, two-stage training, and deterministic future modeling.

  • Problem

    Effective domain-independent unsupervised visual dynamics learning remains difficult because models must capture objects, motion, and their interactions.

  • Method

    SlotFormer is an autoregressive Transformer that performs joint spatiotemporal reasoning over object-centric features from multiple video frames.

  • Results

    SlotFormer achieves state-of-the-art results in video prediction and VQA, with competitive performance in goal-conditioned planning.

  • Takeaways & Limitations

    Unsupervised dynamics knowledge can support long-term visual simulation and downstream future reasoning without object-level labels.

  • Takeaways & Limitations

    SlotFormer depends on pretrained object-centric models that do not yet scale to real-world data, and its deterministic model cannot represent uncertainty in future dynamics.

Abstract

from arXiv · show

Understanding dynamics from visual observations is a challenging problem that requires disentangling individual objects from the scene and learning their interactions. While recent object-centric models can successfully decompose a scene into objects, modeling their dynamics effectively still remains a challenge. We address this problem by introducing SlotFormer -- a Transformer-based autoregressive model operating on learned object-centric representations. Given a video clip, our approach reasons over object features to model spatio-temporal relationships and predicts accurate future object states. In this paper, we successfully apply SlotFormer to perform video prediction on datasets with complex object interactions. Moreover, the unsupervised SlotFormer's dynamics model can be used to improve the performance on supervised downstream tasks, such as Visual Question Answering (VQA), and goal-conditioned planning. Compared to past works on dynamics modeling, our method achieves significantly better long-term synthesis of object dynamics, while retaining high quality visual generation. Besides, SlotFormer enables VQA models to reason about the future without object-level labels, even outperforming counterparts that use ground-truth annotations. Finally, we show its ability to serve as a world model for model-based planning, which is competitive with methods designed specifically for such tasks.

1 INTRODUCTION

Unsupervised visual dynamics learning must capture objects, their motion, and interactions, yet effective domain-independent approaches remain elusive. SlotFormer addresses this gap with a Transformer operating on object-centric features and reports strong results across video prediction and downstream reasoning tasks.

  • Visual dynamics learning requires capturing object instances, appearance, position, motion, and spatiotemporal interactions from video.
  • Pixel-space methods use object-agnostic global representations, often producing unrealistic long-horizon predictions because foreground and background receive equal treatment.
  • SlotFormer combines unsupervised object-centric representations with a Transformer dynamics module that predicts future object features from multiple input frames.
  • The model jointly captures spatiotemporal object relationships while preserving consistency of object properties and motion in synthesized frames.
  • SlotFormer achieves state-of-the-art video prediction on two datasets, gains in long-term dynamics modeling, state-of-the-art VQA results, and competitive planning results.

2 RELATED WORK

Prior visual dynamics methods often struggle with long-horizon prediction or rely on restrictive object assumptions and separate interaction modules. SlotFormer instead performs joint spatiotemporal reasoning over object slots from multiple frames.

  • Global image-level dynamics models commonly struggle with long-horizon predictions, while local priors or extra inputs mainly help in the short term.
  • Earlier object-centric video methods often use strong representation assumptions or separate modules for object interactions and temporal dynamics.
  • Some prior recurrent dynamics models use only a single timestep as context, limiting their temporal view of scene dynamics.
  • OCVT applies Transformers to slots from multiple frames but relies on manually disentangled features and Hungarian matching for latent alignment.
  • SlotFormer jointly reasons over object slots across space and time, enabling consistent long-term dynamics modeling while remaining agnostic to the underlying object-centric representation.

3 SLOTFORMER: OBJECT-ORIENTED DYNAMICS LEARNING

SlotFormer extracts temporally aligned object slots, processes them with a Transformer, and autoregressively predicts future slots. Its training procedure addresses long-horizon error accumulation by simulating test-time rollout with predicted slots.

  • Pipeline: The pipeline extracts object slots from multiple frames, performs joint Transformer reasoning, and predicts future slots autoregressively.
  • Object-centric representations: Slot Attention binds slots to objects through iterative attention, while transition-based initialization promotes temporal alignment across frames.
  • Object-centric representations: The object-centric model is pretrained with reconstruction loss so slots capture foreground objects and the background environment.
  • Dynamics architecture: Temporal positional encoding is applied identically to slots at each timestep, preserving permutation equivariance among slots.
  • Autoregressive prediction: The Transformer uses the final timestep's output slots to predict the next timestep, then feeds predictions back for arbitrary-length autoregressive rollout.
  • Training: Training on predicted slots reduces train-test discrepancy by simulating autoregressive generation and improves long-term generation quality.

4 EXPERIMENTS

SlotFormer is evaluated as an unsupervised object-centric dynamics model for video prediction, VQA, and action planning. It improves long-term visual and object-dynamics modeling and supports downstream reasoning, while ablations identify important design choices and limits.

  • Experimental setup: SlotFormer evaluates object-centric dynamics across video prediction, VQA, and action planning on OBJ3D, CLEVRER, Physion, and PHYRE.The experiments test both generated future states and their usefulness for downstream reasoning and planning.
  • Video prediction: SlotFormer outperforms video-prediction baselines in LPIPS and maintains consistent object attributes over long rollouts, despite competitive rather than best PSNR and SSIM.The authors attribute this advantage to modeling dynamics in object-centric space instead of directly in frames.
  • Video prediction: SlotFormer achieves the best reported AR, ARI, and FG-mIoU results for object dynamics on CLEVRER, with competitive FG-ARI.Predicted bounding boxes and segmentation masks are evaluated from future model predictions.
  • Visual question answering: 3.4% and 6.0% improvements from SlotFormer dynamics boost Aloe on CLEVRER predictive VQA in per-option and per-question accuracy, respectively.The unsupervised dynamics model outperforms supervised-detector-based DCL and VRDP on the reported comparison and ranks first or second on the public leaderboard settings.
  • Visual question answering: 1.9% improvement on Physion makes learned dynamics surpass all baselines in the reported readout accuracy.The comparison evaluates observation frames against observation-plus-rollout frames.
  • Action planning: 82.0 AUCCESS improves SAVi without rollout by 1.3 and is on par with methods using ground-truth object information.SlotFormer is used as a world model whose predicted object states support action ranking in PHYRE.
  • Ablation study: A history length of 6 is sufficient for accurate OBJ3D dynamics, while Physion accuracy improves with longer burn-in; 10 rollout frames balance accuracy and computation.Increasing the Transformer depth helps Physion but can overfit OBJ3D, and 12 layers make training unstable.
  • Ablation study: Permutation-equivariant slot modeling, autoregressive error accumulation, and image reconstruction loss each materially affect performance across visual quality, VQA, and long-term dynamics.Sinusoidal positional encoding degrades VQA, teacher forcing harms results, and image reconstruction preserves object attributes while adding memory cost.

5 CONCLUSION

SlotFormer provides consistent long-term dynamics modeling with object-centric representations and transfers unsupervised dynamics knowledge to VQA and goal-conditioned planning.

  • SlotFormer is a Transformer-based autoregressive model for consistent long-term dynamics modeling with object-centric representations.
  • SlotFormer models complex spatio-temporal object interactions and generates future predictions with high visual quality.
  • Unsupervised SlotFormer dynamics knowledge supports state-of-the-art or comparable results on VQA and goal-conditioned planning.

REPRODUCIBILITY STATEMENT

The paper situates its approach among object-centric dynamics models, physical-reasoning methods, and Transformer-based autoregressive generation. It also reports an implementation intended to support reproducibility and accessibility.

  • Reproducibility: The methods are implemented in PyTorch and can be trained on servers with four modern GPUs in less than five days.The authors state that dataset processing, model hyperparameters, and baseline sources are documented in appendices.
  • Physical reasoning: Physical-reasoning approaches increasingly infer physical laws implicitly by modeling interactions between objects in a scene.Earlier methods instead explicitly encode physical laws and estimate latent variables from inputs.
  • Object-centric dynamics: Prior object-centric models extract object features from pixels and model interactions, but their approaches differ in background modeling and scene dynamics.R-NEM uses a graph neural network for interactions, while SCALOR extends SQAIR to multiple moving objects.
  • Transformers: SlotFormer is motivated by Transformer-based autoregressive methods for image and video generation.The related work connects its design to the broader adoption of Transformers in computer vision.

B DATASET DETAILS

The evaluation uses datasets spanning collisions, object entry, varied physical phenomena, visual appearances, and planning tasks. Dataset-specific preprocessing and evaluation protocols address their differing dynamics and task structures.

  • OBJ3D: OBJ3D contains 3 to 5 static objects followed by a launched sphere, with larger, more visible objects and fewer collisions and occlusions than CLEVRER.All objects are visible at the beginning of the videos.
  • CLEVRER: CLEVRER contains objects that may enter from random directions during videos, making it more challenging than OBJ3D because of smaller objects and more diverse interactions.Ground-truth segmentation masks are used to generate bounding boxes and calculate Average Recall at a 50% IoU threshold.
  • VQA: CLEVRER VQA contains descriptive, explanatory, predictive, and counterfactual questions, with the latter three evaluated as multiple-choice classification.The questions are generated by template-based programs.
  • Physion: Physion covers eight physical scenarios, including collisions, falling, rolling, and sliding, with varied object appearance, backgrounds, and camera poses.Its videos are truncated to 150 frames and subsampled by a factor of three for dynamics-model training.
  • PHYRE: PHYRE-B evaluates task success under within-template and cross-template train-test settings using videos simulated at 1 FPS.The benchmark contains 25 task templates with similar initial object configurations within each template.
  • Preprocessing caveat: Light-colored objects are difficult for SAVi to detect because their pixels receive small optimization gradients when they resemble the white background.The authors address this with a simple background-color workaround rather than improving the object-centric model.
  • Planning evaluation: PHYRE evaluation scores 10,000 predefined actions by rendering a red ball in the current task and ranking the actions by predicted success.The evaluation metric is AUCCESS, a weighted sum of Area Under Curve.
  • Object discovery: Vanilla SAVi can miss newly entering objects because empty slots attend jointly to the first new object, leaving no free slot for subsequent objects.The figure illustrates this failure when objects enter a CLEVRER scene over time.

C IMPLEMENTATION DETAILS

The implementation combines SAVi-based slot extraction, Transformer dynamics, and task-specific readout models designed around object-slot properties. Training and evaluation choices address slot symmetry, permutation equivariance, and long-horizon error accumulation.

  • Baselines: Vanilla SAVi uses a CNN encoder, decoder, Slot Attention corrector, and Transformer predictor, with 128-dimensional slots and six-frame training clips.On PHYRE, the decoder spatial broadcast size is reduced to 16 × 16 to capture small objects.
  • Stochastic SAVi: Stochastic SAVi adds randomness to slot transitions because similar empty slots can share newly entering objects and miss later objects.A two-layer MLP predicts the mean and log variance of the next slot distribution, and a KL loss enforces stochasticity.
  • Stochastic SAVi: The stochastic SAVi model uses a prior standard deviation of 0.1 and weights the KL divergence loss by 1 × 10^-4 alongside frame reconstruction loss.The chosen randomness is intended to break slot symmetry without destroying temporal alignment.
  • Transformer: The Transformer uses eight self-attention heads, Pre-LN encoder blocks, Adam optimization, a 2 × 10^-4 initial learning rate, cosine decay, and linear warmup.The feed-forward hidden size is 4 × D_e.
  • VQA readout: The CLEVRER VQA model concatenates object slots and question tokens before processing them with a stack of NAloe Transformer encoder layers.With SlotFormer, predicted future slots are unrolled and supplied to Aloe for predictive questions.
  • Physion readout: The Physion readout uses max pooling because it is invariant to slot permutations, whereas concatenating slots makes predictions depend on their input order.This preserves compatibility with object-centric representations whose slot ordering is arbitrary.
  • Physion readout: Readout models are fitted only to the first 75 of 150 rollout frames because later prediction errors introduce artifacts that cause severe overfitting.Baselines use the same readout training setting for fair comparison.
  • PHYRE classifier: The PHYRE task-success classifier concatenates predicted slots with a learnable CLS token, adds temporal positional encoding, and predicts scores with a two-layer MLP.Using the CLS representation makes predicted scores invariant to slot order.

E.1 QUALITATIVE RESULTS

SlotFormer produces sharp, object-consistent trajectories on OBJ3D and more challenging CLEVRER scenes, while supporting downstream VQA and planning evaluation. Its qualitative results also expose decoder-quality limitations in Physion.

  • SlotFormer achieves excellent object trajectories and very low LPIPS on OBJ3D.
  • The supplied qualitative-results tables cover VQA accuracy, planning AUCCESS, and visual-quality evaluation across the reported datasets.
  • On CLEVRER, SlotFormer preserves accurate object detection and dynamics despite smaller objects and longer-term interactions.The comparison uses object-aware AR and mIoU metrics.
  • SlotFormer’s Physion VQA visualization synthesizes contact between the red object and yellow ground.
  • The Physion predictions remain visually low quality because STEVE’s decoder is not designed for pixel-space reconstruction.The paper states that improving this decoder is outside its scope.

E.2 QUANTITATIVE RESULTS

Quantitative evaluation reports strong long-horizon object-dynamics performance and downstream-task results for SlotFormer. The evaluation includes CLEVRER segmentation metrics, VQA accuracy, and PHYRE planning AUCCESS.

  • Object-dynamics evaluation on CLEVRER reports percentage-valued results in Table 15.
  • SlotFormer preserves object identity and achieves high FG-ARI and FG-mIoU over long rollout steps on CLEVRER.SAVi-dyn produces blurry objects and false-positive segmentation masks, while G-SWM does not scale well to complex multi-object dynamics.
  • The quantitative evaluation reports CLEVRER VQA results and Physion per-scenario accuracy for rollout-based settings.
  • The PHYRE planning evaluation reports AUCCESS for all 10 folds.

E.3 COMPARISON WITH ADDITIONAL BASELINES

Against additional baselines, SlotFormer benefits from temporally aligned SAVi slots and joint spatial-temporal reasoning over a longer context window. These design choices improve both video quality and object-dynamics accuracy.

  • The comparison evaluates OCVT and Slot-LSTM on video prediction using OBJ3D and CLEVRER.
  • OCVT underperforms SlotFormer in visual video quality and object-dynamics accuracy on OBJ3D and CLEVRER.The paper attributes this partly to SAVi’s stronger slots and SlotFormer’s naturally temporally aligned representations.
  • SlotFormer outperforms Slot-LSTM on all reported metrics, especially on the more challenging CLEVRER dataset.
  • Joint spatial-temporal reasoning over a larger context window distinguishes SlotFormer from Slot-LSTM’s single-timestep recurrent context.
  • Slot-LSTM generates blurry objects and inconsistent long-horizon dynamics because its recurrent module sees only one timestep.

E.4 ATTENTION ANALYSIS

Attention analysis shows SlotFormer uses object interactions, earlier visible frames, and burn-in context to predict future slots. More burn-in generally improves downstream context, though OBJ3D LPIPS peaks at T = 6.

  • SlotFormer derives spatial attention maps from the last Transformer self-attention layer while predicting future slots from encoded frames.
  • On OBJ3D, attention links collision history and imminent interactions to the future motion of specific objects.The model attends to earlier frames when occlusion hides an object and uses other visible objects to predict its motion.
  • On CLEVRER, attention maps capture two collision events, although smaller objects make the maps less clear.
  • On OBJ3D, LPIPS improves with more burn-in frames and then degrades after peaking at T = 6.
  • On Physion, rollout accuracy increases consistently with more burn-in frames because additional frames provide more context.
  • SAVi detects objects entering during burn-in, allowing SlotFormer to simulate their future dynamics from frames where they become visible.

F LIMITATIONS AND FUTURE WORKS

SlotFormer’s current scope is limited by its reliance on pretrained object-centric models, two-stage training, deterministic prediction, and unconditional future generation. The authors propose joint training and conditional-generation extensions as directions for future work.

  • Limitations: Pretrained object-centric models do not yet scale to real-world data, preventing SlotFormer’s application to real-world videos.
  • Limitations: The two-stage training strategy harms performance during early rollout steps.
  • Limitations: Because SlotFormer is deterministic, it cannot model uncertainty in future dynamics common in real-world videos.
  • Future Works: Jointly training scene decomposition and the temporal dynamics module is proposed as a way to simplify training and potentially improve both models.
  • Future Works: Future work will extend SlotFormer from unconditional prediction to conditional generation, including action-conditioned generation.
Loading 2210.05861v2…