Source-linked AI summary
Unsupervised Learning of Object Structure and Dynamics from Videos
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, Honglak Lee
TL;DR
The paper addresses unsupervised learning of object structure and dynamics from videos, where pixel prediction and unstructured representations are limited for motion-related understanding. It learns keypoint representations and stochastic keypoint dynamics, reconstructing future frames with a reference image. The resulting spatially structured representation improves video prediction and downstream object-level tasks, while leaving some phenomena and task integrations outside the model’s explicit scope.
Problem
Unsupervised video models must extract useful object structure and dynamics without costly annotations, while pixel-level prediction and reconstruction representations may not support motion-related downstream tasks.
Method
The model learns an explicit keypoint-based representation from video and a stochastic dynamics model over keypoint coordinates, reconstructing frames from keypoints and a reference frame.
Results
The spatially structured representation outperforms unstructured representations in pixel-level video prediction and downstream tasks requiring object-level understanding, including tracking, action recognition, and reward prediction.
Takeaways & Limitations
Keypoint-based spatial structure provides a middle ground between unstructured representations and explicitly object-centric models while supporting stable training across diverse datasets.
Takeaways & Limitations
The model does not explicitly represent object masks, occlusions, or depth, and applying it to planning and reinforcement-learning control tasks remains future work.
Abstract
from arXiv · showhide
Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning. To address this challenge, we adopt a keypoint-based image representation and learn a stochastic dynamics model of the keypoints. Future frames are reconstructed from the keypoints and a reference frame. By modeling dynamics in the keypoint coordinate space, we achieve stable learning and avoid compounding of errors in pixel space. Our method improves upon unstructured representations both for pixel-level video prediction and for downstream tasks requiring object-level understanding of motion dynamics. We evaluate our model on diverse datasets: a multi-agent sports dataset, the Human3.6M dataset, and datasets based on continuous control tasks from the DeepMind Control Suite. The spatially structured representation outperforms unstructured representations on a range of motion-related tasks such as object tracking, action recognition and reward prediction.
1 Introduction
The paper targets unsupervised extraction of object structure and dynamics from videos, addressing limitations of pixel-level prediction and unstructured representations. It uses keypoint-based structure with stochastic dynamics to improve prediction and object-level downstream tasks.
- Unsupervised video understanding remains challenging because extracting object structure and dynamics typically requires expensive human annotations.
- Pixel-level video prediction becomes difficult beyond about one second because stochastic motion makes small semantic deviations produce large pixel errors.
- Pixel-reconstruction representations are not guaranteed to support downstream tasks such as tracking, motion prediction, and control.
- The model learns an explicit keypoint representation from video, then models the dynamics of those keypoints without supervision beyond pixel data.
- Modeling dynamics in keypoint coordinates avoids compounding pixel-space errors and makes prediction errors more semantically meaningful.
- Spatial structure improves video prediction and downstream performance, including trajectory prediction, action recognition, and reward prediction.
2 Related work
Prior work learns keypoints, stochastic video predictions, and spatially structured representations, but this paper combines these directions for unsupervised video prediction. It compares against unstructured and adversarially trained approaches while retaining a keypoint-based representation.
- Unsupervised keypoint methods use an autoencoding bottleneck that represents images with a small number of keypoint coordinates.
- Stochastic sequence prediction methods model uncertainty; this paper adopts a VRNN to sample possible video futures.
- Compared with SVG, the paper replaces an unstructured image representation with structured keypoints for video prediction and downstream tasks.
- EPVA predicts dynamics in a high-level feature space with adversarial loss, whereas this method reports improvement without adversarial training.
- Other spatially structured approaches use object parts, segmentation masks, edge masks, or human pose as representations for video prediction.
3 Architecture
The architecture encodes frames as keypoints, reconstructs them with a reference frame, and predicts their stochastic dynamics using a VRNN. Future frames can therefore be generated in keypoint space without observing future images.
- The model combines a keypoint detector that encodes frames with a dynamics model that predicts in keypoint space.
- 3.1 Unsupervised keypoint detector: The detector produces K feature maps and converts each normalized map into one spatial (x, y) coordinate by expectation.
- 3.1 Unsupervised keypoint detector: A convolutional generator reconstructs frame v_t from its keypoints and reference frame v_1, creating a bottleneck for frame structure.
- 3.1 Unsupervised keypoint detector: The detector’s mean feature-map intensity indicates whether each modeled object is present in a frame.
- 3.2 Stochastic dynamics model: The stochastic dynamics model uses a VRNN whose prior predicts current keypoint locations from the previous recurrent hidden state.
- 3.2 Stochastic dynamics model: During future prediction, decoded keypoints replace detector outputs, allowing the recurrence to continue without observed images while representing long-term dependencies and uncertainty.
4 Training
Training combines image reconstruction with sparsity and temporal-separation losses for the keypoint detector, while a VRNN dynamics model uses KL regularization and multistep reconstruction. The design prevents dynamics errors from reshaping the keypoint representation and uses best-of-many prediction in low-dimensional coordinate space.
- Keypoint detector: The keypoint detector is trained with an L2 image reconstruction loss, and dynamics-model errors are not backpropagated into it.This preserves a keypoint-structured representation instead of allowing an incompletely trained dynamics model to impose a poorly structured code.
- Keypoint detector: Temporal separation penalizes overlap between keypoint trajectories, encouraging correlated image features to be represented jointly by a single keypoint.Trajectory distances are computed after subtracting each trajectory’s temporal mean, within a Gaussian radius σsep.
- Keypoint detector: An L1 sparsity penalty on keypoint scales encourages keypoints to be sparsely active.The separation and sparsity losses are reported to contribute to stable keypoint detection.
- Dynamics model: The VRNN dynamics model maximizes an evidence lower bound combining reconstruction and KL terms, with β balancing reconstruction fidelity against prediction diversity.The KL term regularizes the latent representation and trains the recurrent model to predict the posterior from past information.
- Dynamics model: A pure multistep reconstruction loss without the KL term encourages the dynamics model to learn long-term dependencies.The standard KL-based objective trains the dynamics model for single-step predictions because observations are supplied after each step.
- Dynamics model: Best-of-many training samples multiple predictions at each timestep, using low-dimensional keypoint distances to evaluate quality without reconstructing pixels.This encourages coverage of likely data modes while avoiding the expense of evaluating several samples in pixel space.
5 Results
Across video prediction and downstream evaluations, the structured stochastic model improves perceptual quality, object tracking, controllability, and action recognition over unstructured representations. Its learned keypoints provide useful spatial information without supervision and support stable, interpretable manipulation.
- The structured representation improves prediction quality on two video datasets and supports downstream tasks requiring object-level information.
- 5.1 Structured representation improves video prediction: Struct-VRNN matches or outperforms comparison models on VGG feature cosine similarity and Fréchet Video Distance.Lower FVD is better; the comparison includes Struct-RNN, CNN-VRNN, SVG, SAVP, and EPVA.
- 5.1 Structured representation improves video prediction: Structured representation, stochastic belief, and the best-of-many objective jointly contribute to performance and sample diversity.The full model produces the best ground-truth reconstructions and diverse high-quality samples, including samples farthest from ground truth.
- 5.2 The learned keypoints track objects: Unsupervised Struct-VRNN keypoints are nearly as predictive of object trajectories as supervised keypoints, while unsupervised CNN-VRNN representations perform poorly.With supervision, CNN-VRNN reaches similar performance to supervised Struct-VRNN, showing that the structured model learns spatial information without supervision.
- 5.3 Simple inductive biases improve object tracking: Removing temporal separation or sparsity losses reduces video prediction and tracking performance and increases variability across model initializations.These losses improve the stability and learnability of the discovered keypoint structure.
- 5.4 Manipulation of keypoints allows interaction with the model: Manipulating keypoints enables counterfactual player trajectories and locally targeted changes to human poses.Player dynamics are rolled out after changing observed keypoints, while body-part manipulations affect spatially local regions.
- 5.5 Structured representation retains more semantic information: Structured-model keypoints outperform unstructured CNN-VRNN features for Human3.6M action recognition.The evaluation trains a three-layer RNN to classify actions from keypoint sequences.
6 Discussion
The discussion frames the keypoint-based model as a spatially biased middle ground that improves downstream object-level tasks while retaining a simple architecture. It also emphasizes stochastic prediction as important and identifies downstream planning and reinforcement-learning applications as future work.
- The spatial inductive bias produces better downstream results than an unstructured CNN-derived representation on tasks requiring object-level understanding.
- The keypoint prior is a middle ground between unstructured representations and explicitly object-centric models, leaving masks, occlusions, and depth unmodeled or learned from data.This choice keeps the architecture simple and supports stable training across diverse datasets.
- Increasing sample diversity improves FVD because video prediction must represent many plausible futures rather than a single deterministic continuation.
- Applying the architecture to planning and reinforcement learning for control tasks is identified as an important future direction.
S1 Model implementation details
The implementation uses convolutional keypoint detection and reconstruction around a stochastic VRNN dynamics model. Training combines coordinate-based representations, latent stochasticity, scheduled sampling, and standard optimization procedures.
- S1.1.1 Keypoint detector: The detector produces K normalized feature maps and converts each map into a spatial-expectation coordinate representing a keypoint.The keypoint count K is a hyperparameter representing the maximum expected number of keypoints.
- S1.1.1 Keypoint detector: Each keypoint is represented by an (x, y, µ) triplet, where µ indicates the modeled object’s presence or absence.For reconstruction, keypoints are converted into Gaussian heatmaps scaled by µ.
- S1.1.1 Keypoint detector: The reconstructor combines keypoint heatmaps with appearance features from the first frame and upsamples them to reconstruct later images.
- S1.1.2 Dynamics model: The dynamics model uses a VRNN with prior, encoder, decoder, and recurrent components to model stochastic keypoint trajectories.The decoder outputs a linearized keypoint vector of length K × 3, while the recurrent component is a 512-unit GRU.
- Optimization and training: The implementation uses ADAM with β1 = 0.9, β2 = 0.999, batch size 32, and an initially 10^-3 learning rate halved every 3 × 10^4 steps.
- S1.1.2 Dynamics model: Training uses teacher forcing followed by scheduled sampling to reduce compounding errors while narrowing the train–test mismatch.The probability of using model samples is increased to a final value of 1.0.
S2 Experimental details
Experiments evaluate video prediction, representation quality, object tracking, and reward prediction across Basketball, Human3.6M, and DeepMind Control Suite datasets. The reported evaluations use stochastic prediction settings, structured and unstructured baselines, and separate downstream models where applicable.
- Video prediction: All compared video-prediction models use 8 observed frames to predict 8 future frames, including SVG and SAVP comparisons trained on the same datasets.
- Action recognition: Human3.6M action recognition uses sequences of 8 observed steps and a separate recurrent classifier over 15 action categories.
- Reward prediction: DeepMind Control Suite reward prediction covers six dense-reward tasks using a single action-conditional model trained across all domains.A separate reward-prediction model receives sequences of learned RNN hidden states for each domain.
- Video prediction: Basketball video quality is evaluated with Fréchet Video Distance, where lower values are better and each dot represents a separate mode.The figure compares Struct-VRNN with deterministic and unstructured baselines and SVG.
- Stochastic prediction: The best-of-many-samples objective produces more diverse Basketball trajectory predictions than training without that objective.