Source-linked AI summary
Unsupervised Learning for Physical Interaction through Video Prediction
Chelsea Finn, Ian Goodfellow, Sergey Levine
TL;DR
Interactive agents need to predict how actions affect objects, but existing physical-interaction methods often require impractical labeled object information. The paper predicts pixel motion while reusing previous-frame appearance, introduces a large robot-pushing dataset, and reports more accurate video predictions than prior methods. Its approach supports action-conditioned visualization of possible futures, while leaving uncertainty modeling and explicit object-centric representation as limitations.
Problem
Learning physical interaction dynamics from scalable real-world video is difficult because many existing methods require labeled object information, while raw video contains complex, stochastic, high-dimensional, partially observed motion.
Method
The paper develops an action-conditioned video predictor that predicts distributions or transformations of pixel motion from previous frames and composites multiple transformed images.
Results
The model produces plausible video sequences more than 10 time steps, about one second, into the future and achieves the best results on several previously proposed metrics.
Takeaways & Limitations
Action-conditioned multi-frame prediction can let agents imagine alternative futures for planning, anticipating problems, and exploring new phenomena.
Takeaways & Limitations
The model encodes uncertainty as blur and does not explicitly extract an internal object-centric representation.
Abstract
from arXiv · showhide
A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment. Many existing methods for learning the dynamics of physical interactions require labeled object information. However, to scale real-world interaction learning to a variety of scenes and objects, acquiring labeled data becomes increasingly impractical. To learn about physical object motion without labels, we develop an action-conditioned video prediction model that explicitly models pixel motion, by predicting a distribution over pixel motion from previous frames. Because our model explicitly predicts motion, it is partially invariant to object appearance, enabling it to generalize to previously unseen objects. To explore video prediction for real-world interactive agents, we also introduce a dataset of 59,000 robot interactions involving pushing motions, including a test set with novel objects. In this dataset, accurate prediction of videos conditioned on the robot's future actions amounts to learning a "visual imagination" of different futures based on different courses of action. Our experiments show that our proposed method produces more accurate video predictions both quantitatively and qualitatively, when compared to prior methods.
1 Introduction
The paper addresses unsupervised prediction of physical interactions in complex real-world videos, where prior approaches depend on labeled object information or struggle with appearance and pixel dimensionality. It proposes pixel-motion prediction that reuses previous-frame appearance and combines multiple motion transformations for action-conditioned future-video prediction.
- Unlabeled video could let interactive agents autonomously collect extensive experience for predicting physical interactions and support planning and decision making.
- Real-world physical interactions are complex and stochastic, while raw-video learning must handle high-dimensional pixels and partially observed object motion.
- The model predicts pixel motion relative to the previous image, allowing appearance information to be reused rather than reconstructed internally.
- DNA predicts distributions over previous-frame locations, CDNA predicts normalized convolution kernels, and STP predicts affine transformations.
- Multiple predicted transformations are combined with a compositing mask so separate object motions form one next-image prediction.
2 Related Work
Prior work addressed video prediction and physical interaction using restricted domains, short horizons, or labeled object information. This paper contributes pixel-transformation models and a real-world robot-interaction dataset designed to capture complex motion, collisions, occlusions, and interaction.
- Video prediction: Prior video-prediction methods commonly targeted synthetic videos, short-term real-video prediction, small patches, or single-frame settings.
- Learning physics: Prior physical-interaction methods required object poses, segmentation masks, camera viewpoints, or image-patch trackers, limiting their use of unlabeled video.
- Pixel transformations: The proposed pixel-advection architecture predicts transformations of previous-frame pixels and combines them through masking to model object motion without reconstructing appearance.
- Video datasets: The new robot-interaction dataset supplies real-world videos with rich object motion, collisions, realistic occlusions, and a clear interactive-robot learning use case.
3 Motion-Focused Predictive Models
The paper introduces video-prediction models that explicitly predict pixel or object motion from previous frames, allowing motion-focused and partially appearance-invariant predictions. These models compose transformed image regions with masks and use convolutional recurrence to support action-conditioned, multi-step prediction.
- The models predict image motion from previous frames rather than reconstructing object appearance, making them partially invariant to appearance.This design is intended to improve generalization to previously unseen objects.
- Dynamic Neural Advection (DNA): DNA predicts a distribution over previous-frame locations for each new pixel and computes its value as the distribution’s expectation.The method constrains motion to a local region, reducing prediction dimensionality while retaining flexibility.
- Convolutional Dynamic Neural Advection (CDNA): CDNA predicts multiple shared motion distributions applied across the image, assuming pixels on the same rigid object move together.The resulting transformed images are later combined into one prediction.
- Spatial Transformer Predictors (STP): STP predicts multiple 2D affine transformations, applies them with bilinear sampling, and composites the resulting images using masks.The transformations warp pixels from the previous image into generated image coordinates.
- Results: The motion-focused models outperform same-architecture ablations that predict raw pixels or frame differences, and generalize better to unseen objects than direct pixel-reconstruction models.The comparison includes models with and without skip connections and evaluates longer-horizon and action-conditioned training.
- Composing motion predictions: CDNA and STP use masks to combine motion predictions, while their learned masks can isolate objects moving in consistent directions.This reuses transformations across pixels and yields a more object-centric unsupervised representation; DNA remains more flexible but lacks these benefits.
- Action-conditioned prediction: Stacked convolutional LSTMs provide recurrent spatial representations for multi-step video prediction, while action and robot-state inputs condition future image predictions.The action and state vectors are spatially tiled and concatenated into a low-dimensional activation map.
4 Robotic Pushing Dataset
The paper introduces a real-world robotic pushing dataset for action-conditioned video prediction, collected with multiple robot arms interacting with many objects. It includes training interactions and test sets that assess both familiar and novel-object generalization.
- Each interaction records RGB images together with gripper poses as internal state and commanded gripper poses as actions.The setup supports prediction conditioned on the robot’s future actions.
- Two test sets contain 1,250 recorded motions each, covering held-out object subsets and objects entirely absent from training.The second test set directly evaluates generalization to novel objects.
- Figure 2 shows the robot data-collection setup above example camera images captured during the interactions.The figure pairs the physical apparatus with representative visual observations.
5 Experiments
The experiments evaluate action-conditioned and unconditional video prediction across robotic pushing and human motion, comparing motion-predictive models with prior and appearance-reconstructing methods. The results show stronger prediction quality, action-sensitive futures, and unsupervised segmentation, while predictions degrade as the horizon extends.
- Action-conditioned prediction for robotic pushing: Our models significantly outperform prior video prediction methods on all reported metrics for robotic pushing.The models use 12.5 million parameters, compared with 12.6 million for the feedforward multiscale model and 78 million for the FC LSTM.
- Action-conditioned prediction for robotic pushing: Explicitly modeling pixel motion outperforms models that predict raw pixels or frame differences using the same stacked convolutional LSTM architecture.The comparison isolates the effect of the motion-prediction representation.
- Action-conditioned prediction for robotic pushing: Training for longer horizons and conditioning on robot actions improve the action-conditioned prediction setting.These effects are examined in the ablation study.
- Limitations: Prediction quality degrades over time because future uncertainty increases, and the mean-squared-error objective represents that uncertainty as blur.Modeling uncertainty directly is identified as future work.
- Action-conditioned prediction for robotic pushing: On novel-object tests, motion-predictive models have a larger advantage over models that reconstruct appearance.The comparison evaluates models with the same architecture but different prediction targets.
- Human motion prediction: On human motion videos with a held-out subject, motion-predictive models quantitatively outperform prior methods and produce plausible motion for at least 10 timesteps before degrading.The human-motion evaluation holds out a subject and clothing combination unseen during training.
- Action-conditioned prediction for robotic pushing: With the same starting image and different future actions, CDNA predicts stationary futures for zero action and more object motion for an action 150% larger than the original.The qualitative predictions use objects absent from the training set.
- Human motion prediction: The model learns background masks that segment the human subject without explicit supervision.These masks are predicted internally to mask the previous frame.
6 Conclusion & Future Directions
The model combines prior-frame appearance with predicted pixel motion to generate plausible action-conditioned futures. The paper reports strong video-prediction results while identifying object-centric representation learning as a future direction.
- The action-conditioned model incorporates previous-frame appearance with motion predicted by the model.
- The model produces plausible video sequences more than 10 time steps into the future, corresponding to about one second.
- The method achieves the best results on a number of previously proposed metrics in comparisons with prior methods.
- The action-conditioned predictions can support imagining different futures for planning, anticipating problems, and exploration.
- Although the model groups pixels that move together, it does not explicitly extract an internal object-centric representation.Learning such a representation is proposed as a future direction, especially for reinforcement learning methods that benefit from concise state representations.
A Data Collection Details
The dataset was collected with multiple robot arms performing randomized pushing interactions on varied objects. The setup recorded robot state, commands, torques, and camera images for model training.
- Data collection used 10 seven-degree-of-freedom robot arms and recorded robot states, commands, torques, and RGB camera images.
- Images were center-cropped and downsampled from 640 × 512 to 64 × 64 for faster training.
- Each robot interacted with 10–20 random objects, which were replaced after approximately 4,000 randomized interactions.
- The robots repeatedly performed either a random push or a randomized sweep toward the bin’s middle.
B Model Details
The models use convolutional recurrent cores and transform previous images to predict future frames. DNA, CDNA, and STP differ primarily in how they parameterize these transformations and composite them with masks.
- Each model uses a stride-2 convolution followed by convolutional LSTMs, with skip connections preserving high-resolution information.
- Action-conditioned robot models tile the 10-dimensional robot state and action into an 8 × 8 map and concatenate it with the input to LSTM 5.
- CDNA and STP output transformation parameters after LSTM 5, using normalized 5 × 5 filters or affine transformation parameters, respectively.
- DNA outputs a spatially varying 5 × 5 kernel for each image pixel at the last layer, unlike the other transformation models.
- Transformed images and the previous image are combined through masks, with the previous image serving as a static background.
C Additional Experimental Results
Additional experiments compared losses for a feedforward multiscale baseline on the robot-motion dataset. The adversarial objective failed to train successfully, while GDL+l1 performed best among the evaluated objectives.
- The adversarial loss could not be trained successfully for this model and dataset.
- The feedforward multiscale architecture was evaluated with four proposed objectives on the robot motion dataset.
- GDL+l1 performed best on the robot dataset and was therefore used for the main evaluation results.