Source-linked AI summary
Visual Interaction Networks
Nicholas Watters, Andrea Tacchetti, Theophane Weber, Razvan Pascanu, Peter Battaglia, Daniel Zoran
TL;DR
AI systems lack a general-purpose mechanism for predicting future physical states from sensory observations. VIN combines a CNN visual encoder with an interaction-network dynamics predictor to learn object states and roll them forward from video. It predicts accurately across varied physical systems, including partially observable settings, while its evaluation is bounded by simulated system assumptions.
Problem
AI still lacks a general-purpose mechanism for predicting future physical states from present sensory observations, despite the importance of physical reasoning.
Method
VIN jointly uses a CNN-based visual encoder and recurrent interaction-network dynamics predictor to infer object states and predict future trajectories from video.
Results
VIN accurately predicts dynamics across all evaluated datasets, outperforms nearly all baselines, and remains strong on long rollout sequences.
Takeaways & Limitations
The results support learning factored object-, relation-, and physics-based representations from raw visual data for physical prediction.
Takeaways & Limitations
The evaluated systems are simulations where positions, masses, and radii are generally visually observable or global constants, limiting the initial benchmark’s hidden-quantity realism.
Abstract
from arXiv · showhide
From just a glance, humans can make rich predictions about the future state of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains and require direct measurements of the underlying states. We introduce the Visual Interaction Network, a general-purpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions and dynamics, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. Our results demonstrate that the perceptual module and the object-based dynamics predictor module can induce factored latent representations that support accurate dynamical predictions. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.
1 Introduction
The paper addresses the lack of a general-purpose mechanism for predicting future physical states from sensory observations. It introduces VIN, which combines visual perception with interaction-based dynamics prediction to learn physical systems from video.
- Physical reasoning remains difficult for AI because general-purpose prediction of future physical states from present sensory observations is still lacking.
- The model is trained from supervised video sequences and can approximate interacting physical systems by learning their dynamics and interactions.
- VIN combines a CNN visual encoder with a recurrent interaction-network dynamics predictor to infer object states and predict their future trajectories.
- Existing approaches include state-to-state physics models, image-based physical judgments, and pixel-to-future-state models tailored to domains or reliant on side information.
2 Model
VIN converts short video sequences into factored object state codes, then rolls those codes forward with interaction-based dynamics prediction. Its architecture combines visual encoding, temporal-offset aggregation, relational interaction modeling, and state decoding.
- Visual Encoder: The visual encoder applies a sliding window over frame triplets to produce state codes containing one vector for each object.Each vector represents the corresponding object’s position and velocity in distributed form.
- Dynamics Predictor: The dynamics predictor uses multiple interaction-network cores at different temporal offsets and aggregates their candidate state codes into the next predicted state code.The practical temporal offsets are {1, 2, 4}.
- Visual Encoder: The frame-pair encoder shares weights across consecutive frame pairs, approximating a temporal convolution over three input frames.
- Visual Encoder: Coordinate channels provide image-position information throughout the convolutional visual encoder.Without them, the architecture would need to infer position from image boundaries.
- State Decoder: A linear state decoder independently maps each state-code slot to an object position/velocity vector for encoder and prediction losses.
- Dynamics Predictor: Each Interaction Network combines pairwise relation effects with self-dynamics and post-processes their sum to produce a predicted object slot.
3 Experiments
The experiments evaluate VIN on simulated interacting-object systems, including hidden objects and variable masses, against visual, state-based, and ablated baselines. Training uses long-horizon prediction, and evaluation measures normalized performance and rollout position error.
- Physical Systems Simulations: The study uses five simulated physical-system types involving pairwise forces, including springs, gravity, billiards, magnetic billiards, and drift.Each system is evaluated with datasets containing three and six objects.
- Physical Systems Simulations: Hidden-quantity settings test whether VIN can infer an invisible object’s location and determine object mass from radius under constant density.
- Physical Systems Simulations: The simulated data use rendered objects over randomized static CIFAR-10 backgrounds, with anti-aliasing to support fine-grained position learning.
- Baseline Models: VIN is compared with visual RNN and LSTM predictors, a no-relations ablation, a ground-truth-dynamics model, and privileged state-to-state baselines.
- Training Procedure: Training predicts eight unseen future states with a discounted multi-step loss whose discount factor increases toward one during training.
- Results: VIN outperforms every baseline on every dataset except the no-relations ablation on Drift, where performance is comparable.
4 Results
The VIN predicts physical dynamics accurately across datasets, including systems with invisible objects, and maintains accuracy over long rollouts. It outperforms pixel-to-state and state-to-state baselines while producing visually plausible trajectories.
- Inverse Normalized Loss: On the invisible-object dataset, VIN performance is comparable to the fully visible 3-object Spring system.It locates the invisible object within 4% of frame width during the first 8 rollout steps.
- Euclidean Prediction Error: The VIN remains within 6% error after objects travel nearly one framewidth and within 7.5% error after 50 timesteps.These are mean Euclidean Prediction Errors over 3-object datasets.
- Euclidean Prediction Error: The VIN outperforms all other models, including privileged state-to-state models, on 50-timestep rollout position errors.The authors attribute part of this advantage to training the dynamics predictor to tolerate visual encoder noise.
- Visualized Rollouts: Rendered rollouts remain visually plausible for trajectories of up to 200 timesteps.The trajectory-trail visualizations cover 40–60 steps depending on the dataset, while accompanying videos show longer rollouts.
5 Conclusion
The Visual Interaction Network infers multiple objects’ physical states from frame sequences and predicts their future trajectories across varied, visually complex, and partially observable systems. Its visual encoder and interaction-based predictor support accurate long-term predictions and offer a step toward learning object-, relation-, and physics-based representations from raw data.
- The model infers multiple objects’ physical states from frame sequences and predicts their future trajectories.
- The CNN-based visual encoder obtains object-state measurements, while Interaction Networks provide relational computation for predictions far into the future.
- The model performs well across a variety of physical systems and remains robust to visual complexity and partially observable data.
- Visual-encoder noise can help the model overcome temporally compounding prediction errors during long-term rollouts, surpassing a pure state-to-state Interaction Net.
- The VIN provides a step toward understanding how object, relation, and physics representations can be learned from raw data.
6 Supplementary Videos
The supplementary videos show VIN rollouts alongside ground-truth simulations and provide examples of the datasets used for training and evaluation. They illustrate long rollouts for systems with three and six objects, using six input frames.
- The videos compare rendered VIN rollout position predictions with ground-truth system simulations for three-object and six-object datasets.
- Rollouts last 200 steps for three-object systems and 100 steps for six-object systems, except Drift, which uses 35 steps to keep objects in view.
- The VIN and all baselines receive six frames as input during training and rollouts on the three-object and six-object datasets.
7 Training Parameters
The models use a common training setup with 5 · 10^5 training steps, Adam optimization, a decaying learning rate, and a temporal discount for future rollout frames.
- 5 · 10^5 training steps are used for all models.
- Adam uses learning rate 5 · 10−4e−t/α, with α = 1.5 · 10^5 and t denoting the training step.
- Future rollout frames are weighted by discount 1 − γ, where γ = e−t/β and β = 2.5 · 10^4.
8 VIN Model Details
VIN encodes short image sequences into object-slot state codes, then predicts future codes with temporally offset Interaction Network computations. Its architecture combines visual pair encoding, slot-wise aggregation, object self-dynamics, pairwise relations, and recurrent or state-based variants.
- Visual Encoder: The visual encoder processes three images and produces an encoded state code from two consecutive image-pair candidate codes.
- Visual Encoder: The image-pair outputs are mapped to Nobject × 64 tensors, concatenated slot-wise into Nobject × 128, and transformed by a shared MLP into the encoded state code.
- Image Pair Encoder: Each image pair is channel-stacked, processed by two convolutional networks with different kernel sizes, and augmented with x- and y-coordinate channels.
- Dynamics Predictor: The dynamics predictor uses temporal offsets 1, 2, and 4, applies corresponding Interaction Network cores, and aggregates their candidate codes slot-wise.
- Interaction Network Core: The Interaction Network core combines per-slot self-dynamics with pairwise relation computations, sums them into updated slots, and applies an Affector MLP.
- Model Variants: The State model uses the same dynamics predictor without an encoder, whereas Visual RNN and Visual LSTM replace the VIN dynamics core with recurrent predictors.
9 Datasets
The datasets simulate several interacting physical systems under specified force laws, initialization procedures, rendering conditions, and stabilization measures. Evaluation reports Mean Euclidean Prediction Error for 8- and 50-step rollouts.
- Physical systems: The benchmark includes Spring, Gravity, Billiards, Magnetic Billiards, and Drift systems with distinct interaction rules.Spring uses Hooke’s law; Gravity uses gravitational interactions; Billiards uses elastic collisions; Magnetic Billiards uses Coulomb interactions; Drift has no forces.
- Initialization: The simulations use tuned force parameters and randomized object positions and velocities, with special initialization for Gravity.Gravity velocities are initialized tangentially around the frame center with a small random perturbation.
- Stabilization: Unbounded systems enforce zero net momentum and center selected objects to keep the simulated system within view.Gravity and Springs additionally center the objects’ center of mass in the frame.
- Stabilization: Weak friction is applied to every system except Drift to prevent accumulated numerical errors from destabilizing long simulations.The frictional force is linearly proportional to each ball’s area and supports stability over hundreds of timesteps.
- Rendering: The systems are rendered as 32 × 32 RGB videos over CIFAR10 backgrounds, with fixed color-based foreground/background ordering where occlusion is possible.Occlusion ordering is used for every system except Billiards.
- Evaluation: 8-step evaluation uses Mean Euclidean Prediction Error scaled as a percentage of frame width, while 50-step evaluation compares models over longer rollouts.The VIN outperforms pixel-to-state baselines on all datasets, some LSTM state-to-state baselines at 8 steps, and all other models at 50 steps.