Source-linked AI summary
A Compositional Object-Based Approach to Learning Physical Dynamics
Michael B. Chang, Tomer Ullman, Antonio Torralba, Joshua B. Tenenbaum
TL;DR
Learning physics simulators that generalize across object counts and scene configurations remains challenging. The NPE addresses this with object-based, compositional representations and pairwise dynamics, achieving strong generalization and latent-mass inference in two-dimensional worlds.
Problem
The paper asks how to build an intuitive-physics prior into an agent that generalizes across variable object counts and scene configurations without additional retraining.
Method
The NPE combines object-based scene factorization, context selection, compositionality, and pairwise interaction modeling in a trainable neural simulator.
Results
The NPE achieves low prediction error, extrapolates to unseen object counts and world configurations, and infers latent properties such as mass.
Takeaways & Limitations
Structured object interactions can support physics-simulator generalization across variable object counts and scene configurations.
Takeaways & Limitations
The experiments use simple two-dimensional ball-and-obstacle worlds, with angular velocity, gravity, friction, and pairwise forces set to zero.
Abstract
from arXiv · showhide
We present the Neural Physics Engine (NPE), a framework for learning simulators of intuitive physics that naturally generalize across variable object count and different scene configurations. We propose a factorization of a physical scene into composable object-based representations and a neural network architecture whose compositional structure factorizes object dynamics into pairwise interactions. Like a symbolic physics engine, the NPE is endowed with generic notions of objects and their interactions; realized as a neural network, it can be trained via stochastic gradient descent to adapt to specific object properties and dynamics of different worlds. We evaluate the efficacy of our approach on simple rigid body dynamics in two-dimensional worlds. By comparing to less structured architectures, we show that the NPE's compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.
1 INTRODUCTION
The paper builds a learned physics prior around objects, interactions, and compositional structure to generalize across changing scenes without retraining. The NPE combines symbolic-style inductive biases with neural adaptability and targets two-dimensional ball-and-obstacle worlds.
- The NPE treats a physical simulator as a program that maps scene inputs and past object states to future states and physical properties.
- The NPE combines symbolic physics structure with gradient-based learning, allowing adaptation to object properties and dynamics while extrapolating through local computation.
- The framework represents objects as primitives, selects relevant context objects, and assumes factorized interactions to support generalization across object counts and scene configurations.
- Factorization and compositionality operate in both scene representations and network architecture, including pairwise decomposition of object dynamics.
- The approach separates visual properties from physical dynamics and therefore focuses on learning dynamics in an intermediate state space.
- The experiments focus on simple rigid-body dynamics in two-dimensional worlds of balls and obstacles, using a neighborhood mask to select context objects.
2 APPROACH
The approach predicts each focus object’s dynamics from locally selected context and compositional pairwise interactions. It compares this structured architecture with baselines that remove pairwise processing or use sequential context attention.
- Object-Based Representations: The NPE predicts each object’s future state from its own past states and the past states of neighboring objects using spatially local computation.
- Pairwise Factorization: The NPE encodes each focus-context object pair, sums the pairwise encodings, concatenates that result with the focus state, and decodes the next velocity.
- State Updates: The model predicts velocity change and updates velocity and position, using velocity prediction to reduce dependence on training-world layouts.
- Context Selection: A neighborhood mask selects context objects whose positions fall within an object-dependent distance threshold, constraining the search space before collision reasoning.
- Baselines: The baselines form a lesion study of pairwise factorization and function composition by removing aspects of the NPE structure.
- Baselines: The NP removes pairwise interactions while retaining the NPE’s encoder and decoder, whereas the LSTM sequentially processes context objects before predicting the focus velocity.
3 EXPERIMENTS
The experiments evaluate NPE prediction, generalization, mass inference, neighborhood selection, and scene-configuration transfer against less structured baselines. Across these settings, NPE’s object-based and pairwise structure supports robust long-horizon prediction, extrapolation, and latent-property inference.
- Evaluation setup: The experiments compare NPE with NPE-NN, NP, and LSTM on prediction, generalization, inference, neighborhood-mask, and scene-configuration tasks.The evaluation uses simulated worlds of balls, walls, and obstacles, including held-out object counts and configurations.
- Prediction: NPE predictions initially follow ground truth and preserve intuitive physical dynamics over long simulations, while NP and LSTM trajectories diverge more substantially.Models are trained only for next-timestep prediction, then recursively generate future states over 50 timesteps.
- Generalization: 0.5 to 1 order of magnitude: NPE’s improvement over baselines in velocity prediction on unseen 6-, 7-, and 8-ball worlds.Training uses 3-, 4-, and 5-ball worlds, while test worlds contain previously unseen object counts and are combinatorially more complex.
- Inference: About 90% accuracy: NPE’s probability of correctly inferring a focus object’s mass, outperforming all baselines.The method evaluates candidate masses by selecting the hypothesis whose predicted collision outcome has the lowest MSE.
- Neighborhood mask: The neighborhood mask improves velocity prediction by about an order of magnitude and prevents associations with objects unable to influence the focus object.Its bounded context also reduces computational complexity from O(k) to O(1), where k is the number of scene objects.
- Scene configurations: NPE remains robust when testing on complex wall geometries and internal obstacles absent from training, whereas less structured models struggle with these configurations.Training uses simpler “O” and “L” worlds, while testing includes more complex “U” and “I” worlds.
4 RELATED WORK
Related work spans top-down symbolic or probabilistic physics engines, bottom-up observation-to-judgment or motion-prediction models, and hybrid object-based neural approaches. NPE is positioned as a differentiable, compositional bridge that retains structural assumptions while learning dynamics from data.
- Top-down and bottom-up approaches: Top-down approaches infer parameters of symbolic physics engines, while bottom-up approaches directly map observations to motion predictions or physical judgments.Top-down models require physical-law and geometry specifications; bottom-up models historically have not been compositional.
- Hybrid approaches: Hybrid approaches such as NeuroAnimator and interaction networks also learn neural models of physical systems involving objects and their relations.The interaction network applies to multiple physical systems, including n-body gravitational interactions.
- NPE positioning: NPE combines rough symbolic structure with gradient-based learning, using object-specific properties and object interactions as explicit inductive biases.Its differentiable architecture adapts the specific nature of interactions to the dynamics of a given world.
- Object-based approaches: Compared with earlier object-based motion prediction, NPE preserves more intuitive collision dynamics and demonstrates stronger generalization to eight balls, variable mass, and complex scenes.The comparison reports that earlier simulations showed attraction-like or magnetic-repulsion artifacts absent from NPE behavior.
- Architectural contrast: Unlike interaction networks, NPE learns relations by constraining attention to neighborhood objects rather than taking object relations as explicit inputs.NPE also infers properties such as mass without an additional classifier, whereas the compared interaction-network approach trains one.
5 DISCUSSION
The discussion argues that object-based, compositional structure helps the NPE learn and generalize physical dynamics while retaining broad applicability. It also positions the framework as a bridge between symbolic physics engines and adaptable neural networks.
- 5 DISCUSSION: Object-based representations, context selection, factorization, and compositionality support generalization across object counts and scene configurations with local computation.The paper attributes this generalization to making testing distributions more similar to training distributions.
- 5 DISCUSSION: The NPE uses inductive biases about objects to constrain physical modeling while learning dynamics largely from observation.The assumptions are described as both structurally useful and general enough for observation-driven learning.
- 5 DISCUSSION: In two-dimensional bouncing-ball worlds, the NPE achieved low prediction error, generalized to unseen object counts and configurations, and inferred latent properties such as mass.It outperformed baselines when all proposed ingredients were combined.
- 5 DISCUSSION: The NPE aims to emulate a general-purpose physics engine and could serve as a prior for model-based planning and reinforcement learning.The paper identifies linking the NPE to perceptual models as a next step.
A IMPLEMENTATION
The implementation trains all models with RMSProp and Euclidean loss under a fixed train-validation-test split, using a multilayer NPE encoder with masked interactions. Hyperparameters were explored informally rather than exhaustively.
- A IMPLEMENTATION: All models were trained for 1,200,000 iterations with RMSProp, Euclidean loss, learning rate 0.0003, and minibatches of size 50.The learning-rate decay was 0.99 every 2,500 iterations beginning at iteration 50,000; data used a 70-15-15 split.
- A IMPLEMENTATION: The NPE encoder used a pairwise layer with 25 hidden units and five feedforward layers with 50 rectified-linear units each.Bias-free encoder layers ensured masked non-neighboring objects did not contribute to activations.
- A IMPLEMENTATION: Hyperparameter exploration varied layers from 2 to 5, hidden dimensions from 50 to 100, and learning rates from 10^-5 to 3 × 10^-3.The authors state that this search was far from exhaustive but that the selected settings worked well.
B QUANTITATIVE ANALYSIS
Figure 6 reports normalized velocity and position errors across train-test variants. The NPE consistently outperforms baselines, while variable mass causes only a small error increase despite substantial mass variation.
- B QUANTITATIVE ANALYSIS: The NPE consistently outperforms all baselines by 0.5 to 1 order of magnitude in velocity and position error.Gray columns show normalized velocity MSE, while white columns show normalized Euclidean position error.
- B QUANTITATIVE ANALYSIS: Variable-mass experiments exhibit only slightly higher error than constant-mass variants despite masses differing by a factor of 25.The comparison concerns the corresponding train-test variants summarized in Figure 6.