Source-linked AI summary
Interaction Networks for Learning about Objects, Relations and Physics
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, Koray Kavukcuoglu
TL;DR
Physical reasoning requires handling objects and relations across combinatorially varied systems, but existing approaches provide limited general-purpose mechanisms for this task. The paper introduces interaction networks, which use graph-based, object- and relation-centric neural reasoning analogous to simulation. Across several physical domains, they learn accurate predictions, abstract-property inference, long-horizon rollouts, and generalization across system sizes and configurations.
Problem
Physical reasoning involves combinatorially many arrangements of objects and relations, motivating models that can explicitly reason about their interactions.
Method
Interaction networks take graphs as input and separately model relation-centric interactions and object-centric dynamics using shared neural functions.
Results
The model accurately predicts physical dynamics, rolls out trajectories over thousands of steps, infers abstract properties such as energy, and generalizes to systems with different numbers and configurations of elements.
Takeaways & Limitations
Explicit decomposition into objects and relations supports flexible learning and combinatorial generalization to novel physical systems and configurations.
Takeaways & Limitations
Scaling interaction networks to very large systems with many interactions remains an open challenge, motivating methods that cull negligible interaction computations.
Abstract
from arXiv · showhide
Reasoning about objects, relations, and physics is central to human intelligence, and a key goal of artificial intelligence. Here we introduce the interaction network, a model which can reason about how objects in complex systems interact, supporting dynamical predictions, as well as inferences about the abstract properties of the system. Our model takes graphs as input, performs object- and relation-centric reasoning in a way that is analogous to a simulation, and is implemented using deep neural networks. We evaluate its ability to reason about several challenging physical domains: n-body problems, rigid-body collision, and non-rigid dynamics. Our results show it can be trained to accurately simulate the physical trajectories of dozens of objects over thousands of time steps, estimate abstract quantities such as energy, and generalize automatically to systems with different numbers and configurations of objects and relations. Our interaction network implementation is the first general-purpose, learnable physics engine, and a powerful general framework for reasoning about object and relations in a wide variety of complex real-world domains.
1 Introduction
Interaction networks target physical reasoning by combining structured object–relation representations, simulation-like interaction processing, and deep learning. The paper evaluates this framework across physical domains and positions it as a general approach to reasoning about complex systems.
- Motivation: Interaction networks combine structured models, simulation, and deep learning to reason about interactions and dynamics in complex systems.Structured representations expose relations, simulation approximates dynamical systems, and deep learning supports scalable learning and inference.
- Model idea: For physical reasoning, the network represents systems as graphs, computes pairwise interaction effects, aggregates them with object states and external effects, and predicts new object states.The relational model computes interaction effects before the object model applies their aggregated influence to object dynamics.
- Model idea: The model separates relation-centric reasoning from object-centric reasoning, enabling generalization across variable numbers and novel configurations of objects and relations.Explicit relations allow selective processing of potential interactions rather than requiring every interaction or a fixed connectivity pattern.
- Evaluation: The evaluation covers n-body problems, rigid-body collision, and non-rigid dynamics, testing future-state prediction and inference of abstract properties such as energy.The experiments also examine long rollouts and generalization to systems with different numbers and configurations of elements.
- Related work: The interaction-network approach draws on prior neural methods for graphs, relations, composable computation, and learned connectivity.Related work includes graph neural networks, recursive autoencoders, neural programmer-interpreters, spatial transformers, and relational reasoning for visual tasks.
2 Model
The interaction network represents a system as objects and explicit relations, then applies separate shared models to interactions and objects in a simulation-like pipeline. Aggregation enables variable-sized, arbitrarily ordered systems and supports both future-state prediction and global-property inference.
- Graph representation: An interaction network represents a system as a graph whose nodes are objects and edges are attributed relations, with external effects supplied separately.Object states include dynamic and static attributes; external effects include control inputs or gravitational acceleration.
- Interaction pipeline: The relational model predicts each interaction’s effect, while the object model combines aggregated effects with object states and external inputs to predict object outputs.The marshalling function constructs one interaction term per relation, and aggregation collects effects for each receiver.
- Abstract inference: An additional abstraction model aggregates per-object outputs to infer a single global property for the whole system.The paper uses this variant to predict potential energy.
- Generalization: Shared relation- and object-level models let the network handle variable numbers of arbitrarily ordered objects and relations.The aggregation function must be commutative and associative; summation satisfies this constraint.
- Implementation: The learnable implementation uses multilayer perceptrons for relation and object processing, while marshalling and aggregation use parameter-free matrix operations.The relation and object models are trained by optimizing their learnable parameters.
3 Experiments
The experiments test interaction networks on future-state prediction and potential-energy estimation across three simulated physical domains. Training uses one-step data, while evaluation examines held-out prediction, long rollouts, and generalization across system sizes and configurations.
- Tasks and domains: The experiments evaluate future-state prediction and abstract-property estimation in n-body systems, bouncing balls, and spring-based strings colliding with rigid objects.The systems are simulated in two dimensions with a physics engine.
- n-body systems: Training scenes contain six bodies in the n-body domain, while test scenes contain 3, 6, or 12 bodies.Bodies exert distance- and mass-dependent gravitational forces, producing n(n −1) input relations.
- Bouncing balls: Training scenes contain six balls and four variably sized walls, while test scenes contain 3, 6, or 9 balls.Walls are represented as objects, and collisions are modeled through object relations.
- Strings and collisions: The string domain combines sparse spring relations between neighboring masses, rigid relations to a static circle, and variable gravitational acceleration.Each scene contains a spring-connected string and a rigid object positioned below it.
- Data and evaluation: Training and evaluation use simulated scenes and randomly sampled one-step input/target pairs, with performance measured on held-out test data.Each split contains 2000 scenes simulated over 1000 time steps; one-step samples number 1 million for training and 200k each for validation and testing.
- Training choices: Effect regularization improves generalization to different object counts and reduces drift over many rollout steps.Parameter regularization generally improves performance and reduces overfitting, while input noise slightly improves visual realism in rollouts.
4 Results
Interaction networks accurately predict physical dynamics, generalize across system sizes and configurations, and estimate abstract potential energies. Single-step training also supports visually realistic rollouts over thousands of steps, though nonlinear dynamics can eventually produce divergent states.
- Prediction accuracy: Orders of magnitude lower test error than alternative models was achieved for next-step dynamics across n-body, bouncing-ball, and string domains.The interaction network exploited object interactions; the dynamics-only IN performed similarly to the constant-velocity model.
- Generalization: The IN generalized well to systems with fewer and greater numbers of objects, including smaller n-body and spring systems where it exceeded models trained on those smaller systems.The authors suggest this may reflect greater experience with object and relation behavior from more complex training systems.
- Long-horizon rollouts: Single-step training produced effective trajectory simulations over thousands of steps, especially in the n-body and string domains.Figure 2 shows 1000-step rollouts for n-body systems, bouncing balls, and strings, including changes in system size and structure.
- Long-horizon rollouts: Model-generated trajectories were usually visually indistinguishable from ground-truth physics-engine trajectories, although small errors could rapidly create different states in nonlinear systems.The incoherent rollouts nevertheless did not violate people’s expectations and might be roughly on par with human understanding of these domains.
- Abstract-property estimation: Potential-energy estimation was more accurate than the MLP baseline in both evaluated domains: n-body MSE 1.4 versus 19, and string MSE 1.1 versus 425.The abstract-estimation variant was evaluated in n-body and string systems; ball potential energy was always 0.
5 Discussion
Interaction networks combine explicit object-relation reasoning with simulation and deep learning to model complex systems. The paper reports accurate physical simulation, generalization to novel systems, and inference of abstract properties, while identifying scalability and perception as future challenges.
- Discussion: The model learns accurate physical simulations and generalizes to novel numbers and configurations of objects and relations.It also infers abstract properties such as potential energy.
- Discussion: The interaction network is presented as a learnable physics engine and a template for reasoning about other physical and mechanical systems.
- Discussion: Future work must address very large systems with many interactions, including by culling interactions with negligible effects.
- Discussion: Future extensions include perceptual front-ends, control through external effects, recurrent prediction, and probabilistic inference over unknown properties and relations.
- Discussion: The approach combines relational reasoning, simulation, and deep learning for flexible learning and inference in challenging domains.
A.1.1 Physics engine details
The physics engine generates trajectories by repeatedly updating object states over 1000 one-millisecond steps. At each step, it computes interaction forces and converts them into accelerations using object masses.
- Physics engine details: Each simulated trajectory records object states across 1000 one-millisecond time steps.System parameters were selected to produce diverse dynamics within each trajectory.
- Physics engine details: At every step, the engine calculates forces for inter-entity interactions and applies them as accelerations after dividing by entity mass.
A.1.2 Physical domains
The experiments cover gravitational, collision-based, and spring-based physical domains with different relation structures and sources of variability. These settings include chaotic n-body dynamics, bouncing-ball collisions, and strings interacting with rigid objects.
- n-body: n-body systems model pairwise gravitational forces whose distances and masses produce highly complex dynamics sensitive to initial conditions.Masses varied across runs while other non-dynamic variables were held constant; forces use a denominator clipped near zero distance.
- Bouncing balls: Bouncing-ball scenes include all-to-all collisions among objects and walls, with variability in shape, scale, mass, and coefficient of restitution.These collision interactions are more difficult to simulate than the gravitational interactions in n-body systems.
- Bouncing balls: Rigid-body collisions are handled through collision detection followed by resolution when overlapping objects approach one another.
- Evaluation: The generalization experiments report prediction mean squared errors for systems with different numbers and configurations of elements.
- String: String scenes contain point masses connected by springs, a static rigid circle, and varied gravitational acceleration that usually produces collisions.Spring forces follow Hooke’s law and include damping proportional to velocity differences.
A.2 Results details
The prediction experiment reports generalization performance using mean squared errors. These results are presented in Table 3.
- Results details: Prediction-experiment generalization performance is measured with mean squared errors in Table 3.