PaperlayerView exact revision on arXiv

Source-linked AI summary

Flexible Neural Representation for Physics Prediction

Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B. Tenenbaum, Daniel L. K. Yamins

arXiv:1806.08047v2cs.AIcs.CVcs.LGcs.NE

TL;DR

Neural physics predictors need representations that handle arbitrary shapes, multiple scales, and complex deformable objects. This paper introduces a hierarchical particle representation and the differentiable Hierarchical Relation Network, which learns physical constraints and predicts plausible long-horizon dynamics, including complex collisions and nonrigid deformation. The results support generalization across varied physical properties and unseen scenarios, while the authors identify limitations for materials and object interactions such as shattering or merging.

  • Problem

    The paper addresses the challenge of generalizing neural interaction-network physics prediction to arbitrary shapes, multiple scales, and complex non-rigid objects.

  • Method

    The method combines a hierarchical particle-based object representation with an efficient end-to-end differentiable neural network based on hierarchical graph convolution.

  • Results

    The HRN predicts plausible trajectories for complex physical interactions over extended time horizons and generalizes across shapes, masses, forces, and material properties.

  • Takeaways & Limitations

    The approach supports complex collisions, nonrigid deformations, and large or unseen scene configurations, outperforming controls most of the time.

  • Takeaways & Limitations

    The reported scope does not yet include inflatables or gases, and future work targets dynamic scenarios in which objects can shatter or merge.

Abstract

from arXiv · show

Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail. Inspired by this ability, we propose a hierarchical particle-based object representation that covers a wide variety of types of three-dimensional objects, including both arbitrary rigid geometrical shapes and deformable materials. We then describe the Hierarchical Relation Network (HRN), an end-to-end differentiable neural network based on hierarchical graph convolution, that learns to predict physical dynamics in this representation. Compared to other neural network baselines, the HRN accurately handles complex collisions and nonrigid deformations, generating plausible dynamics predictions at long time scales in novel settings, and scaling to large scene configurations. These results demonstrate an architecture with the potential to form the basis of next-generation physics predictors for use in computer vision, robotics, and quantitative cognitive science.

1 Introduction

The paper targets neural physics prediction that can flexibly represent complex rigid and deformable objects across multiple levels of detail. It proposes hierarchical representations and networks to address geometric, multiscale, nonrigid, and computational challenges.

  • Motivation: Human physical reasoning flexibly captures object structure, interactions, deformability, and long-horizon trajectories.These abilities motivate a neural system for physics prediction in computer vision, robotics, reinforcement learning, and related applications.
  • Challenges: Existing neural interaction-network approaches face challenges representing arbitrary shapes, multiple scales, and complex non-rigid objects.These limitations hinder generalization to real-world physical dynamics.
  • Challenges: Particle-based representations allow constituent particles to move independently and model deformation, but dense pairwise relations become computationally infeasible and cognitively unnatural.Objects with hundreds or thousands of particles can produce an exploding number of relations.
  • Approach: The proposed hierarchical graph-based representation captures complex rigid and deformable bodies, while the hierarchical graph-convolutional network learns physics prediction within that representation.The approach is evaluated on complex 3D scenarios against strong baselines.
  • Results: The evaluation reports substantial quantitative and qualitative improvements over strong baselines, with evidence of generalization to complex unseen scenarios.The paper frames these results as addressing the representation and prediction challenges introduced in the introduction.

2 Related Work

Related work spans image-based future prediction, graphics simulation, hierarchical particle representations, and cognitive models of internal physics simulation. The paper positions its contribution as a differentiable neural approach for learned 3D physical dynamics.

  • Neural prediction: Image- and video-based future predictors are important for vision and robotics but struggle with sharp object boundaries, occlusions, and remembering unseen objects.These difficulties motivate structured physical representations beyond raw 2D image representations.
  • Physics simulation: Computer graphics commonly models rigid-body collisions, while particle and point-mass methods extend simulation to more complex physical objects.The related work includes both traditional and neural simulation approaches.
  • Hierarchical particles: NeuroAnimator used a hierarchical particle representation to advance physical-object movement.The paper’s particle-based representation draws inspiration from this line of work.
  • Positioning: Unlike the cited non-neural simulation work, the proposed solution is an end-to-end differentiable neural network that learns from data.This distinguishes the paper’s approach from physics engines and other non-neural particle simulations.
  • Cognitive science: Computational cognitive science proposes that humans run physics simulations mentally at flexibly adapted levels of detail.This perspective parallels the paper’s proposed flexible decomposition of environments into objects and parts.

3 Hierarchical Particle Graph Representation

The representation models each 3D object as particles arranged in a multiscale hierarchy with material and interparticle relations. Hierarchical graph construction reduces propagation complexity while preserving fine-to-coarse structure.

  • Representation motivation: Rigid-body assumptions exclude soft bodies, cloths, fluids, gases, and objects that break or combine.The paper therefore uses a representation based on potentially many smaller sub-parts.
  • Particles: Each object is represented as a set of particles, whose states contain position, velocity, and mass.The particle state is a vector in R7.
  • Particles: Particles are distributed across an object’s volume, with fewer particles for simpler parts and more particles for complex parts.For example, eight particles can fully define a cube, while a long rod can use more particles.
  • Hierarchy: Particles are grouped into a hierarchy that represents multiple object scales, with relations within groups and between ancestors and descendants.The hierarchy supports multiscale propagation through the object representation.
  • Relations: Within-object pairwise relations encode material properties such as local stiffness for soft bodies.The relation representation can vary across an object rather than assuming uniform material properties.
  • Graph construction: Fully connected propagation has O(NP^2) pairwise complexity, whereas the constructed hierarchy achieves O(NP log(NP)) complexity.The hierarchy starts from particle leaves, adds object roots, and inserts intermediate clusters.

4 Physics Prediction Model

The model predicts physical dynamics by propagating learned effects through a hierarchical particle graph, combining history, collisions, and external forces before predicting future particle states. Its training objective combines local and global motion accuracy with preservation of connected-particle structure.

  • Hierarchical Graph Convolutions For Effect Propagation: Pairwise graph convolution computes each sender’s effect on a receiver and sums the effects over all connected senders.A fully connected network processes sender state, receiver state, and their relation.
  • Hierarchical Graph Convolutions For Effect Propagation: Hierarchical graph convolution propagates effects from leaves to ancestors, between siblings, and from ancestors to descendants.This three-stage mechanism enables effects to flow across the object hierarchy in one model step.
  • The Hierarchical Relation Network Architecture: The HRN processes histories of hierarchical particle graphs with separate modules for past states, collisions, and external forces, followed by hierarchical effect propagation.Its state-prediction module then uses the propagated effects to compute the next particle states.
  • The Hierarchical Relation Network Architecture: Collision relations are dynamically added between nearby leaf particles from different objects and processed in both directions.The maximum distance for creating a collision relation is controlled by the hyperparameter D_C.
  • The Hierarchical Relation Network Architecture: The state-prediction module predicts local future position changes for particles using their states and propagated effects, while root particles receive global dynamics and gravity.Local predictions are converted into future position changes in world coordinates.
  • Learning Physical Constraints through Loss Functions and Data: The training loss combines local position-change accuracy, global position-change accuracy, and preservation of connected-particle distances.The preservation term maintains rigid distances and learns local deformations in soft bodies.

5 Experiments

Experiments evaluate HRN on rigid and deformable bodies, external forces, collisions, parameter changes, and unseen multi-object scenarios. HRN generally produces accurate or plausible predictions, while ablations show benefits from hierarchical processing, modular design, multiple input steps, and global-plus-local losses.

  • Experimental setup: HRN is evaluated across rigid bodies, soft bodies, cloth, fluids, collisions, external actions, parameter variations, and complex generalization scenarios.Roll-outs are generated recursively by feeding the one-step prediction back into the model.
  • Experimental setup: The training data is generated in a custom interactive particle-based environment that extracts particle representations from 3D meshes and randomizes physics scenes.The environment is based on the FleX physics engine in Unity3D and supports force-based interactions.
  • Qualitative evaluation of physical phenomena: HRN correctly predicts new trajectories caused by varied force vectors and application points, including parabolic motion, torque-induced rotation, impulses, and elastic floor collisions.These examples use instances of the same rigid shapes but test trajectories not seen during training.
  • Qualitative evaluation of physical phenomena: HRN generalizes from simple training shapes to complex non-convex objects, held-out shape collisions, and unseen scenarios including block towers, dominoes, and multi-sphere bowls.Predictions sometimes differ from ground truth in detail but remain plausible to human observers.
  • Qualitative evaluation of physical phenomena: HRN predicts non-rigid dynamics for soft bodies, cloth folding and deformation, and liquid-drop motion, with self-collisions added for cloth.For fluids, flatter hierarchies with small groupings work better because fluid effects are mostly local.
  • Qualitative evaluation of physical phenomena: Varying mass, gravity, and stiffness produces corresponding trajectory changes, while manually constructed heterogeneous-material objects extend beyond the simulator’s ground-truth capabilities.Objects accelerate more rapidly under stronger gravity, and the half-rigid half-soft example has no ground truth because the simulator cannot model heterogeneous materials.
  • Quantitative evaluation and ablation: The full HRN generally outperforms controls on cumulative position, local position-delta, and distance-preservation errors, while hierarchy and the proposed loss structure are especially important.Using multiple input steps helps estimate acceleration during collisions, and separate modules allow specialization for collisions, forces, history, and effect propagation.
  • Discussion: Long recursive roll-outs lose shape preservation and can jitter at rest, partly because training contained only fast-moving object interactions.The model also predicts multi-block towers collapsing faster than ground truth.

6 Conclusion

The paper presents a hierarchical graph-based scene representation and neural predictor for complex physical interactions. HRN generates plausible long-horizon trajectories and generalizes across varied shapes, masses, forces, and material properties, while future work targets broader materials and dynamics.

  • The hierarchical graph-based representation scales to arbitrary geometrical shapes and diverse material properties.
  • HRN uses hierarchical graph convolution to generate plausible trajectories for complex physical interactions over extended time horizons.
  • The model generalizes across shapes, masses, external and internal forces, and material properties.
  • Future work includes handling inflatables, gases, shattering, merging, and richer material-property representations.
  • Potential applications include grasping rigid and soft objects in robotics and modeling 3D point-cloud physics for simulations.

Supplementary Material

The grouping algorithm recursively builds a hierarchical scene graph from connected objects and particle relations. It repeatedly clusters sufficiently large particle groups, creates ancestor nodes, and connects hierarchy levels and within-group particle pairs.

  • The algorithm initializes an empty hierarchical graph and processes each connected component as an object.
  • Each object receives a root particle whose position and displacement are averaged over its particles and whose mass is summed.
  • The algorithm recursively processes particles while their leaf count reaches the target cluster size threshold.
  • It uses k-means to divide leaves into NC subcomponents and creates new root particles for multi-particle groups.
  • Ancestor-leaf and within-subcomponent relations are added before the completed hierarchical graph is returned.

B Comparison of different grouping methods

The grouping-method comparison evaluates agglomerative clustering against k-means hierarchies with maximum group sizes of 10 and 8 particles. The 8-particle k-means configuration is used in HRN because it offers the best reported trade-off.

  • The study compares agglomerative clustering with k-means grouping capped at 10 or 8 particles per group.
  • The authors report that k-means with 8-particle groups works best, balancing particles per group against the number of hierarchical layers.
  • The 8-particle k-means hierarchy is the configuration used in HRN.
  • The qualitative comparison presents agglomerative grouping on top, 10-particle k-means in the middle, and 8-particle k-means at the bottom.

C Comparison of different losses and graph structures

The ablations compare loss terms and graph structures using qualitative and quantitative evaluations. Combining global and local losses and using a hierarchical graph produces the strongest reported predictions, especially for preserving object shape.

  • Loss terms: Predictions trained with combined global and local losses are visually closest to ground truth.
  • Graph structures: The sparse flat graph is comparable to the hierarchical graph on position and delta position MSE but performs much worse on preserve distance MSE.
  • Graph structures: The authors attribute the sparse graph’s acceptable particle positions but deformed objects to less effective effect propagation than in the hierarchical graph.
  • Loss terms: The full combination of global and local losses outperforms each individual loss on the reported quantitative metrics.
  • Graph structures: The hierarchical graph significantly improves predictions compared with sparse flat and fully connected graph structures.

D.2 Training procedure

The experiments use a custom Unity3D particle environment to generate physics data and support interaction through Python. HRN training uses TensorFlow, Adam, large batches, multiple GPUs, and augmentation for generalization experiments.

  • Training procedure: HRN training uses TensorFlow, Adam, batch size 256, multiple Nvidia Titan Xp GPUs, and an initial learning rate of 0.001 with stepwise decay.The learning rate is decayed three times, alternating factors of 2 and 5.
  • Training procedure: Generalization experiments use data augmentation through random grouping, mass, and translation.
  • Training procedure: The custom Unity3D environment automatically decomposes 3D meshes into particle representations and generates randomized physics scenes through a Python interface.It supports forces applied to whole objects or individual particles and provides particle states and camera images each frame.

E.2 Shapes and surfaces used during experiments

The experiments cover diverse rigid shapes, deformable cloth and fluid-like materials, and multiple collision and force-driven settings. Moving objects use 50–300 particles, while surfaces use more than 5000 particles.

  • Shapes and surfaces used during experiments: Training and testing include cubes, cuboids, pyramids, flat pyramids, octahedra, prisms, cylinders, and ellipsoids.The dynamic shapes shown consist of 50–300 particles.
  • Shapes and surfaces used during experiments: Single-object experiments place randomized shapes above surfaces, vary stiffness, and repeatedly apply upward forces during collisions.The shapes fall onto surfaces and are lifted by forces perpendicular to the surface.
  • Shapes and surfaces used during experiments: The surfaces used in experiments are represented with 5000–7000 particles.Multiple leaf-node resolutions are possible by changing particle spacing, although one resolution is shown in the figures.
  • Shapes and surfaces used during experiments: Cloth experiments test dropping and force-driven deformation, as well as swinging cloth attached at two corners.The cloth is randomly positioned, perturbed by forces, and reset between trials.
  • Shapes and surfaces used during experiments: Additional scenarios test fluid spreading, airborne object collisions, multi-object collisions with gravity, falling cube towers, domino chains, and balls dropped into bowls.These setups vary object positions, stiffness, forces, and initial perturbations across repeated trials.

F Qualitative prediction examples

Qualitative examples test HRN on rigid collisions, deformable materials, complex multi-object interactions, and changes in stiffness, gravity, and mass. Predictions are generally physically responsive, but failure cases remain in difficult scenarios.

  • Qualitative prediction examples: Stiffness values from 0.1 to 0.9 produce decreasing deformation at frame t + 5 as stiffness increases.The model is trained on varying soft-body stiffness and accepts manually changed stiffness during testing.
  • Qualitative prediction examples: Objects fall faster as gravity increases across tested values from 1 to 20 m/s2, while heavier objects are displaced less after collisions.Mass variation during training reaches a factor of up to three times.
  • Qualitative prediction examples: Examples include nested-object motion, indirect collisions among five spheres, rigid shape collisions, and soft objects colliding with floors.The qualitative comparisons cover spheres, pyramids, cylinders, ellipsoids, octahedra, teddy bears, and ducks.
  • Qualitative prediction examples: Additional examples cover highly deformable sticks, falling dominoes, cubes on stairs or random surfaces, balls on slopes, and multi-object collisions.The deformable-stick prediction required the ground-truth shape as input, while HRN incorrectly moved one domino sideways.
  • Qualitative prediction examples: Cloth predictions include dropping cloth and cloth hanging from two fixed points while swinging back and forth.
Loading 1806.08047v2…