Source-linked AI summary
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, Jiajun Wu
TL;DR
Object-based physical reasoning can require difficult-to-obtain object-property supervision. O2P2 learns object representations without those labels and correctly built 17 of 25 goal configurations.
Problem
Learning object-based physical representations is challenging because prior approaches assume an idealized object space or object-property labels that are difficult to scale and can be ill-defined.
Source: §1 · pp. 1–2Method
O2P2 jointly learns perception, pairwise physical interaction, and rendering functions from image-level prediction without object-attribute supervision.
Source: §1 · p. 2Results
O2P2 correctly built 17 of 25 tested goal configurations.
Source: §3.4 · p. 9Takeaways & Limitations
The learned representations support planning for difficult object configurations that differ from training configurations, without further adaptation.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties. Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects, and a rendering function to map objects back to pixels. For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics. After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training.
1 INTRODUCTION
Machines still struggle with physical scene understanding, while object-based approaches face challenges connecting images to structured representations without difficult-to-obtain object-property supervision. O2P2 learns object representations for physical interactions without those labels.
1 INTRODUCTION
- Physical scene understanding at a human-like level remains elusive for machines, despite its importance in everyday tasks.The introduction frames physical intuition as something humans acquire and refine through experience.Source: §1 · p. 1
- Object-based methods must bridge high-dimensional observations and structured representations, but prior work assumed an idealized object space or supervision for object properties.Examples of supervised properties include category, position, and orientation.Source: §1 · p. 1
- Object-property labels are hard to scale, while semantic attributes can be ill-defined or constrain identity to a predefined vocabulary.Orientation requires a class-specific canonical reference, and classification-based identity limits categories to a fixed vocabulary.Source: §1 · pp. 1–2
- O2P2 learns object representations for physical interactions without attribute supervision, using segments or proposal regions and jointly trained perception, physics, and rendering modules.The model is trained on pixel-level prediction and is evaluated both on prediction quality and on downstream physical reasoning tasks.Source: §1 · p. 2
2 OBJECT-ORIENTED PREDICTION AND PLANNING (O2P2)
O2P2 jointly learns object representations, physical interactions, and image rendering without labeled object attributes. Its learned representations are used to plan action sequences that construct target block towers.
2 OBJECT-ORIENTED PREDICTION AND PLANNING (O2P2)
- O2P2 jointly learns perception, a physics engine, and rendering from image-level supervision rather than labeled object attributes.The model uses segment or region proposals and needs image-to-object and object-to-image mappings for training.Source: §2 · p. 2
2.1 PERCEPTION MODULE
- The perception module independently encodes each input segment into an object vector, without direct supervision for semantic properties such as position or orientation.Its outputs are trained through their use by the physics and rendering modules.Source: §2.1 · p. 3
2.2 PHYSICS MODULE
- The physics module predicts objects’ forward evolution using per-object transitions and pairwise interactions, then estimates a steady-state configuration with a single prediction.This avoids long-horizon sequential prediction while retaining information relevant to planning under physical constraints.Source: §2.2 · p. 3
2.3 RENDERING ENGINE
- The learned renderer combines object-specific image predictions using heatmaps to weight visibility at each pixel.Lower predicted heatmap values favor visibility, but the model receives no true depth maps during training.Source: §2.3 · p. 3
2.4 LEARNING OBJECT REPRESENTATIONS
- The modules are trained jointly on before-and-after image pairs depicting objects before and after a new object is dropped.The initial image has an available segmentation; perception and rendering reconstruct it, while physics and rendering predict the later image.Source: §2.4 · pp. 3–4
2.5 PLANNING WITH LEARNED MODELS
- Tower construction requires ordered actions that O2P2 was not trained on; planning samples placements and selects those whose predicted stable outcomes best match goal objects.The procedure iteratively re-plans and executes actions, using representation-space distance to evaluate candidate placements.Source: §2.5 · pp. 4–5
3 EXPERIMENTAL EVALUATION
The evaluation tests whether O2P2’s physics predictions support actionable planning, from synthetic block-tower tasks to a Sawyer robot. The results highlight the value of physics-aware planning and show successful real-world tower building across many tested goals.
3 EXPERIMENTAL EVALUATION
- Training uses 60,000 simulated block images, and held-out predictions can produce plausible steady-state configurations even when predicted images differ from ground truth.Training data vary block position, color, and orientation; each simulation contributes only initial and final frames.Source: §3.1 · p. 5
3.2 BUILDING TOWERS
- O2P2 outperforms Oracle (pixels) on block-tower build accuracy, while the No-physics ablation builds no towers correctly and SAVP has low stacking success on complex configurations.Tower success requires every object to have the correct shape; Oracle (pixels) is hampered by pixel-space comparisons.Source: Table 1 · p. 6, §3.2 · p. 6
3.3 THE IMPORTANCE OF UNDERSTANDING PHYSICS
- In tower planning, physics-based action scores select stable sequences, whereas the No-physics ablation can favor goal-image locations without accounting for blocks moving after release.O2P2 scores bottom blocks highly and is relatively insensitive to their release height because the blocks fall to the ground.Source: §3.3 · p. 7, Figure 6 · p. 7
- O2P2 also plans for alternate goals: maximizing tower height and stabilizing an observed block with other blocks.The height-maximization task scores samples by the highest non-zero pixels after rendering; stabilizing a free-floating block requires multi-step planning.Source: §3.3 · p. 7, Figure 7 · p. 8
3.4 TRANSFER TO ROBOTIC ARM
- On a Sawyer robot, O2P2 correctly built 17 of 25 goal configurations using real images and a separate action-to-object embedder for tractable planning.Planning also penalized predicted movement to favor stable placements because real physical interactions were less predictable than simulated ones.Source: §3.4 · pp. 8–9
4 RELATED WORK
O2P2 combines learned object-factorized physics with a focus on actionable downstream reasoning, contrasting with property-supervised and object-agnostic approaches. The authors report substantially better block-building results than representative video- and physical-prediction methods.
4 RELATED WORK
- O2P2 learns physical representations from data without object-property supervision, unlike markup-style scene representations designed for off-the-shelf renderers and simulators.The comparison concerns training a perception module to map scenes into a markup language.Source: §4 · p. 9
- Unlike object-factorized scene methods that use property labels or fully unsupervised object discovery, O2P2 uses segmentation masks and evaluates downstream physical reasoning.The model's learned physics can also be employed for decision making.Source: §4 · p. 9
- O2P2 achieves substantially better results on block-building tasks than representative video-prediction and physical-prediction methods.The video-prediction methods are Lee et al. (2018) and Babaeizadeh et al. (2018); the physical-prediction methods are Ehrhardt et al. (2017), Mottaghi et al. (2016), Li et al. (2017), and Lerer et al. (2016).Source: §4 · p. 9
5 CONCLUSION
The conclusion presents object-centric representations learned without object-property labels, using segment proposals and a factorized physics engine. The method supports prediction and planning in unseen configurations, and beats object-agnostic pixel-space approaches on block-tower matching.
5 CONCLUSION
- The method learns object-centric representations for physical interactions without labels for position, orientation, velocity, or shape.Segment proposals and a factorized learned physics engine guide representation learning.Source: §5 · p. 9
- The learned representations support difficult planning with configurations unseen during training, without further adaptation.The approach was also shown appropriate for a standard physics-prediction task.Source: §5 · p. 9
- On block-tower matching, the model outperformed object-agnostic approaches that compare directly in pixel space.Source: §5 · p. 9
A IMPLEMENTATIONS DETAILS
The implementation uses 256-dimensional object vectors, convolutional perception and rendering networks, and two multilayer perceptrons in the physics engine. Training uses Adam with a learning rate of 1e-3.
A IMPLEMENTATIONS DETAILS
- Objects use 256-dimensional vectors, and the perception module has four convolutional layers with 32, 64, 128, and 256 channels followed by a fully connected output layer.The convolutional layers use kernel size 4 and stride 2.Source: A IMPLEMENTATIONS DETAILS · p. 11
- Each of the physics engine's two MLPs has two hidden layers of size 512.Source: A IMPLEMENTATIONS DETAILS · p. 11
- Rendering networks use convolutional layers with 128, 64, 32, and 3 channels, or 1 output channel for the heatmap predictor.The kernels are 5, 5, 6, and 6, with stride 2.Source: A IMPLEMENTATIONS DETAILS · p. 11
- Training uses Adam with a learning rate of 1e-3.Source: A IMPLEMENTATIONS DETAILS · p. 11
B SAWYER RESULTS
The Sawyer-arm planning results use real image inputs, with seven action sequences counted as correct and three as incorrect. The figures also show all actions for one goal configuration.
B SAWYER RESULTS
- 7 action sequences were counted as correct and 3 as incorrect in Sawyer-arm planning with real image inputs.Figure 9 outlines correct sequences in solid black and incorrect sequences with dashed lines.Source: Figure 9 · p. 12
- Figure 9 distinguishes the seven correct action sequences with solid black outlines and the three incorrect sequences with dashed outlines.Source: Figure 9 · p. 12
- Figure 10 shows all actions taken by the planning procedure for one goal configuration from Figure 8.Source: Figure 10 · p. 12