Source-linked AI summary
Learning Physical Intuition of Block Towers by Example
Adam Lerer, Sam Gross, Rob Fergus
TL;DR
The paper asks whether deep feed-forward models can learn common-sense physical intuition from visual interactions with block towers. Using synthetic tower data and convolutional models, it evaluates stability, falling trajectories, and generalization, finding human-level prediction and transfer to real images.
Problem
The paper studies whether deep models can acquire intuitive physical knowledge, such as stability and momentum, without explicitly invoking Newton’s laws.
Method
The authors render randomized block towers in a 3D game engine and train convolutional models to predict stability, block locations, and falling trajectories.
Results
The models accurately predict tower stability and trajectories, generalize to new physical scenarios and real images, and match human performance on held-out real data.
Takeaways & Limitations
Bottom-up deep CNNs can capture aspects of common-sense physical intuition and provide evidence of learned tower dynamics rather than simple configuration memorization.
Takeaways & Limitations
Deep models require many thousands of training examples and have difficulty generalizing to cases far from their training data compared with simulation-based models.
Abstract
from arXiv · showhide
Wooden blocks are a common toy for infants, allowing them to develop motor skills and gain intuition about the physical behavior of the world. In this paper, we explore the ability of deep feed-forward models to learn such intuitive physics. Using a 3D game engine, we create small towers of wooden blocks whose stability is randomized and render them collapsing (or remaining upright). This data allows us to train large convolutional network models which can accurately predict the outcome, as well as estimating the block trajectories. The models are also able to generalize in two important ways: (i) to new physical scenarios, e.g. towers with an additional block and (ii) to images of real wooden blocks, where it obtains a performance comparable to human subjects.
1. Introduction
The paper asks whether deep feed-forward models can learn intuitive physical reasoning from visual examples of block towers. It introduces convolutional models that predict stability and trajectories, generalize across scenarios and real images, and support simulation-based data generation.
- Motivation: Deep neural networks are evaluated for learning physical intuition beyond perceptual tasks such as recognition and speech understanding.The motivation is to model questions about what will happen next in a scene without explicitly invoking mechanics.
- Motivation: Wooden block towers provide a simple testbed for center-of-mass, stability, momentum, and transfer from synthetic to real examples.The scenario also permits controlled construction of new physical configurations.
- Tasks and models: The study predicts both whether blocks fall and where they end up, requiring visual analysis of configuration and system dynamics.The models adapt convolutional architectures so lower layers perceive block arrangements while upper layers implicitly capture physics.
- Contributions: Standard convnet models refined on synthetic data accurately predict static stability and generalize to real images and unseen physical scenarios.The approach is purely bottom-up rather than relying on complex top-down graphics engines.
- Contributions: The models predict falling-block trajectories, indicating learned notions of acceleration and momentum in a feed-forward setting.Trajectory prediction complements the binary stability task by requiring dynamic prediction.
- Results and tools: Model performance matches humans on held-out real data, exceeds them on synthetic data, and correlates reasonably highly with human judgements.The paper also presents UETorch, an open-source Unreal Engine and Torch combination for machine-learning experiments.
2. Methods
UETorch embeds Torch directly into the Unreal Engine 4 game loop, enabling fine-grained control and efficient data capture for simulation-based learning.
- UETorch: UETorch embeds the Lua/Torch machine-learning environment directly into UE4, enabling online scripting and control of simulations.Torch runs inside the UE4 process, avoiding additional interprocess communication protocols.
- UETorch: The package captures screenshots, segmentation masks, optical flow, user input, and direct game-state modifications.Its interfaces support both observation and control of the simulated environment.
- UETorch: UETorch simulations can run faster than real time, supporting large-scale training and other vision, physics, and agent-learning experiments.The package is open source and downloadable from GitHub.
2.2. Data Collection Synthetic
The synthetic dataset uses randomized Unreal Engine simulations of colored block towers, recording outcomes, screenshots, and segmentation masks for learning stability and dynamics.
- Synthetic data: The simulation generates vertical stacks of 2, 3, or 4 colored blocks in random configurations.Block pose, camera, background, and lighting are randomized to improve transferability of learned features.
- Synthetic data: Each simulation records whether the tower falls and captures screenshots and segmentation masks at 8 frames/sec.Representative screenshots and masks are shown for a 4-block simulation.
- Human comparison: The human-experiment interface presents a tower image for a fall prediction and provides the outcome image as training feedback.Subjects receive no time limit for their predictions.
Real
The real dataset consists of manually stacked, spray-painted wooden cubes filmed as unstable or stable towers fall against a controlled background.
- Real data: Four wooden cubes were spray painted red, green, blue, and yellow for real-world tower examples.Manufacturing imperfections introduced randomness into stack stability.
- Real data: The cubes were manually stacked in configurations 2, 3, and 4 blocks high against a white bedsheet.The real and synthetic blocks were not physically matched.
- Real data: A tripod-mounted DSLR camera filmed the blocks falling at 60 frames/sec after a pole supporting the top block was lifted.The setup produced real image sequences for evaluating transfer beyond simulation.
2.3. Human Subject Methodology
The human-subject experiment evaluated binary judgments of whether block configurations would fall. Participants trained on labeled examples before completing a separate test set.
- 10 human subjects evaluated whether block configurations would fall during a controlled experiment.
- Participants trained on 50 randomly drawn examples with feedback about whether their predictions were correct.
- The subsequent test used 100 randomly drawn examples disjoint from the training set.
2.4. Model Architectures
The study trained several convolutional-network architectures on the synthetic blocks dataset, using either fall prediction alone or joint fall and mask prediction.
- Several convolutional-network architectures were trained on the synthetic blocks dataset.
- Some architectures were trained only for binary fall prediction, while others jointly learned fall and mask prediction.
Fall Prediction
ResNet-34 and GoogLeNet models were fine-tuned for fall prediction after adapting their final layers to produce a single logistic output.
- ResNet-34 and GoogLeNet networks were trained on the fall-prediction task.
- The models were initialized from ImageNet pretraining and their final linear layers were replaced with a single logistic output.
- The entire network was fine-tuned with SGD on the blocks dataset, with learning rates selected by grid search.
Fall+Mask Prediction
Fall-and-mask models predict whether blocks fall and estimate their segmentation trajectories at multiple future times using multi-class mask outputs and a combined training loss.
- Deep mask networks predict block segmentation trajectories at 0s, 1s, 2s, and 4s from an input image, alongside a fall prediction.
- Each mask pixel is classified among a background class and four foreground block-color classes.
- DeepMask was adapted by replacing its binary mask head with a multi-class SoftMax and replicating it for multiple prediction times.
- PhysNet uses spatially local, translation-invariant upsampling and added coarse-resolution depth to reason about block movement.
- Training combines binary cross-entropy for fall prediction with pixelwise multi-class cross-entropy for each mask.
- Baseline methods include logistic regression on image pixels or pretrained GoogLeNet features, plus k-nearest neighbors for fall prediction.
2.5. Evaluation
The evaluation measures fall prediction and future-mask quality, including transfer to held-out tower sizes and uncertainty-sensitive metrics.
- Fall prediction is evaluated on synthetic and real images, including models trained with a held-out tower size.The held-out-size setup tests transfer learning to an unseen number of blocks.
- Mean mask IoU and log likelihood per pixel evaluate predicted block masks.Mean mask IoU uses the binarized t = 4s prediction, averaged over foreground classes present in each label.
- Mean mask IoU compares each foreground mask label with its binarized prediction using intersection-over-union.The metric averages over foreground classes present in the mask label.
- Mask IoU can score a probabilistic prediction as zero after binarization, even when its confidence indicates the block occupies that region.Log likelihood better captures the quality of predicted mask confidences.
- Log likelihood per pixel measures the likelihood of the correct final mask under the predicted SoftMax distribution.It is essentially the negative mask training loss.
- Real-example model accuracy uses an estimated confidence interval based on the standard deviation of a binomial distribution.The binomial probability is approximated by the model’s observed accuracy across N = 493 real examples.
3. Results
The results show strong fall and trajectory prediction, with PhysNet outperforming alternatives on mask metrics and transferring to unseen tower sizes, while real-image predictions remain more uncertain.
- 3.1. Fall Prediction Results: Convolutional networks substantially outperform baselines for fall prediction, with PhysNet achieving the best synthetic-data accuracy.Accuracy on real data is within the models’ standard deviations across convolutional networks.
- 3.1. Fall Prediction Results: Imagenet pretraining preserves Googlenet’s synthetic performance but improves its generalization to real images.Without pretraining, Googlenet generalizes poorly to real examples.
- 3.1. Fall Prediction Results: Occlusion heatmaps indicate that PhysNet uses local features at unstable interfaces and stabilizing blocks when predicting falls.In one example, only occluding the unstable interface between the middle and top blocks changes the prediction.
- 3.2. Mask Prediction Results: PhysNet achieves the highest Mean Mask IoU and Log Likelihood per pixel, substantially outperforming DeepMask and baselines.The initial-mask IoU baseline is artificially strong because mask IoU uses binarized masks.
- 3.2. Mask Prediction Results: PhysNet masks usually capture trajectories for falling 2- and 3-block synthetic towers, but become more diffuse for 4-block and real examples.Real-example predictions are skewed toward stability, likely because simulated and real blocks have different physical properties.
- 3.3. Evaluation on Held-Out Number of Blocks: On an unseen tower size, accuracy decreases but remains comparable to human performance, while predicted masks still capture fall dynamics.Four-block examples show some degradation when that size is excluded from training.
- Model vs. Human Performance: PhysNet outperforms human subjects on synthetic data and performs similarly to humans on real data.The figure compares ROC performance and performance by number of blocks.
4. Discussion
The results support bottom-up deep CNNs as a route to human-level prediction of falling block towers, with evidence that models learn dynamics rather than memorize configurations. Synthetic physical simulations and mask prediction also provide a broader framework, but deep models require much more data than simulation-based alternatives and generalize poorly far from training data.
- Bottom-up deep CNN models attain human-level performance at predicting how towers of blocks will fall.
- Small degradation on unseen tower sizes, occlusion results, and poor nearest-neighbor performance provide evidence that models learn dynamics rather than memorize configurations.
- Deep models require many thousands of training examples for high performance and have difficulty generalizing to examples far from their training data.Simulation-based models begin with strong priors encoded in a physics simulation engine, whereas deep models learn physics from scratch.
- Synthetic data from realistic physical simulations in UETorch is useful for machine-learning experiments in vision, physics, and agent learning.
- The combination of synthetic data and mask prediction offers a framework for learning concepts including object permanence, 3D extent, occlusion, containment, solidity, gravity, and collisions.