PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Learning Mechanisms in Physical Reasoning

Shiqian Li, Kewen Wu, Chi Zhang, Yixin Zhu

arXiv:2210.02075v1cs.LGcs.AIcs.CV

TL;DR

The paper asks whether physical reasoning requires dynamics prediction, a computational question left largely untested. It compares direct intuition-based classification with dynamics-based approaches and finds that LfI competes with or exceeds LfD on PHYRE, while precise dynamics can improve performance.

  • Problem

    The paper asks whether physical reasoning requires predicting future dynamics, since this computational question has received limited attention despite the use of dynamics-prediction modules.

    Source: §1 · p. 2

  • Method

    The authors empirically compare direct outcome classification from initial scenes (LfI) with LfD, including tests with ground-truth and predicted dynamics.

    Source: §1 · pp. 2–3

  • Results

    On PHYRE, LfI is competitive with or better than LfD, while noisy predicted dynamics can lower reasoning performance and precise dynamics improve performance.

    Source: §1 · p. 3, §6 · pp. 9–10

  • Takeaways & Limitations

    LfI offers a simpler route when accurate dynamics are hard to predict, although improving dynamics prediction remains promising.

    Source: §6 · p. 10

    LfD needs additional architecture changes and supervisory signals to learn dynamics, while learning physical laws is demanding because of long-term prediction challenges.

    Source: §3 · p. 4

Abstract

from arXiv · show

Is dynamics prediction indispensable for physical reasoning? If so, what kind of roles do the dynamics prediction modules play during the physical reasoning process? Most studies focus on designing dynamics prediction networks and treating physical reasoning as a downstream task without investigating the questions above, taking for granted that the designed dynamics prediction would undoubtedly help the reasoning process. In this work, we take a closer look at this assumption, exploring this fundamental hypothesis by comparing two learning mechanisms: Learning from Dynamics (LfD) and Learning from Intuition (LfI). In the first experiment, we directly examine and compare these two mechanisms. Results show a surprising finding: Simple LfI is better than or on par with state-of-the-art LfD. This observation leads to the second experiment with Ground-truth Dynamics, the ideal case of LfD wherein dynamics are obtained directly from a simulator. Results show that dynamics, if directly given instead of approximated, would achieve much higher performance than LfI alone on physical reasoning; this essentially serves as the performance upper bound. Yet practically, LfD mechanism can only predict Approximate Dynamics using dynamics learning modules that mimic the physical laws, making the following downstream physical reasoning modules degenerate into the LfI paradigm; see the third experiment. We note that this issue is hard to mitigate, as dynamics prediction errors inevitably accumulate in the long horizon. Finally, in the fourth experiment, we note that LfI, the extremely simpler strategy when done right, is more effective in learning to solve physical reasoning problems. Taken together, the results on the challenging benchmark of PHYRE show that LfI is, if not better, as good as LfD for dynamics prediction. However, the potential improvement from LfD, though challenging, remains lucrative.

1 Introduction

The introduction asks whether physical reasoning requires dynamics prediction and compares direct intuition-based learning with dynamics-based approaches. Its experiments find that simple LfI can match or outperform existing LfD methods, while inaccurate predicted dynamics can undermine performance.

1 Introduction

  • The paper asks whether machines should reason from static physical structure or predict future states, a computational question left largely unexamined.Prior methods commonly use future-prediction modules, while the possibility that some tasks do not require dynamics prediction remains open.Source: §1 · p. 2
  • Physical-reasoning benchmarks such as PHYRE include long-term dynamics and complex interactions, with agents predicting whether an action achieves an explicit goal.For example, an agent may predict whether placing a red ball will make the green object touch the blue object.Source: §1 · p. 2
  • Simple LfI reaches state-of-the-art performance and outperforms existing LfD methods in unseen scenarios; precise dynamics improve performance, but approximations can reduce it to LfI or worse.The experiments compare direct intuition-based classification, ground-truth dynamics, and predicted approximations.Source: §1 · pp. 2–3
  • The authors conduct experiments comparing LfI and LfD, including a ground-truth-dynamics test as an upper bound and a predicted-dynamics test of practical LfD.They also examine several LfI models as a simpler alternative.Source: §1 · pp. 2–3

2 Related Work

Prior work spans intuitive-physics judgments and more complex physical reasoning, but existing approaches do not establish that dynamics prediction is necessary. The section also notes limitations in long-term prediction and object-centric methods.

2 Related Work

  • Prior intuitive-physics work addresses judgments such as object weight, tower stability, and liquid behavior, but primarily studies tasks without long-term dynamics.These studies include knowledge-based and learning-based approaches.Source: §2 · p. 3
  • Methods for more complex physical reasoning include simulation, Bayesian symbolic methods, and graph-based interaction networks, yet none fully justifies the necessity of dynamics prediction.The paper positions direct intuition-based learning as a simpler alternative to investigate.Source: §2 · p. 3
  • PHYRE methods include reinforcement learning and pixel- or object-based forward prediction; interaction networks achieve state-of-the-art performance using ground-truth object-state information.Other forward-prediction architectures did not yield significant performance improvement.Source: §2 · p. 3
  • Pixel-based future-prediction approaches can accumulate errors over long horizons, while object-centric methods have task-specific inductive biases and lose performance with multiple occluded objects.The cited object-centric examples use spatial information or assumptions about object count.Source: §2 · p. 3

3 The Two Learning Mechanisms

The section formalizes physical reasoning as predicting an outcome from initial conditions and distinguishes direct outcome learning from a dynamics-first pipeline. LfD may be optimized jointly or in stages, but learning dynamics requires additional architecture and supervision.

3 The Two Learning Mechanisms

  • Physical reasoning is framed as learning the final-outcome distribution from initial conditions comprising the background image and a random distribution of actions.The initial state is denoted X0 = {A, I}, and y denotes the possible outcome.Source: §3 · p. 4
  • In LfI, a task-solution model learns the outcome directly from initial images and actions, treating the model as an intuitive map from initial conditions to outcome.The model parameters are denoted θ.Source: §3 · p. 4
  • In LfD, a dynamics module first predicts object states over a time window, and a task-solution model then predicts the outcome from those dynamics.The predicted sequence is denoted D = {Xt | t = 0, 1, ..., T}.Source: §3 · p. 4
  • LfD can jointly optimize its dynamics and task-solution models or train the dynamics model first, freeze it, and then train the outcome model.The parallel schedule combines dynamics and cross-entropy losses, while the serial schedule optimizes the two models in sequence.Source: §3 · p. 4
  • LfD requires additional architecture changes and supervisory signals to learn dynamics; without them, it degenerates into LfI, and long-term physical-law learning is demanding.A physics engine or simulator would ideally provide future prediction.Source: §3 · p. 4

4 Experimental Setup

The experimental setup uses PHYRE-B to compare intuition-based classification with dynamics-based approaches, including simulator-grounded and learned predicted trajectories.

4 Experimental Setup

  • PHYRE-B comprises 25 red-ball puzzle templates, with within-template and cross-template evaluation settings.The section refers readers to Appendix A for additional training details.Source: §4 · p. 5
  • AUCCESS evaluates success rates with weights that encourage solving puzzles in fewer attempts.The weights are based on the number of attempts, k, from 1 to 100.Source: §4 · p. 5
  • The benchmark is challenging because solutions occupy a small part of the action space and tasks involve varied objects and complex physical interactions.The interactions include falling, rotation, collision, and friction.Source: §4 · p. 5
  • Experiment 1 compares RPIN, which predicts ten future steps before an MLP judges task success, with ViT, an image-based classifier.RPIN represents Learning from Dynamics (LfD), while ViT represents Learning from Intuition (LfI).Source: §4 · p. 5
  • Experiment 2 feeds simulator-derived image sequences to TimeSformer as an upper-bound test for Learning from Dynamics.This replaces predicted dynamics with ground-truth dynamics.Source: §4 · p. 5
  • Experiment 3 keeps TimeSformer as the task-solution model but supplies trajectories predicted by a learned dynamics module.PredRNN learns the dynamics, with curriculum training for better long-term prediction.Source: §4 · p. 5
  • Experiment 4 evaluates additional Learning from Intuition models, BEiT and Swin Transformer, alongside ViT.BEiT and Swin Transformer have been used for image classification.Source: §4 · p. 5

5 Comparison of Learning from Intuition and Learning from Dynamics

This section compares learning from intuition (LfI) with learning from dynamics (LfD), testing ground-truth and approximate dynamics before evaluating additional LfI models.

5.1 LfD vs. LfI: A surprising finding

  • ViT’s LfI is better than or on par with RPIN’s LfD, and its cross-template AUCCESS is significantly higher, indicating stronger generalization to unseen puzzles.RPIN predicts future object states before classifying outcomes, while ViT predicts outcomes directly from the initial scene.Source: §5.1 · p. 6, Table 1 · p. 6

5.2 LfD under GD: Is ground-truth dynamics better than intuition?

  • With 8 ground-truth input frames, AUCCESS improves by 23.11 within-template and 48.16 cross-template, reaching about 95% in both settings.The gains appear with four or more frames; with fewer than three, the model underperforms ViT.Source: §5.2 · p. 6, Figure 2 · p. 6

5.3 LfD under AD: How do approximate dynamics perform?

  • Approximate dynamics perform far below ground-truth dynamics and equal or worse than LfI across optimization schedules; serial training predicts more accurately but does not improve problem-solving over parallel training.The authors hypothesize that long-horizon uncertainty accumulates; parallel optimization can instead focus on predicting outcomes, effectively degenerating into LfI.Source: §5.3 · p. 7, Table 2 · p. 7

5.4 More on LfI: How does LfI perform?

  • Almost 6 points on average: ViT exceeds SOTA in cross-template AUCCESS, while LfI models remain competitive with RPIN within-template.ViT’s predicted solution distributions also closely match the ground truth.Source: §5.4 · p. 8, Table 4 · p. 8, Figure 4 · p. 9
  • LfI models use only initial scenes and final-outcome labels, avoiding extra object priors and complex dynamics-prediction tweaks.The paper also notes that these models can be pretrained on computer-vision tasks and concludes that LfI is a simpler, more effective paradigm for physical reasoning.Source: §5.4 · p. 9

6 Conclusion and Discussion

The discussion finds that LfI performs strongly when approximate dynamics fail to improve downstream reasoning, while accurate ground-truth dynamics leave LfD with a promising upper bound. It identifies prediction, representation, and evaluation limitations that shape these conclusions.

Conclusion and Discussion

  • LfI achieves strong cross-template generalization while remaining competitive within-template, whereas approximate dynamics can degrade reasoning to LfI-level performance or worse.The authors retain LfD as a promising route if dynamics can be predicted much more accurately.Source: §6 · p. 10

Why dynamics prediction struggles

  • Long-horizon prediction is difficult because unforeseeable events and accumulating frame errors increase future uncertainty and noise.The authors say current methods cannot robustly predict accurate dynamics in physical scenes.Source: §6 · p. 10
  • Pixel representations may preserve detailed physical cues but are noisy, while concise object-based representations can lose collision clues and their aftermath.The authors note that no feature representation yet summarizes all information needed for physical modeling.Source: §6 · p. 10
  • Self-attention may contribute to LfI's strong performance, although the authors describe task-solution model design as a less significant possible factor.Source: §6 · p. 10

Limitations and future work

  • The experiments use PHYRE alone because richer physical-reasoning environments are lacking, so the authors call for a broader task suite.Source: §6 · p. 10
  • The study uses original LfI models without task-specific design; improved perception and spatial analysis may further benefit LfI, particularly for collision dynamics.The authors describe collision in PHYRE as non-smooth angular movement outside traditional linear Euclidean space.Source: §6 · p. 10
  • The authors leave open how accurate predicted dynamics must be to benefit reasoning and suggest testing predictors with targeted masking or noise.Source: §6 · p. 10
  • Despite the difficulty, the authors consider LfD worth further effort given its upper bound and potential relevance to counterfactual and hypothetical reasoning.Source: §6 · p. 10

A Training Details

The appendix gives shared hardware information and separate training configurations for LfI, ground-truth-dynamics LfD, and approximate-dynamics LfD.

Hardware

LfI

  • LfI models used balanced successful and failed samples, batches of 224 × 224 images, and 10 fine-tuning epochs with cosine-annealed Adam training.The learning rate decreased from 1 × 10−4 to 1 × 10−6, using binary cross-entropy loss.Source: A Training Details · p. 15

LfD under GD

  • Ground-truth-dynamics LfD fine-tuned TimeSformer on eight-frame sequences sampled at 1-second intervals and padded shorter sequences with the last frame.Sequences of lengths 1, 2, 4, and 8 were extracted; training used 10 epochs, cosine annealing, Adam, and binary cross-entropy.Source: A Training Details · p. 15

LfD under AD

  • Approximate-dynamics LfD sent PredRNN-generated frames to TimeSformer, training both models under the LfI setting in serial and parallel schedules.PredRNN used reverse scheduled sampling; the parallel schedule set α and β to 1.Source: A Training Details · p. 15

B Additional Visualizations

The appendix adds visualizations of predicted dynamics and solution-distribution heat maps across tasks and models.

Predicted dynamics

Solution-set predictions

  • Figures A3 and A4 compare ground-truth and predicted P(y|X0) heat maps on PHYRE-B for Swin Transformer and BEiT, respectively.The heat maps are generated in the same way as for ViT.Source: Figure A3 · p. 17, Figure A4 · p. 17

Related papers

Loading 2210.02075v1…