Source-linked AI summary
Learning to Perform Physics Experiments via Deep Reinforcement Learning
Misha Denil, Pulkit Agrawal, Tejas D Kulkarni, Tom Erez, Peter Battaglia, Nando de Freitas
TL;DR
The paper asks whether agents can discover hidden physical properties through active interaction rather than passive observation or built-in physics knowledge. It trains deep reinforcement learning agents on interactive mass and object-counting tasks, finding that they learn task-dependent experimentation strategies that balance information gathering against mistake costs.
Problem
The paper addresses how agents can learn physical properties that cannot be reliably inferred from passive vision or appearance correlations alone.
Method
The authors train deep reinforcement learning agents to manipulate objects, observe consequences, and answer property questions in the Which is Heavier and Towers environments.
Results
Agents learn experimentation strategies that improve accuracy and often reduce time relative to randomized interaction policies across both environments.
Takeaways & Limitations
The findings suggest agents can learn to check hidden object properties while balancing information-gathering costs against the costs of incorrect answers.
Takeaways & Limitations
The paper does not address theory building or transferring learned mass and numerosity knowledge to new tasks.
Abstract
from arXiv · showhide
When encountering novel objects, humans are able to infer a wide range of physical properties such as mass, friction and deformability by interacting with them in a goal driven way. This process of active interaction is in the same spirit as a scientist performing experiments to discover hidden facts. Recent advances in artificial intelligence have yielded machines that can achieve superhuman performance in Go, Atari, natural language processing, and complex control problems; however, it is not clear that these systems can rival the scientific intuition of even a young child. In this work we introduce a basic set of tasks that require agents to estimate properties such as mass and cohesion of objects in an interactive simulated environment where they can manipulate the objects and observe the consequences. We found that state of art deep reinforcement learning methods can learn to perform the experiments necessary to discover such hidden properties. By systematically manipulating the problem difficulty and the cost incurred by the agent for performing experiments, we found that agents learn different strategies that balance the cost of gathering information against the cost of making mistakes in different situations.
1 INTRODUCTION
The paper studies how artificial agents can actively experiment to infer hidden physical properties, rather than relying on prior physics knowledge or visual appearance alone. Deep reinforcement learning agents learn strategies for mass and object-counting tasks that trade information gathering against answer accuracy and time.
- Motivation: The work explores whether artificial agents can learn scientific-like experimentation to infer hidden object properties through grounded interaction.The agents form beliefs about the world and update them by observing the consequences of their actions.
- Tasks: Agents are evaluated in Which is Heavier and Towers, inferring the heaviest block or the number of rigid bodies by interacting with objects.The agents receive no prior knowledge of physical properties or physics laws.
- Contribution: The approach extends prior work by learning from active interaction without assuming physical priors or relying on appearance-property correlations.The paper emphasizes grounded learning rather than an account of human cognition.
- Results: Learned interaction policies make agents more accurate and often faster than randomized interaction policies in both environments.In Which is Heavier, the strategies resemble algorithms designed with knowledge of the environment’s structure; in Towers, agents learn an adaptive closed-loop policy.
2 WHAT IS THIS PAPER ABOUT?
The paper frames physical understanding as a problem of learning hidden object properties through interaction, because visual appearance alone can be insufficient. It introduces task environments that reward agents for discovering and correctly labeling those properties.
- Scope: The paper does not propose a new model or algorithm; it uses reinforcement learning tasks to study learned experimentation.The authors expect other existing approaches could perform equally well in the same setting.
- Motivation: Understanding objects matters for planning, reasoning, exploration, and intuitive control in physical worlds, but staring at objects is not enough.The paper connects this motivation to children’s interaction-based understanding.
- Task design: The introduced environments make tasks depend on hidden properties and provide a way to probe whether agents learned those properties.Agents are rewarded for inferring the relevant properties correctly.
- Novelty: The agents learn how to check whether blocks are heavy even when material appearance is disconnected from physical properties.This distinguishes experimentation from merely associating visual appearance with heaviness.
- Limitations of prior approaches: Manually specified physical models are difficult to scale, generalize, and ground in perception, while visual prediction can confuse similarly appearing objects.The paper gives a sack of rocks and a sack of tennis balls as an example.
3 ANSWERING QUESTIONS THROUGH INTERACTION
The paper formulates experimentation as answering environment-grounded questions through interaction, with the agent choosing when to stop gathering information and label an answer. Discounting controls the trade-off between answering quickly and reducing mistakes through further experimentation.
- Three-phase interaction: Agents interact freely, issue a labeling action, and receive positive or negative reward before the episode ends.Failure to label before the maximum time limit produces a negative reward.
- Agent-controlled termination: The agent initiates the transition from interaction to labeling, deciding when it has gathered enough information to answer.This forces a choice between answering with current knowledge and delaying for more information.
- Trade-off: The information-versus-error trade-off depends on question difficulty and information cost.Difficulty is environment-specific, while information cost is controlled by the discount factor.
- Trade-off: A small discount factor encourages faster answers, whereas a large discount factor encourages longer information gathering to improve correctness.The discount factor changes the relative emphasis on future rewards.
- Question semantics: In this framework, each episode is the question and labeling actions are answers whose correctness is assessed by reward-grounded accuracy.This formulation avoids making the question depend on a particular information-acquisition strategy.
4 AGENT ARCHITECTURE AND TRAINING
The experiments use a recurrent agent architecture and pixel-processing pipeline, alongside a simulated Which is Heavier environment in which vertically constrained blocks vary in mass across episodes.
- Agent architecture: Agents use recurrent LSTMs with 100 hidden units, receiving features directly or pixel observations processed by three convolutional layers.Pixel observations are scaled to 84x84 pixels before convolutional processing.
- Environment: The Which is Heavier environment contains four vertically constrained blocks with fixed size, episode-varying masses, and constant agent strength.The environment is diagrammed in Figure 1.
- Agent architecture: The pixel-based convolutional layers use 32, 64, and 64 filters with sizes 8, 4, and 3 and strides 4, 2, and 1.Each convolutional layer is followed by a ReLU non-linearity.
5 WHICH IS HEAVIER
The Which is Heavier environment requires agents to infer randomly assigned block masses through interaction, while difficulty and information costs shape their experimentation strategies. Learned policies adapt information gathering to both task-level and instance-level difficulty and outperform randomized interactions in labeling accuracy.
- Environment: Masses are randomized independently of appearance, so agents must poke four vertically constrained blocks and observe their responses to identify the heaviest.Each episode assigns one heavy block and three light blocks; visual features alone provide no information about current mass.
- Environment: The task encodes a latent best-arm identification problem, but agents observe mass indirectly through object dynamics rather than direct rewards.This creates a more demanding setting than standard bandit algorithms, which assume rewards are directly observed.
- Difficulty: Problem-level difficulty is controlled by β, while instance-level difficulty depends on the mass gap between the two heaviest blocks.Lower mass gaps make individual episodes harder; distributions that generate more small gaps make the overall problem more difficult.
- Learned strategies: As difficulty increases, learned policies shift from guessing after finding one heavy block to poking all blocks before labeling.This matches the expected trade-off between information gathering and the risk of confusing similarly rewarding alternatives.
- Learned strategies: Agents spend longer gathering information on difficult instances, with episode length correlated with the mass gap.A small gap between the best blocks requires more information to distinguish them accurately.
- Evaluation: Compared with randomized interaction policies, learned policies produce more accurate labels without a consistent increase or decrease in episode length.This pattern holds across difficulty and observation combinations.
6 TOWERS
The Towers environment requires agents to infer hidden rigid-body structure through interaction, using direct or fist actuation. Agents learn effective, closed-loop experimentation strategies that remain accurate across interaction conditions and outperform randomized policies.
- Environment: The Towers task asks agents to count cohesive rigid bodies that cannot be identified from the tower’s initial visual or feature observations alone.Each episode randomly binds subsets of five primitive blocks into larger rigid bodies, so agents must interact and observe the tower’s collapse.
- Actuators: Agents can manipulate towers through direct forces on blocks or indirectly by pushing and hitting them with a spherical fist actuator.Direct actuation applies forces to individual blocks, whereas fist actuation controls the fist’s planar velocity.
- Experiments: The learned policy waits for informative observations before labeling, adapting its action timing rather than waiting a fixed number of steps.Testing varied the control time step, including smaller delays than during training, to distinguish closed-loop information use from fixed-step behavior.
- Learning: Agents solve the Towers task using different pairings of actuators and observations, including 3D features and 84 × 84 pixel images.Learning curves evaluate agents across actuator–observation combinations using multiple random seeds.
- Learning: Agents solve the task nearly perfectly, and fist actuators produce the fastest learning despite manipulating blocks indirectly.The authors suggest the fist may affect multiple blocks in one action, unlike direct actuation.
- Experiments: Compared with randomized interaction, learned policies produce labels more quickly and with much higher accuracy.The comparison records episode outcomes and the number of steps before labeling after 10,000 evaluation steps per agent.
7 RELATED WORK
Related work studies physical understanding through passive observation, active sensing, explicit physics models, and learned visual representations. This paper instead focuses on continual experimentation for non-visual object properties that require interaction.
- Learned representations: Other research learns physical representations by predicting embodiment quantities, synthesizing images or videos, and modeling action-conditioned dynamics.These lines include inverse dynamics, forward models, and synthetic-data approaches to physical judgments.
- Data and simulation: Synthetic and interactable datasets provide alternative routes for learning dynamics, including simulated force–trajectory supervision and block-tower stability judgments.These approaches can connect visual inputs to physical outcomes without the continual experimentation studied here.
- Physical reasoning: Psychophysics research links human judgments about block towers and rigid-body dynamics to inference over latent physical models.Related studies examine stability judgments and mass inference from observed dynamics.
- Physics models: Galileo and Physics 101 infer or encode physical parameters to predict dynamics, primarily from observed videos or visual tracking.These approaches use explicit simulators, object tracking, regression, or architectures incorporating physical laws.
- Active sensing: Many prior approaches are restricted to passive sensing, while several others use sequential data collection or active sensing for visual object learning.Active sensing is also motivated by developmental studies of toddlers.
- Distinction: Active-vision work uses deep reinforcement learning to acquire views for semantic classification, whereas this paper studies experiments for intrinsic object properties.The paper’s target tasks cannot be solved by vision alone and require interaction.
8 CONCLUSION AND FUTURE DIRECTIONS
The paper introduces deep reinforcement learning agents that actively interact with objects to infer hidden physical properties. It demonstrates this approach on mass and object-counting tasks while identifying limits in transfer, interaction forms, and data efficiency.
- Deep reinforcement learning agents actively interact with physical objects to infer hidden properties.
- The approach addresses inferring mass and counting objects under strong visual ambiguities.
- The work does not address theory building or transferring learned mass and numerosity knowledge to new tasks.
- Future work should examine how interaction shape influences behavior, including whether agents can use tools such as scales for mass judgments.
- Data efficiency was not optimized, leaving learning physical properties from fewer samples as an important direction.