Source-linked AI summary
Propagation Networks for Model-Based Control Under Partial Observation
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, Russ Tedrake
TL;DR
Existing interaction networks are limited by full-state requirements and local pairwise reasoning, motivating a simulator that supports partial observation and long-range interactions. PropNet uses latent dynamics and learned graph propagation, and experiments report stronger forward simulation and control performance than prior approaches.
Problem
Interaction networks require fully observable states and only model pairwise interactions within a time step, limiting practical dynamics modeling for hidden-object and multi-body scenarios.
Method
PropNet is a differentiable learnable physics engine that uses latent dynamics for partial observations and multi-step propagation on graph-directed interaction paths.
Results
PropNet consistently outperforms interaction networks in forward simulation and improves model-based control across partially observable settings.
Takeaways & Limitations
Model-based control with PropNet is reported to be more sample-efficient, accurate, and generalizable than interaction-network and model-free deep-reinforcement-learning alternatives.
Abstract
from arXiv · showhide
There has been an increasing interest in learning dynamics simulators for model-based control. Compared with off-the-shelf physics engines, a learnable simulator can quickly adapt to unseen objects, scenes, and tasks. However, existing models like interaction networks only work for fully observable systems; they also only consider pairwise interactions within a single time step, both restricting their use in practical systems. We introduce Propagation Networks (PropNet), a differentiable, learnable dynamics model that handles partially observable scenarios and enables instantaneous propagation of signals beyond pairwise interactions. Experiments show that our propagation networks not only outperform current learnable physics engines in forward simulation, but also achieve superior performance on various control tasks. Compared with existing model-free deep reinforcement learning algorithms, model-based control with propagation networks is more accurate, efficient, and generalizable to new, partially observable scenes and tasks.
I. INTRODUCTION
Existing differentiable simulators support learned dynamics for control but remain limited by pairwise interaction modeling and full-state observability. PropNet addresses these challenges with latent dynamics and multi-step signal propagation, improving simulation and control outcomes.
- Physics engines support action planning by simulating environment effects, making accurate dynamics prediction important for reaching target configurations.
- Interaction networks model only pairwise interactions and require fully observable states, limiting scenarios with simultaneous multi-body effects or hidden objects.
- PropNet handles partial observability through a latent dynamics representation, addressing tasks where only selected objects are visible.
- PropNet represents scenes as interaction graphs and propagates signals along directed paths to model multi-body interactions within each time step.
- Experiments report that PropNet outperforms interaction networks in forward simulation and yields more sample-efficient, accurate, and generalizable model-based control.
B. Model-Predictive Control with a Learned Simulator
The paper frames control as trajectory optimization with a learned simulator operating on graph-structured object observations. PropNet differs from abstract-state approaches by modeling physical interactions directly and optimizing continuous-control trajectories.
- Model-predictive control methods can learn abstract-state transitions, whereas PropNet learns a general physics simulator from raw object observations.
- PropNet integrates the learned simulator with classic trajectory optimization and directly optimizes trajectories for continuous control.
- The physical system is represented as a directed graph whose vertices are objects and whose edges encode object relations.
- The simulator learns a dynamics function that predicts future scene states from the observed interaction graph by minimizing prediction error.
- Interaction Networks use object and relation functions to compose predictions while supporting variable numbers of objects and relations.
B. Propagation Networks
Interaction Networks rely on local information at each time step and therefore cannot model instantaneous force propagation across connected bodies. PropNet addresses this limitation with learned multi-step propagation inspired by message passing.
- Interaction Networks perform object- and relation-centric reasoning and can handle variable numbers of objects and relations.
- Interaction Networks cannot propagate forces instantaneously across a graph, limiting their applicability to rigid-body dynamics such as Newton’s cradle.
- PropNet is proposed to handle instantaneous force propagation efficiently through learned propagation over interaction graphs.
1) Effect propagation:
PropNet propagates effects through multiple directed-graph steps within one simulation timestep, allowing signals to travel beyond local pairwise interactions. The propagation depth is task-specific, with small depths often balancing performance and efficiency.
- 1) Effect propagation:: PropNet propagates relation and object effects across multiple directed-graph steps within each simulation timestep.The model indexes propagation steps from 1 to a maximum L and uses relation and object propagators at each step.
- 1) Effect propagation:: The relation and object propagators may share weights across propagation steps, depending on task complexity.
- 1) Effect propagation:: L = 3 often provides a good trade-off between performance and efficiency, although the appropriate propagation depth is task-specific.
2) Object- and relation-encoding with residual connections:
The efficient PropNet architecture reuses encoded object and relation information across propagation steps and adds residual links to support effect propagation. Small propagator networks are used under an assumption of simple transformations between steps.
- 2) Object- and relation-encoding with residual connections:: Vanilla PropNet is not efficient for fast online control, motivating reuse of information fixed at a timestep across propagation steps.
- 2) Object- and relation-encoding with residual connections:: Object and relation encoders compute shared representations before sequential propagation, avoiding recomputation at each step.
- 2) Object- and relation-encoding with residual connections:: Residual links connect adjacent propagation steps to address gradient vanishing and exploding and provide access to historical effects.
- 2) Object- and relation-encoding with residual connections:: The updated architecture is called Propagation Networks (PropNet) and uses small propagator networks for efficiency when inter-step effects are simple transformations.
C. Partially Observable Scenarios
For partially observable environments, PropNet learns dynamics in a latent space rather than requiring full state estimates. The representation uses object-centric encoding, global aggregation, reconstruction, and a history window.
- C. Partially Observable Scenarios: PropNet handles partial observability by encoding observations into a latent dynamics representation and predicting latent transitions.
- C. Partially Observable Scenarios: The encoder maps observations to object-centric representations, aggregates them into a fixed-dimensional global representation, and uses that representation because observable objects vary over time.
- C. Partially Observable Scenarios: A history window of length Thistory concatenates recent latent representations as input to the dynamics function.
- C. Partially Observable Scenarios: An additional reconstruction loss prevents trivial zero encodings by requiring the decoded representation to preserve scene-observation information.
IV. CONTROL USING LEARNED DYNAMICS
The paper uses differentiable learned dynamics models for trajectory optimization and online attribute estimation. Model-predictive control repeatedly simulates and updates controls to reduce goal error while compensating for accumulated prediction error.
- Model predictive control using shooting methods: Model-based control optimizes task-specific control inputs by minimizing the gap between simulated outcomes and a specified goal.The controls can include initial states, external forces, or object and relation attributes.
- Forward simulation: The learned dynamics model maps a dynamics graph at time t to a predicted graph at the next time step and supports multi-step forward simulation.Simulation results are stored while gradients update the control inputs through stochastic gradient descent.
- Model predictive control using shooting methods: Gradient-based trajectory optimization backpropagates from a loss over the history and predicted trajectory to update control inputs.The paper identifies this procedure as the shooting method in trajectory optimization.
- Model predictive control using shooting methods: Model-predictive control stabilizes long-horizon trajectories by simulating at every time step to compensate for accumulated prediction errors.Without this correction, the learned model may deviate from the ground truth as the horizon grows.
- Online adaptation: Online adaptation estimates unobserved attributes by minimizing differences between predicted future states and actual future states.The attributes may include physical quantities that are not directly observable.
V. EXPERIMENTS
The experiments evaluate the learned dynamics model across three physical scenarios and test both control and adaptation capabilities. The evaluation includes simulation, control, generalization, and online adaptation.
- Experimental scenarios: Experiments cover Newton’s Cradle, Rope Manipulation, and Box Pushing.The paper evaluates both simulation and control across these scenarios.
- Evaluation scope: The evaluation tests how the model generalizes to new scenarios and learns to adapt online.These tests complement the task-specific simulation and control experiments.
A. Physics Simulation
The physics-simulation experiments compare Interaction Networks with PropNet on rigid-body and deformable-object dynamics. PropNet improves long-horizon prediction and supports multi-body force propagation, while its propagation depth involves an accuracy–efficiency trade-off.
- Newton’s Cradle: Newton’s Cradle tests force transmission through a chain of rigid balls using fully observable dynamics graphs.The setup contains n = 5 balls and corresponding fixed pinpoints connected by directed relations.
- Newton’s Cradle: For 1,000 forward steps, PropNet achieves an MSE of 7.85 versus 336.46 for IN in tracking object positions.The qualitative comparison attributes IN’s error to incorrect force propagation before the first collision.
- Rope simulation: Rope experiments train on 15-particle ropes while evaluating lengths from 10 to 20, and PropNet tracks ground truth more accurately than IN.The reported improvement persists despite the rope’s shorter underlying force-propagation path than Newton’s Cradle.
- Box pushing: Box pushing represents observable boxes as graph vertices and automatically generates directional relations between every pair of observable boxes.The setting uses only observable boxes, with n(n−1) directed edges when n boxes are visible.
- Ablation studies: Propagation depth L creates an accuracy–efficiency trade-off: larger L models longer paths but is harder to train and more prone to overfitting.PropNet reaches a good trade-off at L = 3, while Vanilla PropNet performs best at L = 2 but generalizes less well as L increases.
B. Control
The control experiments apply learned dynamics to open-loop and feedback tasks under fully and partially observable conditions. PropNet improves control against interaction networks and other baselines, adapts to uncertain attributes, and generalizes across rope lengths.
- Newton’s Cradle: Newton’s Cradle control optimizes the initial angle so the rightmost ball reaches a target height using forward simulation and a goal loss.PropNet’s output-angle MSE is 3.08 from the ground-truth initial angle, versus 296.66 for IN.
- Rope Manipulation: Rope manipulation controls only the top two masses and requires exploiting rope dynamics to reach a target configuration.The goal loss is the L2 distance between the resulting and target configurations.
- Rope Manipulation: PropNet outperforms PD, DRL, and IN on rope manipulation when physical attributes are known.The comparison includes a hand-tuned PD controller with an MSE of 2.50 in the Normal condition.
- Online adaptation: Online attribute adaptation updates estimates over the first 20 steps and brings manipulation performance back toward the accurate-attribute condition.The method uses actual environment outputs as feedback and standard SGD to correct the estimates.
- Generalization: PropNet maintains good performance on rope lengths from 10 to 20 despite training only on length 15, while PD has an MSE of 2.72 for generalization.This experiment evaluates generalization to varied rope lengths.
- Box Pushing: In partially observed box pushing, PropNet outperforms DRL and IN using the L2 distance between resulting and target scene encodings for MPC.Only red boxes are observable, and final performance is evaluated with Chamfer Distance.
VI. CONCLUSION
PropNet is a general learnable physics engine for model-based control that improves simulation and supports both fully and partially observable environments. Its propagation steps capture long-range interactions within one time step, while residual connections and shared encoding improve efficiency.
- PropNet outperforms previous state-of-the-art learnable physics engines by a large margin.
- PropNet applies to model-based control in both fully and partially observable environments.
- Propagation steps transmit effects along relations to model long-range interactions within a single time step.
- Residual connections and shared encoding improve PropNet’s efficiency.