Source-linked AI summary
DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B. Tenenbaum, Shuran Song
TL;DR
Robots need physical object representations because static appearance rarely reveals properties such as mass or friction. DensePhysNet learns dense pixel-wise representations from multi-step sliding and collision interactions using a deep predictive model, and these representations support physical-property decoding, broader scene generalization, and improved manipulation.
Problem
Static appearance rarely reveals latent physical properties such as mass or friction, limiting object-physics learning for manipulation.
Method
DensePhysNet actively performs self-supervised sliding and collision interactions and uses predictive visual modeling with recurrent multi-step aggregation to learn dense physical representations.
Results
DensePhysNet learns rich physical information, achieves 88% simulated and 91% real-world material-classification accuracy, and improves downstream manipulation and novel-task control.
Takeaways & Limitations
Dense physical representations enable decoding object properties and applying them to manipulation tasks and physics-engine-based control in novel settings.
Abstract
from arXiv · showhide
We study the problem of learning physical object representations for robot manipulation. Understanding object physics is critical for successful object manipulation, but also challenging because physical object properties can rarely be inferred from the object's static appearance. In this paper, we propose DensePhysNet, a system that actively executes a sequence of dynamic interactions (e.g., sliding and colliding), and uses a deep predictive model over its visual observations to learn dense, pixel-wise representations that reflect the physical properties of observed objects. Our experiments in both simulation and real settings demonstrate that the learned representations carry rich physical information, and can directly be used to decode physical object properties such as friction and mass. The use of dense representation enables DensePhysNet to generalize well to novel scenes with more objects than in training. With knowledge of object physics, the learned representation also leads to more accurate and efficient manipulation in downstream tasks than the state-of-the-art.
I. INTRODUCTION
DensePhysNet addresses the difficulty of learning latent physical properties that static appearance and limited interactions do not reliably reveal. It uses multi-step dynamic interactions and predictive visual modeling to learn dense representations for physical-property decoding and manipulation.
- Motivation: Static appearance rarely reveals physical attributes such as mass or friction, making unsupervised physical-property learning less explored.Objects with similar appearance can differ substantially in physical properties, such as aluminum being lighter than steel.
- Challenges: Physical attributes may emerge only under particular interactions, requiring systems to explore multiple interaction types and decouple properties from their effects.Sliding distance depends on friction and mass under initial momentum but only friction under initial velocity.
- Approach: DensePhysNet executes self-supervised sequences of dynamic interactions, including sliding and collisions, and learns physical properties from visual observations.The system is designed to learn without explicit supervision.
- Approach: The model predicts pixel-wise optical flow from scenes and actions, recurrently aggregates information across interactions, and separates physical from visual representations.Predicting future object states under different interactions provides an implicit representation of physical properties and their effects on motion.
- Results: Experiments in simulation and real settings show stronger physical-property learning than other representation methods and improved downstream control, including novel tasks with a physics engine.The learned representations are used for tasks such as planar sliding and decoding object properties.
II. RELATED WORK
Prior work estimates or learns object physics using sensors, explicit physical models, or self-supervised interaction representations. DensePhysNet differs by combining dense pixel-wise representations with multiple dynamic interactions to support generalization across objects and multi-object scenes.
- Physics estimation: Physics-estimation methods use force-torque sensing or appearance and motion, while explicit physical models improve generalization but require difficult real-world annotations.These approaches parameterize models with properties such as mass and force.
- Self-supervised representations: Self-supervised interaction methods learn object representations without explicitly modeling physical properties, including motion prediction, visual recognition, and reinforcement-learning approaches.These methods use robot interactions or physical experiments to build object models.
- Relation to closest work: Compared with Push-Net, DensePhysNet learns dense representations and supports novel objects and multi-object scenes rather than a single global latent representation.Push-Net focuses on single-object scenes and encodes the entire image into one representation.
- Relation to closest work: DensePhysNet uses planar sliding and collisions, whereas Push-Net considers planar quasi-static pushing; experiments identify multiple interaction types as important for revealing latent physical properties.The interaction design is intended to expose complementary physical information.
- Dense representations: Dense visual representation work primarily captures appearance, while DensePhysNet targets object-centric physical properties learned through interaction.The contrast is between visual descriptors and representations encoding properties such as mass and friction.
III. METHOD
DensePhysNet trains a deep predictive model on depth images from dynamic robotic interactions to learn latent, object-centric physical representations. Predicting future states under varied interactions encourages the model to encode how physical properties affect motion.
- Predictive training: DensePhysNet trains on a large dataset of observed dynamic robotic interactions and predicts future depth-image states conditioned on actions.The method uses depth images from objects on an inclined ramp.
- Physical representation: The predictive objective makes the model acquire implicit representations of object-centric physical properties and their influence on observed motion.The learned representations are intended to encode properties such as mass and friction through self-supervision.
A. Dynamic Interactions
DensePhysNet uses dynamic sliding and collision interactions because object physics is more visible in motion beyond direct manipulator contact. It balances interaction diversity through randomized action selection and parameter sampling.
- Sliding and collision interactions expose physical-property cues that static or quasi-static manipulation often obscures.Dynamic motion after contact can reveal differences such as sliding distance and friction.
- Sliding: Sliding is parameterized by approach direction and velocity, with high-speed pushes enabling objects to continue sliding after contact.The robot constrains joint speeds to achieve high end-effector velocity within safety limits.
- Collision: Collision interactions roll a fixed auxiliary cylinder down an inclined ramp into a target object, with starting position controlling the setup.The cylinder rolls from either the left or right ramp and has fixed dimensions and weight.
- Interaction policy: A balanced-random policy selects action types and parameters while encouraging broad exploration and keeping objects inside the workspace.Action probabilities depend on how often each action has already been applied to each object.
B. DensePhysNet
DensePhysNet predicts pixel-wise optical flow from depth observations and actions while recurrently aggregating interaction history. Its modular components separate visual, physical, and action-state representations for self-supervised physical reasoning.
- DensePhysNet combines five modules to learn visual Rv, physical Rp, and action-state Ra representations from depth images and interactions.The modules are an image encoder, multi-step aggregator, action encoder, cross convolutional layer, and motion predictor.
- The image encoder extracts Rv, while the recurrent aggregator combines current visual information with the previous Ra to infer Rp.The aggregator uses interaction history because prior actions can place objects in physically informative positions and poses.
- The action encoder converts actions into convolution kernels, and the cross convolutional layer applies them to Rp to produce Ra.This layer functions as a learned latent physical simulator for action effects.
- The motion predictor maps Ra to optical flow between consecutive images, providing the sole training supervision.Optical flow is predicted during training but the motion predictor is unnecessary during testing.
IV. EXPERIMENTS
The experiments evaluate whether DensePhysNet’s learned physical representation encodes object properties accurately and supports manipulation. The evaluation combines qualitative analysis with quantitative tests.
- The experiments assess whether Rp encodes physical properties, how accurately it does so, and whether it improves object manipulation.
A. Decoding Object Material
DensePhysNet’s physical representation distinguishes object materials more effectively than visual features or sliding-only representations. This pattern appears in both simulation and real-world experiments.
- Setup and baselines: The comparison includes visual representations and a model trained with sliding only to isolate the contribution of active interaction types.Simulation uses visually indistinguishable objects whose material-specific mass and friction ranges differ.
- Real-world setup: Real-world evaluation uses a UR5 arm, RG2 gripper, calibrated RGB-D sensing, and sequences containing fifteen interaction steps.The model is pretrained in simulation and tested on twenty sequences involving three materials.
- DensePhysNet clusters objects by material, whereas visual features do not distinguish materials and sliding alone mainly separates friction differences.Sliding-only struggles to distinguish mass differences such as plastic versus metal.
- 88% simulation and 91% real-world material-classification accuracy show that DensePhysNet outperforms sliding-only and visual representations.Sliding-only reaches 69% in simulation and 72% in real-world testing, while visual representations reach 34% and 35%.
B. Decoding Physical Object Properties
DensePhysNet decodes physical properties from latent representations learned through multi-step interactions, outperforming comparison models and improving as interactions accumulate. Dynamic and diverse interactions are especially informative: sliding helps infer friction, while mass requires combining sliding and collisions.
- The comparison includes single-step forward and inverse models and Push-Net, alongside DensePhysNet with different interaction types.
- The evaluation trains 30-way linear classifiers for friction and mass and uses weighted distance error to compare predictions with ground truth.
- DensePhysNet outperforms the baselines in physical-property decoding, showing the value of aggregating information across multi-step interactions.
- Sliding substantially improves decoding over slow pushing, while mass is inferred only from combining sliding and collisions.
- Prediction error decreases quickly across interaction steps, indicating that the model progressively accumulates knowledge of object physics.
C. Application in Sliding Objects with Unknown Physics
DensePhysNet uses interaction-driven physical representations to choose pushes for objects with unknown physics. Exploration improves action accuracy and reduces sliding errors, particularly for heavy, novel, and uncommon objects, in simulation and real-world tests.
- The task varies target locations and requires selecting push direction and initial speed for objects whose physical properties are randomly sampled.
- The simulation evaluates 0, 3, and 7 interaction steps without optical-flow supervision or finetuning during policy testing.
- DensePhysNet enumerates candidate push actions, predicts each resulting motion, and selects the action whose predicted position is closest to the target.
- After a few exploration steps, DensePhysNet and Push-Net outperform other baselines on objects with uncommon physical properties.For objects with common properties, all models achieve a mean distance error of 0.06m.
- In real-world sliding, exploration produces more accurate actions and lower errors, with larger gains for heavy and novel objects than for normal objects.
D. Generalization
DensePhysNet generalizes from training scenes and tasks to settings with more objects and different collision-control requirements. Its object-wise representation supports multi-object scenes, while combining decoded properties with a physics engine improves novel-task performance.
- Generalizing to scenes with more objects: The error gap remains small across scenes with two, three, and four objects, indicating generalization to more objects than used during training.Training uses two-object sequences, while testing includes scenes with two, three, and four objects.
- Generalizing to a novel task: The novel task slides an auxiliary cube into the target object, requiring selection of the cube’s mass and speed.
- Generalizing to a novel task: For novel-task evaluation, the robot interacts for seven steps without finetuning, decodes properties, simulates candidate collisions, and selects the best mass-speed pair.
- Generalizing to a novel task: DensePhysNet outperforms baselines significantly on a novel task that combines decoded object properties with a physics engine for planning and control.
- Generalizing to a novel task: Using both sliding and collision interactions yields better novel-task generalization than using sliding alone.
V. DISCUSSION AND FUTURE WORK
DensePhysNet learns physical object representations from self-supervised interactions and uses them to improve downstream manipulation. The design remains limited by its action space and omission of color information.
- DensePhysNet learns about object materials and physics, enabling more accurate action policies for downstream planar sliding control.
- Using both planar sliding and collisions, the model infers object mass and friction, but richer interactions are needed to fully demonstrate its potential.
- DensePhysNet uses depth-based motion cues without color images, motivating models that combine color and motion for richer object representations.