PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, Jiajun Wu

arXiv:2205.02909v1cs.ROcs.AI

TL;DR

Elasto-plastic objects are important to household and industrial tasks, but their high degrees of freedom make state representation and dynamics modeling challenging. RoboCraft learns particle-based dynamics from raw RGBD observations and uses model-based planning; with 10 minutes of real-world interaction data, it shaped complex unseen targets and performed on par with or better than amateur humans on tested tasks.

  • Problem

    Elasto-plastic objects are useful in household and industrial tasks, but their high degrees of freedom make state representation, dynamics modeling, and control challenging.

    Source: §I · p. 1

  • Method

    RoboCraft builds particle representations from raw RGBD observations, learns object dynamics with GNNs and distribution-based losses, and plans actions using MPC with gradients.

    Source: §I · pp. 1–2, A. Problem Statement · p. 2

  • Results

    With 10 minutes of real-world interaction data, RoboCraft shaped complex targets and was on par with or sometimes more precise than amateur human subjects on tested tasks.

    Source: §I · p. 2, E. Results on Comparing with Human… · p. 9

  • Takeaways & Limitations

    The framework manipulates plasticine-like material into complex target shapes in simulation and on a real robot, including unseen targets.

    Source: §I · p. 2, §VI · p. 9

Abstract

from arXiv · show

Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned model-based planning framework is comparable to and sometimes better than human subjects on the tested tasks.

I. INTRODUCTION

RoboCraft addresses the challenge of modeling and manipulating elasto-plastic objects from raw RGBD observations. It combines particle-based dynamics learning with gradient-based planning and reports accurate shaping after brief real-world training.

I. INTRODUCTION

  • Elasto-plastic object manipulation remains underexplored despite its relevance to household and industrial settings and its challenges in state estimation and dynamics modeling.Prior particle-GNN methods often require full-state information and temporal particle correspondence, which are difficult to obtain from raw sensory data.Source: §I · p. 1
  • RoboCraft represents objects as particles, learns their dynamics with GNNs using distribution-based losses on raw visual data, and plans actions with MPC and gradients.Its perception module samples particles from reconstructed meshes; its planning module uses gradients from the learned model.Source: §I · pp. 1–2
  • With 10 minutes of real-world interaction data, RoboCraft accurately shaped plasticine into complex targets and performed on par with, and sometimes more precisely than, amateur human subjects.The reported targets include hearts and alphabetical letters.Source: §I · p. 2

A. Modeling the Dynamics of Deformable Objects

  • Particle-based simulation models deformable objects through interactions among neighboring particles, while data-driven work has studied several deformable materials and representations.RoboCraft uses a GNN to model object structure without assuming governing equations or one-to-one temporal particle correspondences.Source: A. Modeling the Dynamics of Deformable… · p. 2

B. Manipulating Deformable Objects

RoboCraft addresses the less-explored task of shaping elasto-plastic objects by planning pinching actions from an observed initial state. Its learned dynamics model predicts future observations for MPC-based action selection.

B. Manipulating Deformable Objects

  • RoboCraft uses a particle representation and expressive GNN to shape dough-like objects into complex, semantically meaningful shapes such as hearts and letters, unlike prior low-dimensional approaches.Prior work on elasto-plastic manipulation includes model-based and adaptive methods, but simulation transfer is hindered by state and parameter estimation, while demonstrations are expensive.Source: B. Manipulating Deformable Objects · p. 2

A. Problem Statement

  • The task is to use a parallel two-finger gripper to match an elasto-plastic object to a target shape through a sequence of pinching actions.Each action changes the plasticine state in response to the observed initial state.Source: A. Problem Statement · p. 2
  • A GNN learns the transition function from observation and action histories to a future observation, which RoboCraft uses to formulate model-predictive control.The MPC cost measures the final plasticine state’s distance from the target shape, and the action sequence is selected by minimizing that cost.Source: A. Problem Statement · pp. 2–3

B. Particle Sampling from Raw RGBD Data

RoboCraft converts raw RGBD observations into particles suitable for its dynamics model. Its sampling pipeline combines point-cloud processing, reconstruction, and refinements to address occlusion and missing observations.

1) Preprocessing:

  • RGBD images are converted to point clouds, then the largest-support plane and other noisy background points are removed.RANSAC detects the plane, identified as the object’s supporting platform.Source: 1) Preprocessing · p. 3

2) Surface Reconstruction:

  • Poisson surface reconstruction smooths occluded surfaces, unlike alpha shapes and ball pivoting, which are not robust to occlusion.Point normals are estimated from the principal axes of adjacent points before reconstruction.Source: 2) Surface Reconstruction · p. 3

3) Refinement with Physics Prior:

  • When the gripper is not touching the object, unchanged shape is used as a physics prior to patch particles missing from the current frame.Points from the previous frame fill gaps such as shadow artifacts.Source: 3) Refinement with Physics Prior · pp. 3–4

4) Refinement with Shape Prior:

  • A shape prior addresses incomplete point clouds during pinching by sampling inside a watertight mesh reconstructed from the object and gripper, then removing particles inside the fingers.Particles are sampled using the mesh’s signed distance function and the fingers’ SDFs.Source: 4) Refinement with Shape Prior · p. 4

5) Post-processing:

The particle representation is post-processed for the GNN and organized as a graph whose particle relations are recomputed from spatial proximity.

5) Post-processing:

  • Voxel down-sampling makes the point distribution uniform, statistical outliers are removed, and farthest-point sampling reduces the plasticine points to a reasonable number for the GNN.The resulting points are the final output of the sampling pipeline.Source: 5) Post-processing · p. 4

C. Learning Object Dynamics via Graph Networks

1) Graph Building:

  • The graph’s vertices are object particles with positions and attributes, while edges connect neighbors within a predefined distance and are recomputed from spatial relationships over time.Edge types distinguish relations such as object-internal and gripper-to-object connections.Source: 1) Graph Building · p. 4

2) Model Training:

The GNN predicts future states from graph states and actions, with multi-step message passing during training. Distribution-based losses address the lack of pointwise correspondence between sampled frames.

2) Model Training:

  • The GNN infers system dynamics by predicting the next graph state from the current graph and action, using multi-step message passing during training to handle instantaneous force propagation.Object and relation features are encoded and decoded to produce the prediction.Source: 2) Model Training · p. 4

D. Loss Functions

  • Because sampled point clouds lack one-to-one correspondence across frames, RoboCraft compares their distributions using Earth mover’s distance and Chamfer distance.Earth mover’s distance uses a bijection to match distributions and prevent outliers; Chamfer distance sums squared nearest-neighbor distances.Source: D. Loss Functions · p. 4
  • The training loss is a weighted sum of Earth mover’s distance and Chamfer distance, with experimentally selected weights w1 = 0.9 and w2 = 0.1.The loss combines the two point-cloud distance functions.Source: D. Loss Functions · p. 4

E. Model Predictive Control

The MPC controller searches a simplified parameterized pinch-action space to shape plasticine toward a target. It scores action sequences by the distance between the predicted final object and the target.

1) Action Space:

  • Each grip is parameterized by its center position, z-axis rotation, and minimum finger separation, with x- and y-axis rotations fixed at zero.Source: 1) Action Space · p. 4

E. Model Predictive Control

2) Goal-Conditioned MPC:

Goal-conditioned MPC uses the learned dynamics model to optimize grip actions toward a target. It executes one grip at a time and obtains camera feedback only after each grip.

2) Goal-Conditioned MPC:

  • Random shooting scores candidate action sequences with the GNN dynamics model, then limited-memory BFGS refines the lowest-cost trajectories using gradients.Source: 2) Goal-Conditioned MPC · p. 5
  • The controller receives camera-derived intermediate states only after each grip, trading feedback within a grip for time efficiency.Source: 2) Goal-Conditioned MPC · p. 5
  • MPC executes one grip at a time with a fixed look-ahead horizon, repeating until the loss is small enough or the maximum grip count is reached.Source: 2) Goal-Conditioned MPC · p. 5

C. Tasks and Benchmarks

The benchmark covers shaping tasks with varied motion constraints and targets in simulation and on a real robot. In sampling evaluation, crop-based particle sampling outperforms the patch-based baseline on both reported distance metrics.

C. Tasks and Benchmarks

  • The tasks test fixed and movable objects, rigid-motion shaping, unrestricted 7-DoF gripper motion, and selection between two tool sizes.Source: C. Tasks and Benchmarks · p. 5
  • Targets comprise 26 letters and five other shapes in simulation, and seven letters and five other shapes on the real robot; letters with holes are evaluated by contour only.Source: C. Tasks and Benchmarks · p. 5

D. Evaluation Metrics

  • The framework evaluation includes particle sampling, dynamics learning, and manipulation, using CD and EMD as main metrics and HD to assess whether all points are close across sets.Source: D. Evaluation Metrics · pp. 5–6, §V · p. 6

1) Sampling Results:

  • Crop-based sampling achieves lower CD and EMD losses than the patch-based baseline.Table I reports average distances between sampled and simulator ground-truth particles over 120 frames.Source: 1) Sampling Results · p. 6, Table I · p. 6

2) Dynamics Model Learning:

Dynamics models trained with combined CD and EMD losses improve Hausdorff distance relative to single-loss alternatives, while CD and EMD performance is similar. Predictions cover both rigid and non-rigid motion, with errors increasing over longer horizons but remaining in a reasonable range.

2) Dynamics Model Learning:

  • The GNN uses proximity-based graph edges and three-layer MLP encoders and motion prediction networks, trained for 100 epochs with Adam.The proximity threshold is d = 0.05; hidden and output sizes are 150, with batch size 4 and learning rate 1e-4.Source: 2) Dynamics Model Learning · p. 6
  • Combined CD and EMD training loss significantly improves Hausdorff distance, while CD, EMD, and combined-loss models perform similarly on CD and EMD losses.Source: 2) Dynamics Model Learning · p. 7
  • Prediction distances increase with longer horizons but remain in a reasonable range, and the model predicts particles under both rigid and non-rigid motion.Source: 2) Dynamics Model Learning · p. 7

3) Manipulation Results:

RoboCraft’s learned dynamics and planning support real-world shaping with limited interaction data. Evaluations cover alternative dynamics models, planning methods, complex action spaces, and unseen target shapes.

3) Manipulation Results:

  • Gradient-based optimization with the learned model outperformed the tested alternatives, with limited-time BFGS best in practice.Comparisons included random shooting, gradient-free planning, and model-based Soft Actor-Critic.Source: 3) Manipulation Results · p. 7
  • With a 7-DOF gripper, the method learned to rotate toward target shapes; offering two grippers of different sizes further improved performance.The reported improvement with more tool choices is averaged over all target shapes.Source: 3) Manipulation Results · p. 7, Table III · p. 7

B. Learning Real-World Manipulation of Deformable Objects

C. Comparing with Other Dynamics Models in the Real World

  • RoboCraft outperforms manually parameterized MPM and simulation-trained GNN dynamics models in real-world transfer.The authors attribute this to learning from real-world data while the alternatives face a simulation-to-reality domain gap.Source: C. Comparing with Other Dynamics… · p. 8

D. Generalizing to Novel Initial Shapes and Material Types

RoboCraft generalizes across initial shapes and to modeling foam, and its performance is comparable to or stronger than amateur humans on tested tasks. The conclusion also points to more complex action spaces, motion modes, and tool choices as directions explored.

D. Generalizing to Novel Initial Shapes and Material Types

E. Results on Comparing with Human Performance

VI. CONCLUSION

  • The authors characterize RoboCraft as the first model-based framework to shape complex, unseen elasto-plastic targets in simulation and on a real robot.They also explore harder tasks involving complex action spaces, mixed motion modes, and multiple tools.Source: §VI · p. 9

Related papers

Loading 2205.02909v1…