PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools

Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, Jiajun Wu

arXiv:2306.14447v2cs.RO

TL;DR

Long-horizon deformable-object manipulation requires robots to coordinate tool selection with continuous action planning. RoboCook learns tool-object dynamics from visual data and combines tool classification with self-supervised policies. It handles dumpling-making under perturbations, outperforms four baselines on letter-cookie tasks, and generalizes across materials without retraining.

  • Problem

    Complex soft-object tasks combine tool-use decisions with continuous motion planning conditioned on the selected tool.

    Source: §1 · p. 1

  • Method

    RoboCook learns tool-object dynamics with GNNs and combines tool classification with self-supervised policies for manipulation planning.

    Source: §1 · p. 1

  • Results

    RoboCook succeeds at dumpling-making under external perturbations, outperforms four strong baselines on alphabet letter cookies, and makes shapes on different materials without retraining.

    Source: §1 · p. 1

  • Takeaways & Limitations

    RoboCook demonstrates elasto-plastic object manipulation with a general-purpose robotic arm and everyday tools.

    Source: §5 · p. 8

    RoboCook relies on human-defined subgoals, and higher-level temporal abstraction and task-level planning are needed to remove them.

    Source: §5 · p. 8

Abstract

from arXiv · show

Humans excel in complex long-horizon soft body manipulation tasks via flexible tool use: bread baking requires a knife to slice the dough and a rolling pin to flatten it. Often regarded as a hallmark of human cognition, tool use in autonomous robots remains limited due to challenges in understanding tool-object interactions. Here we develop an intelligent robotic system, RoboCook, which perceives, models, and manipulates elasto-plastic objects with various tools. RoboCook uses point cloud scene representations, models tool-object interactions with Graph Neural Networks (GNNs), and combines tool classification with self-supervised policy learning to devise manipulation plans. We demonstrate that from just 20 minutes of real-world interaction data per tool, a general-purpose robot arm can learn complex long-horizon soft object manipulation tasks, such as making dumplings and alphabet letter cookies. Extensive evaluations show that RoboCook substantially outperforms state-of-the-art approaches, exhibits robustness against severe external disturbances, and demonstrates adaptability to different materials.

1 Introduction

Long-horizon soft-object tasks require both discrete tool choices and continuous motion planning conditioned on those choices. RoboCook combines learned tool-object dynamics, tool classification, and self-supervised policies, and is reported to handle dumpling-making, disturbances, letter cookies, and different materials.

1 Introduction

  • It addresses tasks that require decisions about both which tool to use and the continuous motion conditioned on that tool.Source: §1 · p. 1
  • RoboCook combines a GNN model of tool-object interactions, PointNet-based tool classification, and self-supervised policies for gripping, rolling, and pressing.Other skills use hand-coded policies.Source: §1 · p. 1
  • RoboCook succeeds at dumpling-making despite external perturbations, outperforms four strong baselines on alphabet letter cookies, and makes shapes on different materials without retraining.Source: §1 · p. 1

2 Related Work

Prior work tackles deformable-object manipulation through model-based, adaptive, demonstration-based, and learned-dynamics approaches, while tool-use research has explored tool representations and hierarchical policies. The paper positions its approach against the limits of short-horizon, single-tool learning and the cost of demonstrations.

Long-horizon deformable object manipulation

  • Prior learned-dynamics methods show results on short-horizon tasks using one tool, while long-horizon tasks require decisions about tool choice and action parameters.Source: §2 · p. 2
  • Model-based and adaptive methods face complex system-identification problems, and approximate simulators can create a sim-to-real gap.Expert demonstrations have also been used, but obtaining them can be expensive or prohibitive.Source: §2 · p. 2

Tool usage

  • Prior tool-use work learns tool representations or semantics for downstream policies and uses demonstrations for hierarchical policies; this work instead uses cheaper real-world robot random-play data.Source: §2 · p. 2

3 Method

RoboCook represents dough and tools as particles, learns tool-aware dynamics, and uses a tool classifier with closed-loop policies to plan long-horizon manipulation. Its method proceeds from perception and dynamics modeling to tool selection and action planning.

3 Method

  • The framework combines particle-based scene representation, a GNN dynamics model, tool classification, and self-supervised closed-loop control.Source: §3 · pp. 2–3

3.1 Perception

  • Four-camera point clouds are processed into a watertight dough mesh, then sampled as 300 surface particles and combined with sampled tool particles.Reconstruction choices depend on occlusion; the processing also removes points that penetrate the tool.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Dynamics Model

  • The GNN predicts future dough states from the current state and tool actions, using rigid and non-rigid motion predictions and only 20 minutes of real-world data per tool.Training on two-step predictions supports 15-step inference; the loss combines Chamfer Distance and Earth Mover’s Distance and sums prediction errors across steps.Source: §3.2 · p. 4

3.3 Closed-Loop Control

  • A PointNet++ classifier scores 15 tools from the dough’s input and target states, and the robot plans actions for the three highest-probability candidates before choosing the closest predicted final state.Source: §3.3 · pp. 4–5
  • For grippers, rollers, presses, and punches, random real-world exploration trains GNN models, which generate synthetic data for self-supervised policies; other tools use straightforward planners based on human priors.The robot repeats planning with visual feedback after each manipulation.Source: §3.3 · p. 5

4 Experiments

RoboCook is evaluated on dumpling making and alphabet-letter cookie shaping, including tests of robustness to disturbances and generalization across materials. The experiments also compare letter shaping with four baselines and assess quantitative and human judgments of results.

4 Experiments

  • The experiments demonstrate long-horizon manipulation with multiple interchangeable tools and report robustness to substantial human interference that changes dough shape and volume.The tasks include making dumplings and shaping alphabet-letter cookies.Source: §4 · p. 6

4.1 Making Dumplings

  • The dumpling task requires dough to form a skin thin enough to completely cover the filling, and RoboCook reliably completes it.The experiment also compares its manipulation results with human subjects in supplementary materials.Source: §4.1 · p. 6
  • RoboCook succeeds after disturbances that deform or add dough, and even after the original dough is replaced with an irregular piece.In the replacement case, it restarts with a knife and successfully makes a dumpling.Source: §4.1 · p. 6

4.2 Making Alphabet Letter Cookies

  • RoboCook shapes dough closer to the target than four baselines, and Table 1 reports better performance than all baselines across its listed metrics.The table covers Chamfer Distance, Earth Mover’s Distance, surface-normal Chamfer Distance, and action-planning time.Source: Figure 5 · p. 7, Table 1 · p. 7
  • RoboCook shapes R, O, B, C, and K to compose “RoboCook,” using different grippers and a circle punch for letter-specific features.For C, closed-loop visual feedback supports grip placement despite the large distance between the initial and target shapes.Source: §4.2 · p. 7
  • The letter-shaping evaluation includes Chamfer Distance, Earth Mover’s Distance, surface-normal Chamfer Distance, and human letter-recognition accuracy.The authors note that the geometric metrics are prone to local noise, while people capture holistic dough features; the policy also plans faster than methods that repeatedly run dynamics models.Source: §4.2 · p. 8
  • The dynamics model shapes a K on Play-Doh, Air Dry Clay, and Model Foam without retraining, despite being trained only on the original flour, water, and salt dough.The tested materials have notably different dynamics from the training dough.Source: §4.2 · p. 8, Figure 6 · p. 8

5 Conclusion and Limitations

RoboCook combines tool-aware dynamics, tool selection, and self-supervised policy learning for elasto-plastic object manipulation. Its limitations include occasional tool sticking, human-specified action spaces and subgoals, and additional topology-estimation needs for cables and cloth.

5 Conclusion and Limitations

  • RoboCook’s contributions include tool-aware GNN dynamics, tool selection learned through self-exploratory trials, and self-supervised policy learning.The conclusion describes these components as improving dynamics modeling and manipulation performance and speed.Source: §5 · p. 8
  • Dough occasionally sticks to the tool, and the authors identify automatic error correction as a potential solution.Source: §5 · p. 8
  • RoboCook relies on human priors to specify tool action spaces, and humans define its subgoals.The authors say higher-level temporal abstraction and task-level planning are needed to eliminate human-defined subgoals.Source: §5 · p. 8
  • Applying RoboCook to cables and cloth requires additional topology estimation, which is outside this work’s scope.Source: §5 · p. 8

6 Supplementary Materials

RoboCook’s modules transform camera observations and tool poses into dough-and-tool point clouds, predict dough dynamics and select tools, then produce tool-conditioned actions. Training data comes from per-tool interaction and tool-selection point clouds, with preprocessing heuristics for frame consistency.

6.1 Inputs and Outputs of RoboCook’s Respective Modules

  • The perception module combines four-camera point clouds and the robot pose into a sparse dough surface cloud concatenated with a sampled tool-surface cloud.The tool cloud pose is computed from the end-effector pose and tool geometry.Source: §6.1 · p. 13
  • The dynamics model predicts the next state from the current state and action, while inference uses the initial state and an action sequence to predict the final state.For gripping, pressing, and rolling, the prediction horizon is 15 steps.Source: §6.1 · p. 13
  • The tool predictor maps current and target dough point clouds to the next tool, and the policy maps current dough and its tool-linked subgoal to an action.Each tool has a corresponding subgoal, such as a rolling pin paired with flattened dough.Source: §6.1 · p. 13

6.2.1 Data Collection

  • GNN training uses about 20 minutes of real-world interaction data per tool, collected through random actions in predefined spaces and manually reset episodes.The dough is reshaped after every five grips, three rolls, and three presses; tool data collection uses different episode and sequence counts by tool group.Source: §6.2.1 · p. 14
  • Tool-classifier training reuses point clouds from tool interactions, including nonconsecutive pairs and demonstrations from ten executions of a pre-coded dumpling pipeline.Most tool-selection data is reused from dynamics-model training, avoiding extra exploration.Source: §6.2.1 · p. 14

6.2.2 Data Preprocessing

  • Preprocessing preserves particle correspondence when the tool is outside the dough’s convex hull and subsamples videos to 16 frames each.The same sampled particles are retained from the preceding frame when the tool is not in contact or moves away.Source: §6.2.2 · p. 14

6.3 Dynamics Model

The dynamics and policy pipeline uses locally constructed particle graphs, temporally abstracted predictions, and synthetic action data. Tool-specific action spaces and overlapping-bin action prediction support self-supervised policy training, while demonstrations define subgoals.

6.3.1 Graph Building

  • Dough-particle edges connect nearest neighbors within 0.1 cm, while each tool particle has at most four undirected edges to dough particles.Local-neighborhood features make the GNN translation-invariant, supporting predictions regardless of absolute world location.Source: §6.3.1 · p. 14

6.3.2 Model Training

  • Temporal abstraction trains predictions across multiple frames, such as predicting t = 3 directly from t = 0, shortening the horizon and reducing forward passes.The stated goals are to ease the task and improve inference speed.Source: §6.3.2 · p. 14

6.3.3 Building Synthetic Datasets

  • Synthetic policy data pairs randomly sampled actions with predicted dough states, using 128 actions for each of 300 initial gripping states.This yields 38,400 input-output point-cloud pairs; starting states span varied shapes rather than only block-shaped dough.Source: §6.3.3 · pp. 14–15

6.4.1 Action Space

  • Tool action spaces are categorized by dimension: gripping, rolling, pressing, and punching use parameterized spaces, while the hook action is precoded.Grippers use {r, θ, d}; rollers and square tools use {x, y, z, θ}; circular tools use {x, y, z}; several other tools use {x, y}.Source: §6.4.1 · p. 15
  • Action spaces remove redundant degrees of freedom and add tool-specific parameters, but hand-designed spaces may be undesirable in general manipulation settings.The authors state that new tools can be assigned action spaces for transfer; automatically pruning spaces by tool and task is a proposed direction.Source: §6.4.1 · pp. 15–16
  • RoboCook’s approach can transfer to other dough tasks with ground-truth meshes of new tools and 20 minutes of interaction data per tool.The text names alphabet cookies, pizza, and noodles as potential tasks, while only alphabet cookies are shown in the paper.Source: §6.4.1 · pp. 15–16

6.4.3 Subgoal Definitions

  • Subgoals are selected from the point clouds recorded after each tool use in ten hand-coded dumpling-making demonstrations.Source: §6.4.3 · p. 16

6.5 Experiment Setup

The experiment setup uses a Franka Panda robot with four RGB-D cameras and a collection of 3D-printed dough tools. A tool-switching design lets the robot pick up, use, and put away tools without human assistance.

6.5.1 Robot and Sensors

  • The setup uses a 7-DoF Franka Panda arm and four calibrated D415 cameras capturing 1280×720 RGB-D images at 30 Hz.Source: §6.5.1 · p. 16

6.5.2 Tool Design

  • The team designed 14 3D-printed tools to mimic common kitchen-tool prototypes, with varied geometries for rolling, cutting, pressing, gripping, spatula handling, and dumpling molding.The printed tools are used instead of real tools to make acquisition and manipulation easier for the robot arm.Source: §6.5.2 · pp. 16–17, Figure 7 · p. 17

6.5.3 Tool-Switching Setup

  • Tool connectors attach to and release from the Franka gripper, while upright racks and calibrated world coordinates support autonomous tool pickup and placement.Source: §6.5.3 · p. 17

6.6 Comparison with Human Subjects on Dumpling-making

In dumpling-making comparisons, RoboCook performed better than human subjects choosing tools and subgoals independently, while human performance with a prescribed sequence and subgoals was comparable to or worse than RoboCook. The comparison involved five subjects and constrained tool use.

6.6 Comparison with Human Subjects on Dumpling-making

  • Human subjects performed notably worse than RoboCook when choosing tools and plans independently, while prescribed tool sequences and subgoals brought their results to comparable or worse performance.The comparison is shown in Figure 9.Source: §6.6 · p. 17, Figure 9 · p. 18
  • Five subjects completed both planning conditions, and each subject’s best of three trials was compared with RoboCook under limits on hand contact and tool use.Subjects could not touch the dough directly or apply any tool more than five times.Source: §6.6 · p. 17
  • The tool classifier’s test-set accuracy was very close to 1, as reported with its confusion matrix.Source: Figure 8 · p. 18

6.7 Tool Classification

RoboCook’s PointNet-based tool classifier achieves high instance accuracy and generalizes better than a ResNet-based alternative. Its depth encoding is associated with robustness to lighting, dough color, and dough transformations.

6.7 Tool Classification

  • 0.996 instance accuracy is reported for the tool classifier.The classifier is trained with supervised learning and a cross-entropy loss.Source: §6.7 · pp. 18–19
  • PointNet-based classification generalizes better than ResNet-based classification, attributed to its ability to encode depth information.Source: §6.7 · p. 19
  • The PointNet-based classifier is empirically more robust to changes in lighting, dough color, and dough transformations.Source: §6.7 · p. 19

6.8 Human Evaluation of Alphabet Letters

The human evaluation addresses a mismatch between geometric metrics and holistic human perception of dough shapes. It measures how well people recognize the intended alphabet letters and ranks the methods using responses from 100 subjects.

6.8 Human Evaluation of Alphabet Letters

  • The evaluation notes that Chamfer Distance can be prone to local noise, whereas people capture holistic dough features.Source: §6.8 · p. 19
  • One hundred human subjects were asked which alphabet letter the robot was trying to shape in each image.Source: §6.8 · p. 19
  • To reduce predictive bias, the 20 images were shuffled and split into four groups, each containing one image per letter from different methods.Human perceived accuracy and method rankings were averaged over five letters.Source: §6.8 · p. 19

6.9 Baseline Implementation Details

RoboCook’s policy outperforms alternative planning methods using the same learned dynamics models and tool predictors. The authors attribute this result to offline action-space coverage and describe an additional advantage over a simulator-based baseline.

6.9 Baseline Implementation Details

  • RoboCraft, CEM+GNN, and RL+GNN retain RoboCook’s GNNs and tool predictor but replace its policy with gradient descent, CEM, and RL, respectively.Source: §6.9 · p. 19
  • The RL+GNN baseline uses a learned GNN dynamics model with Soft Actor-Critic and is reported as noticeably worse than RoboCook.It was trained online for 2500 steps per action prediction to fit the closed-loop control module.Source: §6.9 · p. 19

6.10 Analysis of Comparison with Baselines

  • RoboCook outperforms RoboCraft, CEM+GNN, and RL+GNN by a large margin while using the same GNNs and tool predictors.The authors attribute this to offline synthetic data covering the action space and more efficient online action synthesis.Source: §6.10 · p. 19
  • The PointNet-based policy can capture higher-level shape changes such as concavities when comparing the current and target dough configurations.The authors illustrate this with pinching concave locations in the target shape.Source: §6.10 · p. 19
  • RoboCook also outperforms CEM+MPM; the authors cite the simulator’s sim2real gap and need for careful system identification, contrasting it with a GNN trained directly on real-world data.Source: §6.10 · p. 19
Loading 2306.14447v2…