PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances

Aidan Curtis, Xiaolin Fang, Leslie Pack Kaelbling, Tomás Lozano-Pérez, Caelan Reed Garrett

arXiv:2108.04145v2cs.ROcs.AI

TL;DR

The paper addresses manipulation of previously unseen objects in partially observable scenes, where goals may require many coordinated actions. It combines task-and-motion planning with perception modules to estimate object properties and affordances, then demonstrates real-world multi-step manipulation across broad classes of objects, arrangements, and goals without retraining.

  • Problem

    The challenge is to manipulate partially observed, previously unseen objects toward goals that may require many coordinated actions.

    Source: §I · p. 1

  • Method

    The system combines task-and-motion planning with engineered and learned perception modules in a closed-loop policy for manipulation.

    Source: §I · p. 1, §X · p. 11

  • Results

    The system performs multi-step manipulation across broad classes of objects, arrangements, and goals, including scenes where re-perception reveals previously hidden objects.

    Source: §I · pp. 1–2, §X · p. 11

  • Takeaways & Limitations

    A modular system can support varied manipulation operations, with new operations combined with existing ones to achieve new goals.

    Source: §I · p. 2

    This paper focuses on prehensile pick-and-place manipulation, rather than the other operations the framework has previously implemented.

    Source: §IV · p. 3

Abstract

from arXiv · show

We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closed-loop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable objects, their geometries, and their affordances. We explore several different ways of implementing such perceptual modules for segmentation, property detection, shape estimation, and grasp generation. We show how these modules are integrated within the PDDLStream task and motion planning framework. Finally, we demonstrate that this strategy can enable a single system to perform a wide variety of real-world multi-step manipulation tasks, generalizing over a broad class of objects, object arrangements, and goals, without any prior knowledge of the environment and without re-training.

I. INTRODUCTION

The paper proposes a general manipulation policy combining task-and-motion planning with engineered and learned perception, so it can plan for unfamiliar objects and goals from sensory data. Its closed-loop, modular design supports multi-step behavior, re-planning after new observations, and extension without retraining.

I. INTRODUCTION

  • The target policy must safely handle partially observed, previously unseen objects and goals that may require many coordinated actions.The stated objective also includes cumulative acquisition of competencies through incremental engineering and learning.Source: §I · p. 1
  • The approach combines task-and-motion planning with engineered and learned perception to manipulate novel objects in novel arrangements toward novel goals.The system resolves planning queries from observed point clouds rather than requiring prior object models.Source: §I · p. 1, §II · p. 2
  • After initially placing a cracker box, the robot re-perceives two hidden objects and plans collision-free placements for all three objects.It temporarily moves the box to make room for the tape measure and cup before choosing a final collision-avoiding placement.Source: §I · pp. 1–2, Figure 1 · p. 1

II. APPROACH

  • The modular system is designed to generalize without specialized re-engineering or retraining and to acquire new competencies by adding modules.New operations such as pushing or pouring can be combined with existing operations to achieve new goals.Source: §II · p. 2, §I · p. 2
  • M0M estimates the scene, plans multiple steps toward the goal, executes initial steps, then reobserves and replans if needed.It can recover from perceptual errors through continual re-perception and re-planning.Source: §II · p. 2

III. RELATED WORK

The related-work discussion contrasts M0M’s general task-and-motion planning with prior systems that depend on known object poses or narrower rearrangement planners. The framework focuses here on prehensile pick-and-place manipulation, supported by robot-specific setup and perception modules that estimate objects, surfaces, and affordances.

III. RELATED WORK

  • Gualtieri et al. use analogous grasping and shape-estimation components but assume a task-specific rearrangement planner, unlike M0M’s more general task-and-motion planning.Their approach does not consider broader manipulation such as moving objects out of the way or more complex goals enabled by TAMP.Source: §III · pp. 2–3
  • Related work also includes interaction-based information gathering, interactive segmentation, and POMDP studies of object search under partial observability.Some cited work learns policies to uncover hidden objects in piles.Source: §III · p. 3

IV. MANIPULATION WITH ZERO MODELS

  • M0M focuses on prehensile manipulation: grasping objects rigidly, moving them without contacting other objects, and placing them stably on a surface.Other operations, including pushing, pouring, scooping, and unscrewing bottle caps, had been implemented previously, but are outside this paper’s focus.Source: §IV · p. 3
  • The framework requires a robot kinematic description and position controller, plus RGB-D perception modules that identify object hypotheses and support surfaces.Further modules estimate grasps, collision volumes, stable orientations, and goal-relevant object properties; their representations can vary by task.Source: §IV · p. 3

V. M0M USING PDDLSTREAM

M0M uses PDDLStream to combine manipulation actions with engineered and learned perceptual operations, allowing planning from segmented point clouds rather than known object models. Its formulation represents action constraints, perception queries, and goals—including quantified goals—in a common planning framework.

A. Overview

  • PDDLStream treats streams as black-box samplers that generate continuous values, combining engineered and learned operations during planning without requiring a shared implementation approach.The planner can query only operations relevant to the problem, lazily avoiding unnecessary perceptual computation.Source: A. Overview · pp. 3–4

B. PDDLstream formulation

  • PDDLStream models manipulation with PDDL actions and streams: actions specify preconditions and effects, while streams generate values satisfying declared constraints.The formulation describes move and place actions; the framework can also be implemented with other TAMP systems that provide a similar interface.Source: §V · p. 3, B. PDDLstream formulation · p. 4
  • Axioms are derived at each state, enabling PDDL planners to reason efficiently about complex goal conditions, including quantified specifications.The predicates used by actions and streams connect state descriptions to estimated perceptual properties.Source: B. PDDLstream formulation · p. 4, C. Predicate semantics · p. 4

C. Predicate semantics

  • M0M represents objects as segmented point clouds rather than named instances, with predicates encoding their poses, grasps, properties, and current relations to the robot and other objects.The initial state assigns each observed object cloud an identity pose and records the current robot configuration and empty hand.Source: C. Predicate semantics · pp. 4–5

D. Engineered and learned streams

M0M combines engineered robot-centric computations with learned or estimated object-centric streams. Its policy repeatedly plans from the current observation, executes one action, and observes again to check progress toward the goal.

D. Engineered and learned streams

  • Engineered streams use the robot's URDF for inverse kinematics and motion planning, while learned streams estimate grasps, stable placements, collisions, and object properties.Placement prediction combines point-cloud completion, plane estimation, shape estimation, and stable-orientation estimation; collision prediction uses estimated convex bodies.Source: D. Engineered and learned streams · p. 5

VI. MANIPULATION POLICY

  • The policy builds a PDDLStream problem from the segmented scene, robot configuration, goal, actions, and engineered and learned streams, then solves it for a plan.If a necessary attribute is not detected, the planner can return None to indicate that the goal is unreachable from the current state.Source: §VI · pp. 5–6
  • M0M executes the first action of a nonempty plan, reobserves the scene, and replans; an empty plan means the goal is satisfied.Reobserving prevents the policy from declaring success solely from an open-loop plan.Source: §VI · p. 6

VII. IMPLEMENTATION

The implementation combines PDDLStream with RGB-D perception modules for segmentation, shape estimation, and grasp generation. Comparisons across these modules informed the choices used in the system experiments.

VII. IMPLEMENTATION

  • The implementation uses RGB-D images from a PR2 Kinect 1 and standard position trajectory controllers, with gripper opening after closure used to detect grasp failures.Experiments compare alternative implementations of several perception modules.Source: §VII · p. 6

A. Segmentation of objects and surfaces

  • Segmentation identifies rigid point collections that move together; the implementation compares UOIS-net-3D, DBSCAN clustering, and a combined approach, while UOIS-net-3D can split a contacting cracker box.The combined method applies DBSCAN to UOIS-net-3D output and filters degenerate clusters.Source: A. Segmentation of objects and surfaces · p. 6
  • Across ARID-20 and GraspNet-1Billion, Euclidean clustering favored simple, scattered scenes, learned segmentation often favored clutter, and the combined method beat pure learned segmentation.The combined method was used in the system experiments.Source: A. Segmentation of objects and surfaces · pp. 6–7

B. Shape estimation

  • Shape estimation takes partial point clouds and combines neural completion with geometric projection, then filters predictions against the observed depth image.The geometric heuristic projects visible points onto the table plane.Source: B. Shape estimation · p. 7

1) Mesh interpolation:

  • An alpha-shape concave hull avoids the substantial volume overestimation convex hulls can cause for non-convex objects, while V-HACD approximates the mesh for collision checking.The hull uses visible, network-predicted, and projected points.Source: 1) Mesh interpolation · p. 7
  • VLPF generally performed best across four domains, although table projection could overestimate volume when the true base was smaller than the visible base.The evaluation used 2,000 images per domain.Source: 1) Mesh interpolation · p. 7

C. Grasp affordances

  • Grasp affordances are local hand-to-object poses for stable grasps; GPD and GraspNet use partial point clouds, while EMA samples antipodal contacts on an estimated mesh.EMA rejects contact pairs exceeding gripper width or with unsuitable surface normals, and checks for collision with the estimated mesh.Source: C. Grasp affordances · pp. 7–8
  • GPD and EMA outperformed GraspNet in speed and accuracy, with EMA edging GPD; the system experiments used EMA.The three approaches were compared in a real-world experiment.Source: C. Grasp affordances · p. 8

D. Object properties

M0M combines property and grasp perception with closed-loop planning to solve real-world manipulation tasks. Across five repeated trials for each of five tasks, it succeeded in every trial except one Task 3 trial.

D. Object properties

  • Property detection covered category and color, using learned detectors for bowls, cups, and goal-mentioned YCB objects alongside color statistics from segmented images.The bowl/cup detector was trained on the bowl-cup subset of IIT-AFF; the YCB classifier used real YCB Video and synthetic PyBullet data.Source: D. Object properties · p. 8
  • GPD and EMA outperformed GraspNet in speed and accuracy, with EMA edging GPD and serving in the system experiments.Figure 7 shows grasps from all three modules on point clouds across scenes with varying clutter.Source: Figure 7 · p. 8

VIII. WHOLE-SYSTEM EXPERIMENTS

  • For a mustard-bottle goal, M0M moved two obstructing objects before placing the bottle, and replanned after an initial grasp attempt failed.The system aborted, re-observed, and executed a new successful grasp.Source: §VIII · p. 8, Figure 8 · p. 9

A. Repeated trials

  • Reported iterations include estimation and planning, while average estimation, planning, and execution times are reported per iteration.Extra iterations may follow undersegmentation, an aborted failed grasp, or an action with unanticipated effects.Source: A. Repeated trials · p. 9
  • The system achieved the goal in every trial except one Task 3 trial, demonstrating robust performance across diverse long-horizon manipulation tasks.Tasks were repeated five times each; the system was also applied to over 25 individual problem instances.Source: A. Repeated trials · p. 9

1) Task 1:

The four tasks test novel-object placement, collision-aware planning, response to unexpected observations, and target selection by color. The reported Task 3 trial is the only failure among the five tasks.

1) Task 1:

  • Task 1 tested stable placement of novel, non-box-like objects; its five trials used a bowl, power drill, plastic banana, cup, and tennis ball.Figure 10 depicts a successful bowl trial in which the robot grasped the bowl's interior.Source: 1) Task 1 · p. 9, Figure 10 · p. 9

2) Task 2:

  • Task 2 tested placing multiple objects in a tight region, requiring collision-avoiding placements and motions for a mustard bottle and toy drill.In the depicted successful trial, the drill was placed first and the mustard bottle in the remaining collision-free area.Source: 2) Task 2 · p. 9, Figure 11 · p. 10

3) Task 3:

  • Task 3 had the only failed trial: lifting the cracker box displaced an occluded tennis ball beyond reach, and replanning could not achieve the goal.After re-observation, the robot identified the objects and recognized that the goal remained unsatisfied.Source: 3) Task 3 · p. 10, Figure 12 · p. 10

4) Task 4:

  • Task 4 tested collision reasoning for a color-selected target: the robot moved objects covering the blue region to place the red-hued toy drill there.The goal selected the object closest in hue to red.Source: 4) Task 4 · p. 10, Figure 13 · p. 10

5) Task 5:

Task 5 demonstrates moving obstructions to reach a color-selected object, while the extensions describe additions and boundaries of the M0M framework. The conclusion characterizes the system as a closed-loop combination of analytical and empirical methods for open-world sequential manipulation.

5) Task 5:

  • To reach the mustard bottle, M0M relocated obstructing bottles; after the detergent bottle fell during placement, it inferred this at the next observation and continued successfully.The goal selected the object closest in color to yellow.Source: 5) Task 5 · p. 10, Figure 14 · p. 10

IX. EXTENSIONS TO M0M

  • Recognizing an object category can support inference of latent properties and parts; identifying a bowl, for example, supports inferring that objects can be dropped into it.The planning model can be augmented to infer additional properties from perceived properties or class membership.Source: §IX · pp. 10–11
  • Adding operations such as pushing requires a controller, an operator description of predicted pose changes, and a sampler for feasible push paths.The added operations can combine with prehensile actions to generate richer plans; analogous characterizations are described for pouring and scooping.Source: §IX · p. 11
  • The basic system has no memory and selects actions from the current view; robustness would benefit from integrating observations over time and predicted action effects.The authors also note potential benefits from tactile and auditory sensing.Source: §IX · p. 11
  • The current perception system gives the planner a single state hypothesis, so it cannot account for uncertainty when deciding whether to gather information before pursuing the goal.The authors identify integration with belief-space TAMP approaches as future work.Source: §IX · p. 11

X. CONCLUSION

  • The demonstrated policy maps images to position commands and generalizes across broad classes of objects, arrangements, and goals, solving more open-world sequential problems than purely analytical or empirical methods.The conclusion describes the system as combining analytical and empirical methods.Source: §X · p. 11

APPENDIX

The appendix details the PDDLStream operators and perception modules used by M0M, then evaluates segmentation and shape-estimation strategies. The shape-estimation comparison favors combining visible, projected, and learned points with filtering, with results conditioned by the tested scene setup.

A. PDDLStream

  • PDDLStream defines operators and streams for manipulation, including a stream that predicts whether a held-object trajectory avoids collision with another posed object.The prediction certifies a collision-free trajectory when the specified conditions are met.Source: A. PDDLStream · p. 13

B.1 Segmentation

  • The combined segmentation method uses neural networks for table and object segmentation and returns separate table and object point clouds.The comparison evaluates three approaches on ARID-20 and GraspNet-1Billion images, reporting F-measure.Source: §B.1 · p. 13, Figure 17 · p. 14

B.2 Surfaces

  • Surface estimation uses RANSAC to find a plane, then builds a polygonal surface from nearby points when the plane meets orientation and distance tolerances.The surface is represented by a plane pose and a polygon within that plane.Source: §B.2 · p. 13

B.3 Shape estimation

  • Shape estimation compares six strategies that form meshes from visible points, table-projected points, learned completions, and filtered combinations of these inputs.The MSN completion model was trained on rendered views of 14 YCB object models and evaluated on a larger variety of objects.Source: §B.3 · p. 13
  • The fully combined VLPF method generally performs best across the tested domains, while filtering improves learned and projection-based predictions.Visible points alone perform worst in each domain; the experiments use stable objects on flat surfaces viewed from above, which likely contributes to VP outperforming VL before filtering.Source: §B.3 · p. 14, Figure 18 · p. 14

B.4 Grasps

The appendix describes EMA, a mesh-based grasp generator, and compares it with GPD and GraspNet in real-world trials. GPD and EMA both succeed more often than GraspNet in uncluttered trials, while EMA has the highest success rate in clutter.

B.4 Grasps

  • EMA samples antipodal surface-point pairs on an estimated mesh, generates grasp poses, and rejects poses that collide with the end effector.The algorithm yields a possibly infinite sequence of candidate grasps.Source: §B.4 · pp. 14–15
  • In clutter, EMA has a higher success rate than GPD and GraspNet, possibly because its mesh-based generator can consider object-gripper collisions.All three methods performed worse in clutter, where segmentation undersegmentation and collisions can make grasping harder.Source: §B.4 · p. 15
  • 27 out of 35 attempts succeeded for both GPD and EMA in uncluttered trials, outperforming GraspNet.The evaluation counts a grasp as successful when the robot cannot fully close its fingers after picking and lifting; 35 trials were run in each setting.Source: Figure 20 · p. 15, §B.4 · p. 15

C. Demonstrations

The demonstrations qualitatively show M0M solving multi-object placement tasks with goals expressed over objects and target regions. The examples include packing objects, changing an object's resting face, and using multiple target regions to disperse objects.

C. Demonstrations

  • The demonstrations are qualitative examples of M0M solving individual problems.The appendix points readers to videos of the tasks.Source: C. Demonstrations · p. 16

1) Packing three objects:

2) Changing the resting face:

4) Manipulation in clutter:

The demonstrations show M0M handling clutter, occlusion, and obstructed placement goals through different placement and rearrangement plans. In clutter, it moves between large objects to pick and place a mustard bottle while avoiding collisions.

4) Manipulation in clutter:

5) Occluded objects:

6) Obstructed place:

  • For obstructed placement, the robot moves a sugar box off the blue target region to make room for a mustard bottle.The mustard bottle's goal is to be on the blue target region.Source: Figure 26 · p. 17, 6) Obstructed place · p. 17

7) Moving a destructible object:

The system handles a destructible object by perceiving newly created pieces and replanning so each reaches the blue target. Additional demonstrations show a two-arm re-grasp strategy and long-horizon color-based sorting of closely placed objects.

7) Moving a destructible object:

8) Sorting:

  • The robot successfully sorts closely placed objects into bowls according to which bowl is closest in color to each object.The task requires a long-horizon sequence of picks.Source: 8) Sorting · p. 17, Figure 29 · p. 18

Related papers

Loading 2108.04145v2…