Source-linked AI summary
A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois R. Hogan, Joshua Tenenbaum, Pulkit Agrawal, Alberto Rodriguez
TL;DR
Long-horizon manipulation of unfamiliar rigid objects from sensory observations requires planning over feasible subgoals while accounting for how the robot can achieve them. This paper combines point-cloud-conditioned subgoal and contact sampling with primitive manipulation skills, and reports improved planning efficiency over privileged-knowledge baselines alongside simulated and real-world demonstrations.
Problem
Long-horizon manipulation methods either rely on object-shape-informed TAMP heuristics or sensory-based approaches that can struggle to plan across multiple skills.
Source: §1 · p. 2Method
The framework learns point-cloud-conditioned distributions over feasible object subgoals and contact locations for primitive skills, representing rigid-body subgoals as transformations under sticking contact.
Source: §1 · p. 2Results
The method outperforms carefully designed baselines with privileged geometry and task knowledge in planning efficiency on simulated and real YuMi experiments.
Source: §1 · p. 3Takeaways & Limitations
The approach manipulates previously unseen object geometries into target configurations requiring long-term, sequential skill execution.
Source: §1 · p. 3The formulation assumes rigid segmented point-cloud objects, sticking contact, quasi-static interactions, and a supplied plan skeleton.
Source: §2 · p. 3
Abstract
from arXiv · showhide
We present a framework for solving long-horizon planning problems involving manipulation of rigid objects that operates directly from a point-cloud observation, i.e. without prior object models. Our method plans in the space of object subgoals and frees the planner from reasoning about robot-object interaction dynamics by relying on a set of generalizable manipulation primitives. We show that for rigid bodies, this abstraction can be realized using low-level manipulation skills that maintain sticking contact with the object and represent subgoals as 3D transformations. To enable generalization to unseen objects and improve planning performance, we propose a novel way of representing subgoals for rigid-body manipulation and a graph-attention based neural network architecture for processing point-cloud inputs. We experimentally validate these choices using simulated and real-world experiments on the YuMi robot. Results demonstrate that our method can successfully manipulate new objects into target configurations requiring long-term planning. Overall, our framework realizes the best of the worlds of task-and-motion planning (TAMP) and learning-based approaches. Project website: https://anthonysimeonov.github.io/rpo-planning-framework/.
1 Introduction
The paper addresses efficient long-horizon manipulation of unfamiliar objects from sensory observations by planning over feasible object subgoals and predicting how to achieve them. It combines point-cloud-based learning with manipulation primitives and reports planning-efficiency gains on simulated and real YuMi experiments.
1 Introduction
- Planning over object subgoals can reduce the long-horizon search space, but infeasible subgoals make decoupled high- and low-level planning inefficient.The framework predicts subgoals and how to achieve them while biasing search toward reachable options.Source: §1 · p. 1
- Conventional TAMP subgoal heuristics can depend on prior object-shape information, while sensory-based manipulation methods can struggle with long-horizon tasks.The paper seeks to retain generalization to novel objects while planning across multiple skills.Source: §1 · p. 2
- For rigid bodies under sticking contact, an object subgoal is an SE(3) transformation that the manipulator can execute after establishing contact.This reduces predicting the action sequence to predicting the initial contact configuration.Source: §1 · p. 2
- Self-supervised data from privileged manipulation of cuboidal objects trains point-cloud samplers to predict compatible feasible subgoals and contact locations.The sampler predicts distributions over both quantities for each skill.Source: §1 · p. 2
- The framework combines point-cloud planning for rigid bodies with geometry-grounded reorientation subgoals, graph attention, and joint prediction of subgoals and their execution.These design choices are presented as improving planning performance and efficiency.Source: §1 · p. 3
- On simulated and real-world YuMi experiments, the method outperforms carefully designed baselines using privileged geometry and task knowledge in planning efficiency.The experiments involve previously unseen object geometries and target configurations requiring sequences of skills.Source: §1 · p. 3
2 Problem Setup and Multi-step Planning
The planner searches over object transformations and contact poses for a given sequence of manipulation skills. Its formulation relies on rigid, segmented point-cloud objects and sticking contact, which lets each skill use the planned object transformation as the manipulator motion after contact.
2 Problem Setup and Multi-step Planning
- The formulation assumes rigid objects represented by segmented point-clouds, SE(3) subgoals, sticking contact, quasi-static interactions, and a supplied plan skeleton.It also assumes segmentation of the entire point-cloud and that the target is reachable within the skeleton's steps.Source: §2 · p. 3
- Under sticking contact, planning the contact pose and object subgoal is sufficient because the manipulator applies the same transformation after contacting the object.The framework reuses six skills: left/right pulling, left/right pushing, pick-and-place, and reorientation.Source: §2 · p. 3
- Each skill's inverse-kinematics controller produces joint configurations, with feasibility requiring collision avoidance, joint-limit compliance, and avoidance of singularities.The resulting joint-sequence length varies with skill type and inputs.Source: §2 · p. 3
- For a given plan skeleton, the planner chooses each skill's object transformation and contact poses to move the segmented point-cloud toward the target configuration.The object point-cloud at the next step is obtained by applying the selected object transformation.Source: §2 · p. 3
- A learned sampler proposes subgoal transformations and contact poses for the current point-cloud, and infeasible proposals can be discarded.The planner samples sequences until it finds a feasible one or times out; the target transformation can be analytically solved.Source: §2 · p. 3
3 Learning the Skill Sampler
The skill sampler uses a CVAE to predict contact poses and object subgoals from point clouds. Its modeling choices encode object geometry, ground reorientation predictions in table contact, and jointly predict compatible skill parameters.
Learning the Skill Sampler
- For each skill, a CVAE predicts a distribution over contact pose and object subgoal from a point-cloud observation, using successful single-step executions as training data.Sampling from the learned distribution is intended to make high-level planning efficient while retaining generalization from sensory observations.Source: §3 · p. 4
Learning Pointcloud Features for Predicting Contact Poses
- Graph attention models pairwise point relationships, while per-node contact-pose prediction encourages features to capture global geometry.At test time, the model averages the per-node contact-pose predictions.Source: §3 · p. 4
Obtaining Accurate Subgoal Prediction
- Reorientation is predicted by labeling object points expected to touch the table, then registering that mask to the table with ICP to infer orientation.The inferred orientation is combined with predicted translation to obtain the object transformation.Source: Figure 3 · p. 4, §3 · p. 5
Modeling the joint distribution of Subgoals and Contact Points
- A shared latent distribution jointly predicts subgoals, contact poses, and reorientation masks, helping produce mutually compatible skill parameters.The decoder uses shared self-attention layers and separate output heads.Source: §3 · p. 5
Training Data
- The training dataset contains 24K samples from cuboids of different sizes, collected by executing one primitive skill at a time.Separate CVAEs with the same architecture were trained for each skill.Source: §3 · p. 5
4 Experiments and Results
The learned samplers improve multistep planning efficiency over hand-designed cuboid-based samplers, while single-step evaluations favor GAT, mask-based reorientation subgoals, and joint modeling. Qualitative demonstrations include shelf placement, novel object geometries, and real-robot execution.
4.1 Performance Evaluation on Single-Step Manipulation Tasks
- GAT outperforms PointNet++ on grasping and pushing, while both perform similarly on the easier pulling task.PointNet++ predictions more often misalign the robot’s palms with the object, producing less stable edge contacts and inaccurate or infeasible manipulation.Source: §4.1 · pp. 5–6, Figure 4 · p. 6
- Mask-based reorientation subgoals fix physically infeasible transformations caused by directly predicting T o, while direct T o prediction suffices for pull and push.Pull and push operate in SE(2) and do not share this error mode.Source: §4.1 · p. 6, Figure 5 · p. 6
- Jointly modeling subgoals and contacts yields significant performance gains over modeling them independently.The comparison includes grasping, pushing, and pulling success rates.Source: §4.1 · p. 6, Figure 4 · p. 6
4.2 Performance Evaluation on Multi-step Manipulation Tasks
- Learned samplers provide a large planning-efficiency benefit over hand-designed cuboid-based samplers, while feasible plans from both have similar pose errors.The baseline performs worse primarily because it samples many infeasible sequences; the learned model biases samples toward likely feasible parameters.Source: §4.2 · p. 7, Table 1 · p. 7
4.3 Qualitative Results and Capabilities
- The mask-based representation supports a five-step move from a table to a vertical pose on an elevated shelf.The demonstrated shelf subgoal is sampled using the segmentation mask-based reorientation representation.Source: §4.3 · p. 7, Figure 6 · p. 7
- Grasp-reorient predictions trained on cuboids were executed successfully in simulation on cylindrical and bottle-shaped objects, and the planner found a three-step bottle plan.Real ABB YuMi demonstrations include skills trained only in simulation and a two-step plan on a bottle; noisy simulated point-clouds caused negligible performance change.Source: §4.3 · pp. 7–8, Figure 7 · p. 7, Figure 8 · p. 8
5 Related Work
Related work spans learned perception and affordances, manipulation planning, and methods for guiding TAMP. This framework maps sensory observations to distributions over promising skill parameters while planning through object subgoals.
Perception-based Multi-step Manipulation
- Fang et al. [24] also learn samplers for multistep manipulation from segmented point-clouds, but learn forward simulation in both subgoal and action spaces and consider planar straight-line pushing.This framework instead propagates SE(3) subgoals and avoids learning low-level action dynamics.Source: §5 · p. 8
Task-Oriented Contact-Affordances
- Task-oriented contact-affordance methods select or learn contacts for downstream tasks, whereas this work models the joint distribution of reachable subgoals and corresponding contacts for primitive skills.The cited approaches include task-reward grasp selection, keypoint affordances [8] [23], and push affordances ranked with a mechanics model [9].Source: §5 · p. 8
Manipulation with 3D Deep Learning
Learning to guide TAMP
- TAMP-guidance methods learn parameters for whole plan skeletons or skill-specific samplers [38], often assuming object poses and shapes are perfectly estimated.This framework extends the setting to unknown object shapes and poses by mapping sensory observations to distributions over manipulation-skill parameters.Source: §5 · p. 8
6 Conclusion
The paper presents point-cloud-conditioned sampling for multistep planning with primitive manipulation skills. Its learned samplers improve planning efficiency over a manually designed baseline, and qualitative results show transfer to real point-clouds, more complex geometry, and multistep tasks.
6 Conclusion
- The framework maps segmented object point-clouds to distributions over likely contact poses and subgoals for primitive-skill multistep planning.Source: §6 · p. 8
- Learned samplers provide planning-efficiency gains over a manually designed baseline, while qualitative results show transfer to real point-clouds, more complex geometry, and sophisticated multistep tasks.Source: §6 · p. 8
A.1.1 Point-cloud Registration
For mask-based reorientation, ICP registers the object to a segmented support-surface point-cloud to obtain a subgoal transformation. Pull and push instead use predicted transformations projected to SE(2).
A.1.1 Point-cloud Registration
- ICP registers the object to the segmented table point-cloud to obtain the mask-based subgoal transformation, initialized from predicted planar translation and a forward π/2 pitch.The paper notes that other registration techniques and initial transformations could also be used.Source: §A.1.1 · p. 11
- Pull and push subgoals are obtained by projecting the model’s predicted T o to SE(2), since these skills operate in SE(2).Source: §A.1.1 · p. 11
A.1.2 Sampling-based Planning Algorithm
The planner grows a buffered search tree over skill subgoals and contact poses, filtering candidate steps with point-cloud preconditions and motion-feasibility checks. It backtracks from a feasible final node to return the plan, or fails if no plan is found within five minutes.
A.1.2 Sampling-based Planning Algorithm
- Each search node stores a subgoal, contact pose, transformed point-cloud, and parent, while per-step buffers hold feasible nodes for later sampling.A randomly popped node supplies the point-cloud input for sampling at that step.Source: §A.1.2 · p. 11
- Point-cloud preconditions reject configurations likely to fail, using reachability checks such as whether the object’s average planar position lies within a valid table region.Grasp-reorient uses a valid region near the table’s front-center; pull and push check table boundaries.Source: §A.1.2 · p. 11
- A skill is feasible only if initial and final collision checks pass, followed by collision and kinematic checks on the planned motion.The more expensive motion checks run only after the cheaper endpoint checks pass; each step uses at most Kmax = 10 samples.Source: §A.1.2 · p. 11
- At the final step, the planner computes the required transformation from the accumulated subgoals; an infeasible final transformation restarts sampling at the first step.New start point-clouds continue to be added to the buffers as sampling restarts.Source: §A.1.2 · p. 11
- The planner backtracks from a feasible final node to return the plan; it returns failure if no feasible plan is found within five minutes.The returned plan contains the contact poses and object transformations for each step.Source: §A.1.2 · p. 11
A.1.3 Conditional Variational Auto-encoder Skill Samplers
The skill samplers use conditional variational auto-encoders (CVAEs) to model distributions over skill parameters from point-cloud observations. The planner can sample joint predictions or use independent models for subgoals and contact poses.
A.1.3 Conditional Variational Auto-encoder Skill Samplers
- The CVAE models the conditional distribution of output skill parameters given an observation, using a latent variable and neural-network encoder and decoder.The encoder approximates a latent posterior; the decoder models the output conditioned on the observation and latent variable.Source: §A.1.3 · p. 13
- At inference, the trained decoder samples a latent vector from its prior and generates skill parameters using the corresponding network and output heads.The ELBO objective is also described as the CVAE’s training objective in this setup.Source: §A.1.3 · pp. 11, 13
- Training optimizes the conditional log-likelihood’s evidence lower bound, combining a reconstruction term with a KL-divergence term that regularizes the encoder’s latent distribution toward a unit-Gaussian prior.The model uses the reparameterization trick to estimate the ELBO for gradient-based optimization.Source: §A.1.3 · p. 13
- Joint skill samplers predict the object transformation, contact pose, and point-cloud mask together from the point-cloud observation.The joint model represents their conditional distribution through a shared latent variable.Source: §A.1.3 · p. 13
- Independent-model baselines split outputs into two CVAEs: one predicts the object transformation and mask, and the other predicts the contact pose.Source: §A.1.3 · p. 13
A.1.4 Neural Network Architecture Details
The architecture details describe point-cloud preprocessing and CVAE models for predicting subgoals, contact poses, and masks. They include graph-attention and PointNet++ variants, with joint auxiliary prediction of transformations and masks reported to improve training and evaluation performance.
A.1.4 Neural Network Architecture Details
- Subgoals and contact poses are represented by 7D vectors containing a 3D position and a unit quaternion, which is normalized after prediction.The quaternion is regressed as an unnormalized 4D vector.Source: §A.1.4 · p. 13
- Jointly predicting the subgoal transformation and point-cloud mask improved overall training and evaluation performance, even when the transformation was not used to execute the skill.All decoder models used for subgoal prediction were set up to predict the transformation and mask jointly.Source: §A.1.4 · p. 13
- Before network input, point-clouds are uniformly downsampled, centered by subtracting their mean, and augmented with that mean as a feature for each point.Learned models use 100 points; baseline samplers use the full dense point-cloud.Source: §A.1.4 · p. 14
- In the GAT encoder, per-point input features include the point-cloud and the output variables, and graph-attention features are averaged to produce a global encoding for latent-distribution heads.The encoder predicts μ and σ, which are reparameterized to obtain the latent variable.Source: §A.1.4 · pp. 14–15
- The GAT decoder combines the point-cloud representation with the latent variable, then predicts contact poses and masks per point and the object transformation from pooled features.The contact-pose output head predicts one pose per point; a separate head predicts the binary mask.Source: §A.1.4 · p. 15
- The PointNet++ variant uses global point-cloud features for contact-pose and transformation predictions, with a separate pointwise head for mask prediction.Its encoder computes a global feature used to predict the latent distribution.Source: §A.1.4 · p. 15, Figure A2 · p. 15
- At test time, the decoder samples the latent vector from its prior; for the GAT model, one per-point contact-pose prediction is randomly selected for execution.Joint models use latent vectors of dimension 256; independent contact and subgoal models use dimension 128.Source: §A.1.4 · pp. 15–16
- The grasp-reorient ablation figure reports position and orientation errors, contact-maintenance fraction, and the fraction finding feasible motion within 15 learned-model samples.Source: Figure A3 · p. 16
A.1.5 Training Data Generation and Training Details
Training examples come from simulated executions of primitive skills and successful object transitions. The section also specifies the optimization procedure and reconstruction losses used to train the CVAEs.
A.1.5 Training Data Generation and Training Details
- Training data are generated by simulating primitive skills, sampling cuboid transitions between stable poses and object-mesh contact poses, and retaining successful feasible executions.The point-cloud mask labels identify points whose final positions fall within a thresholded Euclidean distance of the table.Source: §A.1.5 · p. 16
- The number of objects and data points per skill varies because data generation is stochastic and successful samples take different amounts of time to obtain.Source: §A.1.5 · p. 16
- The CVAE is trained with Adam at learning rate 0.001, using position MSE, quaternion geodesic orientation loss, and binary cross-entropy for the mask.For GAT models with multiple contact-pose predictions, reconstruction loss is computed for each prediction against the single ground-truth example.Source: §A.1.5 · p. 16
A.1.6 Evaluation Details
The appendix evaluates sampler design choices, generalization to novel geometries and noisy point clouds, and practical implementation constraints. It also discusses planning efficiency and limitations inherited from the primitive skills and open-loop execution.
A.1.6 Evaluation Details
- Mask-based subgoals reduced position and orientation errors and improved sticking success over directly predicting the transformation, which could place objects above the table or penetrate it.Those configurations could cause contact breaks or unexpected table collisions during execution.Source: §A.1.6 · p. 17
- Compared with the GAT model, PointNet++ had larger orientation error and lower sticking and feasibility success, with predicted contacts more prone to environmental collision.The paper links the orientation error to end-effector/object misalignment during initial grasp.Source: §A.1.6 · p. 17
- Jointly predicting contact and subgoal parameters improved feasibility success over independent prediction because the shared representation captured their dependence.Independent models needed more samples to find compatible contact poses and subgoals.Source: §A.1.6 · p. 17
- The hand-designed multi-step baseline samples contact poses and object transformations using point-cloud heuristics for top-surface pulls, antipodal reorientation grasps, stable configurations, and table clearance.The grasp-reorient sampler filters points on the subgoal and opposite planes, as well as low points, before sampling contacts; the pull sampler uses a point with an approximately upward normal.Source: §A.1.6 · p. 17
A.1.7 Additional Quantitative Results
- On 10 novel cylindrical objects with 10 start poses each, learned samplers retained successful executions and outperformed a baseline approximating cylinders as cuboids despite a distribution shift.The evaluation reports larger pose errors and lower success rates than the cuboid results; the authors hypothesize that training on more diverse objects could close the gap.Source: §A.1.7 · p. 18, Figure A4 · p. 18
- Under simulated depth-camera noise, the system remained robust to moderate input noise when evaluated in the multi-step setting.The learned samplers were trained on point clouds without noise.Source: §A.1.7 · p. 18, Table 3 · p. 18
A.1.8 System Implementation Details
- Contact-pose refinement searches along rays aligned with the palm-plane normal and updates the contact position to the nearest point in the dense object point cloud.The refinement is used for all skills and sampler variants and provides a large practical improvement.Source: §A.1.8 · p. 18
- Real-robot demonstrations used four calibrated RGB-D cameras, AR tags to obtain the desired relative transformation, and image masking and point-cloud cropping to segment objects before planning and execution.The system also relies on quasi-static rigid-object manipulation, sticking contact, and a plan skeleton specifying skill types; its primitive skills do not model interactions such as rolling or toppling, limiting demonstrated generalization to novel geometries.Source: §A.1.8 · p. 19, §A.1.9 · p. 19