PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning compositional models of robot skills for task and motion planning

Zi Wang, Caelan Reed Garrett, Leslie Pack Kaelbling, Tomás Lozano-Pérez

arXiv:2006.06444v2cs.ROcs.AIcs.LG

TL;DR

Robots need models of learned skills to combine them into plans for long-horizon manipulation, but prior robot-learning work has largely focused on acquiring individual skills. This paper learns skill models and integrates them with task-and-motion planning; its experiments report efficient model acquisition and robust use of new skills in complex problems.

  • Problem

    Robot-learning research has largely focused on acquiring individual sensorimotor skills, while combining and executing them to address real-world problems has received little attention.

    Source: §1 · p. 1

  • Method

    The paper learns skill-effect models using Gaussian processes and integrates the models with a task-and-motion planner that samples diverse, risk-aware control parameters.

    Source: §3.2 · p. 4, §4.1 · p. 5, §9 · p. 17

  • Results

    The experiments demonstrate efficient model acquisition and robust use of new skills in complex problems; active learning reduces robot trials, and diverse sampling helps planners find values admitting a full robot plan.

    Source: §1 · p. 2, §9 · p. 17

  • Takeaways & Limitations

    Learned models can be integrated with existing skills to plan sequences of skill executions toward long-horizon goals in complex environments.

    Source: §1 · p. 1

    The work addresses deterministic planning and open-loop execution, whereas real-world settings are stochastic and partially observable.

    Source: §9.1 · p. 19

Abstract

from arXiv · show

The objective of this work is to augment the basic abilities of a robot by learning to use sensorimotor primitives to solve complex long-horizon manipulation problems. This requires flexible generative planning that can combine primitive abilities in novel combinations and thus generalize across a wide variety of problems. In order to plan with primitive actions, we must have models of the actions: under what circumstances will executing this primitive successfully achieve some particular effect in the world? We use, and develop novel improvements on, state-of-the-art methods for active learning and sampling. We use Gaussian process methods for learning the constraints on skill effectiveness from small numbers of expensive-to-collect training examples. Additionally, we develop efficient adaptive sampling methods for generating a comprehensive and diverse sequence of continuous candidate control parameter values (such as pouring waypoints for a cup) during planning. These values become end-effector goals for traditional motion planners that then solve for a full robot motion that performs the skill. By using learning and planning methods in conjunction, we take advantage of the strengths of each and plan for a wide variety of complex dynamic manipulation tasks. We demonstrate our approach in an integrated system, combining traditional robotics primitives with our newly learned models using an efficient robot task and motion planner. We evaluate our approach both in simulation and in the real world through measuring the quality of the selected primitive actions. Finally, we apply our integrated system to a variety of long-horizon simulated and real-world manipulation problems.

1 Introduction

The paper addresses the gap between learning individual robot skills and combining them to achieve long-horizon goals. It learns skill models for planning and evaluates their integration in simulation and on a real robot.

1 Introduction

  • Home robots need to acquire specialized skills such as pouring and scooping from relatively few examples and combine them across varied situations.The motivation is to build on foundational capabilities such as locomotion and basic object manipulation.Source: §1 · p. 1
  • The framework learns models of new skills and uses them to plan skill sequences for long-horizon goals in complex environments.The paper reports extensive simulation and real-robot empirical results demonstrating efficient model acquisition and robust use of new skills in complex problems.Source: §1 · pp. 1–2
  • TAMP combines discrete choices of operations with continuous choices such as object grasps and placements, while also planning robot motion.The motivating coffee task combines picking, placing, and pushing with learned pouring and scooping amid arbitrary tabletop object arrangements.Source: §1 · p. 2
  • Skill models must characterize broad sets of control values, quantify uncertainty, and be learned with relatively few costly physical trials.Examples include pour and scoop primitives across different object dimensions and relative poses.Source: §1 · p. 2
  • The work extends Wang et al. (2018) with an integrated-system discussion, high-dimensional simulated manipulation experiments, learner and sampling comparisons, and real-world validation.The real-world validation includes learning individual primitives and deploying the full system on multi-step manipulation problems.Source: §1 · p. 2

2 Approach

The approach learns models of parameterized skills so a task-and-motion planner can combine them. A plan must select skill instances and parameters whose state transitions satisfy the learned constraints.

2 Approach

  • The robot plans over a sequence of skill instances and parameters because it lacks a complete model of each skill’s effects.Each skill is treated as an option with an initiation set, policy, and termination set.Source: §2 · p. 3
  • The learned skill model estimates a constraint that holds when executing a skill from a state with given parameters produces the specified next state.Training examples are state, skill-parameter, and future-state triplets; constraints may be decomposed into atomic constraints, with some known analytically.Source: §2 · p. 3
  • For planning, skill constraints can be expressed over fixed context parameters and free control parameters, such as object dimensions and a cup trajectory relative to a bowl.The learned constraint is distinct from the planner’s existing kinematic-path and collision constraints.Source: §2 · p. 3
  • A correct plan must satisfy each skill’s constraint across its alternating sequence of states and parameters, and this work searches for such values with sampling-based PDDLStream.The planner searches both the skill sequence and parameter values.Source: §2 · pp. 3–4

3 Estimating the constraint

The paper learns the successful control-parameter regions for skills across contexts, while representing uncertainty so planners can select parameters with confidence. It uses Gaussian-process active sampling to characterize those regions efficiently.

3 Estimating the constraint

  • The learning objective is to characterize the full set of successful control parameters for each fixed context, with uncertainty represented explicitly for robust planning.The formalization maps context parameters to sets of control parameters and represents success as a Boolean constraint.Source: §3 · p. 4, §3.1 · p. 4

3.1 Contextual super-level set estimation

  • Success is modeled as a super-level set of a real-valued score, which can represent the proportion of poured liquid reaching a target minus a required proportion.The scoring example assumes real-valued training outcomes rather than only binary success or failure labels.Source: §3.1 · p. 4

3.2 Active sampling for learning

  • A Gaussian process and level-set estimation guide queries toward characterizing each context’s successful region, rather than learning all function values or only finding a maximum.The straddle algorithm selects control parameters for evaluation; its acquisition score is high near the zero boundary or where the score is uncertain.Source: §3.2 · p. 4

4 Planning with a new skill

The section explains how learned skill models enter TAMP planning, where continuous action parameters must satisfy both skill-effect constraints and broader planning constraints. It develops risk-aware, efficient, and diversity-aware sampling to supply useful parameter candidates.

4.1 The need for sampling during planning

  • TAMP needs diverse control-parameter samples because skill-valid values may still violate broader constraints, such as robot reachability.The planner combines the learned skill constraint with other constraints when selecting samples.Source: §4.1 · pp. 4–5

4.2 Risk-aware sampling

  • Risk-aware sampling selects control parameters with high predicted success probability, relaxing the threshold to include values within 5% of the most confident score when the strict set is empty.The illustrated high-probability set is a subset of the true super-level set.Source: §4.2 · p. 5, Figure 4 · p. 5

4.3 Efficient adaptive sampling

  • An adaptive sampler avoids inefficient uniform rejection by proposing near likely-feasible values, reweighting accepted samples, and retaining uniform proposals to explore farther regions.It adjusts its search radius according to the acceptance rate of truncated-Gaussian samples.Source: §4.3 · pp. 5–6

4.4 Diversity-aware sampling for planning

  • DPP-style determinant scores quantify sample diversity, while online kernel updates treat planner-rejected samples as similar and adjust feature length scales.The adapted diversity metric is intended to suit the distribution of planning tasks.Source: §4.4 · pp. 6–7

5 Related work

The related work spans learning individual motor primitives, probabilistic skill models, and sampling for task-and-motion planning. This paper focuses on learning hybrid skill models and using them to plan multi-step execution.

Learning skill models

  • Prior work learns motor primitives such as pushing, scooping, and pouring, while this work targets models of those primitives for multi-step planning.It extends constraint-based preimage models with a probabilistic characterization used during planning (Kaelbling and Lozano-Perez 2017).Source: §5 · p. 7
  • Related skill-model approaches include symbolic models (Konidaris et al. 2018), image classifiers that do not support general-purpose planning (Kroemer and Sukhatme 2016b), and state-transition models for sequencing motor skills (Wang and Kroemer 2019).The paper's models are hybrid and include continuous parameters.Source: §5 · p. 7

Learning and sampling for planning

  • The paper uses GP level-set estimation and active sampling for skill constraints, while learned samplers can complement existing TAMP generators and methods for selecting or generating plan parameters.The authors focus on level-set estimation with GP regression while noting that other active-learning approaches could also be used.Source: §5 · pp. 7–8

6 Experimental domains

The evaluation spans 2D and 3D simulation and a real PR2; Kitchen3D bridges the earlier 2D domain and real-world robot operation. In Kitchen3D, pouring and scooping use parameterized contexts and waypoint controls, with skill-specific constraints.

6 Experimental domains

  • Experiments cover Kitchen2D, the new Kitchen3D simulation, and a real-world PR2 setting.Kitchen3D uses PyBullet and a dual-arm PR2 to study manipulation with 3D objects.Source: §6 · p. 8, §6.1 · p. 8

6.1 Implementation of Kitchen3D

  • Kitchen3D randomizes object and particle properties, while its planners use inverse kinematics, free-space motion planning, and constrained Cartesian-path planning.The simulation varies bowl and cup dimensions, physical properties, and liquid-particle properties across trials.Source: §6.1 · p. 8

6.2 Parameterization

  • Pouring and scooping each use five control parameters that define object waypoints, with distance controls normalized relative to bowl size.Contexts encode bowl and cup dimensions; scooping also includes spoon length.Source: §6.2 · pp. 8–9, Figure 6 · p. 9
  • Skill-specific collision constraints reject unsafe trajectories, but satisfying them does not guarantee a feasible full robot path.Failed-to-plan samples receive a small negative score rather than the same penalty as low-quality executions.Source: §6.2 · p. 9

7 Experiments in simulation

Simulation experiments compare supervised learners, active learning, and sampling strategies, then test learned skills in integrated planning. The reported results favor learned-kernel diverse sampling for two tasks and demonstrate complex integrated manipulation.

7.1 Supervised learning

  • The GP ultimately achieves the best average success rate, while random-forest and GP methods outperform neural-network methods.Figure 8 compares learner success curves; the GP uses a multi-layer perceptron kernel.Source: Figure 8 · p. 10, §7.1.1 · p. 10
  • On test F1, regressors outperform classifiers, in part because thresholding real-valued scores can make labels unstable near zero.The passage warns that this thresholding can also mislead GP uncertainty estimates and active sampling.Source: §7.1.2 · p. 10
  • The multi-layer perceptron kernel slightly outperforms the squared exponential and Matérn kernels on F1.Figure 9 compares the three GP kernels.Source: Figure 9 · p. 11, §7.1.3 · p. 11

7.2 Active learning

  • Active learners outperform the non-active GP; GP-LSE2 has the best success rate, and active learners more conclusively outperform it on F1.GP-LSE2 samples contexts randomly while actively optimizing control parameters; F1 reflects accuracy across more of the decision space.Source: §7.2 · pp. 11–12, Figure 11 · p. 12
  • With 100 training examples across 200 runs, GP-LSE2 has the best average F1 performance despite non-trivial variance.The authors hypothesize that actively focusing on control parameters may neglect context dimensions.Source: §7.2 · p. 12, Figure 12 · p. 13

7.3 Adaptive and diverse sampling

  • DIVERSE yields notably greater sample diversity than ADAPTIVE, with slightly more samples needed and a slightly higher false-positive rate.The false-positive rate can be reduced by increasing the confidence bound on the level set.Source: Table 1 · p. 13, §7.3 · p. 13

7.4 Learning kernels for diverse sampling

  • DIVERSE-LK generally finds better solutions than the alternatives in both WASH and UNCLOG.Table 2 reports timing and success-rate results after training.Source: Table 2 · p. 13, §7.4 · p. 13

7.5 Integrated system

  • The integrated system combines learned pouring and scooping with seven existing operations to plan complex goals.The paper demonstrates preparing coffee in simulation and on a real robot while avoiding obstacles.Source: §7.5 · pp. 13–14

8 Real-world experiments

The real-world system combines perception, planned data collection, Gaussian-process learning, and task-and-motion planning for pouring and scooping. Active learning improves test-object skill predictions, and the integrated planner solves multi-step manipulation problems.

8.1 Perception

  • Perception combines RGB object detections with depth-based table estimation and mesh registration to estimate object poses.The real-world setup assumes objects rest on one table, are visible to a head-mounted Kinect RGB-D camera, and have approximate mesh models.Source: §8.1 · p. 14

8.2 Data collection

  • The data-collection planner executes prescribed pour or scoop trials, scores their outcomes, and resets the robot and objects for the next trial.The typical planning solution takes less than 15 seconds; humans still clean spills and swap objects between trials.Source: §8.2 · p. 15
  • Learners trained on 10 bowls and 12 cups were tested on 5 bowls and 6 cups, while the same three spoons appeared in training and testing.Object contexts were sampled uniformly at random for each trial.Source: §8.2 · p. 15, Figure 16 · p. 15

8.3 Training

  • Active learning enables the GP learner to more quickly classify successful pours and scoops than without active learning; Figure 20 compares F1-score learning curves for both learners on this data set.The test set includes 81 scoop examples sampled uniformly at random on the test objects; the 1/4 standard-deviation error bounds come from retraining each GP 10 times on the same data.Source: §8.3 · p. 16

8.4 Most confident prediction

  • The active learner outperformed non-active learners on Valid, Success, and Filled for both pouring and scooping.Table 3 reports these test-object metrics for the learners’ most confident predictions.Source: §8.4 · p. 16, Table 3 · p. 17

8.5 Integration

  • The planner solved two multi-step tasks by relocating an obstructing block or pushing a bowl into reach before pouring.In the first task it found a grasp compatible with both a collision-free pickup and a predicted-successful pour; in the second it selected the bowl-pushing solution, likely because it used fewer actions.Source: §8.5 · p. 17

9 Conclusion

The paper concludes that GP skill models, diverse sampling, and conventional robotics operations can support complex manipulation planning. It identifies risk-aware planning, sim-to-real transfer, and stochastic partially observable settings as areas for further work.

9 Conclusion

  • GP-learned pour and scoop models support active training and diverse sampling, and the integrated planner applies them with conventional operations to challenging manipulation problems.The paper reports that active learning reduces the number of robot trials needed to learn a skill, while diverse successful control parameters help the planner find values admitting a full robot plan.Source: §9 · p. 17

9.1 Future work

  • Future work could incorporate expected skill score or super-level-set probability into action costs so the planner can weigh candidate plans.The paper frames this as an avenue for deterministic planning with nonnegative additive costs.Source: §9.1 · p. 17
  • The approach addresses deterministic planning with open-loop execution, not the stochastic and partially observable real world; sim-to-real transfer is also unaddressed.The authors note that active learning on simulated data was challenging for real-world models and identify integrating active learning with sim-to-real transfer as further work when simulators are available.Source: §9.1 · pp. 18–19

A Gaussian processes

This appendix describes Gaussian-process modeling and the PDDLStream framework used to combine learned skill samplers with conventional planning procedures. It also summarizes the framework’s use in the simulated Kitchen2D domain.

A Gaussian processes

  • Gaussian processes represent distributions over functions, and conditioning on noisy function observations yields a posterior with a mean and variance used to model skill outcomes and uncertainty.The paper considers squared exponential, Matérn, and multi-layer perceptron kernels and optimizes kernel hyperparameters using GPy.Source: A Gaussian processes · p. 19

B.1 PDDL

  • PDDL actions specify parameters, preconditions, and effects; pour and scoop actions additionally require GoodPour, GoodScoop, and Motion conditions on their parameter values.These learned conditions constrain skill parameters to those likely to succeed and connect robot-arm paths to manipulated-object paths.Source: §B.1 · pp. 19–20

B.2 Streams

  • PDDLStream streams pair conditional samplers with declarative guarantees, allowing learned pour and scoop path sampling to compose with model-based motion planning.The skill streams propose manipulated-object paths predicted to lie in the success super-level set; another stream finds robot paths that follow them.Source: §B.2 · p. 20

B.3 Algorithms

  • PDDLStream’s INCREMENTAL algorithm alternates sampling and discrete search, while FOCUSED first searches plan skeletons to target relevant samplers.Both operate on the same PDDLStream problem description.Source: §B.3 · p. 20

C Simulated Kitchen2D domain

  • In Kitchen2D, learned models of skills such as pouring, scooping, and pushing let a sampling-based planner plan efficiently using those skills.The appendix presents Kitchen2D as a simulated domain from an earlier version of the work.Source: C Simulated Kitchen2D domain · p. 20

C.1 Implementation of Kitchen2D

Kitchen2D models parameterized manipulation skills and scores their effects against task-specific criteria. The implementation learns models and samplers for pouring, scooping, and pushing.

C.1 Implementation of Kitchen2D

  • Kitchen2D represents material as circular particles and plans gripper motions with RRT-Connect.The gripper has 2D position and rotation degrees of freedom and can open and close its fingers.Source: §C.1 · p. 21
  • The implementation learns models and samplers for pouring with 4 context and 4 predicted parameters, scooping with 2 context and 7 predicted parameters, and pushing with 2 context and 6 predicted parameters.Robot trajectories are sequences of gripper waypoints relative to the manipulated object.Source: §C.1 · p. 22
  • Pouring succeeds when at least 95% of particles reach the target cup, scooping is scored by spoon capacity filled, and pushing by distance from the goal position.The respective score functions are defined for pour, scoop, and push outcomes, with cup dimensions as pour context, cup dimensions as scoop context, and the push goal position as pushing context.Source: §C.1 · p. 22

C.2 Experiments in Kitchen2D

The Kitchen2D experiments evaluate the method’s components separately and show their collective use in planning long-horizon tasks. The active-learning comparison includes GP-LSE, neural-network methods, and random sampling.

C.2 Experiments in Kitchen2D

  • The experiments assess individual components and then demonstrate their collective performance in long-horizon planning.The evaluation spans a high-dimensional continuous domain.Source: §C.2 · p. 22
  • GP-LSE estimates parameter constraints with a straddle algorithm, while NNc, NNr, and random sampling serve as comparison methods.NNc predicts success probability; NNr predicts a score, and both select the parameter with the highest predicted output.Source: §C.2.1 · p. 22
  • The component comparison evaluates GP-LSE, the neural-network approaches, and random sampling for pushing, pouring, and scooping.The neural-network approaches are classification- and regression-based variants.Source: §C.2.1 · p. 22

C.2.1 Active learning:

In Kitchen2D, GP-LSE learns effective action parameters more efficiently than the comparison methods. The section also presents how the first recommended action and generated samples are evaluated and compared.

C.2.1 Active learning:

  • Figure 28 compares first-action success rates for random selection, two neural-network methods, and GP-LSE across pouring, scooping, and pushing.The tasks have 8, 9, and 6 parameters, respectively; some are context parameters.Source: Figure 28 · p. 23, §C.2.1 · p. 23
  • Figure 29 contrasts the first five pouring samples from DIVERSE and ADAPTIVE, with transparency indicating sampling order.The section next compares rejection sampling, adaptive sampling, and diversity-aware sampling.Source: Figure 29 · p. 23, §C.2.2 · p. 23
  • GP-LSE achieves high accuracy well before the neural-network and random-selection methods across the tested cases.The reported measure is the success rate of the first recommended action parameter vector as actively gathered training examples increase.Source: §C.2.1 · pp. 22–23, Figure 28 · p. 23

C.2.2 Adaptive and diverse sampling:

DIVERSE yields more diverse candidate actions than ADAPTIVE, while using more samples to obtain five positives and having a slightly higher false-positive rate. The section also introduces comparisons of sampling effectiveness and learned-kernel sampling.

C.2.2 Adaptive and diverse sampling:

  • DIVERSE produces notably more diverse samples than ADAPTIVE, but uses more samples to find five positives and has a slightly higher false-positive rate.The false-positive rate can be reduced by increasing the confidence bound on the estimated level set.Source: §C.2.2 · p. 23
  • For 5 positive ones, DIVERSE uses more samples than ADAPTIVE and has a slightly higher false-positive rate, but notably higher sample diversity; increasing its confidence bound on the level set can decrease that rate.Figure 29 illustrates the ending poses of the 5 pouring actions generated by adaptive sampling with DIVERSE and ADAPTIVE.Source: §C.2.2 · p. 23
  • The next comparison evaluates ADAPTIVE, fixed-kernel DIVERSE-GK, and learned-kernel DIVERSE-LK using a GP-estimated high-probability super-level set.The section defines the planning reward for comparing samplers.Source: §C.2.3 · p. 23, Table 5 · p. 23

C.2.3 Learning kernels for diverse sampling:

The experiments evaluate learned diversity kernels on pushing and more complex pouring tasks, then integrate learned pour and scoop models with existing robot operations for a coffee-serving goal. DIVERSE-LK finds better solutions than alternatives on Tasks II and III, while diverse samplers are more stable after training.

C.2.3 Learning kernels for diverse sampling:

  • Figure 30 shows mean learning reward J(φ), with 1.96 standard deviation, versus training-task count for pushing an object off the table, pouring into a cup next to a wall, and picking up a cup in a holder and pouring into a cup next to a wall.Source: Figure 30 · p. 24
  • The planning reward discounts samples by order, and timed-out planner evaluation yields a lower bound averaged over tasks from a predefined distribution.The discount factor used is 0.6.Source: §C.2.3 · p. 24
  • On Task I, DIVERSE-LK learns the optimal side-switching strategy after a few planning instances, despite obstacles being absent from the sampler context.Task I is a pushing task with an obstacle equally likely to be on either side of a 2D table.Source: §C.2.3 · p. 24
  • Tasks II and III test multi-step pouring plans, with Task III adding a cup holder and a grasp location constraint near the cup top.The evaluation uses 50 task specifications and repeats it five times.Source: §C.2.3 · p. 24
  • The authors attribute sharp turns in Tasks II and III learning curves to a high kernel-length-scale penalty and the limited 50-task test set.They state these effects remain for future investigation.Source: §C.2.3 · p. 24
  • On Tasks II and III, DIVERSE-LK finds better solutions than the alternatives, while both diverse samplers have lower success-rate variance and are more stable after training.These comparisons use three sampling algorithms, each with a GP-estimated high-probability super-level set.Source: §C.2.3 · p. 24

C.2.4 Integration

  • The integrated system combines learned pour and scoop models with seven existing operations to plan coffee service; it solves tested arrangements in 20–40 seconds.The goal includes placing the prepared cup on a coaster near the table edge.Source: §C.2.4 · p. 24

Related papers

Loading 2006.06444v2…