PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Learning of Abstract Plan Feasibility

Michael Noseworthy, Caris Moses, Isaiah Brand, Sebastian Castro, Leslie Kaelbling, Tomás Lozano-Pérez, Nicholas Roy

arXiv:2107.00683v1cs.ROcs.AI

TL;DR

Long-horizon manipulation planning depends on predicting whether abstract plans can be concretely planned and reliably executed, yet real-world phenomena make Abstract Plan Feasibility difficult to model. This paper uses task-independent active learning and the infeasible subsequence property to select informative robot experiments. On a real Panda robot, the system learned a useful APF model in 400 experiments and its learned model achieved aggregate stability similar to the simulation model across tasks.

  • Problem

    Predicting whether an abstract action sequence can be concretely planned and reliably executed is difficult because real-world phenomena affect plan outcomes.

    Source: §I · p. 1

  • Method

    The robot actively explores real plan executions, selecting informative sequences and using infeasible prefixes to prune candidate plans while learning APF.

    Source: §I · pp. 1–2

  • Results

    The robot learned a useful APF model with 400 experiments, and the learned model yielded 27 stable towers out of 30 versus 24 for the simulation model in aggregate across tasks.

    Source: C. Real Robot Experiments · p. 8, §VII · p. 9

  • Takeaways & Limitations

    The learned APF model was used to build towers with previously unseen blocks under multiple objectives, including tallest tower and longest overhang.

    Source: §I · p. 2

Abstract

from arXiv · show

Long horizon sequential manipulation tasks are effectively addressed hierarchically: at a high level of abstraction the planner searches over abstract action sequences, and when a plan is found, lower level motion plans are generated. Such a strategy hinges on the ability to reliably predict that a feasible low level plan will be found which satisfies the abstract plan. However, computing Abstract Plan Feasibility (APF) is difficult because the outcome of a plan depends on real-world phenomena that are difficult to model, such as noise in estimation and execution. In this work, we present an active learning approach to efficiently acquire an APF predictor through task-independent, curious exploration on a robot. The robot identifies plans whose outcomes would be informative about APF, executes those plans, and learns from their successes or failures. Critically, we leverage an infeasible subsequence property to prune candidate plans in the active learning strategy, allowing our system to learn from less data. We evaluate our strategy in simulation and on a real Franka Emika Panda robot with integrated perception, experimentation, planning, and execution. In a stacking domain where objects have non-uniform mass distributions, we show that our system permits real robot learning of an APF model in four hundred self-supervised interactions, and that our learned model can be used effectively in multiple downstream tasks.

MIT, CSAIL

Long-horizon manipulation planning relies on abstract plans that can be realized by concrete motion plans and reliable real-world execution, but real-world effects make feasibility difficult to predict. The paper proposes task-independent active learning that selects informative plans and uses the infeasible subsequence property to improve data efficiency.

I. INTRODUCTION

  • Long-horizon manipulation requires searching over abstract action sequences and generating concrete motion plans, while successful lazy planning depends on predicting feasibility.Feasibility requires both a concrete planner solution and reliable execution with the intended outcome.Source: §I · p. 1
  • Because coarse simulation requires a dynamics model that may miss real-world phenomena, the paper learns feasibility from real plan executions without a task at hand.Each labeled training instance requires finding and executing a concrete motion plan, potentially taking several minutes.Source: §I · p. 1
  • The active learner generates and scores candidate plans, then executes informative plans and updates the APF model from their feasibility labels.The technical challenge is to generate candidates and assess their information value under the current model.Source: §I · pp. 1–2
  • If a plan prefix is infeasible, every longer prefix is also infeasible, so the learner can focus on informative sequences that can actually be experienced.This property supports pruning candidate plans during data acquisition.Source: §I · p. 2
  • The robot learned a useful Abstract Plan Feasibility model in 400 real-robot experiments through information-theoretic active learning.The system integrates autonomous perception, experimentation, planning, and execution in a block-stacking domain.Source: §I · p. 2
  • The contributions include synthesizing hypothetical plans, exploiting infeasible subsequences during sampling, and integrating autonomous self-supervised robot learning.The learned predictor is used for downstream tower-building objectives, including height and overhang.Source: §I · p. 2

II. PROBLEM FORMULATION

The problem is to learn parameters for predicting whether a robot will successfully execute an abstract action sequence, using as few labeled sequences as possible. The system separates model-learning experimentation from later goal-directed planning and assumes the first action is feasible independently of the initial state.

II. PROBLEM FORMULATION

  • The objective is to learn APF parameters that predict sequence success while using as few labeled action sequences as possible.The label φ is binary success or failure for an abstract action sequence.Source: §II · p. 2
  • The formulation assumes the first abstract action in a sequence is feasible and independent of the initial state.The method is stated to apply whether or not this assumption is made.Source: §II · p. 2
  • The system first explores plans to learn APF, then uses the learned model to plan over abstract sequences for specified goals.Both phases require a system that can execute concrete plans from abstract sequences and observe success or failure.Source: §II · p. 2

A. Experimentation phase

The robot actively selects and executes abstract action sequences to learn APF, then uses the learned model to choose sequences for downstream objectives. The model represents uncertainty and exploits the infeasible-subsequence structure of plans.

A. Experimentation phase

  • At each iteration, the robot generates and scores candidate action sequences, executes the highest-scoring plans, and uses their feasibility labels to update the model.The acquisition function and sampling strategy determine which sequences can be tested.Source: A. Experimentation phase · pp. 2–3

B. Evaluation phase

  • After learning APF, a Monte Carlo planner samples action sequences and selects the one with the highest expected value under the objective function.The expectation accounts for the likelihood that each sequence is feasible on the real robot.Source: B. Evaluation phase · p. 3

III. ACTIVE LEARNING OF ABSTRACT PLAN FEASIBILITY

  • The active-learning objective is to select experiments that reduce uncertainty over model parameters, using a distribution updated as data are collected.The method seeks to minimize posterior entropy as quickly as possible.Source: §III · p. 3

A. Abstract plan feasibility model

  • The APF model captures aleatoric uncertainty, while a Bayesian ensemble of parameter vectors represents epistemic uncertainty and is updated with new data.The ensemble is initialized with independently random parameters and updated via gradient descent.Source: A. Abstract plan feasibility model · p. 3
  • The Θss model uses the infeasible-subsequence property by predicting each action’s feasibility given feasible preceding actions; unlike Θcomp, it needs labels after each action.The method assumes the initial action is feasible.Source: A. Abstract plan feasibility model · p. 3
  • Graph neural networks aggregate local properties and relations to predict plans of varying lengths with a fixed-dimensional parameterization.Source: A. Abstract plan feasibility model · p. 3

B. Entropy reduction

The method selects plans to reduce posterior uncertainty, approximating sequential experiment design with a myopic information-gain objective. BALD makes this objective computable in label space and favors plans where ensemble models disagree confidently.

B. Entropy reduction

  • The objective selects the plan expected to reduce posterior parameter entropy most, and a myopic choice is a good approximation because the objective is submodular.The robot cannot choose the plan outcome, so the expected reduction accounts for possible outcomes.Source: B. Entropy reduction · p. 3
  • BALD reformulates the parameter-space entropy objective so that entropy calculations occur in the lower-dimensional outcome-label space.This avoids estimating entropy directly over a high-dimensional parameter space.Source: B. Entropy reduction · p. 4
  • BALD favors plans with high overall outcome uncertainty but low uncertainty within individual models, indicating epistemic disagreement rather than aleatoric uncertainty.Confident models that predict different outcomes can be distinguished by observing the execution result.Source: B. Entropy reduction · p. 4
  • An equally weighted ensemble supports global feasibility prediction and estimation of the BALD objective for candidate experiments.Source: B. Entropy reduction · p. 4

C. Sampling Strategies

The sampling strategies differ in how they search for informative plans and account for infeasible prefixes. The sequential strategy weights information gain by the chance that the relevant plan steps will be attempted, while each strategy has distinct sampling constraints.

C. Sampling Strategies

  • The complete strategy uniformly samples and scores full plans, but consistent sampling density requires exponentially more samples as plan length grows and may emphasize infeasible plans.Source: C. Sampling Strategies · p. 4
  • The greedy strategy chooses each next action to maximize BALD after optimistically constructing the preceding actions, but ignores that early failure prevents learning from the full execution.Source: C. Sampling Strategies · p. 4
  • The infeasible-subsequence property provides structure for active learning with Θss, motivating strategies that account for which actions are likely to be reached.Source: C. Sampling Strategies · p. 4
  • The sequential strategy weights information gain by the predicted probability that each plan step will be attempted, accounting for possible failure at earlier steps.It is implemented by sampling and scoring entire plans; this needs exponentially more samples for longer plans, though the tested domain had a relatively short horizon.Source: C. Sampling Strategies · p. 4
  • The incremental strategy considers only plans whose preceding prefix has been observed feasible, but reusing prefixes yields no novel observations and may be overly restrictive.It finds more feasible plans than the complete strategy.Source: C. Sampling Strategies · p. 4

IV. IMPLEMENTATION

The framework is implemented for robot object manipulation to construct towers, with experiments in simulation and on a real Franka Emika Panda.

IV. IMPLEMENTATION

  • The implementation targets tower construction through object manipulation and uses a 7-DOF Franka Emika Panda in both simulation and real-world experiments.Source: §IV · p. 4

A. Domain

The domain uses cuboids with non-uniform mass distributions and abstract actions that place objects on a stack. The integrated robot system perceives, plans, and executes those placements, including recovery and replanning when towers fall.

A. Domain

  • Objects are cuboids described by dimensions, center-of-mass offset, and mass; some contain lead balls that alter their mass distributions.Each face carries a unique ArUco marker for identification and localization.Source: A. Domain · p. 4, Figure 3 · p. 5
  • An abstract action places an object at a relative pose to the preceding object or, for the first placement, to the table.A plan is feasible only if detailed planning and execution place the objects and leave a stable tower, including every prefix subtower.Source: A. Domain · p. 5
  • The learned model is evaluated on objectives to build the tallest tower, maximize overhang, or maximize unsupported block area.Source: A. Domain · p. 5

B. Perception, planning, and execution

  • Two fixed depth cameras localize tagged blocks, and a wrist-mounted camera refines pose estimates for grasping.Fixed-camera estimates can err by up to 1 cm; the wrist camera refines them near the object.Source: B. Perception, planning, and execution · p. 5
  • PDDLStream combines task-level planning with bidirectional RRT motion planning, while the robot executes the resulting plans with joint-space controllers.After each placement, the wrist camera checks tower stability; if a tower falls, execution and parallel planning are interrupted for replanning and clearing fallen blocks.Source: B. Perception, planning, and execution · p. 5
  • The TGN uses domain-specific connectivity, the FCGN connects every node, and the LSTM processes blocks from the top of the tower.Source: Figure 4 · p. 6

C. Learning

The system learns APF from informative tower-building experiments and evaluates the resulting model on downstream planning tasks. In simulation, sampling strategy and architecture affect accuracy and task performance; real-robot learning also captures noise that may be difficult to model in simulation.

C. Learning

  • The TGN ensemble represents each block placement as a node connected to placements above it, reflecting the need to evaluate subtower stability.The compared alternatives are a fully connected graph network and an LSTM that processes blocks from the top of the tower.Source: C. Learning · p. 6
  • Each ensemble network is randomly initialized, trained with binary cross-entropy, and early-stopped using validation loss.The ensemble contains 10 networks.Source: C. Learning · p. 6
  • Before active experimentation, models are initialized with 40 randomly generated towers; each iteration selects the 10 highest-scoring towers for robot labeling.Twenty percent of collected data goes to validation, and collected towers are augmented by rotations of 90, 180, and 270 degrees.Source: C. Learning · p. 6

V. EVALUATION

  • Real-robot learning produces an APF model that performs downstream tasks and is robust to noise that would be difficult to model in simulation.Source: §V · p. 6
  • Task performance is measured by normalized regret against the highest-reward stable tower considered by the planner, with unstable towers assigned zero reward.Source: §V · p. 6

A. Impact of Sampling Strategy

Sampling strategy affects how quickly the learned APF model performs across downstream tasks, while graph-network architecture improves prediction compared with an LSTM. The TGN learns faster than the alternatives and has similar long-term performance to the FCGN.

A. Impact of Sampling Strategy

  • The sequential strategy minimizes Tallest Tower regret after 200 towers, while the naive greedy strategy takes roughly 800 towers to achieve decent performance on that task.Greedy sampling performs worst and does not account for subtower feasibility; the complete strategy has high variance on the harder tasks.Source: A. Impact of Sampling Strategy · p. 6, Figure 5 · p. 7
  • Longest Overhang and Maximum Unsupported Area require deeper understanding of the stability decision boundary than Tallest Tower, yet active learning achieves low regret and low variance on the harder tasks.Source: A. Impact of Sampling Strategy · p. 6

B. Model Architecture and Generalization

C. Real Robot Experiments

The robot collected 400 training towers over 55 hours and used its learned APF model on held-out blocks. The learned model performed competitively with the noisy simulation model and had similar aggregate stability, while the analytical model’s towers were rarely stable across the tasks.

C. Real Robot Experiments

  • The robot built 400 training towers over 55 hours, then evaluated its learned model on a separate set of held-out blocks.The training set contained 10 blocks, and the learned model was initialized using 40 simulated two-block towers with added relative-pose noise.Source: C. Real Robot Experiments · p. 7
  • The Learned model produced 27 stable towers out of 30, versus 24 for the Simulation model, with similar aggregate stability across tasks.The Simulation model showed higher variability across tasks; its assumptions specify Gaussian noise only in the plane normal to the table.Source: Table I · p. 7, C. Real Robot Experiments · p. 8
  • The Analytical model could select high-reward towers when stable, but its towers were rarely stable across the three tasks.Source: C. Real Robot Experiments · p. 8
  • The Learned and Simulation models could build large-overhang towers while accounting for noisy real-robot action execution.Some towers built with the Analytical and Simulation models were unstable in the Longest Overhang task.Source: C. Real Robot Experiments · p. 8, Figure 7 · p. 8

VI. RELATED WORK

The related work situates active abstract-plan feasibility learning among hierarchical planning, feasibility prediction, and learned dynamics, while contrasting it with data-intensive or vision-dependent approaches. The conclusion reports real-robot APF learning in 400 experiments and points to other domains as future work.

VI. RELATED WORK

  • Hierarchical planning separates abstract action reasoning from concrete planning; prior work studied high-level feasibility and action-feasibility prediction, while this work learns feasibility actively without detailed physical models.The cited approaches address long-horizon planning, high-dimensional planning efficiency, and reducing calls to expensive solvers.Source: §VI · p. 8
  • Active learning has also explored environments for model-based reinforcement learning; unlike Pathak et al.’s forward-model disagreement, this work predicts abstract-plan feasibility for long-horizon tasks.Source: §VI · p. 8
  • Stacking research spans block-placement stability, scene understanding with geometry and stability, and stone-stack construction using detailed object models.Source: §VI · p. 8
  • Tower-stability predictors have used deep learning, including graph networks, but prior datasets can be orders of magnitude larger than this work’s and difficult to collect on a real robot.Vision-based systems also cannot capture non-visual properties such as center of mass.Source: §VI · p. 8

VII. CONCLUSION

  • The conclusion reports information-theoretic active learning for APF and a useful model learned on a real Franka Emika Panda in 400 experiments.Incorporating plan feasibility into active learning improved sample efficiency.Source: §VII · p. 9
  • The authors identify applying curious, self-supervised exploration to other domains as future work.They suggest it may eventually help robots use powered-off laboratory downtime.Source: §VII · p. 9

APPENDIX

The appendix compares active and random sampling strategies in simulation, illustrates real-robot tower evaluations, and describes which components must change when applying the method to new domains.

A. Real Robot Evaluations

  • Real-robot evaluations show towers for Tallest, Longest Overhang, and Maximum Unsupported Area using analytical, simulation, and learned APF models.Ten towers per task were built with five blocks; fallen towers or towers with fewer than five blocks counted as infeasible.Source: A. Real Robot Evaluations · p. 11, Figure 8 · p. 12, Figure 9 · p. 12

B. Supervised vs Active Learning Comparisons

  • Sequential and incremental active methods using Θss outperform random-ss, while random-ss only outperforms greedy in simulation.Random-ss samples action sequences and labels each subsequence; the comparison reports median normalized regret across downstream tasks.Source: B. Supervised vs Active Learning… · p. 11, Figure 11 · p. 13
  • For Θcomp, complete—which selects sequences using BALD—outperforms random-comp, which randomly samples sequences and labels only full sequences.The comparison covers simulation downstream tasks and reports median normalized regret.Source: B. Supervised vs Active Learning… · p. 11, Figure 12 · p. 13

C. Method Generalizability

  • The method is most beneficial when action feasibility depends strongly on the preceding sequence, as in adding a fifth block to a four-block tower.Construction tasks can benefit from non-myopic information gathering and predictors that incorporate prior actions.Source: C. Method Generalizability · p. 11
  • Adapting to a new domain requires domain-appropriate abstract actions, autonomous physical planning and execution, and a detector that labels the desired outcome; added learner structure is optional.The authors state that a general-purpose graph network can be used without additional inductive bias.Source: C. Method Generalizability · p. 11
Loading 2107.00683v1…