PaperlayerView exact revision on arXiv

Source-linked AI summary

Partially Observable Task and Motion Planning with Uncertainty and Risk Awareness

Aidan Curtis, George Matheos, Nishad Gothoskar, Vikash Mansinghka, Joshua Tenenbaum, Tomás Lozano-Pérez, Leslie Pack Kaelbling

arXiv:2403.10454v2cs.ROcs.AI

TL;DR

Typical task and motion planning assumes full observability and deterministic action effects, limiting its treatment of uncertainty and information gathering. TAMPURA learns a sparse abstract belief MDP from controller descriptions and simulated outcomes, and outperforms several baselines on tasks requiring risk sensitivity, information gathering, and robustness.

  • Problem

    Typical TAMP assumes full observability and deterministic action effects, despite uncertainty and partial observability in real robotic settings.

    Source: §I · p. 1

  • Method

    TAMPURA uses coarse controller preconditions and possible effects to guide task-relevant model learning, then plans probabilistically in a sparse abstract belief MDP.

    Source: §IV · p. 4, §V · p. 5

  • Results

    TAMPURA significantly outperforms reinforcement learning, Monte Carlo tree search, and determinized belief-space planners on tasks requiring risk sensitivity, information gathering, and robustness.

    Source: §I · p. 2

  • Takeaways & Limitations

    The planner produces efficient risk- and uncertainty-aware plans across real and simulated robotic tasks, including with designed or learned low-level controllers.

    Source: §VIII · p. 9

    Planning time can be burdensome in real-time systems.

    Source: §VIII · p. 9

Abstract

from arXiv · show

Integrated task and motion planning (TAMP) has proven to be a valuable approach to generalizable long-horizon robotic manipulation and navigation problems. However, the typical TAMP problem formulation assumes full observability and deterministic action effects. These assumptions limit the ability of the planner to gather information and make decisions that are risk-aware. We propose a strategy for TAMP with Uncertainty and Risk Awareness (TAMPURA) that is capable of efficiently solving long-horizon planning problems with initial-state and action outcome uncertainty, including problems that require information gathering and avoiding undesirable and irreversible outcomes. Our planner reasons under uncertainty at both the abstract task level and continuous controller level. Given a set of closed-loop goal-conditioned controllers operating in the primitive action space and a description of their preconditions and potential capabilities, we learn a high-level abstraction that can be solved efficiently and then refined to continuous actions for execution. We demonstrate our approach on several robotics problems where uncertainty is a crucial factor and show that reasoning under uncertainty in these problems outperforms previously proposed determinized planning, direct search, and reinforcement learning strategies. Lastly, we demonstrate our planner on two real-world robotics problems using recent advancements in probabilistic perception.

I. INTRODUCTION

TAMPURA extends task and motion planning to uncertain, partially observable settings where controller effects cannot be described exactly. Across simulated and real-world tasks, it outperforms several planning and learning baselines on problems requiring risk sensitivity, information gathering, and robustness.

I. INTRODUCTION

  • Short-horizon behavior cloning, reinforcement learning, and model-based control can be robust under narrow training distributions, dense rewards, or modeling assumptions, but typically do not generalize to arbitrary long-horizon goals.TAMP sequences short-horizon controllers to address long-horizon tasks.Source: §I · p. 1
  • Typical TAMP assumes deterministic controller effects and full observability, although real-world stochasticity and partial observability make exact predictions and symbolic controller descriptions difficult.In open-world settings, limited knowledge can lead to actions with potentially catastrophic effects and leaves no reason to seek information.Source: §I · p. 1
  • TAMPURA uses coarse controller preconditions and possible effects to guide deterministic symbolic planning, then learns a sparse MDP of uncertain outcomes for risk-aware planning.The learned model captures tradeoffs among utility, risk, and information gathering and is small enough for solvers such as LAO* [10].Source: §I · pp. 1–2
  • TAMPURA significantly outperforms reinforcement learning, Monte Carlo tree search, and determinized belief-space planners on tasks requiring risk sensitivity, information gathering, and robustness, even with the same controllers.The evaluation includes simulated tasks and two real-world robotics tasks: searching clutter for objects and operating safely around an unpredictable human.Source: §I · p. 2

II. RELATED WORK

  • Some stochastic TAMP methods plan in state space using hardcoded transition probabilities or example plans, whereas TAMPURA learns belief-space transition probabilities through exploration.The cited approaches can produce contingent plans or likely successful open-loop plans [11].Source: §II · p. 2
  • Belief-space TAMP methods can gather information, but their determinization limits risk awareness, action-cost reasoning, and performance under large observational branching factors.The cited non-determinized alternative focuses on contingent planning without probabilities for nondeterministic outcomes.Source: §II · p. 2

III. BACKGROUND

The background frames planning as control over beliefs and controllers, then introduces reductions that make this problem tractable. TAMPURA abstracts beliefs, uses controller descriptions to guide exploration, and plans probabilistically in a sparse abstract MDP.

III. BACKGROUND

  • A POMDP represents uncertainty with state, observation, and action spaces, transition and observation probabilities, an initial-state belief, reward, and discount factor.Its belief-state MDP has probability distributions over states as states, with transitions induced by observations and belief updates.Source: §III · p. 3, A. Belief-State MDP · p. 3

B. Belief updates

  • When exact belief updates are intractable, approximate Bayesian inference such as particle filtering or sequential Monte Carlo can produce approximate beliefs; the real-world experiments use Bayes3D.Bayes3D generates belief distributions over latent 3D scene states from RGB-Depth images.Source: B. Belief updates · p. 3

C. Belief-State Controller MDP

  • A belief-space controller takes the current belief as input and acts in closed loop over an extended horizon, lifting the primitive-action belief MDP to one whose actions are controllers.The controller-level transition gives the probability of the resulting belief after a controller finishes executing.Source: C. Belief-State Controller MDP · pp. 3–4

IV. PLANNING WITH AN ABSTRACT BELIEF-STATE MDP

  • Direct search over controllers and continuous belief outcomes can be intractable, so TAMPURA reduces the controller MDP to a sparse abstract MDP for probabilistic planning.It abstracts beliefs, uses symbolic preconditions and uncertain effects to guide determinized planning and task-relevant model learning, then plans over learned transitions with a solver such as LAO*.Source: §IV · p. 4

A. Belief state propositions

TAMPURA represents beliefs with Boolean propositions, forms a discrete abstract belief MDP, and describes controller preconditions and uncertain effects with operators. These structures restrict applicable actions and possible successor states, while the planning objective is expressed as a goal in belief space.

A. Belief state propositions

  • Belief-state propositions are Boolean functions of a robot’s belief, including tests for an object’s most likely category or probable pose within a tolerance.In practice, they are implemented as lifted symbol-grounding functions that return a Boolean value from entities and the current belief.Source: A. Belief state propositions · p. 4

B. The abstract belief-state MDP

  • Evaluating propositions maps each belief to a discrete abstract belief; under an assumed condition, this yields an abstract belief-state controller MDP.The condition is that the abstraction be stationary; the paper assumes it holds.Source: B. The abstract belief-state MDP · p. 4, A. Belief state propositions · p. 4
  • The planning objective is a set of abstract belief states, and episodes terminate when a goal belief is reached, making the controller-level problem a belief-space stochastic shortest paths problem.A goal may require believing with high probability that the world lies in a desired set of states.Source: B. The abstract belief-state MDP · p. 4

C. Operators with uncertain effects

  • An operator specifies a controller’s belief-proposition preconditions, guaranteed effects, uncertain effects, and conditions on which uncertain-effect probabilities may depend.The probability distribution over uncertain effects need not be known in advance.Source: C. Operators with uncertain effects · pp. 4–5
  • Operator structure leaves few applicable actions in an abstract belief state and restricts successors to states that add guaranteed effects and possibly uncertain effects.This structure supports planning over a smaller set of feasible choices.Source: C. Operators with uncertain effects · p. 5

D. Operator schemata

TAMPURA uses controller schemata with uncertain effects to guide planning and learn outcome probabilities. Its Bayes-optimistic strategy focuses model-learning simulations on symbolically feasible, goal-directed plans.

D. Operator schemata

  • The PDDL extension describes controller schemata with preconditions, guaranteed effects, uncertain conditions, and possible uncertain effects.For example, grasp outcomes can depend on whether an object is believed to be glass, potentially prompting inspection before grasping.Source: D. Operator schemata · p. 5

V. LEARNING THE SPARSE ABSTRACT MDP

  • Learning outcome probabilities is costly, so TAMPURA uses symbolic structure and the goal to select which distributions to learn.Randomly sampling belief-state/controller pairs is described as inefficient.Source: §V · p. 5

A. Solution-guided model learning

  • Naive maximum-likelihood model learning can underexplore actions and converge to local optima; epsilon-greedy explores locally feasible actions rather than task-relevant ones.The paper gives picking unrelated objects as an example of unhelpful local exploration.Source: A. Solution-guided model learning · p. 5

B. Bayes optimistic model learning

  • Bayes-optimistic learning plans with optimistic outcome costs, then raises those costs as evidence accumulates; low-probability outcomes can also be penalized.The Bayesian-UCB criterion sets costs using an upper confidence bound on estimated outcome probabilities.Source: B. Bayes optimistic model learning · pp. 5–6
  • Bayes-optimistic model learning outperforms epsilon-greedy across tested values, while full probabilistic planning on the learned model remains risk-aware.Its exploration follows symbolically feasible, goal-directed policies rather than random action selection.Source: B. Bayes optimistic model learning · p. 6

C. The TAMPURA model-learning algorithm

The model-learning algorithm compresses transition probabilities into tables indexed by uncertain preconditions and effects, then uses simulated controller outcomes to update those tables and build a sparse abstract MDP.

C. The TAMPURA model-learning algorithm

  • The transition table depends on assignments to each operator’s uncertain conditions and effects, compressing the abstract transition model.For symbolically feasible transitions, the probability depends on the assignments to those proposition sets rather than the full beliefs.Source: C. The TAMPURA model-learning algorithm · p. 6
  • Across 50 environments, the figure compares estimated with true irrecoverable-transition probabilities; its reward plot shows TAMPURA reaching near-optimal performance quickly and surpassing weighted all-outcomes determinization.The reward comparison uses the determinized solution under ground-truth outcome probabilities.Source: Figure 4 · p. 7
  • The algorithm tracks controller simulations and resulting uncertain-effect assignments, then selects planned transitions with maximal posterior entropy for further simulation.It updates counts from the simulated outcomes and compiles them into a model over explored abstract beliefs reachable from the initial belief.Source: C. The TAMPURA model-learning algorithm · pp. 6–7

D. Progressive widening

Progressive widening gradually expands the grounded operator set with new continuous-parameter samples, controlling growth of the abstract planning problem during model learning.

D. Progressive widening

  • The algorithm assumes continuous inputs are initially fixed, but can extend the operator set by progressively adding instances with newly sampled parameters.The added instances bind controller schemata to new continuous parameter values.Source: D. Progressive widening · p. 7
  • Operator-set expansion must be gradual so that growth of the abstract MDP does not outpace optimistic exploration.Source: D. Progressive widening · p. 7
  • The branching factor of a lifted operator expands as a function of how many times that operator has been sampled, with α < 1 and coefficient k controlling the relationship.The passage gives the qualitative relationship but no further parameter values.Source: D. Progressive widening · p. 7

E. Learning UConds from controller feedback.

TAMPURA learns additional uncertain preconditions from controller feedback and is evaluated on simulated and real-robot tasks involving uncertainty. Across the simulated tasks, Bayes-optimistic learning with probabilistic decision-making performs best among the compared methods.

E. Learning UConds from controller feedback.

  • Controller simulations can return newly relevant UConds, which TAMPURA adds immediately to the operator’s uncertain-condition set.The paper notes that this can improve efficiency without increasing the ability to find correct plans in the limit.Source: E. Learning UConds from controller… · p. 7

VI. SIMULATED EXPERIMENTS & ANALYSIS

  • Evaluation spans five simulated and two real-world robotics problems, with comparisons to MCTS, reinforcement learning, and belief-space TAMP baselines.The simulated experiments use PyBullet; the real-world tasks include cluttered-object search and safe operation around an unpredictable human.Source: §VI · pp. 7–8

A. Simulated Domains

  • The simulated suite covers class, pose, partial-observability, physical, and SLAM uncertainty, including tasks that require risk awareness or information gathering.Source: A. Simulated Domains · p. 8

B. Baselines

  • Bayes-optimistic learning with full probabilistic decision-making performs best across the tested tasks; determinized planning performs poorly with irreversible outcomes and higher observational branching.The table reports average and standard error of discounted return for the compared strategies and tasks.Source: B. Baselines · p. 8, Table I · p. 8

VII. REAL-WORLD IMPLEMENTATION

TAMPURA was implemented on a Franka robot for object search and human-aware manipulation, using probabilistic perception and occupancy beliefs. The discussion presents uncertainty- and risk-aware planning as a way to plan over long horizons without precise controller-effect descriptions, while noting planning-time and engineering burdens.

VII. REAL-WORLD IMPLEMENTATION

  • The Franka robot used Bayes3D pose inference with known object meshes for two real-world tasks involving partial observability and human safety.The setup used Realsense D415 RGB-D cameras with known intrinsics and extrinsics.Source: §VII · p. 9

A. Searching for Objects in Clutter

  • For cluttered-object search, Bayes3D posterior distributions let TAMPURA estimate which objects might hide the cube and produce qualitatively sensible search plans.It moved larger objects with greater probability of hiding the cube before smaller ones; failures involved perception, camera-cord tension, and sometimes insufficient grasp or camera-perspective samples.Source: A. Searching for Objects in Clutter · p. 9

B. Safety in Human-Robot Interaction

  • For human-robot interaction, occupancy probabilities decay over time and are reset to 1 for voxels observed in the current point cloud.The planner integrates occupancy probabilities along a trajectory to estimate collision probabilities, enabling safety decisions such as choosing a lower-collision object order or waiting for the human to clear the workspace.Source: B. Safety in Human-Robot Interaction · p. 9

VIII. DISCUSSION

  • TAMPURA reasons about uncertainty and risk at task and motion levels, supporting efficient plans across real and simulated tasks without precise low-level controller-effect descriptions.The approach is intended to work with arbitrary learned or designed controllers.Source: §VIII · p. 9
  • Planning time can burden real-time use, and the planner requires user-provided belief representations, abstractions, and belief-updating functionality that may demand considerable engineering and expertise.Faster motion planning or GPU-based parallel simulation could ease the planning-time burden.Source: §VIII · pp. 9–10

APPENDIX

TAMPURA compiles simulation outcome counts into a sparse abstract MDP for probabilistic planning. The appendix also describes class-uncertainty and pose-uncertainty tasks that require inspection or manipulation to improve success under uncertainty.

B. Compiling simulation outcome counts into the sparse abstract MDP

  • The learned transition probability is the fraction of simulations from a given uncertain-precondition assignment that produced each uncertain-effect assignment.The sparse MDP includes reachable abstract beliefs and explored outcomes; unvisited operators are omitted, keeping the action branching factor sparse for LAO*.Source: B. Compiling simulation outcome counts… · p. 13

1) CLASS UNCERTAINTY:

  • In class uncertainty, the robot must place objects of a target class in a bowl despite noisy classifications and unknown grasp-success probabilities.Closer inspection can increase category certainty, while stable grasps support successful placement.Source: 1) CLASS UNCERTAINTY · p. 13

2) POSE UNCERTAINTY:

  • In pose uncertainty, the robot must stack three cubes with uncertain initial poses and no wrist camera, using a hook to improve reach or reduce pose uncertainty.The described controllers include picking, placing, stacking, and pulling objects.Source: 2) POSE UNCERTAINTY · pp. 13–14

3) PARTIAL OBSERVABILITY:

The appendix specifies uncertainty-focused tasks and experimental details for TAMPURA, including hidden-object search, friction learning, and localization under action noise. It also describes the controllers and collision-probability calculation used in real-robot object finding and human interaction.

3) PARTIAL OBSERVABILITY:

  • In partial observability, the robot must find and hold a hidden die among 2 to 10 objects, potentially manipulating other objects subject to kinematic, geometric, or visibility constraints.The available controllers are Pick, Place, Look, and Move.Source: 3) PARTIAL OBSERVABILITY · p. 14

4) PHYSICAL UNCERTAINTY:

  • Under physical uncertainty, the robot pushes a puck with unknown friction toward a target, with local pushes used to gather friction information before the target push.The task provides PushTo and fixed-velocity PushDir controllers.Source: 4) PHYSICAL UNCERTAINTY · p. 14

5) SLAM UNCERTAINTY:

  • In SLAM uncertainty, action noise increases the robot’s position uncertainty, while beacon localization is available during navigation and manipulation.The task varies objects, obstacles, and starting position; a manipulation-free variant checks baseline implementations.Source: 5) SLAM UNCERTAINTY · p. 14

D. Experimental Details

  • Experiments ran on one Intel Xeon Gold 6248 processor with 9 GB of memory, and reported planning times for each algorithm-environment combination.The algorithms were not optimized for planning time and run anytime, so earlier termination can yield lower success rates.Source: D. Experimental Details · p. 14

1) Object finding:

  • For real-robot object finding, Pick requires object-pose variance below a threshold, Place requires placement collision probability below a threshold, and Look captures an RGB-D image at a joint configuration.Source: 1) Object finding · p. 14

2) HRI:

  • In human-robot interaction, collision probability is calculated for each trajectory segment using occupancy probabilities along the path.The trajectory description defines T as the number of time steps, n_t as the cells encountered at each step, and (x_i, y_i, z_i) as cell coordinates.Source: 2) HRI · p. 14

E. Additional Baseline Details

The section specifies how DQN, MCTS, and Q-learning baselines were implemented, and identifies additional statistics reported for simulated TAMPURA experiments.

1) DQN:

  • DQN uses online simulations from the initial belief, with a vectorized belief state, pre-discretized actions, and a 1,000-sample-per-action training limit.The authors focus on zero-shot performance on novel, out-of-distribution tasks; offline RL generalization requires a separate setup outside the paper’s scope. Table II reports average and standard deviation of per-step planning times, including simulated controller execution.Source: 1) DQN · pp. 14–15, Table II · p. 15

2) MCTS:

  • MCTS plans in abstract belief space with pre-discretized actions and selects actions using UCB, setting its exploration constant C to 1.The UCB score combines a child node’s average reward with an exploration term based on parent and child simulation counts.Source: 2) MCTS · p. 15

3) Q-Learning:

  • Q-learning maintains a sparse Q table over the abstract belief space and pre-discretized actions, selecting the initial-belief action with the highest Q value.The experiments use α = 0.2 and random-action probability ϵ = 0.2.Source: 3) Q-Learning · p. 15

F. Additional Experimental Statistics

G. Continuous Action Parameter Samplers

Continuous action parameters are sampled through streams that encode input-output relationships, while the planner uses specified simulation and planning settings and Fast Downward-based symbolic planning.

G. Continuous Action Parameter Samplers

  • Continuous action parameter samplers use streams to generate values conditioned on related inputs, such as object-specific grasps or inverse-kinematics solutions for a grasp, object, and pose.Stream domain and certified atoms are nonfluents, so action preconditions can enforce these input relationships.Source: G. Continuous Action Parameter Samplers · p. 15

H. Planner Hyperparameter Details

  • Simulated experiments cap controller simulations at 1,000 per real step, use γ = 0.98, and query the Top-K symbolic planner in batches of 20 plans.All methods have at most 20 environment steps; progressive-widening baselines use k = 3 and α = 0.2.Source: H. Planner Hyperparameter Details · p. 15

I. Fast Downward Planning Details

  • Fast Downward converts each possible stochastic action outcome into a separate deterministic action and assigns costs using estimated log probabilities.SymK supplies batches of determinized plans; Fast Downward uses A* with the Landmark Cut heuristic.Source: §I · p. 16

J. Stationarity of the abstract belief state MDP

The abstract belief-state MDP requires a stationary distribution over concrete beliefs within each abstract state; the paper assumes this condition and discusses imperfect abstractions that may violate it.

J. Stationarity of the abstract belief state MDP

  • The conditional belief distribution is defined by considering the robot’s belief after a controller sequence and conditioning on its corresponding abstract belief state.Controller transition probabilities describe the belief-state distribution after each controller.Source: J. Stationarity of the abstract belief… · p. 16
  • The abstract belief-state MDP is well defined when the concrete-belief distribution within each abstract state is invariant across time and controller sequences, a stationarity condition assumed for user-provided abstractions.Stationarity means the distribution does not depend on elapsed time or which controllers have been executed.Source: J. Stationarity of the abstract belief… · p. 16
  • User-provided abstractions may violate stationarity; TAMPURA approximates the distribution using concrete and abstract belief-state pairs encountered in simulations.When distributions across histories have bounded divergence, the method is expected to approximate an abstract belief-state MDP, but formal analysis is left for future work.Source: J. Stationarity of the abstract belief… · p. 16
Loading 2403.10454v2…