PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs

Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Yuke Zhu

arXiv:2012.07277v2cs.RO

TL;DR

Long-horizon manipulation requires planning over visually observed scenes despite expensive continuous and combinatorial action search. The paper introduces a hierarchical neuro-symbolic planner using geometric and symbolic scene graphs, and reports over 70% real-robot success with nearly 90% subgoal completion.

  • Problem

    Long-horizon manipulation is difficult because continuous robot actions create high-dimensional search spaces whose planning complexity grows with task length.

  • Method

    The method combines neuro-symbolic task planning and graph-based motion generation over geometric and symbolic scene graphs constructed from visual observations.

  • Results

    The real robot achieved an overall success rate of 70.6% and a subgoal completion rate of 89.6%.

  • Takeaways & Limitations

    The framework demonstrates long-horizon and novel-task generalization in simulation and real-world manipulation experiments.

Abstract

from arXiv · show

We present a visually grounded hierarchical planning algorithm for long-horizon manipulation tasks. Our algorithm offers a joint framework of neuro-symbolic task planning and low-level motion generation conditioned on the specified goal. At the core of our approach is a two-level scene graph representation, namely geometric scene graph and symbolic scene graph. This hierarchical representation serves as a structured, object-centric abstraction of manipulation scenes. Our model uses graph neural networks to process these scene graphs for predicting high-level task plans and low-level motions. We demonstrate that our method scales to long-horizon tasks and generalizes well to novel task goals. We validate our method in a kitchen storage task in both physical simulation and the real world. Our experiments show that our method achieved over 70% success rate and nearly 90% of subgoal completion rate on the real robot while being four orders of magnitude faster in computation time compared to standard search-based task-and-motion planner.

I. INTRODUCTION

Long-horizon manipulation is difficult because continuous action spaces and growing task length make planning expensive. The paper addresses this with visually grounded hierarchical planning over symbolic and geometric scene graphs.

  • I. INTRODUCTION: Long-horizon manipulation is challenging because continuous robot actions create high-dimensional search spaces whose complexity grows combinatorially with task length.Existing task-and-motion planning also depends on manually specified rules, known physical states, and expensive combined search.
  • I. INTRODUCTION: The paper introduces a visually grounded hierarchical planner that combines neuro-symbolic task planning with graph-based motion generation.The framework operates on visual observations and conditions planning and motion generation on the specified task goal.
  • I. INTRODUCTION: The approach targets limitations of both task-and-motion planning and end-to-end learning by avoiding predefined symbolic rules while improving generalization to longer tasks.The paper positions its graph-based representation as a way to integrate visual input with structured planning.
  • I. INTRODUCTION: Two-level object-centric scene graphs represent symbolic relations for task planning and geometric poses and spatial relations for motion generation.The symbolic graph provides abstract semantic structure, while the geometric graph captures 6-DoF entity poses and relative spatial relations.
  • I. INTRODUCTION: The authors validate the approach on tabletop manipulation in simulated and real-world environments.The contribution summary explicitly includes both simulated and real-world evaluation.

II. RELATED WORK

Prior work separates symbolic task reasoning from continuous motion generation or learns neural planning models from sensory data. The paper situates its approach between these paradigms to address search complexity and limited long-horizon generalization.

  • II. RELATED WORK: Classical task-and-motion planning generalizes to arbitrary goals within a domain but suffers from combinatorial search over feasible plans.These methods use predefined symbolic rules and known dynamic models together with logic search and continuous motion planning.
  • II. RELATED WORK: Data-driven methods integrate sensory inputs and require less domain knowledge, but their long-term prediction and out-of-distribution generalization remain limited.Hierarchical neural methods improve long-horizon prediction within training distributions, while generic neural representations remain weaker on longer or unseen tasks.

III. PROBLEM STATEMENT

The problem is to generate low-level robot actions from visual observations and specified goals over prolonged manipulation sequences. The formulation factorizes this into high-level task planning and low-level motion generation using two scene-graph abstractions.

  • III. PROBLEM STATEMENT: The goal is to learn a policy mapping raw visual observations and task goals to low-level motor actions.The formulation defines observation, action, and goal spaces and seeks actions that achieve long-term goals.
  • III. PROBLEM STATEMENT: The problem is factorized into high-level task planning and low-level motion generation to address the computational burden of long action sequences.This factorization separates intermediate symbolic decisions from continuous control.
  • III. PROBLEM STATEMENT: A geometric scene graph represents object poses in SE(3) and pairwise spatial relations computed from visual observations.The graph is obtained through a 6-DoF pose estimation model.
  • III. PROBLEM STATEMENT: A symbolic scene graph represents object states and semantic relations as visually grounded predicates, including both current states and goal specifications.The symbolic graph uses a vocabulary of semantic symbols to form an abstract scene description.
  • III. PROBLEM STATEMENT: The task model predicts an immediate symbolic subgoal, while the motion model grounds that subgoal in the geometric graph to generate a low-level action.The immediate subgoal is intended to be achievable by a single motion primitive.

IV. METHOD

The method constructs hierarchical scene graphs, predicts subgoals through neuro-symbolic regression planning, and generates motions with graph-based models. Regression planning works backward from the goal and checks whether candidate subgoals are reachable by a single primitive.

  • IV. METHOD: The pipeline builds geometric and symbolic scene graphs from RGB observations, then predicts a symbolic subgoal before grounding it for motion generation.The high-level planner conditions on the current symbolic graph and specified goal, while the low-level stage uses the geometric graph.
  • IV. METHOD: Graph neural networks provide relational inductive biases by exchanging information among neighboring nodes and edges across graph structures.This design is intended to support processing scenes with different object configurations and graph topologies.
  • A. Task Planning: Regression Planning Networks recursively predict predecessor subgoals backward from the final goal until reaching one achievable by a single motion primitive.Backward regression reduces search space and supports training on shorter demonstrations while generalizing to longer tasks.
  • A. Task Planning: The task planner uses a preimage network, subgoal serialization, and reachability network to generate, select, and test candidate subgoals.The preimage network predicts preceding relations, serialization selects a candidate, and reachability checks whether it can be achieved.
  • A. Task Planning: The preimage network predicts possible previous symbolic subgoals by using message passing to infer relations between graph nodes.The method uniformly samples among possible next subgoals for subsequent reachability checking.
  • A. Task Planning: The reachability network predicts whether a candidate subgoal can be achieved from the current state with one motion primitive.If the candidate is unreachable, the planning process repeats with another possible subgoal.

B. Motion Generation

The motion generation model grounds an immediate symbolic subgoal in the geometric scene graph and predicts parameters for a selected motion primitive. It combines graph-based scene encoding with subgoal conditioning to produce low-level motor commands.

  • Motion Generation: The planner uses predicted motion parameters to execute primitives such as pick, place, or push through low-level motor control.The motion generation model operates on the geometric graph after it is grounded with the next immediate subgoal.
  • Motion Generation: The selected primitive type and manipulated object are determined from the next symbolic subgoal before geometric grounding.Grounding computes new object-centric node attributes relative to the relevant object or target region.
  • Motion Generation: The model takes a grounded geometric scene graph and immediate subgoal as input, then predicts parameters for the selected motion primitive.The grounded graph contains object-centric attributes computed relative to the target or object specified by the subgoal.
  • Motion Generation: A graph neural network embeds the grounded scene graph while a fully connected network embeds the subgoal before an MLP predicts motion parameters.The two embeddings are concatenated before parameter prediction.

C. Scene Graph Generation

Scene graph generation converts visual observations into geometric and symbolic object-centric representations. Geometric poses and spatial relations are mapped into semantic predicates that also incorporate fixtures and robot state.

  • Scene Graph Generation: A 6-DoF pose estimator converts RGB observations into geometric scene graphs whose nodes encode object poses and types.The geometric graph is fully connected so GNN message passing can encode constraints across all nodes.
  • Scene Graph Generation: The geometric scene graph represents object locations and relative spatial relations, while the symbolic graph abstracts these into semantic relations.The two levels provide geometric and symbolic descriptions of the same manipulation scene.
  • Scene Graph Generation: The symbol mapping function generates symbolic predicates from geometric graph edges while adding environment-fixture and robot-state information.Robot state includes whether an object is being held, supporting robot-related predicates.

D. Model Training

The task planner is trained from symbolic state transitions derived from short demonstrations, while motion generation is trained self-supervised from successful simulated interactions. Pose estimation relies on DOPE models trained with synthetic imagery and domain randomization.

  • Model Training: Task-planning training uses symbolic scene-graph transitions extracted from short demonstration trajectories.An algorithmic planner automates demonstration generation, and subsequent symbolic graphs provide training pairs.
  • Model Training: The reach model is trained with positive demonstration transitions augmented by negative examples, while the preimage model receives previous symbolic graphs as supervision.Both models use pairs of subsequent symbolic scene graphs from demonstrations.
  • Model Training: Motion generation is trained self-supervised by sampling primitive parameters in simulated environments and retaining successful executions as training data.Random object configurations and sampled primitive actions provide task-agnostic interaction data.
  • Model Training: DOPE pose estimators are trained on synthetic photo-realistic images with domain randomization for estimating known-object poses from single RGB images.The resulting models are used for real-world pose estimation.

V. EXPERIMENTS

Experiments evaluate the planner in a kitchen storage environment with shelf-placement and stacking goals, using pick, place, and push primitives. The setup discretizes motion parameters and trains task and motion models from demonstrations and sampled executions.

  • Environment: The experiments use a two-shelf kitchen storage environment in which a 7-DoF Franka Panda manipulates cans and boxes.Evaluation includes physical simulation and real-world household-object experiments using HOPE objects and DOPE pose estimation.
  • Environment: The task goals specify objects being inside a shelf, on a shelf, or on top of another object, represented with symbolic predicates.An example goal is On(Can, shelf A).
  • Motion Primitive Parametrization: Place and push motions are discretized into 10 bins per place coordinate and 21 bins for each push parameter.Place uses x and y coordinates, while push uses angle θ and travel distance d.
  • Attributes for Grounded Scene Graphs: Grounded scene-graph attributes encode object type, region, pose, and task-specific relations to target regions or objects.Push grounding additionally represents pose relative to shelf A.
  • Model Training: Task planning uses 18 demonstration trajectories, while motion training uses 14,573 place samples and 8,000 push samples.The demonstrations cover placing one object, stacking two objects, and putting one can into shelf A.
  • Model Training: Both task-planning and motion-generation models are trained with cross-entropy loss and graph-based message passing.The motion model uses two-round GNN message passing and 32-dimensional embeddings for shared graph and subgoal representations.

C. Evaluation Setup

The evaluation uses kitchen storage tasks with two- to six-object goals, measured by whole-task success and subgoal completion in simulation and on a real robot.

  • Tasks contain two to six objects, with goals specifying symbolic states for each object; two-object stacking cases are excluded because they appear in demonstrations.
  • Success rate measures complete-task trials, whereas subgoal completion rate measures completed subgoals relative to all subgoals in a goal.The former captures overall performance, while the latter captures partial task completion after intermediate failure.
  • The storage tasks use pick, place, and push motion primitives.
  • The framework is evaluated with ground-truth poses in simulation and with visual observations on a real robot.

D. Simulation Experiments with Ground-Truth Poses

Simulation compares the proposed graph-based planner and motion generator with direct subgoal prediction, random sampling, MLP motion generation, and PDDLStream timing.

  • Direct subgoal prediction achieves zero success rate for every problem size because it does not generalize beyond the training demonstrations.
  • The proposed graph-based motion generator outperforms random sampling for single-shot θ prediction, while GNN relational inductive biases improve performance over the MLP baseline.
  • Failures are primarily execution failures, increasing from 7% for two-object tasks to 33% for six-object tasks, while overstep failure never exceeds 1%.Overstep failure includes infinite primitive loops or failure to predict a reachable subgoal within 50 steps.
  • The method substantially accelerates task- and motion-level inference relative to the search-based PDDLStream solver, especially for task planning.The supplied passage attributes this efficiency to learning models reducing the search required for long-horizon inference.

E. Real Robot Experiments with Visual Observations

Real-robot experiments use raw visual input in a kitchen storage environment and achieve 70.6% overall success with 89.6% subgoal completion.

  • The real robot achieves 70.6% overall success rate and 89.6% subgoal completion rate across 34 trials.Per problem size, success/subgoal completion rates are 0.80/0.90, 0.80/0.90, 0.63/0.88, 0.67/0.93, and 0.33/0.89 for two through six objects.
  • The framework is evaluated with raw visual input, using multiple viewpoints to obtain robust pose estimates for the geometric scene.
  • The success-rate gap from subgoal completion is attributed to motion-primitive execution failures on the last few subgoals of larger tasks.
  • The paper concludes that the visual hierarchical planner generalizes to longer and novel tasks after training on short demonstrations.
Loading 2012.07277v2…