Source-linked AI summary
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Yuke Zhu
TL;DR
Long-horizon manipulation requires planning across extended action sequences despite high-dimensional continuous actions and growing planning complexity. The paper presents visually grounded hierarchical planning with two-level scene graphs and reports 70.6% overall success and 89.6% subgoal completion on a real robot.
Problem
Long-horizon manipulation planning must address high-dimensional continuous action spaces and planning complexity that grows with task length.
Source: §I · p. 1Method
The approach jointly predicts high-level task plans and low-level motions from visual observations using geometric and symbolic scene graphs.
Source: §I · p. 1Results
70.6% overall success rate and 89.6% subgoal completion rate were achieved across 34 real-robot trials.
Source: E. Real Robot Experiments with Visual… · p. 6Takeaways & Limitations
The authors report generalization to longer and novel tasks using models trained on short demonstrations.
Source: §VI · p. 6
Abstract
from arXiv · showhide
We present a visually grounded hierarchical planning algorithm for long-horizon manipulation tasks. Our algorithm offers a joint framework of neuro-symbolic task planning and low-level motion generation conditioned on the specified goal. At the core of our approach is a two-level scene graph representation, namely geometric scene graph and symbolic scene graph. This hierarchical representation serves as a structured, object-centric abstraction of manipulation scenes. Our model uses graph neural networks to process these scene graphs for predicting high-level task plans and low-level motions. We demonstrate that our method scales to long-horizon tasks and generalizes well to novel task goals. We validate our method in a kitchen storage task in both physical simulation and the real world. Our experiments show that our method achieved over 70% success rate and nearly 90% of subgoal completion rate on the real robot while being four orders of magnitude faster in computation time compared to standard search-based task-and-motion planner.
I. INTRODUCTION
The introduction presents a visually grounded hierarchical planner for long-horizon manipulation, combining symbolic subgoal planning with geometric motion generation over two-level scene graphs. The approach operates from visual observations and a specified goal, and is designed to avoid predefined symbolic rules and complex motion-planning procedures.
I. INTRODUCTION
- Long-horizon manipulation is difficult because continuous robot actions create high-dimensional search and planning complexity grows combinatorially with task length.Existing task-and-motion planning methods require manually specified symbolic rules, abstract away raw perception using known physical states, and incur costly searches over discrete symbols and continuous motions.Source: §I · p. 1
- End-to-end sensor- or latent-space learning integrates sensory input and supports reactive behavior, but is less able to generalize to long-horizon tasks than task-and-motion planning.Learning methods that integrate with task-and-motion planning still rely on strong domain knowledge and costly search procedures.Source: §I · p. 1
- The planner uses visual observations and a specified goal to combine high-level task planning with low-level motion generation, avoiding predefined symbolic rules and complex motion-planning procedures.The authors state that its graph-based representations enable generalization to longer task instances that are difficult for end-to-end learning approaches.Source: §I · p. 1
- The two-level scene graph pairs a geometric graph of entities’ 6-DoF poses and spatial relations with a symbolic graph of abstract semantic relations among entities and the robot.The graphs are obtained for real-world household objects using RGB-camera 6-DoF pose estimation and symbol mapping.Source: §I · pp. 1–2, Figure 1 · p. 1
- At each step, a neuro-symbolic regression planner derives the next subgoal from the final goal and symbolic graph, then graph-based motion generation predicts motion parameters using the grounded geometric graph.Both planning and motion generation use Graph Neural Networks.Source: §I · p. 2
- The authors report contributions in long-horizon planning with two-level scene graphs, graph-based learning and training for novel-task generalization, and tabletop manipulation demonstrations in simulation and the real world.Source: §I · p. 2
II. RELATED WORK
Related work spans task-and-motion planning, learned planning, and graph-based scene representations. This work combines regression-based task planning with motion generation through hierarchical scene graphs.
Task and Motion Planning
- Classic TAMP uses symbolic search and continuous sampling or trajectory optimization; unlike Deep Visual Reasoning, our GNNs predict task and motion plans directly for efficient inference.Classic TAMP relies on predefined symbolic rules and known dynamics and faces combinatorial search; learned heuristics guide search toward promising plans.Source: §II · p. 2
Learning to Plan
- Hierarchical learned planners decompose plans into subgoals and motions, but neural models have limited out-of-distribution or longer-task generalization; RPN generalizes zero-shot but abstracts away motion.Our model performs regression planning at the task level and connects hierarchical scene graphs from real images to low-level motion generation.Source: §II · p. 2
Graph-based Scene Representations
- Object-factorized representations aid reasoning about novel scenes [2], and graph representations support manipulation skills or high-level planning; ours unifies task and motion abstractions.The paper identifies hierarchical scene representations for visual planning as its novelty relative to prior hierarchical 3D scene graphs [29].Source: §II · p. 2
III. PROBLEM STATEMENT
The section formulates long-horizon manipulation as vision-based, goal-conditioned planning and factorizes it into high-level task planning and low-level motion generation using two-level scene graphs.
III. PROBLEM STATEMENT
- The policy maps visual observations and task goals to low-level robot actions, while long action sequences make direct planning computationally prohibitive.The problem is factorized into high-level task planning and low-level motion generation.Source: §III · pp. 2–3
- The geometric scene graph represents object poses in SE(3) and relative spatial relations, while the symbolic graph encodes visually grounded object states and semantic relations as predicates.The geometric graph is computed from visual observations using a 6-DoF pose estimation model.Source: §III · p. 3
- The task model predicts an immediate subgoal from the symbolic scene graph and goal, and the motion model uses the observation and subgoal to generate a low-level action.Intermediate subgoals are states achievable by a single motion primitive; the motion model computes a geometric graph from the observation and grounds the subgoal.Source: §III · p. 3
IV. METHOD
The method alternates high-level symbolic subgoal planning with low-level motion generation grounded in geometric scene graphs, then replans from a new observation after each subgoal is achieved.
IV. METHOD
- The high level predicts the next symbolic subgoal from the current symbolic scene graph and specified goal, while the low level grounds it in the geometric scene graph to predict motion parameters.After the subgoal is achieved, the method replans using the new observation.Source: §IV · p. 3
- GNNs process graph nodes and edges with parameter-shared neural modules and message passing to aggregate contextual information through local neighborhoods.This architecture supports generalization across graph topologies, including scene graphs with different numbers of objects.Source: §IV · p. 3
A. Task Planning · B. Motion Generation
Task planning recursively regresses from the specified goal to a reachable immediate subgoal, while motion generation uses that subgoal and a grounded geometric scene graph to produce motion-primitive commands.
A. Task Planning
- Backward regression planning reduces search space and enables models trained on shorter tasks to generalize to longer unseen tasks.The planner recursively predicts previous subgoals until one can be achieved from the current state with a single motion primitive.Source: A. Task Planning · p. 3
- A GNN preimage network predicts preceding symbolic relations, and the planner uniformly samples a candidate subgoal for reachability checking.Unlike RPN’s dependency network, subgoal serialization uniformly samples among possible next subgoals.Source: A. Task Planning · p. 3
- A GNN reachability network checks whether each target semantic relation is achievable with one motion primitive, repeating regression when the candidate subgoal is unreachable.Once a subgoal is reachable, the planner selects a motion primitive based on it.Source: A. Task Planning · p. 3
B. Motion Generation
- The overall pipeline estimates object poses from observations, constructs geometric and symbolic scene graphs, plans to an immediate subgoal, and grounds it for low-level primitive control.The geometric graph is constructed from pose estimates, and a symbol-mapping function maps it to the symbolic graph.Source: Figure 2 · p. 4
- The planner grounds the immediate subgoal in the geometric scene graph with object-centric contextual attributes that support collision avoidance and trajectory selection under environmental constraints.A GNN encodes these constraints, while a heuristic selects the motion primitive type and object to manipulate.Source: B. Motion Generation · p. 4
- Motion generation takes the grounded graph and immediate subgoal as inputs, then predicts continuous motion-primitive parameters using separate graph and goal embeddings concatenated into an MLP.The graph embedding uses a GNN, while the subgoal embedding uses a fully connected network; node attributes vary by task design and primitive type.Source: B. Motion Generation · pp. 3–4
C. Scene Graph Generation
The method builds geometric scene graphs from estimated object poses, then maps them to symbolic graphs that encode semantic relations, fixtures, and robot state.
C. Scene Graph Generation
- A CNN estimates dynamic objects’ 6-DoF poses from RGB images, forming geometric graph nodes with pose and object-type attributes alongside fixture and object nodes.The scene includes predefined fixtures such as tables and shelves, as well as manipulable objects such as cans and boxes.Source: C. Scene Graph Generation · p. 4
- The geometric scene graph is fully connected, enabling GNN message passing across all nodes to encode environmental constraints.Source: C. Scene Graph Generation · p. 4
- A symbol-mapping function converts geometric-graph edges into symbolic predicates, adding fixture and robot-state information while representing relations with directed edges and unary predicates with self-edges.For these manipulation domains, the function is defined by heuristics; it may also be manually defined or learned.Source: C. Scene Graph Generation · p. 4
D. Model Training
The models are trained from algorithmically generated task demonstrations, simulated task-agnostic interactions, and synthetic-data-trained object pose estimation. Task-planning training uses symbolic scene transitions, while motion training uses successful primitive executions.
D. Model Training
- An algorithmic planner generates short task demonstrations that are converted into symbolic scene graphs and state transitions for task-planning training.The preimage model uses previous symbolic graphs as supervision, while the reachability model is trained on positive and predicate-altered negative examples with binary labels for edge relations and node predicates.Source: D. Model Training · pp. 4–5
- Motion model ψ learns primitive parameters from task-agnostic interactions, using successful executions of sampled actions in procedurally generated simulated object configurations.The self-supervised procedure is similar to CAVIN.Source: D. Model Training · p. 5
- DOPE predicts known-object SE(3) poses from single RGB images; synthetic photorealistic training with domain randomization yields reliable real-image estimates.Source: D. Model Training · p. 5
V. EXPERIMENTS · A. Environment · B. Data Collection and Training Details
The experiments use a kitchen-storage environment to evaluate symbolic task planning and motion generation. The section also describes the demonstrations, simulation samples, and model configuration used for training.
A. Environment
- The kitchen setup uses two shelves, three pick/place/push primitives, household objects, and a 7-DoF Franka Panda arm in real-world and simulated experiments.Objects come from HOPE, with poses estimated by DOPE; simulation uses RoboVat and NViSII.Source: A. Environment · p. 5
- Four predicates—InGrasp, Clear, In, and On—describe symbolic object states and support goals such as On(Can, shelf A).In represents objects inside a shelf; On represents objects on a region, shelf, or another object.Source: A. Environment · p. 5
- Place locations use 10 bins per dimension, while push angle and distance each use 21 bins; RRT plans all motion trajectories.Picking orientation and stacking position are determined from the geometric scene graph rather than predicted.Source: A. Environment · p. 5
- Grounded place nodes encode object type, current region, and pose relative to the target; push nodes also encode pose relative to the pushed object and shelf A.The push attributes include object type and region as well.Source: A. Environment · p. 5
B. Data Collection and Training Details
- Task-planning training uses 18 short demonstration trajectories, while simulated motion training uses 14,573 successful place samples and 8,000 successful push samples.Demonstrations cover placing one object on a shelf, stacking two objects, or putting one can into shelf A.Source: B. Data Collection and Training Details · p. 5
- Both models use cross-entropy loss; the motion model combines two-round GNN message passing and a subgoal embedding in 32 dimensions each to predict primitive parameters.Shared modules use fully connected layers with Leaky ReLU; the combined embeddings feed a two-layer network, and all layers have 256 hidden units.Source: B. Data Collection and Training Details · p. 5
C. Evaluation Setup
The evaluation covers manipulation tasks with two to six objects, using success and subgoal completion rates to measure whole-task performance and partial progress. It tests the framework with ground-truth poses in simulation and visual observations on a real robot.
C. Evaluation Setup
- Tasks range from two to six objects, with goals specifying symbolic states for each object; two-object stacking cases are excluded because they appear in the demonstrations.Source: C. Evaluation Setup · p. 5
- Success rate measures whole-task completion across trials, while subgoal completion rate measures completed subgoals relative to all subgoals in the goal.Subgoal completion rate captures progress even when a task fails before completion.Source: C. Evaluation Setup · p. 5
- The framework is evaluated with ground-truth poses in simulation and on a real robot using visual observations.Source: C. Evaluation Setup · p. 6
D. Simulation Experiments with Ground-Truth Poses
In ground-truth-pose simulation, the method’s graph-based motion generation outperformed random sampling and MLP baselines, while direct next-subgoal prediction failed to generalize to longer tasks. The neuro-symbolic planner also substantially reduced inference time compared with PDDLStream.
D. Simulation Experiments with Ground-Truth Poses
- The evaluation generated 100 symbolic goal specifications per problem size and repeated each evaluation five times.The study compared regression planning with direct next-subgoal prediction conditioned on the goal and current subgoal.Source: D. Simulation Experiments with… · p. 6
- Direct subgoal prediction had zero success across all problem sizes, while graph-based motion generation outperformed random sampling and MLP baselines.The authors attribute direct prediction’s failure to poor generalization to tasks longer than the training demonstrations.Source: D. Simulation Experiments with… · p. 6, Table I · p. 6
- Execution failure rose from 7% for 2-object tasks to 33% for 6-object tasks, while overstep failure never exceeded 1%.Execution failures arose during motion execution; overstep failures involved infinite primitive loops or failure to predict a reachable subgoal within 50 steps.Source: D. Simulation Experiments with… · p. 6
- 4 orders of magnitude faster task planning than PDDLStream, with significantly faster inference at both task and motion levels.The computation-time comparison includes RRT motion planning.Source: Table II · p. 6, D. Simulation Experiments with… · p. 6
E. Real Robot Experiments with Visual Observations · VI. CONCLUSIONS
The method combines geometric and symbolic scene graphs for visual hierarchical planning, and its real-robot evaluation achieved 70.6% overall success and 89.6% subgoal completion. The paper concludes with directions for automating symbol mapping and representing articulated or deformable objects.
E. Real Robot Experiments with Visual Observations
- The real-robot evaluation achieved 70.6% overall success and 89.6% subgoal completion across 34 trials.The authors attribute the gap between these rates to motion-primitive execution failures on the last few subgoals in larger problem sizes.Source: E. Real Robot Experiments with Visual… · p. 6
- For 2–6 objects, success/subgoal completion rates were 0.80/0.90, 0.80/0.90, 0.63/0.88, 0.67/0.93, and 0.33/0.89, respectively.The corresponding trial counts were 10, 10, 8, 3, and 3.Source: E. Real Robot Experiments with Visual… · p. 6
- Robust geometric-scene computation used multiple images from different viewpoints to cover the workspace.The evaluation used raw visual input.Source: E. Real Robot Experiments with Visual… · p. 6
VI. CONCLUSIONS
- The framework integrates neuro-symbolic task planning and graph-based motion generation using geometric and symbolic scene graphs as two-level scene abstractions.The paper presents it as a visual hierarchical planning algorithm for long-horizon manipulation.Source: §VI · p. 6
- Models trained on short demonstrations generalized to longer and novel tasks, with quantitative evaluation in simulation and real-world experiments.Source: §VI · p. 6
- Future work includes learning the symbol-mapping function self-supervised from demonstrations to further automate the framework and learning process.Source: §VI · p. 6
- A second future direction is developing more flexible scene-graph representations for articulated and deformable objects.Source: §VI · p. 6