PaperlayerView exact revision on arXiv

Source-linked AI summary

GRID: Scene-Graph-based Instruction-driven Robotic Task Planning

Zhe Ni, Xiaoxin Deng, Cong Tai, Xinyue Zhu, Qinghongbing Xie, Weihang Huang, Xiang Wu, Long Zeng

arXiv:2309.07726v2cs.RO

TL;DR

Instruction-driven robotic planning needs richer environmental representations than raw images typically provide. GRID uses scene and robot graphs with LLM and graph-attention processing to iteratively predict action–object subtasks, outperforming GPT-4 while operating in real time.

  • Problem

    Existing LLM-based robotic planners mainly use images, which limit global scene comprehension and require extensive multimodal data and large-scale models.

    Source: §I · p. 1

  • Method

    GRID encodes instructions, scene graphs, and robot graphs with an LLM and GAT modules, then iteratively predicts action–object subtasks using a synthetic training-data pipeline.

    Source: §I · pp. 1–2

  • Results

    GRID outperforms GPT-4 by over 25.4% in subtask accuracy and 43.6% in task accuracy, with 0.11s inference time.

    Source: §I · p. 2

  • Takeaways & Limitations

    GRID maintains task-planning performance across unseen scenes and scenes with different object counts, with at most a 3.8% task-accuracy decline.

    Source: §I · p. 2

Abstract

from arXiv · show

Recent works have shown that Large Language Models (LLMs) can facilitate the grounding of instructions for robotic task planning. Despite this progress, most existing works have primarily focused on utilizing raw images to aid LLMs in understanding environmental information. However, this approach not only limits the scope of observation but also typically necessitates extensive multimodal data collection and large-scale models. In this paper, we propose a novel approach called Graph-based Robotic Instruction Decomposer (GRID), which leverages scene graphs instead of images to perceive global scene information and iteratively plan subtasks for a given instruction. Our method encodes object attributes and relationships in graphs through an LLM and Graph Attention Networks, integrating instruction features to predict subtasks consisting of pre-defined robot actions and target objects in the scene graph. This strategy enables robots to acquire semantic knowledge widely observed in the environment from the scene graph. To train and evaluate GRID, we establish a dataset construction pipeline to generate synthetic datasets for graph-based robotic task planning. Experiments have shown that our method outperforms GPT-4 by over 25.4% in subtask accuracy and 43.6% in task accuracy. Moreover, our method achieves a real-time speed of 0.11s per inference. Experiments conducted on datasets of unseen scenes and scenes with varying numbers of objects demonstrate that the task accuracy of GRID declined by at most 3.8%, showcasing its robust cross-scene generalization ability. We validate our method in both physical simulation and the real world. More details can be found on the project page https://jackyzengl.github.io/GRID.github.io/.

I. INTRODUCTION

GRID addresses the challenge of representing rich environmental information for instruction-driven robotic planning by using scene and robot graphs instead of relying primarily on images. It iteratively predicts executable subtasks and achieves strong accuracy, real-time inference, and cross-scene robustness.

Motivation and representation

  • GRID introduces scene graphs and separate robot graphs to represent environmental semantics, robot state, object relationships, and attributes for planning.The graphs provide broader and richer scene information than raw images, while the robot graph records location, nearby objects, and grasped objects.Source: §I · pp. 1–2

Deployment

  • GRID is designed as a lightweight model for offline embodied agents and can operate across robot forms and varied environments.The reported inference time is 0.11s, supporting real-time deployment on resource-constrained robots.Source: §I · pp. 1–2, Figure 1 · p. 1

GRID architecture

  • The model encodes instructions and both graphs with INSTRUCTOR, refines graph structure using GAT modules, and decodes action–object subtasks.A feature enhancer integrates instruction and graph features, while iterative planning adapts to scene changes and human interference.Source: Figure 1 · p. 1, §I · p. 2

Results

  • 43.6% higher task accuracy and 25.4% higher subtask accuracy than GPT-4 demonstrate GRID’s main planning advantage.GRID requires only 0.11s per inference and maintains task accuracy within a 3.8% maximum decline across unseen and differently sized scenes.Source: §I · p. 2

Dataset construction

  • A synthetic dataset construction pipeline generates scene-graph datasets because existing scene-graph datasets are not designed for robotic task planning.The pipeline supports evaluation using subtask accuracy and task accuracy.Source: §I · p. 2

II. RELATED WORK

Prior robotic research uses scene graphs for semantic navigation and planning, while LLM-based approaches commonly rely on images and difficult vision–language fusion. GRID frames instruction-driven planning as graph-conditioned prediction of an action and scene-object ID at each stage.

A. Scene Graph in Robotic Research

  • Scene graphs provide broad contextual information for navigation and long-term planning, but instruction-driven robotic task planning has received limited attention.GRID distinguishes itself from SayPlan by predicting object IDs, helping handle multiple objects of the same type and reducing dependence on text-only outputs.Source: A. Scene Graph in Robotic Research · p. 2

B. Understand Scene Information With LLMs

  • Image-based LLM systems face difficult vision–text alignment and often extract sparse image-bound semantics while relying on large-scale multimodal data and models.These limitations motivate representations that expose richer scene information directly to the language model.Source: B. Understand Scene Information With… · p. 2
  • GRID uses scene graphs instead of images so LLMs can process rich semantic scene knowledge without additional vision–text alignment or massive multimodal data.The broader observable domain supports navigation and tasks beyond the robot’s immediate field of view.Source: B. Understand Scene Information With… · p. 2

III. PROBLEM STATEMENT

  • GRID’s architecture combines a shared LLM encoder, graph-attention modules, feature enhancement, and a task decoder to generate action and object outputs.The instruction, robot graph, and scene graph are transformed into tokens before structural graph features are integrated.Source: Figure 2 · p. 3
  • The planning problem predicts the current subtask from a human-language instruction, robot graph, and scene graph, with instructions decomposed into sequential subtasks.The output is an action paired with a target object ID from the scene graph.Source: §III · pp. 2–3
  • Scene and robot graphs contain nodes for objects and edges for relationships, with node attributes including colors, positions, and other properties.The robot graph uses nR and eR, while the scene graph uses nS and eS for their respective nodes and edges.Source: §III · p. 3
  • At each stage, GRID predicts an action–object pair, executes it, updates the robot and scene graphs, and repeats until all subtasks are completed.The formulation defines the subtask space as the Cartesian product of robot actions and scene-node IDs.Source: §III · p. 3

IV. METHOD

GRID predicts each robotic subtask as an action–object pair from an instruction, robot graph, and scene graph at the current planning stage.

IV. METHOD

  • GRID predicts an action and object ID for each instruction stage using the instruction, robot graph, and scene graph as inputs.Prediction proceeds iteratively across the instruction’s associated subtasks.Source: §IV · p. 3
  • The method uses an encoder–decoder architecture with a shared-weight LLM, graph attention modules, a feature enhancer, and a task decoder.The LLM extracts semantic features, GAT modules capture graph structure, and the decoder generates task predictions.Source: §IV · p. 3

A. Feature Extractor

GRID maps instructions and graph nodes into a shared semantic space, then uses graph attention to refine structural features while incorporating instruction context.

A. Feature Extractor

  • A frozen shared-weight INSTRUCTOR LLM maps instructions and graph descriptions into a common latent space without additional feature alignment.Instructions use word tokens and sentence embeddings, while graph nodes are represented through sentence embeddings.Source: A. Feature Extractor · pp. 3–4
  • Vanilla instruction and graph features are passed through parallel cross-attention layers to enhance information shared between the two modalities.The feature extractor supplies graph-aware and instruction-aware representations to the later feature enhancer.Source: Figure 3 · p. 4
  • Each graph node’s attributes are treated as a sentence, and its sentence embedding becomes the node token for graph processing.The robot and scene graphs therefore provide separate sequences of node tokens.Source: A. Feature Extractor · p. 4
  • The instruction sentence embedding is concatenated with every node token before the robot and scene graph features are processed by separate GAT modules.This incorporates instruction information while extracting structural features from each graph.Source: A. Feature Extractor · p. 4

B. Feature Enhancer

The feature enhancer repeatedly exchanges information between instruction and graph tokens, producing instruction fusion features and graph queries for decoding.

B. Feature Enhancer

  • GRID uses stacked parallel multi-head cross-attention layers to strengthen crossover information between instructions and graphs.The design includes instruction-to-graph and graph-to-instruction attention with residual connections.Source: B. Feature Enhancer · p. 4
  • The cross-attention structures emphasize shared information, particularly when the same object appears in both the instruction and a graph.After the final layer, instruction tokens become fusion features and graph tokens become graph queries.Source: B. Feature Enhancer · p. 4
  • Enhanced tokens are separated into fusion features for the transformer encoder and graph queries for the transformer decoder.This separation prepares instruction and graph representations for task prediction.Source: Figure 4 · p. 4

C. Task Decoder

The task decoder combines enhanced instruction and graph representations, then predicts the robot action and the corresponding scene-graph object for each subtask.

C. Task Decoder

  • GRID divides each subtask into separate action and object predictions, enabling combinations of atomic robot skills with scene objects.Scene-graph object IDs avoid textual ambiguity and avoid enumerating large sets of complete subtasks.Source: C. Task Decoder · p. 4
  • Fusion features enter a transformer encoder, while robot and scene graph queries enter a transformer decoder before separate action and object branches.The decoder outputs embeddings associated with the robot and scene graph queries.Source: C. Task Decoder · p. 4
  • The robot-graph output is processed by an MLP to produce action scores, and the highest-scoring action is selected.The action branch yields Nact scores based on the current robot state representation.Source: C. Task Decoder · p. 4
  • Each scene-graph node receives an MLP-derived score indicating its probability of being the operated object.Object selection is therefore performed over node-specific scene-graph scores.Source: C. Task Decoder · p. 4
  • The 70-object dataset organizes subtasks by action type, with each subtask associated with a unique action.The table lists the dataset’s subtask types and their respective counts.Source: Table I · p. 5

D. Loss Function

GRID trains with cross-entropy losses for action and object prediction, combined with L1 and L2 regularization.

D. Loss Function

  • GRID combines categorical cross-entropy losses for action and object branches with L1 and L2 regularization terms.The loss weights are α and β, while γ and δ control the regularization strengths.Source: D. Loss Function · p. 5

V. DATASET

The authors construct synthetic datasets linking instructions, subtasks, robot graphs, and scene graphs for graph-based robotic task planning. The pipeline generates staged graph states from randomized domestic environments and varied language instructions.

V. DATASET

  • The synthetic pipeline connects instructions, subtasks, robot graphs, and scene graphs, providing benchmarks for graph-based robotic task planning.Subtasks serve as ground truth for training and evaluation, while the graph states represent successive task stages.Source: §V · p. 5
  • Each example begins with randomized domestic and predefined robot states, then iteratively updates both graphs after feasible subtasks are selected.Human-language instructions are synthesized in diverse expressions with ChatGPT-4 for each subtask sequence.Source: §V · p. 5
  • The datasets vary scene size from 30 to 70 objects, with the 70-object dataset containing about 20K instruction tasks and 82K subtasks.An additional unseen-scene dataset evaluates generalization beyond training environments.Source: §V · p. 5

VI. EXPERIMENTS

GRID is evaluated against LLM baselines, through module ablations, and across scene sizes and unseen environments. It achieves higher planning accuracy than GPT-4 while retaining fast inference and relatively stable cross-scene performance.

VI. EXPERIMENTS

  • The experiments include baseline comparison, module ablation, varying-scene-size generalization, unseen-scene evaluation, and simulation and real-world demonstrations.These components define the scope of the experimental evaluation.Source: §VI · p. 5

A. Training Details

  • The experiments use an 80/20 split of the 70-object dataset, with other datasets reserved for evaluation, and train GRID for 500 iterations.Training uses AdamW, a peak learning rate of 1e-4, batch size 240, and two RTX4090 GPUs.Source: A. Training Details · p. 5

B. Evaluation

  • Evaluation reports action, object, subtask, and task accuracy, with task accuracy requiring all subtasks to be predicted sequentially and correctly.Subtask accuracy requires both the action and object to be correct.Source: B. Evaluation · p. 5
  • With complete and accurate input graphs, GRID achieved 83.0% subtask accuracy and 64.1% task accuracy on the 70-object dataset, outperforming GPT-4’s 47.5% and 8.0%.Source: B. Evaluation · p. 5
  • GRID averages 0.11s per inference, compared with GPT-4’s 2.52s and LLaMA2’s 3.10s or 4.87s, with substantially fewer parameters.The reported speed and parameter profile support offline deployment and real-time inference on edge robots.Source: B. Evaluation · p. 6
  • Both GAT modules and the feature enhancer improve accuracy over the simplified model in the 70-object ablation study.The full model captures graph relationships and fuses instruction and graph information.Source: B. Evaluation · p. 6, Table IV · p. 6
  • GRID’s task accuracy remains within 62.2% ± 2% across scenes with different object counts, whereas GPT-4’s accuracy decreases by 8.7% as scenes grow.The result indicates robustness to varying scene sizes under the evaluated datasets.Source: B. Evaluation · p. 6, Table III · p. 6
  • On unseen scenes, GRID’s subtask accuracy decreases by about 2% and task accuracy by about 3% relative to the 70-object dataset.These results are obtained without additional training on the unseen scenes.Source: B. Evaluation · p. 6, Table V · p. 6

C. Simulation Experiment

GRID is integrated with ROS2 and Unity for simulation testing across robot types and environments. A mobile-pick demonstration shows sequential execution of an instruction-driven transport task.

C. Simulation Experiment

  • The Unity simulation platform connects GRID and robot control in ROS2 to the physics environment through ROS-TCP Connector.The setup validates GRID’s functionality in a simulated robotic system.Source: C. Simulation Experiment · p. 7, Figure 5 · p. 7
  • GRID runs on turtlebot, mobile, and mobile-pick robots across office, shop, and home scenarios.The deployments test multiple robot platforms and environmental settings.Source: C. Simulation Experiment · p. 7, Figure 6 · p. 7
  • A mobile-pick robot completes the instruction to move a teacup from the dining table to the kitchen table through staged navigation, retrieval, and placement.The demonstration proceeds from the living room to the dining room and then the kitchen.Source: C. Simulation Experiment · p. 7, Figure 7 · p. 7

D. Real Deployment Experiment

GRID was deployed on physical robots using scene graphs built from scene-graph generation and 6DoF pose estimation. A mobile-pick robot demonstrated instruction-driven delivery, supporting deployment across robot configurations and real-world scenarios.

D. Real Deployment Experiment

VII. CONCLUSIONS

  • GRID uses instructions, robot graphs, and scene graphs to produce action-object subtasks while encoding graph relationships with GAT modules.Source: §VII · p. 7
  • The reported results show that GRID outperforms GPT-4 in scene-graph-based instruction-driven robotic task planning and adapts to diverse scenes.Source: §VII · p. 7

Related papers

Loading 2309.07726v2…