PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-vocabulary Queryable Scene Representations for Real World Planning

Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S. Ryoo, Austin Stone, Daniel Kappler

arXiv:2209.09874v2cs.ROcs.AIcs.CV

TL;DR

Real-world robot planning needs language understanding grounded in the surrounding scene, while prior systems are limited by fixed object and executable-option sets. NLMap builds an open-vocabulary, language-queryable scene representation, uses an LLM to propose and query relevant objects, and conditions planning on the results. In real-robot evaluation, it achieves 61.8% success across 55 kitchen tasks, including 35 tasks impossible for prior planners without NLMap.

  • Problem

    Prior language-based robot planners lack sufficient grounding in surrounding scenes and are limited by predefined objects or executable options.

  • Method

    NLMap builds a visual-language scene representation, uses an LLM to propose instruction-relevant objects, queries their availability and locations, and conditions LLM planning on the returned scene information.

  • Results

    61.8% success across 55 real-world kitchen tasks, including 35 tasks impossible for prior planners without NLMap.

  • Takeaways & Limitations

    NLMap enables real-robot operation on open-world tasks involving novel objects that previous state-of-the-art planners could not address.

Abstract

from arXiv · show

Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable scene representation with Visual Language models (VLMs). An LLM based object proposal module parses instructions and proposes involved objects to query the scene representation for object availability and location. An LLM planner then plans with such information about the scene. NLMap allows robots to operate without a fixed list of objects nor executable options, enabling real robot operation unachievable by previous methods. Project website: https://nlmap-saycan.github.io

I. INTRODUCTION

NLMap addresses the challenge of grounding language-based robot planning in persistent, open-vocabulary scene information. It combines a natural-language-queryable representation with LLM object proposal and planning to support real-world tasks.

  • Motivation: Robots executing varied commands must identify relevant objects, locations, affordances, and sequential actions within the surrounding scene.The motivating example requires locating watering equipment, a sink, and plants before planning and executing the task.
  • Motivation: Persistent scene representations avoid repeated exploration, but prior representations generally locate only object categories known during construction.This limits their use when human instructions mention open-vocabulary objects.
  • Approach: NLMap builds a queryable representation from exploration, proposes instruction-relevant objects with an LLM, and supplies their presence and locations to LLM-based planning.The overview describes the pipeline from exploration through object querying to context-conditioned planning.
  • Motivation: LLM planners provide language reasoning but are not inherently grounded in environmental context, creating a challenge for embodied operation.The paper identifies connecting semantic scene representations to planners as a separate challenge.
  • Approach: NLMap combines class-agnostic visual-language features with natural-language queries that can target individual objects or object families.The representation uses CLIP and ViLD features and supports queries such as “snack” or “fruit”.

A. Scene Representation

NLMap constructs a persistent spatial-semantic representation during exploration and queries it at plan time using language-derived object names. The resulting object availability and location information conditions executable-option generation and LLM planning.

  • Scene construction: During exploration, NLMap detects class-agnostic regions and encodes each region with visual-language features, 3D position, and estimated size.The representation is agnostic to the exploration policy, which may use frontier exploration or predetermined waypoints.
  • Scene construction: The scene representation is the collection C={c_i} of context elements c_i=(φ_i,p_i,r_i), combining embeddings, positions, and object sizes.CLIP and ViLD provide the visual encoders used to form φ_i.
  • Planning: The system generates executable options from objects found in the scene and then plans and executes using those scene objects.This connects natural-language object proposal to the downstream SayCan-style planning loop.
  • Scene construction: NLMap is built once per scene and can subsequently be loaded for instruction-specific querying.The high-level algorithm separates scene construction from later language-conditioned planning.
  • Object querying: An LLM parses each instruction into relevant object names, whose text features query the stored map for object scores, presence, and locations.Multiview fusion combines query results before filtering the scene-object list.
  • Object querying: Few-shot prompting is used to make object proposal parse unstructured instructions into structured object names for scene lookup.The paper identifies this language-to-structure bridge as the core challenge of querying scene information.

2) Object Query:

NLMap extends SayCan by querying a natural-language scene representation before generating executable options. This adds global object availability and location information to planning beyond SayCan’s fixed option and local-visibility assumptions.

  • Object Query: CLIP and ViLD embeddings are combined for object querying because CLIP better detects out-of-distribution objects while ViLD is more robust to common objects.Their normalized inner products support nearest-neighbor retrieval and threshold-based filtering of low-confidence detections.
  • Integration with SayCan: NLMap generates instruction-relevant object names and locations conditioned jointly on the instruction and the scene.This supplies the missing scene-query component that connects perception to SayCan planning.
  • Integration with SayCan: Unlike SayCan’s hard-coded object and option set, NLMap proposes objects, queries their availability, and generates find/pick/place options dynamically.The resulting framework can expand navigation and manipulation choices beyond the finite predefined set.
  • Comparison: SayCan fails on “water the plant” when plant is absent from its predefined options, whereas NLMap queries relevant objects before planning.The figure contrasts fixed-list scoring with NLMap’s object proposal, map query, and template-based option generation.

V. EXPERIMENTS

NLMap is evaluated with SayCan on real-world tasks in an office kitchen, where it adds scene context and open-vocabulary object handling to language-based planning. Across benchmarks, it retains performance on existing tasks, enables novel-object tasks, and can detect some infeasible tasks caused by missing objects.

  • Experimental setup: NLMap + SayCan is evaluated end-to-end on natural-language tasks with a mobile manipulator in a real office kitchen.The robot uses an RGBD camera, imitation-learned manipulation policies, and PaLM 540B for planning.
  • System capability: NLMap combines with SayCan to remove fixed sets of objects, locations, and executable options, enabling previously unachievable tasks.The system generates relevant object names and locations conditioned on both the instruction and the scene.
  • SayCan tasks: 66.7% success on 18 SayCan tasks versus 84% for privileged SayCan measures the performance gap after adding perception and context-aware planning.Including two typo instructions, overall success was 60% versus 65% for privileged SayCan in real-robot experiments.
  • Novel objects: 80% end-to-end success on novel-object tasks contrasts with SayCan’s failure to plan or execute any of them.The novel-object benchmark includes out-of-distribution instructions, although manipulation remains limited to objects visually similar to training objects.
  • Missing objects: 40% success in the missing-object setting shows that context-aware planning can identify some infeasible tasks, while false-positive detections caused 56% of failures.Vanilla SayCan achieved 0% success in comparison.

B. Benchmarking Object Proposal

The object proposal module is tested on instructions requiring inferred, unstructured, fine-grained, and appropriately granular object descriptions. It performs strongly on inferred and crowd-sourced instructions, but fine-grained descriptions and proper granularity remain harder cases.

  • Benchmark design: Object proposal addresses four cases: inferred objects, unstructured instructions, fine-grained descriptions, and decomposition to proper granularity.Examples include inferring a microwave for heating a taco and extracting ingredients from a breakfast instruction.
  • Decomposition to proper granularity: Proper granularity is the hardest task family, and its performance declines sharply with smaller models while other families decline only slightly.The benchmark summary reports very high success for the other task families.
  • Unstructured crowd-sourced instructions: 96% success on 25 crowd-sourced instructions demonstrates robustness to highly unstructured language, including multi-step tasks.The module succeeded in 8 of 9 multi-step tasks in this study.
  • Fine-grained descriptions: 72% success on fine-grained descriptions shows that identifying instances specified by adjectives or clauses remains more difficult.These descriptions are intended to preserve distinctions needed to identify particular visual instances.

4) Decomposition to proper granularity:

NLMap combines open-vocabulary object querying, context-aware planning, and multi-view perception to support long-horizon real-world tasks. Object-proposal granularity remains difficult, while the integrated system improves planning and enables tasks beyond prior methods.

  • Decomposition to proper granularity: Object proposal achieves a 60% success rate on 15 tasks requiring expansion of an instruction-mentioned category.The result indicates that selecting the proper proposal granularity remains challenging for LLMs.
  • Object querying: The open-vocabulary query module evaluates 50 common objects across two kitchen scenes, including deliberately missing objects, using ensemble scores and multi-view fusion.Baselines include CLIP, ViLD, and maximum ensemble without multi-view fusion.
  • RoI retrieval: The maximum ensemble of CLIP and ViLD features provides the best RoI retrieval results because the two features exhibit complementary failure cases.CLIP and ViLD individually fail on distinct cases.
  • Context-grounded planning: 85% success on positive object sets and 60% on negative sets show that context-conditioned generative planning handles object availability, though negation remains harder.The benchmark contains 80 cases from 40 instructions, with positive and negative available-object sets.
  • Integrated system: NLMap grounds LLM-based planners in their environments, significantly improves long-horizon natural-language planning, and enables tasks prior algorithms failed to address.The system integrates a queryable scene representation with SayCan.
  • Limitations and future work: NLMap currently represents static scenes and leaves dynamic objects and humans for future work.The authors also plan fine-tuning and more efficient exploration for scene-representation construction.

APPENDIX

Context-aware SayCan extends the original planner with detected scene objects and template functions, translating executable language options into available robot skills before execution.

  • Inputs: The planner takes a high-level instruction, detected scene object names, template functions, an initial state, skills, affordance functions, and skill descriptions.These inputs support context-conditioned option generation and execution.
  • Option construction: For each detected object and template function, the algorithm creates an executable option and binds it to the nearest neural policy.A template maps an object name to an option such as “pick up [x]”.
  • Execution: The selected option is translated into its bound policy, executed in the environment, and followed by a state update.The loop repeats until the planner emits “done”.

B. Multi-view fusion algorithm

NLMap fuses multiple context elements associated with queried objects by clustering Gaussian spatial representations and retaining clusters that pass score criteria.

  • Representation: Each context element stores an object centroid and width, while overlapping predictions for one real object must be aggregated.The procedure is motivated by post-processing for duplicate object detections.
  • Similarity metric: The method models each cylindrical bounding volume as a Gaussian and uses KL divergence instead of bounding-box IoU to measure similarity.The Gaussian center is the estimated centroid and its standard deviation is proportional to object width.
  • Clustering: For a queried object name, the algorithm scores context elements, selects the top k, and clusters them using Gaussian distributions, a KL threshold, and score thresholds.A monotonic function increases confidence according to the number of elements in a cluster.
  • Output: The fusion algorithm outputs clustered locations for objects queried by name.These clustered locations provide the spatial result returned by the query process.

C. Prompt used for object proposal and for planning

NLMap + SayCan uses language prompts to propose objects from instructions and condition plans on which proposed objects are available. Examples cover single-step, sequential, semantic, and unavailable-object cases.

  • Object proposal: The object-proposal prompt maps instructions to potentially involved objects, including fine-grained items, destinations, containers, and semantic alternatives.Examples include fruits for “put the fruits in the fridge,” cleaning tools for spilled coke, and drink brands for soft drinks.
  • Object proposal: The prompt includes task-specific proposals for food, beverages, tools, appliances, containers, destinations, and actions such as checking time or discarding objects.The examples span coffee, water, dishes, knives, bananas, and trash-related tasks.
  • Task planning: Planning examples condition action sequences on available objects and can decompose multi-object instructions into ordered subtasks.For grapes and cheese, the plan completes the bowl placement before moving cheese to the table.
  • Availability conditioning: When required objects are unavailable, the planner explains that it will do nothing and emits only a terminal “done” action.This behavior is shown for missing mug, cardboard box, orange, apple, and water bottle examples.

D. Object proposal experiment task list

The object-proposal experiment spans instructions requiring complex language understanding, fine-grained object descriptions, category interpretation, and decomposition to an appropriate granularity.

  • The examples include locating, fetching, moving, discarding, combining, and preparing diverse household objects and foods.
  • The task list tests whether robots can infer relevant objects from complex human language inputs.
  • Fine-grained descriptions require identifying objects through attributes such as color, material, shape, or contextual references.
  • Category-based requests test proposing appropriate object types and decomposing broad categories to the right granularity.

E. Robot experiment task list

The robot experiment uses SayCan-style office-kitchen scenes to evaluate object finding and manipulation instructions, including absent-object cases and multi-object tasks.

  • The experiment uses task lists from a SayCan scene setup and an office kitchen containing common and testing objects.
  • Tasks require moving, placing, watering, washing, microwaving, locating, and arranging objects relative to destinations.
  • Several instructions combine multiple objects or actions, such as moving an apple, watering a plant, or preparing items for viewing television.
  • A second scene omits several named testing objects, including apples, cans, drinks, bags, and a protein bar.

F. Additional qualitative experiment results

Additional qualitative results show NLMap paired with SayCan executing object-finding and long-horizon tasks, while visualizations illustrate natural-language scene queries and matches.

  • The qualitative experiments include robot executions and additional NLMap query visualizations.
  • In another example, the robot finds an apple and compost, then plans to put the apple in the compost.
  • For “Move the apple to the plant and water the plant,” object proposal identifies apple, water, and plant before a nine-step robot sequence.
  • Long-horizon examples contain sequences of 9 and 8 steps, demonstrating object proposal and planning across extended tasks.
  • NLMap query plots show the query word, heatmap matches overlaid on the map, and the top four region-of-interest matches.
Loading 2209.09874v2…