PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifiably Following Complex Robot Instructions with Foundation Models

Benedict Quartey, Eric Rosen, Stefanie Tellex, George Konidaris

arXiv:2402.11498v3cs.ROcs.AI

TL;DR

Robots need to disambiguate open-ended instructions and ground their referents while allowing people to verify intended behavior. LIMP combines foundation models, temporal logic, and task-and-motion planning; it outperforms baselines in most task sets, particularly complex temporal planning and constraint avoidance.

  • Problem

    Robot instructions can contain ambiguous phrasing, complex spatiotemporal constraints, and unique referents that must be grounded and disambiguated.

    Source: §I · p. 1

  • Method

    LIMP uses foundation models to translate instructions into temporal logic, dynamically ground open-vocabulary referents, and synthesize task-and-motion plans.

    Source: §I · p. 2, C. Task and Motion Planning Module · p. 4

  • Results

    LIMP outperforms baselines in most task sets, notably complex temporal planning and constraint avoidance, but underperforms NLMap-Saycan on NLMC.

    Source: §VI · p. 6

  • Takeaways & Limitations

    Combining language and vision foundation models with methods for safety, explainability, and verifiable behavior synthesis can lead to more reliable and capable robotic systems.

    Source: §VII · p. 6

    The planner currently handles only co-safe temporal-logic formulas, which cover finite sequences.

    Source: §VII · p. 6

Abstract

from arXiv · show

When instructing robots, users want to flexibly express constraints, refer to arbitrary landmarks, and verify robot behavior, while robots must disambiguate instructions into specifications and ground instruction referents in the real world. To address this problem, we propose Language Instruction grounding for Motion Planning (LIMP), an approach that enables robots to verifiably follow complex, open-ended instructions in real-world environments without prebuilt semantic maps. LIMP constructs a symbolic instruction representation that reveals the robot's alignment with an instructor's intended motives and affords the synthesis of correct-by-construction robot behaviors. We conduct a large-scale evaluation of LIMP on 150 instructions across five real-world environments, demonstrating its versatility and ease of deployment in diverse, unstructured domains. LIMP performs comparably to state-of-the-art baselines on standard open-vocabulary tasks and additionally achieves a 79\% success rate on complex spatiotemporal instructions, significantly outperforming baselines that only reach 38\%. See supplementary materials and demo videos at https://robotlimp.github.io

I. INTRODUCTION

Robots must resolve ambiguous referents and complex spatiotemporal constraints while making their understanding verifiable. LIMP combines instruction-conditioned semantic mapping, temporal logic, and task-and-motion planning to address these challenges without prebuilt semantic maps.

I. INTRODUCTION

  • Natural-language instructions can combine ambiguous phrasing, intricate spatiotemporal constraints, and unique referents that robots must ground and disambiguate.Existing approaches largely focus on navigation or handle only simple temporal constraints in unambiguous language.Source: §I · p. 1
  • The Figure 1 example plans a pick-and-place sequence while grounding spatially described objects without prior semantic knowledge of the environment.The robot travels from its start to a green plush toy, picks it up, then navigates to a whiteboard and places it there.Source: Figure 1 · p. 1
  • LIMP uses foundation models and temporal logic to generate instruction-conditioned semantic maps for verifiable navigation and mobile manipulation with open-vocabulary referents.The approach dynamically grounds referents rather than relying on prebuilt semantic maps with prespecified landmarks.Source: §I · pp. 1–2
  • LIMP translates instructions into temporal logic, grounds open-vocabulary objects in 3D maps, and uses task-and-motion planning to synthesize constraint-satisfying plans.Its three stated contributions include a modular framework, spatial grounding with constraints, and long-horizon planning over goal and avoidance regions.Source: §I · p. 2

II. BACKGROUND AND RELATED WORKS

Prior work grounds open-vocabulary queries or translates language into temporal logic, but the cited approaches have limits in modeling relationships, resolving landmarks, or supporting open-ended goals.

II. BACKGROUND AND RELATED WORKS

  • NLMap grounds open-vocabulary queries to spatial locations but cannot handle complex constraints between multiple objects because it lacks object-relationship modeling.LIMP adds spatial grounding of relationships and uses task-and-motion planning to satisfy those constraints.Source: §II · p. 2
  • Lang2LTL and its extension translate instructions into temporal logic and ground referents, but require prebuilt semantic maps or databases of landmarks.LIMP dynamically generates landmarks from open-vocabulary instructions.Source: §II · p. 2
  • Action-Oriented Semantic Maps integrate manipulation-skill models with task-and-motion planning, but remain constrained to a fixed set of goals once generated.LIMP provides a similar TAMP-compatible spatial representation while supporting open-vocabulary tasks.Source: §II · p. 2

A. Linear Temporal Logic

The paper uses Linear Temporal Logic to express complex robot mission requirements and to connect instructions with verifiable behavior synthesis.

A. Linear Temporal Logic

  • LIMP chooses Linear Temporal Logic (LTL) to represent complex robot mission requirements, although the framework is compatible with other specification languages and planning frameworks.The passage describes LTL using atomic propositions, logical and temporal operators.Source: A. Linear Temporal Logic · p. 2
  • LTL can represent temporal properties with operators including next, until, globally, and finally, but constructing specifications has traditionally required expert knowledge.Recent work has translated natural language directly into LTL,,,,,.Source: A. Linear Temporal Logic · p. 2
  • Temporal-logic specifications let the system synthesize plans that are correct-by-construction instead of relying on explicit verification methods such as model checking,,.The passage identifies this as an approach to verifying generated-plan correctness a priori.Source: A. Linear Temporal Logic · p. 2

III. PROBLEM DEFINITION

The problem is to sequence navigation and manipulation behaviors that satisfy an instruction’s spatial and temporal constraints. LIMP grounds instruction-specific referents without assuming a prebuilt semantic map.

III. PROBLEM DEFINITION

  • The objective is to synthesize a policy whose navigation and manipulation behaviors satisfy the instruction’s spatial and temporal constraints.Spatial constraints depend on the robot poses traversed, while temporal constraints specify how those constraints are sequenced as the task progresses.Source: §III · p. 2
  • The setup assumes RGB-D images and camera poses from a prior traversal to build a metric map, but does not assume a semantic map of prespecified object locations or predicates.A vision-language model detects text-specified objects, while a large language model samples language tokens.Source: §III · p. 2
  • Navigation seeks paths to goal poses through feasible regions while avoiding obstacles and instruction-defined semantic regions that violate constraints.The goal and infeasible regions are defined over the robot’s possible poses.Source: §III · p. 2
  • Manipulation behaviors are object-parameterized options, and executing a skill requires first navigating near the target object.The setup assumes access to a library of manipulation skills.Source: §III · p. 2
  • The system translates instructions into temporal logic, grounds referents in a 3D semantic map, and compiles the specification for task-and-motion planning.The example evaluation includes multi-object goal navigation and open-vocabulary mobile pick-and-place,.Source: Figure 2 · p. 3, §III · p. 3

IV. LANGUAGE INSTRUCTION GROUNDING FOR MOTION PLANNING

LIMP interprets expressive natural-language instructions to generate instruction-conditioned semantic maps, enabling robots to verifiably solve long-horizon tasks with complex spatiotemporal constraints.

IV. LANGUAGE INSTRUCTION GROUNDING FOR MOTION PLANNING

  • LIMP interprets expressive natural-language instructions to generate instruction-conditioned semantic maps.The section introduces a modular approach, with implementation details in supplementary materials.Source: §IV · p. 3
  • The maps support verifiable solutions to long-horizon tasks with complex spatiotemporal constraints.Source: §IV · p. 3
  • The approach is modular, with comprehensive implementation details provided in supplementary materials.Source: §IV · p. 3

A. Language Instruction Module

The language module translates instructions into LTL specifications with composable referent descriptions, then presents extracted referents and task structure for human verification. It does not require fixed phrasing or keywords, and initial formulas encode correct referents and temporal structure in the reported experiments.

A. Language Instruction Module

  • A two-stage LLM strategy translates an instruction into conventional LTL and then into a formula whose predicates represent parameterized robot skills.The second stage uses both the instruction and conventional formula as input.Source: A. Language Instruction Module · p. 3
  • Composable Referent Descriptors (CRDs) chain spatial comparators to represent specific referent instances for near, pick, and release predicates.The example describes a yellow cabinet above a fridge that is next to a stove.Source: A. Language Instruction Module · p. 3
  • The LLM samples formulas with context-appropriate spatial comparators, without requiring specific phrasing or keywords.The authors contrast this flexibility with recent works,.Source: A. Language Instruction Module · p. 3
  • An interactive dialog presents extracted CRDs and implied task structure for human verification, with user corrections prompting new formulas.The task structure is translated back into English statements through a deterministic scheme.Source: A. Language Instruction Module · pp. 3–4
  • Initial formulas sampled by the language module encode the correct referents and temporal task structure, even without human verification or reprompting.Source: A. Language Instruction Module · p. 4

B. Spatial Grounding Module

The spatial grounding module detects instruction referents in prior observations, maps them into 3D, and resolves spatial descriptions to localize the specified instances. Users can inspect the resulting maps, while interactive correction of VLM misclassifications is outside the work’s scope.

B. Spatial Grounding Module

  • The module detects and segments referent occurrences in prior observations, then backprojects their mask pixels onto a 3D map to create an initial semantic map.Source: B. Spatial Grounding Module · p. 4
  • Spatial comparators applied to detected referent and landmark positions filter instances to produce a Referent Semantic Map of the described objects.Comparator resolution uses the origin coordinate frame; object center-mask pixels represent positions.Source: B. Spatial Grounding Module · p. 4
  • VLM misclassification is identified as the module’s main error source; interactive correction is out of scope, but 3D visualization tools let users inspect map localization.Source: B. Spatial Grounding Module · p. 4

C. Task and Motion Planning Module

The planning module compiles the instruction’s LTL formula into an automaton and sequences navigation and manipulation behaviors along a path to an accepting state. Navigation objectives use semantic maps encoding goal and constraint regions for continuous path planning.

C. Task and Motion Planning Module

  • The planner compiles the LTL formula into an equivalent finite-state automaton and selects a path from its initial to accepting state to sequence navigation and manipulation objectives.Transition expressions determine low-level behaviors: parameterized manipulation skills or navigation planning with FMT∗.Source: C. Task and Motion Planning Module · p. 4, Figure 5 · p. 5
  • For navigation subgoals, Task Progression Semantic Maps localize goal and constraint regions for continuous path planning.The maps are converted into 2D obstacle maps for constraint-aware planning.Source: C. Task and Motion Planning Module · p. 4, Figure 5 · p. 5
  • A TPSM encodes navigation constraints using regions of interest defined by a nearness threshold, which may be set globally or inferred from the instruction.The approach is robot-agnostic and supports direct planning in 3D TPSMs for embodiments such as drones.Source: C. Task and Motion Planning Module · p. 5

V. EVALUATION

The evaluation tests whether LIMP can interpret complex instructions, resolve described object instances, and produce constraint-satisfying plans. It compares LIMP with two LLM-based planners across five real-world environments and 150 instructions.

V. EVALUATION

  • The evaluation asks whether LIMP interprets complex ambiguous instructions, resolves specific described objects, and generates plans that satisfy constraints.These questions address the language instruction, spatial grounding, and task-and-motion-planning modules.Source: §V · p. 5
  • LIMP is compared with NLMap-Saycan and Code-as-Policies, with both baselines using GPT-4-0613 and the same prompting structure and in-context examples.The baselines also receive LIMP's CRD syntax, spatial grounding module, and low-level robot control.Source: §V · p. 5
  • Two language-module ablations use randomly selected examples or skip the conventional-LTL stage and directly generate the parameterized-skill syntax.The full approach selects relevant in-context examples by cosine similarity.Source: §V · p. 5
  • The benchmark covers 150 instructions across five real-world environments, spanning five task sets from fine-grained object descriptions to complex spatiotemporal constraints.The sets include NLMD, NLMC, OKRB, CT, and CST.Source: §V · p. 5
  • Instruction understanding is assessed with referent, avoidance-constraint, and spatial-relationship resolution metrics, while plan evaluation measures temporal alignment and success.A plan is temporally aligned when its subgoal sequence matches the instructor’s intention, and successful when it satisfies all stated spatial and temporal constraints.Source: §V · pp. 5–6

VI. DISCUSSION

LIMP outperforms baselines in most task sets, especially complex temporal planning and constraint avoidance. Its exception is NLMC, where NLMap-Saycan performs better on instructions requiring implicit preference inference.

VI. DISCUSSION

  • LIMP outperforms baselines in most task sets, notably complex temporal planning and constraint avoidance.The authors attribute its advantage on CT and CST tasks to ensuring each robot step follows constraints while achieving subgoals.Source: §VI · p. 6
  • In NLMC, LIMP underperforms NLMap-Saycan, which is better suited to infer options from implicit preferences such as a request for a liked fruit.The authors say their few-shot LTL translation process is not designed for this inference.Source: §VI · p. 6

VII. LIMITATIONS AND CONCLUSION

This section describes LIMP’s limitations and instruction-verification and spatial-grounding mechanisms, including its reliance on finite-horizon specifications, reliable recognition, static scenes, and fixed-threshold spatial comparisons. It concludes that combining foundation models with established safety and verifiable-synthesis methods could support more reliable, capable robots.

VII. LIMITATIONS AND CONCLUSION

  • LIMP’s planner handles only co-safe LTL formulas, grounding depends on VLM recognition, and mapping assumes static scenes; its motion planner is complete but not optimal.The authors identify editable scene representations as future work and note that LIMP can be used with existing TAMP planners to enhance efficiency.Source: §VII · p. 6

A. Foundation Models in Robotics

  • Unlike NLMap and related systems, LIMP supports relational open-vocabulary grounding, LLM and off-the-shelf TAMP planning, verifiable instructions, and instruction-specific semantic maps.NLMap handles descriptions of individual objects but cannot represent relationships between multiple objects; LIMP’s scene representation supports those relationships and works with both LLM planners and TAMP solvers.Source: A. Foundation Models in Robotics · p. 9

B. Planning Models in Robotics

  • Unlike AOSMs, LIMP uses foundation models without online learning and supports open-vocabulary tasks; its representation supplies geometry for bi-level TAMP solvers,.AOSMs support a closed set of goals once generated, whereas LIMP can handle open-vocabulary task specifications.Source: B. Planning Models in Robotics · pp. 9–10

A3. LANGUAGE INSTRUCTION MODULE

  • The instruction module first translates commands into conventional LTL using K examples selected by cosine similarity, then maps formulas to parameterized robot skills and spatial CRDs using eight comparators.The second stage uses the instruction and first-stage LTL formula as inputs to generate predicates corresponding to parameterized skills.Source: A3. LANGUAGE INSTRUCTION MODULE · p. 10

A. Interactive Symbol Verification

  • Users review sampled formulas’ referents and translated task sequences, then approve them or provide corrective feedback that prompts formula regeneration.The system presents extracted CRDs and the implied sequence of state-machine transition expressions in English.Source: A. Interactive Symbol Verification · p. 11
  • Interactive verification presents an English translation of the symbolic task and asks whether it matches the user’s intention.The example dialog includes a logical expression and translated task statements.Source: A. Interactive Symbol Verification · p. 11

A4. SPATIAL GROUNDING MODULE

  • Spatial grounding detects and segments open-vocabulary object instances, then filters them using spatial details in composable referent descriptors.For example, a whiteboard is retained when it is in front of at least one green plush toy.Source: A4. SPATIAL GROUNDING MODULE · p. 11

A. 3D Spatial Comparators

  • 3D spatial comparators support relative-frame reasoning between referents, with user-settable sensitivity thresholds held fixed across experiments.The described relations include directional, vertical, between, and proximity comparisons.Source: A. 3D Spatial Comparators · p. 11

A5. TASK AND MOTION PLANNING MODULE

The task-and-motion-planning module builds task-progression semantic maps to mark goal and constraint-violating regions, then uses them to support navigation and manipulation skills.

A5. TASK AND MOTION PLANNING MODULE

  • The planner builds a task-progression semantic map by evaluating nearby points around referents against automaton transitions, marking goal, undesired, and unaffected regions.The map combines the environment map with a referent semantic map and uses the desired automaton state and nearness threshold.Source: A5. TASK AND MOTION PLANNING MODULE · p. 12

A6. ROBOT SKILLS

  • Navigation, picking, and placing use near, pick, and release predicates; picking detects and segments the object before sending its mask center to a grasp-planning API.The pick routine uses Owl-Vit for detection and Segment Anything for segmentation.Source: A6. ROBOT SKILLS · p. 12
  • Placement currently releases the grasped object by moving the arm according to the receptacle’s 3D position; more complex semantic placement is future work.The release routine is described as a simple routine.Source: A6. ROBOT SKILLS · p. 13

A7. EVALUATION AND BASELINE DETAILS

This section details the evaluation setup, baseline implementations, and 150-instruction test set across five real-world environments. It also presents examples of object-manipulation and landmark-navigation requests involving ordered actions, route constraints, temporal conditions, and instruction correction.

A7. EVALUATION AND BASELINE DETAILS

  • NLMap-Saycan and Code-as-Policies share LIMP’s GPT-4, prompting structure, and examples, while using its referent syntax, spatial grounding, FMT* planner, and robot control as APIs.The APIs let the baselines query object positions, plan paths, and execute manipulation options.Source: A7. EVALUATION AND BASELINE DETAILS · p. 13
  • Figure A.2 illustrates that interactive symbol verification and reprompting improve instruction satisfaction with minimal chat turns across instruction sets.The passage identifies this as the A3-A strategy.Source: A7. EVALUATION AND BASELINE DETAILS · p. 13, Figure A.2 · p. 14

A. NLMap-Saycan Implementation Prompt

  • The NLMap-Saycan prompt [10] restricts plans to finite robot and spatial predicates, composes spatial predicates where needed, and sequences repeated actions.Its examples use navigation, pick, and release actions with spatially described referents.Source: A. NLMap-Saycan Implementation Prompt · pp. 13–14

B. Code-as-Policies Implementation Prompt

  • The Code-as-Policies prompt [12] likewise restricts the predicate set, composes spatial referents, and sequences repeated actions such as picking and releasing.The example for delivering two beers uses a loop to repeat the actions.Source: B. Code-as-Policies Implementation… · pp. 14–15

C. Instruction set

  • The test set covers 150 instructions across five real-world environments: 24 NLMD, 25 NLMC, 25 OKRB, 37 CT, and 39 CST tasks.NLMD and NLMC come from, OKRB from, and CT from; CST tasks are proposed by the authors and combine descriptive spatial constraints with temporal structures.Source: C. Instruction set · p. 15
  • NLMC examples include object transfers and multi-object requests, such as bringing two different sodas to a cabinet.The listing also includes requests involving cleanup and choosing items based on a stated preference.Source: C. Instruction set · p. 15
  • OK-Robot examples range from moving household objects between landmarks to navigation instructions with order, visit-count, and avoidance constraints.Some examples also contain conflicting requirements, such as visiting a counter while never visiting it.Source: C. Instruction set · p. 15
  • Lang2LTL complex-temporal examples combine landmark descriptions with ordered visits, prohibitions, and revisions to earlier requests.Examples include locating objects by spatial relations and returning to a previously visited landmark.Source: C. Instruction set · p. 16
  • Tasks also require sequencing and temporal structure, including ordered visits, repeating a sequence three times, and avoiding the fridge on the second repetition.One instruction explicitly refers to objects by their order of mention, then combines that reference with a destination and an avoidance requirement.Source: C. Instruction set · p. 16
  • Examples pair goal-directed requests with route restrictions: avoid cabinets while retrieving and moving a mug, stay away from a trash bin throughout, avoid robots while carrying a bag, and avoid a table on return with a toy.Source: C. Instruction set · p. 16
  • Examples specify objects through spatial relations to landmarks and destinations, including items beside, between, under, or left of named objects.Tasks also use people and environmental cues to identify destinations, such as a speaker sitting by the whiteboard or meeting the robot at the bed.Source: C. Instruction set · p. 16
  • One example revises the requested object mid-instruction, canceling a bag retrieval and substituting a green toy at the same location.Source: C. Instruction set · p. 16
Loading 2402.11498v3…