Source-linked AI summary
Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation
Vishnu Sashank Dorbala, James F. Mullen, Dinesh Manocha
TL;DR
LGX addresses language-driven zero-shot object navigation, where an embodied agent must find a uniquely described object in an unseen environment. It combines LLM-guided sequential exploration with vision-language target grounding, achieving a 27% improvement on RoboTHOR and demonstrating real-world performance.
Problem
L-ZSON requires agents to navigate to arbitrarily described objects in unseen environments, beyond fixed categories and conventional supervised settings.
Method
LGX uses scene observations to prompt an LLM for exploratory navigation decisions while a vision-language model grounds the target object.
Results
LGX improves zero-shot SR and SPL on RoboTHOR by 27% and is evaluated in real-world experiments with visually unique objects.
Takeaways & Limitations
The approach combines LLM commonsense reasoning and open-vocabulary grounding for language-guided zero-shot navigation in simulation and real-world settings.
Takeaways & Limitations
LGX has failure cases when the LLM incorrectly localizes the target object, including entering rooms that do not contain it.
Abstract
from arXiv · showhide
We present LGX (Language-guided Exploration), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to a uniquely described target object in a previously unseen environment. Our approach makes use of Large Language Models (LLMs) for this task by leveraging the LLM's commonsense reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via \textit{real-world} experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.
I. INTRODUCTION
The paper targets language-driven zero-shot object navigation, where agents must find arbitrarily described objects in unseen environments. LGX combines LLM-guided exploration with vision-language grounding and reports simulation and real-world validation.
- LGX addresses L-ZSON, requiring an agent to find a freeform-described target object in an unseen environment without prior exposure to the environment or target.
- The task separates sequential exploratory decision-making from target-object grounding, both of which must generalize beyond fixed object categories.
- The approach studies prompt formulations and LLM commonsense reasoning for navigation, alongside open-vocabulary grounding for unconstrained object references.
- LGX uses visual scene information to prompt an LLM for object sub-goals or directions while a vision-language model grounds the natural-language target.
- The paper reports a 27% improvement in zero-shot SR and SPL on RoboTHOR and evaluates transfer to a real-world robotics platform.
II. RELATED WORK
Related work spans language-guided robotics, zero-shot embodied navigation, and open-vocabulary grounding. LGX builds on these directions by combining GLIP with LLM-driven exploration in unseen environments.
- Prior language-guided robotics work parses unconstrained instructions and uses dialogue or grounding structures for manipulation and navigation.
- Zero-shot embodied-navigation studies use CLIP or OWL-ViT to navigate unseen environments, including L-ZSON with frontier-based exploration.
- LGX contrasts with these approaches by using GLIP for zero-shot object grounding and GPT-3 to make navigational decisions during exploration.
C. Language-Guided Scene Manipulation
The paper distinguishes LGX from language-guided manipulation and prior LLM planning systems. Its LLM is used to explore an unseen environment rather than execute plans in a known one.
- C. Language-Guided Scene Manipulation: Language-guided scene-manipulation methods process language to manipulate or place objects, whereas LGX focuses on language-guided exploration.
- C. Language-Guided Scene Manipulation: Prior LLM robotics systems decompose high-level instructions into actionable subtasks while assuming environmental information is already known.
- C. Language-Guided Scene Manipulation: LGX instead uses an LLM as a prior to predict exploration directions in an unseen environment without requiring a map.
E. LLMs for Language-Guided Navigation
LGX differs from related navigation work by using an LLM for environment exploration rather than planning objectives. It combines scene-derived prompts, commonsense reasoning, and visual feedback to make sequential navigation decisions.
- Relation to prior work: Unlike LM-Nav and VLMaps, LGX translates visual scene semantics into action guidance for exploration rather than using an LLM for path planning or code writing.
- LLM-guided exploration: LGX uses visual scene descriptions and an LLM to select navigational directions or objects during sequential exploration.The scene input can consist of detected object labels or image captions, while the LLM supplies the next exploratory decision.
- Target verification: GLIP checks the described target during exploration and terminates the loop when its confidence exceeds Gth and the target is in view.The resulting episode is successful when the target is in the agent’s view during rotation.
- Scene understanding: The agent rotates in place to observe its surroundings before navigation, giving the LLM information from the full current viewpoint.Depth observations simultaneously support construction of an environmental costmap for navigation.
C. Intelligent Exploration with Large Language Models
LGX combines LLM-guided exploration with open-vocabulary GLIP grounding to navigate toward uniquely described objects. The LLM uses extracted scene semantics for movement, while GLIP verifies and localizes the target.
- LLM exploration: LGX prompts GPT-3 with detected objects or image captions and uses its response to choose an object or direction for exploration.The YOLO→LLM variant selects an object from detected labels, while BLIP→LLM selects among four directions from captions.
- Exploration loop: During each 360-degree observation, GLIP checks the target and ends exploration when confidence exceeds Gth; otherwise, LLM-guided exploration continues.
- Target grounding: GLIP grounds the target object from the image and prompt, returning detections and bounding boxes that localize a navigation direction.Its inputs are the image and target prompt; its outputs include detected objects and bounding boxes.
- Target grounding: GLIP can distinguish a uniquely described cat-shaped mug from related cat and mug objects, supporting grounding of unconstrained target descriptions.
B. Examining LLM Prompts for Exploration
Because GPT-3’s commonsense navigation decisions depend on prompt wording, LGX evaluates multiple prompt formulations. The prompts vary perspective, information order, requested response format, and semantic input type.
- Prompt motivation: LGX compares seven prompts because GPT-3’s output is strongly influenced by prompting and drives commonsense navigation decisions.Invalid responses cause the agent to move randomly.
- Open-vocabulary grounding: GLIP demonstrations show that language such as “cat-shaped mug” can identify the unique target while distinguishing related cat and mug objects.
- Prompt variants: The evaluated prompts vary the speaker perspective, including robot, first-person, and third-person formulations.
- Prompt variants: Other variants change whether object information appears first, ask for the closest object, or place the one-word response instruction first.
- Prompt variants: The BLIP prompt replaces object selection with a directional choice among Front, Right, Behind, and Left using captions from the four views.
V. EXPERIMENTS AND RESULTS
LGX is evaluated in RoboTHOR and real-world experiments covering open-vocabulary grounding and LLM-based exploration. The experiments assess success, path efficiency, prompt validity, and two-phase room-to-room navigation.
- Simulation setup: The RoboTHOR validation setup contains 1,800 episodes across 15 environments and 12 goal-object categories, with exploration limited to five turns or target detection above Gth.
- Prompt evaluation: Prompt ablations evaluate seven prompts on 500 best- and worst-performing episodes, using Prompt Success Rate to measure valid LLM responses.A valid response selects a detected object or navigation direction, depending on the semantic extraction scheme.
- Real-world evaluation: Real-world exploration uses two phases: reaching a hallway from the starting room, then selecting the target-containing room from four possibilities.The setup is intended to test navigation through partially observable household environments.
- Simulation results: RoboTHOR experiments compare LGX with prior L-ZSON methods using Success Rate and SPL, with Table III reporting improvements over CoW and OWL.The supplied table caption identifies SR and SPL as the reported comparison metrics.
- Class breakdown: LGX improves localization for baseball bat, bowl, laptop, spray bottle, and vase classes, while performance is similar for television and garbage can.
B. Baselines and Ablations
LGX is compared with frontier-based and random exploration baselines, including variants that replace its LLM-based exploration or target-grounding components.
- LGX is evaluated against CLIP-on-Wheels, OWL CoW, GLIP on Wheels, and Random with GLIP.GLIP on Wheels removes the LLM-based exploration mechanism, while Random with GLIP uses uninformed random exploration.
C. Comparison with Baselines in Simulation
In RoboTHOR, LGX improves over established baselines, while prompt design affects exploration performance and exposes limitations of the simulation setting.
- LGX significantly improves success rate and SPL over OWL CoW and CoW, and improves success rate over GoW.The authors attribute the success-rate improvement over GoW to the LLM-based exploration scheme combined with GLIP grounding.
- LGX outperforms CoW and OWL on smaller targets such as bowls and vases, but performs similarly to OWL on larger televisions.The comparison is reported across individual RoboTHOR target objects.
- Object-based prompts outperform natural-language BLIP prompts, which are associated with a restricted four-direction action space and occasional directional loops.The BLIP prompting scheme achieved perfect PSR but lower task performance, according to the authors.
- Prompt perspective produces no significant task-SR difference despite large PSR variation, with the robot-perspective prompt having the highest SR and lowest PSR.Robot-perspective prompts more often elicited “no” or “nothing” responses than empty responses.
- Changing prompt structure generally does not significantly affect SR, but placing the “ONE word” instruction first substantially worsens SR and PSR.Object-set-first and get-closest-object prompts retain similar SR, while their PSR patterns suggest easier object reference or selection for the LLM.
- The authors argue that weak task-SR differences despite large PSR changes indicate RoboTHOR may provide a skewed basis for context-dependent exploration.This limitation concerns how well the simulation tests intelligent, context-dependent scene exploration.
E. Comparison with Baselines in the Real World
In real-world experiments, LGX outperforms the selected baselines, while performance remains constrained by grounding errors and incorrect LLM-based localization.
- GLIP detects present goal objects 87.5% of the time and falsely detects absent objects 8.3% of the time.These grounding errors affect success rates, including failures involving stevia sugar packets and false positives for Red Bull cans.
- LGX improves real-world SR over GoW by 26.4% and over Random with GLIP by 47.3%.All three methods use GLIP for target detection, but the baselines do not integrate scene context into exploration.
- LGX follows a two-phase real-world behavior in which successful runs navigate from the starting room through the hallway into the target room before GLIP detects the object.This success case occurs 54.2% of the time.
- A Phase 1 failure occurs when the LLM selects a plausible object in the starting scene instead of directing the agent toward the hallway.For example, the LLM selected “desk” for a Red Bull can target even though the can was in the kitchen.
- A Phase 2 failure occurs when the agent enters a room without the target, which happened 20.8% of the time and reflected poor object associations by the LLM.The method failed to localize the bust, although baselines indicate GLIP could detect it after entering the correct room.
- The paper concludes that LGX combines LLM navigation with open-vocabulary grounding, achieves state-of-the-art RoboTHOR results, and is validated in real-world experiments.The authors also study prompt structure and phrasing as part of the navigation analysis.
- LGX retains failure cases when the LLM incorrectly localizes the target object, motivating future work on richer context and visited-object history.The authors also call for more informative SR and SPL metrics for zero-shot navigation.
VII. DISCUSSION
LGX addresses real-world L-ZSON through open-vocabulary grounding and exploration under free-form language and partial observability. The discussion highlights GLIP’s role in grounding ambiguous descriptions and compares alternative detectors qualitatively.
- Real-World Applicability: The discussion presents additional experiments and analyses addressing LGX’s real-world applicability.
- Real-World Applicability: Zero-shot ObjectNav is challenging because free-form language and partial observability make exploration and visual grounding central real-world factors.
- Visual Object Grounding: GLIP detects only objects mentioned in the text prompt rather than proposing every object from an open set.
- Visual Object Grounding: GLIP grounds ambiguous free-form descriptions such as “cat-shaped mug,” identifying the unique object among images containing a cat and a mug.The qualitative comparison in Figure 8 supports GLIP’s use for ambiguous language grounding.
- Visual Object Grounding: LGX’s ObjectNav performance is sensitive to the quality of its visual grounding model.