Source-linked AI summary
Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill
Wenzhe Cai, Siyuan Huang, Guangran Cheng, Yuxing Long, Peng Gao, Changyin Sun, Hao Dong
TL;DR
Zero-shot object navigation needs visual grounding, commonsense inference, and locomotion, but map-based locomotion is difficult to integrate with RGB-trained foundation models and often relies on depth sensing and localization. PixNav uses a pixel in the initial RGB frame as a navigation goal, while an LLM planner guides exploration sequences across rooms. PixNav achieves competitive performance against baseline results without depth or localization inputs and retains high success in 1m-3m navigation despite 50% relative camera-height and 25% relative HFOV changes.
Problem
Zero-shot object navigation needs visual grounding, commonsense inference, and locomotion, but map-based locomotion is difficult to integrate with RGB-trained foundation models and often relies on depth sensing and localization.
Source: §I · p. 1Method
PixNav uses a pixel in the initial RGB frame as a navigation goal, while an LLM planner guides exploration sequences across rooms.
Source: §I · pp. 1–2Results
PixNav achieves competitive performance against baseline results without depth or localization inputs and retains high success in 1m-3m navigation despite 50% relative camera-height and 25% relative HFOV changes.
Source: C. Performance on Zero-Shot ObjectNav · p. 5, D. Ablation Study on PixNav · p. 5Takeaways & Limitations
The authors present PixNav as a versatile RGB-based path-planning policy that can generalize across camera settings and integrate with foundation models for long-horizon navigation.
Source: §I · p. 2, §V · p. 6The authors state that PixNav’s current long-term path-planning performance is not satisfactory.
Source: §V · p. 6
Abstract
from arXiv · showhide
Zero-shot object navigation is a challenging task for home-assistance robots. This task emphasizes visual grounding, commonsense inference and locomotion abilities, where the first two are inherent in foundation models. But for the locomotion part, most works still depend on map-based planning approaches. The gap between RGB space and map space makes it difficult to directly transfer the knowledge from foundation models to navigation tasks. In this work, we propose a Pixel-guided Navigation skill (PixNav), which bridges the gap between the foundation models and the embodied navigation task. It is straightforward for recent foundation models to indicate an object by pixels, and with pixels as the goal specification, our method becomes a versatile navigation policy towards all different kinds of objects. Besides, our PixNav is a pure RGB-based policy that can reduce the cost of home-assistance robots. Experiments demonstrate the robustness of the PixNav which achieves 80+% success rate in the local path-planning task. To perform long-horizon object navigation, we design an LLM-based planner to utilize the commonsense knowledge between objects and rooms to select the best waypoint. Evaluations across both photorealistic indoor simulators and real-world environments validate the effectiveness of our proposed navigation strategy. Code and video demos are available at https://github.com/wzcai99/Pixel-Navigator.
I. INTRODUCTION
Zero-shot object navigation combines visual grounding, commonsense inference, and locomotion, but map-based locomotion is difficult to align with foundation models and often requires costly depth sensing and localization. PixNav uses pixels as goals for RGB navigation, while an LLM planner guides exploration across rooms.
I. INTRODUCTION
- Map-based zero-shot navigation remains difficult to align with RGB-trained foundation models and commonly depends on costly, disturbance-prone depth sensing and localization.The section frames pure RGB navigation as the motivating question.Source: §I · p. 1
- PixNav uses pixels as goal specifications to connect foundation models with navigation and avoid task-specific multimodal alignment while supporting navigation to varied objects.Unlike word or one-hot object goals, a designated pixel specifies the target area.Source: §I · pp. 1–2
- Assigning different pixels on an object as goals can generate diverse navigation trajectories without explicit semantic annotations, reducing demonstration-collection cost.Off-the-shelf planning methods can generate actions toward the corresponding 3-D point.Source: Figure 1 · p. 1, §I · p. 2
- An LLM planner uses commonsense priors and a step-by-step prompt to guide exploration sequences across rooms, with potential for self-localization and structural memory organization.The authors report competitive performance against mapping-based methods without depth input.Source: §I · p. 2
A. Open-Vocabulary Object Navigation
Prior open-vocabulary navigation methods use text-image alignment or object tags, while map-based alternatives depend on depth and localization. This section situates PixNav among foundation-model and LLM navigation work, highlighting its step-by-step prompt for location reasoning and memory organization.
A. Open-Vocabulary Object Navigation
- Open-vocabulary object navigation addresses the close-set object limits of earlier work, but end-to-end policies can struggle to learn multimodal alignment without large-scale data, while map-based alternatives rely on depth and localization.PixNav is presented as a pixel-navigation alternative to traditional map-based path planning.Source: A. Open-Vocabulary Object Navigation · p. 2
B. Foundation Models on Vision and Language
- Foundation models provide zero-shot detection, object tagging, segmentation, and language capabilities; this work uses LLama-Adapter to convert images into text and GPT-4 for textual planning.The passage also describes vision-language models that combine image and language inputs.Source: B. Foundation Models on Vision and… · p. 2
C. Large-Language Model for Visual Navigation
- Unlike prior LLM navigation approaches that caption observations or score frontiers, the proposed step-by-step prompt is designed to pinpoint the agent’s location and organize memory.The authors identify self-localization and structural memory organization as potential extensions to other navigation tasks.Source: C. Large-Language Model for Visual… · pp. 2–3, Figure 2 · p. 3
III. METHOD
PixNav uses an initial-frame pixel as a goal for an RGB-based navigation policy. Its transformer processes image history and the goal pixel, with supervision for action prediction, pixel tracking, and temporal distance.
A. Path-Planning with Pixel Navigation
- PixNav uses a pixel in the initial frame as its goal and predicts primitive actions from the image history.Its six primitive actions are Stop, MoveAhead, TurnLeft, TurnRight, LookUp, and LookDown.Source: A. Path-Planning with Pixel Navigation · p. 3
- More than 100,000 simulated trajectories, each limited to 64 time steps, train the policy.Trajectories are generated in Habitat-Lab with HM3D by sampling an RGB pixel and using a path planner to interpolate actions.Source: A. Path-Planning with Pixel Navigation · p. 3
- A transformer combines a goal token from the initial image and pixel mask with RGB observation tokens, then predicts actions, tracked pixels, and remaining steps.The goal-fusion layer appends a 256-dimensional feature to each observation token before the transformer decoder integrates information across tokens.Source: A. Path-Planning with Pixel Navigation · pp. 3–4
- Training combines imitation learning for actions with MSE losses for pixel tracking and temporal distance, using ground-truth actions, distances, and pixels.The temporal-distance loss is divided by 10; all training processes are supervised learning.Source: A. Path-Planning with Pixel Navigation · p. 4
B. Perception with Foundation Model
The perception pipeline turns panoramic RGB observations into text for planning and assigns a pixel goal to a proposed object. It uses VLM room- and object-level queries alongside object segmentation models.
B. Perception with Foundation Model
- The VLM answers structured room-type and object-description queries for each of six images captured as the robot rotates to build a panoramic view.Each image is converted into a textual paragraph for the LLM planner.Source: B. Perception with Foundation Model · p. 4, Figure 3 · p. 4
- GroundingDINO and SAM produce object masks, and the center of the proposed target's mask becomes the pixel goal for low-level navigation.Source: B. Perception with Foundation Model · p. 4
C. Planning with Large-Language Model
For long-horizon navigation, the LLM planner uses structured descriptions and an estimated room layout to choose promising search directions. Its prompts support both room-level organization and exploration-exploitation planning.
C. Planning with Large-Language Model
- The LLM summarizes VLM captions as JSON containing ID, angle, type, and objects, then estimates the robot's room and position to group views into room instances and build a local-layout graph.Adjacent-angle rooms of the same type and connected rooms of the same type are merged.Source: C. Planning with Large-Language Model · p. 4
- Using the graph, the LLM plans room-to-room exploration while balancing exploration and exploitation, and proposes a new subgoal periodically or after the low-level policy completes its goal.Source: C. Planning with Large-Language Model · p. 4
IV. EXPERIMENTAL RESULTS
The evaluation uses HM3D validation scenes and samples pixel goals at two geodesic-distance ranges. It reports navigation success, path efficiency, and distance to goal; the supplied passages give evaluation setup but no results.
A. Benchmarks and Metrics
- The evaluation uses HM3D, with 80 scenes for training and 20 for validation, and samples 500 episodes at each goal-distance level.Object-navigation evaluation follows objectnav-challenge-2022 settings.Source: A. Benchmarks and Metrics · pp. 4–5
- The evaluation reports Success and SPL, with SPL described as a measure of navigation-path efficiency.Source: A. Benchmarks and Metrics · p. 4
- Figure 4 depicts a planning prompt pipeline that structures panoramic captions, estimates robot location, forms a room graph, and requests an exploration-exploitation plan.Source: Figure 4 · p. 5
- Pixel-goal evaluation includes a 3–5 m level, representing geodesic distance from assigned pixel goals to the initial robot location; DTG is geodesic distance from trajectory end to pixel goal.For each level, 500 episodes are sampled for evaluation.Source: A. Benchmarks and Metrics · p. 5
B. Baselines
The evaluation includes both non-zero-shot and zero-shot object-navigation baselines. These include closed-set imitation-learning and representation-learning methods, a map-based zero-shot method, and an RGB-based zero-shot method.
- Habitat-Web and OVRL are closed-set object-navigation baselines, using human demonstrations for imitation learning and a large-scale indoor dataset for representation learning, respectively.Source: B. Baselines · p. 5
- ESC is a map-based zero-shot object-navigation baseline that uses an LLM for frontier planning.Source: B. Baselines · p. 5
- ZSON is an RGB-based zero-shot object-navigation baseline using CLIP as its image and object-goal encoder.Source: B. Baselines · p. 5
C. Performance on Zero-Shot ObjectNav
PixNav achieves competitive object-navigation performance without depth or localization inputs, tracks the initial pixel target, and adapts its view to avoid collisions. Camera-setting tests and component ablations indicate robustness across some shifts, while tracking prediction and goal fusion matter more for longer navigation distances.
C. Performance on Zero-Shot ObjectNav
- PixNav achieves competitive performance without depth or localization inputs, consistently tracks the initial pixel target, and looks down to avoid collisions.Source: C. Performance on Zero-Shot ObjectNav · p. 5
- Long-horizon trajectories show heuristic exploration, including moving between rooms and returning to a corridor to search for a bathroom.Source: C. Performance on Zero-Shot ObjectNav · p. 5
D. Ablation Study on PixNav
- With camera height changed by 50% and HFOV by 25%, PixNav retains high success in 1–3 m tasks and degrades little in most cases.Source: D. Ablation Study on PixNav · p. 5
- Most tested components have minor effects at 1–3 m, but tracking prediction and goal fusion significantly influence performance at 3–5 m.Source: D. Ablation Study on PixNav · p. 5
E. Analysis of LLM Planner
The analysis examines whether the LLM planner can use indoor spatial awareness and illustrates PixNav and long-horizon navigation trajectories. It evaluates room localization and room clustering against human responses.
- Figure 5 pairs first-person PixNav views with a top-down trajectory; its caption describes persistent goal prediction as the view changes and semantic exploration by the LLM planner.Source: Figure 5 · p. 6
- The table captions identify camera-setting variations and PixNav component ablations, but the supplied passages do not report their results.Source: Table II · p. 6, Table III · p. 6
- The LLM spatial-awareness evaluation tests room localization and room clustering by comparing its answers with three humans across 100 episodes.Source: E. Analysis of LLM Planner · p. 6
F. Sim2Real Transfer
Real-world experiments test PixNav and its LLM planner on an iRobot Create3 using RGB observations. The conclusion notes that current long-term path-planning performance is not satisfactory.
F. Sim2Real Transfer
- Real-world tests use an iRobot-Create3 Mobile base with an OAK-D camera, but use only RGB observations to evaluate PixNav and the LLM planner.Source: F. Sim2Real Transfer · p. 6