Source-linked AI summary
UniGoal: Towards Universal Zero-shot Goal-oriented Navigation
Hang Yin, Xiuwei Xu, Lingqing Zhao, Ziwei Wang, Jie Zhou, Jiwen Lu
TL;DR
Existing zero-shot navigation methods use task-specific pipelines, while universal alternatives require training and lack zero-shot generalization across goal types. UniGoal unifies object category, instance image, and text description through graph-based scene-goal representations and matching-guided exploration, achieving state-of-the-art zero-shot performance across three tasks with one model.
Problem
Existing zero-shot methods use task-specific inference pipelines, whereas universal methods require training and lack zero-shot generalization across different goal types.
Method
UniGoal converts observations and diverse goals into consistent graphs, then uses graph matching and multi-stage LLM-guided exploration with a blacklist mechanism.
Results
UniGoal achieves state-of-the-art zero-shot performance on three navigation tasks with a single model, outperforming task-specific zero-shot and supervised universal methods.
Takeaways & Limitations
A single graph-based framework handles object-goal, instance-image-goal, and text-goal navigation without training or finetuning.
Takeaways & Limitations
The framework is scoped to three navigation sub-tasks and assumes goals can be specified by free-form language or image.
Abstract
from arXiv · showhide
In this paper, we propose a general framework for universal zero-shot goal-oriented navigation. Existing zero-shot methods build inference framework upon large language models (LLM) for specific tasks, which differs a lot in overall pipeline and fails to generalize across different types of goal. Towards the aim of universal zero-shot navigation, we propose a uniform graph representation to unify different goals, including object category, instance image and text description. We also convert the observation of agent into an online maintained scene graph. With this consistent scene and goal representation, we preserve most structural information compared with pure text and are able to leverage LLM for explicit graph-based reasoning. Specifically, we conduct graph matching between the scene graph and goal graph at each time instant and propose different strategies to generate long-term goal of exploration according to different matching states. The agent first iteratively searches subgraph of goal when zero-matched. With partial matching, the agent then utilizes coordinate projection and anchor pair alignment to infer the goal location. Finally scene graph correction and goal verification are applied for perfect matching. We also present a blacklist mechanism to enable robust switch between stages. Extensive experiments on several benchmarks show that our UniGoal achieves state-of-the-art zero-shot performance on three studied navigation tasks with a single model, even outperforming task-specific zero-shot methods and supervised universal methods.
1. Introduction
Goal-oriented navigation must support diverse goal types in unknown environments, yet existing approaches are either task-specific or training-dependent. UniGoal addresses this gap with a unified graph-based, zero-shot framework for three navigation tasks.
- Goal-oriented navigation requires agents to reach specified goals in unknown environments.
- The three representative goal types are object category, instance image, and text description, corresponding to ON, IIN, and TN.
- Universal navigation is needed because flexible human instructions require one model to handle all sub-tasks.
- Training-based universal methods use shared goal representations but require costly reinforcement-learning training and may generalize weakly beyond simulation.
- UniGoal represents scenes and goals as graphs, enabling consistent graph matching and explicit LLM-based reasoning across the three goal types.
- UniGoal uses matching-guided exploration strategies that progress from subgraph search through goal-location inference as scene-goal overlap increases.
2. Related Work
Zero-shot navigation research is organized around goal-specific tasks and increasingly uses vision-language models or LLMs for exploration and goal-location reasoning.
- Zero-shot navigation is divided mainly into object-goal, instance-image-goal, and text-goal navigation according to goal type.
- CoW uses open-vocabulary CLIP embeddings with frontier-based exploration as a zero-shot object-goal navigation baseline.
- ESC, OpenFMNav, and VLFM use LLM-derived common-sense object correlations to reason about goal locations.
- Mod-IIN provides a zero-shot approach for instance-image-goal navigation.
3. Approach
UniGoal presents a single training-free framework for navigating to object categories, instance images, and text descriptions using consistent scene and goal graphs. Graph matching determines whether to expand exploration, infer the goal location from partial overlap, or verify a fully matched goal.
- Task Specification: UniGoal defines universal zero-shot navigation as one unmodified, training-free method for object-category, instance-image, and text-description goals.The agent receives RGB-D observations and acts in an unknown environment until stopping within the task’s success radius and step budget.
- Graph Construction and Matching: The framework incrementally maintains a scene graph and converts each goal type into a uniform goal graph with object nodes and spatial or semantic relation edges.This consistent representation supports graph-based reasoning while retaining structural information across scene and goal descriptions.
- Multi-stage Scene Exploration: As matching becomes perfect, the framework transitions toward goal verification after the central object is matched, while a blacklist prevents unsuccessful matches from causing repeated exploration.The blacklist can be extended after failed stage transitions or verification and cleared locally when scene-graph correction refines affected elements.
- Graph Construction and Matching: Graph matching combines node, edge, and topology similarities to score how well the goal graph is observed in the scene graph.Node and edge embeddings are paired through bipartite matching, while topology is measured from graph editing similarity; the final score averages the three components.
- Multi-stage Scene Exploration: With zero matching, UniGoal expands the explored region and searches for goal elements, including separate goal subgraphs when the goal contains weakly related parts.This stage uses frontier exploration guided by semantic relationships between the scene graph and goal.
- Multi-stage Scene Exploration: With partial matching and an anchor pair, the method aligns scene and goal bird’s-eye-view graphs to project the unobserved goal nodes and infer exploration locations.Relative spatial relations in the goal graph are used to estimate coordinates, while observed world coordinates provide the scene reference.
4. Experiments
UniGoal is evaluated across three navigation tasks and compared with supervised, zero-shot, and universal methods. It achieves strong benchmark performance, while ablations show that matching judgments, blacklist handling, multi-stage exploration, and stage-specific modules materially support navigation.
- Experimental Settings: UniGoal is evaluated on object-goal, instance-image-goal, and text-goal navigation using success rate and path-length-weighted success rate.Experiments use MP3D, HM3D, and RoboTHOR for object-goal navigation, and HM3D for instance-image-goal and text-goal navigation.
- Comparison with State-of-the-art: UniGoal surpasses zero-shot SG-Nav and Mod-IIN by 0.8% on ON and 4.1% on IIN, while also exceeding supervised universal baselines across all three tasks.Against PSL and GOAT, the reported leads are +3.9/1.0 on ON, +22.8/7.6 on IIN, and +3.2/2.6 on TN.
- Comparison with State-of-the-art: Even when the ON goal graph degenerates to one node and disables stage 1 decomposition and stage 2 anchor alignment, UniGoal outperforms SG-Nav on all benchmarks.The authors attribute this result to graph correction and goal verification in stage 3.
- Ablation Study: Removing matching-score judgment, the blacklist mechanism, or stage 2 causes more failures or significant performance degradation on HM3D IIN.Without matching-degree judgment, stage switching occurs at suboptimal times; without blacklisting, repeated matching can trap the agent; removing stage 2 simplifies exploration but lowers performance.
- Ablation Study: Ablations indicate that stage 1 components, structure-based coordinate inference in stage 2, and scene-graph correction and goal verification in stage 3 each improve performance.The stage 1 tests cover goal-graph decomposition and frontier selection, while stage 2 tests compare coordinate projection and anchor-pair alignment against simpler alternatives.
- Qualitative Results: Qualitative results show UniGoal progressively increases matching scores and produces trajectories for ON, IIN, and TN across diverse HM3D scenes with one model.Figure 5 visualizes the three path types using green, orange, and blue trajectories.
5. Conclusion
UniGoal is a universal zero-shot navigation framework that handles object-goal, instance-image-goal, and text-goal navigation in one model without training or finetuning. Experiments on three datasets and a real-world robotic deployment validate its effectiveness and generalization.
- UniGoal handles object-goal, instance-image-goal, and text-goal navigation in a single model without training or finetuning.
- The framework converts observations and goals into consistent graphs and uses graph matching to guide multi-stage scene exploration.Different overlap states determine strategies for locating the goal, while a blacklist avoids repeated exploration.
- Experiments on three widely used datasets validate UniGoal’s effectiveness.
- A real-world robotic deployment demonstrates UniGoal’s generalization ability and application value.
Supplementary Material
Table 4 illustrates the goals used in the three navigation tasks and marks their central objects in red.
- Table 4 illustrates the goal format for each navigation task.
- The table uses red coloring to identify central objects.
- Central-object marking distinguishes the primary object within each illustrated goal.
A. Overview
The supplementary material covers task details, UniGoal’s pipeline and approach, and the prompts used for the LLM.
- Section B provides details of the three studied navigation tasks.
- Section C presents UniGoal’s overall pipeline in algorithm form.
- Section D details the approach, while Section E documents the LLM prompts.
B. Definition of Each Task
The three studied tasks differ in how their goals are represented: category text for ON, an object-centered image for IIN, and an object description for TN.
- Object-goal Navigation uses an object category in text format as its goal.
- Instance-image-goal Navigation uses an image with the central object located at its center.
- Text-goal Navigation uses a description of an object and its relationships with relevant scene objects.
C. Pipeline of UniGoal
Algorithm 1 provides UniGoal’s algorithm diagram, where each gray box denotes one stage of the pipeline.
- Algorithm 1 presents the algorithm diagram of UniGoal.
- The diagram represents the pipeline using stages.
- Each gray box in the diagram corresponds to one stage.
D. Details of Approach
UniGoal constructs uniform goal graphs for object categories, instance images, and text descriptions, incrementally updates a scene graph, and uses graph matching to guide staged exploration.
- D.1. Goal Graph Construction: Object-goal graphs contain one category node without edges, while instance-image and text goals are converted into relational graphs.Instance-image goals use object identification and relationship prediction; text goals use LLM-based object and relationship extraction.
- D.4. Matching-Guided Navigation: The pipeline applies coordinate-based navigation in intermediate matching states, then corrects the scene graph and verifies the goal before stopping.The algorithm includes navigation to a predicted position and a separate scene-graph-correction and goal-verification step.
- D.2. Graph Embedding: Node embeddings concatenate CLIP text features with node degree, whereas edge embeddings use CLIP features alone.These embeddings support graph matching between the scene and goal representations.
- D.3. Zero-Matching Stage: During zero matching, LLM-guided decomposition divides the goal graph into correlated subgraphs and frontier scoring selects an exploration goal.Frontiers are scored using their distance to the predicted subgraph position and to the agent.
D.4. Hyperparameters
UniGoal uses task-specific navigation limits and matching thresholds, alongside prompts for goal construction, decomposition, frontier scoring, graph correction, and relationship inference.
- D.4. Hyperparameters: The maximum navigation step number is T = 500 for ON and T = 1000 for IIN and TN.
- D.4. Hyperparameters: The success distance is r = 1.6m for ON and r = 1.0m for IIN and TN.
- D.4. Hyperparameters: The node-and-edge matching similarity threshold is τ = 0.9, with matching score thresholds σ1 = 0.5 and σ2 = 0.9.
- Prompt Design: Prompts extract objects and relationships from text, decompose goals into correlated subsets, score frontier distances, and revise scene-graph descriptions.Relationship prompts infer spatial relations from images, while correction prompts compare graph descriptions with newly observed images.
- Prompt Design: Frontier scoring prompts the model to predict the distance between an object and a goal subgraph in meters.