PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenFrontier: General Navigation with Visual-Language Grounded Frontiers

Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys, Hermann Blum

arXiv:2603.05377v3cs.ROcs.CV

TL;DR

Open-world navigation must connect high-level semantic goals with low-level geometry, while conventional mapping-based and learned approaches have constraints including dense mapping, training demands, or grounding difficulty. OpenFrontier uses visual frontiers as sparse semantic anchors, scoring them with vision-language priors and navigating toward selected subgoals. The paper reports strong zero-shot benchmark performance and real-world robotic deployment, while noting that recovery from stalled progress remains limited.

  • Problem

    Open-world navigation requires reasoning over semantics and geometry under partial observability, while existing approaches can depend on dense mapping or substantial training and still struggle to ground semantic reasoning in metric navigation.

    Source: §I · p. 1

  • Method

    OpenFrontier uses visual frontiers as sparse, physically grounded anchors, evaluating candidates with vision-language models and combining semantic relevance with exploration information gain.

    Source: §I · pp. 1–2, §III · p. 3

  • Results

    The authors report strong zero-shot generalization across multiple navigation benchmarks and robust real-world deployment, with competitive performance against recent baselines.

    Source: §I · p. 2, §VI · p. 9

  • Takeaways & Limitations

    The framework grounds vision-language priors in actionable frontier subgoals while avoiding dense semantic mapping, task-specific policy training, and fine-tuning.

    Source: §VI · p. 9

    When progress stalls, the system has limited ability to recover and may continue in incorrect directions or become trapped in local minima.

    Source: B. When and Why It Fails · p. 8

Abstract

from arXiv · show

Open-world navigation requires robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in vision-language navigation (VLN) and vision-language-action (VLA) models enable end-to-end policies conditioned on natural language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent. We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation. Based on this insight, we select visual frontiers as semantic anchors and propose OpenFrontier, a navigation framework that requires no task-specific training or fine-tuning and seamlessly integrates diverse vision-language prior models. OpenFrontier enables efficient navigation with a lightweight system design, without dense 3D semantic mapping, task-specific policy training, or model fine-tuning. We evaluate OpenFrontier across multiple navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile robot.

I. INTRODUCTION

OpenFrontier grounds language-conditioned navigation in visual frontiers, aiming to connect semantic reasoning with physically actionable decisions. The introduction situates this approach among mapping-based and learning-based navigation methods and reports zero-shot benchmark performance and real-world deployment.

I. INTRODUCTION

  • Visual frontiers serve as sparse, interpretable, physically grounded anchors for connecting vision-language priors to actionable navigation goals.The framework evaluates candidate frontiers with vision-language models and combines semantic relevance with exploration-driven information gain.Source: §I · pp. 1–2
  • OpenFrontier scores frontier relevance using in-image context, then manages the resulting candidates to update navigation targets, plan paths, and determine termination.The robot iteratively clears frontiers according to grounded priors while balancing exploration and exploitation.Source: Figure 2 · p. 2, §I · p. 2
  • OpenFrontier demonstrates strong zero-shot performance across multiple navigation benchmarks and validates the approach in real-world robotic deployment.The framework is described as requiring no dense semantic mapping, task-specific policy training, or fine-tuning.Source: §I · p. 2

A. Map-based Exploration and Navigation

  • Map-based exploration commonly selects frontier goals using geometric heuristics such as distance or information gain, relying on mapping and geometric criteria.FrontierNet detects and evaluates frontiers directly from RGB images, while VLFM uses dense semantic frontier maps.Source: A. Map-based Exploration and Navigation · p. 2
  • Prior approaches include dense semantic maps, scene graphs, learned belief representations, or interactive learning, while object-goal benchmarks span closed-set and open-vocabulary tasks.The passage names HM3D ObjectNav, OVON, and GOAT-Bench as benchmarks.Source: A. Map-based Exploration and Navigation · pp. 2–3

B. Vision-Language Navigation and Action Models

Vision-language navigation and action systems often depend on large-scale training or fine-tuning and can be difficult to ground in metric navigation. OpenFrontier instead reasons over image-domain frontiers and uses them as navigation goals.

B. Vision-Language Navigation and Action Models

  • Many vision-language navigation and action approaches require large-scale datasets or task-specific fine-tuning, and may generalize poorly beyond training distributions.Frequent model queries can also be computationally expensive and difficult to ground in metric navigation space.Source: B. Vision-Language Navigation and… · p. 3

III. METHOD

  • OpenFrontier detects and evaluates navigation targets directly in the 2D image domain, using visual frontiers as candidates for goal-conditioned navigation.The system processes RGB observations with camera extrinsics and intrinsics, then grounds a selected frontier in 3D as a goal pose for a low-level planner.Source: §III · p. 3
  • A set-of-marks query lets a vision-language model score marked frontiers against the instruction in their local image context.The resulting relevance probabilities re-weight FrontierNet's exploration-driven information gain.Source: Figure 3 · p. 3

A. Visual-Frontier Identification

OpenFrontier identifies visual frontiers from RGB observations, combines their exploration value with language-conditioned relevance, and manages candidates as navigation goals. Its manager updates and selects frontiers, supports target verification, and hands goals to a low-level planner.

A. Visual-Frontier Identification

  • Visual frontiers are detected from a single 2D image and back-projected into 3D, avoiding dense mapping for frontier identification.Each cluster carries an exploration information-gain estimate; a vision-language model assigns its likelihood of leading to the language-specified goal.Source: A. Visual-Frontier Identification · pp. 3–4
  • The frontier utility combines exploration-driven information gain with semantic relevance to balance exploration and goal-directed exploitation.The relevance score comes from the set-of-marks vision-language query.Source: A. Visual-Frontier Identification · p. 4
  • Set-of-marks prompting asks the model to score all marked frontier clusters against the goal using surrounding image context.A joint forward pass uses actionable 2D locations and avoids requiring explicit 3D spatial reasoning from the model.Source: A. Visual-Frontier Identification · p. 4

B. Frontier Management

  • The global manager selects the highest-utility frontier for the low-level planner, favoring candidates that are relevant and spatially efficient to reach.The low-level planner can be map-free or map-based; Table I compares Habitat ObjectNav performance and methods' primary global representations.Source: B. Frontier Management · p. 4, Table I · p. 5
  • The management loop updates, merges, and prunes frontiers, tracks progress, and clears candidates when progress stalls or the robot nears them.When an object mask is detected, the system can add a viewpoint frontier, query the model to verify the target, and use its estimated centroid as the final target position if confirmed.Source: B. Frontier Management · pp. 4–5

C. Flexible Framework Design and Open-Context Query

OpenFrontier separates visual-frontier perception and reasoning from global goal management, with frontiers providing the interface between them. The vision-language model can be replaced without changing the rest of the system, and lightweight geometric cues are optional.

C. Flexible Framework Design and Open-Context Query

  • Navigation frontiers form a lightweight, interpretable interface between visual reasoning and global goal management, allowing the components to be modified independently.This separation is presented as providing flexibility in system design.Source: C. Flexible Framework Design and… · p. 5
  • The framework requires no navigation-specific VLM pre-training, fine-tuning, or in-context learning, and the model can be replaced directly.Dense 3D reconstruction and semantic mapping are not required; an optional lightweight volumetric map supplies basic filtering cues asynchronously.Source: C. Flexible Framework Design and… · p. 5

IV. EXPERIMENT AND RESULTS

OpenFrontier is evaluated on closed- and open-vocabulary object-goal benchmarks using a uniform system configuration, then assessed for model flexibility and contextual goal reasoning. It achieves competitive benchmark performance while supporting alternative VLMs and language goals with spatial context.

A. Experimental Setup

  • The evaluation covers HM3D and MP3D closed-vocabulary goals and OVON open-vocabulary language descriptions, using the same system parameters across benchmarks.It uses SR and SPL, Gemini-2.5-flash for object-goal experiments, and a pretrained DD-PPO point-goal policy for low-level navigation.Source: A. Experimental Setup · p. 5

B. ObjectNav Benchmark Evaluation

  • OpenFrontier outperforms most baselines across HM3D, MP3D, and OVON; it is within 1.5% SPL on HM3D and 0.5% SR on OVON of Uni-NaVid, and over 20% higher SR than UniGoal on HM3D.It is also 0.3% lower in SR than UniGoal on OVON; Uni-NaVid is fine-tuned on these benchmarks and action spaces, while UniGoal uses dense mapping and scene graphs.Source: B. ObjectNav Benchmark Evaluation · pp. 5–6
  • Qualitative examples show OpenFrontier navigating branching corridors toward targets while avoiding redundant exploration, with examples spanning all three datasets.The reported runs use the same system configuration across datasets.Source: Figure 4 · p. 5, B. ObjectNav Benchmark Evaluation · p. 6, Figure 5 · p. 6
  • At matched VLM backbones, OpenFrontier substantially outperforms UniGoal, History-Augmented VLM, and InstructNav; swapping its VLM causes at most 3% SR variation on HM3D.The reported comparison attributes the gains to the visual-frontier design rather than VLM strength alone.Source: B. ObjectNav Benchmark Evaluation · p. 6, Table II · p. 6
  • For context-rich goals such as “plant in bathroom,” OpenFrontier changes frontier relevance and prioritizes bathroom-related regions rather than regions likely to contain plants generally.The comparison uses the same starting location for “plant in the bathroom” and “plant.”Source: B. ObjectNav Benchmark Evaluation · p. 6

C. Real-World Deployments

OpenFrontier is deployed on a legged robot, and the paper explains its design through shared image-space reasoning, complementary frontier utilities, and a sparse interface linking semantic decisions to navigation. A history-augmented query illustrates how recent observations can refine frontier relevance.

C. Real-World Deployments

  • With similar frontier locations and the same start, the query “plant in the bathroom” prioritizes bathroom-related regions, while “plant” directs the robot toward the living room.The final observation triggers termination when the target enters the field of view.Source: Figure 6 · p. 7
  • In a small real-world evaluation, OpenFrontier achieved 70% success across 10 navigation trials involving five target objects, and autonomously navigated to a fire extinguisher.The Spot robot operated in a large indoor environment without prior knowledge of its layout.Source: C. Real-World Deployments · p. 7, Figure 8 · p. 8

A. Why the Design Is Effective

  • The paper presents OpenFrontier as a lightweight pipeline whose effectiveness depends on the visual-frontier abstraction.Its design is described as an intermediate point between approaches using rich 3D maps and approaches requiring navigation-model training.Source: A. Why the Design Is Effective · p. 7, §V · p. 7
  • Frontiers and their relevance are evaluated in the same RGB image, supporting direct VLM reasoning without fine-tuning and allowing recent keyframes to be added as context.The history-augmented variant appends recent keyframes while leaving frontier detection and global management unchanged; one example lowers Frontier A’s probability from 0.65 to 0.10.Source: A. Why the Design Is Effective · pp. 7–8, Figure 9 · p. 8
  • Frontier utility combines geometric exploration value from FrontierNet with the VLM’s goal-conditioned semantic relevance, balancing unexplored-volume coverage with goal-directed bias.The paper distinguishes FrontierNet’s strength in estimating unobserved 3D volume from its lack of task semantics.Source: A. Why the Design Is Effective · p. 8
  • Sparse frontiers are grounded from 2D detection into 3D navigation targets, providing a lightweight global memory and separating high-level semantic decisions from low-level execution.The paper identifies tighter feedback between these levels as a possible future direction while preserving modularity.Source: A. Why the Design Is Effective · p. 8

B. When and Why It Fails

OpenFrontier’s failures include false-positive target detections and step-budget exhaustion, alongside difficulty recovering when progress stalls. Its conclusion emphasizes competitive benchmark performance and real-world transfer, while identifying effective failure detection and recovery as future work.

B. When and Why It Fails

  • OpenFrontier lacks a mechanism to recognize stalled progress and trigger semantic-level rereasoning or replanning; point-goal execution can move incorrectly or become trapped after target detection.Termination criteria are also challenging in open-world settings, and some failures are influenced by the Habitat benchmark.Source: B. When and Why It Fails · p. 8

VI. CONCLUSION

  • False-positive target detections and step-budget exhaustion are the two most common failure modes across HM3D, MP3D, and OVON.The conclusion reports competitive performance across multiple object-goal benchmarks and transfer to real-world deployment on a mobile legged robot.Source: Figure 10 · p. 9, §VI · p. 9

APPENDIX

The appendix specifies the simulation setup and operational details for frontier scoring, target verification, and recovery when no frontiers are available.

A. Configuration Settings and Key Parameters

  • Simulation uses Habitat 0.2.4 with default ObjectNav actions, a 640×480 RGB-D camera, 0.25 m steps, 30° turns, and a 500-step episode cap.FrontierNet, SAM3, and applicable InternVL runs use an RTX 4090; Gemini and Gemma are accessed by API.Source: A. Configuration Settings and Key… · p. 12

B. Emergency rotation mechanism

  • When no valid frontier is visible, three artificial rotation frontiers trigger in-place turns; if exploration remains stalled, the filtering threshold is repeatedly halved up to a limit.After the maximum emergency iterations, the system triggers the no_frontiers failure condition.Source: B. Emergency rotation mechanism · p. 12

1) Frontier Probability Estimation (Set-of-Marks):

  • The Set-of-Marks prompt asks the model to estimate each labeled frontier’s probability of reaching or being near the target, using local context and long-term exploration potential.It assigns unseen labels a probability of 0.5 and requests a short explanation with each score.Source: 1) Frontier Probability Estimation… · p. 12

2) Target Presence Verification:

  • Target presence verification estimates whether the object is visible, within five meters, and reachable, treating reflected, obscured, barely visible, or unreachable objects as absent.The prompt asks for probabilities near 0 or 1 and one sentence of reasoning.Source: 2) Target Presence Verification · p. 12

3) Optional add-on for history enhancement:

The appendix describes optional history and object-viewpoint handling, then details the real-world robot, planning pipeline, and deployment adjustments.

3) Optional add-on for history enhancement:

  • The history add-on lowers scores for frontiers leading to already visited or visually similar places; InsertViewpointIfAny segments target objects, projects them into world coordinates, clusters centroids, and creates object-linked frontiers.The algorithm uses RGB and depth images together with the robot pose.Source: 3) Optional add-on for history… · pp. 12–13
  • A stall counter increments when distance progress falls below a threshold and marks the robot to drop when the counter reaches its threshold.The routine returns the drop decision, updated stall counter, and previous distance.Source: 3) Optional add-on for history… · p. 13

D. Real-World Experiment

  • Real-world deployment uses a Spot robot, arm-mounted RGB camera, estimated depth, occupancy-grid mapping, and RRT-star planning with pure-pursuit execution.Frontier prediction, segmentation, and mapping run asynchronously around 3 Hz, while frontier management and goal selection run around 16 Hz.Source: D. Real-World Experiment · p. 13
  • Real-world frontier filtering is strengthened to handle clutter, glass surfaces, and high-elevation detections.Deployment-specific parameters are summarized in Table S2.Source: D. Real-World Experiment · p. 13, Table S2 · p. 13

E. Metric Calculation

The appendix defines the reported navigation metrics and presents supplementary trajectories and contextual-goal examples across benchmarks.

E. Metric Calculation

  • Success requires stopping within 1.0 m of a target instance with oracle visibility; SPL is averaged over episodes, and infeasible episodes are excluded from the listed benchmark evaluations.The evaluated splits are HM3D (V2), MP3D, and OVON Val Unseen; infinite shortest-path episodes are excluded.Source: E. Metric Calculation · pp. 13–14

F. Additional Exploration Examples on HM3D, MP3D, and OVON

  • Supplementary figures show representative HM3D, MP3D, and OVON trajectories from the start location to the success region, with final-step frontier distributions.The trajectories complement the quantitative evaluations and illustrate both closed-set and more open-set navigation scenarios.Source: F. Additional Exploration Examples on… · p. 14
  • In contextualized examples, the robot successfully navigates to the same object in the same scene under different goal descriptions, including chair contexts shown in Fig. S1.The examples contrast a chair in a bedroom with a chair near a monitor.Source: F. Additional Exploration Examples on… · p. 14, Figure S1 · p. 14

G. Additional results of ablations with Different Vision– Language Models

The appendix compares vision-language models on HM3D and gives examples of real-world navigation targets.

G. Additional results of ablations with Different Vision– Language Models

  • The full pipeline is evaluated with different VLMs on the entire HM3D evaluation set, extending qualitative comparisons from the main paper.The quantitative results are summarized in Table II.Source: G. Additional results of ablations… · p. 14
  • Gemini-2.5-flash performs best; Gemma-3 and InternVL3 incur less than 3% success-rate drops and approximately 2% SPL drops.The compared public models have 4B parameters for Gemma-3 and 8B for InternVL3.Source: G. Additional results of ablations… · p. 14

H. Real-World Navigation Results

  • Real-world navigation targets include objects present in the environment, such as fire extinguishers, microwaves, scissor lifts, and televisions.The evaluation uses a large-scale indoor environment with diverse layouts and room types and random robot starting locations.Source: H. Real-World Navigation Results · p. 14

I. Failure Cases

Benchmark failures fall into three categories, with distinct causes despite the shared outcome of an unsuccessful episode. The analysis traces failures to benchmark annotations, the low-level navigator, and OpenFrontier’s frontier handling.

I. Failure Cases

  • Failures comprise missed targets, local-minimum stalls after target detection, and semantically valid objects missing from benchmark annotations.All terminate unsuccessfully after the step limit, but for different reasons.Source: §I · p. 14
  • Failure sources include Habitat annotation limits, DD-PPO navigation problems, and OpenFrontier missing informative frontiers near targets.The method may also detect an incorrect object as the target.Source: §I · p. 14
  • More reliable frontier detection and stronger low-level navigation policies could improve robustness, while benchmark ambiguities can affect evaluation independently of the policy.Source: §I · p. 15
Loading 2603.05377v3…