PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseNav: Instruction-conditioned Sparse Semantic Perception for Training-Free Vision-Language Navigation

Quanhua Chen, Juhan Kang, Runfeng Lin, ZiFei Zhang, Enquang Feng, Chunran Zheng, Xiwang Dong, Jiarong Lin

arXiv:2609.26408v1cs.RO

TL;DR

Map-based VLN must connect language understanding with geometric planning, while the value of broader semantic coverage remains open. SparseNav maintains persistent geometry and selectively grounds instruction-relevant landmarks. It reports success rates of 42.8% on R2R-CE and 40.7% on RxR-CE, alongside ablations and real-robot validation.

  • Problem

    Map-based VLN needs persistent spatial representations to connect language understanding with geometric planning, but whether broader semantic coverage benefits instruction following remains open.

    Source: §I · p. 1

  • Method

    SparseNav retains persistent geometric memory and uses the active sub-instruction to trigger metric grounding of visible, relevant landmarks for waypoint selection.

    Source: §I · p. 2, 4) 2D-to-3D Landmark Projection · p. 4

  • Results

    SparseNav achieves success rates of 42.8% on R2R-CE and 40.7% on RxR-CE Val-Unseen, with ablations favoring instruction-related on-demand grounding over evaluated dense-semantic settings.

    Source: §V · p. 8

  • Takeaways & Limitations

    SparseNav’s successful Unitree Go2 deployment across multiple indoor environments validates its framework using instruction-conditioned waypoint navigation.

    Source: 2) Navigation Task and Execution · p. 7, §V · p. 8

    VLM inference remains a major computational bottleneck that can delay robot motion and increase task completion time in the sequential pipeline.

    Source: §VI · p. 8

Abstract

from arXiv · show

Map-based vision-language navigation (VLN) relies on persistent spatial representations to connect language understanding with geometric planning. However, acquiring semantics beyond the needs of the current instruction can introduce unnecessary perception cost and irrelevant annotations. Continuously accumulating unrelated objects may not only waste computation, but also clutter the visual-spatial representation consumed by the vision-language model (VLM) planner. To address this problem, we present SparseNav, a training-free framework that follows a less-is-more principle for semantic navigation. SparseNav persistently maintains a lightweight geometric bird's-eye-view (BEV) map and sparse landmark memory, acquiring new semantics on demand using the active sub-instruction to decide what is worth grounding. An instruction manager first tracks navigation progress and identifies the active landmark query. An instruction-conditioned perception mechanism then invokes open-vocabulary segmentation when the queried landmark is visible and its metric location can inform the next decision. The resulting landmark memory supports VLM selection among hybrid frontier and local directional waypoint candidates. Without any additional training, SparseNav achieves success rates of 42.8% on R2R-CE and 40.7% on RxR-CE, both on the Val-Unseen splits. Controlled ablations examine semantic perception strategies and the contributions of individual framework components. Furthermore, we successfully deployed SparseNav on a Unitree Go2 quadruped equipped with an Intel RealSense D455 RGB-D camera for geometric mapping and landmark grounding and a Livox MID-360 LiDAR for localization, without a prebuilt map. We validated its effectiveness across multiple indoor environments using instruction-conditioned waypoint navigation.

I. INTRODUCTION

SparseNav separates persistent geometric memory from selective semantic acquisition, using active instructions to decide which landmarks to ground. It frames this approach as a training-free response to the demands of continuous VLN and the open question of how much semantic coverage is useful.

I. INTRODUCTION

  • Continuous VLN requires visual and language grounding, spatial reasoning, exploration, progress tracking, and low-level motion execution in a consistent connection between semantics and geometric planning.Training-free VLN avoids navigation-specific demonstrations or task-specific model training.Source: §I · p. 1
  • Persistent maps connect semantic reasoning with physical motion, but whether broader semantic coverage improves instruction following remains an open question.The paper motivates selective semantics because unrelated annotations can consume computation and clutter the VLM planner's representation.Source: §I · p. 1
  • SparseNav grounds landmarks named by the active sub-instruction only when visible and when metric grounding can affect waypoint selection.It retains geometry while limiting semantic acquisition to instruction-relevant landmarks.Source: §I · pp. 1–2
  • SparseNav combines persistent geometry, sparse landmark memory, recent observations, and instruction progress in a BEV interface for VLM waypoint selection and classical path planning.Its hybrid frontier and local candidates support that selection.Source: §I · p. 2

A. Vision-Language Navigation in Continuous Environments

  • SparseNav evaluates training-free navigation on R2R-CE and RxR-CE, with controlled ablations and legged-robot validation across multiple environments.The introduction frames this as its benchmark and deployment evaluation.Source: §I · p. 2, A. Vision-Language Navigation in… · p. 2

B. Map-Based Memory for VLN

Map-based VLN uses persistent spatial representations, while SparseNav distinguishes persistent geometry from selectively acquired semantics. It makes landmark grounding an instruction-triggered event rather than continuously annotating the scene.

B. Map-Based Memory for VLN

  • Persistent spatial representations support long-horizon navigation because raw observation histories grow and encode spatial relationships only implicitly.Map-based systems described here use occupancy maps, semantic annotations, or other structured top-down representations.Source: B. Map-Based Memory for VLN · p. 2
  • SparseNav retains persistent geometry while the active sub-instruction determines landmark queries and the VLM selectively triggers metric grounding.Previously grounded landmarks remain available in sparse memory, making semantic acquisition an explicit decision.Source: B. Map-Based Memory for VLN · p. 2

C. Open-Vocabulary Visual Grounding

  • SparseNav treats segmentation as an event-driven tool, triggered when the VLM identifies a potentially present, task-relevant landmark.Its open-vocabulary segmentation model is used as a callable perception tool.Source: C. Open-Vocabulary Visual Grounding · p. 2
  • For “turn left in between the couches and the kitchen counter,” SparseNav grounds the relevant landmark pair while retaining geometry, unlike dense annotation of unrelated objects.Sparse memory stores landmark labels and cluster centers.Source: Figure 1 · p. 3

III. METHOD

SparseNav represents navigation with persistent geometry, sparse landmark memory, recent visual history, and instruction progress. RGB-D or range observations and odometry update geometric memory, while a VLM selects waypoints from hybrid candidates for classical planning.

A. Problem Formulation

  • The agent selects a geometrically valid waypoint from candidate set C_t using instruction I and observations O_t, then executes it with a classical planner.The state also includes persistent geometry, sparse landmark memory, recent visual history, and instruction progress.Source: A. Problem Formulation · p. 3
  • A VLM visibility check gates open-vocabulary segmentation, and mask-depth projection updates landmark labels and cluster centers in sparse memory.RGB-D observations and odometry update persistent geometry; the VLM selects among hybrid candidates using geometry, recent observations, and instruction progress.Source: A. Problem Formulation · p. 3

B. Geometry-Centric Spatial Memory

  • RGB-D or range points within specified height bounds are rasterized into an obstacle map, then dilation and closing bridge nearby observations and fill small holes.Obstacle inflation accounts for the robot footprint using a radius of 0.1 m; free-space accuracy is limited by observed geometry.Source: B. Geometry-Centric Spatial Memory · p. 3
  • For VLM input, the map is translated and rotated to center the robot and align its heading upward.The local +x forward and +y left axes map to image up and image left, aligning image directions with linguistic forward, left, and right.Source: B. Geometry-Centric Spatial Memory · p. 3

C. Hybrid Exploration-Exploitation Waypoint Proposal

SparseNav combines frontier and local directional waypoint candidates because frontiers support exploration but may miss useful waypoints in already explored space. The VLM selects among candidates informed by grounded landmarks.

C. Hybrid Exploration-Exploitation Waypoint Proposal

  • Frontiers can miss useful waypoints in explored space, such as continuing forward before turning at a sofa.SparseNav therefore combines frontier candidates with local directional candidates, with grounded landmarks guiding selection.Source: C. Hybrid Exploration-Exploitation… · p. 3

1) Frontier Candidates:

  • Frontiers are navigable cells adjacent to unknown space; connected cells are clustered, small clusters rejected, and a reachable point with obstacle clearance represents each remaining cluster.The VLM visibility check gates segmentation, and mask-depth projection updates landmark labels and cluster centers.Source: 1) Frontier Candidates · pp. 3–4
  • The navigation state combines persistent geometry, recent observations, instruction progress, and frontier/local candidates before VLM selection and path planning.Execution feedback supports sub-instruction switching or stopping.Source: Figure 2 · p. 4

2) Local Directional Candidates:

SparseNav uses frontier and local directional candidates for exploration and within-map motion. Its instruction manager tracks progress and activates only landmark queries relevant to the current or imminent objective.

2) Local Directional Candidates:

  • Frontier candidates support discovery, while local left/front/right proposals support instruction-following maneuvers within explored space.Proposals in obstacles or disconnected regions are discarded or projected to free space.Source: 2) Local Directional Candidates · p. 4

D. Hierarchical Instruction Management

  • The instruction manager advances through ordered sub-instructions when completion conditions are met, retaining completed, current, and upcoming objectives for the VLM.Progress tracking also avoids repeating completed objectives and determines the active landmark query.Source: D. Hierarchical Instruction Management · p. 4

1) Task-Relevant Landmark Query:

  • Landmark queries come from the active sub-instruction and may include multiple labels; an upcoming landmark can activate at a transition when it constrains the next waypoint, without advancing progress.Grounding is triggered when the queried landmark is likely visible and its metric position can affect the next decision; a cooldown suppresses redundant calls unless localization is unreliable.Source: 1) Task-Relevant Landmark Query · p. 4, 2) Semantic Trigger · p. 4

2) Semantic Trigger:

When a queried landmark is selected for grounding, SparseNav uses its description to prompt open-vocabulary segmentation. It projects valid mask pixels into the map frame and stores clustered landmark instances in sparse semantic memory.

3) Open-Vocabulary Landmark Segmentation:

  • The active landmark description prompts open-vocabulary segmentation, instantiated with SAM 3 for text-conditioned concept segmentation.Each output is an independent instance mask with a label.Source: 3) Open-Vocabulary Landmark Segmentation · p. 4

4) 2D-to-3D Landmark Projection:

  • Valid-depth pixels from each mask are back-projected using camera intrinsics and transformed into the map frame; projected observations are clustered after outlier rejection.Sparse semantic memory stores landmark labels and cluster centers, keeping same-label instances separate and retaining previously grounded landmarks when they are not queried.Source: 4) 2D-to-3D Landmark Projection · p. 4

5) Landmark-Conditioned Candidate Reasoning:

Landmark centers provide semantic references rather than destinations: the VLM uses them with the active spatial relation to select a collision-checked candidate. A classical planner then executes the selected waypoint using persistent map information and recent observations.

5) Landmark-Conditioned Candidate Reasoning:

  • The VLM combines landmark positions with the active spatial relation to choose among collision-checked frontier and local candidates; a landmark center is not necessarily a traversable target.The path planner executes the selected candidate on the geometric map.Source: 5) Landmark-Conditioned Candidate… · pp. 4–5

F. Visual-Spatial Memory and Waypoint Execution

  • The geometric layer stores obstacles, free and unknown space, trajectory, and visited candidates, while the sparse layer retains landmark labels and centers; recent observations preserve appearance context.The VLM selects a valid candidate and provides a concise rationale, then a classical planner computes a collision-free trajectory on the inflated map, separating semantic selection from metric execution.Source: F. Visual-Spatial Memory and Waypoint… · p. 5

A. Benchmarks

On Val-Unseen benchmark splits, SparseNav reports success rates of 42.8% on R2R-CE and 40.7% on RxR-CE. The section also describes benchmark protocols, a qualitative episode, and controlled ablations of perception strategies and framework components.

A. Benchmarks

  • Evaluation uses the Val-Unseen splits of R2R-CE and RxR-CE, with SR, SPL, and NE reported on both benchmarks, OSR on R2R-CE, and nDTW on RxR-CE.RxR-CE has longer trajectories and richer language descriptions; GPT-5 is used as the VLM in all experiments.Source: A. Benchmarks · p. 5

B. Results on Public Benchmarks

  • 42.8% Success Rate on R2R-CE Val-Unseen and 40.7% SR on RxR-CE; reported SR exceeds NaVid and MapNav on both, while ETPNav is higher on both and Dynam3D on R2R-CE.These are cross-paper comparisons, not controlled reruns; evaluated episodes, model backbones, and training data differ.Source: B. Results on Public Benchmarks · p. 5

C. Qualitative Analysis

  • A representative episode shows instruction-following to the specified destination, with new grounding events when relevant landmarks provide spatial references and reuse of prior landmarks between events.The episode illustrates navigation without semantic acquisition at every sub-instruction.Source: C. Qualitative Analysis · p. 5

D. Ablation Studies

1) Semantic Perception Strategy:

  • Instruction-related queries achieve 40.2% SR with continuous perception and 42.8% with on-demand perception, versus 36.7% and 38.8% for dense semantics, respectively.The comparisons separate semantic coverage from perception frequency; the SparseNav configuration uses instruction-related queries with on-demand grounding.Source: 1) Semantic Perception Strategy · p. 5

2) Hybrid waypoint proposal:

Ablations examine waypoint candidates, landmark localization, instruction and memory choices, and semantic-trigger quality. The results support combining geometric options with instruction-conditioned landmark grounding and explicit progress tracking.

2) Hybrid waypoint proposal:

  • 42.8% SR with frontier, forward, left, and right candidates outperforms 37.4% with forward added, 35.2% with frontiers alone, and 29.1% with local candidates alone.The results support combining frontier exploration with local motion options in explored free space.Source: 2) Hybrid waypoint proposal · p. 5, Table III · p. 6

3) Landmark Grounding:

  • 42.8% SR and 0.12 m localization error with segmentation localization compare with 40.2% SR and 0.51 m error using annotation-box localization.Localization error is the Euclidean distance between the predicted 3-D cluster center and the true object center.Source: 3) Landmark Grounding · p. 5

4) Instruction and Memory:

  • On RxR-CE, persistent landmark memory raises SR from 35.0% to 40.7%; adding temporal memory, sub-instructions, and progress tracking also increases SR across the reported stages.With the full instruction, temporal memory raises SR from 29.8% to 31.5%; sub-instructions raise it to 33.2%, and progress tracking to 35.0%.Source: 4) Instruction and Memory · p. 5

5) Semantic Trigger Quality:

  • 90.4% precision and 99.8% recall are achieved by sub-instruction-aware triggering on R2R-CE; a trigger is correct when the queried object is visible and relevant to the instruction.The semantic query is derived from the active sub-instruction and changes as it changes.Source: 5) Semantic Trigger Quality · p. 6

6) BEV Coordinate Representation:

The ablations favor an agent-centered heading-up BEV, while the robot deployment and representative navigation sequence demonstrate the framework in multiple indoor environments. The section also summarizes how geometry, selective grounding, waypoint proposals, and instruction memory contribute to navigation.

6) BEV Coordinate Representation:

  • 42.8% SR with an agent-centered heading-up BEV compares with 36.2% for agent-centered non-rotating and 30.3% for global north-up representations.The comparison changes map center and orientation together, so it does not isolate orientation alone.Source: 6) BEV Coordinate Representation · p. 6, Table VII · p. 7
  • The ablations support persistent geometry and instruction-related on-demand grounding, with hybrid waypoint proposals, precise localization, and instruction memory also contributing to navigation performance.Instruction-related, on-demand grounding achieves the strongest SR among the evaluated semantic strategies.Source: 6) BEV Coordinate Representation · p. 7

1) Platform and environment:

  • SparseNav was deployed without a prebuilt map on a Unitree Go2 with a RealSense D455 for geometric mapping and landmark grounding and a Livox MID-360 for localization.The deployment was validated across multiple indoor environments.Source: 1) Platform and environment · p. 7

2) Navigation Task and Execution:

  • The navigation task uses landmark-conditioned routes: SparseNav tracks sub-instruction progress, selectively grounds relevant landmarks, retains their positions, and plans from VLM-selected hybrid candidates.A path planner generates trajectories for execution.Source: 2) Navigation Task and Execution · p. 7

3) Real-World Deployment:

SparseNav reports benchmark results, real-robot validation, and evidence that task-relevant sparse grounding can support navigation without a semantically complete scene map. Its waypoint flexibility, inference cost, and benchmark gap remain limitations, motivating further work on planning, efficiency, and broader evaluation.

3) Real-World Deployment:

  • SparseNav was validated across multiple indoor environments on a Unitree Go2, where grounded landmark positions and instruction progress guide waypoint selection while the geometric map supports path planning.Source: 3) Real-World Deployment · p. 7

V. DISCUSSION AND CONCLUSION

  • Persistent geometric memory is paired with instruction-conditioned sparse semantic grounding; hybrid frontier and local waypoint candidates, landmark memory, and explicit progress tracking support VLM navigation.Source: §V · pp. 7–8
  • 42.8% success on R2R-CE and 40.7% on RxR-CE, both Val-Unseen; ablations favor instruction-related on-demand grounding over evaluated dense-semantic settings.Ablations also support hybrid waypoint proposals, precise landmark localization, instruction memory, and agent-centered map orientation; Unitree Go2 deployment across multiple indoor environments further validates the framework.Source: §V · p. 8
  • The findings suggest task-relevant sparse grounding can support instruction following with persistent geometry, without requiring a semantically complete scene map.They do not establish that reduced visual clutter alone causes the performance gain, because selective grounding also changes the semantic evidence available to the planner.Source: §V · p. 8

VI. LIMITATIONS AND FUTURE WORK

  • The rule-based waypoint generator limits choices to predefined frontier and local directional candidates, VLM inference remains a major computational bottleneck, and reported success rates trail some navigation-trained methods.Differences in training data and evaluation settings prevent attributing the performance gap solely to the absence of navigation-specific training.Source: §VI · p. 8
  • Future work will investigate learned instruction-conditioned waypoint proposals, more efficient VLM inference with asynchronous planning and execution, and evaluations across more diverse real-world settings.Asynchronous execution would require detecting stale plans and responding to environmental changes.Source: §VI · p. 8

Related papers

Loading 2609.26408v1…