PaperlayerView exact revision on arXiv

Source-linked AI summary

PASS-VLN: Passability-Aware Local-Command Vision-and-Language Navigation

Cong Zou, Xiangsheng Shi, Xuanhui Wang, Fubin Wang, Guangjun Ge, Chao Wang, Yu Wang

arXiv:2610.22812v1cs.RO

TL;DR

Existing VLN benchmarks generally assume instructed motion is executable, leaving body-dependent local execution underexamined. PASS-VLN pairs a controlled local-command benchmark with a body-width-conditioned model that grounds commands in geometry and executes latent actions in a closed loop. The paper reports adaptation to unseen widths and improved collision-free execution, including in hierarchical navigation and real-world deployment.

  • Problem

    Existing VLN benchmarks typically assume executable instructions and emphasize long-horizon tasks, leaving body-dependent local passability and collision-free execution underexamined.

    Source: §I · p. 1

  • Method

    PASS-VLN varies robot width while holding the scene, instruction, start pose, and route fixed, and uses body-conditioned geometry with latent-action prediction for local execution.

    Source: §I · pp. 1–2

  • Results

    Experiments report generalization to unseen body widths and improved collision-free success over the evaluated passability-gating baseline, with higher success and fewer local commands in hierarchical navigation.

    Source: §I · p. 2

  • Takeaways & Limitations

    PASS-VLN demonstrates a local executor that adapts reach-versus-stop behavior to body constraints, including in hierarchical navigation and real-world deployment.

    Source: §VI · p. 8

Abstract

from arXiv · show

Reliable vision-and-language navigation (VLN) requires not only understanding instructions but also determining whether and how a robot can physically execute them. However, existing VLN benchmarks typically assume executable instructions and focus on long-horizon tasks. This leaves body-dependent local execution, including passability and collision-free execution, largely unexamined. Evaluating these factors within long-horizon tasks would entangle local physical execution with task planning and long-term spatial memory, making navigation failures difficult to attribute. We therefore focus on grounding local navigation commands in executable motion under environmental and robot body constraints. We present PASS-VLN, a passability-aware local-command VLN benchmark and model. The benchmark holds the scene, instruction, start pose, and reference route fixed while varying robot body width, requiring target completion when the instructed passage is feasible and stopping before blockage otherwise. Our model encodes body width through obstacle inflation in a depth-derived local bird's-eye view, fuses geometry with visual and linguistic context, and predicts discrete latent action codes for continuous closed-loop execution. Experiments show that PASS-VLN improves collision-free success by approximately 12 percentage points over a strong VLN model fine-tuned on our benchmark, while adapting reach-versus-stop behavior to body constraints and generalizing to unseen body widths. When integrated with a high-level planner, PASS-VLN achieves higher success with fewer local commands, while real-world deployment further demonstrates its effectiveness as a low-level executor. The benchmark and model will be released.

I. INTRODUCTION

Existing VLN benchmarks generally assume instructed motion is executable, while long-horizon evaluation entangles local execution with planning and memory. PASS-VLN isolates local-command execution under body-width constraints with a controlled benchmark and conditioned model.

I. INTRODUCTION

  • Existing benchmarks assume executable instructions, while evaluating physical execution in long-horizon tasks entangles it with planning, progress tracking, and spatial memory.Route-level reasoning gains do not necessarily translate into improved local physical execution.Source: §I · p. 1
  • PASS-VLN frames local commands as an interface: a high-level agent specifies an immediate objective and spatial cues, and a low-level executor grounds them in observations and motion.A command can specify landmarks, spatial relations, and a stopping condition while leaving broader planning to the issuer.Source: §I · p. 1
  • The benchmark holds scene, instruction, start pose, and route fixed while varying body width, testing whether the same command calls for reaching or stopping before blockage.It covers commands requiring local observations or broader scene and route knowledge.Source: §I · p. 1
  • The model incorporates body width through obstacle inflation in a depth-derived local bird’s-eye view, then fuses geometry with language, current vision, and visual history.It predicts discrete latent action codes learned from short trajectory windows, decoded into planar relative motions for closed-loop control.Source: §I · pp. 1–2
  • Experiments report generalization to unseen body widths and improved collision-free success over the evaluated passability-gating baseline; hierarchical use achieves higher success with fewer local commands.Real-world deployment further demonstrates the model’s utility as a low-level executor.Source: §I · p. 2

II. RELATED WORK

PASS-VLN complements route-oriented VLN benchmarks by isolating local-command grounding and body-width-dependent physical feasibility. It differs from recent embodied and large-model work by focusing on local execution for hierarchical navigation.

A. Vision-and-Language Navigation Benchmarks

  • PASS-VLN complements diverse VLN benchmarks by evaluating whether an instructed local motion remains executable under different body widths, including unseen widths.The cited benchmarks span multilingual, continuous-control, object-grounding, outdoor, aerial, and physics-aware settings.Source: A. Vision-and-Language Navigation… · p. 2

B. Large-Model-Based VLN and Physical Execution

  • Unlike work centered on route-level navigation, general embodied reasoning, or platform adaptation, PASS-VLN studies body-width-dependent local execution between high-level commands and low-level motion.Related methods include VLA planning with locomotion skills, embodiment-specific affordances, and continuous local policies.Source: B. Large-Model-Based VLN and Physical… · p. 2

III. PASS-VLN BENCHMARK

The benchmark samples bounded local trajectories with limited turning, then pairs them with multimodal evidence to generate commands at different spatial-context levels. Instructions are verified for grounding and executable motion specification.

A. Controlled Local-Trajectory Sampling

  • Trajectories are bounded local objectives with at most one major heading change, categorized as straight, initial-turn, or mid-turn patterns.This limits repeated turns, long-term progress tracking, and accumulated execution errors while representing common local navigation structures.Source: A. Controlled Local-Trajectory Sampling · p. 2

B. Evidence-Grounded Instruction Generation

  • Commands are generated from chronological egocentric keyframes, a top-down map, semantic objects, and trajectory type, with map-free and map-informed context variants.Map-free commands use observations and commonsense knowledge; map-informed commands can specify route-level turns, landmarks, and stopping locations beyond view.Source: B. Evidence-Grounded Instruction… · p. 2
  • A VLM proposes instructions, which undergo geometric and VLM-based checks for grounding, executability, brevity, and unambiguous motion specification.The two command settings cover local goal discovery and precise route execution.Source: B. Evidence-Grounded Instruction… · p. 3

C. Body-Width-Aware Passability Annotation

Passability labels are tied to the commanded route and robot width: feasible cases require reaching the target, while blocked cases require stopping before the blockage. The benchmark evaluates this behavior with width-specific configurations and collision-aware metrics.

C. Body-Width-Aware Passability Annotation

  • A command feasible for a narrow robot may be infeasible for a wider one; map-free cases with a comparable alternative detour under 5 m are discarded.Width-aware path search and exact rectangular-footprint validation determine the longest collision-free prefix and pre-blockage stopping reference.Source: C. Body-Width-Aware Passability… · p. 3

D. Benchmark Configuration and Metrics

  • 0.35 m and 0.70 m are training widths, while 0.20 m and 0.50 m are held out to test generalization to unseen widths.The evaluation uses MP3D validation unseen scenes, with 2,971 map-free and 3,470 map-informed episodes; 95.6% of 500 sampled instructions were judged correct and unambiguous.Source: D. Benchmark Configuration and Metrics · p. 3
  • Passable episodes succeed by stopping within 1.5 m of the endpoint; impassable episodes succeed within 1.5 m before the first blockage, with a 100-step limit.The metrics include collision-free rate and collision-free success rate in addition to navigation and success measures.Source: D. Benchmark Configuration and Metrics · pp. 3–4

IV. PASS-VLN MODEL

PASS-VLN processes instructions, visual observations, and body-width-conditioned geometry in a streaming model that predicts latent actions for closed-loop execution. Its depth-derived geometry represents clearance for the robot’s configured width and can be recomputed for unseen widths.

A. Model Overview

  • At each step, PASS-VLN combines language, visual history and current vision, and body-width-conditioned geometry to predict four latent-action tokens for a relative target transformation executed by a low-level controller.It repeats prediction after acquiring new observations.Source: A. Model Overview · p. 4

B. Body-Width-Conditioned Geometry Fusion

  • Depth points form a binary occupancy map, and obstacle cells are expanded laterally using body width, a safety margin, and BEV cell size; only the lateral dimension is explicitly conditioned.Longitudinal effects are handled by fixed dynamics and trajectory supervision.Source: B. Body-Width-Conditioned Geometry… · p. 4
  • The four-channel body-conditioned BEV is encoded into geometry tokens and fused with current visual tokens, instruction tokens, and visual-history tokens; recomputing it for arbitrary widths supports operation on unseen widths.The channels represent inflated occupancy, body-conditioned free space, unknown space, and normalized obstacle clearance.Source: B. Body-Width-Conditioned Geometry… · p. 4

C. Latent Action Learning and Execution

PASS-VLN encodes short planar-motion windows as discrete latent actions, predicts those codes autoregressively, and decodes them into relative targets for closed-loop execution. Its evaluation tables cover fixed-width local-command performance and generalization across seen and unseen widths.

C. Latent Action Learning and Execution

  • Each one-second trajectory segment is encoded into two translation and two rotation codes, using separately trained vector-quantized temporal encoders.The trajectories are collected under maximum linear and angular velocities of 0.5 m/s and 30°/s.Source: C. Latent Action Learning and Execution · p. 4
  • The latent codes decode into a planar relative target pose, and latent-action learning uses translation reconstruction, rotation geodesic, and vector-quantization losses.During VLN training, the code indices are added as special output tokens while the decoder and codebooks remain frozen.Source: C. Latent Action Learning and Execution · pp. 4–5
  • Table I covers local-command performance at a fixed 0.35 m body width, while Table II covers seen- and unseen-width generalization and includes a simulator-passability-gated baseline.The supplied table passages give scope and baseline descriptions, not cell values.Source: Table I · p. 5, Table II · p. 5
  • Four latent-action codes are predicted autoregressively using a sliding-window cache of recent observation–action dialogues, with older visual context compressed into slow memory tokens.At inference, the decoded target is converted into body-frame velocities for closed-loop execution.Source: C. Latent Action Learning and Execution · p. 5
  • A decoded transformation is treated as STOP below 0.05 m translation and 3° absolute rotation, avoiding a separate termination head while retaining smooth displacement and heading adjustment.The latent representation is also described as supporting passage alignment and closed-loop obstacle avoidance.Source: C. Latent Action Learning and Execution · p. 5

A. Implementation Details

PASS-VLN combines body-width-aware geometry and latent-action prediction for local-command execution. Its implementation fine-tunes a streaming VLN model, and its evaluation compares the model with existing and adapted VLN systems.

A. Implementation Details

  • The model starts from LLaVA-Video 7B, is fine-tuned on R2R-CE and RxR-CE before PASS-VLN training, and uses a 96×96 local BEV encoded into 16 geometry tokens.Training uses 8 NVIDIA A800 GPUs for one epoch; each latent action has two translation and two rotation codes with a codebook size of 8.Source: A. Implementation Details · p. 5

B. Local-Command Execution

  • Local-command evaluation compares released VLN checkpoints at a fixed 0.35 m body width and fine-tunes StreamVLN as a controlled adaptation baseline.The shared base streaming architecture supports comparison of task adaptation and model design.Source: B. Local-Command Execution · p. 5
  • PASS-VLN further improves physical execution through body-width-aware geometry and latent-action prediction.The evaluation compares it with StreamVLN, DualVLN, JanusVLN, and a fine-tuned StreamVLN baseline.Source: B. Local-Command Execution · pp. 5–6

C. Body-Width-Aware Navigation

PASS-VLN evaluates whether local execution changes with robot body width, including on paired routes where only a narrower robot can pass. The results distinguish body-conditioned execution from a binary passability gate.

C. Body-Width-Aware Navigation

  • PASS-VLN is evaluated at seen widths of 0.35 and 0.70 m and unseen widths of 0.20 and 0.50 m, testing extrapolation and interpolation, respectively.Passability labels are recomputed for each body width.Source: C. Body-Width-Aware Navigation · p. 6

1) Body-Width-Conditioned Execution versus Passability Gating:

  • The GT passability gate can correctly terminate more routes at larger widths, but width-agnostic VLN has more collisions and larger CFR and CFSR drops.The authors conclude that reliable execution also requires body-width-dependent clearance, alignment, and avoidance.Source: 1) Body-Width-Conditioned Execution… · p. 6

2) Paired Body-Width Evaluation:

  • On paired routes passable for a narrower body but blocked for a wider one, PASS-VLN changes its behavior according to body width rather than following a width-independent strategy.The narrow robot should reach the target, while the wider robot should stop before the first blockage.Source: 2) Paired Body-Width Evaluation · p. 6

D. Local Execution in an Oracle Hierarchical Pipeline

In a shared oracle hierarchical pipeline, PASS-VLN has the best overall performance, with fewer local commands. Ablations find benefits from latent trajectories and geometric conditioning, while real-world figures illustrate width-aware and extended local-command execution.

D. Local Execution in an Oracle Hierarchical Pipeline

  • PASS-VLN achieves the best overall performance in the shared hierarchical pipeline, where more reliable local execution is associated with fewer corrective interactions and improved navigation efficiency.CFR and CFSR are measured over the long-horizon episode, so a collision in any local command makes the episode non-collision-free.Source: D. Local Execution in an Oracle… · p. 6

E. Ablation Studies

  • Replacing atomic actions with latent trajectories improves CFR and CFSR, while depth and raw BEV improve physical execution to some extent but do not encode body width.With the same multi-width supervision, geometric conditioning generalizes better than the evaluated width-prompt baseline; that baseline has only 13.0% GSR on the unseen pair.Source: E. Ablation Studies · p. 6
  • The real-world deployment figure demonstrates width-dependent traversal, blockage stopping, corridor selection, and sequential blockage detection.The deployment uses a Unitree Go2-W.Source: Figure 3 · p. 7

D. Local Execution in an Oracle Hierarchical Pipeline

  • The local-command execution figure covers indoor and outdoor environments, with sequential map-informed commands for long-range navigation and map-free commands for local spatial grounding.The caption identifies these as distinct command settings.Source: Figure 4 · p. 7

F. Real-World Deployment

Real-robot trials show PASS-VLN adjusting traversal and stopping to passage width, while local commands also ground nearby landmarks and spatial relations in tested indoor and outdoor environments. The conclusion reports execution at unseen body widths and in hierarchical navigation.

F. Real-World Deployment

  • 0.6–1.0 m passages were traversed collision-free in all trials; at 0.5 m, the robot traversed in two of three trials with minor boundary contact, and at 0.4 m it consistently stopped.Passages were tested in 0.1 m increments, with three trials per setting.Source: F. Real-World Deployment · p. 7
  • In asymmetric layouts PASS-VLN selected the feasible alternative, and in sequential passages it detected downstream blockages.The robot used a configured width of 0.45 m, including a 0.01 m safety margin.Source: F. Real-World Deployment · p. 7
  • Sequential local commands guided map-informed navigation across an approximately 500 m2 indoor workspace and a 3600 m2 outdoor environment, while map-free commands grounded nearby landmarks and spatial relations.The reported demonstrations show local commands composed for extended navigation in the tested environments.Source: F. Real-World Deployment · pp. 7–8

VI. CONCLUSION

  • PASS-VLN reached feasible targets and stopped before body-width-induced blockages at unseen widths, with effectiveness also demonstrated in reference-route-assisted hierarchical navigation and real-world deployment.The conclusion identifies finer-grained stopping supervision, collision-free action learning, and more robust geometry-aware control as future work, particularly for wide embodiments and narrow clearances.Source: §VI · p. 8
Loading 2610.22812v1…