OWLS: Open-world Language-conditioned Speed Fields for Zero-Shot Rover Navigation

I. INTRODUCTION

• Traversability depends on obstacle scale, load bearing, and slip, properties that a camera cannot measure directly and that matter in severe planetary conditions.Planetary rovers operate without GPS, and communication delays require long stretches of onboard autonomy.Source: §I · p. 1
• Geometric heuristics require per-platform engineering, learned terrain classes need annotated data and assigned costs, and self-supervised estimators presume target-terrain experience.Open-vocabulary VLMs remove fixed labels, but prior navigation systems still derive free space from depth or LiDAR rather than planning over dense VLM output.Source: §I · p. 1
• OWLS uses ground-projected VLM pixel scores as a minimum-time planner’s speed field, with morphological adaptation encoding rover size and clearance.The method uses no task-specific training or terrain annotation, and carries the same VLM and prompt across three differently sized rovers without retraining.Source: §I · pp. 1–2
• OWLS is evaluated on a physical rover, simulated Mars and Moon rovers over 200 procedurally generated terrains, and real planetary imagery.The simulated evaluation includes prompt and morphology ablations.Source: §I · p. 2

Traversability from Appearance

• Semantic traversability methods use appearance as a proxy for physical properties, but commonly rely on annotated fixed classes and manually assigned planning costs.Other approaches use robot experience or pretrained visual features, yet still train platform-specific estimators; OWLS shifts platform dependence to two morphology radii set from rover geometry.Source: §II · p. 2

Mapless Navigation and Planning over Speed Fields

• Learned image-goal policies model traversability implicitly, whereas OWLS keeps an explicit planner whose speed comes from terrain appearance as well as geometry.Classical visual servoing also operates without a persistent map, but relies on hand-engineered geometric features.Source: §II · p. 2

VLMs for Navigation

• Existing VLM navigation systems use depth, LiDAR, or learned models to determine traversability, while the VLM selects goals, annotates maps, or ranks paths.OWLS instead uses the ground-projected VLM feature map, adapted by two radii, as the planner’s speed field from a single calibrated camera and without training.Source: §II · p. 2

III. METHOD

• The VLM’s continuous pixel-aligned response to a terrain prompt is ground-projected, adapted by morphology, and used as the speed field for minimum-time planning.A waypoint controller tracks the resulting path; planning runs at perception rate and tracking at control rate.Source: §III · p. 2, A. Planning over VLM Features · p. 2, Figure 2 · p. 3

A. Planning over VLM Features

• Ground projection lifts pixel scores into a top-down robot-frame map using calibrated camera geometry and a flat-ground assumption.Grid cells outside the image or above the horizon receive value zero.Source: A. Planning over VLM Features · p. 3
• Dilation removes low-scoring regions smaller than the platform’s traversable obstacle scale, while erosion adds a distance-decaying clearance buffer.The continuous map is not thresholded; the clearance radius depends on track width, while the obstacle radius is shared across platforms.Source: A. Planning over VLM Features · p. 3
• The planner uses the continuous adapted map directly as cell speed, with a positive floor that keeps every cell passable.It takes low-traversability terrain only when going around would take longer; unobserved cells and confident obstacles have the same floor speed.Source: A. Planning over VLM Features · p. 3
• The eikonal navigation function gives travel time to the goal, and the trajectory descends it by repeatedly choosing a step that most reduces remaining time.Clearance buffers shape the path around obstacles while VLM scores steer it across terrain in one minimum-time optimization; the goal seed is recomputed each cycle as new terrain enters view.Source: A. Planning over VLM Features · p. 3

B. Trajectory Tracking

• Each new image provides a trajectory, which wheel odometry transforms into the rover’s current frame between images.The controller targets the first waypoint beyond a lookahead distance.Source: B. Trajectory Tracking · p. 3
• The experimental platforms include a real four-wheeled rover and simulated Perseverance and Pragyan rovers.The figure shows them from left to right in that order.Source: Figure 3 · p. 4
• On a mobile RTX 4090, a GSAM2 planning cycle takes 692 ms, including 490 ms for the VLM and 199 ms for morphology.Waypoint and action generation take less than a millisecond, consistent with a 1 Hz planning rate.Source: B. Trajectory Tracking · p. 4

C. Implementation Details

• The 640 × 640 grid uses 0.0125 m cells, spanning 8 m ahead and ±4 m laterally; planning runs at 1 Hz.Source: C. Implementation Details · p. 4
• Morphological radii are 2.15 m for Perseverance and 0.50 m for Pragyan and the terrestrial rover, with ρt = 0.19 m throughout.Source: C. Implementation Details · p. 4
• Blind and self-occluded cells receive maximum traversability rather than V = 0, so they are not treated as obstacles.Source: C. Implementation Details · p. 4
• The speed floor is ϵ = 0.1, with gains s = 8 for Grounded SAM 2 and s = 16 for SAM 3.Source: C. Implementation Details · p. 4

IV. EXPERIMENTS

• The evaluation uses real-rover trials to test feasibility under real compute and sensor constraints.Source: §IV · p. 4
• A simulated sweep over 100 terrains per platform tests transfer across space embodiments and appearances.Source: §IV · p. 4
• Ablations vary the prompt and morphology, described as the only inputs set per deployment.Source: §IV · p. 4
• Table I covers navigation success rate for a real four-wheeled rover across two Earth scenarios, with dynamic and static obstacles and five trials per configuration.Source: Table I · p. 4

A. Physical Rover Experiments on Earth

• The rover used a front-facing RGB camera and wheel-encoder odometry; trials used the prompt “drivable ground” and compared three dense feature maps.Source: A. Physical Rover Experiments on Earth · p. 4
• Success meant traversing the environment end to end without collision, and differences of one or two trials are treated as indicative only.Source: A. Physical Rover Experiments on Earth · p. 4
• Grounded SAM 2 avoided all static obstacles indoors and outdoors in all 5 trials, and dynamic ones indoors 80% of the time; only 2 of 5 dynamic outdoor trials succeeded.Source: A. Physical Rover Experiments on Earth · p. 4, Figure 4 · p. 5
• Across successive perception cycles, the heatmap excludes crossing people and OWLS replans over ground unoccupied at that moment.Source: Figure 4 · p. 5

B. Simulated Rover Experiments on Mars and Moon

• OWLS reached the goal on 58.0 and 60.0 terrains per 100 for Perseverance and 75.0 and 77.0 for Pragyan, versus NoMaD’s 23.0 and 35.0.Source: B. Simulated Rover Experiments on Mars… · p. 5
• On shared solved terrains, NoMaD took 30.4 vs. 27.3 s against OWLS with GSAM2 on Perseverance, and 46.4 vs. 38.6 s on Pragyan.Source: Table II · p. 6
• Under the deployed prompt, VLM variants were within two points on both platforms (58.0 vs. 60.0; 75.0 vs. 77.0); replacing “ground” with “area” or “region” took the same SAM 3 pipeline to 0.0 on both platforms.Source: B. Simulated Rover Experiments on Mars… · p. 6
• OWLS failures more often stalled than timed out, while Fig. 5 also depicts forgotten out-of-view obstacles and slopes misread as cliffs.Source: B. Simulated Rover Experiments on Mars… · p. 6, Figure 5 · p. 6

Prompt ablations

• Naming “rocks” instead of free space cuts success to 10.0 (GSAM2) and 34.0 (SAM 3) on Mars, and 10.0 for SAM 3 on the Moon.GSAM2 on the Moon changes from 75.0 to 71.0 because it often grounds “rocks” onto regolith when no rock is visible.Source: C. Prompt and Morphology Ablations · p. 6, Table III · p. 6
• SAM 3 reaches 0.0 success on both platforms under “area” and “region” as empty-mask rates rise to 99–100% on Perseverance and 97–93% on Pragyan.GSAM2 stays within four points of its deployed value on Perseverance and eight on Pragyan for other free-space nouns; the authors retain “ground” and recommend validating wording before deployment.Source: C. Prompt and Morphology Ablations · p. 6, Table III · p. 6
• SAM 3 under “surface” reaches 90.0 success on Pragyan but scores over 97% of the frame traversable, while Perseverance remains at 59.0.Its median peak score is 0.54, versus about 0.1 for the other nouns.Source: C. Prompt and Morphology Ablations · p. 6, Table III · p. 6

Morphology scan

• The geometry-derived radii match or beat every scanned value; erosion peaks at 2.15 m, while dilation at 1.0 m costs 13 points.Erosion without dilation costs 7 points and 3.0 m costs 6; dilation is flat from off to the deployed 0.19 m. The metric scores goal-reaching, not clearance.Source: C. Prompt and Morphology Ablations · p. 7, Table IV · p. 7

• SAM 3 separates ground from hazards in real Perseverance and Pragyan imagery, but misclassifies under extreme lighting.The labeled imagery is too scarce for quantitative scoring, so the evidence is visual inspection of feature maps.Source: D. Ground Detection on Images from… · p. 7

Conclusion

• OWLS reaches the goal on most of 100 unseen terrains per platform with either VLM and beats a learned navigation foundation model in all four simulated settings.Real-rover trials establish feasibility; hazard-noun prompts cut success in three of four settings, and prompt rewording alone can zero it out.Source: §V · p. 7
• OWLS projects prompted VLM pixel scores into a platform-adapted speed field for minimum-time planning, without task-specific training, terrain annotation, or a persistent map.A waypoint controller tracks the planned trajectory.Source: §V · p. 7

Limitations and Future Work

• The memory-less design forgets obstacles that leave view, while flat-ground projection can be inaccurate on steep slopes and does not model overhead clearance or tall obstacles.The paper identifies forgetting out-of-view obstacles as its most common failure.Source: §V · p. 7
• The off-the-shelf VLM runs on compute beyond current flight hardware, and planetary optical effects were not included in closed-loop simulation.The paper also notes that changing a free-space noun can take SAM 3 to zero success and that current VLMs work best with appearance-based prompts.Source: §V · p. 8