PaperlayerView exact revision on arXiv

Source-linked AI summary

RECAST: Recasting Vision-Language Semantics into an Actionable Cost Map for Robot Navigation

Incheol Cho, Jintae Park, Jinkyu Kim, Jungbeom Lee, Jaegul Choo, Seokha Moon

arXiv:2609.32595v1cs.ROcs.AI

TL;DR

Navigation needs to translate complex-scene understanding into stable action, but learned policies may fail outside training data and VLM reasoning is often ungrounded or misaligned. RECAST grounds four VLM scene judgments in an AC map used to condition and score trajectory proposals, with pivot and current maps. It achieves the highest success and lowest collision rate among methods in simulation and the real world.

  • Problem

    Existing navigation approaches do not reliably turn high-level scene semantics into accurate action: learned policies can fail outside their training data, while VLM reasoning may be ungrounded or reduced to a fixed cue.

    Source: §I · p. 2

  • Method

    RECAST grounds VLM judgments about terrain, collision risk, heading, and passable gaps in an AC map that conditions trajectory decoders and scores their proposals using pivot and current frames.

    Source: §I · p. 2

  • Results

    RECAST achieves the highest success and lowest collision rate among methods in simulation and on a real quadruped.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    A new instruction reshapes the path through the map alone, while the two-frame map design keeps navigation aligned with the live scene as the VLM reasons.

    Source: §V · p. 7

    The method depends on its grounding models: anything they miss never enters the map.

    Source: §V · p. 7

Abstract

from arXiv · show

Safe and robust robot navigation across diverse environments requires a high-level understanding of complex scenes and the ability to carry it into stable motion. Recent works tackle this with learning-based models trained at scale and with approaches built on vision-language models (VLMs). However, learning-based models break down outside their training distribution, while VLM-based approaches bring that understanding but rarely ground it in the scene or align the action with it. To address these limitations, we present RECAST, a robot navigation framework that combines the reasoning of a VLM with the spatial grounding of vision foundation models to build an Actionable Cost map. Given the robot's front view and the user's instruction, we first decompose the scene with the VLM, judging which surfaces are traversable, which objects pose a risk, which heading to prefer, and which gaps are passable. Vision foundation models then ground these surfaces and objects in the image, and all four judgments are spatially recast into one compact cost map. This map both conditions the trajectory decoders and scores their proposals to select the one to execute. As the VLM's answers trail the live scene, both steps draw on cost maps from two points in time: the pivot frame the VLM judged, which carries all four judgments, and the current frame, whose terrain and collision costs are rebuilt from the current image. RECAST improves success over the strongest prior method by 13.3 points in simulation and 31.4 points on a real quadruped, and reduces the collision rate relative to it by 9.6 and 14.3 points, reaching the lowest collision rate among all methods. The project page is available at https://recast-nav.github.io/

I. INTRODUCTION

RECAST addresses the gap between understanding complex scenes and translating that understanding into accurate, scene-aligned action. It grounds four VLM judgments in an Actionable Cost map that conditions and selects trajectories using pivot and current frames.

  • Prior approaches leave scene semantics ungrounded or reduce VLM reasoning to a fixed cue, while large-scale learned policies may fail on scenes outside their training data.Source: §I · pp. 1–2
  • The AC map grounds terrain traversability, object collision risk, heading preference, and passable gaps, then both conditions trajectory decoders and selects their proposals.Source: §I · p. 2
  • A pivot–current design uses all four judgments from the VLM-analyzed frame and refreshes terrain and collision costs from the current frame.Source: §I · p. 2
  • RECAST achieves the highest success and lowest collision rate among methods in simulation and on a real quadruped.Source: §I · p. 2

II. RELATED WORKS

Prior navigation work uses learned policies and VLM reasoning, but the latter is rarely grounded in the scene and can lag behind it. RECAST’s framework builds on learned decoders while grounding scene judgments and refreshing map components as the scene changes.

A. Navigation Foundation Models

  • Learned navigation foundation models generalize across embodiments and environments, but their demonstrations limit deployment-time constraints and unfamiliar scenes can end in collisions.RECAST retains learned decoders, conditions them on the AC map, and uses the map to evaluate proposals without retraining.Source: A. Navigation Foundation Models · p. 2

B. VLM Reasoning for Navigation

  • VLM navigation methods often lack spatial grounding; BehAV grounds instruction-derived rules, whereas RECAST grounds judgments about the scene and uses a pivot–current design to handle delayed reasoning.Source: B. VLM Reasoning for Navigation · p. 2

III. METHODS

  • The task is instruction-guided point-goal navigation on a quadruped: the policy uses egocentric observations, frame and pose history, an instruction, and a point goal to predict body-frame waypoints.A low-level controller converts the waypoints to velocity commands.Source: §III · p. 2

A. The Actionable Cost Map (AC Map)

The AC map turns VLM judgments about a scene into spatially grounded cost components that can guide navigation. It combines these components on a pivot frame and refreshes terrain and collision costs on each current frame.

  • The AC map combines terrain traversability, collision risk, heading preference, and passable-gap judgments into spatially registered image-plane cost maps.Vision foundation models ground named surfaces and objects to image pixels; Gaussian blurring makes costs change gradually, penalizing paths nearer obstacles more.Source: A. The Actionable Cost Map (AC Map) · pp. 2–3
  • Terrain costs use segmentation probabilities weighted by traversability scores, with higher values marking preferred ground and lower values marking obstacles or hazards.Language-stated constraints directly change the traversability scores.Source: A. The Actionable Cost Map (AC Map) · p. 3
  • Collision costs place graded Gaussian holes at detected obstacles’ ground points; instruction-constrained obstacles can be cut out entirely, and the radius can be enlarged at inference.Source: A. The Actionable Cost Map (AC Map) · p. 3
  • Heading costs encode goal-oriented traversable directions, while passable-gap costs boost corridors between named object pairs to reopen narrow passages.Source: A. The Actionable Cost Map (AC Map) · p. 3
  • The pivot map contains all four components, while each current map refreshes only terrain and collision costs using pivot-frame labels and scores, without another VLM call.The refresh updates where things are, not what they mean.Source: A. The Actionable Cost Map (AC Map) · p. 3, Figure 2 · p. 3

B. AC Map-Conditioned Trajectory Decoders

The AC map is consumed by decoder variants that generate multiple trajectory proposals. The decoders use pivot and current information to produce trajectories and scores, while the training objective also penalizes jerk.

B. AC Map-Conditioned Trajectory Decoders

  • Two independently trained, decoder-agnostic variants consume the AC map: a regression decoder predicts anchor-relative residuals, while a diffusion decoder denoises truncated-noise anchors.Source: B. AC Map-Conditioned Trajectory… · pp. 3–4
  • Both decoders use K-means trajectory anchors and predict one trajectory per anchor, preserving multimodality in feasible paths.Source: B. AC Map-Conditioned Trajectory… · p. 4
  • The decoder receives separate pivot and current map streams, the lag between them, past frames and poses, and pivot-frame object and semantic streams to predict trajectories and scores.The goal reaches the decoder only through the AC map, where heading preference encodes it.Source: B. AC Map-Conditioned Trajectory… · p. 4
  • Training supervises the mode with the nearest anchor and combines trajectory, jerk, and classification losses.The jerk term penalizes zig-zagging between waypoints, which the trajectory term alone does not penalize.Source: B. AC Map-Conditioned Trajectory… · p. 4

C. AC Map-Scored Mode Selection

RECAST scores candidate trajectories on the Actionable Cost map, complementing the decoder’s learned confidence with judgments about progress, safety, and traversability. Current-frame costs can veto unsafe candidates, while pivot-frame costs and goal alignment rank those that pass.

C. AC Map-Scored Mode Selection

  • The decoder’s confidence head imitates demonstration choices but does not evaluate goal progress, safety, or traversability, so RECAST scores each candidate against the AC map instead.The map is projected along each candidate trajectory.Source: C. AC Map-Scored Mode Selection · p. 4
  • Current and pivot scoring maps are distinct: the pivot map includes all four components, while the current map uses terrain and collision costs.Both maps are rasterized onto a metric bird’s-eye-view grid.Source: C. AC Map-Scored Mode Selection · p. 4
  • The highest-scoring candidate is executed, and the robot stops if no candidate clears the current-frame safety threshold.The selected trajectory is tracked by the low-level controller.Source: C. AC Map-Scored Mode Selection · p. 4
  • Candidates with no waypoint inside trusted camera coverage score zero, treating unseen ground as unsafe.Current-frame trusted waypoints are restricted to a forward window.Source: C. AC Map-Scored Mode Selection · p. 4
  • The current-map factor uses a low percentile of trusted samples to veto risky candidates, while pivot-map scoring weights later waypoints more heavily.Only the current map can reject a candidate, so a stale pivot map cannot admit an unsafe path.Source: C. AC Map-Scored Mode Selection · pp. 4–5

A. Experimental Setup

The evaluation uses trained decoders, an 85-episode simulation benchmark, and repeated deployments across seven campus scenes on a quadruped. It reports success, navigation error, path-length-weighted success, and collision rate.

A. Experimental Setup

  • Simulation follows DynaNav’s 85-episode protocol across hospitals, offices, warehouses, and outdoor scenes with moving pedestrians, using a Spot quadruped for every method.Real-world evaluation uses a Vision60 quadruped in seven campus scenes, each run five times with goals 9 to 16 m away.Source: A. Experimental Setup · p. 5
  • Both decoders are trained on SCAND, GND, and a teleoperated Vision60 dataset, with evaluation spanning simulation and real-world deployment.The VLM is Gemini 3.5 Flash-Lite; CLIPSeg grounds surfaces and YOLO-World grounds objects.Source: A. Experimental Setup · p. 5
  • The reported metrics are success rate within 1.5 m before timeout, navigation error, success weighted by path length, and collision rate including sustained contact or passing within 0.2 m of a person.Real-world success is checked against outdoor UTM tracks and indoor video.Source: A. Experimental Setup · p. 5

B. Comparison with Prior Methods

RECAST’s diffusion decoder achieves the highest simulation success and path-length-weighted success, while its regression decoder has the lowest navigation error and collision rate. On the real quadruped, regression leads all reported metrics and has the lowest collision rate.

B. Comparison with Prior Methods

  • 80.0% SR with diffusion is the highest among methods on DynaNav, alongside 70.0% SPL; regression has the lowest NE (4.07 m) and CR (22.4%).All three RECAST variants have higher SR than prior methods; the MPC variant reaches 71.8% SR versus 66.7% for NavDP†.Source: Table I · p. 5, B. Comparison with Prior Methods · p. 5
  • 77.1% SR, 1.94 m NE, 60.2% SPL, and 28.6% CR make RECAST regression the leader on every metric in the real-world comparison.NoMaD, the strongest prior method, reaches 45.7% SR and has a 42.9% collision rate.Source: Table II · p. 5, B. Comparison with Prior Methods · p. 5

C. Ablations

The ablations test the AC map’s components, its use across pivot and current frames, and its role in scoring trajectory proposals. The full map improves over a terrain-only map, while removing current-frame information or map-based proposal scoring degrades performance.

Effect of the AC map components

  • The full AC map improves success by 18.8 points and collision rate by 12.9 points over Model A’s terrain-only map.Adding collision risk, heading preference, and passable gaps contributes successive success gains; the heading map also reduces collisions from 34.1% to 22.4%.Source: C. Ablations · pp. 5–6, Table III · p. 6

Pivot–current design

  • Using only the pivot map reduces success from 77.6% to 65.9% and raises collision rate from 22.4% to 32.9%.The single-frame variant still navigates, but planning and vetoing on a scene from seconds earlier degrades accuracy, path quality, and safety.Source: C. Ablations · p. 6, Table III · p. 6

AC map-scored mode selection

  • Replacing map-based proposal scoring with a classification head drops success to 14.1%, showing that map conditioning alone is not enough.The comparison keeps the candidate trajectories and safety veto the same.Source: C. Ablations · p. 6, Table IV · p. 6

D. Qualitative Analysis

The qualitative examples show instructions reshaping the AC map and selected path, including a safety veto when no candidate is acceptable. Across simulation and real-world episodes, RECAST follows constrained routes while prior methods fail or deviate.

Effect of the instruction on the AC map

  • In two real scenes, added instructions reshape the AC map and can change the selected path or trigger a stop when every candidate is rejected.For example, permitting gravel shifts the path around a construction fence, while forbidding grass can lead to a safety veto on a blocked sidewalk.Source: D. Qualitative Analysis · pp. 6–7, Figure 4 · p. 7

Simulation

  • In the hospital and warehouse episodes, RECAST avoids approaching pedestrians and threads a gap between workers without touching a rack.ViNT and NaVIDA have already failed by the later hospital frame; prior methods cut through the warehouse crowd or swing wide.Source: D. Qualitative Analysis · p. 7, Figure 4 · p. 7

Real world

  • In two campus runs, RECAST passes between curb stones and tables as instructed, while three prior methods fail outdoors and two fail indoors by the later frame.Other methods that cannot take or follow the instruction wander across the open floor.Source: D. Qualitative Analysis · p. 7, Figure 4 · p. 7

V. CONCLUSION

RECAST grounds VLM scene judgments in an AC map that conditions trajectory decoders and selects their proposals, using pivot and current frames to stay aligned with the live scene. It achieves the highest success and lowest collision rate in simulation and the real world; grounding-model omissions remain its main limitation.

Conclusion

  • RECAST grounds VLM scene judgments with vision foundation models in an AC map that conditions trajectory decoders and selects proposals to execute.The map uses both pivot and current frames to stay aligned with the live scene while the VLM reasons.Source: §V · p. 7
  • RECAST achieves the highest success and lowest collision rate in both simulation and the real world.Source: §V · p. 7
  • The grounding models are RECAST’s main limitation: anything they miss never enters the map; future work will predict obstacle motion to bridge pivot and current frames.Source: §V · p. 7

Related papers

Loading 2609.32595v1…