PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Language-Conditioned Traversability Representations for Adaptive Visual Navigation

Senda Chen, Changxu Cheng, Fangdi Li, Tao Wang, Wuyue Zhao

arXiv:2610.11622v1cs.RO

TL;DR

Traversability depends on robot capabilities and language requirements, but predefined costs require manual tuning and viewpoint-dependent masks complicate asynchronous planning. LaTraNav learns a traversability representation shared by a slow VLM and fast flow-matching planner; it improves planning over explicit masks and enables more frequent path updates at the same semantic-update rate.

  • Problem

    Predefined terrain costs require heuristic rules and manual tuning, while viewpoint-dependent traversability masks complicate planning updates between slow VLM inferences.

    Source: §I · p. 1

  • Method

    LaTraNav uses a slow VLM to predict goal and traversability representations that condition a fast flow-matching planner, with segmentation and path supervision jointly grounding the representation.

    Source: §I · pp. 1–2

  • Results

    Path updates rise from 1.21 to 7.32 Hz (6.05×) at a 1.21 Hz semantic-update rate, and latent conditioning improves planning over explicit masks.

    Source: §I · p. 2, E. Structural and Training Ablations · p. 7

  • Takeaways & Limitations

    The results support Trav. Rep. as a reusable interface between language understanding and adaptive visual navigation.

    Source: §VI · p. 8

Abstract

from arXiv · show

Traversability is essential for visual navigation but varies with robot capabilities and user preferences. Conventional pipelines often rely on explicit costmaps or segmentation masks with predefined criteria, requiring hand-crafted rules and careful tuning. Moreover, viewpoint-dependent segmentation masks complicate asynchronous planning under perception latency. We present LaTraNav, a framework that learns language-conditioned traversability representations for adaptive visual navigation. Its asynchronous architecture combines a slow vision-language model that produces latent representations of traversability and navigation goals, with a fast flow-matching planner conditioned on these representations. To train the system, we develop a simulation-based data generation pipeline with controllable trajectories, producing observations paired with language instructions, traversability maps, goal locations, and diverse trajectories. Photorealistic image translation further enhances visual realism. Evaluations on datasets from multiple sources demonstrate effective language-guided traversability segmentation and goal localization by the slow VLM, alongside adaptive pixel-space path planning by the fast planner. Latent conditioning improves planning performance over explicit segmentation masks, while asynchronous scheduling increases the path-update rate by $6.05\times$ at the same semantic-update rate.

I. INTRODUCTION

Traversability depends on both robot capabilities and language-specified requirements, while predefined costs and viewpoint-dependent masks complicate adaptive, asynchronous planning. LaTraNav connects a slow VLM to a fast planner through a learned traversability representation, with improved planning results and more frequent path updates.

I. INTRODUCTION

  • Traversability varies with robot capabilities and user requirements, while heuristic terrain costs require manual tuning and viewpoint-dependent masks complicate path updates between VLM inferences.A mask is tied to its prediction viewpoint and may need spatial alignment or regeneration after camera motion.Source: §I · p. 1, Figure 1 · p. 1
  • A slow VLM predicts goals and traversability representations that a fast flow-matching planner reuses between semantic updates for asynchronous path generation.The planner incorporates current visual observations without repeating VLM inference at every planning step.Source: §I · p. 2
  • The controllable data pipeline pairs observations with language, traversability masks, goals, and paths across agent conditions, using simulation, photorealistic generation, and visual datasets.Source: §I · p. 2
  • LaTraNav improves planning over explicit-mask conditioning and raises path updates from 1.21 to 7.32 Hz (6.05×) while semantic updates remain at 1.21 Hz.Across 13,580 evaluation samples, traversability IoU is 0.8817 and joint goal-arrival and region-adherence success is 63.95%.Source: §I · p. 2
  • The learned representation is jointly grounded by pixel-level segmentation supervision and path learning, linking robot capabilities and task requirements to route selection.Source: §I · p. 2

II. RELATED WORK

Prior work addresses visual navigation, traversability, and language-conditioned region understanding through distinct representations and interfaces. LaTraNav combines agent- and language-conditioned region prediction with image-space planning through a learned representation.

A. Visual Navigation

  • LaTraNav couples free-form route constraints with agent- and language-conditioned region prediction and image-space planning from a single view.Related approaches include learned semantic planners,,, capability-conditioned features,, joint traversability and trajectory prediction, and preference or language costs,.Source: A. Visual Navigation · p. 2
  • Navigation interfaces range from spatial instructions passed to locomotion to affordance scoring, path-mask conversion, bird’s-eye-view prediction, and cached costmaps.LaTraNav instead uses a goal and learned Trav. Rep. grounded by segmentation and path supervision for asynchronous image-space planning.Source: A. Visual Navigation · p. 2

B. Referring Expression Segmentation

  • Referring-expression segmentation grounds language in image regions, with LAVT integrating language into visual encoding and LISA using segmentation-token hidden states for implicit-intent queries.These methods support traversability targets that vary with descriptions of agent capabilities and permitted regions.Source: B. Referring Expression Segmentation · p. 2
  • LaTraNav builds on UFO’s mask-token interface, exposing hidden states as Trav. Rep. for the planner, with segmentation and path supervision grounding and adapting the representation.Source: B. Referring Expression Segmentation · pp. 2–3

III. METHOD

LaTraNav predicts a local image-space path from an egocentric RGB observation and an instruction specifying the agent, destination, and traversal constraints. A VLM supplies goal and traversability conditions to a flow-matching planner, which can update paths between semantic updates.

III. METHOD

  • The task predicts an agent-conditioned route-feasibility decision, a goal point, and an agent- and language-conditioned traversable region represented by Trav. Rep. Z.Source: §III · p. 3
  • The VLM generates a goal and mask-token hidden states Z, while an adapter maps Z to planner tokens H and a decoder produces the traversability mask.The flow-matching planner generates a path when the VLM predicts route feasibility; otherwise, the VLM returns a refusal.Source: §III · p. 3
  • Between semantic updates, the planner combines cached goal and traversability conditions with current visual features, replacing both cached conditions at the next update.Source: §III · p. 3

A. Learning Traversability Representations for Planning

LaTraNav shares a traversability representation between mask decoding and path generation, grounding it with region labels and adapting it to route selection. Conditional flow matching then generates paths from noise using the goal and permitted regions as conditions.

A. Learning Traversability Representations for Planning

  • Trav. Rep. is shared by mask decoding and path generation: pixel-level annotations ground its region semantics, while path supervision adapts it to route selection.The flow-matching planner consumes adapted tokens H, and path-loss gradients backpropagate through the adapter to the parameters producing Z, without passing a thresholded mask to the planner.Source: A. Learning Traversability… · p. 3

B. Learning Region-Conditioned Path Distributions

  • Conditional flow matching transforms Gaussian noise into consecutive-point path increments using a conditional vector field, with goal and H conditioning each update.Euler integration accumulates the generated increments relative to the path anchor s.Source: B. Learning Region-Conditioned Path… · p. 3

C. Joint Optimization

LaTraNav jointly trains its VLM and flow-matching planner through shared mask-token hidden states, while its data pipeline pairs controllable agent- and language-conditioned routes with visual observations and aligned annotations. Simulation controls route geometry and robot capabilities; photorealistic generation broadens appearance and is followed by mask realignment.

C. Joint Optimization

  • The joint objective combines text-response, segmentation, and path losses, with segmentation and path losses applied only to compatible samples.The segmentation loss comprises focal and Dice losses, and the response sequence uses autoregressive cross-entropy.Source: C. Joint Optimization · p. 3
  • Path supervision reaches the VLM through the Trav. Rep., while goal localization uses text supervision and the representation receives pixel-level segmentation and path supervision.Segmentation and path losses are computed only for compatible samples; during training, the planner uses the reference goal.Source: C. Joint Optimization · p. 3

IV. CONTROLLABLE DATA CONSTRUCTION

  • The shared planning-task format pairs observations with instructions, traversability masks, goals, reference paths, and agent-conditioned route-feasibility labels.The first reference-path point supplies the path anchor; at inference, the VLM generates goal coordinates and mask tokens before the planner samples paths.Source: C. Joint Optimization · p. 3, §IV · p. 4

A. Simulation-Based Controllable Generation

  • Simulation varies road topology, obstacles, agent dimensions, and safety margins to derive traversable regions and A* reference paths, then projects supervision into camera images.Holding scene geometry and appearance fixed while changing the agent yields corresponding masks and route-feasibility annotations.Source: A. Simulation-Based Controllable… · p. 4, Figure 3 · p. 4
  • Photorealistic image generation changes simulated visual style, after which traversability-mask annotations are realigned to the generated images.The union of agent-specific masks conditions image generation while each agent keeps its own mask annotation.Source: A. Simulation-Based Controllable… · p. 4, Figure 3 · p. 4
  • Samples count agent–instruction planning tasks rather than unique images.Table I is identified as the training and test set composition table.Source: A. Simulation-Based Controllable… · p. 5, Table I · p. 5

B. Adaptation of Existing Visual Sources

Existing visual sources are adapted into agent- and language-conditioned planning supervision through a common annotation pipeline. The resulting dataset uses a unified task format and spans eight source groups.

B. Adaptation of Existing Visual Sources

  • ADE20K, FPV Crosswalk, CARLA, RoomTour3D, and EgoWalk are processed through a common pipeline that retains usable masks and annotates or corrects unsuitable ones.Qwen3.8-27B selects goals from candidate locations, and a region-constrained planner generates reference paths.Source: B. Adaptation of Existing Visual Sources · p. 5

C. Unified Format and Dataset Composition

  • The adapted samples pair images, agent descriptions, instructions, masks, goals, reference paths, and agent-conditioned route-feasibility labels.Existing scene annotations are extended with agent- and language-conditioned planning supervision; the simulation branch adds shared-scene agent variations.Source: C. Unified Format and Dataset… · p. 5
  • The dataset contains 639,343 training samples and 13,580 test samples across eight source groups, including 11,768 feasible and 1,812 infeasible test tasks.Samples are planning tasks, and the source mix covers indoor layouts, outdoor walking, crossings, simulated routes, and generated passages.Source: C. Unified Format and Dataset… · p. 5

V. EXPERIMENTS

The experiments evaluate region, goal, and route-feasibility predictions before conditional path generation and full-pipeline performance, then examine ablations, execution schedules, and robot deployment. Comparisons use a shared test set and baselines selected for the outputs each method supports.

V. EXPERIMENTS

  • The experimental sequence covers VLM predictions, conditional path generation, full-pipeline evaluation, ablations, synchronous versus asynchronous execution, and robot deployment.The ablations examine the Trav. Rep. interface, pixel-level segmentation supervision, and joint optimization.Source: §V · p. 5
  • Tables II and III are scoped to agent- and language-conditioned traversable regions and agent-conditioned route feasibility, respectively.The supplied table titles identify their evaluation topics but not cell values or comparative outcomes.Source: Table II · p. 5, Table III · p. 6

A. Experimental Setup

  • LaTraNav and the independent cascade share training data, while the cascade trains the VLM and planner separately before connecting them at inference.Other comparisons include original-weight UFO and LISA region predictors, RoboPoint goal localization, zero-shot InternVL2.5-8B route feasibility, and several path-planning baselines.Source: A. Experimental Setup · p. 5
  • The full-pipeline Strict A* comparison uses LaTraNav’s region and goal predictions and path anchor, but searches the decoded mask toward the predicted goal.Conditional planning is evaluated separately with ground-truth goals; Strict A* returns no path when endpoints lie outside the mask or are disconnected.Source: A. Experimental Setup · p. 5
  • All methods use the same 13,580 samples, and InternVL2.5-8B is evaluated without task-specific training.False-feasible rate is measured over the 1,812 ground-truth infeasible samples.Source: A. Experimental Setup · p. 6

B. Evaluation Metrics

The evaluation metrics distinguish region and goal prediction, route feasibility, conditional path similarity, path violations, and full-pipeline success. Full-pipeline success is measured across feasible tasks, while violation metrics describe generated paths.

B. Evaluation Metrics

  • Region prediction uses IoU and Dice on samples with ground-truth masks, while goal localization uses normalized-coordinate L2 error and SR@α on feasible tasks.Refusals and missing goals receive the unit-square maximum distance for Goal L2 and count as failures for SR@α.Source: B. Evaluation Metrics · p. 6
  • Conditional planning reports ADE between resampled predicted and reference paths, using ground-truth goals and each method’s generated paths.ADE is auxiliary because multiple acceptable paths may reach the same goal; full-pipeline evaluation does not supply the ground-truth goal.Source: B. Evaluation Metrics · p. 6
  • Path-violation metrics compare predicted and reference points outside the agent-specific mask, with excess violation defined as ΔPVR = PVRpred − PVRref.The evaluation also reports mean excess violation and RelTVR@0.05; metric subsets may differ between methods.Source: B. Evaluation Metrics · p. 6
  • Jointall measures full-pipeline success over all feasible tasks using path production, endpoint error below 0.15, and ΔPVR ≤ 0.05.Missing paths count as failures, and the endpoint criterion evaluates the generated path endpoint rather than the VLM-predicted goal.Source: B. Evaluation Metrics · p. 6

C. Region, Goal, and Route Feasibility Predictions

LaTraNav predicts agent- and language-conditioned regions, goals, and route feasibility, then evaluates whether those predictions support full-pipeline navigation. Its flow-matching planner achieves higher full-pipeline success than the compared alternatives, while conditional-planning results also examine generated-path quality.

C. Region, Goal, and Route Feasibility Predictions

  • LaTraNav reaches a traversability IoU of 0.8817 on 13,093 annotated samples, versus 0.311 for UFO and 0.332 for LISA with original weights.The independent cascade attains a similar IoU of 0.8864.Source: C. Region, Goal, and Route Feasibility… · p. 6
  • On the full feasible set, LaTraNav records Goal L2 0.155 and SR@0.15 76.8%, versus 0.312 and 25.8% for RoboPoint.The independent cascade scores Goal L2 0.142 and SR@0.15 79.1%.Source: C. Region, Goal, and Route Feasibility… · p. 6
  • LaTraNav reaches 96.40% route-feasibility accuracy and 92.99% macro-F1, versus 88.37% and 68.71% for zero-shot InternVL2.5-8B.Its false-feasible rate is 0.00%, compared with 65.89% on 1,812 annotated infeasible tasks.Source: C. Region, Goal, and Route Feasibility… · p. 6

D. Conditional Planning and Full-Pipeline Evaluation

  • With ground-truth goals, Strict A* generates paths for 3,054 tasks using UFO masks and 9,069 using LaTraNav masks; the cascade and LaTraNav generate 11,324 and 11,279 paths.ADEs are 0.0313 for the cascade and 0.0151 for LaTraNav on their generated subsets; ground-truth goals isolate conditional planning from goal-prediction error.Source: D. Conditional Planning and… · p. 6
  • Using LaTraNav's region and goal predictions, Strict A* reaches 51.6% Jointall and produces no path for 30.5% of feasible tasks.Its lower full-pipeline success reflects path-generation failures under disconnected masks or out-of-mask goals or anchors; generated-path quality and full-pipeline success are complementary measures.Source: D. Conditional Planning and… · pp. 6–7
  • 63.95% Jointall for LaTraNav exceeds 57.70% for the independent cascade, 28.1% for VENTURA, and 3.1% for VAMOS-VLM.LaTraNav's gain over the cascade occurs despite slightly lower standalone region and goal scores, supporting joint optimization's contribution to full-pipeline success.Source: D. Conditional Planning and… · p. 6, Table IV · p. 7
  • Fig. 4 shows region and goal predictions with generated paths across crossings, ramps, nighttime scenes, and narrow passages.Feasible-task rates in the full-pipeline evaluation use 11,768 samples.Source: D. Conditional Planning and… · p. 7

E. Structural and Training Ablations

Structural and training ablations test the Trav. Rep. interface, pixel-level segmentation supervision, and joint optimization. The results distinguish the contribution of latent conditioning from segmentation accuracy alone and show that segmentation supervision grounds the representation.

E. Structural and Training Ablations

  • At similar mask IoU, LaTraNav reaches 63.95% Jointall versus 59.83% with explicit-mask conditioning, improving success by 4.12 percentage points.With Trav. Rep. withheld, mask IoU remains 0.8802 but Jointall falls to 54.74%; mean ∆PVR changes from 0.0731 to 0.0544 for explicit-mask conditioning versus LaTraNav.Source: E. Structural and Training Ablations · p. 7
  • Removing pixel-level segmentation supervision reduces IoU to 0.2631 and Jointall to 59.47%, supporting its role in grounding Trav. Rep.The ablation table covers structural and training variants.Source: E. Structural and Training Ablations · p. 7
  • Compared with the independent cascade, LaTraNav reduces mean ∆PVR from 0.0895 to 0.0544 alongside a 6.25-point Jointall gain.The ablations support the Trav. Rep. interface, pixel-level segmentation supervision, and joint optimization as complementary components.Source: E. Structural and Training Ablations · p. 7

F. Asynchronous Execution Design

Asynchronous execution reuses cached semantic outputs while the flow-matching planner continues generating paths. The system is also deployed on wheeled and quadrupedal robots, and the paper concludes that Trav. Rep. serves as an interface between language understanding and adaptive visual navigation.

F. Asynchronous Execution Design

  • Asynchronous execution reuses the cached goal and Trav. Rep. for path generation while the next VLM inference runs concurrently.The planner combines those cached outputs with current visual features.Source: F. Asynchronous Execution Design · p. 8
  • Path-update rate rises 6.05×, from 1.21 to 7.32 Hz, while semantic-update rate remains 1.21 Hz.The semantic-update interval is approximately 0.83 s; planner latency increases from 63 to 86 ms, while reuse still allows multiple path updates between semantic updates.Source: F. Asynchronous Execution Design · p. 8

G. Real-Robot Deployment

  • LaTraNav is deployed on a wheeled AgileX RANGER MINI 3.0 and quadrupedal UNIUBI Cyvet, with VLM and planner inference on an external workstation.Image-space paths are converted to robot-frame waypoints using camera calibration and tracked by each platform's MPC controller.Source: G. Real-Robot Deployment · p. 8

VI. CONCLUSION

  • The conclusion presents Trav. Rep. as a reusable interface between language understanding and adaptive visual navigation, grounded by segmentation and path supervision.The reported experiments show improved full-pipeline success over explicit-mask conditioning and more frequent path updates under asynchronous execution.Source: §VI · p. 8
Loading 2610.11622v1…