PaperlayerView exact revision on arXiv

Source-linked AI summary

UDAV: Uncertainty-Driven Adaptive VLM Waypoint Planner

Ghazal Farhani, Shabnam Shabani

arXiv:2609.16368v1cs.ROcs.AI

TL;DR

Off-road VLM routes lack an indication of their reliability, and locally deployable VLM planning for aerially guided off-road navigation remains largely unexplored. UDAV uses stochastic trajectory medoids and spatial disagreement to trigger reconsideration. On 400 held-out queries, it reduces mean ADE by 25.1% relative to deterministic planning, while its reconsideration-stage benefit is not guaranteed for every uncertain query.

  • Problem

    Locally deployable VLM planning for aerially guided off-road navigation remains largely unexplored, and deterministic predictions provide no indication of route reliability.

    Source: §I · p. 1

  • Method

    UDAV uses stochastic trajectory samples for a medoid nominal route and geometric uncertainty estimation, triggering a reconsideration pass when uncertainty exceeds a threshold.

    Source: §I · p. 1

  • Results

    110.4 px mean ADE: UDAV reduces error by 25.1% relative to deterministic planning, with valid trajectories for all 400 queries and the lowest P90 and P95 errors among evaluated configurations, below K = 10 consensus.

    Source: §V · p. 7

  • Takeaways & Limitations

    Stochastic sampling improves nominal trajectory accuracy, while disagreement offers a selective failure-mitigation signal rather than guaranteeing improvement on every uncertain query.

    Source: §I · p. 1, §V · p. 7

    Evaluation covers only two flights from one dataset and environment type, and reconsidered trajectories were not executed in closed loop.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-language models (VLMs) can generate routes directly from aerial imagery for off-road navigation, but their predictions provide no indication of reliability. We present UDAV, an Uncertainty-Driven Adaptive VLM Waypoint Planner for UAV-guided UGV navigation. UDAV draws multiple stochastic trajectory predictions, selects their medoid as a self-consistent nominal route, and estimates predictive uncertainty from their spatial dispersion. When the maximum uncertainty across interior waypoints exceeds a threshold, UDAV invokes a reconsideration stage; otherwise, it returns the medoid directly. We evaluate UDAV on 400 held-out trajectory queries from two UAV flights. Stochastic medoid selection reduces the mean average displacement error (ADE) from 147.4 pixels for a deterministic prediction to 115.9 pixels. The complete planner achieves a mean ADE of 110.4 pixels, a 25.1% reduction relative to deterministic planning, while producing valid trajectories for all queries. UDAV also yields the lowest 90th- and 95th-percentile errors among all evaluated configurations, including a higher-budget K=10 consensus baseline. Relative to the K=5 medoid, UDAV reduces these errors from 225.3 and 326.0 pixels to 199.0 and 290.8 pixels, respectively. These results demonstrate that stochastic VLM predictions provide both a stronger nominal route and an actionable uncertainty signal for selectively mitigating large planning errors.

I. INTRODUCTION

The introduction frames aerially guided off-road planning as a setting where terrain ambiguity and limited route-reliability information challenge navigation. UDAV uses stochastic trajectory consensus and uncertainty-triggered reconsideration, and its evaluation reports improved accuracy over deterministic planning.

I. INTRODUCTION

  • Locally deployable VLM planning for aerially guided off-road navigation remains largely unexplored, especially when trajectory reliability must be judged before execution.Classical terrain-cost pipelines require predefined semantic classes, class-to-cost mappings, and environment-specific supervision, which the introduction describes as restrictive in unseen or variable terrain.Source: §I · p. 1
  • A single deterministic prediction provides no reliability signal, while stochastic disagreement may indicate when a self-consistent plan warrants reconsideration.The introduction reports that disagreement is associated with trajectory error and that medoid self-consistency improves accuracy over one deterministic prediction.Source: §I · p. 1
  • The evaluation uses 400 held-out trajectory queries from 193 spatially separated aerial observations with five candidate UGV paths each.The observations come from two flight sequences in the GA3T UAV–UGV off-road dataset [10], with 965 annotated path examples overall.Source: §I · p. 1
  • Fig. 1 sketches initial planning and uncertainty-triggered refinement, with the complete experimental prompts available in the reproducibility package.The introduction situates UAV–UGV collaboration as combining UGV local sensing and control with UAV global views and route guidance,.Source: §I · p. 1, Figure 1 · p. 2
  • 110.4 px mean ADE: UDAV reduces error by 25.1% relative to deterministic planning and produces valid trajectories for all 400 queries.UDAV also attains the lowest P90 and P95 errors among evaluated configurations, below a higher-budget K = 10 consensus baseline.Source: §I · p. 2
  • UDAV uses the medoid of stochastic predictions as its nominal route and invokes a second planning pass only when trajectory disagreement exceeds a threshold.The work also releases annotations, fine-tuned model weights, and data-preparation and reproduction instructions.Source: §I · pp. 1–2

II. RELATED WORK

Related work covers VLM navigation, off-road traversability estimation, and UAV–UGV collaboration. The paper distinguishes UDAV's direct waypoint generation from approaches that select supplied proposals, rely on terrain classes or prior traversal, or coordinate mission tasks.

A. Vision-Language Models for Navigation

  • PIVOT [11] and CoNVOI narrow externally supplied proposals rather than generating routes directly; CoNVOI's marker ablation raises unacceptable paths from 0% to 85.7%.CoNVOI uses cloud-hosted VLMs and ground-level imagery, and its authors identify the lack of a global top-down view as a limitation.Source: A. Vision-Language Models for Navigation · p. 2

B. Traversability Estimation for Off-Road Navigation

  • GA-Nav and Shaban et al. require predefined terrain classes and class-to-cost mappings, while WayFAST learns traversability from traction but presumes prior representative-terrain traversal.The classical approaches are also described as platform- and environment-specific.Source: B. Traversability Estimation for… · pp. 2–3

C. UAV–UGV Collaboration

  • Cladera et al. coordinate and sequence interdependent air–ground tasks, whereas this work asks the language model to generate the UGV waypoint sequence and explain its visual evidence.The cited work describes UAV–UGV sub-task relaying as supporting effective autonomous operation.Source: C. UAV–UGV Collaboration · p. 3

III. METHOD

UDAV samples trajectories from a fine-tuned VLM given aerial imagery and start and target locations. It uses the medoid of those samples as the nominal trajectory, preserving a generated route rather than averaging waypoint coordinates.

III. METHOD

  • The fine-tuned VLM receives the full aerial image, corridor crop, start, and target, and generates K stochastic trajectories of 16 ordered image-space waypoints.The experiments use K = 5.Source: §III · p. 3

A. Stochastic Consensus

  • The nominal trajectory is the medoid under the trajectory-similarity criterion, making it one of the VLM-generated trajectories rather than a pointwise average.This stochastic consensus supplies the self-consistent nominal route used by UDAV.Source: A. Stochastic Consensus · p. 3, §III · p. 3

B. Uncertainty-Triggered Reconsideration

UDAV estimates waypoint uncertainty from covariance across stochastic predictions and normalizes it. A threshold determines whether the nominal medoid is returned or a selected uncertain region is reconsidered.

B. Uncertainty-Triggered Reconsideration

  • The empirical covariance across stochastic predictions is computed for each interior waypoint, with the corresponding sample mean denoted µi.The resulting normalized uncertainty is used in the reconsideration decision.Source: B. Uncertainty-Triggered Reconsideration · p. 3
  • When maximum normalized uncertainty exceeds τ, UDAV selects the contiguous region with the largest uncertainty and prompts the model to reassess the terrain.A high-resolution crop of that region is optional; the core method does not require it.Source: B. Uncertainty-Triggered Reconsideration · p. 3
  • The reconsideration stage generates Kref trajectories conditioned on the uncertainty cue, and their medoid is used when maximum uncertainty exceeds τ.The cue is Cτ; crop I3 is included only in the crop-augmented variant, and the refined medoid uses the same trajectory-distance criterion.Source: B. Uncertainty-Triggered Reconsideration · p. 3

C. Dataset Overview

The study builds its evaluation around selected, visually distinct frames from two UAV flights, then specifies the VLM and classical planner used for comparison.

C. Dataset Overview

  • D1 contains 495 trajectories from 99 selected frames, while D2 contains 470 from 94; both include permanently held-out test frames.D1’s split is performed by blocks of 10 frames; D2 reserves 50 frames (250 trajectories) for testing and uses 44 frames (220 trajectories) for adaptation.Source: C. Dataset Overview · pp. 3–4
  • Manual annotations of scene objects and terrain features support segmentation training and construction of the obstacle-aware cost maps used by A∗.The trajectory annotations are available in the PathGuide annotation dataset.Source: C. Dataset Overview · p. 4

1) VLM model:

  • Among four initially evaluated VLMs, Qwen2.5-VL-7B and InternVL3-8B had the lowest waypoint errors, but Qwen2.5-VL-7B adapted more effectively and was used thereafter.Qwen2.5-VL-7B can also be deployed offline.Source: 1) VLM model · p. 4

D. A∗Planning

  • A∗ finds a minimum-cost path between start and goal while excluding non-traversable regions; its grid cost map encodes traversal costs and clearance margins.The comparison planner’s cost map is derived from semantic predictions; its segmentation model is trained on 113 non-test frames, with no permanent-test annotations used.Source: D. A∗Planning · p. 4

E. Metrics of Comparison

Trajectory comparisons use four image-space measures of geometric similarity, and the model is fine-tuned sequentially on spatially separated flight data.

E. Metrics of Comparison

  • Four image-space metrics assess trajectory geometry: ADE uses corresponding-point distances, MidADE excludes endpoints, Chamfer uses symmetric nearest neighbors, and Hausdorff measures the maximum nearest-neighbor mismatch.Lower Hausdorff values indicate greater geometric similarity.Source: E. Metrics of Comparison · p. 4

F. Fine-Tuning Setup

  • The model is adapted first on 70% of D1 samples, then on D2, using spatially separated training and test samples and BF16 computation on an NVIDIA A100 80 GB GPU.D1 adaptation runs for three epochs and D2 adaptation for one; the respective initial learning rates are 2×10−4 and 5 × 10−5.Source: F. Fine-Tuning Setup · p. 4

IV. RESULTS

Sequential fine-tuning improves trajectory accuracy across the two flights, while stochastic predictive dispersion tracks trajectory error and motivates selective reconsideration.

A. Sequential Fine-Tuning and Cross-Flight Generalization

  • D1 fine-tuning lowers D1 ADE from 241.8 px to 149.3 px and transfers to unseen D2, reducing its ADE from 218.8 px to 194.7 px.The D1-adapted model was not exposed to D2 during training; a noticeable domain gap remains.Source: A. Sequential Fine-Tuning and… · p. 4

B. Stochastic Consensus and Uncertainty as a Reliability Signal

  • ρ = 0.557 on D1 and ρ = 0.661 on D2: K = 5 predictive dispersion increases across trajectory-error quintiles, with both associations statistically significant.The plotted relationship is positive in both flights.Source: Figure 2 · p. 5, B. Stochastic Consensus and… · p. 5

A. Sequential Fine-Tuning and Cross-Flight Generalization

  • 147.4 px combined ADE follows sequential D1→D2 adaptation, down from 177.8 px, with no evidence of catastrophic forgetting in the evaluated setting.D2 adaptation reduces D2 ADE from 194.7 px to 156.8 px, a 19.5% improvement over the D1-only model.Source: A. Sequential Fine-Tuning and… · pp. 4–5

B. Stochastic Consensus and Uncertainty as a Reliability Signal

  • K = 5 sampling raises mean latency from 12.95 s to 15.57 s, about 20%; K = 10 requires 18.51 s, motivating K = 5 as the default budget.Multiple trajectories are generated within the same inference call, so latency grows sublinearly with sample count.Source: B. Stochastic Consensus and… · p. 5
  • Predictive disagreement provides an online signal for deciding when an otherwise self-consistent plan warrants additional reasoning.The reported association between dispersion and error supports using uncertainty to trigger reconsideration rather than define the nominal route.Source: B. Stochastic Consensus and… · p. 5

C. Consensus Planning and Uncertainty-Guided Reconsideration

Stochastic medoids improve nominal trajectory accuracy, while UDAV selectively reconsiders high-uncertainty plans. At τ = 0.6, it has lower tail errors than the higher-budget K = 10 medoid, though reconsideration is not guaranteed to help every query.

C. Consensus Planning and Uncertainty-Guided Reconsideration

  • The K = 5 medoid reduces mean ADE from 147.4 px for a deterministic prediction to 115.88 px and produces valid trajectories for every query.The deterministic prediction is valid for 398 of 400 queries.Source: C. Consensus Planning and… · p. 5, Table III · p. 6
  • UDAV uses a K = 5 ensemble to form the nominal medoid and decide whether more reasoning is needed; at τ = 0.6, reconsideration activates for 174 of 400 queries (43.5%).The remaining 56.5% return the initial stochastic medoid directly.Source: C. Consensus Planning and… · p. 5
  • At τ = 0.6, UDAV’s mean ADE is 110.36 px versus 108.58 px for K = 10, but its P90/P95 are lower (198.98/290.78 vs 204.92/292.01 px).These are full-test-set results over 400 queries; Table III summarizes the comparison.Source: C. Consensus Planning and… · p. 5, Table III · p. 6
  • On the 174 triggered queries, mean ADE falls from 153.71 px to 141.01 px, but reconsideration improves only 50.0% of them and is not a guaranteed per-query gain.The paired one-sided Wilcoxon test is not significant (p = 0.335), and the bootstrap interval for the all-query mean improvement includes zero.Source: C. Consensus Planning and… · p. 5
  • At τ = 0.3, 71.0% of queries are escalated yet mean ADE is 115.03 px; thresholds above approximately 1.0 trigger too few reconsiderations for consistent benefit.The threshold was selected from the sweep rather than calibrated on a separate validation split; mean ADE varies by less than 4 px across τ ∈ [0.5, 0.8], while tail percentiles are more sensitive.Source: C. Consensus Planning and… · p. 5
  • Qualitative examples show that reconsideration’s benefit is concentrated in difficult planning cases rather than distributed uniformly across the test set.In the examples, stochastic predictions disagree in locally ambiguous regions and the second pass modifies the route.Source: C. Consensus Planning and… · pp. 5–6

D. Visual Re-observation Ablation

The visual re-observation ablation compares reconsideration with no added image, a broad crop, and a crop localized to peak uncertainty. The localized crop has the lowest mean ADE, but crop variants are not significantly different from reconsideration without a crop.

D. Visual Re-observation Ablation

  • At τ = 0.6, 174 of 400 test queries (43.5%) trigger reconsideration, while the crop ablation itself examines 143 queries at τ = 0.8.The figure reports the distribution of maximum normalized interior-waypoint uncertainty over the D1 and D2 permanent test sets.Source: Figure 3 · p. 6, D. Visual Re-observation Ablation · p. 6
  • At τ = 0.8, peak-local crops yield 138.0 px mean ADE, versus 144.6 px without an added image and 150.3 px with a broad crop.The comparison uses 143 queries triggered at the stricter threshold τ = 0.8.Source: D. Visual Re-observation Ablation · p. 6
  • Neither crop variant differs significantly from reconsideration without a crop under paired Wilcoxon tests, so crop design remains a secondary component.The authors suggest localized visual re-observation can provide useful terrain detail, whereas broad re-observation does not necessarily improve the decision.Source: D. Visual Re-observation Ablation · p. 6
  • Qualitative examples show stochastic disagreement identifying difficult regions where an additional planning pass can modify the route toward the human-annotated reference.Source: Figure 4 · p. 7

E. Comparison with the Physically Executed UGV Trajectory

This comparison measures how closely VLM and other planning representations match the physically executed UGV path. Its results are contextual rather than a closed-loop evaluation of UDAV because the tested physical route never triggered reconsideration.

E. Comparison with the Physically Executed UGV Trajectory

  • None of the PathGuide 1 queries triggered reconsideration, so this physical-path experiment evaluates the underlying VLM representation, not closed-loop UDAV.PathGuide 1 corresponds to the UGV's actual forward route during data collection.Source: E. Comparison with the Physically… · p. 7
  • Many high-uncertainty examples require lateral, T-turn, or return maneuvers absent from the physically executed forward route.These maneuvers are not represented by PathGuide 1.Source: E. Comparison with the Physically… · p. 7
  • On combined data, Qwen achieves 131.2 px ADE, versus 124.2 px for human PathGuide and 146.7 px for SegFormer+A∗; on D2, Qwen has the lowest ADE and Chamfer distance.The comparison evaluates similarity to the physically executed UGV trajectory.Source: E. Comparison with the Physically… · p. 7
  • Fig. 5 shows two scenes, D1 and D2, comparing human annotations, Qwen, SegFormer+A∗, and executed paths reconstructed from consecutive UAV frames.Source: Figure 5 · p. 7

V. CONCLUSION

UDAV uses stochastic trajectory samples for both a self-consistent nominal route and geometric uncertainty, with uncertainty activating a second planning pass. On the evaluated test set, it improves mean ADE and reduces high-percentile errors, while the evaluation remains limited in scope and lacks closed-loop execution.

V. CONCLUSION

  • The K = 5 stochastic medoid reduces mean ADE from 147.4 px for a deterministic generation to 115.9 px over the complete test set.Source: §V · p. 7
  • At τ = 0.6, UDAV achieves 110.4 px mean ADE, a 25.1% reduction relative to deterministic planning, and the lowest P90 and P95 errors among evaluated configurations, including K = 10 consensus.It returns valid trajectories for all 400 queries, with an average budget of 7.2 sampled trajectories per query rather than 10.Source: §V · p. 7
  • The physical-path comparison is contextual: the reconsideration trigger did not activate on the physically executed PathGuide 1 routes.The comparison also considers human annotations and a SegFormer+A∗ planner; similarity to one human annotation is not the only meaningful measure of planning quality.Source: §V · p. 7
  • The evaluation covers only two flights from one dataset and environment type, and reconsidered trajectories were not executed in closed loop.Future work will evaluate more diverse terrain and deploy the complete policy in closed-loop UGV execution.Source: §V · p. 8

Related papers

Loading 2609.16368v1…