Source-linked AI summary
From Language Priors to Field Adaptation: Preference Learning for Traversability Estimation
Simon Schwaiger, David Seyser, Alessandro Scherl, Zlatan Ajanović, Wilfried Wöber, Gerald Steinbauer-Wagner
TL;DR
Traversability preferences vary across robots and domains, making target-domain adaptation from limited annotations important. The paper learns language-initialised vMF prototypes in a frozen vision-language space and adapts them with relative image annotations. Its WayFAST results are competitive with end-to-end estimators, while language initialization supports sample-efficient adaptation.
Problem
Traversability costs vary by robot and domain, while relative-label methods trade accuracy against adaptation and lack initialization from generic open-world knowledge.
Source: §I · p. 1, §II · p. 2Method
The method represents traversability with vMF prototypes in a shared frozen VLM space, initialized from relative language rules and adapted using relative image annotations.
Source: §I · p. 1, §III · p. 2, A. Shared Vision-Language Embedding · p. 2Results
The fine-tuned estimator achieves accuracy competitive with the state of the art, and language initialization improves sample efficiency.
Source: §V · p. 8Takeaways & Limitations
The estimator can operate on VLM semantic maps by predicting traversability at map query points, in addition to image patches.
Source: D. Dense Traversability Inference · p. 4
Abstract
from arXiv · showhide
Image-based traversability estimation is inherently dependent on the robot platform, deployment domain, and mission preferences, which limits the applicability of purpose-trained models. To facilitate domain adaptation, this work aims to reduce the number of required annotations in the target domain using sample-efficient preference learning. Our method represents traversability through von Mises-Fisher mixture prototypes in a frozen vision-language feature space. Relative natural-language rules provide a commonsense prior, while sparse relative image annotations adapt the prototype directions and utilities to a target domain through computationally and sample-efficient fine-tuning. Experiments on WayFAST demonstrate accuracy competitive with end-to-end trained estimators while enabling sample-efficient image-based adaptation. Qualitative experiments further demonstrate the language prior's zero shot applicability and the fine-tuned estimator's improved dense prediction on semantic maps. Evaluation is complemented via semantic interpretation of learned prototypes by dissecting semantically close natural language prompts. Code and trained estimators available at https://resireg.github.io
I. INTRODUCTION
Traversability depends on robot, domain, and mission, while existing estimation approaches require robot experience, platform-specific annotations, or expensive computation. This paper uses language-initialised prototypes and relative image annotations to support zero-shot use and sample-efficient adaptation.
I. INTRODUCTION
- Terrain with similar geometry can have different traversability costs across robots and application domains, so a single ground truth is not generally applicable.Existing approaches rely on robot experience, platform-specific annotations, or computationally expensive broad predictions.Source: §I · p. 1
- The method reuses a frozen VLM feature space and models traversability with vMF mixture responsibilities over prototypes learned from relative language and image annotations.Language provides a platform-agnostic prior that can bootstrap image-based fine-tuning.Source: §I · p. 1
- With the same annotation amount, prediction accuracy is competitive with end-to-end trained models while offering language-based initialisation and sample-efficient learning.The experiments also evaluate zero-shot and data-efficient adaptation, fine-tuning accuracy, and alignment of learned prototypes with language.Source: §I · pp. 1–2
II. BACKGROUND
Traversability has no domain-independent ground truth because costs vary with terrain, robot, and mission. Relative labels offer flexibility but discard magnitude information, leaving a trade-off between accuracy, adaptation, and open-world prior knowledge.
II. BACKGROUND
- Traversability cost varies with terrain, robotic embodiment, and application domain, so no training data source supplies ground truth across domains.The estimated cost informs terrain-dependent path and motion planning.Source: §II · p. 2
- Relative annotations preserve ordering across arbitrary absolute scales, but lose information about how much easier one region is, making sparse supervision harder to learn from.The literature balances this flexibility against information loss through end-to-end training or few-shot adaptation.Source: §II · p. 2
- Preference learning fits traversability because it learns from relative feedback without requiring an objective traversability scale.The introduced method combines this training objective with a commonsense language prior and sample-efficient domain fine-tuning.Source: §II · p. 2
III. METHOD
The method learns traversability preferences from relative image and language annotations in a shared frozen vision-language embedding space. Language supervision initializes the prototype bank, which is then adapted to image-domain preferences for dense inference.
III. METHOD
- A relative ranking loss fits per-prototype traversability utilities, prototype directions, and shared concentration for a vMF mixture, which estimates dense traversability from query features.The overview shows cached language and image annotation pairs feeding prototype learning and per-patch or per-point inference.Source: §III · p. 2, B. Prompt Learning · p. 2, Figure 2 · p. 3
A. Shared Vision-Language Embedding
- A frozen VLM encodes image regions and natural-language prompts into a shared feature space, with image features sampled at annotated pixel locations.Relative supervision uses tuples (xA, xB, r), where r indicates which region is more traversable or whether they are tied.Source: A. Shared Vision-Language Embedding · p. 2
B. Prompt Learning
- Language training fixes prototype directions at spherical k-means centroids and fits utilities and concentration; after entropy-matched concentration transfer, image fine-tuning unfrees directions for domain adaptation.The language directions are anchored because the comparison graph does not constrain all prototype directions; this preserves similarity-based assignment to language concepts.Source: B. Prompt Learning · p. 3
C. Latent Prompt Bank
The latent prompt bank represents traversability with visual prototypes and bounded utilities in the VLM feature space. Shared-concentration vMF responsibilities assign query features softly to prototypes, whose utilities determine patch-level traversability.
C. Latent Prompt Bank
- The prompt bank pairs normalized visual prototypes with bounded traversability utilities in the VLM feature space.The prototypes serve as prompts for downstream traversability queries.Source: C. Latent Prompt Bank · p. 3
- A shared concentration controls assignment sharpness, while vMF responsibilities softly weight prototypes by query-feature cosine similarity.The assignments combine with learned per-prototype utilities to retrieve patch traversability.Source: C. Latent Prompt Bank · pp. 3–4
D. Dense Traversability Inference
The estimator predicts traversability from individual VLM feature vectors, applying it patchwise for dense image inference. It can also run on VLM embeddings at semantic-map query points.
D. Dense Traversability Inference
- For dense image inference, the estimator scores each VLM patch embedding and bilinearly upsamples the resulting traversability map to pixel resolution.The estimator predicts an absolute value in [0, 1] from each feature vector and can be queried during field fine-tuning.Source: D. Dense Traversability Inference · p. 4
- On semantic maps, the estimator operates on VLM embeddings at map query points rather than processing the whole image.This supports deployment beyond end-to-end whole-image processing.Source: D. Dense Traversability Inference · p. 4
IV. EXPERIMENTS
The experiments evaluate zero-shot application, dataset transfer, and WayFAST fine-tuning, using relative human annotations and a common frozen VLM encoder. Comparisons include task-specific baselines and capacity-matched prediction heads, with a nested-subset experiment assessing fine-tuning sample efficiency.
IV. EXPERIMENTS
- WayFAST evaluation uses human disagreement rate on relative traversability labels, with the same frozen encoder across VLM predictors.HDR is reported at thresholds 0.1, 0.25, and 0.5; higher thresholds treat smaller score differences as equally traversable.Source: §IV · p. 4
- Evaluation covers zero-shot target-domain performance, RUGD-to-WayFAST dataset transfer, and fine-tuned performance on the WayFAST training set.The zero-shot setting evaluates the generic commonsense language prior without target-domain training or tuning.Source: §IV · p. 4
- Comparators include a statically prompted VLM, a transfer baseline, the end-to-end relative-annotation SOTA, and linear, MLP, and transformer heads.The MLP and transformer heads are parameter-matched to the proposed estimator for fine-tuning comparisons.Source: §IV · p. 4
- Language-prior configurations start from generic relative traversability rules that are aligned with the target domain during image-based fine-tuning.The vocabulary contains 300 inequality samples and 7 equality samples, and is also used for the statically prompted baseline.Source: §IV · pp. 4–5
- The language-prior estimator lies lower-left than the compared models in the WayFAST fine-tuning sample-efficiency plot.Configurations use approximately 8.2k trainable parameters and identical nested image subsets across five seeds; shading shows 95% confidence intervals.Source: Figure 3 · p. 5
A. Human Disagreement Traversability Estimation
On fine-tuned WayFAST results, language-prior initialization wins at the two higher HDR thresholds, where clearer distinction between semantic traversability modes is required. Across the reported settings, zero-shot performance depends on the HDR threshold, and dataset-transfer outcomes differ by threshold and initialization.
A. Human Disagreement Traversability Estimation
- In zero-shot evaluation, the static prompt wins HDR0.1, while the language prior wins the other reported thresholds.The outcome depends on the HDR threshold.Source: A. Human Disagreement Traversability… · p. 5
- For RUGD-to-WayFAST transfer, random prototype initialization slightly improves HDR0.1 and HDR0.25, while language-prior initialization improves HDR0.5 and surpasses the baseline.The reported transfer baseline uses an approximately 49.4k-parameter head, compared with approximately 8.2k for the proposed estimator.Source: A. Human Disagreement Traversability… · p. 5
- Language-prior initialization wins at the two higher HDR thresholds in fine-tuned WayFAST evaluation, where clearer semantic-mode distinction is required.The comparison is against end-to-end model, the benchmark SOTA trained on the same dataset with the same loss; random initialization also measurably improves over linear probing.Source: A. Human Disagreement Traversability… · p. 5, Table I · p. 5
B. Language-Guided Adaptation Across Annotation Budgets
Language-prior initialisation improves downstream accuracy at low training-sample sizes for the stricter HDR thresholds. The budget evaluation compares this approach with randomly initialised and parameter-matched heads.
B. Language-Guided Adaptation Across Annotation Budgets
- Fig. 4 compares language-prior and WayFAST-adapted predictions for RoboNav scenes on image features and reconstructed 3D maps using the same [0, 1] scale.Higher values indicate greater estimated traversability.Source: Figure 4 · p. 6
- At HDR0.25 and HDR0.5, the language-prior initialised vMF mixture improves downstream accuracy over other configurations at low training-sample sizes.Configurations without a language prior show similar sample efficiency; HDR0.1 results have greater spread across runs.Source: B. Language-Guided Adaptation Across… · p. 6
- Nested training subsets range from 2 to 8192 images plus the complete 13,248-image split, with identical subsets and minibatch orders across methods.Each disjoint image pair provides three relative comparisons, so an N-image budget corresponds to 3N/2 annotations; results average five seeds.Source: B. Language-Guided Adaptation Across… · pp. 5–6
C. 3D Semantic Map Queries
The estimator can query language-embedded 3D maps without additional training or scene-specific adaptation. On RoboNav map excerpts, fine-tuning yields clearer traversability distinctions than the language prior.
C. 3D Semantic Map Queries
- Because s(x) predicts directly from a query VLM feature, the estimator can perform traversability estimation on language-embedded 3D maps.The language-initialised estimator is applied without additional training or scene-specific adaptation to a map assembled from RoboNav data.Source: C. 3D Semantic Map Queries · p. 6
- The aerial semantic map uses 240×424-pixel orthophoto tiles resized to 480×848 for VLM inference, then interpolates a 4-metre feature grid to 1-metre resolution before grid-wise estimation.Source: C. 3D Semantic Map Queries · p. 6
- On semantic-map excerpts, the fine-tuned estimator pushes untraversable obstacles such as trees toward 0 and captures a traversability gradient across street, grass, and obstacles.The language prior shows less clear separation between the most and least traversable areas.Source: C. 3D Semantic Map Queries · p. 6
D. Ablations
The ablations examine prototype count, language-prior vocabulary, and the semantic interpretation of learned prototypes. They report a capacity–speed tradeoff, vocabulary-dependent accuracy patterns, and both plausible and ambiguous prototype descriptions.
D. Ablations
- Prototype-count sensitivity is low overall, with differing top scores across HDR thresholds; K = 16 is chosen for its capacity–processing-speed tradeoff and diminishing returns from more prototypes.The highest scores in the dataset-transfer evaluation arise at K = 100.Source: D. Ablations · pp. 6–7, Table II · p. 7
- The 73-description vocabulary achieves the lowest HDR at all three thresholds despite only 307 comparisons, while larger nested vocabularies show different patterns across thresholds.The authors conclude that description number, composition, and balance seem important to downstream accuracy, and suggest tailoring the vocabulary to an expected use case.Source: D. Ablations · p. 7, Table III · p. 7
- CLIP-Dissect descriptions show plausible score associations, including high scores for asphalt and soil and low scores for retaining wall and building, but also semantic inconsistencies.Ditch receives a high score, and prototype 4 shifts from sky to motorcycle after fine-tuning; ambiguity in the frozen language-embedding space may contribute.Source: D. Ablations · pp. 7–8, Figure 5 · p. 7
V. CONCLUSION
The paper presents language-initialised vMF prototypes in a frozen vision-language space for bootstrapping domain-specific traversability fine-tuning. Experiments report improved sample efficiency, competitive accuracy, and applicability to language-embedded 3D maps.
V. CONCLUSION
- The estimator represents terrain preferences as language-initialised vMF prototypes in a frozen vision-language feature space to bootstrap domain-specific fine-tuning.Quantitative experiments show improved sample efficiency and competitive accuracy against the state of the art; qualitative results demonstrate 3D semantic-map applicability and examine prototype semantics.Source: §V · p. 8