PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Traversability Estimation for Wild Visual Navigation

Jonas Frey, Matias Mattamala, Nived Chebrolu, Cesar Cadena, Maurice Fallon, Marco Hutter

arXiv:2305.08510v2cs.ROcs.CVcs.LG

TL;DR

Natural terrain such as high grass can be mistaken for rigid obstacles, and labeled training data may be difficult to collect or poorly matched to robot capabilities. WVN learns traversability online from vision and short human demonstrations, using self-supervised visual features and robot-generated supervision. The authors report adaptation within minutes and closed-loop navigation in natural scenes, including a 1.4 km path-following experiment.

  • Problem

    High grass and other natural growth can be misclassified as untraversable by geometry-based systems, while labeled datasets and fixed class labels are difficult to collect and may not match robot capabilities.

    Source: §I · p. 1

  • Method

    WVN combines online self-supervised traversability learning from pretrained visual-transformer features with supervision generated from vision and robot traversability measurements.

    Source: §I · p. 2

  • Results

    WVN adapted within minutes and achieved closed-loop navigation in natural scenes, including a 1.4 km path-following experiment.

    Source: §I · p. 2, §VI · p. 14

  • Takeaways & Limitations

    The authors conclude that WVN can support autonomous navigation by learning from small demonstrations in new environments.

    Source: §VI · p. 14

    WVN runs at 2.5 Hz on an NVIDIA Orin board with unoptimized code.

    Source: D. Limitations · p. 13

Abstract

from arXiv · show

Natural environments such as forests and grasslands are challenging for robotic navigation because of the false perception of rigid obstacles from high grass, twigs, or bushes. In this work, we propose Wild Visual Navigation (WVN), an online self-supervised learning system for traversability estimation which uses only vision. The system is able to continuously adapt from a short human demonstration in the field. It leverages high-dimensional features from self-supervised visual transformer models, with an online scheme for supervision generation that runs in real-time on the robot. We demonstrate the advantages of our approach with experiments and ablation studies in challenging environments in forests, parks, and grasslands. Our system is able to bootstrap the traversable terrain segmentation in less than 5 min of in-field training time, enabling the robot to navigate in complex outdoor terrains - negotiating obstacles in high grass as well as a 1.4 km footpath following. While our experiments were executed with a quadruped robot, ANYmal, the approach presented can generalize to any ground robot.

I. INTRODUCTION

WVN learns robot-specific traversability online from brief human demonstrations, addressing the difficulty of judging natural terrain from geometry and the limits of fixed semantic labels. The paper reports fast adaptation and closed-loop navigation in forests and parks, including kilometer-scale path following.

I. INTRODUCTION

  • Natural growth such as high grass can appear to 3D occupancy-based systems as an untraversable obstacle even when the robot can pass through it.Traversability depends on the robot's capabilities, so semantic understanding is important for judging passability.Source: §I · p. 1
  • Existing semantic-segmentation and anomaly-detection approaches face costly data collection, while fixed class labels do not directly represent a robot's capabilities.Earlier self-supervised methods still required robot-specific datasets and retraining for a new environment or platform.Source: §I · pp. 1–2
  • WVN combines online self-supervised adaptation with real-time supervision generation from vision, proprioceptive traversability measurements, and control performance.The system learns after a few minutes of manual demonstrations in the field.Source: §I · p. 2
  • WVN uses high-dimensional features from pretrained self-supervised vision transformers and trains a small neural network online.The authors also integrate traversability estimates with local mapping and planning for autonomous navigation tasks.Source: §I · p. 2
  • WVN adapted within minutes and supported closed-loop forest navigation and kilometer-scale park path following, including a path whose semantic class was unlabeled.The reported experiments used ANYmal; the authors state that the principles can apply to other ground platforms.Source: §I · p. 2

A. Traversability from geometry

  • Geometry-only methods can model robot-specific traversability, but the authors argue they do not succeed in natural environments with high grass, branches, or bushes.Their approach instead focuses on vision to represent semantic features that are difficult to capture otherwise.Source: A. Traversability from geometry · p. 2

B. Traversability from semantics

The related work motivates vision-based traversability, online self-supervision, and adaptation from demonstrations as responses to limits of fixed semantic labels and robot-specific training. WVN builds on these directions with pretrained visual-transformer features and online learning.

B. Traversability from semantics

  • Semantic-navigation methods assign costs to classes, but they often rely on labeled or fine-tuned models whose class definitions can be difficult to reuse across environments.Self-supervised visual models such as DINO-ViT can produce semantically meaningful segmentation without manual supervision.Source: B. Traversability from semantics · p. 2

C. Traversability from self-supervision

  • Prior self-supervised approaches derive labels from robot visits, footholds, sounds, or MPC trajectories, but supervision remains tied to hardware and adaptation can take substantial time.TerraPN is reported to take 25 minutes including data collection and learning; WVN aims for concurrent supervision generation and learning.Source: C. Traversability from self-supervision · pp. 2–3

D. Traversability from anomalies

  • Anomaly detection models traversed terrain and treats out-of-distribution inputs as untraversable; WVN instead uses anomaly detection as a confidence metric for sparse traversability supervision.Source: D. Traversability from anomalies · p. 3

E. Traversability from demonstrations

  • Inverse reinforcement learning learns rewards from demonstrations; the paper frames demonstrated terrain choices as preferences for navigating particular areas.The system overview describes monocular RGB and proprioceptive inputs being processed for online learning and traversability inference.Source: E. Traversability from demonstrations · p. 3, Figure 2 · p. 3

F. Adaptive traversability estimation

  • Earlier adaptive systems used clustering, binary classifiers, or superpixels, while WVN extends online learning with pretrained visual-transformer semantic features for varied natural environments.BADGR adapted from experience but was not framed as online learning.Source: F. Adaptive traversability estimation · p. 3

III. METHOD

WVN estimates dense traversability from RGB images using online self-supervision from brief human demonstrations. It combines segment-level DINO-ViT features with scores derived from how closely the robot tracks a reference velocity command.

A. System Overview

  • WVN targets dense traversability prediction from RGB images, with labels generated by a robot interacting with its environment and only a brief human demonstration for data collection and learning.Source: A. System Overview · pp. 3–4

B. Feature Extraction

  • Each image is encoded with dense DINO-ViT features, then SLIC divides it into 100 segments whose features are averaged into per-segment embeddings.The embeddings are used to estimate a traversability score for each segment.Source: B. Feature Extraction · p. 4, Figure 3 · p. 4

C. Traversability Score Generation

  • Traversability scores derive from the discrepancy between the robot’s current velocity and an external reference command: closer tracking indicates more traversable terrain.The velocity error is smoothed with a 1-D Kalman filter before a sigmoid maps it to a score; sigmoid parameters can be calibrated to platform motion specifications.Source: C. Traversability Score Generation · p. 4, Figure 4 · p. 4

D. Supervision and Mission Graphs

WVN generates supervision online by linking recent footprint and traversability measurements to mission images. A separate learning process uses the resulting segment labels to estimate traversability and uncertainty.

D. Supervision and Mission Graphs

  • Unlike post-processing approaches, WVN generates supervision online using a short-horizon Supervision Graph and a global Mission Graph that retains training data over the mission.The Supervision Graph keeps temporary footprint information in a sliding window; the Mission Graph stores data required for online learning.Source: D. Supervision and Mission Graphs · pp. 4–5, Figure 5 · p. 5

1) Supervision Graph:

  • The Supervision Graph stores a footprint track with traversability scores, generating supervision signals in hindsight for reprojection onto earlier camera views.Its ring buffer retains a fixed number of nodes, with the product of node count and spacing setting its maximum physical length.Source: 1) Supervision Graph · p. 5

2) Mission Graph:

  • Mission nodes retain RGB images, weak segmentation masks, per-segment features, and traversability supervision for online learning.A node is added when the robot has moved farther than the specified distance from the last added node.Source: 2) Mission Graph · p. 5

3) Graph interaction for supervision generation:

  • When a mission node is added, the graphs reproject recent footprint scores onto nearby mission images, averaging scores within segments and labeling segments outside the footprint as untraversable.Restricting reprojection to nodes within the Supervision Graph’s range keeps the transferred information locally consistent despite potential state-estimator drift.Source: 3) Graph interaction for supervision… · p. 5

E. Traversability and Anomaly Learning

  • The online learner maps segment features to traversability scores and uses anomaly detection to estimate confidence for unvisited, unlabeled areas.The formulation also addresses nonstationary data distributions as training data and model weights are continuously updated.Source: E. Traversability and Anomaly Learning · p. 5

1) Confidence Estimation:

The method estimates confidence from how well an encoder-decoder reconstructs traversed segment features, then trains a traversability predictor using confidence-weighted supervision. It also selects a traversability threshold from self-supervised ROC performance.

1) Confidence Estimation:

  • An encoder-decoder learns to reconstruct labeled traversed-segment features; low reconstruction loss indicates familiar features, while unobserved anomalous features tend to incur high loss.The reconstruction-loss distribution becomes multimodal during online training; a Gaussian fit to traversed-segment losses supplies a confidence estimate, with confidence set to 1 below the positive-loss mean and otherwise scaled by likelihood.Source: 1) Confidence Estimation · pp. 5–6

2) Traversability Estimation:

  • The traversability predictor regresses labeled segment scores, assumes unlabeled segments are untraversable, and weights their loss by confidence so familiar areas contribute less than anomalous ones predicted as traversable.The method selects a traversability threshold by computing an ROC during training and setting the desired false-positive ratio.Source: 2) Traversability Estimation · p. 6, Figure 6 · p. 6

3) Implementation details:

Online training combines traversability and reconstruction losses in a shared network, then feeds visual predictions into local terrain mapping for navigation. The implementation evaluates alternative learning designs and fuses projected predictions into a robot-centric map.

3) Implementation details:

  • The network uses shared hidden layers with traversability and reconstruction heads, and trains per segment with a weighted sum of both losses.The traversability head ends in a sigmoid; the reconstruction head has E outputs.Source: 3) Implementation details · p. 6
  • The implementation compares learning methods, features, architectures, and loss formulations, including anomaly detection and regression without uncertainty modeling.Source: 3) Implementation details · p. 7

IV. CLOSED-LOOP INTEGRATION

  • The learned traversability is integrated into a standard navigation pipeline, and Fig. 7 presents the network architecture and per-segment online losses.Source: §IV · p. 7, Figure 7 · p. 7

A. Local terrain mapping

  • Visual traversability is raycast into a robot-centric 2.5D map, accounting for terrain occlusions and allowing temporal fusion while preserving prior values through exponential averaging.The projection can inherit artifacts from spikes in the elevation map.Source: A. Local terrain mapping · p. 7

B. Local planning

The system turns predicted traversability into local planning commands and a simple forward exploration behavior. On hardware, it adapts during teleoperated loops, improving its segmentation over training steps.

B. Local planning

  • The local planner filters the traversability costmap, computes a Signed Distance Field, and uses it with the local goal to command locomotion around untraversable terrain.The twist command is sent to a robust learning-based controller for rough terrain.Source: B. Local planning · p. 7

C. Smart carrot for autonomous exploration

  • A smart carrot selects the farthest-obstacle-clear position in front of the robot and updates it as the SDF is recomputed, enabling footpath following without a global plan.The authors note that a more sophisticated exploration planner could improve this simple strategy.Source: C. Smart carrot for autonomous… · p. 7

A. Platform Description

  • WVN ran onboard an ANYmal C with an added Jetson Orin AGX; monocular wide-FoV color images were its main input, while LiDAR and depth served local terrain mapping.Source: A. Platform Description · p. 7

B. Real-world deployments

1) Fast adaptation on hardware:

  • After 800 training steps (2 min), the system correctly segmented dirt as traversable while keeping trees untraversable, improving from poor segmentation after 9 steps (21 s).Across later iterations, it extended predictions across additional grass patches; a wooden-wall segment was once clustered with grass.Source: 1) Fast adaptation on hardware · pp. 7–8, Figure 8 · p. 8

2) Benefits of visual traversability vs geometric methods:

Visual traversability classified high grass as passable where geometry-based methods treated elevation spikes as obstacles. In a woodland test, the robot reached all eight deliberately challenging goals after two minutes of teleoperation.

2) Benefits of visual traversability vs geometric methods:

  • WVN classified high-grass patches as traversable, unlike geometric methods that treated elevation spikes as obstacles, matching the human operator’s demonstrated assessment.Fig. 9 compares traversability maps and corresponding SDFs; visual estimates were less affected by grass and branch spikes.Source: 2) Benefits of visual traversability… · p. 8, Figure 9 · p. 9

3) Point-to-point autonomous navigation between trees:

4) Kilometer-scale autonomous navigation in the park:

After less than two minutes of footpath training, WVN followed park paths for hundreds of meters across three runs, though muddy patches and intersections sometimes required intervention. Offline ablations used three held-out outdoor datasets and operator-derived binary labels for assessment.

4) Kilometer-scale autonomous navigation in the park:

  • Across three park runs, the robot followed footpaths for hundreds of meters, mostly centered and avoiding nearby obstacles; muddy visually similar areas and intersections sometimes required manual correction.Run distances were 0.55 km, 0.5 km, and 1.4 km; the less-conservative run 2 settings led toward muddy grass requiring heading correction.Source: 4) Kilometer-scale autonomous… · pp. 9–10, Figure 11 · p. 10
  • A short teleoperated gravel-footpath demonstration supplied cues for desired path following, where geometric boundaries were often indistinguishable and semantic segmentation would have required learning multiple terrain classes.Source: 4) Kilometer-scale autonomous… · p. 10

C. Offline validation via ablation studies

1) Dataset overview:

  • Test labels were manually divided into traversable and untraversable classes based on an expert operator’s judgment of safe access, and self-supervision labels were binarized for fair comparison.Binary ground truth was used to assess design choices; the continuous regression formulation was retained and thresholded for classification metrics.Source: 1) Dataset overview · p. 10

2) Evaluation method:

Evaluation used the stated environment splits and pixel-wise binary accuracy, then compared WVN with classical learning methods. WVN's MLP performed consistently across three environments and outperformed the prior machine-learning approaches overall while allowing mission-time adaptation.

2) Evaluation method:

  • Testing used training and validation splits within the same environment unless stated otherwise.The evaluation materials include aerial views showing the training trajectories and remaining validation paths.Source: 2) Evaluation method · p. 10, Figure 12 · p. 11
  • Accuracy was reported for binary classification in image space, so pixel-level errors include mixed traversable and untraversable segments.Compared neural models were trained for 1000 steps using five random seeds, with confidence intervals reported.Source: 2) Evaluation method · p. 11

3) Study 1: Learning Methods:

  • WVN's MLP performed consistently across three environments and overall outperformed the compared machine-learning methods while allowing gradient-descent adaptation during the mission.The comparison reports traversability accuracy against both ground-truth and self-supervised labels.Source: 3) Study 1: Learning Methods · p. 11, Table III · p. 11

4) Study 2: Training Objective:

The ablations compare training objectives, feature extractors, and cross-environment performance. They report gains from WVN's confidence-weighted loss and online threshold scaling, while showing that feature choice and environment matching affect accuracy.

4) Study 2: Training Objective:

  • WVN consistently outperformed the alternative objectives; adding confidence-weighted traversability loss improved accuracy by 5.17% over Trav, and online threshold scaling added a further 7.33%.The fixed-threshold setting is insufficient during online network adaptation; the Trav setting tends to classify more regions as untraversable than the ground truth.Source: 4) Study 2: Training Objective · pp. 11–12, Table IV · p. 12

5) Study 3: Features:

  • Self-supervised feature extractors outperformed ImageNet-pretrained models, including when the architecture was the same; DINO-ViT also had short inference time on Jetson Orin.The feature comparison isolated the extractor by feeding each feature set to the Trav-setting model.Source: 5) Study 3: Features · p. 12, Table V · p. 12

6) Study 4: Scene Adaptation:

  • Accuracy generally fell when a model was tested outside its training environment, while the Hilly-trained model performed best overall and especially well on Grass.The authors suspect the Hilly-to-Grass result reflects visual similarity; performance declined despite similar semantic classes and nearby deployment conditions.Source: 6) Study 4: Scene Adaptation · p. 12, Table VI · p. 12

7) Study 5: Adaptation Speed & Dataset Size:

WVN achieved good performance within 200 training steps across the tested environments, and the paper reports closed-loop navigation in natural scenes over 1.4 km. The discussion also identifies limits in runtime, segmentation, traversability supervision, and map integration.

7) Study 5: Adaptation Speed & Dataset Size:

  • Good performance was reached within 200 training steps across environments, and the authors attribute this in part to each image providing 100 segment-based training samples.The adaptation-speed figure notes that a small dataset can suffice, while performance is similarly degraded after fewer than 50 training steps regardless of dataset size.Source: 7) Study 5: Adaptation Speed & Dataset… · p. 12, Figure 13 · p. 13

D. Limitations

  • The current system runs at 2.5 Hz on an Orin board with unoptimized code, and SLIC superpixels limit segmentation accuracy by averaging features across semantically different regions.The authors identify both pipeline efficiency and segment extraction as targets for improvement.Source: D. Limitations · p. 13
  • Velocity tracking error is only a proxy for traversability and does not fully characterize robot–environment interactions; the local-map projection also brings perspective, field-of-view, and 2.5D-map drawbacks.The map integration required additional filtering stages.Source: D. Limitations · p. 13
  • WVN does not explicitly address continual learning between test environments, which the authors identify as a potential area for improvement.They also propose multiple cameras and feedback from the locomotion policy as future extensions.Source: D. Limitations · p. 13, §VI · p. 14

VI. CONCLUSION

  • WVN's real-world validation included 1.4 km of closed-loop navigation experiments in natural scenes.The conclusion also reports consistent traversability predictions for local planning and learning from small amounts of data in the field.Source: §VI · p. 14

Related papers

Loading 2305.08510v2…