PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning

Jannik Zürn, Wolfram Burgard, Abhinav Valada

arXiv:1912.03227v1cs.ROcs.CVcs.LG

TL;DR

Changing terrain appearance and the effort of manually labeling new terrains and conditions challenge visual terrain classification. This paper uses vehicle-terrain interaction sounds to self-supervise visual segmentation, reporting state-of-the-art unsupervised audio classification and segmentation comparable to supervised training.

  • Problem

    Terrain appearance changes with environmental conditions, while supervised approaches require substantial manual labeling for newly encountered terrains and variations.

    Source: §I · p. 1

  • Method

    The method learns clustered audio embeddings using triplets formed from visual terrain features, projects traversed tracks into images for sparse labels, and trains a pixel-wise segmentation network.

    Source: §I · p. 2, §V · p. 13

  • Results

    The unsupervised proprioceptive classifier achieves state-of-the-art performance among unsupervised methods, and visual segmentation performs comparably to supervised training with manual labels.

    Source: §I · p. 2, §V · p. 13

  • Takeaways & Limitations

    The framework demonstrates utility for semantic terrain mapping and trajectory planning, with self-supervised adaptation to new environments.

    Source: §V · p. 13

Abstract

from arXiv · show

Mobile robots operating in unknown urban environments encounter a wide range of complex terrains to which they must adapt their planned trajectory for safe and efficient navigation. Most existing approaches utilize supervised learning to classify terrains from either an exteroceptive or a proprioceptive sensor modality. However, this requires a tremendous amount of manual labeling effort for each newly encountered terrain as well as for variations of terrains caused by changing environmental conditions. In this work, we propose a novel terrain classification framework leveraging an unsupervised proprioceptive classifier that learns from vehicle-terrain interaction sounds to self-supervise an exteroceptive classifier for pixel-wise semantic segmentation of images. To this end, we first learn a discriminative embedding space for vehicle-terrain interaction sounds from triplets of audio clips formed using visual features of the corresponding terrain patches and cluster the resulting embeddings. We subsequently use these clusters to label the visual terrain patches by projecting the traversed tracks of the robot into the camera images. Finally, we use the sparsely labeled images to train our semantic segmentation network in a weakly supervised manner. We present extensive quantitative and qualitative results that demonstrate that our proprioceptive terrain classifier exceeds the state-of-the-art among unsupervised methods and our self-supervised exteroceptive semantic segmentation model achieves a comparable performance to supervised learning with manually labeled data.

I. INTRODUCTION

Robots need terrain classifications that remain useful as terrain appearance changes, yet manual labeling across new terrains and conditions is costly. The paper uses vehicle-terrain sounds to self-supervise visual terrain segmentation and reports strong unsupervised audio classification and segmentation comparable to supervised learning.

I. INTRODUCTION

  • Terrain appearance varies with lighting, weather, precipitation, dirt, and snow, while supervised models require repeated manual labeling for new data distributions.The introduction argues that relying only on pre-existing maps is insufficient for safe, efficient navigation.Source: §I · p. 1
  • The authors present cross-modal metric learning as a way to use audio supervision in unseen visual conditions and fine-tune the visual classifier for new appearances.They frame this as a step toward lifelong visual terrain learning.Source: §I · p. 2
  • Visual features form audio-clip triplets for metric learning; clustered audio embeddings label terrain patches, which train a pixel-wise segmentation network.Robot poses associate the visual patch seen ahead with its auditory features when later traversed.Source: Figure 1 · p. 1, §I · p. 2
  • The unsupervised proprioceptive classifier achieves state-of-the-art performance, while self-supervised visual segmentation performs comparably to supervised learning.The framework is also reported to adapt to visual changes from illumination, time of day, weather, and seasonal conditions.Source: §I · p. 2
  • The Freiburg Terrains dataset contains more than four hours of audio-video traversals tagged with SLAM poses and covers five terrain categories across varied environments and conditions.Recordings span different times of day and weather conditions.Source: §I · p. 2

II. RELATED WORK

Prior terrain-classification work explored visual, proprioceptive, and multimodal approaches, but the cited methods faced limits in class generalization, sensor robustness, or use of complementary modalities. This paper positions its approach as combining proprioception and exteroception without manual labels.

II. RELATED WORK

  • Earlier vision-based terrain methods demonstrated long-range classification but were shown on few classes and did not use complementary proprioceptive modalities.The cited examples include deep belief networks and pixel-wise texture-histogram clustering.Source: §II · p. 2
  • Multimodal terrain work combined vibration or acceleration signals with vision, while platform shaking can make vibration characteristics susceptible to misclassification.Brooks et al. used proprioceptive vibration and exteroceptive vision; Bekhti et al. used Canonical Correlation Analysis for terrain maneuverability assessment.Source: §II · pp. 2–3
  • Metric-learning methods commonly rely on ground-truth labels for contrastive tuples, while unsupervised clustering approaches omit complementary cross-modal training and improve clustering only modestly.The cited metric-learning work includes, and the clustering approaches include and Gou et al..Source: §II · p. 3
  • Prior visual segmentation methods used vehicle trajectories, semi-supervised traversability prediction, proximity-sensor prediction, or foothold force-torque labels, but some lacked class generalization or multimodal complementarity.The cited work includes Barnes et al., Hirose et al., Nava et al., and Wellhausen et al..Source: §II · p. 3
  • The paper proposes exploiting complementary proprioception and exteroception without manual labels, reusing an unsupervised audio classifier to adapt visual classification in adverse lighting.The authors also report that automatic collection and self-labeling steadily improve model performance and robustness.Source: §II · p. 3

III. TECHNICAL APPROACH

The framework connects audio-based terrain clustering to pixel-wise visual segmentation by projecting the robot’s traversed path into birds-eye camera images. It converts audio clips to spectrograms and uses pose-tagged sensor data to create weak labels only for ground the robot traversed.

III. TECHNICAL APPROACH

  • The complete pipeline uses audio embeddings clustered into labels for robot-path regions in birds-eye images, which then train a semantic segmentation network.The camera images and robot trajectory are projected into a birds-eye view before cluster labels are assigned to the traversed path.Source: §III · p. 3, Figure 2 · p. 4

A. Audio Preprocessing

  • Each audio clip is converted to a spectrogram before embedding with the Siamese Encoder with Reconstruction loss (SE-R).The short-time Fourier transform produces the spectrogram representation used for audio processing.Source: A. Audio Preprocessing · pp. 3–4, §III · p. 3

B. Trajectory Projection Into Image Coordinates

  • Robot poses from SLAM project the trajectory into birds-eye camera images, and ground within 0.4 m of either side of the trajectory is treated as traversed.Pixels outside the traversed path remain unlabeled because the robot has no terrain information for those regions.Source: B. Trajectory Projection Into Image… · p. 4

C. Unsupervised Acoustic Feature Learning

The method uses visual patch similarity to form audio triplets, learns discriminative audio embeddings with a Siamese Encoder, and clusters those embeddings into terrain labels.

C. Unsupervised Acoustic Feature Learning

  • Visual terrain-patch embeddings provide a heuristic substitute for audio labels when forming triplets for unsupervised acoustic feature learning.The nearest visual embedding is selected as the positive, while negatives are sampled from a different visual cluster; the authors note some triplets may be incorrect.Source: C. Unsupervised Acoustic Feature… · pp. 4–5
  • The Siamese Encoder with Reconstruction loss (SE-R) adds a convolutional decoder to the triplet-trained encoder and minimizes triplet and reconstruction losses together.The encoder and decoder map spectrograms to embeddings and back to the spectrogram domain.Source: C. Unsupervised Acoustic Feature… · p. 5
  • K-means clusters the learned embeddings into K groups corresponding to the dataset’s terrain classes, and cluster indices supply terrain labels.The method requires setting the number of terrain classes and assigning names to cluster indices.Source: C. Unsupervised Acoustic Feature… · p. 5, Figure 3 · p. 5

D. Self-Supervised Visual Terrain Classification

The visual classifier is trained from weak labels on traversed ground, with other pixels excluded from the segmentation loss. The evaluation covers real-world data and describes quantitative, qualitative, ablation, generalization, and trajectory-planning analyses.

D. Self-Supervised Visual Terrain Classification

  • Weakly labeled birds-eye scenes train segmentation using known labels on the robot path, while pixels outside the path are excluded as background.Class weights are proportional to each class’s log frequency in the training data.Source: D. Self-Supervised Visual Terrain… · p. 5
  • The segmentation objective uses class-weighted cross-entropy over image pixels with ground-truth labels and network log probabilities.The passage defines the class weight, pixel label, and softmax log probability used in the loss.Source: D. Self-Supervised Visual Terrain… · p. 5

IV. EXPERIMENTAL EVALUATION

  • The evaluation uses real-world data collected with the Obelix robot across diverse environments and perceptual conditions.The evaluation plan includes acoustic clustering, semantic segmentation, challenging-scene analysis, ablations, generalization, and trajectory planning; examples include similar terrain appearances or sounds and illumination changes.Source: §IV · pp. 5–6, Figure 4 · p. 6

A. Dataset

The dataset combines synchronized terrain images and vehicle–terrain audio across five terrain classes, with weakly labeled training imagery and separate labeled evaluation sets. The section also specifies the clustering and segmentation metrics used for evaluation.

A. Dataset

  • The dataset covers Asphalt, Grass, Cobblestone, Parking Lot, and Gravel, terrains with some visually or acoustically similar classes.The robot is unsuitable for wet mud, crushed stone, or puddles; collection varied weather, time of day, and robot speed.Source: A. Dataset · p. 6
  • Over four hours of recordings support weakly labeled training images, 200 densely pixel-annotated validation images, and a separate weakly labeled evaluation set of 500 images.Recordings were collected over three weeks in varying weather and at different times of day.Source: A. Dataset · p. 6

B. Evaluation Metrics

  • Clustering is evaluated with NMI and clustering accuracy, while segmentation uses class-wise IoU and mIoU on dense labels and average class-wise recall on weak labels.Because at most 5% of pixels are labeled in weakly labeled images, IoU is not considered meaningful there; clustering accuracy uses the best class-to-cluster assignment.Source: B. Evaluation Metrics · pp. 6–7

C. Training Details

The audio and segmentation models use distinct training configurations, and the proposed audio embeddings yield the strongest reported clustering metrics among the compared methods. The embeddings also become increasingly separable as training progresses.

1) Audio Clustering:

  • SE and SE-R were trained for 100 epochs, while DEC, IDEC, and DCEC were further optimized on the dataset after using their original hyperparameters.The audio encoder and decoder follow Engel et al., and ImageNet-pretrained MobileNet V2 extracts visual features for triplet formation.Source: 1) Audio Clustering · p. 7

2) Self-Supervised Semantic Segmentation:

  • Semantic segmentation uses AdapNet++ with an EfficientNet backbone, data augmentations, and Adam optimization with polynomial learning-rate decay.Augmentations include rotations, flips, and color jitter in hue, saturation, brightness, and contrast.Source: 2) Self-Supervised Semantic Segmentation · p. 7

D. Evaluation of Unsupervised Acoustic Feature Clustering

  • The proposed audio methods report clustering accuracies of 94.07% and 94.81%, and achieve the best clustering accuracy and NMI among the compared methods.DEC and DCEC do not exceed 85.35% clustering accuracy; the table compares NMI and clustering accuracy across embedding models.Source: Table I · p. 7, D. Evaluation of Unsupervised Acoustic… · p. 7
  • In t-SNE-reduced two-dimensional SE-R audio embeddings, clusters become well separable and highly correlated with ground-truth classes as training progresses, despite some label correlation already visible before training.Source: D. Evaluation of Unsupervised Acoustic… · p. 7

E. Evaluation of Self-Supervised Semantic Segmentation

SE-R-generated weak labels outperform DEC-generated labels for semantic segmentation, with reported gains on weakly and fully labeled validation sets. Qualitative results show both challenging scenes and remaining visual-confusion failure cases.

E. Evaluation of Self-Supervised Semantic Segmentation

  • SE-R labels improve recall by 50.3% on the weakly labeled validation set and mIoU by 30.2% on the fully labeled validation set versus DEC labels.The model trained with SE-R labels achieves performance comparable to the model trained on manually annotated data.Source: E. Evaluation of Self-Supervised… · p. 8
  • On the fully labeled validation set, gravel is sometimes confused with grass and cobblestone with parking lot.The passage attributes these confusions to grass growing over gravel paths and similar rectangular patterns in cobblestone and parking-lot terrain.Source: E. Evaluation of Self-Supervised… · p. 8

F. Qualitative Semantic Terrain Segmentation Results

  • The SE-R model is reported to segment most scenes accurately, including cluttered scenes and scenes with multiple adjoining terrains, while DEC-trained outputs often miss terrain classes.The qualitative comparison attributes DEC errors to inconsistencies in its image labels; Fig. 7 marks SE-R improvements over DEC in green and SE-R errors in red.Source: F. Qualitative Semantic Terrain… · p. 8

G. Ablation studies

  • The section introduces ablation studies to examine model performance under different settings and the effects of the paper’s contributions.No specific ablation outcome is stated in the supplied passage.Source: G. Ablation studies · p. 8

1) Noise Resistance of Audio Embeddings:

The ablations examine noise robustness, training-set size, and triplet sampling. SE-R retains an accuracy advantage over SE across nearly all tested noise levels, while more triplets and a strong sampling heuristic support better clustering.

1) Noise Resistance of Audio Embeddings:

2) Number of Samples for Clustering:

3) Triplet Sampling Mechanism:

  • The tested triplet heuristics include random, distance-based, cluster-based, and ground-truth formation, with ground-truth labels serving only as a reference.Only ground-truth triplet formation requires sample-class knowledge; the other heuristics do not.Source: 3) Triplet Sampling Mechanism · p. 10
  • The combination of distance-based positives and cluster-based negatives forms 81.35% correct triplets and yields 94.8% clustering accuracy with SE-R.Random triplet formation performs only marginally above chance; the authors relate lower correctly formed triplet fractions to lower accuracy and NMI.Source: Table III · p. 10, 3) Triplet Sampling Mechanism · pp. 10–11, Figure 10 · p. 11

4) Ratio of Correctly Formed Triplets:

Triplet quality strongly affects clustering accuracy, while the segmentation architecture study selects AdapNet++ with the Efficient backbone for the framework.

4) Ratio of Correctly Formed Triplets:

  • Clustering accuracy increases with the fraction of correctly formed triplets and falls below 90% for SE-R when that ratio is 0.8.For correct triplet ratios below 0.2, accuracy declines to 30%.Source: 4) Ratio of Correctly Formed Triplets · p. 11
  • SE-R consistently achieves higher clustering accuracy than SE across the tested correctly formed triplet ratios.The experiment deliberately assigns incorrect samples as positive or negative triplets to vary the correct-triplet ratio.Source: 4) Ratio of Correctly Formed Triplets · p. 11

5) Segmentation Network Architectures:

  • Larger networks such as DeepLabv3+ have several million parameters, making them inefficient for online robotics and demanding substantial training compute.ERFNet, BiSeNet, and F-SCNN instead emphasize fewer parameters and faster training and inference on a given hardware budget.Source: 5) Segmentation Network Architectures · p. 11
  • Table IV presents a comparison of self-supervised semantic terrain segmentation performance across network architectures.The caption does not provide the table’s individual values.Source: Table IV · p. 11

6) Influence of the Perspective Transformation:

  • Table V compares self-supervised semantic terrain segmentation performance when learning from different perspectives.The caption does not provide the table’s individual values.Source: Table V · p. 11

5) Segmentation Network Architectures:

  • AdapNet++ yields the highest mean IoU, while ENet yields the highest recall; the authors therefore adopt AdapNet++ with the Efficient backbone.The comparison covers DeepLabv3+ backbones and smaller segmentation networks on the weakly labeled Freiburg Terrains dataset.Source: 5) Segmentation Network Architectures · p. 11
  • The architecture evaluation compares four DeepLabv3+ backbones and three smaller segmentation networks on the weakly labeled Freiburg Terrains dataset.The tested DeepLabv3+ backbones are ResNet-50, DRN-D-105, Xception, and MobileNetV2; the smaller networks are ENet, BiSeNet, and Fast-SCNN.Source: 5) Segmentation Network Architectures · p. 11

6) Influence of the Perspective Transformation:

Birds-eye-view terrain images produce higher mean IoU than standard camera-perspective images. The paper also reports low-light adaptation through audio-generated labels and uses semantic terrain maps for cost-aware route planning.

6) Influence of the Perspective Transformation:

  • Birds-eye-view terrain images yield higher mean IoU than standard camera-perspective images, attributed to terrain patterns being distance-independent in that view.The study compares birds-eye-view and standard camera-perspective representations.Source: 6) Influence of the Perspective… · p. 11

H. Generalization to Terrain Appearance Changes

  • 93.1% of audio clips in the new low-light dataset are labeled accurately by the SE-R model trained on the combined dataset.The dusk data include low light, artificial lighting, motion blur, and terrain-hue variation; the framework generates labels without human labeling, and the segmentation model is fine-tuned for 10 additional epochs.Source: H. Generalization to Terrain… · pp. 11–12
  • After 10 additional fine-tuning epochs, the segmentation model predicts terrains accurately in most low-light scenes, unlike the model without fine-tuning.The authors report that lighting, hue changes, motion blur, and other low-light artifacts are not easily modeled by the training augmentations.Source: H. Generalization to Terrain… · p. 12, Figure 11 · p. 12

I. Semantic Terrain Mapping and Trajectory Planning

  • The mapping procedure projects birds-eye-view segmentation predictions into a global map using SLAM poses and selects the most frequent predicted class for pixels seen from multiple views.The authors also tested Bayesian fusion of multiple-view predictions and report similar results.Source: §I · p. 12
  • The predicted terrain maps are mostly globally consistent, with some shadow-related errors in cobblestone and grass regions.Asphalt and Grass are accurately predicted in the majority of pixels, and some class boundaries align well with the ground truth.Source: §I · p. 12
  • The planning procedure represents each terrain-map pixel as a graph node and each neighboring pixel pair as an edge, with costs set by terrain class.The paper uses Dijkstra’s algorithm as a proof of concept for minimum-cost trajectory planning.Source: §I · p. 12
  • Cost-aware routes avoid high-cost Grass and Parking Lot terrain in favor of lower-cost Asphalt or Cobblestone, at the cost of a longer route.Fig. 12 contrasts a cost-aware white route with a cost-unaware blue route on tiled views and semantic maps.Source: §I · pp. 12–13, Figure 12 · p. 13

V. CONCLUSION

The paper combines unsupervised audio terrain classification with visual terrain segmentation trained from weak labels. It reports state-of-the-art unsupervised audio classification, segmentation comparable to fully supervised training, and demonstrations of adaptation, mapping, and trajectory planning.

V. CONCLUSION

  • The framework uses an unsupervised proprioceptive classifier to supervise pixel-wise exteroceptive terrain segmentation, with audio embedding clusters labeling visual terrain patches for training.A triplet-sampling heuristic uses a complementary modality rather than ground-truth labels.Source: §V · p. 13
  • The audio classifier achieves state-of-the-art performance among unsupervised methods, while self-supervised segmentation performs comparably to training on fully supervised, manually annotated labels.The work introduces and publicly releases the Freiburg Terrains dataset.Source: §V · p. 13
  • Experiments demonstrate semantic terrain mapping and trajectory planning, as well as adaptation of the segmentation model to a new environment in a self-supervised manner.The conclusion frames these results as a step toward lifelong learning.Source: §V · p. 13

Related papers

Loading 1912.03227v1…