PaperlayerView exact revision on arXiv

Source-linked AI summary

GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation

Noriaki Hirose, Amir Sadeghian, Marynel Vázquez, Patrick Goebel, Silvio Savarese

arXiv:1803.03254v1cs.ROcs.CVcs.LG

TL;DR

Safe robot navigation requires recognizing non-traversable spaces, while collecting unsafe examples can be costly and dangerous. GONet uses GAN-based fisheye-image estimation trained with many safe examples and few unsafe ones; the authors report greater robustness than supervised and fully unsupervised baselines.

  • Problem

    Recognizing non-traversable spaces is essential for robot safety, but collecting negative examples can be time-consuming, costly, and dangerous.

    Source: §I · p. 1

  • Method

    GONet and its extensions use generative models to estimate traversability from fisheye images, learning from many positive examples and a small set of negative examples.

    Source: §I · p. 1

  • Results

    GONet and its extensions are reported as more robust than supervised and fully unsupervised baselines, and full GONet outperformed the other models evaluated on GS1.

    Source: C. GONet Evaluation on GS1 · p. 5, §VII · p. 8

  • Takeaways & Limitations

    Real-time fisheye predictions can serve as an automatic visual emergency stop and complement 2D lidar during robot navigation.

    Source: §I · pp. 1–2

Abstract

from arXiv · show

We present semi-supervised deep learning approaches for traversability estimation from fisheye images. Our method, GONet, and the proposed extensions leverage Generative Adversarial Networks (GANs) to effectively predict whether the area seen in the input image(s) is safe for a robot to traverse. These methods are trained with many positive images of traversable places, but just a small set of negative images depicting blocked and unsafe areas. This makes the proposed methods practical. Positive examples can be collected easily by simply operating a robot through traversable spaces, while obtaining negative examples is time consuming, costly, and potentially dangerous. Through extensive experiments and several demonstrations, we show that the proposed traversability estimation approaches are robust and can generalize to unseen scenarios. Further, we demonstrate that our methods are memory efficient and fast, allowing for real-time operation on a mobile robot with single or stereo fisheye cameras. As part of our contributions, we open-source two new datasets for traversability estimation. These datasets are composed of approximately 24h of videos from more than 25 indoor environments. Our methods outperform baseline approaches for traversability estimation on these new datasets.

I. INTRODUCTION

Safe mobile-robot operation depends on recognizing non-traversable spaces, but collecting unsafe examples is costly and risky. GONet and its extensions use fisheye images and few negative examples to estimate traversability, with real-time demonstrations and indoor datasets supporting their practical use.

I. INTRODUCTION

  • Existing bumper and depth-sensing approaches have limitations, including missed edges or small obstacles and difficulty with reflective or transparent surfaces.These limitations motivate image-based traversability estimation alongside earlier RGB classification work.Source: §I · p. 1
  • Many positive examples and few negative examples let the methods learn traversability despite the cost and danger of collecting unsafe cases.Positive images can be collected by driving through safe areas; negative examples may require positioning a robot near hazards.Source: §I · p. 1
  • GONet uses a robot-mounted fisheye image and generative adversarial models to predict whether the visible area is safe to traverse.The approach requires only an off-the-shelf camera and uses a feed-forward structure for real-time mobile operation.Source: §I · p. 1, Figure 1 · p. 1
  • The approach is described as robust and scalable across fisheye views, including use on a back-facing camera after training on a forward-facing view.The paper attributes robustness to broad environmental coverage and learning from varied data.Source: §I · p. 1
  • GONet+T enforces temporal consistency using automatically annotated image sequences, while GONet+TS adds stereo fisheye images and retains temporal consistency.The extensions modestly increase computational power while improving prediction smoothness and accuracy in challenging cases such as strong shadows.Source: §I · p. 2
  • The authors contribute new indoor traversability datasets, extensive experiments, and real-time robot demonstrations of an emergency-stop use and lidar-complementary navigation.The emergency-stop demonstrations target collisions and falls down stairs.Source: §I · p. 2

II. PRELIMINARIES ON DEEP LEARNING

The paper introduces GAN-based GONet for traversability estimation from fisheye images, then extends it to temporal and stereo-temporal predictions. Its single-view model learns a positive-image manifold and compares each input with a generated image, using a small labeled set to classify traversability.

II. PRELIMINARIES ON DEEP LEARNING

  • GANs train a generator to model training-data examples and a discriminator to distinguish generated from real samples; the paper uses DCGANs to generate images from latent variables.GAN training is unsupervised in the sense that it does not rely on labels.Source: §II · p. 2

III. SEMI-SUPERVISED TRAVERSABILITY ESTIMATION

  • GONet+T adds temporal consistency, and GONet+TS uses stereo images while retaining temporal consistency.Both are extensions of the single-view traversability method.Source: §III · p. 2

A. GONet: Single-View Traversability Classification

  • GONet predicts traversability from a fisheye image by generating a similar image from the positive-example manifold and comparing it with the input.The model assumes generated and input images will be similar for traversable areas and different for non-traversable scenarios.Source: A. GONet: Single-View Traversability… · p. 2
  • The classifier combines image residuals, discriminator features, and discriminator-feature differences through separate fully connected layers and a final sigmoid layer.The scalar outputs of the three feature-processing paths are concatenated before the final layer.Source: A. GONet: Single-View Traversability… · pp. 2–3
  • An auxiliary autoencoder uses InvGen to find a latent variable whose frozen-Gen output resembles the input under residual and discriminator losses.The loss weights these two terms with an empirically chosen parameter.Source: A. GONet: Single-View Traversability… · p. 3
  • Training proceeds through positive-only DCGAN training, positive-example InvGen training, and final classifier training with a small positive-and-negative set.The final classifier stage uses early stopping; positive examples are automatically collected from sustained robot motion, while negative examples are gathered near hazards.Source: A. GONet: Single-View Traversability… · p. 3

B. GONet+T: Enforcing Temporal Consistency

GONet+T addresses the fact that robot image observations are temporally dependent by adding an LSTM to GONet. Its training combines sequence labels with semi-automatic annotation, and its loss balances prediction accuracy and temporal smoothness.

B. GONet+T: Enforcing Temporal Consistency

  • GONet+T uses an LSTM to reduce prediction variance over time, addressing the temporal dependence of robot observations.GONet’s original assumption that each image is independent and identically distributed does not hold for robot-captured sequences.Source: B. GONet+T: Enforcing Temporal… · p. 3
  • Three feature vectors are reduced to 10 dimensions each, concatenated, and passed through an LSTM before a sigmoid layer predicts traversability probability.The features come from GONet’s Feature Extraction Module.Source: B. GONet+T: Enforcing Temporal… · p. 3
  • The multi-objective loss trades off prediction error against temporal smoothness, with λ = 0.5 selected using the validation set.Training optimizes the model on labeled image sequences.Source: B. GONet+T: Enforcing Temporal… · pp. 3–4
  • The annotation procedure labels positive examples from robot velocity, manually labels a small negative subset, then uses GONet’s probabilities to label remaining images collected while stationary or moving slowly.The sequence model requires fully annotated training sequences, while annotating many negative examples is costly and time-consuming.Source: B. GONet+T: Enforcing Temporal… · p. 3
  • ε = 0.04 mean prediction difference after re-annotation; accuracy on the positive training set increased.The reported mean difference compares GONet’s output probabilities before and after retraining on re-annotated data.Source: B. GONet+T: Enforcing Temporal… · p. 4

C. GONet+TS: Estimating Traversability With Stereo Views

GONet+TS extends GONet+T to stereo fisheye images because appearance effects such as strong shadows sometimes affected performance. The authors hypothesized that two views could improve robustness in challenging scenarios.

C. GONet+TS: Estimating Traversability With Stereo Views

  • Strong shadows sometimes induced false-negative predictions, motivating the stereo extension and the hypothesis that two views could improve robustness in challenging scenarios.GONet and GONet+T had worked well in different situations, but their performance was sometimes affected by environmental appearance.Source: C. GONet+TS: Estimating Traversability… · p. 4
  • GONet+TS retains GONet+T’s architecture and training procedure but takes a six-channel input stacking the stereo pair’s left and right RGB images.The model uses the same loss and data collection and annotation procedure as GONet+T.Source: C. GONet+TS: Estimating Traversability… · p. 4

IV. EVALUATION

The evaluation section describes experiments assessing the proposed traversability-estimation methods in indoor environments.

IV. EVALUATION

  • The proposed methods are evaluated in indoor environments, with results computed using an NVIDIA GeForce GTX TITAN X GPU.The passage specifies the evaluation setting and hardware.Source: §IV · p. 4

A. Datasets

The evaluation uses two fisheye-camera datasets collected across Stanford buildings and tests data efficiency using unseen labeled images. GONet reached 94.25% accuracy with 800 training examples.

A. Datasets

  • GS1 contains 7.2 hours of sensor data from 15 buildings, while GS2 contains 16.74 hours from 27 buildings; both datasets are split by building for training, validation, and testing.GS1 uses 9/3/3 buildings across the splits, and GS2 uses 19/4/4.Source: A. Datasets · p. 4
  • The datasets include 2,400 manually annotated negative examples apiece, complemented by an equal number of automatically annotated positive images.Images were resized to 3×128×128 for the experiments.Source: A. Datasets · p. 4

B. Data efficiency

  • 94.25% accuracy was reached with 800 training examples—400 positive and 400 negative—on unseen labeled GS1 test images.The authors then used 800 images to train supervised models in subsequent experiments.Source: B. Data efficiency · p. 4

C. GONet Evaluation on GS1

On GS1, the full GONet architecture outperformed the evaluated models, while using less memory and computation time than ResNet. The reported comparisons also favor GONet’s inverse-generator feature extraction over the autoencoder and iterative back-propagation baselines.

C. GONet Evaluation on GS1

  • Adding a small amount of negative training data generally improved accuracy.Table II separates positive-only training from training with positive and negative examples.Source: C. GONet Evaluation on GS1 · p. 5
  • GONet’s Feature Extraction Module achieved the highest accuracy among positive-only unsupervised methods and was more accurate and faster than iterative back-propagation.It also outperformed the baseline autoencoder.Source: C. GONet Evaluation on GS1 · p. 5
  • The full R+D+F GONet model outperformed all other evaluated GS1 models while reducing memory requirements and computation time compared with ResNet.The authors connect this efficiency to mobile robots’ often limited onboard computing resources.Source: C. GONet Evaluation on GS1 · p. 5
  • The GS1 ablation indicates that each feature and classification component contributes to GONet’s accuracy.Source: C. GONet Evaluation on GS1 · p. 5

D. Evaluation of GONet and its extentions on GS2

On GS2, GONet+TS had the strongest reported performance, and re-annotation and temporal models improved accuracy. Qualitative analyses show how generated images, saliency, temporal consistency, and stereo input relate to traversability predictions.

D. Evaluation of GONet and its extentions on GS2

  • On GS2, re-annotation increased accuracy, and models using LSTM units also performed better than the other evaluated rows.The authors attribute the LSTM models’ success to reasoning about dependencies between input images.Source: D. Evaluation of GONet and its… · p. 5

E. Qualitative Analysis

D. Evaluation of GONet and its extentions on GS2

  • GONet+TS had the best GS2 performance, and GONet and its extensions outperformed the supervised depth-map and pre-trained-feature baselines.The results support the authors’ hypothesis that stereo fisheye images can help estimate traversability.Source: D. Evaluation of GONet and its… · p. 5, Table III · p. 6

E. Qualitative Analysis

  • The bottom portions of the input images were most salient for GONet+TS and corresponded to the space in front of the robot.Source: E. Qualitative Analysis · p. 6, Figure 5 · p. 6
  • In the shadow scenario, only GONet+TS correctly predicted traversability at the final highlighted time point; temporal models also produced smoother predictions in the glass-door scenario.The authors report that temporal consistency and stereo vision often helped in challenging scenarios.Source: E. Qualitative Analysis · p. 6

V. APPLICATIONS

The applications section demonstrates GONet and GONet+T in emergency stopping, lidar-assisted mapping, and assistive navigation. The demonstrations used a fisheye camera and laptop, with image processing capped at the training frame rate.

V. APPLICATIONS

  • The applications demonstrate visual emergency stopping, complementing lidar-based exploration and mapping, and navigation assistance for visually impaired people.Source: §V · p. 6
  • The demonstrations focus mainly on GONet+T because it was more accurate than GONet in the quantitative evaluation, while GONet+TS results are omitted due to space constraints.Source: §V · p. 6
  • GONet and GONet+T ran at the camera’s maximum 15 Hz on the laptop, but processing was limited to the 3 Hz training frame rate.Source: §V · p. 6

A. Visual Emergency Stop (E-Stop) Switch

The demonstrations use traversability predictions to stop a robot, update a navigation costmap, and warn a blindfolded pedestrian about hazards. Failures with a tangled wire and small, carpet-colored objects mark practical limits of the emergency-stop demonstration.

A. Visual Emergency Stop (E-Stop) Switch

B. Mobile Robot Exploration & Mapping

C. Assistive Traversability for the Visually Impaired

  • GONet+T issued audible warnings in unseen indoor and outdoor pedestrian settings despite not being trained on outdoor or pedestrian images.The authors suggest that a warning system could be mounted on a belt or integrated into glasses.Source: C. Assistive Traversability for the… · p. 7

VI. RELATED WORK

The paper situates GONet among learned traversability and anomaly-detection approaches, then concludes that semi-supervised methods using many positive and few negative examples are practical and robust. Its evaluation is limited to indoor environments, with outdoor accuracy and joint policy learning left for future work.

VI. RELATED WORK

  • Unlike hand-tuned-feature approaches, the methods process raw images and learn task-relevant information from data.Source: §VI · p. 7
  • Adding a small number of negative images increased GONet’s accuracy compared with positive-only training.Source: §VI · p. 7
  • GONet treats traversable scenes as expected inputs and non-traversable examples as outliers, using query-resembling image manipulation in a technique the authors identify as a first for traversability estimation.Source: §VI · p. 7
  • The authors suggest GONet could support learned robot motion behaviors, including steering prediction, while combining traversability estimation with policy learning is outside this paper’s scope.Source: §VI · p. 8

VII. CONCLUSION & FUTURE WORK

  • The methods’ uneven-data training makes them more robust than supervised and fully unsupervised baselines while remaining practical.They use many positive images and just a few negative examples.Source: §VII · p. 8
  • The feed-forward methods run in real time on mobile hardware and were demonstrated for avoiding stairs and glass, complementing 2D lidar, and warning visually impaired users.Source: §VII · p. 8
  • Experiments covered indoor environments; outdoor accuracy and end-to-end joint traversability and policy learning remain future-work directions.Source: §VII · p. 8

Related papers

Loading 1803.03254v1…