PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networks

Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, Gijs Dubbelman

arXiv:1804.02176v3cs.ROcs.CV

TL;DR

The paper addresses end-to-end monocular semantic-metric occupancy-grid mapping from weak binocular ground truth. It uses a variational encoder-decoder to transform a front-view RGB image into a top-view semantic map, outperforming flat-plane mapping by more than 12% mean IoU while showing robustness, generalization, and real-time inference.

  • Problem

    Perception remains a bottleneck for forming consistent world models, motivating end-to-end learning of semantic occupancy-grid maps from monocular input.

  • Method

    A variational encoder-decoder maps one front-view monocular RGB image to a top-view occupancy grid with a semantic class assigned to each cell.

  • Results

    The monocular neural methods surpass the flat-plane assumption by nearly 12% mean IoU, while the VED approach shows robustness to pitch and roll perturbations and generalization to unseen scenarios.

  • Takeaways & Limitations

    Variational sampling over a relatively low-dimensional embedding space supports robustness and generalizability compared with the canonical neural-network baseline.

Abstract

from arXiv · show

In this work, we research and evaluate end-to-end learning of monocular semantic-metric occupancy grid mapping from weak binocular ground truth. The network learns to predict four classes, as well as a camera to bird's eye view mapping. At the core, it utilizes a variational encoder-decoder network that encodes the front-view visual information of the driving scene and subsequently decodes it into a 2-D top-view Cartesian coordinate system. The evaluations on Cityscapes show that the end-to-end learning of semantic-metric occupancy grids outperforms the deterministic mapping approach with flat-plane assumption by more than 12% mean IoU. Furthermore, we show that the variational sampling with a relatively small embedding vector brings robustness against vehicle dynamic perturbations, and generalizability for unseen KITTI data. Our network achieves real-time inference rates of approx. 35 Hz for an input image with a resolution of 256x512 pixels and an output map with 64x64 occupancy grid cells using a Titan V GPU.

I. INTRODUCTION

The paper targets monocular end-to-end construction of semantic-metric occupancy-grid world models, motivated by the bottleneck between perception and consistent environmental representation. It proposes a variational encoder-decoder approach designed for real-time mapping, robustness, and generalization.

  • Motivation: Semantic occupancy grids extend local free-space maps with road, sidewalk, terrain, and non free-space subclasses.This representation adds semantic information to metric occupancy mapping for mobile robots and intelligent vehicles.
  • Motivation: CNN feature maps preserve semantic abstractions and inter-pixel spatial relations that also occur between occupancy-grid cells.This correspondence motivates learning occupancy-grid maps directly from image data.
  • Contributions: The proposed system performs end-to-end monocular learning from front-view imagery to semantic-metric occupancy-grid maps.The paper presents this as a contribution aimed at real-time inference.
  • Contributions: The approach is reported as robust to pitch and roll perturbations and capable of generalizing to unseen data from different cameras.These properties are presented as contributions of the end-to-end monocular method.
  • Contributions: The network can be trained from weak ground truth and is reported to be robust to sparse input data.The evaluation uses Cityscapes and KITTI data.

II. RELATED WORK

Prior work supports semantic understanding and metric mapping through separate sensing and processing pipelines, but these approaches are not directly compatible with unified vehicle mapping and planning. The paper motivates learning semantic-metric occupancy maps holistically from monocular images.

  • Occupancy-grid mapping: Classical occupancy grids represent cells as occupied or not occupied, without semantic classes.Semantic information is introduced as a way to support navigation.
  • Semantic understanding: Semantic segmentation methods estimate pixel-level labels, while image-based semantic mapping commonly projects those labels into 3-D using stereo or additional processing.The related work includes FCN, SegNet, CRF-based, and adversarial segmentation approaches.
  • Limitations of prior pipelines: Image-based semantic segmentation is usually not directly compatible with vehicle mapping and planning because metric mapping and semantic segmentation are performed separately.This separation motivates a more integrated mapping approach.
  • Motivation for learned mapping: Prior monocular and binocular deep-learning work on depth estimation suggests that metric information can be learned directly from photometric data.This motivates mapping environments into semantic-metric occupancy grids.
  • Proposed architecture: The proposed network structure uses a pre-trained VGG-16 feature extractor together with encoder and decoder components.The figure legend identifies convolutional, fully connected, sampling, up-convolutional, and convolutional blocks.

III. SEMANTIC OCCUPANCY GRID MAPPING

The paper defines a semantic-metric occupancy-grid representation and describes its grid resolution and class encoding for neural-network-based mapping. The representation extends classical occupancy grids with semantic ground-area categories.

  • Representation: The proposed representation extends classical occupancy grids to contain semantic and metric information suitable for modern deep neural networks.This establishes the map representation used by the method.
  • Grid size and perceiving distance: Each grid map contains 64×64 cells, with each cell measuring 0.5×0.5 meters in 2-D vehicle coordinates.The grid is designed to maintain reliable cell status over the perceiving region.
  • Semantic encoding: Each cell is assigned one of four semantic classes: road, sidewalk, terrain, or non free-space.Non free-space includes undetected grids behind foreground objects and outside the camera’s field of view.
  • Semantic encoding: The semantic encoding replaces binary free-space labeling with semantic ground-area categories that may benefit navigation.The representation retains a non free-space class alongside the ground-surface classes.

B. Network structure and training

The proposed system uses a feature extractor and variational encoder-decoder to transform monocular front-view images into semantic top-view occupancy grids. It is trained end-to-end with weak ground truth generated from registered 3-D information and combines latent and mapping objectives.

  • Network structure: The system combines a low-level feature extractor with a modified variational encoder-decoder network for end-to-end mapping.The network takes extracted image features and decodes them into a semantic occupancy grid.
  • Network structure: A monocular RGB image is encoded into a latent embedding, then decoded into a 2-D top-view grid whose cells receive semantic classes.The latent embedding includes spatial information and semantics.
  • Training: Training uses variational sampling with a latent Kullback-Leibler loss and a mapping cross-entropy loss.The latent embedding is regularized toward N(0, I), while the mapping loss compares softmax outputs with one-hot ground truth.
  • Training: Weak ground truth is generated by reconstructing 3-D points, assigning semantic labels, projecting points onto the ground plane, and using majority voting per grid cell.The automatically generated labels can be noisy or sparse because the registered depth or disparity is imperfect.
  • Training: Only 70 top-view maps are manually annotated for evaluation, so the network is trained using automatically generated weak ground truth.The paper identifies training from weak ground truth as an important feature of the approach.

IV. EXPERIMENTS

The experiments evaluate accuracy, disturbance invariance, cross-dataset generalizability, variational sampling, resolution effects, and the information encoded in the latent representation.

  • Quantitative evaluation: Cityscapes quantitative evaluation measures mapping performance using semantic image-segmentation metrics.
  • Input disturbance invariance: Pitch and roll perturbation experiments test whether the mapping approach remains invariant to simulated camera movements.
  • Generalizability to unseen scenarios: Unseen KITTI data evaluates generalizability across a semantic domain gap and different camera parameters.
  • Ablation study: An ablation study compares the proposed network with and without variational sampling to assess the role of that design choice.
  • Mapping quality invariance w.r.t. resolutions: Resolution experiments compare two deterministic baselines with the neural network method across different output-map resolutions.
  • Semantic latent embedding: A small experiment examines the high-level information encoded in the VED latent embedding.

A. Dataset and ground truth

Cityscapes supplies stereo imagery, disparity, and fine pixel-level semantic annotations for training and evaluation, while KITTI provides an unseen cross-dataset test setting. Because automatically generated maps are noisy, the authors manually improve a subset for quantitative evaluation.

  • Dataset: Cityscapes contributes 2,975 training images and 500 validation images, resized from 1024×2048 to 256×512 pixels.
  • Ground truth: Semi-global matching disparity and semantic annotations are used to generate weak top-view ground truth from Cityscapes.
  • Ground truth: 70 validation-set top-view maps are manually improved and annotated to provide a more reliable evaluation reference.These maps are based on the automatically generated weak ground truth and visual cues in the corresponding front-view images.
  • Dataset: KITTI supplies 200 images with semantic annotations, depth maps, and camera parameters for testing generalizability.

B. Baseline methods

The study compares the proposed VED mapping approach with flat-plane monocular, binocular disparity-based, and SegNet baselines. The flat-plane baseline deterministically projects semantic ground-like pixels under a ground-height assumption, while the binocular baseline uses disparity-derived 3-D points.

  • Compared methods: The comparison includes two canonical point-cloud methods and the CNN-based SegNet method.
  • Monocular mapping with flat-plane assumption: The flat-plane monocular baseline maps semantic-segmentation outputs by assuming road, sidewalk, and terrain pixels lie at zero height.Its mapping uses camera calibration and can produce offsets on slopes or under noisy inertial measurements.
  • Binocular mapping: The binocular baseline uses image pairs and disparity to generate 3-D point clouds for occupancy-grid construction.Its semantic information is estimated with a VGG-16-based fully convolutional network rather than labeled Cityscapes annotations.

3) Mapping using canonical CNNs (SegNet):

The study compares CNN-based semantic-metric occupancy-grid mapping with deterministic and binocular baselines, evaluating accuracy under weak and manually improved ground truth.

  • SegNet setup: SegNet is adapted to map 256×512 input images to 64×64 outputs using a VGG-16 backbone.Its trainable parameter count is approximately 28.9M, compared with approximately 27.5M for the proposed VED network.
  • Evaluation: The evaluation reports mean accuracy and mean intersection-over-union (IoU), averaged over test samples.Metrics are evaluated against both weak and manually improved ground truth, while cells outside the camera FOV are ignored.
  • Results: Nearly 5% mean accuracy and 12% mean IoU separate the two neural methods from the flat-plane assumption method.SegNet performs slightly better than VED by less than 2%, while the binocular method outperforms all monocular methods.
  • Runtime: Approximately 35 Hz is achieved by the VED method for 256×512 inputs on an Nvidia Titan V GPU.The reported inference time is about 28 milliseconds per input.

2) Input disturbance invariance:

The disturbance experiment tests whether monocular mapping remains stable under simulated camera pitch and roll, avoiding dependence on an IMU for perturbation compensation.

  • Motivation: Pitch and roll perturbations can significantly degrade monocular flat-plane mapping if they are not accounted for.The study motivates an image-to-top-view mapping that is intrinsically invariant to these disturbances without using an IMU.
  • Experimental design: The experiment simulates pitch with vertical pixel offsets and roll with in-plane rotations around the imaging center.Perturbed images are evaluated with multiple mapping methods using quantitative metrics and visual examples.
  • Findings: The VED network exhibits invariance to the simulated pitch and roll disturbances.Table II reports the metrics, while Figure 4 visualizes examples under different orientation disturbances.
  • Visualization: Figure 5 compares unseen-KITTI mappings from deterministic, flat-plane, SegNet, and VED methods with and without pitch perturbations.The black FOV mask is shared with Cityscapes because the input images are cropped to align the two cameras’ fields of view.

4) Ablation study:

The ablation examines variational sampling and higher-resolution mapping, linking the VED latent representation to robustness and map quality.

  • Ablation study: The variational-sampling ablation directly passes the encoder embedding to the decoder instead of randomly sampling and regularizing it.Table IV compares the proposed VED network with and without this modification.
  • Resolution: At 128×128 resolution, point-cloud-based approaches show artifact patterns, whereas VED produces maps with acceptable quality.At large distances, deterministic methods lose semantic information in some grids because registered points are insufficient for reliable majority voting.
  • Latent representation: The latent embedding has 512 dimensions and is intended to encode high-level semantic and spatial information.Correlated attributes make direct analysis difficult, motivating principal component analysis on 500 test-image embeddings.
  • PCA analysis: The network’s first principal axis mainly encodes drivable-space width, while the second mainly encodes the central non-free-space area’s size.The analysis connects the embedding to learned semantic-spatial understanding and to robustness against pitch and roll.

V. CONCLUSION

The conclusion presents VED as a real-time monocular system for semantic-metric top-view occupancy maps, with robustness and generalizability linked to variational sampling.

  • Conclusion: The proposed system estimates a top-view semantic-metric occupancy grid map from a single front-view monocular image.The method is designed as a real-time neural-network-based end-to-end mapper.
  • Conclusion: VED outperforms monocular flat-plane mapping and shows better robustness and generalizability than the canonical neural-network baseline.The conclusion attributes these properties to variational sampling over a relatively low-dimensional embedding space.
  • Conclusion: PCA analysis verifies that the latent embedding captures semantic and metric spatial information from monocular images.The embedding analysis supports the interpretation that the network learns both scene semantics and spatial structure.
  • Implications: The authors conclude that occupancy grids remain relevant and can complement, enhance, or partially replace traditional point-cloud processing techniques.They identify prediction of road layout beyond the camera’s FOV as future work.
Loading 1804.02176v3…