PaperlayerView exact revision on arXiv

Source-linked AI summary

A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye View

Lennart Reiher, Bastian Lampe, Lutz Eckstein

arXiv:2005.04078v1cs.CVcs.LG

TL;DR

IPM’s flat-world assumption distorts three-dimensional objects, while monocular cameras make distance estimation challenging. This paper learns a semantically segmented BEV transformation from multiple vehicle-mounted cameras using synthetic training without manual BEV labels; the approach improves on the homography baseline and is demonstrated on real-world data.

  • Problem

    IPM assumes a flat world, distorting three-dimensional objects and making their positions harder to estimate accurately.

    Source: §I · p. 1

  • Method

    The approach learns to transform semantically segmented images from multiple vehicle-mounted cameras into a BEV, training on synthetic data without manual BEV labels.

    Source: §I · p. 2

  • Results

    Both proposed variations improve environment-perception results over the homography baseline, and the trained models are successfully applied to real-world data.

    Source: §I · p. 2, A. Results on Synthetic Data · p. 5

  • Takeaways & Limitations

    The semantically segmented BEV preserves object shapes for locating dynamic objects and marks areas occluded in the original camera views.

    Source: §I · p. 2

Abstract

from arXiv · show

Accurate environment perception is essential for automated driving. When using monocular cameras, the distance estimation of elements in the environment poses a major challenge. Distances can be more easily estimated when the camera perspective is transformed to a bird's eye view (BEV). For flat surfaces, Inverse Perspective Mapping (IPM) can accurately transform images to a BEV. Three-dimensional objects such as vehicles and vulnerable road users are distorted by this transformation making it difficult to estimate their position relative to the sensor. This paper describes a methodology to obtain a corrected 360{\deg} BEV image given images from multiple vehicle-mounted cameras. The corrected BEV image is segmented into semantic classes and includes a prediction of occluded areas. The neural network approach does not rely on manually labeled data, but is trained on a synthetic dataset in such a way that it generalizes well to real-world data. By using semantically segmented images as input, we reduce the reality gap between simulated and real-world data and are able to show that our method can be successfully applied in the real world. Extensive experiments conducted on the synthetic data demonstrate the superiority of our approach compared to IPM. Source code and datasets are available at https://github.com/ika-rwth-aachen/Cam2BEV

I. INTRODUCTION

The paper addresses distortions caused by applying IPM to three-dimensional objects and proposes learning a semantically segmented BEV transformation from multiple vehicle-mounted cameras. Its contributions include training without manual BEV labels and a successful real-world application.

I. INTRODUCTION

  • IPM’s flat-world assumption distorts three-dimensional objects, impeding accurate localization of vehicles and vulnerable road users.The paper also identifies a simulation-to-real gap and the manual-labeling demands of deep-learning segmentation as challenges.Source: §I · p. 1
  • The method transforms images from multiple vehicle-mounted cameras into a semantically segmented BEV without IPM’s flatness-assumption errors.The output also color-codes areas occluded in the original camera views.Source: §I · p. 2
  • The approach uses semantically segmented camera images to train neural networks on synthetic data without manually labeled BEV images.The paper’s contributions include comparing two neural-network variations, one designed for this task.Source: §I · p. 2
  • The authors report a successful real-world application of their trained models.The semantic BEV preserves object shapes and includes color-coded unknown areas occluded in the camera views.Source: §I · p. 2

II. RELATED WORK

Prior work includes geometry-based surround-view transformations, deep-learning approaches to BEV correction and scene understanding, and a blog-level proposal for transforming multiple segmented views. The paper characterizes the latter as lacking detailed testing and real-world application.

II. RELATED WORK

  • Automotive surround-view work synthesizes multiple camera images into a top-down view, generally focusing on accurate depiction of the ground.The cited examples are and.Source: §II · p. 2
  • Deep-learning work addresses IPM correction, [7], vehicle detection in BEV, 3D bounding boxes, and coarse static semantic maps.The approaches in and seek to remove dynamic traffic participants, whereas [7] synthesizes a road scene including them.Source: §II · p. 2
  • The only identified prior effort to directly transform multiple semantically segmented images to BEV was a blog article, without detailed testing or real-world application.Its fully convolutional autoencoder is also described as having a relatively low range for accurate object detection.Source: §II · p. 2

III. METHODOLOGY

The methodology uses multiple camera views and IPM as an intermediate guide for learning a BEV output. It treats occlusion explicitly by adding an occlusion class to the BEV labels.

III. METHODOLOGY

  • The methodology addresses multiple viewpoints because concatenating camera images as channels would create spatial inconsistency, and it uses IPM as an intermediate guiding view.The authors note that IPM contains errors but resembles a ground-truth BEV image.Source: §III · p. 2

A. Dealing with Occlusions

  • Traffic participants and static obstacles can hide parts of the scene, making those areas difficult to predict from vehicle-mounted camera images alone.The paper gives the example of a truck blocking the view of what lies in front of it.Source: A. Dealing with Occlusions · p. 2
  • A dedicated occlusion class is added to BEV labels, and objects are marked occluded only when they are occluded in every camera view.Partially occluded objects remain labeled as completely visible.Source: A. Dealing with Occlusions · pp. 2–3

B. Projective Preprocessing

The section derives projective mappings from camera images to the road plane using camera projection parameters and a planar-surface assumption. It applies IPM separately to each camera and merges the views into a homography image covering the ground-truth BEV field of view.

B. Projective Preprocessing

  • The projection matrix P maps homogeneous world coordinates to image coordinates and encodes camera intrinsics and extrinsics.The intrinsic parameters include focal length; the extrinsics are rotation and translation relative to the world frame.Source: B. Projective Preprocessing · p. 3
  • Assuming a road-plane transformation M makes PM invertible, image coordinates can be mapped back to road-plane coordinates.The projection from world to image is generally not invertible because multiple world points can correspond to one image pixel.Source: B. Projective Preprocessing · p. 3
  • Camera calibration methods can determine P for real-world cameras.Source: B. Projective Preprocessing · p. 3
  • Each camera image is transformed separately by IPM to the ground-truth BEV field of view, then merged into one homography image.Pixels in overlapping regions are selected arbitrarily from one of the transformed images.Source: B. Projective Preprocessing · p. 3

C. Variation 1: Single-Input Model

The first variation precomputes a homography image to provide spatial correspondence with the target BEV, then uses a single-input CNN to correct IPM errors. DeepLabv3+ is tested with two different backbones.

C. Variation 1: Single-Input Model

D. Variation 2: Multi-Input Model

The second variation processes the untransformed camera images in separate input streams and integrates projective transformations into the network. Its uNetXST architecture transforms and fuses feature maps across streams before producing one BEV output.

D. Variation 2: Multi-Input Model

  • Unlike the first variation, the second processes untransformed camera images, extracting features in camera views and integrating projective transformations to address spatial inconsistency.Source: D. Variation 2: Multi-Input Model · p. 3
  • The architecture illustration shows two pooling and upsampling layers, while the trained network contains four of each.Source: Figure 3 · p. 4
  • uNetXST replicates the encoder for each input image, transforms feature maps with fixed IPM homographies, and fuses streams through skip connections into a single decoder.The architecture extends U-Net and has approximately 9.6M trainable parameters.Source: D. Variation 2: Multi-Input Model · pp. 3–4
  • The ϑ-block transforms each camera stream's feature maps into the same field of view as the ground-truth BEV before concatenation.Source: Figure 4 · p. 4

IV. EXPERIMENTAL SETUP

The networks are trained entirely on simulated data generated in Virtual Test Drive. The data acquisition setup provides both a four-camera surround-view dataset and a separate front-camera dataset for later real-world testing.

IV. EXPERIMENTAL SETUP

  • The methodology is trained entirely on simulated data generated in Virtual Test Drive, using a recording toolchain to produce images and corresponding labels.Source: §IV · p. 4, A. Data Acquisition · p. 4

A. Data Acquisition

  • The surround-view dataset uses four wide-angle cameras covering 360° and a virtual drone camera for ground-truth BEV over an approximately 70 m × 44 m field of view.Source: A. Data Acquisition · p. 4
  • The surround-view images and ground truth are recorded at 964 px × 604 px, with nine semantic classes and approximately 33,000 training and 3,700 validation samples.The simulated world's static elements remain the same between training and validation data.Source: A. Data Acquisition · p. 4
  • A separate front-camera dataset for later real-world testing contains three visible-area classes and approximately 32,000 training and 3,200 validation samples.Its ground-truth images are left-aligned with the ego vehicle and cover only the area in front.Source: A. Data Acquisition · pp. 4–5

B. Training Setup

Training uses cropped, resized one-hot inputs and a class-weighted loss; performance is evaluated with class IoU and mean IoU.

B. Training Setup

  • Inputs and target labels are center-cropped to 2:1 and resized to 512 × 256 px; input images are one-hot encoded.Source: B. Training Setup · p. 5
  • The loss weights semantic classes by the logarithm of their relative occurrence to counter class imbalance.Source: B. Training Setup · p. 5

C. Evaluation Metrics

  • Class IoU scores are averaged into Mean Intersection-over-Union (MIoU), the main metric for semantic-class prediction.Source: C. Evaluation Metrics · p. 5

V. RESULTS AND DISCUSSION

The synthetic validation study compares model variations with homography IPM and qualitatively tests simulated-to-real generalization. uNetXST leads validation MIoU, and the proposed models outperform IPM while correcting its visible distortions.

A. Results on Synthetic Data

  • uNetXST achieves the highest validation MIoU; both proposed variations generally outperform homography IPM, while IPM-free versions perform worse than their counterparts.Source: Table I · p. 5, A. Results on Synthetic Data · p. 5
  • All three networks perform best on large-area classes such as road and vegetation, while bikes and especially persons remain difficult to predict and localize.The passage attributes difficulty to their small BEV size and low training-set occurrence; uNetXST's results suggest benefit from processing raw, untransformed images.Source: A. Results on Synthetic Data · p. 5, Table II · p. 5
  • In the validation examples, the homography images show IPM's flat-world distortions, whereas both models produce corrected BEV scenes and reasonably predict camera-occluded areas.Source: Figure 5 · p. 6, A. Results on Synthetic Data · p. 6
  • Vehicles are well localized in the first example, but the intersection scene has less accurate object dimensions and an almost fully occluded car is not properly detected.Source: A. Results on Synthetic Data · p. 6

B. Real-World Application

On two real-world scenes, both models make reasonable traffic-participant predictions, with uNetXST producing smoother, qualitatively better results. The fixed IPM transformation may be miscalibrated because camera pose changes with vehicle dynamics.

B. Real-World Application

  • Both models reasonably predict vehicle locations and dimensions in two real-world scenes; uNetXST produces smoother, qualitatively better results, especially for distant vehicles.Source: B. Real-World Application · p. 6, Figure 6 · p. 7
  • Vehicle dynamics can change camera pose relative to the road, potentially miscalibrating the fixed IPM transformation; dynamic transformation changes could improve real-world results.Source: B. Real-World Application · p. 6

VI. CONCLUSION

The methodology transforms multiple vehicle-camera images into semantically segmented BEV images, removes flatness-assumption errors, and predicts occluded areas. Further real-world evaluation is needed with a 360° multi-camera setup and strong segmentation beyond front-camera images.

VI. CONCLUSION

  • The method produces semantically segmented BEV images from multiple vehicle-mounted cameras, removes IPM flatness-assumption errors, and predicts occluded areas.Synthetic training and semantically segmented camera inputs support real-world application without manual BEV-image labeling; uNetXST uses multiple inputs and in-network transformations.Source: §VI · p. 7
  • Real-world testing remains needed with a 360° multi-camera setup, which requires good semantic segmentation beyond front-camera images.Source: §VI · p. 7
Loading 2005.04078v1…