PaperlayerView exact revision on arXiv

Source-linked AI summary

RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving

Khaled El Madawy, Hazem Rashed, Ahmad El Sallab, Omar Nasr, Hanan Kamel, Senthil Yogamani

arXiv:1906.00208v2cs.CVcs.AIcs.LGcs.RO

TL;DR

Classical point-cloud segmentation pipelines are described as prone to low performance, while 3D semantic segmentation combines object localization and classification. The paper maps RGB images to a LiDAR polar-grid representation and fuses them with point clouds; on KITTI, fusion improves SqueezeSeg and PointSeg by 10%.

  • Problem

    Classical point-cloud segmentation pipelines combine ground/foreground separation, clustering, and hand-crafted classification, and are described as prone to low performance; object classification supports interaction planning.

    Source: §I · p. 1

  • Method

    The paper remaps RGB images to the LiDAR polar-grid representation and builds early- and mid-level fusion architectures for point-cloud segmentation.

    Source: §I · p. 1, §V · p. 5

  • Results

    10%: fusion improved SqueezeSeg and PointSeg on KITTI, compared with their LiDAR-only baselines.

    Source: §V · p. 5

  • Takeaways & Limitations

    The reported evaluation covers two architectures on KITTI, with more sophisticated fusion and use of all color pixels left for future work.

    Source: §V · p. 5

Abstract

from arXiv · show

LiDAR has become a standard sensor for autonomous driving applications as they provide highly precise 3D point clouds. LiDAR is also robust for low-light scenarios at night-time or due to shadows where the performance of cameras is degraded. LiDAR perception is gradually becoming mature for algorithms including object detection and SLAM. However, semantic segmentation algorithm remains to be relatively less explored. Motivated by the fact that semantic segmentation is a mature algorithm on image data, we explore sensor fusion based 3D segmentation. Our main contribution is to convert the RGB image to a polar-grid mapping representation used for LiDAR and design early and mid-level fusion architectures. Additionally, we design a hybrid fusion architecture that combines both fusion algorithms. We evaluate our algorithm on KITTI dataset which provides segmentation annotation for cars, pedestrians and cyclists. We evaluate two state-of-the-art architectures namely SqueezeSeg and PointSeg and improve the mIoU score by 10 % in both cases relative to the LiDAR only baseline.

I. INTRODUCTION

3D semantic segmentation combines localization and classification, making it important for autonomous driving. This paper studies camera–LiDAR fusion to add color information to LiDAR-based segmentation.

I. INTRODUCTION

  • 3D semantic segmentation is critical for autonomous driving because it simultaneously localizes and classifies objects.The introduction illustrates this role with the different responses required for a pedestrian and a vehicle appearing after occlusion.Source: §I · p. 1
  • Camera–LiDAR fusion is explored through early fusion of raw inputs and mid-fusion of separately extracted features.The paper also proposes a hybrid strategy combining the two fusion approaches.Source: §I · p. 1
  • The paper constructs early- and mid-fusion CNN architectures to study color information and evaluates fusion on SqueezeSeg and PointSeg.Its stated contributions include an RGB fusion representation and a hybrid fusion strategy.Source: §I · p. 1

II. RELATED WORK

Image semantic segmentation research includes approaches for pixel-wise classification, dense output, and preserving resolution. The reviewed color-based methods typically do not use depth information.

A. Semantic Segmentation for Images

  • Image-segmentation methods reviewed include patch-wise classification, Laplacian-pyramid features, and end-to-end dense prediction.The passage also describes heatmap upsampling and encoder–decoder designs with skip connections or kept pooling indices.Source: A. Semantic Segmentation for Images · p. 2
  • SegNet uses an encoder–decoder that upsamples feature maps using kept indices from corresponding encoder layers.The passage also notes skip connections as a way to avoid losing resolution.Source: A. Semantic Segmentation for Images · p. 2
  • The reviewed color-based segmentation improvements typically do not use depth information.Source: A. Semantic Segmentation for Images · p. 2

B. Semantic Segmentation For Point Clouds

Point-cloud semantic segmentation is less studied than image segmentation, and LiDAR sparsity motivates alternative representations. Those representations trade off structural information, computational demands, and use of the scan structure.

B. Semantic Segmentation For Point Clouds

  • LiDAR point clouds are sparse, and voxel, bird-view, and polar-grid representations are among the approaches used to simplify them.Bird-view flattens the point cloud and loses height information, while voxelization requires substantial memory and processing power.Source: B. Semantic Segmentation For Point… · p. 2
  • PointNet and PointNet++ treat point clouds as unordered sets, providing viewpoint invariance but not accounting for LiDAR scan structure.The passage notes that the specific perspective matters in autonomous driving.Source: B. Semantic Segmentation For Point… · p. 2
  • SqueezeSeg uses a spherical polar-grid representation that models LiDAR scanning and yields a relatively dense 2D plane.This representation made it possible to leverage image-based semantic-segmentation architectures.Source: B. Semantic Segmentation For Point… · p. 2

III. PROPOSED ALGORITHM

The proposed representation maps camera color onto LiDAR’s polar-grid plane using sensor calibration. This produces aligned RGB and LiDAR features while leaving unmapped image pixels unused.

III. PROPOSED ALGORITHM

  • Figure 4 contrasts the SqueezeSeg-based unimodal baseline and early-fusion architecture with the proposed mid-fusion architecture.Early fusion keeps the baseline architecture but changes the number of input planes.Source: Figure 4 · p. 3
  • The LiDAR scan is reshaped into a polar grid whose height is the number of layers and width is the number of points detected per layer.The described front-view subset uses a 90° field of view, 0.175° yaw resolution, 64 layers, and five LiDAR features: X, Y, Z, I, and D.Source: A. Polar Grid Map representation for… · p. 3
  • Camera pixels are mapped to LiDAR scan points using relative calibration, adding RGB to the XYZID tensor to form a 64x512x8 representation.The corresponding ground-truth tensor is 64x512x1.Source: A. Polar Grid Map representation for… · pp. 3–4
  • The RGB-to-polar-grid mapping does not use color pixels that have no corresponding LiDAR point.The representation can also scale to multiple cameras covering 360° around the car.Source: A. Polar Grid Map representation for… · pp. 3–4

B. LiDAR baseline architecture

The LiDAR-only SqueezeSeg baseline uses a lightweight CNN to produce 3D semantic-segmentation probabilities from a five-channel point-cloud tensor.

B. LiDAR baseline architecture

C. Early-fusion architecture

The section compares raw-input early fusion with encoder-level mid-fusion, then introduces a hybrid that combines the approaches.

C. Early-fusion architecture

  • Early fusion concatenates three RGB channels with XYZDI into an eight-channel input for joint CNN feature extraction.The authors note that this input structure cannot directly leverage pre-training on large unimodal datasets such as ImageNet.Source: C. Early-fusion architecture · p. 4

D. Mid-Fusion architecture

  • Mid-fusion uses separate modality encoders and concatenates their processed feature maps, doubling encoder parameters relative to early fusion.Source: D. Mid-Fusion architecture · p. 4
  • The hybrid concatenates RGB with LiDAR depth and intensity channels and significantly improves performance over the baseline.Source: D. Mid-Fusion architecture · p. 4
  • The separate encoders may be leveraged for other tasks in their respective modalities, but this mid-fusion model showed negligible accuracy increase in experiments.Source: D. Mid-Fusion architecture · p. 4

IV. EXPERIMENTS

The experiments use KITTI raw driving data and its provided split to evaluate segmentation of four labeled groups.

A. Datasets

  • KITTI raw supplies 12,919 frames, split into 10,128 training frames and 2,791 validation frames using the split from.Source: A. Datasets · p. 4
  • The labeled groups are Car, Pedestrian, Cyclist, and Background, with Car including cars, vans, and trucks.Source: A. Datasets · p. 4
  • The selected vehicle classes are motivated by their collision risk, and points inside annotated 3D bounding boxes receive the corresponding class label.Source: A. Datasets · p. 4

B. Experimental Setup

The experiments compare LiDAR-only and fused models on KITTI using mean IoU and report segmentation results, class-specific changes, and runtime. Fusion improves over LiDAR-only baselines, with the strongest stated SqueezeSeg result from the hybrid approach.

B. Experimental Setup

  • The setup uses a 64×512 polar-grid representation, y-axis random flips, learning rate 0.01, momentum 0.9, and average class-wise IoU.Source: B. Experimental Setup · p. 4

C. Results

  • XYZDI + DIRGB yields the best SqueezeSeg result, with a 3.7% absolute increase and around 10% relative increase in mean IoU over the XYZDI baseline.XYZDIRGB alone gives an absolute increase of 3% in mean IoU over the baseline.Source: C. Results · p. 5, Table I · p. 5
  • Both early and mid fusion outperform LiDAR-only results with PointSeg, although early fusion outperformed mid fusion in these experiments.The authors hypothesize the atypical enlargement-layer concatenation in PointSeg may account for the unexpected ordering.Source: C. Results · p. 5
  • The paper reports that no-fusion struggles on small-volume classes; fusion improves Pedestrian and Cyclist results, with architecture-dependent gains.For Pedestrian, early and mid fusion yield 3.3% and 5.8%, respectively; Cyclist mIoU improves by 2.3% and 2.5%. PointSeg improvements are 3% and 2.8%.Source: C. Results · p. 5
  • Inference takes around 10 ms per scan; early fusion takes nearly the no-fusion runtime, while mid-fusion adds 3 ms in both architectures.Source: C. Results · p. 5

V. CONCLUSIONS

The paper remaps RGB images to LiDAR polar-grid representations and combines camera and LiDAR data through early- and mid-level fusion. On KITTI, both SqueezeSeg and PointSeg improve by 10% in mIoU relative to LiDAR-only baselines.

V. CONCLUSIONS

  • 10% mIoU improvement over LiDAR-only baselines is reported for both SqueezeSeg and PointSeg on KITTI.The paper reports this as its experimental result.Source: §V · p. 5
  • The approach remaps RGB images to the LiDAR polar-grid representation and constructs early- and mid-level fusion architectures.Source: §V · p. 5
  • Future work will explore fusion architectures using network architecture search and use all available color pixels.Source: §V · p. 5

SUPPLEMENTARY MATERIAL

The supplementary PointSeg comparison presents no-fusion, early-fusion, and mid-fusion outputs alongside ground truth. The text reports improvements in cyclist and car segmentation.

SUPPLEMENTARY MATERIAL

  • The PointSeg comparison reports improvement in cyclist and car segmentation.Source: SUPPLEMENTARY MATERIAL · p. 7
  • The figure compares no-fusion, early-fusion, and mid-fusion outputs with ground truth.Source: Figure 6 · p. 7
  • Red, green, violet, and light blue represent None, Cars, Cyclists, and Pedestrians, respectively.Source: Figure 6 · p. 7
Loading 1906.00208v2…