PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images

Lei Sun, Kailun Yang, Xinxin Hu, Weijian Hu, Kaiwei Wang

arXiv:2002.10570v2cs.CVcs.ROeess.IV

TL;DR

Autonomous-driving segmentation needs efficient RGB-D processing and recognition of unforeseen small obstacles, but prior RGB-D work often lacks real-time performance. RFNet combines complementary RGB-D fusion with multi-dataset training, achieving strong Cityscapes performance while detecting unexpected obstacles. It runs at 22Hz on full-resolution images and is reported as faster and more accurate than existing RGB-D fusion methods.

  • Problem

    Real-time RGB-D semantic segmentation remains limited, while fixed-category driving datasets overlook unforeseen small road hazards.

  • Method

    RFNet uses an attention-based RGB-D fusion architecture and multi-dataset training to add unexpected small-obstacle detection.

  • Results

    22Hz inference at full 2048×1024 Cityscapes resolution is reported alongside higher accuracy and speed than existing RGB-D fusion methods.

  • Takeaways & Limitations

    RFNet broadens recognizable road-scene classes with unexpected small-obstacle detection while retaining real-time operation.

Abstract

from arXiv · show

Semantic segmentation has made striking progress due to the success of deep convolutional neural networks. Considering the demands of autonomous driving, real-time semantic segmentation has become a research hotspot these years. However, few real-time RGB-D fusion semantic segmentation studies are carried out despite readily accessible depth information nowadays. In this paper, we propose a real-time fusion semantic segmentation network termed RFNet that effectively exploits complementary cross-modal information. Building on an efficient network architecture, RFNet is capable of running swiftly, which satisfies autonomous vehicles applications. Multi-dataset training is leveraged to incorporate unexpected small obstacle detection, enriching the recognizable classes required to face unforeseen hazards in the real world. A comprehensive set of experiments demonstrates the effectiveness of our framework. On Cityscapes, Our method outperforms previous state-of-the-art semantic segmenters, with excellent accuracy and 22Hz inference speed at the full 2048x1024 resolution, outperforming most existing RGB-D networks.

I. INTRODUCTION

Autonomous-driving segmentation must be accurate and efficient while detecting small, unforeseen road hazards. RFNet addresses this gap by combining RGB-D fusion with multi-dataset training for unexpected-obstacle detection.

  • Motivation: Unexpected debris, bricks, stones, and cargos are hazardous because they are small and vary in shape and type.Road debris caused more than 200,000 crashes, approximately 39,000 injuries, and more than 500 deaths in the cited U.S. period.
  • Motivation: RGB appearance alone is insufficient for obstacle detection, whereas depth provides complementary location and contour information.The paper motivates combining appearance and depth for road-driving perception.
  • Research gap: Most accuracy-oriented RGB-D segmentation methods target indoor scenes without guaranteeing the inference speed required by autonomous vehicles.This creates a need for real-time RGB-D semantic segmentation in driving settings.
  • Proposed framework: RFNet combines RGB-D semantic segmentation with pixel-wise unexpected small-obstacle detection and classifies 19 Cityscapes categories.The framework uses multi-dataset training to incorporate unexpected obstacles without limiting recognition to fixed road-scene categories.
  • Contributions: RFNet is presented as a real-time fusion network that improves accuracy over state-of-the-art methods while maintaining fast inference on Cityscapes.The introduction attributes its performance to the proposed fusion design and depth information.
  • Contributions: Depth complementary features improve accuracy over a single RGB stream, while multi-dataset training and the depth stream support unexpected small-object detection.These are identified as the network’s principal contributions.

A. RGB-D Semantic Segmentation

Prior RGB-D segmentation work established the value of depth but often emphasized indoor accuracy, while road-scene methods faced sparse depth and real-time constraints. RFNet is introduced against this background as a real-time fusion architecture.

  • RGB-D segmentation: Earlier RGB-D methods used channel stacking, depth-aware CNNs, lightweight depth branches, or multilevel feature fusion to combine modalities.These approaches demonstrate several strategies for exploiting depth complementary information.
  • RGB-D segmentation: RGB-D semantic segmentation can outperform single-RGB methods because depth contains location and contour information.The related work presents this as a recurring finding across prior studies.
  • Road-scene depth: Outdoor traffic-scene depth maps are sparser than indoor Kinect or RealSense depth maps, motivating specialized road-scene approaches.Prior work addressed this using stacked smoothed depth, scale selection, and depth enhancement.
  • Obstacle detection: Unexpected small hazardous obstacles are studied using geometric stereo methods and CNN-based approaches for appearance and context.Examples include Stixel, geometric point clustering, Bayesian fusion, and MergeNet.
  • RFNet: RFNet is positioned as a real-time fusion-based RGB-D semantic-segmentation architecture.Figure 2 summarizes the proposed network at the architecture level.

III. METHODOLOGY

RFNet uses separate RGB and depth encoders, attention-based feature fusion, spatial pyramid pooling, and an efficient decoder to combine complementary information while preserving real-time operation.

  • Network Architecture: RFNet uses independent RGB and depth branches with ResNet-18 backbones, treating RGB as the main branch and depth as subordinate.Depth features are fused into the RGB branch after each ResNet layer.
  • Network Architecture: The Attention Feature Complementary module uses channel attention to emphasize informative features and suppress less useful ones.An SE block provides the channel-attention mechanism for RGB-D fusion.
  • AFC Module: The AFC module exploits complementary information by weighting RGB and depth features before combining them.The module is designed to make the network focus on informative cross-modal features.
  • AFC Module: The fusion operation forms Z by adding attention-weighted RGB and depth feature maps.The weights are produced through global pooling, 1×1 convolution, and sigmoid activation.
  • Decoder: Spatial Pyramid Pooling enlarges the receptive field using aligned grids with different granularities before upsampling.This is intended to cover large objects while maintaining real-time speed.
  • Decoder: A simple decoder uses three upsampling modules and encoder skip connections to restore coarse semantic features to input resolution.The first two modules use bilinear interpolation to match skip-connection resolutions.

B. Multi-Dataset Learning

The framework uses multi-source learning to expand recognizable classes while addressing annotation conflicts between datasets. Cityscapes and Lost and Found are combined so RFNet predicts standard semantic classes alongside unexpected small obstacles.

  • Multi-source learning increases recognizable classes because real-world semantic labels cannot cover every class.
  • Simply mixing datasets can overfit one source because annotation types and sample amounts differ, causing incomplete segmentation.
  • Conflicting classes are separated into class sets A and B, with a standard dataset selected for conflicts during joint training.
  • The loss uses separate cross-entropy terms for class sets A and B, weighted by hyperparameter λ.
  • Using Cityscapes and Lost and Found, RFNet learns 19 Cityscapes classes plus the critical unexpected small-obstacle class.
  • The unexpected-obstacle formulation relies on shared context such as small size and partial surrounding free space to generalize beyond training examples.

A. Datasets

The study uses Cityscapes and Lost and Found, two RGB-D road-scene datasets with 2048×1024 imagery and complementary annotation schemes. Their disparity images are estimated from rectified stereo pairs using semi-global matching.

  • The experiments exploit two RGB-D semantic segmentation datasets: Cityscapes and Lost and Found.
  • Cityscapes contains 2975/500/1525 training, validation, and testing images with fine labels for 19 classes across 50 cities.
  • Lost and Found provides 2014 annotated frames with coarse free-space labels and fine-grained small-obstacle annotations.
  • Both datasets provide disparity images estimated with semi-global matching from rectified stereo image pairs.

B. Implementation Details

The implementation trains RFNet with Adam and cosine-annealed learning rates on a single 2080Ti GPU. Preprocessing crops invalid disparity regions, restores resolution, and applies standard geometric augmentation.

  • Training uses Adam with an initial learning rate of 4×10^-4, cosine annealing, and a final minimum of 1×10^-6.
  • The models are implemented on a single 2080Ti GPU with CUDA 10.0, cuDNN 7.6.0, and PyTorch 1.1.
  • Pre-trained parameters use four-times smaller learning rate and weight decay than the other parameters.
  • Invalid left and bottom disparity regions are cropped, then images are resized to the original resolution with bilinear upsampling.
  • Augmentation includes random scaling from 0.5 to 2, horizontal flipping, and random cropping to 768×768 outputs.

C. Results and Analysis

RFNet’s experiments show that its RGB-D fusion and multi-dataset training improve segmentation accuracy while supporting real-time road-scene inference and unexpected-obstacle recognition.

  • Ablation Study: 72.22% mIoU was achieved by RFNet, exceeding the single-RGB baseline and alternative fusion variants on the blended validation set.The single-RGB method reached 69.20%, RGB-D-Stack 65.20%, RGB-D-Fusion (concatenation) 68.67%, and RGB-RGB-Fusion 69.37%.
  • Numerical Performance Comparison: RFNet improved accuracy in most classes, with notable gains for fence, traffic light, terrain, truck, bus, train, and small obstacle compared with SwiftNet.The comparison used a blended Cityscapes and Lost and Found validation setting with 20 classes.
  • Numerical Performance Comparison: RFNet achieved higher accuracy and real-time performance than the compared RGB-D fusion networks on the Cityscapes validation set.The evaluation measured full-resolution 2048×1024 inference speed on a single RTX 2080Ti.
  • Qualitative Performance Study: Depth information helped RFNet detect small obstacles while reducing confusion between obstacles and manhole covers in qualitative comparisons with SwiftNet.RFNet also performed better on large objects such as buses and trucks in the reported examples.

MEAN IOU OF 20 CLASSES IN DIFFERENT DEPTH RANGES

Depth-aware feature processing improved RFNet’s segmentation across depth ranges and produced complementary representations for obstacle detection.

  • Depth-range comparison: RFNet improved mean IoU across all depth ranges compared with SwiftNet, with especially strong gains in close and middle ranges.The comparison covered 0–20m through 80–100m ranges and evaluated mean IoU over 20 classes.
  • Feature-map visualization: RFNet’s depth feature maps made small obstacles clearer and caused manhole covers to disappear relative to the RGB feature maps.The merged AFC feature maps combined advantages from both branches.

V. CONCLUSION

The paper concludes that RFNet combines complementary RGB-D information with multi-source training to support accurate real-time segmentation and unexpected small-obstacle detection.

  • Conclusion: RFNet operates at 22Hz on full-resolution Cityscapes images and 41.6Hz at half resolution on a single Nvidia GTX2080Ti GPU.The conclusion presents these speeds as suitable for autonomous-driving applications.
  • Conclusion: Multi-source training enables RFNet to detect unexpected small obstacles alongside 19 Cityscapes categories.The method is reported to enrich recognizable classes for unforeseen road hazards.
  • Conclusion: RFNet outperforms state-of-the-art RGB-D fusion methods in accuracy and speed according to the paper’s conclusion.The authors state that future work will streamline RFNet and deploy it on portable TPU devices.
Loading 2002.10570v2…