PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images

Lei Sun, Kailun Yang, Xinxin Hu, Weijian Hu, Kaiwei Wang

arXiv:2002.10570v2cs.CVcs.ROeess.IV

TL;DR

Autonomous-driving segmentation needs efficient RGB-D processing and recognition of unforeseen small obstacles, but prior RGB-D work often lacks real-time performance. RFNet combines complementary RGB-D fusion with multi-dataset training, achieving strong Cityscapes performance while detecting unexpected obstacles. It runs at 22Hz on full-resolution images and is reported as faster and more accurate than existing RGB-D fusion methods.

  • Problem

    Real-time RGB-D semantic segmentation remains limited, while fixed-category driving datasets overlook unforeseen small road hazards.

    Source: §I · p. 1

  • Method

    RFNet uses an attention-based RGB-D fusion architecture and multi-dataset training to add unexpected small-obstacle detection.

    Source: §I · p. 2, A. Network Architecture · p. 3, B. Multi-Dataset Learning · p. 3

  • Results

    22Hz inference at full 2048×1024 Cityscapes resolution is reported alongside higher accuracy and speed than existing RGB-D fusion methods.

    Source: §V · p. 7

  • Takeaways & Limitations

    RFNet broadens recognizable road-scene classes with unexpected small-obstacle detection while retaining real-time operation.

    Source: §I · p. 2, §V · p. 7

Abstract

from arXiv · show

Semantic segmentation has made striking progress due to the success of deep convolutional neural networks. Considering the demands of autonomous driving, real-time semantic segmentation has become a research hotspot these years. However, few real-time RGB-D fusion semantic segmentation studies are carried out despite readily accessible depth information nowadays. In this paper, we propose a real-time fusion semantic segmentation network termed RFNet that effectively exploits complementary cross-modal information. Building on an efficient network architecture, RFNet is capable of running swiftly, which satisfies autonomous vehicles applications. Multi-dataset training is leveraged to incorporate unexpected small obstacle detection, enriching the recognizable classes required to face unforeseen hazards in the real world. A comprehensive set of experiments demonstrates the effectiveness of our framework. On Cityscapes, Our method outperforms previous state-of-the-art semantic segmenters, with excellent accuracy and 22Hz inference speed at the full 2048x1024 resolution, outperforming most existing RGB-D networks.

I. INTRODUCTION

Autonomous-driving segmentation must be accurate and efficient while detecting small, unforeseen road hazards. RFNet addresses this gap by combining RGB-D fusion with multi-dataset training for unexpected-obstacle detection.

Motivation

  • Unexpected debris, bricks, stones, and cargos are hazardous because they are small and vary in shape and type.Road debris caused more than 200,000 crashes, approximately 39,000 injuries, and more than 500 deaths in the cited U.S. period.Source: §I · p. 1
  • RGB appearance alone is insufficient for obstacle detection, whereas depth provides complementary location and contour information.The paper motivates combining appearance and depth for road-driving perception.Source: §I · p. 1

Research gap

  • Most accuracy-oriented RGB-D segmentation methods target indoor scenes without guaranteeing the inference speed required by autonomous vehicles.This creates a need for real-time RGB-D semantic segmentation in driving settings.Source: §I · p. 2

Proposed framework

  • RFNet combines RGB-D semantic segmentation with pixel-wise unexpected small-obstacle detection and classifies 19 Cityscapes categories.The framework uses multi-dataset training to incorporate unexpected obstacles without limiting recognition to fixed road-scene categories.Source: §I · p. 2

Contributions

  • RFNet is presented as a real-time fusion network that improves accuracy over state-of-the-art methods while maintaining fast inference on Cityscapes.The introduction attributes its performance to the proposed fusion design and depth information.Source: §I · p. 2
  • Depth complementary features improve accuracy over a single RGB stream, while multi-dataset training and the depth stream support unexpected small-object detection.These are identified as the network’s principal contributions.Source: §I · p. 2

A. RGB-D Semantic Segmentation

Prior RGB-D segmentation work established the value of depth but often emphasized indoor accuracy, while road-scene methods faced sparse depth and real-time constraints. RFNet is introduced against this background as a real-time fusion architecture.

RGB-D segmentation

  • Earlier RGB-D methods used channel stacking, depth-aware CNNs, lightweight depth branches, or multilevel feature fusion to combine modalities.These approaches demonstrate several strategies for exploiting depth complementary information.Source: A. RGB-D Semantic Segmentation · p. 2
  • RGB-D semantic segmentation can outperform single-RGB methods because depth contains location and contour information.The related work presents this as a recurring finding across prior studies.Source: A. RGB-D Semantic Segmentation · p. 2

Road-scene depth

  • Outdoor traffic-scene depth maps are sparser than indoor Kinect or RealSense depth maps, motivating specialized road-scene approaches.Prior work addressed this using stacked smoothed depth, scale selection, and depth enhancement.Source: A. RGB-D Semantic Segmentation · p. 2

Obstacle detection

  • Unexpected small hazardous obstacles are studied using geometric stereo methods and CNN-based approaches for appearance and context.Examples include Stixel, geometric point clustering, Bayesian fusion, and MergeNet.Source: B. Unexpected Obstacle Detection for… · p. 2

RFNet

  • RFNet is positioned as a real-time fusion-based RGB-D semantic-segmentation architecture.Figure 2 summarizes the proposed network at the architecture level.Source: Figure 2 · p. 3

III. METHODOLOGY

RFNet uses separate RGB and depth encoders, attention-based feature fusion, spatial pyramid pooling, and an efficient decoder to combine complementary information while preserving real-time operation.

Network Architecture

  • RFNet uses independent RGB and depth branches with ResNet-18 backbones, treating RGB as the main branch and depth as subordinate.Depth features are fused into the RGB branch after each ResNet layer.Source: A. Network Architecture · p. 3
  • The Attention Feature Complementary module uses channel attention to emphasize informative features and suppress less useful ones.An SE block provides the channel-attention mechanism for RGB-D fusion.Source: A. Network Architecture · p. 3, Figure 3 · p. 3

AFC Module

  • The AFC module exploits complementary information by weighting RGB and depth features before combining them.The module is designed to make the network focus on informative cross-modal features.Source: A. Network Architecture · p. 3
  • The fusion operation forms Z by adding attention-weighted RGB and depth feature maps.The weights are produced through global pooling, 1×1 convolution, and sigmoid activation.Source: A. Network Architecture · p. 3

Decoder

  • Spatial Pyramid Pooling enlarges the receptive field using aligned grids with different granularities before upsampling.This is intended to cover large objects while maintaining real-time speed.Source: A. Network Architecture · p. 3
  • A simple decoder uses three upsampling modules and encoder skip connections to restore coarse semantic features to input resolution.The first two modules use bilinear interpolation to match skip-connection resolutions.Source: A. Network Architecture · p. 3

B. Multi-Dataset Learning

The framework uses multi-source learning to expand recognizable classes while addressing annotation conflicts between datasets. Cityscapes and Lost and Found are combined so RFNet predicts standard semantic classes alongside unexpected small obstacles.

A. Datasets

The study uses Cityscapes and Lost and Found, two RGB-D road-scene datasets with 2048×1024 imagery and complementary annotation schemes. Their disparity images are estimated from rectified stereo pairs using semi-global matching.

  • The experiments exploit two RGB-D semantic segmentation datasets: Cityscapes and Lost and Found.Source: A. Datasets · p. 4
  • Cityscapes contains 2975/500/1525 training, validation, and testing images with fine labels for 19 classes across 50 cities.Source: A. Datasets · p. 4
  • Lost and Found provides 2014 annotated frames with coarse free-space labels and fine-grained small-obstacle annotations.Source: A. Datasets · p. 4
  • Both datasets provide disparity images estimated with semi-global matching from rectified stereo image pairs.Source: A. Datasets · p. 4

B. Implementation Details

The implementation trains RFNet with Adam and cosine-annealed learning rates on a single 2080Ti GPU. Preprocessing crops invalid disparity regions, restores resolution, and applies standard geometric augmentation.

C. Results and Analysis

RFNet’s experiments show that its RGB-D fusion and multi-dataset training improve segmentation accuracy while supporting real-time road-scene inference and unexpected-obstacle recognition.

Ablation Study

  • 72.22% mIoU was achieved by RFNet, exceeding the single-RGB baseline and alternative fusion variants on the blended validation set.The single-RGB method reached 69.20%, RGB-D-Stack 65.20%, RGB-D-Fusion (concatenation) 68.67%, and RGB-RGB-Fusion 69.37%.Source: C. Results and Analysis · p. 5

Numerical Performance Comparison

  • RFNet improved accuracy in most classes, with notable gains for fence, traffic light, terrain, truck, bus, train, and small obstacle compared with SwiftNet.The comparison used a blended Cityscapes and Lost and Found validation setting with 20 classes.Source: C. Results and Analysis · p. 5, Table II · p. 6
  • RFNet achieved higher accuracy and real-time performance than the compared RGB-D fusion networks on the Cityscapes validation set.The evaluation measured full-resolution 2048×1024 inference speed on a single RTX 2080Ti.Source: C. Results and Analysis · p. 5, Table III · p. 6

Qualitative Performance Study

  • Depth information helped RFNet detect small obstacles while reducing confusion between obstacles and manhole covers in qualitative comparisons with SwiftNet.RFNet also performed better on large objects such as buses and trucks in the reported examples.Source: C. Results and Analysis · pp. 5–6

MEAN IOU OF 20 CLASSES IN DIFFERENT DEPTH RANGES

Depth-aware feature processing improved RFNet’s segmentation across depth ranges and produced complementary representations for obstacle detection.

Depth-range comparison

Feature-map visualization

V. CONCLUSION

The paper concludes that RFNet combines complementary RGB-D information with multi-source training to support accurate real-time segmentation and unexpected small-obstacle detection.

Conclusion

  • RFNet operates at 22Hz on full-resolution Cityscapes images and 41.6Hz at half resolution on a single Nvidia GTX2080Ti GPU.The conclusion presents these speeds as suitable for autonomous-driving applications.Source: §V · p. 7
  • Multi-source training enables RFNet to detect unexpected small obstacles alongside 19 Cityscapes categories.The method is reported to enrich recognizable classes for unforeseen road hazards.Source: §V · p. 7
  • RFNet outperforms state-of-the-art RGB-D fusion methods in accuracy and speed according to the paper’s conclusion.The authors state that future work will streamline RFNet and deploy it on portable TPU devices.Source: §V · p. 7

Related papers

Loading 2002.10570v2…