Source-linked AI summary
SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection
Rui Fan, Hengli Wang, Peide Cai, Ming Liu
TL;DR
Freespace detection supports safe autonomous navigation, but existing approaches rely on RGB, depth, or learned segmentation without the paper’s proposed surface-normal integration. The paper introduces SNE, RoadSeg, and the R2D dataset, and reports that SNE benefits all evaluated CNNs while SNE-RoadSeg achieves the best overall performance across datasets.
Problem
Freespace detection requires pixel-level drivable-area classification for autonomous navigation, while existing methods use geometric models or CNN segmentation and lack the proposed surface-normal integration.
Method
The paper estimates surface normals from dense depth/disparity images with SNE, fuses RGB and surface-normal features with RoadSeg, and releases the R2D dataset.
Results
SNE benefits all ten evaluated state-of-the-art CNNs, while RoadSeg outperforms the other evaluated CNNs across the reported datasets.
Takeaways & Limitations
Surface-normal information and densely connected feature fusion are supported as effective components for freespace detection in the evaluated settings.
Takeaways & Limitations
Similar surface normals on freespace and sidewalks can cause sidewalk pixels to be mistaken for freespace, especially when their textures are similar.
Abstract
from arXiv · showhide
Freespace detection is an essential component of visual perception for self-driving cars. The recent efforts made in data-fusion convolutional neural networks (CNNs) have significantly improved semantic driving scene segmentation. Freespace can be hypothesized as a ground plane, on which the points have similar surface normals. Hence, in this paper, we first introduce a novel module, named surface normal estimator (SNE), which can infer surface normal information from dense depth/disparity images with high accuracy and efficiency. Furthermore, we propose a data-fusion CNN architecture, referred to as RoadSeg, which can extract and fuse features from both RGB images and the inferred surface normal information for accurate freespace detection. For research purposes, we publish a large-scale synthetic freespace detection dataset, named Ready-to-Drive (R2D) road dataset, collected under different illumination and weather conditions. The experimental results demonstrate that our proposed SNE module can benefit all the state-of-the-art CNNs for freespace detection, and our SNE-RoadSeg achieves the best overall performance among different datasets.
1 Introduction
Freespace detection is fundamental to autonomous driving because pixel-level drivable-area predictions support safe navigation modules. The paper addresses this task with SNE, RoadSeg, and the R2D dataset.
- Motivation: Freespace detection classifies each pixel in RGB or depth/disparity images as drivable or undrivable.These pixel-level results are used by trajectory prediction and path planning modules.
- Existing approaches: Existing approaches include geometry-based traditional methods and CNN-based semantic driving scene segmentation.Recent work also uses data-fusion CNNs to improve semantic image segmentation.
- Contributions: SNE infers surface normal information from dense disparity/depth images with high precision and efficiency.The module is designed as an additional source of geometric information for freespace detection.
- Contributions: RoadSeg fuses RGB and inferred surface normal information in a data-fusion CNN architecture for accurate freespace detection.The paper also introduces the publicly available R2D dataset with 11430 RGB-depth image pairs under varied illumination and weather conditions.
- Evaluation: The paper evaluates ten state-of-the-art CNNs, with and without SNE, across KITTI, SYNTHIA, and R2D datasets.The experiments assess both the SNE module and the proposed RoadSeg architecture.
2 Related Work
Related work develops encoder-decoder and multimodal CNNs for dense semantic segmentation, while SNE-RoadSeg combines RGB and surface-normal processing with dense feature fusion.
- Semantic segmentation CNNs: FCN, U-Net, and SegNet established encoder-decoder designs for end-to-end semantic image segmentation.U-Net uses skip connections to recover spatial resolution, while SegNet uses an encoder-decoder architecture.
- Semantic segmentation CNNs: DeepLabv3+ uses depth-wise separable convolutions in ASPP and the decoder, but its generated feature maps may be insufficiently dense for autonomous driving.DenseASPP was designed to address this resolution issue.
- Specialized architectures: DUpsampling reduces computational costs by downsampling fused features before merging them, while GSCNN uses regular and shape branches.These architectures modify decoding or branch structure for dense prediction.
- Data-fusion CNNs: FuseNet combines feature maps from RGB and depth encoders through element-wise summation for RGB-D semantic segmentation.Depth-aware CNN instead uses depth-aware convolution and pooling to incorporate geometric information based on pixel depth similarity.
- SNE-RoadSeg architecture: SNE-RoadSeg uses separate RGB and surface-normal encoders, hierarchically sums their features, and applies densely-connected skip connections in the decoder.The decoder restores feature-map resolution before sigmoid prediction.
- Multimodal segmentation: MFNet and RTFNet target real-time semantic driving-scene segmentation using RGB-thermal data, with RTFNet emphasizing sharp boundaries through decoder shortcuts.Their focus differs from SNE-RoadSeg’s RGB and inferred surface-normal fusion.
3 SNE-RoadSeg
SNE estimates surface normals from dense depth or disparity information, while RoadSeg fuses RGB and inferred-normal features for freespace segmentation. The architecture combines hierarchical encoder fusion with densely connected decoder skips to restore resolution.
- 3.1 SNE: SNE estimates surface normals from dense depth or disparity images using local geometric relationships and neighborhood aggregation.Inverse depth is processed with horizontal and vertical gradient filters; neighboring normalized normals are aggregated on a unit sphere.
- 3.1 SNE: The surface normal estimator derives image-space normal components by differentiating the perspective-camera geometry and filtering inverse depth.Disparity can be used because it is inversely proportional to depth.
- 3.1 SNE: The optimal surface normal is selected where neighboring normalized surface normals are most concentrated on the unit sphere.The estimator represents the normal with inclination and azimuth and obtains it by minimizing an energy under the stated normal-angle assumption.
- 3.2 RoadSeg: Densely connected skip connections relax U-Net's same-scale aggregation constraint during decoder feature fusion.RoadSeg is inspired by DenseNet and uses these connections for flexible feature fusion.
- 3.2 RoadSeg: RoadSeg uses separate RGB and surface-normal encoders, hierarchically fuses their feature maps by element-wise summation, and applies densely connected decoder skips.The decoder fuses features again to restore feature-map resolution before producing a probability map.
4 Experiments
Experiments evaluate SNE and SNE-RoadSeg across DIODE, KITTI, SYNTHIA, and R2D datasets, including ablations against single-modal and data-fusion CNNs. SNE improves surface-normal estimation and freespace detection, while SNE-RoadSeg achieves strong cross-dataset and KITTI benchmark performance.
- Datasets and setup: The evaluation uses KITTI, SYNTHIA, and R2D road datasets, with R2D providing 11,430 stereo-image pairs across varied illumination and weather conditions.KITTI and SYNTHIA provide real-world and synthetic RGB-D data, respectively; R2D is generated with CARLA4.
- SNE evaluation: SNE outperforms SRI and LINE-MOD for surface-normal estimation in both indoor and outdoor DIODE scenarios.The comparison uses qualitative and quantitative results with average angular error evaluation.
- SNE effectiveness: IoU increases approximately 2–12% for single-modal CNNs and 1–7% for data-fusion CNNs when SNE is embedded.F-score increases by approximately 1–7% for single-modal CNNs and 1–4% for data-fusion CNNs.
- RoadSeg evaluation: RoadSeg outperforms the other evaluated CNNs, while densely connected skip connections support feature fusion and smoother gradient flow.RoadSeg with SNE also outperforms the other CNNs with SNE embedded.
- KITTI benchmark: On KITTI, SNE-RoadSeg achieves the highest MaxF, AP, and PRE, while LC-CRF achieves the best REC; the method ranks second overall.The comparison includes five published state-of-the-art CNNs.
- Limitations: SNE-RoadSeg can mistake sidewalks for freespace when their surface normals and textures are similar.The paper suggests surface-normal gradient features as a possible way to use the height difference at the boundary.
5 Conclusion
The paper contributes SNE for inferring surface normals, RoadSeg for fusing RGB and surface-normal information, and a public synthetic dataset. Experiments show SNE benefits ten state-of-the-art CNNs, while SNE-RoadSeg performs best among the evaluated CNNs for drivable road detection.
- SNE infers surface normal information from depth or disparity images with high precision and efficiency.
- RoadSeg fuses RGB and surface normal information for accurate freespace detection.
- The paper releases a publicly available synthetic dataset for semantic driving scene segmentation.
- SNE improved freespace detection in all ten evaluated state-of-the-art CNNs.
- RoadSeg was most compatible with SNE and outperformed the other CNNs when detecting drivable road regions.