Source-linked AI summary
(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network
Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, Bingbing Liu
TL;DR
LiDAR semantic segmentation must provide accurate scene understanding despite unstructured, sparse data and the limitations of vision sensors. AF2-S3Net addresses this with an end-to-end sparse CNN combining attentive feature fusion and adaptive feature selection. The paper reports state-of-the-art performance across several benchmarks, including SemanticKITTI.
Problem
LiDAR semantic segmentation is challenged by unstructured, partially sparse point clouds, while vision sensors have limitations including poor-lighting sensitivity, missing depth, and limited field of view.
Method
AF2-S3Net is an end-to-end 3D sparse convolution network using encoder attentive feature fusion and decoder adaptive feature selection with feature-map re-weighting.
Results
The model achieves state-of-the-art performance in experiments on SemanticKITTI, nuScenes-lidarseg, and ModelNet40.
Takeaways & Limitations
The proposed attention blocks learn local and global context while emphasizing fine details in LiDAR point clouds.
Abstract
from arXiv · showhide
Autonomous robotic systems and self driving cars rely on accurate perception of their surroundings as the safety of the passengers and pedestrians is the top priority. Semantic segmentation is one the essential components of environmental perception that provides semantic information of the scene. Recently, several methods have been introduced for 3D LiDAR semantic segmentation. While, they can lead to improved performance, they are either afflicted by high computational complexity, therefore are inefficient, or lack fine details of smaller instances. To alleviate this problem, we propose AF2-S3Net, an end-to-end encoder-decoder CNN network for 3D LiDAR semantic segmentation. We present a novel multi-branch attentive feature fusion module in the encoder and a unique adaptive feature selection module with feature map re-weighting in the decoder. Our AF2-S3Net fuses the voxel based learning and point-based learning into a single framework to effectively process the large 3D scene. Our experimental results show that the proposed method outperforms the state-of-the-art approaches on the large-scale SemanticKITTI benchmark, ranking 1st on the competitive public leaderboard competition upon publication.
1. Introduction
3D LiDAR semantic segmentation addresses the need for reliable scene understanding despite vision-sensor limitations, but LiDAR’s unstructured and partially sparse data remain challenging. AF2-S3Net combines attentive feature fusion and adaptive feature selection in an end-to-end sparse CNN, with experiments spanning three benchmarks.
- Motivation: Vision sensors struggle with poor lighting, variable sensitivity, missing depth information, and limited field of view, whereas LiDAR provides depth across lighting conditions.The passage frames LiDAR as a reliable information source for safety-critical tasks such as self-driving.
- Motivation: Unstructured and partially sparse LiDAR scans make semantic segmentation difficult, motivating methods that handle varying point densities and preserve granular information.The paper identifies SemanticKITTI as an available dataset supporting research on LiDAR scene understanding.
- Contributions: AF2-S3Net is an end-to-end 3D sparse CNN with a multi-branch attentive feature fusion module that learns global context and local details.The fusion module is placed in the encoder.
- Contributions: Its decoder uses adaptive feature selection with feature-map re-weighting to emphasize contextual information and improve generalizability.This module is presented as a complementary attention mechanism to encoder feature fusion.
- Evaluation: The evaluation covers semantic segmentation and classification on SemanticKITTI, nuScenes-lidarseg, and ModelNet40 using ablations, qualitative results, and quantitative results.Figure 1 compares the proposed method with SalsaNext and MinkNet42 on SemanticKITTI.
2. Related Work
Prior 3D semantic-segmentation methods process LiDAR through projection, voxel, point, or hybrid representations. The related work contrasts their efficiency and accuracy trade-offs, especially for large-scale and unstructured point clouds.
- Projection-based methods: Projection-based methods transform LiDAR into 2D representations and adapt image-segmentation components, with later variants adding contextual, loss, normalization, or spatially adaptive modules.The cited progression includes SqueezeSeg, SqueezeSegV2, SqueezeSegV3, and SalsaNext.
- Point-based methods: Point-based methods such as PointNet and PointNet++ extract features directly from points and can model local and global information, but are slow on large-scale data.PointNet++ introduces sampling at different scales.
- Voxel-based methods: Voxel-based methods capitalize on 3D convolutions, and MinkowskiNet uses 4D sparse convolution for spatio-temporal 3D point clouds.The passage characterizes MinkowskiNet as strong in both accuracy and efficiency among 3D semantic-segmentation categories.
- Point-based methods: RandLA-Net combines random down-sampling with local feature aggregation to enlarge receptive fields while targeting efficiency and accuracy in large-scale segmentation.The aggregation progressively increases each point’s receptive field.
- Hybrid methods: Hybrid methods combine voxel, projection, and point-wise operations; FusionNet and SPVNAS use voxel-point processing to pursue competitive large-scale performance with efficient designs.SPVNAS builds on Minkowski Engine and uses neural architecture search to design its Sparse Point-Voxel Convolution operation.
3. Proposed Approach
(AF)2-S3Net addresses sparse, unstructured large-scale LiDAR segmentation with an end-to-end encoder-decoder sparse CNN. Its AF2M combines local and global features across multiple scales, while AFSM adaptively reweights decoder feature maps.
- Network architecture: (AF)2-S3Net transforms a MinkNet42 baseline into an end-to-end encoder-decoder network for large-scale LiDAR semantic segmentation.The model processes 3D sparse convolution tensors and predicts a class label for each point.
- Network architecture: The network represents each point with 3D coordinates and per-point normal and laser-intensity features in a sparse tensor.Normal features provide additional directional information for distinguishing fine object details.
- Attentive Feature Fusion: AF2M fuses point-based, medium-scale voxel-based, and large-scale voxel-based features through branches designed to capture local details and global context.Its branches use small, medium, and large kernels, respectively, and combine features with attention coefficients α, β, and γ.
- Attentive Feature Fusion: The AF2M branches emphasize fine details of smaller instances or global context from larger structures according to their learned attention maps.The fine-detail branch targets examples such as persons, poles, and traffic signs, while shallower branches focus on vegetation, sidewalks, and roads.
- Adaptive Feature Selection: AFSM processes multi-branch AF2M outputs with shared re-weighting parameters, filtering feature maps that do not contribute to the final results.A skip connection also propagates error gradients from the decoder back to encoder branches for learning stability.
- Network optimization: Training combines geo-aware anisotropic, Exponential-log, and Lovász losses, with the geometric term supporting recovery of fine LiDAR-scene details.The Exponential-log loss focuses on both small and large structures in the highly unbalanced dataset.
4. Experimental results
Experiments across SemanticKITTI, nuScenes-lidarseg, and ModelNet40 evaluate AF2-S3Net’s segmentation, classification, qualitative behavior, ablations, and distance robustness. The method improves semantic segmentation performance, particularly for small objects and distant points, while its attention modules and loss choices contribute complementary gains.
- Quantitative Evaluation: AF2-S3Net achieves state-of-the-art mean IoU on the SemanticKITTI test set, improving 2.7% over the second-best method and 15.4% over MinkNet42.It uses only voxelized input, whereas SPVNAS uses voxelized data and point-wise features.
- Quantitative Evaluation: On nuScenes-lidarseg, AF2-S3Net shows a large margin over MinkNet42 and SalsaNext and handles small objects across substantially different datasets.The authors present this as evidence that the method generalizes well between SemanticKITTI and nuScenes despite different sensor setups and environments.
- Qualitative Evaluation: The attention maps emphasize small instances such as people, poles, and bicycles alongside larger objects and region boundaries, while qualitative results surpass both baselines.The compared baselines are reported to miss fine details such as cars and vegetation.
- Ablation Studies: Combining AF2M and AFSM raises validation mIoU to 68.6%, an 8.8% increase from the MinkNet42 baseline.AF2M alone reaches 65.1%, while AFSM alone gives a 3.5% increase from baseline.
- Ablation Studies: Adding Lovász loss and the combination of Lovász and geo-aware anisotrophic loss produces mIoU values of 70.2% and 74.2%, respectively.These results support the effectiveness of the evaluated loss-function choices in the ablation sequence.
- Distance-based Evaluation: mIoU decreases with increasing distance for all methods, but AF2-S3Net performs better than MinkNet42 and SalsaNext at every evaluated distance.The authors attribute degradation at long range to the relative sparsity of LiDAR point clouds.
5. conclusion
The paper presents (AF)2-S3Net, an end-to-end 3D sparse convolutional CNN for LiDAR semantic segmentation and classification. Across SemanticKITTI, nuScenes-lidarseg, and ModelNet40, experiments demonstrate local-detail capture and state-of-the-art performance.
- (AF)2-S3Net is an end-to-end 3D sparse convolutional network for semantic segmentation and classification of LiDAR point clouds.The model uses an end-to-end CNN architecture based on sparse convolutions.
- The network introduces Attentive Feature Fusion and Adaptive Feature Selection modules to learn local and global contexts while emphasizing fine details.The two attention blocks are AF2M and AFSM.
- Experiments on SemanticKITTI, nuScenes-lidarseg, and ModelNet40 demonstrate the model’s ability to capture local details and achieve state-of-the-art performance.The evaluation covers semantic segmentation and classification across several benchmarks.
- Future work will extend the method to end-to-end 3D instance segmentation and object detection on large-scale LiDAR point clouds.