Source-linked AI summary
LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation
Lin Zhao, Hui Zhou, Xinge Zhu, Xiao Song, Hongsheng Li, Wenbing Tao
TL;DR
Sparse LiDAR points and limited image-LiDAR alignment make camera-LiDAR fusion for point-cloud segmentation challenging. LIF-Seg combines image context with LiDAR features and uses offset rectification to align sensor features. On nuScenes, it outperforms Cylinder3D by 2.1 mIoU and reports gains over projection-based methods.
Problem
LiDAR points can be sparse and lack rich color and texture, while residual spatiotemporal misalignment can place projected points outside their corresponding image instances.
Source: §I · pp. 1–2Method
LIF-Seg fuses low-level image context with LiDAR points early, predicts offsets to align coarse LiDAR and image semantic features, and refines the fused features.
Source: §I · p. 2Results
LIF-Seg outperforms Cylinder3D by 2.1 mIoU and achieves about 6% ∼12% performance gain over RangeNet++ and Salsanext.
Source: C. Performance Results and Analyses · p. 8Takeaways & Limitations
Ablation analyses report that image context and image semantic features are helpful for LiDAR segmentation.
Source: C. Performance Results and Analyses · p. 8
Abstract
from arXiv · showhide
Camera and 3D LiDAR sensors have become indispensable devices in modern autonomous driving vehicles, where the camera provides the fine-grained texture, color information in 2D space and LiDAR captures more precise and farther-away distance measurements of the surrounding environments. The complementary information from these two sensors makes the two-modality fusion be a desired option. However, two major issues of the fusion between camera and LiDAR hinder its performance, \ie, how to effectively fuse these two modalities and how to precisely align them (suffering from the weak spatiotemporal synchronization problem). In this paper, we propose a coarse-to-fine LiDAR and camera fusion-based network (termed as LIF-Seg) for LiDAR segmentation. For the first issue, unlike these previous works fusing the point cloud and image information in a one-to-one manner, the proposed method fully utilizes the contextual information of images and introduces a simple but effective early-fusion strategy. Second, due to the weak spatiotemporal synchronization problem, an offset rectification approach is designed to align these two-modality features. The cooperation of these two components leads to the success of the effective camera-LiDAR fusion. Experimental results on the nuScenes dataset show the superiority of the proposed LIF-Seg over existing methods with a large margin. Ablation studies and analyses demonstrate that our proposed LIF-Seg can effectively tackle the weak spatiotemporal synchronization problem.
I. INTRODUCTION
LiDAR provides geometric and distance information but its sparse, uneven points and limited color and texture can make segmentation difficult; camera images offer complementary semantic cues. LIF-Seg combines image context with LiDAR features and rectifies feature alignment to address weak sensor synchronization.
I. INTRODUCTION
- LiDAR sparsity increases with distance, while its lack of rich color and texture can make small or visually similar objects difficult to segment; camera images provide denser semantic cues.Residual spatiotemporal deviation between sensors can also place projected LiDAR points outside their corresponding image instances.Source: §I · pp. 1–2
- LIF-Seg uses early fusion of low-level image context and LiDAR points, then predicts offsets to align coarse LiDAR features with image semantic features.The aligned features are fused and refined to produce point-level semantic predictions.Source: §I · p. 2
- The paper reports that LIF-Seg is effective on nuScenes and proposes offset rectification to address weak spatiotemporal synchronization between LiDAR and cameras.The introduction presents early contextual fusion and offset rectification as the method's two main contributions.Source: §I · p. 2
II. RELATED WORK
Related work covers deep learning for point clouds and LiDAR segmentation approaches, alongside their challenges in large-scale outdoor scenes. It groups LiDAR segmentation into projection-based, voxel-based, and multi-view fusion-based methods.
A. Deep learning for 3D Point Clouds
- Point-cloud methods build on PointNet-style shared-MLP features, sampling, grouping, ordering, or graph networks, but many indoor approaches do not scale directly to outdoor LiDAR's varying density and large scenes.Large point sets also increase computational and memory costs.Source: A. Deep learning for 3D Point Clouds · p. 2
B. LiDAR Point Cloud Semantic Segmentation
- LiDAR segmentation methods are grouped as projection-based, voxel-based, and multi-view fusion-based; projection methods can lose or alter point-cloud topology, while outdoor sparsity and density variation challenge voxel methods.Cylinder3D uses cylindrical partitioning and an asymmetric residual block to reduce computation.Source: B. LiDAR Point Cloud Semantic… · pp. 2–3
- Multi-view fusion combines point-cloud representations such as BEV, range images, points, or voxels, but these methods remain limited by LiDAR's lack of rich color and texture.Examples include early fusion, uncertainty-based late fusion, point-voxel schemes, and gated range-point-voxel fusion.Source: B. LiDAR Point Cloud Semantic… · p. 3
C. LiDAR and Camera Fusion Methods
Prior camera-LiDAR fusion work has chiefly addressed object detection, while relatively few methods target semantic segmentation and weak sensor synchronization. LIF-Seg combines image context and semantic features with LiDAR features in a coarse-to-fine framework.
C. LiDAR and Camera Fusion Methods
- Camera-LiDAR fusion methods have shown promise in 3D object detection, but few focus on semantic segmentation while addressing weak synchronization between the sensors.Prior methods include point-wise fusion, candidate-level fusion, cross-view fusion, and image semantic scores appended to LiDAR points.Source: C. LiDAR and Camera Fusion Methods · p. 3
D. Image Semantic Segmentation
- Image semantic segmentation methods capture image context, and the paper adopts DeepLabv3+ as its image segmentation submodel to balance efficiency and performance.The passage identifies atrous convolution and ASPP modules in the DeepLab family as mechanisms for capturing image context.Source: D. Image Semantic Segmentation · p. 3
III. PROPOSED METHOD
- LIF-Seg predicts point labels through coarse feature extraction, offset learning, and refinement, fusing low-level image context early and aligned high-level image semantics in the mid-stage.In coarse extraction, a context sampler takes image context at each projected LiDAR point for fusion with LiDAR points.Source: §III · p. 3, Figure 3 · p. 3
A. Coarse Feature Extraction Stage
The coarse stage supplements each LiDAR point with image context, rather than fusing image and point features only in a one-to-one manner. Projected points are used to sample image neighborhoods, which are concatenated with LiDAR data for coarse feature extraction.
A. Coarse Feature Extraction Stage
- The method uses image context because camera imagery is dense and semantically rich, while many prior fusion methods combine image and LiDAR information one-to-one.The paper identifies image contextual information as important for fusion.Source: A. Coarse Feature Extraction Stage · p. 3
- LiDAR points are transformed and projected into camera images to locate their corresponding image pixels.For nuScenes, the transformation composes sweep LiDAR, ego-vehicle, global, image-time ego, and camera transforms.Source: A. Coarse Feature Extraction Stage · p. 4
- Image context is concatenated with LiDAR points before a UNet segmentation sub-network extracts coarse features.The sampled context can use a w × w window, such as 3×3.Source: A. Coarse Feature Extraction Stage · p. 4
B. Offset Learning Stage
The offset-learning stage addresses weak spatiotemporal synchronization by predicting corrections to projected LiDAR points. It uses the corrected positions to back-project image semantic features for fusion with coarse features.
B. Offset Learning Stage
- LIF-Seg predicts offsets to compensate for synchronization mismatch, then fuses aligned image semantic features with coarse features.The predicted offsets update projected-point positions before image features are back-projected into 3D point-wise features.Source: B. Offset Learning Stage · pp. 4–5
- The stage extracts high-level image semantic features and projects coarse LiDAR features into a same-sized pseudo-image feature map.DeepLabv3+ is used as the image segmentation sub-network, balancing efficiency and performance.Source: B. Offset Learning Stage · p. 4
- The refinement stage follows offset learning and combines point-wise image features with coarse features.Figure 5 presents the connection between stages and illustrates refinement.Source: Figure 5 · p. 5, B. Offset Learning Stage · p. 5
C. Refinement Stage
The refinement stage concatenates point-wise image features with coarse LiDAR features and feeds them to a UNet for segmentation. Training also uses auxiliary supervision to learn offsets when direct pixel-level image labels are unavailable.
C. Refinement Stage
- The refinement stage concatenates point-wise image features and coarse features, then feeds them to a UNet for more accurate segmentation predictions.It uses the same segmentation sub-network as the coarse feature extraction stage.Source: C. Refinement Stage · p. 5
- Training supervises segmentation with cross-entropy and Lovasz-softmax losses, and weights the auxiliary offset loss by α = 0.01.The two segmentation losses target point accuracy and intersection-over-union, respectively.Source: C. Refinement Stage · p. 5
- For foreground points in image-plane 2D boxes, auxiliary offset supervision regresses toward the box centroid and constrains the predicted offset direction.The regression uses an L1 loss, while direction loss is based on minus cosine similarity.Source: C. Refinement Stage · p. 5
IV. EXPERIMENTS
The experiments evaluate LIF-Seg on nuScenes, comparing fusion strategies and state-of-the-art methods, then testing offset learning through ablations.
IV. EXPERIMENTS
- The evaluation uses the nuScenes dataset to assess the effectiveness of LIF-Seg.Source: §IV · p. 5
- The experiments compare LiDAR-camera fusion strategies and state-of-the-art methods on nuScenes.They include early fusion of different image contexts and mid-fusion of point and image semantic features.Source: §IV · p. 5
- Ablation studies test the effectiveness of offset learning.Source: §IV · p. 5
A. Dataset and Evaluation Metric
The evaluation uses nuScenes, a large-scale multi-modal dataset with diverse scenes and imbalanced classes. LiDAR semantic segmentation is assessed using mean intersection-over-union (mIoU) across classes.
A. Dataset and Evaluation Metric
- nuScenes contains more than 1,000 scenes from Boston and Singapore, with 28,130 training frames, 6,019 validation frames, and 16 retained segmentation classes.The dataset uses LiDAR, cameras, and radar; this work uses LiDAR point clouds and RGB images from all six cameras.Source: A. Dataset and Evaluation Metric · p. 5
- nuScenes poses challenges from class imbalance, diverse locations and weather, and relatively sparse point clouds.Cars and pedestrians are frequent, while bicycles and construction vehicles have relatively limited training data; the dataset's sensor has fewer beams and lower horizontal angular resolution.Source: A. Dataset and Evaluation Metric · p. 5
- Performance is evaluated using mIoU across all classes, with p_ij denoting points of class i predicted as class j.The metric is formulated using the number of classes C and the class-by-prediction point counts p_ij.Source: A. Dataset and Evaluation Metric · p. 5
B. Implementation Details
The image branch uses DeepLabV3+ pretrained on nuImages, while Cylinder3D serves as the LiDAR segmentation network in both coarse and refinement stages.
B. Implementation Details
- DeepLabV3+ uses a ResNet backbone and FCN head to produce full-resolution 16-dimensional image semantic features, and is trained on nuImages because nuScenes has no public segmentation pretraining model.The nuImages annotations cover 100k images, and its classes are all part of nuScenes; its images hardly overlap with the nuScenes image set corresponding to LiDAR.Source: B. Implementation Details · p. 6
- Cylinder3D is used in both LiDAR stages, with cylindrical point-cloud partitions of size 480 × 360 × 32 and a 3 × 3 image-context window.The partition dimensions represent radius, angle, and height; coarse feature dimension C0 equals the number of classes C.Source: B. Implementation Details · p. 6
C. Performance Results and Analyses
On nuScenes validation, LIF-Seg outperforms the compared LiDAR segmentation methods, while fusion experiments favor image context and a coarse-to-fine fusion design. The experiments use retrained image and LiDAR baselines for comparison.
C. Performance Results and Analyses
- C+3×3 achieves the best mIoU among the early-fusion context-window variants; 1×1 lacks context, while 5×5 adds redundant information.C+Sem. and C+3×3+Sem. also improve segmentation, and the well-designed refinement variant improves over simpler mid-fusion configurations.Source: C. Performance Results and Analyses · pp. 6, 8
- LIF-Seg exceeds Cylinder3D by 2.1 mIoU and projection-based RangeNet++ and Salsanext by about 6%–12% on nuScenes validation.It is reported as dominant in many categories and particularly stronger in sparse categories; the dataset has about 35k points per frame.Source: Table II · p. 7, C. Performance Results and Analyses · p. 8
- The final design fuses LiDAR points with image context in the coarse stage, then combines point features with aligned image semantic features in refinement.The experiments compare early and mid-fusion strategies using a retrained DeepLabV3+ image network and Cylinder3D LiDAR baseline.Source: C. Performance Results and Analyses · pp. 6, 8
D. Ablation Studies
The ablation links offset learning to improved nuScenes validation mIoU and shows projected points moving toward their corresponding objects. The conclusion describes LIF-Seg's three-stage fusion pipeline and identifies unsupervised sensor-transform prediction as future work.
D. Ablation Studies
- Removing offset learning reduces nuScenes validation performance from 78.2 to 77.6 mIoU.The ablation evaluates offset learning within the full LIF-Seg pipeline.Source: D. Ablation Studies · p. 8
- The predicted offsets move projected points horizontally toward their corresponding centroids, helping them fall on the instance object.Fig. 7 marks original projections in red and offset-updated positions in cyan.Source: D. Ablation Studies · p. 8, Figure 7 · p. 8
V. CONCLUSION
- LIF-Seg uses coarse fusion, offset learning to align coarse and image semantic features, then refinement to produce more accurate segmentation.The coarse stage fuses LiDAR points with low-level image context; the refinement stage combines coarse features with aligned image semantic features.Source: §V · p. 8
- The conclusion proposes unsupervised transformation-matrix prediction as future work to address weak LiDAR-camera synchronization completely.The authors state that this could further improve LiDAR segmentation performance.Source: §V · p. 8