LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception

Introduction

• LidarMultiNet reports state-of-the-art results across five major LiDAR benchmarks, including first place in the Waymo 2022 semantic segmentation challenge.It also reports the highest single-model mAPH L2 on Waymo detection and state-of-the-art performance on nuScenes tasks.Source: Introduction · p. 2
• The network jointly performs 3D semantic segmentation, object detection, and panoptic segmentation, with new tasks added through task-specific heads at little additional computational cost.The authors identify distinct specialized architectures and prior multi-task underperformance as the motivation for unification.Source: Introduction · p. 2
• A Global Context Pooling module improves global feature learning in a 3D sparse-convolution encoder-decoder network.The introduction notes that submanifold sparse convolution limits information flow and receptive field, while global context is important for 3D segmentation.Source: Introduction · pp. 1–2
• A second-stage module refines first-stage semantic segmentation for foreground thing classes to produce panoptic segmentation.Source: Introduction · p. 2

Related Work

• Voxel, bird’s-eye-view, and range-view projections encode sparse point clouds differently; segmentation needs larger feature maps for point-wise labels than detection’s dense 2D BEV representation [Zhou and Tuzel 2018; Zhu et al. 2021b; Yang, Luo, and Urtasun 2018; Lang et al. 2019; Zhang et al. 2020; Wu et al. 2018; Sun et al. 2021].Methods also fuse point-level and voxel features to preserve fine-grained information [Tang et al. 2020; Xu et al. 2021; Ye et al. 2021b].Source: Related Work · p. 2
• Anchor-free detectors are predominant on major 3D detection benchmarks, and LidarMultiNet attaches anchor-free detection heads to its 2D branch [Yin, Zhou, and Kr¨ahenb¨uhl 2021].Source: Related Work · p. 2
• Detection frameworks use RCNN-style second stages to refine object boxes, whereas LidarMultiNet’s second segmentation stage uses first-stage detection and segmentation results [Shi et al. 2020; Yin, Zhou, and Kr¨ahenb¨uhl 2021; Li, Wang, and Wang 2021; Sheng et al. 2021].Source: Related Work · p. 2
• Bottom-up panoptic methods derive from segmentation networks, while end-to-end LiDAR panoptic methods still underperform independently combined detection and segmentation models [Zhou, Zhang, and Foroosh 2021; Hong et al. 2021; Razani et al. 2021; Fong et al. 2022].The passage attributes incompatible segmentation and detection designs partly to detection networks’ loss of height information [Xu et al. 2021; Yin, Zhou, and Kr¨ahenb¨uhl 2021].Source: Related Work · p. 2
• Prior LiDAR multi-task work includes LidarMTL, which jointly handles object detection and road understanding; LidarMultiNet targets the major LiDAR perception tasks.MultiNet unified image-based object detection and road understanding [Teichmann et al. 2018].Source: Related Work · p. 2

Main Architecture

• A voxelized 3D sparse U-Net with skip-connected encoder and decoder features supports segmentation, while BEV task heads handle detection and auxiliary segmentation.Voxel predictions are de-voxelized to point-level labels; a second stage refines semantic segmentation and produces panoptic results.Source: Main Architecture · p. 3, Figure 2 · p. 3

Global Context Pooling

• GCP uses a dense BEV feature map and a 2D multi-scale extractor to enlarge the receptive field and provide global context for segmentation.The BEV features can also serve other tasks through additional heads with slight computational cost.Source: Global Context Pooling · p. 4

Multi-task Training and Losses

• The network is trained end-to-end with task losses combined using learned uncertainty weights, so more uncertain tasks contribute less to the total loss.The task-specific losses are grouped with fixed weights before uncertainty-based combination.Source: Multi-task Training and Losses · pp. 4–5

Second-stage Refinement

• The second stage predicts box-class and point-wise mask scores from first-stage detections and features, then fuses refined scores with first-stage semantic scores.Points outside detected boxes retain their first-stage scores; points within boxes are used to derive panoptic instance assignments.Source: Second-stage Refinement · p. 5

Datasets and Metrics

• The evaluation spans Waymo detection and semantic segmentation, plus nuScenes detection, LiDAR segmentation, and panoptic segmentation; each dataset uses task-specific metrics.Waymo reports APH for detection and IoU for semantic segmentation; nuScenes uses mAP and NDS for detection, mIoU for semantic segmentation, and PQ for panoptic segmentation.Source: Experiments · p. 5, Datasets and Metrics · pp. 5–6

Implementation Details

• Waymo inputs merge the current frame with two past frames, while nuScenes merges nine past frames; training uses augmentation and a two-stage schedule.The two-stage model freezes stage one and fine-tunes stage two for six epochs.Source: Implementation Details · p. 6

Waymo Open Dataset Results

• 71.13 mIoU places LidarMultiNet first on the Waymo semantic-segmentation leaderboard, with best accuracy on 15 of 22 classes using LiDAR alone.The leaderboard submission uses test-time augmentation and model ensembling.Source: 3D Semantic Segmentation Challenge… · p. 6, Table 1 · p. 7
• On the Waymo validation set, multi-frame input adds 0.59 mIoU, GCP adds 0.94, auxiliary losses add 0.63, and second-stage refinement adds 0.34.TTA and ensembling further raise mIoU to 73.05 and 73.78, respectively.Source: Table 2 · p. 7, Ablation Study on the 3D Semantic… · p. 7
• 76.35 mAPH L2 is the highest single-model Waymo test result without TTA, outperforming state-of-the-art detectors, including camera-based methods.The same model also reaches 71.93 mIoU on the Waymo semantic-segmentation validation set; jointly trained models outperform independent single-task models on both tasks and are more efficient than combining them.Source: Table 3 · p. 7, Evaluation on the 3D Object Detection… · p. 7, Effect of the Joint Multi-task Training · p. 8

NuScenes Benchmarks

• A single LidarMultiNet without TTA outperforms combined single-task models on nuScenes test sets by 1.0% in mIOU, 1.1% in NDS, and 1.3% in PQ.It also outperforms previous state-of-the-art methods on each of the three tasks.Source: Table 6 · p. 8, Comparison with State-of-the-art Methods · p. 8
• The second stage further improves semantic- and panoptic-segmentation results over direct fusion of first-stage detection and semantic predictions.The refinement targets spatial consistency among thing-class points within detected objects.Source: Table 8 · p. 8, Effect of the Second-Stage Refinement · p. 8

• LidarMultiNet reached official first place in the Waymo Open Dataset 3D semantic segmentation challenge 2022.Source: Conclusion · p. 8
• LidarMultiNet was the first multi-task network to achieve state-of-the-art performance on all five major large-scale LiDAR perception benchmarks.Source: Conclusion · p. 8
• The authors hope LidarMultiNet inspires future work toward unifying LiDAR perception tasks in a single versatile multi-task network.Source: Conclusion · p. 8

Network Hyperparameters

• The encoder has four 3D sparse-convolution stages with channel widths 32, 64, 128, and 256, and downsampling factor 8.Source: Network Hyperparameters · p. 9
• The decoder has four symmetric 3D sparse-deconvolution stages with channel widths 128, 64, 32, and 32.Source: Network Hyperparameters · p. 9
• VFE input features have size 16, while GCP depth and width are 6, 6 and 128, 256, respectively.Implementation details are summarized in Table 9.Source: Network Hyperparameters · p. 9, Table 9 · p. 9

Test-Time Augmentation

• Test-time augmentation uses xz- and yz-plane flips, global scaling from 0.95 to 1.05, and specified yaw rotations.Yaw angles are ±22.5°, ±45°, ±135°, ±157.5°, and 180°.Source: Test-Time Augmentation and Ensemble · p. 9
• Pitch rotations of ±8°, roll rotations of ±5°, and z-axis translations of ±0.2m are also used for segmentation augmentation.Source: Test-Time Augmentation and Ensemble · p. 9

Ensemble

• The leaderboard segmentation submission ensembles seven models, including variants with smaller voxels, different channel widths, more past frames, or no second stage.Other explored variants use a 4× downsampling factor.Source: Test-Time Augmentation and Ensemble · p. 9

Runtime

• 126ms is the nuScenes first-stage runtime and 9ms the second-stage runtime, versus 219ms summed runtime for separate detection and segmentation models.The single-task runtimes are 107ms for detection and 112ms for segmentation.Source: Runtime and Model Size · p. 9
• 145ms is the Waymo first-stage runtime and 18ms the second-stage runtime.Source: Runtime and Model Size · p. 9

Model Size

• The nuScenes first and second stages have model sizes of 135M and 4M, respectively; the Waymo stages are 131M and 4M.Source: Runtime and Model Size · p. 9

Visualizations

• Figure 6 visualizes qualitative LidarMultiNet results on the Waymo Open Dataset validation set.Source: Visualizations · p. 9
• Figure 7 visualizes qualitative LidarMultiNet results on the nuScenes validation set.Source: Visualizations · p. 9
• The visualizations cover validation sets from both Waymo Open Dataset and nuScenes.Source: Visualizations · p. 9