Source-linked AI summary
LargeKernel3D: Scaling up Kernels in 3D Sparse CNNs
Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, Jiaya Jia
TL;DR
3D sparse CNNs have limited receptive fields, while naively applying large kernels introduces severe efficiency and optimization difficulties. The paper proposes spatial-wise partition convolution and LargeKernel3D to address these barriers, and reports improvements across semantic segmentation and object detection, including leading nuScenes results.
Problem
3D sparse CNNs have constrained receptive fields, and directly applying large kernels creates efficiency and optimization difficulties in sparse, limited-scale 3D datasets.
Method
The paper proposes spatial-wise partition convolution and its SW-LK Conv module, which share weights among spatial neighbors to make large 3D kernels efficient and easier to optimize.
Results
LargeKernel3D improves semantic segmentation and object detection, reaching 72.8% NDS on nuScenes and 74.2% NDS with simple multi-modal fusion.
Takeaways & Limitations
The paper shows that large-kernel CNN designs can be realized efficiently and effectively for essential 3D visual tasks.
Takeaways & Limitations
LargeKernel3D relies on hand-designed spatial kernel sizes that may be suboptimal for other datasets or tasks.
Abstract
from arXiv · showhide
Recent advance in 2D CNNs has revealed that large kernels are important. However, when directly applying large convolutional kernels in 3D CNNs, severe difficulties are met, where those successful module designs in 2D become surprisingly ineffective on 3D networks, including the popular depth-wise convolution. To address this vital challenge, we instead propose the spatial-wise partition convolution and its large-kernel module. As a result, it avoids the optimization and efficiency issues of naive 3D large kernels. Our large-kernel 3D CNN network, LargeKernel3D, yields notable improvement in 3D tasks of semantic segmentation and object detection. It achieves 73.9% mIoU on the ScanNetv2 semantic segmentation and 72.8% NDS nuScenes object detection benchmarks, ranking 1st on the nuScenes LIDAR leaderboard. The performance further boosts to 74.2% NDS with a simple multi-modal fusion. In addition, LargeKernel3D can be scaled to 17x17x17 kernel size on Waymo 3D object detection. For the first time, we show that large kernels are feasible and essential for 3D visual tasks.
1. Introduction
3D sparse CNNs struggle to capture long-range context because small kernels and spatial disconnection constrain receptive fields. LargeKernel3D addresses the efficiency and optimization barriers of 3D large kernels with spatial-wise partition convolution, achieving strong results across 3D tasks.
- 3D sparse CNNs have limited receptive fields because small kernels and spatially disconnected sparse features restrict information flow.
- 3D large kernels face cubic growth in computation and parameters, while limited dataset scale makes optimization difficult.
- Spatial-wise partition convolution shares weights among spatially adjacent locations, preserving large spatial kernels while reducing model size and latency.
- Position embeddings supplement spatial-wise group convolution by restoring detail-capturing ability for large kernels.
- 72.8% NDS on nuScenes placed LargeKernel3D first on the nuScenes LIDAR leaderboard, while simple multi-modal fusion reached 74.2% NDS.
- LargeKernel3D produces larger effective receptive fields than plain 3D CNNs, including deeper small-kernel networks, and captures context at object centers and edges.
2. Related Work
Prior work establishes large kernels and local or windowed attention as ways to enlarge receptive fields, but 3D sparse settings impose distinct efficiency and representation challenges. The paper positions spatial-wise partition convolution as a design specifically for efficient 3D large kernels.
- 3D Feature Extractors: Point-cloud and grid-based 3D feature extractors address sparse geometric data through direct processing, sampling, or scene partitioning.
- 3D Feature Extractors: Spatial group convolution partitions spatially but splits input features into groups, whereas the proposed method partitions kernel weights to facilitate large 3D kernels.
- Vision Transformer: Vision Transformers enlarge context through local or shifted windows, with 3D variants applying window-based self-attention to detection.
- Position Encoding: Relative position encoding supplements spatial partitioning because weight sharing can blur position information.
3. Revisiting 3D Sparse CNNs
3D sparse convolutions preserve efficiency through submanifold operation but miss information across spatially disconnected features. Naively enlarging kernels increases computation and overfitting risk, motivating a specialized large-kernel design.
- Convolution Types: Submanifold sparse convolutions preserve output positions and computation cost, but their small local scope misses information flow across disconnected features.
- Obstacles in 3D Large Kernels: A 3D convolution's parameter count rises from 27 · cin·cout to 343 · cin · cout when kernel size increases from 3 to 7.
- Plain 3D CNNs and deeper doubled versions have insufficient effective receptive fields because sparse features remain spatially disconnected.
- Obstacles in 3D Large Kernels: 3D datasets commonly contain no more than one thousand scenes, and increasing MinkowskiNet-34 kernels from 3×3×3 to 7×7×7 gradually worsens over-fitting.
- Popular 2D large-kernel components, including depth-wise convolution, layer normalization, and GELU, can be ineffective or harmful in 3D CNNs.
4. 3D Large-kernel Convolutional Network
The paper replaces ineffective depth-wise convolution and naive large 3D kernels with spatial-wise partitioning, combining shared spatial weights with position embeddings in SW-LK Conv. These modules retain large receptive fields while reducing efficiency and detail-capture problems in sparse 3D CNNs.
- Depth-wise convolutions are not beneficial to 3D tasks, unlike their role in several 2D large-kernel CNNs.
- Spatial-wise partition convolution shares weights among spatial neighbors, remapping a large kernel into smaller grouped operations while preserving its large spatial scope.
- Naively enlarging kernels from 3×3×3 to 7×7×7 sharply increases model size and latency, whereas spatial-wise partitioning avoids parameter growth with limited latency overhead.
- SW-LK Conv combines spatial-wise partition convolution with learnable position embedding to recover detail lost when neighboring positions share weights.
- LargeKernel3D replaces plain sparse-convolution layers with SW-LK Conv in segmentation and detection backbones, including 7×7×7 grouped into 3×3×3 splits for semantic segmentation.
5. Experiments
Experiments show that LargeKernel3D improves 3D segmentation and detection while remaining efficient, and that its design choices support larger kernels across benchmarks.
- ScanNetv2 contains 1,201 training, 312 validation, and 100 test scenes with 20 semantic categories, evaluated using mIoU.
- Ablation Studies: 2D-inspired layer normalization, depth-wise convolution, and GeLU provide no obvious benefit on 3D baseline networks, whereas SW-LK Conv improves them.
- Ablation Studies: LargeKernel3D keeps the central group at 1×1×1 without shifting because enlarging or shifting central groups sharply reduces accuracy.
- Ablation Studies: 17×17×17 kernels improve Waymo vehicle detection by around 2% mAP across categories, while 19×19×19 provides no further gains.
- Ablation Studies: Position embedding adds limited latency and is essential for the 17×17×17 model, where larger kernels suffer from feature blurring.
- Ablation Studies: Inference-time kernel shrinking reduces ScanNetv2 latency from 514ms to 145ms while preserving 73.5% mIoU.
- Main Results: LargeKernel3D improves CenterPoint to 70.6% and 72.8% NDS on nuScenes, while multimodal fusion reaches 74.2% NDS and 71.2% mAP.
- Main Results: On Waymo validation, 17×17×17 LargeKernel3D improves CenterPoint vehicle detection by around 1.5% LEVEL 1 AP / APH using single-frame LIDAR without augmentation or ensembles.
6. Conclusion and Discussion
The paper presents SW Conv, SW-LK Conv, and LargeKernel3D as efficient and effective designs for large-kernel 3D CNNs, improving semantic segmentation and object detection. It also identifies hand-designed spatial kernel sizes as a limitation and notes potential societal risks from downstream tasks or datasets.
- Conclusion and Discussion: SW Conv addresses efficiency and optimization issues in plain 3D large-kernel CNNs.The paper describes it as specifically designed for 3D large kernels.
- Conclusion and Discussion: LargeKernel3D improves performance on both 3D semantic segmentation and object detection with efficient large-kernel networks.The method is reported to achieve these improvements across both task types.
- Limitations: Hand-designed spatial kernel sizes may be suboptimal for other datasets or tasks because scene sizes and data sparsity vary.The authors mention search-based techniques as a possible future direction.
- Broader Impacts: The proposed backbone may be used for tasks or datasets involving negative societal impacts.This is stated as a broader-impact consideration rather than a benchmark limitation.
A.1. Data Processing
The supplementary experiments preprocess each dataset by voxelizing or clipping point clouds with dataset-specific spatial ranges and augmentations. ScanNetv2, nuScenes, and Waymo use different voxel sizes, coordinate ranges, and transformation settings.
- ScanNetv2: ScanNetv2 point clouds are voxelized at 0.02m along all three axes and augmented with dropout, flipping, contrast, color translation, and jittering.The processing follows MinkowskiNet baseline settings.
- nuScenes: nuScenes point clouds are clipped to dataset-specific XYZ ranges and voxelized at (0.075m, 0.075m, 0.2m).Augmentations include flipping, scaling, rotation, ground-truth sampling, and translation.
- Waymo: Waymo point clouds are clipped to dataset-specific XYZ ranges and voxelized at (0.1m, 0.1m, 0.15m).Augmentations include flipping, scaling, rotation, and ground-truth sampling.
A.2. Training Settings
Training settings differ by dataset: ScanNetv2 uses SGD over 600 epochs, while nuScenes and Waymo use Adam-based training for 20 and 30 epochs, respectively. The supplementary material also describes center-group visualizations and settings.
- ScanNetv2: ScanNetv2 models are trained for 600 epochs with batch size 16 using SGD and a poly learning-rate schedule.The initial learning rate is 0.1, with momentum 0.9.
- Center-group settings: Figure S-5 studies center sizes for group splitting and center shifting in the Tab. 4 schemes.The figure labels include Center 1 + shift, Center 3, and Center 5.
- nuScenes: nuScenes models are trained for 20 epochs with batch size 32 using Adam and cosine annealing from 1e-3 to 1e-4.Weight decay is 0.01 and gradient norms are clipped by 35.
- Waymo: Waymo models are trained for 30 epochs with batch size 16 using Adam, weight decay 0.01, and gradient clipping by 10.The learning rate starts at 0.003 and momentum is 0.9.
A.3. Network Architecture Settings
LargeKernel3D replaces selected plain sparse convolutional blocks with SW-LK Conv in semantic-segmentation and object-detection backbones, while retaining dataset-specific backbone components. A lightweight RGB branch supports the multi-modal LargeKernel3D-F variant.
- 3D Semantic Segmentation: LargeKernel3D follows MinkowskiNet-34 and replaces plain blocks with SW-LK Conv in stages 1, 2, and 3.The last stage remains unchanged because its receptive field is considered sufficient.
- 3D Object Detection: The object-detection architecture substitutes plain backbone blocks with SW-LK Conv while retaining the stated CenterPoint stage structure.The supplementary architecture materials include a LargeKernel3D object-detection design.
- Additional Experiments: The supplementary material reports a nuScenes validation comparison involving other methods and ground-truth sampling fading.The table caption identifies this as Table S-12 on the nuScenes validation split.
- Multi-modal Fusion: LargeKernel3D-F fuses LIDAR and RGB features by direct voxel-wise summation using a lightweight RGB branch.The RGB branch contains a convolution, batch normalization, ReLU, pooling stem, and three residual bottlenecks.
B. Additional Experimental Results
Additional visualizations compare plain 3D CNNs with LargeKernel3D through effective receptive fields, using original images and projected gradient intensities. The comparison follows the same setup as the paper’s Figure 2.
- nuScenes Results: 63.9% mAP is achieved by LargeKernel3D on the nuScenes validation split with GT-S Fading.GT-S Fading removes ground-truth sampling during the last five training epochs and is included for test submission.
- Effective Receptive Fields: The supplementary visualization places original images on the left, plain 3D CNN effective receptive fields in the middle, and LargeKernel3D effective receptive fields on the right.The visualization uses the same setting as Figure 2 in the paper.
- Effective Receptive Fields: Effective receptive fields are computed from gradients of each input voxel with respect to a feature of interest, then normalized and projected onto the image plane.Gradient intensity indicates how input-value changes affect the feature, and calibration matrices perform the projection.