Source-linked AI summary
Efficient Semantic Scene Completion Network with Spatial Group Convolution
Jiahui Zhang, Hao Zhao, Anbang Yao, Yurong Chen, Li Zhang, Hongen Liao
TL;DR
Dense 3D prediction faces high computational and memory demands, and semantic scene completion must infer labeled structure beyond the input. The paper introduces Spatial Group Convolution and an efficient sparse completion network; on SUNCG it reports state-of-the-art IoU results.
Problem
Dense 3D CNNs face cubic growth in computation and memory with voxel resolution, while semantic scene completion must predict complete structure different from its input.
Source: §1 · p. 1, §3.1 · p. 5Method
Spatial Group Convolution partitions feature maps along spatial dimensions for sparse convolution, within a multiscale sparse network using coarse-to-fine prediction.
Source: §1 · pp. 2–3Results
84.5% IoU for scene completion and 70.5% IoU for semantic scene completion are reported as state-of-the-art SUNCG results.
Source: §1 · p. 3Takeaways & Limitations
On SUNCG, the proposed network combines reduced computation with state-of-the-art semantic scene completion performance.
Source: §1 · p. 3, §6 · p. 14Partial voxels make small, complex objects difficult to recognize; adaptively sampling small objects more densely is proposed as future work.
Source: §4.2 · p. 12
Abstract
from arXiv · showhide
We introduce Spatial Group Convolution (SGC) for accelerating the computation of 3D dense prediction tasks. SGC is orthogonal to group convolution, which works on spatial dimensions rather than feature channel dimension. It divides input voxels into different groups, then conducts 3D sparse convolution on these separated groups. As only valid voxels are considered when performing convolution, computation can be significantly reduced with a slight loss of accuracy. The proposed operations are validated on semantic scene completion task, which aims to predict a complete 3D volume with semantic labels from a single depth image. With SGC, we further present an efficient 3D sparse convolutional network, which harnesses a multiscale architecture and a coarse-to-fine prediction strategy. Evaluations are conducted on the SUNCG dataset, achieving state-of-the-art performance and fast speed. Code is available at https://github.com/zjhthu/SGC-Release.git
1 Introduction
The paper targets the computational and memory costs of dense 3D prediction by exploiting voxel sparsity and redundancy. It introduces Spatial Group Convolution (SGC) and a sparse network for semantic scene completion, reporting state-of-the-art SUNCG results.
1 Introduction
- Dense 3D CNNs face cubic growth in computation and memory as voxel resolution increases, while empty and redundant voxels offer opportunities to reduce processing.The baseline sparse convolutional network still costs about 80 GFLOPs when outputting predictions at 1/64 the input size.Source: §1 · pp. 1–2
- SGC partitions features into spatial groups, making each group sparser for sparse convolution while later gathering group results to use their complementary information.Unlike Group Convolution, SGC groups spatial positions rather than feature channels and shares weights between groups.Source: §1 · pp. 2–3
- SGC is presented as a 3D acceleration approach distinct from channel-based Group Convolution and methods that deactivate feature voxels, which may be unsuitable for dense prediction.SGC operates on spatial dimensions and shares weights across groups; some desired voxels deactivated by prior feature-sparsity methods cannot be recovered.Source: §1 · p. 2
- Semantic scene completion must predict semantic labels and complete structure that differs from the input, motivating a sparse network that can generate unknown structures.The network uses a multiscale encoder-decoder and coarse-to-fine prediction strategy; its dense deconvolution layer and Abstracting Module generate absent voxels and remove trivial ones, respectively.Source: §1 · pp. 2–3
- 84.5% IoU for scene completion and 70.5% IoU for semantic scene completion are reported as state-of-the-art SUNCG results.SGC reduces computation by about 3/4 while losing 0.7% and 1.2% IoU for scene completion and semantic scene completion, respectively, compared with networks without SGC.Source: §1 · p. 3
2 Related works
Related work covers sparse and hierarchical approaches to 3D deep learning, computation-efficient networks, and semantic segmentation and shape completion. The paper distinguishes its approach by increasing feature sparsity through spatial grouping and by focusing on efficient completion.
2.1 3D Deep Learning
- Prior 3D deep-learning approaches include sparse convolution, octree-based CNNs, blockwise convolution, and point-cloud networks; this work distinguishes itself by increasing feature sparsity for efficiency.Graham et al. [11] proposed hash-table sparse convolution; OctNet and O-CNN use octrees, while PointNet processes point clouds directly.Source: §2.1 · pp. 3–4
2.2 Computation-efficient Networks
- Efficiency methods include pruning, quantization, and channel-wise Group Convolution, while prior feature-sparsity methods can deactivate voxels needed for dense prediction.The paper instead groups spatial positions so only partial valid voxels participate in each convolution.Source: §2.2 · p. 4
2.3 3D Semantic Segmentation and Shape Completion
- Semantic segmentation labels observed voxels and shape completion fills missing ones; prior methods combine these tasks, while this work emphasizes efficient completion.The architecture shares similarities with octree-based completion methods, but its focus is efficient model design.Source: §2.3 · pp. 4–5
3 Method
The method combines sparse convolution and Spatial Group Convolution (SGC) to reduce computation, then extends sparse networks to predict scene structures absent from the input. Its semantic scene completion network uses multiscale, coarse-to-fine prediction and implementation choices tailored to preserve useful sparsity.
3.1 Sparse Convolutional Network
- Sparse convolution stores and processes only non-empty voxels while maintaining sparsity, but its output structure matches the input and cannot directly generate unknown structures.The proposed completion network addresses this restriction because completion requires output with a different, complete structure.Source: §3.1 · p. 5
3.2 Spatial Group Convolution
- SGC partitions voxels along spatial dimensions, applies sparse convolution to each group with shared weights, and gathers group features to use complementary information.Only a portion of the original non-empty voxels—about 1/G per group—participates in each convolution.Source: §3.2 · pp. 5–6, Figure 2 · p. 6
- The method considers random uniform partitioning and a fixed pattern shared across training and testing, which also partitions voxels uniformly.A fixed pattern avoids requiring filters to be invariant to all possible activation patterns.Source: §3.2 · p. 6
3.3 Sparse Convolutional Networks for Semantic Scene Completion
- The completion network uses sparse-convolution encoder modules and dense deconvolution to generate new voxels, while an Abstracting Module removes unnecessary voxels and guides coarse-to-fine refinement.The Abstracting Module keeps voxels with non-empty predictions and nearby voxels; the described setting uses one such module above resolution 32.Source: §3.3 · pp. 7–8
- Voxel-wise softmax losses supervise two prediction scales, and the final loss sums the scale-specific losses with weights α_i.The authors report that α_i = 1 works well.Source: §3.3 · p. 8
3.4 Implementation Details
- Implementation uses flipped TSDF input and a six-scale U-Net, with SGC at scales of at least 32^3 and repartitioning when resolution changes.The input is 256^3; the network is trained and evaluated on SUNCG, whose test set contains 470 depth-map and complete-label pairs.Source: §3.4 · pp. 8–9
4 Evaluation
On SUNCG, the evaluation compares the baseline network with SSCNet and tests SGC’s computation–accuracy trade-off; the network is also evaluated on real-world NYU data.
4.1 Comparision to SSCNet
- The baseline improves over SSCNet by 24.1% in semantic scene completion and 11.0% in scene completion on SUNCG.The scene-completion IoU is measured on unobserved voxels, while semantic scene-completion IoU covers observed and unobserved voxels.Source: §4.1 · p. 9, Table 1 · p. 11
4.2 Spatial Group Convolution Evaluation
- Using only 27.8% computation, SGC loses 0.7% scene-completion IoU and 1.2% semantic-completion IoU on SUNCG.The reported SGC results compare against networks without SGC.Source: Table 2 · p. 11, §4.2 · p. 11
- With four groups, fixed-pattern partitioning gives a 1.7% semantic-completion IoU enhancement over random partitioning.The evaluation compares partition strategies across accuracy and FLOPs.Source: Table 2 · p. 11, §4.2 · p. 11
- SGC’s IoU drops more for small categories such as chair, furniture, and objects than for large categories such as ceiling and floor.The authors suggest fewer voxels may cost small objects geometric information and propose sampling small objects more densely than large ones.Source: §4.2 · pp. 11–12, Table 3 · p. 12
4.3 Evaluation on NYU dataset
- On NYU, the network improves over SSCNet by 2.0% in semantic scene completion and 1.1% in scene completion.The authors report that fixed-pattern partitioning is comparable to or better than the baseline and consistently better than random partitioning on this dataset.Source: Table 4 · p. 12, §4.3 · p. 12
4.2 Spatial Group Convolution Evaluation
- Sparsity-invariant convolution, which normalizes by the number of valid voxels, did not work in this task.Source: §4.2 · p. 12
5 Discussion
The discussion examines the filters learned by SGC and how its spatial groups exchange information. SGC learns sharper weight distributions and irregular fixed-pattern kernels, while communication occurs at later sampling layers.
5.1 What does Spatial Group Convolution learn?
- SGC filters have sharper weight histograms than SCN filters, and the histograms sharpen as the group count increases.The authors suggest this may reflect a need for filter values to stay close across locations to adapt to different sparsity patterns.Source: §5.1 · p. 13
- Fixed-pattern SGC learns irregular kernels; with two groups, the valid 2D kernel has an X shape that persists as it slides.The illustrated 3D valid filter shapes are the ones used in Table 2.Source: §5.1 · p. 13, Figure 6 · p. 14
5.2 Information flow among different groups
- Groups have no information flow during SGC, but their voxels are gathered before down-sampling convolution or up-sampling deconvolution.Shuffled SGC, which changes partition patterns between ResNet-block convolutions, produced no obvious improvement.Source: §5.2 · pp. 13–14
6 Conclusions
The paper presents SGC for efficient semantic scene completion and an end-to-end sparse convolutional network, reporting a new accuracy record on SUNCG.
6 Conclusions
- The paper combines spatially grouped convolution with an end-to-end sparse convolutional network for semantic scene completion, reporting a new accuracy record on SUNCG.Source: §6 · p. 14