Source-linked AI summary
Anisotropic Convolutional Networks for 3D Semantic Scene Completion
Jie Li, Kai Han, Peng Wang, Yu Liu, Xia Yuan
TL;DR
Semantic scene completion must infer occupancy and semantic labels while modeling objects with highly variable 3D shapes, layouts, and visibility. The paper introduces anisotropic convolution, which adaptively selects receptive-field kernels voxel-wise through decomposed 1D convolutions. Across two SSC benchmarks, AIC-Net outperforms existing methods by a large margin and establishes a new state-of-the-art.
Problem
Semantic scene completion must jointly infer volumetric occupancy and voxel-wise semantic labels while modeling 3D context across objects with varied shapes, layouts, and visibility.
Method
Anisotropic convolution decomposes 3D convolution into three consecutive 1D convolutions and adaptively learns kernel combinations voxel-wise to control receptive fields.
Results
AIC-Net outperforms existing methods by a large margin on two SSC datasets, establishing the new state-of-the-art.
Takeaways & Limitations
AIC-Net provides a parameter-economic, plug-and-play way to model flexible 3D context for semantic scene completion.
Abstract
from arXiv · showhide
As a voxel-wise labeling task, semantic scene completion (SSC) tries to simultaneously infer the occupancy and semantic labels for a scene from a single depth and/or RGB image. The key challenge for SSC is how to effectively take advantage of the 3D context to model various objects or stuffs with severe variations in shapes, layouts and visibility. To handle such variations, we propose a novel module called anisotropic convolution, which properties with flexibility and power impossible for the competing methods such as standard 3D convolution and some of its variations. In contrast to the standard 3D convolution that is limited to a fixed 3D receptive field, our module is capable of modeling the dimensional anisotropy voxel-wisely. The basic idea is to enable anisotropic 3D receptive field by decomposing a 3D convolution into three consecutive 1D convolutions, and the kernel size for each such 1D convolution is adaptively determined on the fly. By stacking multiple such anisotropic convolution modules, the voxel-wise modeling capability can be further enhanced while maintaining a controllable amount of model parameters. Extensive experiments on two SSC benchmarks, NYU-Depth-v2 and NYUCAD, show the superior performance of the proposed method. Our code is available at https://waterljwant.github.io/SSC/
1. Introduction
Semantic scene completion must infer both volumetric occupancy and voxel-wise semantic labels from partial visual observations, while modeling objects with varied shapes, layouts, and visibility. The paper addresses these challenges with anisotropic convolution, which adapts receptive fields per voxel while reducing the computational and parameter burden of standard 3D convolution.
- Semantic scene completion jointly infers volumetric occupancy and voxel-wise semantic labels from static depth and/or RGB images.
- Objects’ varied shapes, layouts, and visibility make effective 3D-context modeling the central SSC challenge.
- Anisotropic convolution decomposes 3D convolution into three consecutive 1D convolutions whose kernel combinations are learned voxel-wise, enabling receptive fields with varying sizes.
- AIC-Net automatically chooses proper receptive fields for different voxels to model object variation in 3D scenes.
- The module is less computationally demanding and more parameter-efficient than standard 3D convolution, while serving as a plug-and-play replacement.
- Experiments on two SSC benchmarks show that the method outperforms existing methods by a large margin and establishes a new state-of-the-art.
2. Related Work
Prior SSC methods use 3D, hybrid, two-stream, or decomposed convolutional architectures, but fixed receptive fields and 3D-convolution costs remain recurring constraints. AIC addresses these constraints by learning anisotropic receptive fields voxel-wise while balancing semantic completion accuracy and computational cost.
- SSCNet jointly predicts semantic labels and volumetric occupancy in an end-to-end network, establishing an early 3D-CNN baseline.
- Hybrid and two-stage methods combine 2D and 3D processing or depth and visual information, but two-stage processing can suffer from error accumulation.
- Existing approaches remain constrained by the computational cost of 3D convolution and fixed receptive fields, including DDRNet’s decomposed architecture.
- Fixed-size kernels are less robust and flexible for objects with various sizes, motivating multi-scale and adaptive receptive-field strategies.
- Unlike prior strategies, AIC learns anisotropic receptive fields voxel-wise for 3D SSC and trades off semantic completion accuracy against computational cost.
3. Anisotropic Convolutional Networks
AIC-Net uses anisotropic convolution modules to adapt receptive fields voxel-wise for semantic scene completion. It decomposes 3D convolution into consecutive 1D operations, selects kernels through learned modulation factors, and stacks modules for flexible multi-scale context.
- Network Architecture: AIC-Net processes RGB and depth features in parallel, projects them into 3D, aggregates and fuses multi-scale features, then predicts occupancy and object labels voxel-wise.The architecture uses stacked AICs before feature fusion and three voxel-wise convolutions for prediction.
- Anisotropic Convolution: AIC decomposes 3D convolution into three consecutive 1D convolutions and uses multiple kernel sizes along each dimension to model anisotropic context.Candidate kernels can differ across dimensions, producing combinations of 3D receptive fields.
- Anisotropic Convolution: The bottleneck AIC reduces feature channels before anisotropic convolution and restores them afterward to reduce parameter cost.The first 1×1×1 convolution maps D channels to D′, where D′ < D, and the final convolution returns to D channels.
- Anisotropic Convolution: Learned modulation factors weight candidate kernels per dimension, with nonnegative weights normalized across each dimension to control their contributions.The weighting mechanism provides soft kernel selection rather than requiring a single fixed kernel.
- Training: AIC-Net is trained end-to-end with voxel-wise weighted cross-entropy, using weights to balance semantic categories.The loss uses the predicted probability and ground-truth label at each voxel.
4. Experiments
AIC-Net is evaluated on NYU and NYUCAD using quantitative comparisons, qualitative visualizations, and ablations of its adaptive kernels, modulation factors, plug-and-play use, and efficiency. Results show stronger semantic completion and flexible context modeling, with a favorable performance–cost trade-off.
- Experimental setup: Experiments evaluate AIC-Net on NYU-Depth-V2 and NYUCAD using semantic completion metrics, scene completion metrics, qualitative results, and ablation studies.The evaluation includes implementation details, dataset descriptions, quantitative comparisons, visualizations, and analyses of critical design choices.
- Quantitative comparison: AIC-Net achieves 2.9% higher average IoU than DDRNet for semantic scene completion, while DDRNet slightly outperforms it on class-agnostic scene completion.The authors attribute the smaller advantage in binary completion to AIC-Net’s focus on voxel-wise object variation.
- Quantitative comparison: AIC-Net performs better on categories with severe shape variations, including chairs, tables, and objects.These category-level gains are consistent with the network’s intended ability to model varying object shapes.
- Qualitative results: Qualitative results show more accurate semantic predictions and shape completion than SSCNet and DDRNet for chairs, tables, windows, beds, walls, sofas, and cluttered objects.The examples illustrate smaller receptive fields for delicate parts and larger context for objects such as windows.
- Ablation study: Using three candidate kernels outperforms using only kernel 7, and three kernels outperform two by about 1%.Because all options share the same maximum receptive field of 7, the comparison supports the benefit of multiple candidate kernels and greater context flexibility.
- Ablation study: Removing modulation factors causes degradation on both NYU and NYUCAD, while replacing DDR and ASPP components with AIC improves semantic completion by 1.6% and 0.6%, respectively.The results support adaptive kernel selection and the use of AIC as a plug-and-play module.
- Efficiency analysis: AIC-Net offers a better performance–cost trade-off because its parameters and computation grow linearly with candidate kernels, unlike the cubic growth of standard 3D convolution.A 5 × 5 × 5 3D convolution uses 3 times the parameters and FLOPs, while a 7 × 7 × 7 convolution uses 8 times as much as AIC-Net.
5. Conclusion
AIC-Net addresses object variation in semantic scene completion through voxel-wise adaptive anisotropic convolutions. Experiments on two SSC datasets report large-margin gains over existing methods.
- AIC-Net uses AIC modules that adaptively choose convolution kernels along all three dimensions for each voxel.The modules provide voxel-wise control over the receptive field.
- Stacking AIC modules flexibly controls each voxel’s receptive field while modeling 3D context economically in parameters.The module is also designed as a plug-and-play component for existing networks.
- Experiments on two SSC datasets show that AIC-Net outperforms existing methods by a large margin.The authors describe this result as establishing a new state-of-the-art.
A.1. Detailed Architectures
The detailed network architecture uses point-wise convolution for channel adjustment and combines max-pooling with stride-2 convolution for downsampling. Their outputs are concatenated before subsequent processing.
- PWConv adjusts the number of channels in the feature map.
- Each down-sample layer combines max-pooling with a convolution layer whose stride is 2.
- The outputs of the max-pooling and convolution layers are concatenated before entering subsequent layers.
A.2. Details of Each AIC Module
Each AIC module uses multiple candidate kernels along every dimension within a bottleneck design. The implementation keeps internal channel dimensions lower than output dimensions and uses unit stride and dilation.
- Each AIC dimension uses three candidate kernels with sizes 3, 5, and 7.
- The bottleneck AIC design sets internal channel dimension D′ below the output dimension D.D′ is 32 in the first six AIC modules and 64 in the last two.
- All AIC modules use stride and dilation rates set to 1.
A.3. 2D to 3D Projection
The projection stage voxelizes the 3D space and places 2D feature tensors into the corresponding 3D locations. This converts view-dependent 2D feature maps into a view-independent 3D feature volume.
- The method voxelizes the entire projected 3D space using meshed grids to form a 3D volume.
- Each depth point is projected to a 3D position, where the corresponding feature tensor is placed.
- The feature projection layer converts 2D CNN feature maps into a view-independent 3D feature volume.
B. More Qualitative Results
Qualitative comparisons show that the proposed method produces less cluttered, more accurate, and more detailed semantic scene completions than the compared methods. It also distinguishes difficult structures such as windows more effectively.
- The completed semantic 3D scenes are less cluttered and achieve higher voxel-wise accuracy than DDRNet and SSCNet.
- Compared with two other methods, AIC-Net produces more meticulous chair predictions and reduces interference from irrelevant voxels.
- AIC-Net distinguishes difficult windows effectively, whereas the other methods fail to distinguish them.
- Across rows 3 to 8, AIC-Net predictions are reported as more accurate than those of the other methods.