Source-linked AI summary
ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation
Jinming Cao, Hanchao Leng, Dani Lischinski, Danny Cohen-Or, Changhe Tu, Yangyan Li
TL;DR
RGB and depth encode different properties, and homogeneous convolution may not adequately capture depth shape information for semantic segmentation. ShapeConv separates depth patches into shape and base components, weights them independently, and combines them before convolution. Across three indoor RGB-D benchmarks and five architectures, it improves segmentation performance without added inference computation or memory.
Problem
RGB and depth have intrinsically different representations, while existing homogeneous convolutions overlook depth shape information that is strongly connected to semantics.
Method
ShapeConv decomposes each depth patch into shape and base components, applies separate learnable weights, and convolves their re-weighted combination.
Results
ShapeConv improves performance across three indoor RGB-D benchmarks and five architectures, including segmentation accuracy around object boundaries.
Takeaways & Limitations
ShapeConv can replace vanilla convolution in CNNs while retaining identical inference computation and memory requirements through kernel fusion.
Takeaways & Limitations
The method focuses on translational transformations because fixed-view depth values do not address rotational transformations; extending decomposition to point clouds is identified as future work.
Abstract
from arXiv · showhide
RGB-D semantic segmentation has attracted increasing attention over the past few years. Existing methods mostly employ homogeneous convolution operators to consume the RGB and depth features, ignoring their intrinsic differences. In fact, the RGB values capture the photometric appearance properties in the projected image space, while the depth feature encodes both the shape of a local geometry as well as the base (whereabout) of it in a larger context. Compared with the base, the shape probably is more inherent and has a stronger connection to the semantics, and thus is more critical for segmentation accuracy. Inspired by this observation, we introduce a Shape-aware Convolutional layer (ShapeConv) for processing the depth feature, where the depth feature is firstly decomposed into a shape-component and a base-component, next two learnable weights are introduced to cooperate with them independently, and finally a convolution is applied on the re-weighted combination of these two components. ShapeConv is model-agnostic and can be easily integrated into most CNNs to replace vanilla convolutional layers for semantic segmentation. Extensive experiments on three challenging indoor RGB-D semantic segmentation benchmarks, i.e., NYU-Dv2(-13,-40), SUN RGB-D, and SID, demonstrate the effectiveness of our ShapeConv when employing it over five popular architectures. Moreover, the performance of CNNs with ShapeConv is boosted without introducing any computation and memory increase in the inference phase. The reason is that the learnt weights for balancing the importance between the shape and base components in ShapeConv become constants in the inference phase, and thus can be fused into the following convolution, resulting in a network that is identical to one with vanilla convolutional layers.
1. Introduction
RGB and depth encode different information, so vanilla convolution may not process depth optimally. ShapeConv decomposes depth patches into shape and base components, weights them separately, and preserves inference efficiency while improving segmentation performance.
- RGB captures photometric appearance, whereas depth encodes local geometric shape and its position in a larger context.
- Vanilla convolution can produce different features for same-class patches because differing base components hinder shape-invariant learning.The base component remains useful because it contributes to shape formation in larger-context follow-up layers.
- ShapeConv decomposes each depth patch into base and shape components, applies separate learnable weights, then convolves their combined representation.The patch mean represents base location, while residual values represent relative geometric shape.
- ShapeConv can be fused into a following convolution during inference, adding no computation or memory while replacing vanilla convolution in CNNs.The learned base-kernel and shape-kernel become constants during inference.
- Experiments span three indoor RGB-D benchmarks and five semantic-segmentation architectures, with improved performance and stronger accuracy around object boundaries.The boundary improvement is presented as evidence that ShapeConv effectively leverages depth information.
2. Related Work
Prior RGB-D segmentation methods either adapt RGB architectures through multimodal fusion or design geometry-aware convolutional layers. These approaches face modality-fusion or computational limitations, while ShapeConv adaptively models patch shape and can retain vanilla-convolution inference cost.
- RGB-D architectures commonly use parallel RGB and depth CNN streams whose features are fused at selected stages.
- Two-stream and multilevel fusion methods make fusion-stage selection difficult and often substantially increase computation.
- Geometry-aware layers improve flexibility by exploiting depth characteristics, including depth similarity, depth-axis receptive fields, or 3D-guided sampling offsets.
- ShapeConv adaptively weights shape and base content within each patch and converts to vanilla convolution during inference with zero additional memory or computation.
3. Method
ShapeConv adapts convolution to RGB-D segmentation by separating each depth patch into base and shape components, weighting them independently, and recombining them before convolution. Its kernel-based implementation can replace vanilla convolutions in CNNs while preserving vanilla-convolution inference cost and memory.
- Motivation: ShapeConv addresses depth-patch variation by modeling shape, which is shared by same-class patches, separately from base information tied to observation distance.Vanilla convolution can produce different features for patches of the same chair when their base components differ, hindering shape invariance.
- ShapeConv Formulation: Each depth patch is decomposed into a base-component given by its mean values and a shape-component given by relative values.The base describes where the patch is in a larger context, while the shape expresses what the patch is.
- ShapeConv Formulation: Two learnable weights separately process the base and shape components, then combine their outputs element-wise into a same-sized shape-aware patch.This reweighted patch retains shape information while allowing the layer to balance shape and base information adaptively.
- Training and Inference: ShapeConv shifts the base and shape product operations from patches to kernels, defining KBS = KB + KS for standard convolutional processing.The base-kernel and shape-kernel are constructed from the original kernel’s components, making the training implementation mathematically equivalent to the patch formulation.
- Training and Inference: During inference, WB and WS become constants and fuse into KBS, so ShapeConv has the same tensor size and zero additional inference computation or memory.The fused kernel makes the resulting layer identical to vanilla convolution.
- ShapeConv-enhanced Network Architecture: ShapeConv replaces vanilla convolutions in most CNNs after RGB and depth information are concatenated, allowing existing RGB architectures to consume RGB-D data.Depth input can be represented by depth values or HHA images, and ShapeConv can be used in both backbone and segmentation stages.
4. Experiments
Experiments evaluate ShapeConv across three indoor RGB-D benchmarks, five architectures, and ablations, showing consistent gains over vanilla-convolution baselines, especially near object boundaries.
- Datasets and metrics: Experiments use NYUDv2-13/-40, SUN-RGBD, and SID, reporting Pixel Accuracy, Mean Accuracy, Mean IoU, and frequency-weighted IoU.NYUDv2 contains 1,449 RGB-D scene images, split into 795 training and 654 testing images.
- Comparison protocol: Baseline comparisons replace only vanilla convolutional layers with ShapeConv, isolating the effect of the proposed layer.Other architecture and experimental settings remain unchanged.
- Experiments on different datasets: ShapeConv architectures outperform baselines by a large margin on both NYUDv2-13 and NYUDv2-40 under all reported settings.On NYUDv2-13, ShapeConv also achieves the best values across all four metrics and about 6.3% higher Mean IoU than a recent method.
- Experiments on different datasets: ShapeConv improves results on SUN-RGBD and surpasses state-of-the-art methods on SID, including a reported 6% Mean IoU improvement over a strong ResNet-101 baseline.On SUN-RGBD, the ResNet-50 ShapeConv-enhanced network exceeds several methods using deeper backbones.
- Experiments on different architectures: Across Deeplabv3+, Deeplabv3, UNet, PSPNet, and FPN with ResNet-50 or ResNet-101, ShapeConv brings significant improvements on NYUDv2-40.The results are presented as evidence of generalization across architectures and backbones.
- Visualization and ablation study: ShapeConv improves segmentation around object boundaries, outperforming the baseline across all trimap widths.Ablations also show that removing either the base or shape weight degrades performance; using both achieves the best reported values.
5. Conclusion
ShapeConv decomposes depth patches into shape and base components, reweights them before convolution, and improves RGB-D segmentation without added inference computation or memory. Its broader application remains bounded by the greater difficulty of shape-base decomposition for point clouds.
- ShapeConv decomposes each input patch into shape and base components, then applies corresponding learnable weights before convolution.The method is designed to emphasize shape information when beneficial for RGB-D semantic segmentation.
- ShapeConv yields promising results across several challenging indoor RGB-D semantic segmentation benchmarks.
- Inference adds no computation or memory compared with vanilla convolution while achieving superior performance.
- Extending shape-base decomposition to point clouds is more challenging because of their additional degree of freedom.The authors identify this as a future application challenge rather than a demonstrated failure on the evaluated benchmarks.