PaperlayerView exact revision on arXiv

Source-linked AI summary

SPLATNet: Sparse Lattice Networks for Point Cloud Processing

Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz

arXiv:1802.08275v4cs.CVcs.GR

TL;DR

Point-cloud networks lacked flexible spatial connectivity while preserving surface structure. SPLATNet uses sparse bilateral convolutional layers for hierarchical, spatially aware, and joint 2D–3D processing, outperforming state-of-the-art methods on two segmentation benchmarks while remaining computationally efficient.

  • Problem

    Existing point-cloud architectures lacked flexible filter neighborhoods and could lose surface information by aggregating points without explicitly modeling spatial layouts.

  • Method

    SPLATNet builds hierarchical, spatially aware point-cloud networks from bilateral convolutional layers operating on flexible sparse lattices, including joint 2D–3D processing.

  • Results

    Experiments on RueMonge2014 and ShapeNet segmentation show superior performance over state-of-the-art techniques while remaining computationally efficient.

  • Takeaways & Limitations

    SPLATNet provides a favorable architecture for segmentation from point clouds and jointly processed 2D images and 3D data.

Abstract

from arXiv · show

We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice. Naively applying convolutions on this lattice scales poorly, both in terms of memory and computational cost, as the size of the lattice increases. Instead, our network uses sparse bilateral convolutional layers as building blocks. These layers maintain efficiency by using indexing structures to apply convolutions only on occupied parts of the lattice, and allow flexible specifications of the lattice structure enabling hierarchical and spatially-aware feature learning, as well as joint 2D-3D reasoning. Both point-based and image-based representations can be easily incorporated in a network with such layers and the resulting model can be trained in an end-to-end manner. We present results on 3D segmentation tasks where our approach outperforms existing state-of-the-art techniques.

NVIDIA

This section lists Subhransu Maji and associates him with UMass Amherst.

  • Subhransu Maji is listed with UMass Amherst.

1. Introduction

SPLATNet is introduced to process sparse, unordered point clouds directly, addressing limitations of grid conversions and inflexible point-based aggregation. Built from sparse bilateral convolution layers, it supports efficient hierarchical, spatially aware, and joint 2D–3D feature learning for segmentation.

  • Motivation: Point clouds are sparse and unordered, making traditional CNN architectures difficult to apply directly and motivating a new processing architecture.Existing direct approaches are dominated by hand-crafted features, while CNN-based preprocessing commonly converts clouds into images or voxels.
  • Problem: Converting point clouds to images or voxels can introduce artifacts and discard natural invariances.These representations are attractive because convolutions are easy to implement on regular 2D or 3D grids.
  • Problem: Existing point-cloud architectures provide limited control over spatial connectivity because pooling aggregates information globally or hierarchically.Such aggregation may lose surface information because it does not adequately preserve point spatial layouts.
  • Method: SPLATNet uses bilateral convolution layers that map unordered points to a sparse lattice, convolve there, and interpolate filtered signals back to the points.This construction enables flexible lattice structures and hierarchical, spatially aware feature learning without voxel or image preprocessing.
  • 2D–3D integration: The architecture also supports jointly processing multi-view images and corresponding 3D points in one end-to-end learnable 2D–3D network.It allows mappings between 2D and 3D points in both directions.
  • Results: Experiments on RueMonge2014 facade and ShapeNet part segmentation report superior performance to state-of-the-art techniques while remaining computationally efficient.The proposed versions are SPLATNet3D and SPLATNet2D-3D.

2. Related Work

Prior 3D shape-processing networks use multi-view, voxel, point-cloud, non-Euclidean, and joint 2D-3D representations, each with distinct limitations. SPLATNet addresses these distinctions by mapping 2D and 3D features onto a shared lattice for end-to-end learning.

  • Multi-view and voxel networks: Multi-view networks exploit high-resolution rendered images and transfer learning, but projections can lose surface information through self-occlusion and depend on viewpoint selection.They render surface depth and normals under multiple 2D projections before applying image-based architectures.
  • Multi-view and voxel networks: Regular voxel-grid convolutions are limited to low shape resolutions by high memory and computation costs.Recent methods use adaptively subdivided hierarchical grids with denser surface cells to reduce computational and memory overhead, although convolutions may still occur away from the surface.
  • Point cloud networks: Point-cloud networks directly process points and aggregate learned features, but pooling can lose surface information because point spatial layout is not explicitly considered.Aggregation occurs across the whole point set or hierarchical surface regions through max-pooling.
  • Non-Euclidean networks: Non-Euclidean networks convolve in graph spectral domains, but structurally different shapes can have different spectral bases, leading to poor generalization.Spectral-transformer alignment methods address basis variation but can be undesirable for man-made object tasks involving large deformations that alter shape or part semantics.
  • Joint 2D-3D networks: Unlike late-stage fusion of volumetric and multi-view classification scores, SPLATNet maps 2D and 3D feature representations onto the same lattice for end-to-end learning.FusionNet combines the two networks only after their final fully connected layers and therefore does not jointly consider intermediate local and global representations.

3. Bilateral Convolution Layer

The Bilateral Convolution Layer (BCL) is SPLATNet’s basic building block, applying learnable convolution in a flexibly specified high-dimensional lattice through splat, convolve, and slice operations. Its sparse indexing and separate input/output lattice specifications support efficient, spatially aware point-cloud and cross-dimensional processing.

  • Bilateral Convolution Layer: BCL processes features through three steps: splat onto a lattice, convolve with learnable filters, and slice the filtered signal back to points.Splatting and slicing use barycentric interpolation, while convolution operates in the lattice space.
  • Lattice Representation: BCL uses a permutohedral lattice for efficient projection, with simplex spacing controlled by scaling the lattice features using a diagonal scaling matrix Λ.The lattice dimensionality is determined by the lattice features; position and color can define a 6-dimensional XYZRGB filtering space.
  • Flexible Connectivity: BCL supports different input and output points by specifying separate input and output lattice features, allowing filtered signals to be sliced onto points of interest.The output lattice features Lout ∈ R^m×d_l describe m output points.
  • Cross-Dimensional Processing: BCL can project input signals into a different-dimensional filtering space, including projecting a 2D image into 3D space for filtering.This follows from separately specifying input features and lattice features.
  • Sparse Computation: BCL uses hash tables to index populated lattice vertices and performs convolutions only at those locations, enabling efficient processing of sparse high-dimensional inputs.This avoids applying convolution across unoccupied portions of the lattice.

4. SPLATNet3D for Point Cloud Processing

SPLATNet3D operates directly on 3D point clouds, producing per-point predictions through a network built from pointwise convolutions and sparse bilateral convolutional layers. Its lattice spaces and scales control point connectivity, signal resolution, receptive fields, and hierarchical feature learning.

  • Point-cloud interface: SPLATNet3D takes a 3D point cloud with point locations and optional features such as RGB color, surface normals, and curvature.The input is P ∈ R^n×d, with d ≥ 3 feature dimensions including XYZ locations.
  • Point-cloud interface: The model produces per-point predictions for tasks including 3D semantic segmentation and object part labeling, while global pooling enables classification outputs.Global pooling converts point-wise features into a single output vector.
  • Network architecture: The architecture applies a pointwise 1 × 1 CONV layer, T BCLs on 3D lattices built from XYZ locations, concatenation across BCL scales, two further 1 × 1 CONV layers, and softmax classification.The initial 1 × 1 CONV processes points independently; concatenation aggregates outputs from BCLs operating at different lattice scales.
  • Lattice spaces and their scales: Lattice scales directly control signal resolution and receptive fields, with coarser lattices increasing a filter’s effective receptive field.Increasing neighborhood size also enlarges receptive fields but can substantially increase cost in high dimensions.
  • Lattice spaces and their scales: The network benefits from finer lattices earlier and coarser lattices deeper, gradually increasing receptive fields to build hierarchical representations with varying spatial extents and abstraction levels.This ordering is described as consistent with common 2D CNN practice.
  • Lattice spaces and their scales: BCLs support alternative lattice spaces such as XYZRGB and surface-normal features, and adding an XYZ-normal BCL yielded minor performance improvements.Different lattice spaces impose different connectivity across input points.

5. Joint 2D-3D Processing with SPLATNet2D-3D

SPLATNet2D-3D jointly processes aligned images and point clouds by embedding 2D and 3D features into shared spaces through bilateral convolutional projections. Its differentiable modules support end-to-end training with losses defined on either domain or both.

  • Architecture: SPLATNet2D-3D extends SPLATNet3D with additional computational modules for joint 2D-3D processing.Its architecture is depicted in the green box of Figure 3.
  • 2D-to-3D Projection: A bilateral convolutional layer projects image-pixel features onto 3D point-cloud points using only splat and slice operations.Separate input and output lattice features allow projection even when pixel locations do not exactly correspond to point-cloud points.
  • 2D-3D Fusion: The 2D-3D Fusion module concatenates projected image features with SPLATNet3D features, applies 1 × 1 CONV layers, and predicts point-wise class probabilities.Both signals are embedded in the same 3D space before fusion.
  • 3D-to-2D Projection: The framework can back-project fused 3D features onto 2D images through a reversed bilateral convolutional operation controlled by smoothness hyperparameter Λb.The input and output lattice features are swapped relative to BCL2D→3D.
  • Training and Prediction: All joint-processing components are differentiable and trainable end-to-end, with losses defined on the 2D domain, 3D domain, or both in multi-task learning.CNN2 concatenates CNN1 output, input images, and back-projected features; a simple 2-layered network suffices in the experiments.

6. Experiments

SPLATNet is evaluated on RueMonge2014 facade labeling and ShapeNet Part segmentation using point-based, image-based, and joint 2D–3D inputs. The experiments report strong segmentation performance, runtime comparisons, and a marginal gain from six-dimensional filtering.

  • RueMonge2014 facade segmentation: On RueMonge2014, SPLATNet2D-3D achieves 69.8 IoU for point cloud labeling, outperforming the previous state of the art by 6.9 IoU.The joint model uses both 2D images and 3D data as input.
  • RueMonge2014 facade segmentation: For multi-view image labeling, SPLATNet2D-3D reaches 70.6 IoU, a 1.3 IoU improvement over using CNN1 alone.The experiment demonstrates the benefit of incorporating 3D information into 2D image segmentation.
  • ShapeNet Part segmentation: On ShapeNet Part, SPLATNet evaluates both SPLATNet3D and SPLATNet2D-3D for point-cloud part segmentation across 16 object categories.The dataset contains 16681 objects, with 2–6 part labels per category, and reports class-average, instance-average, and category-level mIoU.
  • ShapeNet Part segmentation: On one Nvidia GeForce GTX 1080 Ti, SPLATNet3D runs at 9.4 shapes/sec, SPLATNet2D-3D at 0.4 shapes/sec, and PointNet++ at 2.7 shapes/sec.For SPLATNet2D-3D, the high-resolution 2D network accounts for more than 95% of computation time.
  • ShapeNet Part segmentation: Adding a six-dimensional position-and-normal lattice feature produces a marginal 0.2 IoU improvement over standard SPLATNet3D for both class- and instance-average mIoU.The additional BCL is inserted between the final two 1 × 1 CONV layers.

7. Conclusion

SPLATNet directly processes point clouds with sparse, efficient lattice filters that learn hierarchical and spatially aware features while enabling bidirectional 2D–3D information mapping for joint point-cloud and multi-view image processing.

  • 7. Conclusion: SPLATNet directly takes point clouds as input and computes hierarchical and spatially-aware features with sparse and efficient lattice filters.The architecture is designed for direct point-cloud processing using lattice-based filtering.
  • 7. Conclusion: SPLATNet enables easy mapping of 2D information into 3D and vice-versa for joint processing of point clouds and multi-view images.This bidirectional mapping supports a unified network architecture for combining point-cloud and image information.
  • 7. Conclusion: Experiments on two different benchmark datasets evaluate the proposed networks.The supplied conclusion passage states the benchmark scope but does not provide the associated comparative results.

A. Point Cloud Density Normalization

BCL addresses uneven point density by normalizing responses at lattice vertices supported by different numbers of data points. It does so by comparing standard filtered responses with responses from a second Gaussian-filtering pass using unit-valued inputs.

  • Normalization mechanism: BCL includes a normalization scheme for uneven point density across lattice vertices.The issue is that some lattice vertices are supported by more data points than others.
  • Normalization mechanism: BCL first filters input signals with learnable filter kernels, then performs a second Gaussian-filtering pass after replacing signal values with 1s.The second pass produces responses that reflect support independently of the original signal values.
  • Normalization mechanism: Responses from the second filtering round normalize the responses from the first, similarly to using homogeneous coordinates.This separates signal filtering from support-based normalization.

B. RueMonge2014 Facade Segmentation

For RueMonge2014 facade segmentation, SPLATNet3D uses five sparse bilateral convolutional layers with skip connections, while SPLATNet2D-3D adds a 2D–3D fusion component and image-processing branches. Training uses sampled 60k-point facade segments, small batches, Adam, and data augmentation.

  • SPLATNet3D architecture: SPLATNet3D uses 5 BCLs followed by 2 1 × 1 CONV layers, with skip connections from every BCL to the penultimate convolution.Its layer channels are B64-B128-B128-B128-B64-C64-C7, and the initial 1 × 1 CONV layer is omitted because it has no observed performance effect.
  • SPLATNet2D-3D architecture: SPLATNet2D-3D combines SPLATNet3D with a 2D-3D Fusion component containing 2 1×1 CONV layers, C64-C7.CNN1 uses DeepLab, while CNN2 has C32-C7 with 3 × 3 filters in its first layer and 1×1 filters in its second.
  • Training: The model is trained by randomly sampling facade segments of 60k points with a batch size of 4 for SPLATNet3D.CNN1 is initialized with Pascal VOC pre-trained weights and fine-tuned for 2D facade segmentation.

C. ShapeNet Part Segmentation

For ShapeNet Part segmentation, SPLATNet3D uses a 1 × 1 convolutional layer, five bilateral convolutional layers, and two final 1 × 1 convolutional layers, while SPLATNet2D-3D adds a two-layer 2D-3D fusion component. The dataset contains labeling errors and inconsistencies that undermine benchmark reliability and make learning unnecessarily difficult.

  • Network architecture: SPLATNet3D comprises an initial 1 × 1 CONV layer, five BCLs, and two final 1 × 1 CONV layers.Its channel configuration is C32-B64-B128-B256-B256-B256-C128-Cx, with x representing 2–6 part categories.
  • Network architecture: SPLATNet2D-3D uses SPLATNet3D as its 3D component and adds two 1×1 CONV layers for 2D-3D fusion.CNN2 and BCL3D→2D are omitted because 2D prediction is unnecessary.
  • Training: Training uses separate category-specific models, Adam with an initial learning rate of 0.0001, and random rotations, translations, and scalings for augmentation.Networks are trained until validation loss plateaus.
  • Training: SPLATNet3D and SPLATNet2D-3D require about 2.5 and 3 days of training, respectively, compared with 3.5 days for PointNet++ on the same hardware.These durations use default settings for PointNet++.
  • Dataset labeling issues: ShapeNet Part includes frequent mislabeling, incomplete part coverage, inconsistent labels across shapes, and ambiguous ‘other’ categories.The first two issues make evaluation and comparison less reliable, while the latter two make learning ill-posed or unnecessarily hard.
Loading 1802.08275v4…