Source-linked AI summary
Tangent Convolutions for Dense Prediction in 3D
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou
TL;DR
Large-scale 3D scene analysis is constrained by volumetric representations and by methods that do not generalize well to unstructured point clouds. The paper introduces tangent convolutions on local tangent planes and builds a fully convolutional segmentation network that scales to millions of points and outperforms baselines across diverse scenes.
Problem
Existing 3D deep-learning systems often use representations with poor scalability or limited support for general, unstructured scene analysis.
Method
Tangent convolutions project local surface geometry onto virtual tangent planes, providing the building block for a fully convolutional U-type point-cloud segmentation network.
Results
The method produces high scores on all evaluated datasets and consistently outperforms the baselines in quantitative comparisons.
Takeaways & Limitations
Tangent-convolution networks can be evaluated efficiently on large point clouds with millions of points for semantic segmentation of indoor and outdoor scenes.
Takeaways & Limitations
Baseline methods were difficult to apply consistently across datasets, and none showed consistent performance across different scene types.
Abstract
from arXiv · showhide
We present an approach to semantic scene analysis using deep convolutional networks. Our approach is based on tangent convolutions - a new construction for convolutional networks on 3D data. In contrast to volumetric approaches, our method operates directly on surface geometry. Crucially, the construction is applicable to unstructured point clouds and other noisy real-world data. We show that tangent convolutions can be evaluated efficiently on large-scale point clouds with millions of points. Using tangent convolutions, we design a deep fully-convolutional network for semantic segmentation of 3D point clouds, and apply it to challenging real-world datasets of indoor and outdoor 3D environments. Experimental results show that the presented approach outperforms other recent deep network constructions in detailed analysis of large 3D scenes.
1. Introduction
The paper addresses the scalability and representation limits of volumetric 3D convolution by introducing tangent convolutions for surface-based scene analysis. It applies the construction to dense semantic segmentation of large indoor and outdoor point-cloud scenes.
- Voxel-based convolutions suffer cubic memory and computation growth, forcing low-resolution grids that limit prediction accuracy for detailed large-scale scenes.Octree methods support higher resolutions, but may still be insufficient for detailed analysis.
- 3D sensor data typically represents surfaces rather than true volumes, motivating convolutional operations that acknowledge latent surface structure.RGB-D cameras and LiDAR capture 2D structures embedded in 3D space.
- The approach is presented as an alternative to point-set, graph, and sphere-surface methods, whose limitations include weak local-structure sensitivity or restrictive topological assumptions.These alternatives address shortcomings of voxel-based methods but introduce their own constraints.
- Tangent convolutions project local surface geometry onto a tangent plane around every point, producing tangent images processed by convolutional networks.The construction assumes locally Euclidean surfaces and requires only approximate normal-vector estimation; latent surfaces need not be known.
- The paper designs a U-type network using tangent convolution as its main building block for dense semantic segmentation of point clouds.The architecture is intended for large-scale scenes and is evaluated on three diverse real-world indoor and outdoor datasets.
2. Related Work
Prior 3D deep-learning systems use voxel grids, images, point sets, graphs, surfaces, and other representations, but often lack general scene coverage or scalability. The paper positions tangent convolutions as a scalable alternative for detailed whole-scene analysis.
- Voxel-grid networks support several 3D tasks but cubic complexity typically restricts them to resolutions no higher than 64^3, limiting accuracy.Octrees and kd-trees improve memory and computation efficiency and permit higher resolutions.
- RGB-D and multi-view methods can leverage powerful 2D networks, but they are not generally applicable to unstructured point clouds with unknown sensor poses.The proposed approach can be viewed as associating a virtual camera with each point while addressing efficient implementation.
- Point-set networks based on independent point processing and global maxpooling have weak communication between points, creating difficulties in large scenes with complex layouts.The cited limitation concerns how local point interactions are represented.
- Other formulations target shape segmentation, correspondence, classification, synthesis, or specialized surface domains rather than large-scale semantic segmentation of whole scenes.These methods provide relevant ideas but address different task or domain scopes.
- Most existing 3D deep-learning systems either lack representations suitable for general scene analysis or have poor scalability.The paper claims tangent-convolution networks scale to millions of points and support detailed analysis of large scenes.
3. Tangent Convolution
Tangent convolution projects local surface geometry onto a tangent plane, forms tangent images, interpolates scattered signals, and applies a convolution kernel. The construction supports point clouds and can be evaluated without explicitly maintaining tangent images.
- Tangent image construction: Tangent convolution projects each point’s local surface neighborhood onto a tangent plane, producing a tangent image for convolution.The tangent plane uses locally estimated surface orientation; projected neighbors become 2D points on the image plane.
- Tangent plane estimation: Surface normals come from the smallest-eigenvalue covariance eigenvector, while the other two eigenvectors define tangent-image axes.This local covariance analysis estimates the tangent-plane orientation around each point.
- Signal interpolation: Projected neighbor signals are interpolated over the tangent image using nearest-neighbor or Gaussian kernel-mixture schemes.The projected points are scattered, so interpolation estimates the full continuous signal S(u).
- Signal interpolation: Nearest-neighbor interpolation is used mostly because more sophisticated schemes showed no significant empirical performance effect.The authors also consider Gaussian mixtures, including variants restricted to top-k neighbors.
- Tangent convolution evaluation: The tangent image need not be explicitly maintained: it supplies the domain and contributes to the geometric weights used in convolution.This observation underlies efficient evaluation on point clouds with millions of points.
4. Efficiency
Tangent convolutions are evaluated by precomputing geometry-dependent selections and weights, gathering point features into intermediate tensors, and applying flattened convolution kernels. The resulting operations support efficient convolutional layers and differentiable training.
- Discrete evaluation: The tangent image is discretized on a regular l×l grid, allowing tangent convolution to use ordinary discrete convolution machinery.Elements u become pixels in the virtual image, and c is a discrete kernel on that grid.
- Nearest-neighbor implementation: Nearest-neighbor evaluation precomputes an N×L index matrix, gathers features into an N×L×Cin tensor, and applies flattened kernels to produce Fout.The selection function depends only on point-cloud geometry, not on the signal F.
- Gaussian-mixture implementation: Gaussian-mixture evaluation restricts each pixel to the top-k neighbors, so each pixel has at most k nonzero interpolation weights.The corresponding selection functions and weights are independent of the signal and can be precomputed.
- Layer computation: For multiple interpolation weights, intermediate tensors are weighted and summed before convolution with the kernel W.The implementation represents selections as index matrices Ii and weights as matrices Hi.
- Training: The implementation consists entirely of differentiable atomic operations, enabling backpropagation through automatic differentiation.The construction was implemented in TensorFlow.
5. Additional Ingredients
The network adds pooling, unpooling, and special handling for tangent-plane distance signals to support multiscale point-cloud analysis. Grid hashing keeps these operations scalable while preserving correspondence between resolutions.
- Pooling: Grid hashing pools signals from points sharing a grid location, with progressively coarser grids providing multiscale aggregation.The grid step doubles after each pooling layer, beginning at an example 5cm resolution.
- Multi-scale analysis: Pooling doubles tangent-plane radius and tangent-image pixel size, reducing point-cloud and tangent-image resolution together.The downsampled geometry and indexing structures are independent of the signals defined on the points.
- Efficiency: All hashing stages have linear complexity in the number of points without constructing grids explicitly.The data structures remain linear in point count, independent of point-set extent or grid resolution.
- Unpooling: Unpooling reuses pooling indices to copy low-resolution features back onto the higher-resolution points that contributed to them.
- Local distance feature: Tangent-plane distance is a point-relative signal precomputed as distance from each neighbor to the reference point’s tangent plane, then concatenated into the first-layer input tensor.This signal receives special treatment because it cannot be expressed as a scalar value independently associated with each point.
6. Architecture
The architecture is a fully convolutional encoder-decoder over point clouds, using tangent convolutions, pooling, unpooling, and skip connections. Its final layer produces class predictions for each point.
- Architecture: The network uses an encoder-decoder design with two pooling layers, two corresponding unpooling layers, and skip connections between matching blocks.
- Architecture: All layers except the classifier use 3 × 3 filters followed by Leaky ReLU, while the final 1×1 convolution produces class predictions.
- Training: The network is trained with cross-entropy loss using Adam with an initial learning rate of 10^-4.
- Receptive field: Each convolutional layer’s receptive field uses R = 2r, and the reported final receptive-field size is 200cm for initial r = 5cm.
7. Experiments
The experiments evaluate tangent-convolution networks for semantic 3D scene segmentation across three indoor and outdoor datasets, using class-balanced measures on original point clouds. The method achieves high scores and consistently outperforms the baselines, while input-signal effects and baseline limitations vary by dataset.
- Datasets: The evaluation covers Semantic3D, S3DIS, and ScanNet, spanning large-scale outdoor and indoor scans with 8, 13, and 20 object classes, respectively.The datasets contain over 3 billion points, six indoor areas, and more than 1,500 indoor scans.
- Measures: Performance is reported with mean accuracy over classes, mean intersection over union, and overall accuracy from confusion matrices on original point clouds.Predictions from downsampled or voxelized methods are mapped back by nearest-neighbor assignment.
- Main results: Overall, the proposed method produces high scores on all datasets and consistently outperforms the compared baselines.Quantitative results are summarized in Table 2 and qualitative comparisons in Figure 6.
- Input signals: Geometry is more important than color indoors, whereas color significantly improves performance on Semantic3D.On ScanNet, RGB is harmful for mean and overall accuracy, while per-scan height normalization can make height-based signals harmful across scans.
- Baseline comparison: The baselines are difficult to apply across datasets because each requires non-trivial dataset-specific data-preparation decisions.None of the baselines shows consistent performance across the different scene types.
- Baseline comparison: PointNet performs poorly on noisy ScanNet and fails to produce meaningful predictions on Semantic3D despite high indoor overall-accuracy scores.Its high overall accuracy is strongly influenced by dominant classes such as walls, floors, and ceilings.
- Baseline comparison: ScanNet’s configuration has reasonable indoor overall accuracy but much worse mean accuracy and mean intersection over union, and it was not evaluated on Semantic3D because of scalability.Its central-column output makes full-scene evaluation extremely time-consuming.
- Baseline comparison: OctNet performs well on Semantic3D but poorly indoors, possibly because of poor generalization from overfitting to training-octree structure.
8. Conclusion
The paper introduces tangent convolutions for 3D data by evaluating convolutions on virtual tangent planes at every point. These convolutions support efficient processing of large point clouds and semantic segmentation of indoor and outdoor scenes.
- Tangent convolutions evaluate convolutions on virtual tangent planes at every point.
- Precomputed tangent planes enable efficient evaluation of deep convolutional networks on large point clouds.
- The method applies tangent convolutions to semantic segmentation of large indoor and outdoor scenes.
- The authors suggest tangent convolutions may also apply to other problems in 3D data analysis, processing, and synthesis.
A. Robustness to noise
The robustness experiment evaluates tangent convolutions under additive Gaussian noise and compares signal interpolation schemes. Reasonable noise can improve overall accuracy, while severe noise harms performance when semantic structure is damaged.
- The experiment trains and tests networks on S3DIS point clouds perturbed with varying additive Gaussian noise.Training used Area 1 and testing used Area 5, so these results are not compatible with the main-paper performance numbers.
- Reasonable amounts of noise improve overall accuracy, but performance suffers when noise severely damages semantic structure.
- No pipeline parameters were tuned for the noise experiments.
- Nearest-neighbor and Gaussian-mixture signal interpolation produce very similar results on S3DIS using distance and height inputs.The paper therefore uses the simpler nearest-neighbor interpolation scheme.
C. Comparison with SnapNet
The paper compares tangent convolutions with SnapNet, which achieves a higher Semantic3D mIoU but could not be applied to the indoor datasets because of camera-pose requirements.
- 67.7 mIoU for SnapNet exceeds the 66.4 mIoU produced by tangent convolutions on Semantic3D.
- SnapNet could not be applied to the indoor datasets because its non-trivial camera-pose sampling procedure was required.
D. Qualitative results
The paper presents qualitative results and comparisons across S3DIS, ScanNet, and Semantic3D. Figure 6 specifically compares results on S3DIS and Semantic3D using color-coded labels.
- Additional qualitative results are provided for S3DIS, ScanNet, and Semantic3D.
- The qualitative comparisons complement the paper's quantitative evaluation of semantic segmentation methods.