PaperlayerView exact revision on arXiv

Source-linked AI summary

Nesti-Net: Normal Estimation for Unstructured 3D Point Clouds using Convolutional Neural Networks

Yizhak Ben-Shabat, Michael Lindenbaum, Anath Fischer

arXiv:1812.00709v1cs.CV

TL;DR

Estimating normals from unstructured point clouds is difficult because noise, density variation, and detail level complicate neighborhood selection and CNNs require structured input. Nesti-Net combines local multiscale point statistics with mixture-of-experts scale selection. It reports state-of-the-art normal-estimation results and robustness to noise and occlusion, while scanned-data results reflect scanner-specific noise.

  • Problem

    Normal estimation from raw point clouds is challenging across sampling densities, noise, outliers, and detail levels, while CNNs require structured input that point clouds lack.

    Source: §1 · p. 1

  • Method

    Nesti-Net feeds local multiscale point statistics into a mixture-of-experts CNN that predicts the neighborhood scale for each point.

    Source: §2.2 · p. 3, §3.1 · p. 4, §3.2 · p. 4

  • Results

    Nesti-Net achieves state-of-the-art results relative to other methods and demonstrates robustness to noise and occlusion data corruptions.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The mixture-of-experts design enables prediction of an optimal local scale and provides insights into the network's resource distribution.

    Source: §5 · p. 8

    Scanned-data results reflect correlated, non-Gaussian scanner noise, and further improvement may require training with scanner noise and ground-truth normals.

    Source: §4.5 · p. 7

Abstract

from arXiv · show

In this paper, we propose a normal estimation method for unstructured 3D point clouds. This method, called Nesti-Net, builds on a new local point cloud representation which consists of multi-scale point statistics (MuPS), estimated on a local coarse Gaussian grid. This representation is a suitable input to a CNN architecture. The normals are estimated using a mixture-of-experts (MoE) architecture, which relies on a data-driven approach for selecting the optimal scale around each point and encourages sub-network specialization. Interesting insights into the network's resource distribution are provided. The scale prediction significantly improves robustness to different noise levels, point density variations and different levels of detail. We achieve state-of-the-art results on a benchmark synthetic dataset and present qualitative results on real scanned scenes.

1. Introduction

Nesti-Net addresses the challenges of estimating normals from noisy, irregular point clouds and of using CNNs with unstructured data. It combines a local multiscale representation with data-driven scale selection and reports state-of-the-art normal-estimation performance.

1. Introduction

  • Raw point clouds lack connectivity and may be incomplete, noisy, and density-varying, making them challenging inputs for CNNs that require structured, grid-like data.Normal estimates also depend on a neighborhood scale: large neighborhoods smooth sharp details, while small ones are more noise-sensitive.Source: §1 · p. 1
  • Pointwise surface normals provide local geometric information that can improve tasks such as over-segmentation, classification, and surface reconstruction.The passage cites over-segmentation [3], classification [21], and surface reconstruction [11].Source: §1 · p. 1
  • Nesti-Net outperforms state-of-the-art methods for normal estimation and increases robustness to different noise levels, outliers, and levels of detail.The method is designed for unstructured 3D point clouds.Source: §1 · p. 1
  • Nesti-Net combines a local point representation suitable for CNN input with mixture-of-experts scale prediction to estimate normals from unstructured point clouds.The pipeline computes MuPS for each point, then uses a scale manager to select the corresponding expert sub-network.Source: §1 · p. 1, Figure 1 · p. 2

2. Related-work

Prior point-cloud learning methods address unstructured inputs in different ways, while normal-estimation methods face a tradeoff between noise robustness and detail preservation. Nesti-Net adapts 3DmFV to local multiscale neighborhoods and learns neighborhood size with a mixture of experts.

2.1. Deep learning for unstructured 3D point clouds

  • PointNet, kd-Networks, and voxel methods address unstructured point clouds differently, while Nesti-Net proposes a point-wise, multiscale variation of 3DmFV.Voxel methods face accuracy-complexity tradeoffs; PointNet uses an order-insensitive function on point representations, and kd-Networks learn shared weights over a kd-tree.Source: §2.1 · p. 2

2.2. Normal estimation

  • Classic neighborhood-fitting methods trade noise robustness against preserving sharp details, and optimal scale estimation can require unknown noise levels and costly iterative parameter estimation.Voronoi methods are robust to sharp features but sensitive to noise; some also rely on preprocessing and fine-tuned parameters.Source: §2.2 · p. 2
  • HoughCNN loses some 3D information during transformation, while PCPNet processes multiple scales jointly without selecting an optimal scale.Joint multiscale processing can encourage averaging rather than specialization; Nesti-Net instead learns neighborhood size with a mixture of experts.Source: §2.2 · pp. 2–3

2.3. Representing point clouds using 3DmFV

  • MuPS extends 3DmFV by computing a point-wise multiscale 3DmFV representation.The earlier 3DmFV representation had achieved good results for point-cloud classification using a 3D CNN.Source: §2.3 · p. 3

3. Approach

Nesti-Net computes a local multi-scale point representation (MuPS), then feeds it to a mixture-of-experts CNN to estimate each point’s normal. MuPS encodes neighborhoods at multiple scales relative to a coarse Gaussian grid, and the network selects an expert based on its predicted scale suitability.

3.1. MuPS - Multi-scale point statistics

  • The network’s 3DmFV descriptors use normalized Fisher Vector gradients and add symmetric minimum and maximum functions to represent unordered point sets.The Fisher Vector is defined using point likelihoods under a Gaussian mixture model.Source: §3.1 · p. 3
  • For each point, MuPS extracts neighborhoods at several radii, normalizes each around that point, computes a 3DmFV per scale, and concatenates the descriptors.Large neighborhoods are capped by randomly sampling up to a design-parameter threshold of points.Source: §3.1 · p. 4
  • MuPS uses symmetric functions to make each scale independent of point count and order, while its Gaussian-mixture grid supplies the structure required by CNNs.The multi-scale representation also incorporates fine detail and noise robustness.Source: §3.1 · p. 4

3.2. The Nesti-Net architecture

  • The scale manager processes MuPS and outputs scores interpreted as the probabilities that each expert will estimate the normal correctly.Its described layers include 3D-Inception-inspired convolutions, max pooling, and fully connected layers.Source: §3.2 · p. 4
  • At test time, Nesti-Net selects the normal from the expert with the highest scale-manager score; its seven experts receive individual scales or the full MuPS representation.Two experts each receive the 1%, 3%, and 5% scales, while the seventh receives all scales.Source: §3.2 · pp. 4–5

4. Evaluation

The evaluation tests Nesti-Net on synthetic benchmark shapes and scanned indoor scenes. It compares normal-estimation performance and scale-selection variants, and examines expert assignments and scanned-data behavior.

4.1. Datasets

  • The synthetic test set contains 22 shapes, while training uses eight densely sampled shapes augmented with three Gaussian-noise levels.The evaluation also uses qualitative scanned scenes from ScanNet and NYU Depth V2.Source: §4.1 · p. 5

4.2. Training details

  • Training uses 32,768 random samples per epoch, 512 neighbors per point, and an eight-cell Gaussian grid for MuPS.The experiments ran in TensorFlow on a single NVIDIA Titan Xp GPU.Source: §4.2 · p. 5

4.3. Normal estimation performance

  • Nesti-Net outperforms the compared methods across all noise levels and most density variations in unoriented normal estimation.Table 1 compares RMS angle error for Nesti-Net, PCA, Jet, PCPNet, and HoughCNN.Source: §4.3 · p. 5, Table 1 · p. 6
  • Errors occur more often near edges, corners, and detailed high-curvature regions, where Nesti-Net suffers least; small-scale experts 1–2 favor high-curvature corners, while medium-scale experts 3–4 handle close surfaces or noise.Experts 5–6 specialize in planar surfaces with specified normal-direction components; expert 7 uses combined scales and handles planar surfaces, complex geometry, and low to medium noise.Source: §4.3 · pp. 5–6

4.4. Scale selection performance

  • Nesti-Net’s learned scale selection beats the tested single-scale, multi-scale, and switching variants; selecting the correct scale outperforms concatenating scales.Unlike switching, its scale prediction does not require noise parameters as training input.Source: §4.4 · p. 7, Table 2 · p. 7

4.5. Results on scanned data

  • On scanned data, training with TV-derived normals significantly improves estimates of the underlying surface, while synthetic Gaussian-noise training reflects raw scan noise.TV smooths fine detail and uses the depth image; Nesti-Net estimates from the raw point cloud without its grid structure.Source: §4.5 · p. 7, Figure 7 · p. 8

5. Summary

The paper presents MuPS and Nesti-Net for normal estimation on unstructured point clouds, reporting state-of-the-art results and robustness to noise and occlusion corruptions.

  • MuPS represents 3D point clouds on a grid while encoding fine and coarse details.Source: §5 · p. 8
  • Nesti-Net applies a CNN to the representation for normal estimation, with a mixture-of-experts architecture that predicts an optimal local scale.The architecture also provides insight into the network’s resource distribution.Source: §5 · p. 8
  • The representation and architecture achieve state-of-the-art results relative to other methods and demonstrate robustness to noise and occlusion corruptions.Source: §5 · p. 8

6. Acknowledegment

The authors acknowledge NVIDIA Corporation for donating the GPU used in the research.

  • NVIDIA Corporation donated the Titan Xp GPU used for this research.Source: §6 · p. 8

A. Scale selection methods: architecture details

The appendix describes three ablation architectures: single-scale, multi-scale, and multi-scale switching.

A. Scale selection methods: architecture details

B. Normal estimation performance analysis

The section explains why RMS error alone is insufficient and presents complementary metrics and visual comparisons for assessing normal estimates and scale choices.

B. Normal estimation performance analysis

  • RMS error favors averaging methods near corners, so PGPα complements it by measuring the percentage of points below an angular-error threshold.PGP10 counts errors below 10 degrees; the section also reports PGP5.Source: B. Normal estimation performance… · p. 10
  • The additional results visualize predicted normals, angular-error heatmaps, and the expert scale selected at each point.Figure 10 maps angular error to a 0–60 heatmap and reports RMS error above each point cloud.Source: B. Normal estimation performance… · p. 10
  • PCA at small scale produces noisy outputs, while large scale over-smooths fine details and corners.The section qualitatively compares Nesti-Net with medium-scale PCA and also presents small- and large-scale PCA results.Source: B. Normal estimation performance… · p. 10

C. Time complexity and timing

The section analyzes MuPS and network costs, then compares timing across Nesti-Net variants and other methods, including the speed–accuracy tradeoff of a lighter representation.

C. Time complexity and timing

  • MuPS computation has time complexity O(nKTmax), while normal-estimation network time is constant and proportional to its number of operators.Here n is the number of scales, K the number of Gaussians, and Tmax the maximum points in a scale subset.Source: C. Time complexity and timing · pp. 10–11
  • Adding experts increases training time but not test time, whereas additional scales add scale-manager operators.Only one expert is evaluated at test time; normal-estimation time is independent of the number of points.Source: C. Time complexity and timing · p. 11
  • Nesti-Net light averages RMS error 13.5, remaining superior to the other methods by a smaller margin as the lower-Gaussian MuPS runs faster.Figure 12 compares single-scale, multi-scale, and mixture-of-experts variants using 83 Gaussians and 33 in the light versions; time is measured in ms per point.Source: C. Time complexity and timing · p. 11, Figure 12 · p. 13
  • PCA and Jet are fastest, learning-based methods have comparable timings, and none achieves real-time performance.Figure 13 compares methods in ms per point; measurements used different software frameworks but the same machine, and Nesti-Net timing is less sensitive to points per scale than the other methods.Source: C. Time complexity and timing · p. 11, Figure 13 · p. 13
Loading 1812.00709v1…