PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Kernel Surface Reconstruction

Jiahui Huang, Zan Gojcic, Matan Atzmon, Or Litany, Sanja Fidler, Francis Williams

arXiv:2305.19590v2cs.CV

TL;DR

Reconstructing surfaces from sparse, noisy point clouds is challenging because real-world inputs can be large and variable, and NKF is limited by dense solves and sensitivity to noise. NKSR uses gradient-based kernel fitting and compactly supported kernels with a sparse voxel hierarchy. The authors report state-of-the-art reconstruction quality and efficiency, with generalization to unseen data.

  • Problem

    Sparse, noisy point clouds vary in size and sampling, while NKF's dense solve and exact occupancy constraints limit scale and noise robustness.

    Source: §1 · p. 1

  • Method

    NKSR pairs gradient-based kernel fitting with compactly supported kernels and a sparse voxel hierarchy to make reconstruction robust to noise and scalable.

    Source: §1 · p. 2

  • Results

    The authors report state-of-the-art reconstruction quality and efficiency, with good generalization to unseen data.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors present NKSR as making deep-learning-based surface reconstruction more practical for general use.

    Source: §5 · p. 8

    Training uses paired noisy input points and dense supervision points, together with sensor origins for both sets.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

We present a novel method for reconstructing a 3D implicit surface from a large-scale, sparse, and noisy point cloud. Our approach builds upon the recently introduced Neural Kernel Fields (NKF) representation. It enjoys similar generalization capabilities to NKF, while simultaneously addressing its main limitations: (a) We can scale to large scenes through compactly supported kernel functions, which enable the use of memory-efficient sparse linear solvers. (b) We are robust to noise, through a gradient fitting solve. (c) We minimize training requirements, enabling us to learn from any dataset of dense oriented points, and even mix training data consisting of objects and scenes at different scales. Our method is capable of reconstructing millions of points in a few seconds, and handling very large scenes in an out-of-core fashion. We achieve state-of-the-art results on reconstruction benchmarks consisting of single objects, indoor scenes, and outdoor scenes.

1. Introduction

The paper targets reconstruction of implicit surfaces from sparse oriented points despite scale, sampling variation, noise, and ambiguous solutions. It builds on Neural Kernel Fields (NKF), changing the kernel formulation and support to pursue noise robustness and scalability while retaining generalization.

1. Introduction

  • Real-world point clouds can be very large, variably sampled, and noisy, while discrete observations admit many valid surface solutions.Reconstruction methods therefore combine agreement with input points and a prior over the rest of space.Source: §1 · p. 1
  • NKF generalizes well, but its globally supported kernel requires a dense solve and its exact positional occupancy constraints degrade under noise.The dense-system limitation prevents reconstruction above ten thousand input points.Source: §1 · p. 2
  • NKSR combines gradient-based kernel fitting with compactly supported kernels and a sparse voxel hierarchy to target noise robustness and large inputs.The authors build on NKF's generalization capability while addressing its practical limitations.Source: §1 · p. 2
  • The method is reported to generalize to out-of-distribution inputs and produce high-fidelity reconstructions amid sparsity and noise.This is presented as a capability of the method, not a quantified benchmark comparison.Source: §1 · p. 2
  • Training can combine diverse datasets using dense oriented points as supervision, without requiring occupancy or signed-distance preprocessing.Existing preprocessing requirements make diverse training data harder to use, particularly for non-watertight or open shapes.Source: §1 · p. 2
  • The method reconstructs millions of points in seconds and handles very large scenes out of core.The introduction also frames Figure 2 as a visual comparison to related works, without stating a specific comparison outcome in the supplied text.Source: §1 · p. 2, Figure 2 · p. 2

2. Related Work

Prior reconstruction work spans learned kernels, neural implicit fields, sparse scene representations, and other output forms. NKSR relates to these approaches through a data-dependent kernel while restricting its support and fitting weights with gradients.

2. Related Work

  • NKSR uses a data-dependent kernel like NKF, but restricts spatial support for efficiency and uses gradient-based fitting for noise robustness.The passage contrasts this with Neural Splines, which derives a non-learned kernel from infinitely wide ReLU networks.Source: §2 · p. 2
  • Prior implicit reconstruction methods include learned occupancy fields, sparse octree architectures, local-patch approaches, and point-upsampling methods followed by Poisson reconstruction.ConvONet and CIRCLE predict occupancy fields; ASR and DOGNN use octree-based architectures; LIG uses local patches; Shape as Points [48] upsamples points before Poisson reconstruction.Source: §2 · p. 2
  • Other reconstruction representations include dense point clouds, polygonal meshes, manifold atlases, and voxel grids, while neural fields also serve other point-cloud tasks.The cited tasks include shape compression, voxel-grid upsampling, and reconstruction from rotated inputs or articulated poses.Source: §2 · p. 3

3. Method

The method represents the predicted surface as the zero level set of an input-conditioned Neural Kernel Field supported on a predicted sparse voxel hierarchy. It fits the field to point positions and normals, and trains the network using dense oriented supervision.

3.1. Predicting a 3D Shape from Points

  • A sparse convolutional network predicts a voxel hierarchy and corner features that define input-conditioned positive-definite kernels; their weighted sum represents the surface as a zero level set.The kernels are centered at hierarchy voxels and have one-voxel-level support.Source: §3 · p. 3, §3.1 · pp. 3–4, Figure 4 · p. 5
  • The method solves for kernel weights by fitting field gradients to normals near voxel centers and field values to zero at input points.Compact kernel support makes the linear system sparse and positive definite, enabling efficient GPU solving.Source: §3.1 · p. 4
  • An optional masking branch excludes regions where its predicted value is at most 0.5 when extracting the final surface.The branch is intended to discard unwanted geometry at coarse hierarchy levels far from the predicted surface.Source: §3.1 · p. 4

3.2. Supervision

  • Training uses noisy input points and dense supervision points, with normals estimated from local neighborhoods and oriented toward their sensor origins.The supervision losses constrain near-surface field values and signs, normal agreement, distant geometry, and surface area; sensor-dependent terms are omitted when sensor information is unavailable.Source: §3.2 · pp. 4–5
  • The training requirements do not restrict the shapes used, and the method can mix object and scene data at different scales.The paper gives the example of noisy object scans paired with dense scans and single LiDAR frames paired with accumulated scene scans.Source: §3.2 · p. 4

4. Experiments

Experiments assess NKSR’s accuracy, scalability, and generalization across object-level and large-scale scene reconstruction. A model trained on a union of datasets performs well across varied inputs, though it slightly trails domain-specific models.

4. Experiments

  • The kitchen-sink model trained on a union of datasets slightly underperforms domain-specific models but outperforms most baselines across varied inputs.The experiments present it as a model usable across a wide variety of inputs.Source: §4 · p. 5

4.1. Accuracy: Object-level Reconstruction

  • NKSR reaches state-of-the-art performance on the object-level datasets; NKF degrades with noise, while SAP and NGSolver are more noise-robust but have less tight fitting.The authors attribute NKSR’s gain to gradient-based energy fitting and learned-kernel inductive biases.Source: Table 1 · p. 6, Table 2 · p. 6, §4.1 · p. 6

4.2. Scalability: Outdoor Driving Scenes

  • In this setting, POCO runs 161x slower than ours; SPSR is sensitive to input noise and sparsity, while POCO’s fitted surfaces fail to faithfully respect the input.Mean/min/max input point counts are 490k/290k/820k.Source: §4.2 · p. 7
  • 10M/11M points are reconstructed in 20s/35s, respectively, using the kitchen-sink model out of core on Waymo scenes.The left/right scenes contain 10M/11M points and take 20s/35s.Source: Figure 9 · p. 8
  • ShapeNet-trained NKSR generalizes best to the tested room datasets, and under extreme CARLA sparsity it still reconstructs complete geometry while baselines degrade.The room-level evaluation uses 10K input points; ShapeNet’s highest accuracy is attributed possibly to dataset diversity and similar geometric distributions.Source: Table 4 · p. 7, §4.3 · p. 8, Table 5 · p. 8

4.4. Ablation Study

  • Increasing kernel feature dimension yields only slight gains, voxel size has a larger influence, and removing hierarchy or gradient-based matrices degrades IoU.Figure 11’s back arrows indicate the setting used for Table 2.Source: Figure 11 · p. 8, §4.4 · p. 8

5. Conclusion

The paper presents NKSR as an accurate, scalable surface reconstruction algorithm using neural kernel fields. It identifies more expressive kernels and lower memory use as directions for future work.

5. Conclusion

  • NKSR is presented as an accurate and scalable surface reconstruction algorithm using the neural kernel field representation.The authors report state-of-the-art quality and efficiency and good generalization to unseen data.Source: §5 · p. 8
  • Future work will explore more expressive kernel models to improve reconstruction quality and reducing memory footprint for larger-scale reconstructions.Source: §5 · p. 8

Appendix

  • The appendix documents method derivations and network designs, experiment settings, pipeline extensions, and additional visualizations.Source: Appendix · p. 12

A.1. Network Architecture

The method predicts a sparse voxel hierarchy from point-cloud features, builds a learned hierarchical kernel field, and fits its coefficients through a linear system. Additional losses supervise hierarchy and masking, while overlapping scene chunks are reconstructed separately and merged in implicit form.

A.1. Network Architecture

  • A sparse-voxel U-Net predicts an adaptive hierarchy from voxelized point features, with structure classes controlling subdivision, retention, or deletion.The point encoder uses local coordinates and, for most demonstrated datasets, point normals; the predicted hierarchy remains covered by coarser voxels.Source: §A.1 · p. 12

A.2. Hierarchical Kernel Formulation

  • The implicit surface is a weighted sum of learned, positive-definite kernels centered on hierarchy voxels, with coefficients fitted to normals and near-zero values at input points.The solve minimizes gradient mismatch at hierarchy locations together with implicit values at input points; its optimal coefficients follow from a linear system.Source: §A.2 · pp. 13–14

A.3. Additional Losses

  • Structure cross-entropy supervises the predicted voxel hierarchy, while masking loss labels points within distance W of dense points as 1 and others as 0.The target hierarchy is recursively subdivided based on normal variation and constrained by the maximum adaptive depth L′.Source: §A.3 · p. 15

A.4. Out-of-Core Reconstruction

  • For very large scenes, overlapping chunks are reconstructed independently and their implicit fields are merged using masking outputs; a combined hierarchy supports mesh extraction.The merged implicit value averages chunk predictions weighted by their masking values; the merged mask takes the maximum across covering chunks.Source: §A.4 · p. 15, Figure 16 · p. 15

B. Experimental Settings

Training uses gradient accumulation, Adam, and a warm-up for predicted structure; the linear solver is Jacobi-preconditioned conjugate gradient. Dataset-specific parameters are chosen empirically, and the kitchen-sink model normalizes data to CARLA scale.

B.1. Hyperparameters

  • Training uses batch size 4 with gradient accumulation, Adam at initial learning rate 10^-4, and learning-rate decay to 70% every 50K iterations.Gradients are clipped at norm 0.5.Source: §B.1 · p. 15
  • A warm-up gradually replaces ground-truth structure with predicted structure, which the authors report stabilizes training.The Jacobi-preconditioned conjugate-gradient solver is used for forward and backward linear solves, with tolerance 10^-5 and typical convergence within several hundred iterations.Source: §B.1 · p. 15
  • Dataset-specific parameters are selected empirically, and kitchen-sink-model data are normalized to CARLA scale during training and testing.After normalization, the average is around 5 points per voxel.Source: §B.1 · p. 15

B.2. Baselines

The baselines use their official or default implementations where stated, with selected adaptations for normal input, retraining, or trimming. The NKF comparisons were run by its original authors.

B.2. Baselines

  • SPSR uses PoissonRecon with the voxel size matched to this method, and its mesh is trimmed using dataset-specific empirical density quantiles.Vertices below the selected density quantile are removed.Source: §B.2 · pp. 15–16
  • POCO uses its official implementation; where no pretrained model is available, it is trained without normals because the normal-input model performed consistently worse.NGSolver uses the official implementation with repository defaults.Source: §B.2 · p. 16
  • SAP and ConvONet use official configurations when possible, with their point encoders modified to concatenate normal inputs for normal-input comparisons.NKF comparisons were run by the original authors because its code was not publicly available.Source: §B.2 · p. 16

B.3. Metrics

Metrics are computed from densely sampled points and normals on ground-truth and predicted meshes. The section defines F-Score, Chamfer distance, and normal consistency, with dataset-dependent F-Score thresholds.

B.3. Metrics

  • Chamfer distance combines completion and accuracy, the mean nearest-neighbor distances from ground-truth to prediction and prediction to ground-truth.The paper notes this is consistent with ConvONet and differs from POCO's measure.Source: §B.3 · p. 16
  • Normal consistency averages absolute dot products between each sampled point's normal and the normal at its nearest neighbor on the other mesh.The metric uses both ground-truth-to-prediction and prediction-to-ground-truth directions.Source: §B.3 · p. 16
  • F-Score is the harmonic mean of precision and recall, with distance threshold ξ set to 0.01 for object and indoor datasets and 0.1 for CARLA.Precision measures the fraction of predicted points within ξ of ground truth; recall measures the corresponding fraction of ground-truth points near the prediction.Source: §B.3 · p. 16

B.4. Details on CARLA Dataset

The CARLA dataset uses simulated LiDAR to generate input and ground-truth training pairs. The sparse neural kernel field can also represent continuous RGB texture, which the paper demonstrates on ShapeNet cars.

B.4. Details on CARLA Dataset

  • The CARLA input LiDAR is mounted 1.8 m above ground, with a −15° to 15° vertical field of view and atmosphere attenuation rate of 4 × 10^-3.Source: §B.4 · p. 16

C.1. Texture Reconstruction

  • The hierarchy supports continuous texture fields in covered regions, with RGB functions fit by solving a kernel Gram-matrix linear system against input colors.Source: §C.1 · p. 17
  • NKSR reconstructs ShapeNet car textures from 10K colored input points using separate red, green, and blue kernel-field branches.The authors report accurate texture recovery alongside shape after 800 iterations per shape.Source: §C.1 · p. 17, Figure 17 · p. 17

C.2. Outlier Detection

NKSR handles outliers by pruning some through voxel support and weighting input constraints for others. The paper reports automatic weight learning with 50% outliers and provides additional reconstructions across several datasets.

C.2. Outlier Detection

  • Voxel support can prune outliers far from surfaces, while learned weights reduce the influence of near-surface outliers that would otherwise impose false constraints.An MLP predicts a weight for each input point from interpolated U-Net backbone features.Source: §C.2 · p. 17
  • With 3K-point inputs containing 50% outliers, the model automatically learns meaningful point weights without extra supervision and reconstructs good geometry.Source: Figure 18 · p. 17, §C.2 · p. 18

D. More Visualizations

Loading 2305.19590v2…