PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Fields as Learnable Kernels for 3D Reconstruction

Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, Or Litany

arXiv:2111.13674v1cs.CVcs.GRcs.LG

TL;DR

Sparse oriented points make it challenging to reconstruct partial shapes while generalizing beyond training categories and across sampling densities. Neural Kernel Fields learn an input-conditioned kernel and fit it with a closed-form linear solve, reporting state-of-the-art reconstruction and generalization. Their reconstructions are designed to respect observed points while retaining data-driven priors.

  • Problem

    3D reconstruction must recover shape geometry from sparse, partial measurements while generalizing beyond the training distribution and respecting the input points.

    Source: §1 · pp. 1–2

  • Method

    Neural Kernel Fields learn a positive-definite kernel conditioned on an input point cloud, then use kernel ridge regression to fit the points through a linear solve.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    NKF achieves state-of-the-art ShapeNet reconstruction and reports state-of-the-art generalization to out-of-distribution shapes, scenes, and point densities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The learned-kernel fit combines data-driven priors with reconstruction that respects observed points and remains robust to changes in input point density.

    Source: §1 · p. 2

    The dense linear solve limits evaluation to around 12k points on a V100 GPU.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present Neural Kernel Fields: a novel method for reconstructing implicit 3D shapes based on a learned kernel ridge regression. Our technique achieves state-of-the-art results when reconstructing 3D objects and large scenes from sparse oriented points, and can reconstruct shape categories outside the training set with almost no drop in accuracy. The core insight of our approach is that kernel methods are extremely effective for reconstructing shapes when the chosen kernel has an appropriate inductive bias. We thus factor the problem of shape reconstruction into two parts: (1) a backbone neural network which learns kernel parameters from data, and (2) a kernel ridge regression that fits the input points on-the-fly by solving a simple positive definite linear system using the learned kernel. As a result of this factorization, our reconstruction gains the benefits of data-driven methods under sparse point density while maintaining interpolatory behavior, which converges to the ground truth shape as input sampling density increases. Our experiments demonstrate a strong generalization capability to objects outside the train-set category and scanned scenes. Source code and pretrained models are available at https://nv-tlabs.github.io/nkf.

1. Introduction

Sparse-point 3D reconstruction must balance learned priors with fidelity to the observed points, while handling partial shapes and out-of-distribution inputs. Neural Kernel Fields learn input-conditioned kernels and fit them through a closed-form solve, with reported state-of-the-art reconstruction and generalization results.

  • Reconstruction methods trade off speed, point fidelity, learned priors, and spatial scale, leaving a need for methods that generalize while respecting observations.Feed-forward methods may act more like retrieval, test-time optimization can be slow or reach poor minima, fixed priors struggle with partial shapes, and local methods depend on patch size and lack global context.Source: §1 · p. 2
  • NKF learns a positive-definite kernel conditioned on the input point cloud and predicts an implicit shape by solving a linear system.Its kernel weights are obtained through a closed-form convex least-squares problem, and its global aggregation of local features avoids tuning scale parameters.Source: §1 · p. 2
  • Figure 2 illustrates that fixed-prior methods cannot complete the partial shape, feed-forward methods miss canoe slats, and LIG is sensitive to patch size and local minima.Source: Figure 2 · p. 2, §1 · p. 2
  • NKF is introduced as a neural-field representation that outputs detailed surfaces while respecting the input points.Source: §1 · p. 2
  • NKF achieves state-of-the-art ShapeNet reconstruction and reports state-of-the-art generalization to out-of-distribution shapes, scenes, and point densities.The method is also reported to outperform baselines in reconstruction detail on in- and out-of-distribution shapes.Source: §1 · p. 2

2. Related Work

Related work spans learned and fixed kernels, neural fields, and reconstruction methods using varied output representations. NKF combines learned priors with globally aggregated local features and a convex test-time fit.

  • NKF combines data-learned priors and local features aggregated globally, with a convex test-time optimization.Source: §2 · p. 2
  • Prior work includes learned kernels, Neural Splines, convolutional occupancy networks, LIG, MetaSDF, and Shape as Points, which uses Poisson reconstruction as its inductive bias.Source: §2 · p. 3
  • Shape reconstruction also uses point clouds, meshes, manifold atlases, and voxel grids, while neural fields support several other 3D tasks.Source: §2 · p. 3

3. Method

The method builds on kernel reconstruction by learning an input-conditioned kernel, then fitting an implicit shape through a linear system. It trains the kernel to support shape-aware predictions and can weight input points to reduce noisy-point contributions.

3.1. Review of Neural Splines

  • Neural Splines formulates reconstruction as fitting an implicit field to augmented point labels with kernel basis functions centered at the augmented points.Its kernel is the closed-form kernel of an infinitely wide shallow ReLU network, and the coefficients are recovered from a linear system.Source: §3.1 · p. 3

3.2. Inductive Bias of Neural Splines

  • The learned kernel addresses the smoothness prior’s weakness on sparse or partial inputs while preserving the input-point constraints.Neural Splines favors smooth functions, but that prior can be too weak for sparse or partial shapes; NKFs condition the kernel on the input.Source: §3.2 · p. 4

3.3. Neural Kernel Fields

  • NKFs learn a data-dependent kernel from input-conditioned features, then solve a positive definite linear system to predict an implicit function.The feature network maps points to features used with the Neural Spline kernel; the resulting coefficients encode the function as basis functions centered at input points.Source: §3.3 · pp. 4–5
  • Training evaluates the predicted function on dense volume and surface samples, using occupancy and surface-agreement losses to update the feature network.Dense volume and surface samples provide training supervision; they are not part of the stated sparse input at inference.Source: §3.3 · p. 5
  • Optional per-point weights let weighted ridge regression filter noisy-point contributions, yielding a more accurate reconstruction in the illustrated noisy-input example.The comparison holds the noisy input points and regularization value fixed.Source: §3.3 · p. 5, Figure 5 · p. 5

4. Experiments

The experiments evaluate NKF on ShapeNet reconstruction and completion, category generalization, ScanNet scenes, varying input densities, and design ablations. Across these settings, the reported results show strong reconstruction performance and generalization, including a 1.1% IoU drop on unseen categories and no degradation across train/test point densities.

4. Experiments

  • The experiments compare NKF with task-specific baselines across object reconstruction, completion, out-of-category shapes, scenes, and sampling densities.The evaluation uses IoU, L2 Chamfer distance, and Normal Correlation for quantitative comparisons.Source: §4 · pp. 5–6

4.1. Single Object Reconstruction on ShapeNet

  • NKF outperforms strong baselines across IoU, Chamfer distance, and Normal Consistency on 13 ShapeNet categories, reaching near 95% IoU on noise-free reconstruction.Figure 6 highlights recovered details including a car’s side-view mirror, a lamp cord, and chair-leg bulges.Source: Table 1 · p. 6, §4.1 · p. 6

4.2. Shape Completion on ShapeNet

  • For completion from partial point clouds, NKF matches C-OccNet on Chamfer distance and Normal correlation while achieving substantially better IoU.The setup uses up to 50% of a shape’s surface as input; the reported example recovers unobserved truck wheels using shape symmetry.Source: §4.2 · p. 6, Table 2 · p. 7

4.3. Out of Category Generalization

  • On the seven held-out ShapeNet categories, NKF outperforms learned and non-learned baselines with a 1.1% IoU drop relative to training on all categories.Models train on six categories and evaluate on the remaining seven; LIG’s all-category pretrained model is noted as an upper bound.Source: §4.3 · p. 7, Table 3 · p. 7

4.4. Scene Reconstruction on ScanNet

  • On ScanNet scenes with 10,000 input points, NKF achieves about half the average Chamfer distance of the next-best method.The qualitative comparison shows better capture of small details such as a stepladder and shelf.Source: §4.4 · p. 7, Table 4 · p. 7

4.5. Point Density Generalization

  • NKF works well with sparse inputs, improves as input density increases, and does not degrade when training and test densities differ.The evaluation varies input samples from 250 to 3,000; the NKF curves for matched and mismatched densities are nearly identical.Source: §4.5 · pp. 7–8, Figure 9 · p. 8

4.6. Ablations

  • The ablation study varies per-point feature dimensions and the surface L1 loss, with the table reporting mean test-set IoU.The table caption states that NKF benefits from the L1 surface loss and works well with small feature dimensions.Source: Table 5 · p. 8, §4.6 · p. 8

5. Conclusion and Limitations

The conclusion reports strong object, completion, and scene reconstruction results alongside generalization across categories and sampling densities. It also identifies dense linear solves and the requirement for oriented points as limitations; the appendix defines the Neural Spline kernel used in the implementation.

5. Conclusion and Limitations

  • The method outperforms state-of-the-art approaches on object reconstruction and completion, as well as scene reconstruction, while generalizing across categories and sampling densities.The current dense linear solve limits evaluation to around 12,000 points on a V100 GPU, and the method requires oriented points, which can be noisy.Source: §5 · p. 8

A. Neural Spline Kernel Equation

  • The implemented Neural Spline kernel is the limiting kernel of an infinitely wide ReLU network under Gaussian or Uniform initialization.The implementation uses the Gaussian-initialized version; its angle computation uses Kahan’s formula, which is numerically stable especially for small angles.Source: A. Neural Spline Kernel Equation · p. 12

B. The Effect of Noise Filtering in 3D

Figure 10 compares reconstructions without learned weights, with learned weights, and the ground-truth surface. The weighted model is smoother and does not interpolate input noise.

B. The Effect of Noise Filtering in 3D

C. More Extreme Generalization

The experiments compare a model trained only on chairs with one trained on all ShapeNet categories for reconstruction of the other 12 categories. The chair-only model’s performance drops only slightly relative to the all-category model.

C. More Extreme Generalization

  • A model trained only on chairs reconstructs the other 12 ShapeNet categories with only a slight performance drop relative to a model trained on all categories.The setup uses 1,000 input points and otherwise matches the out-of-category generalization experiment.Source: C. More Extreme Generalization · p. 12
  • The chair-only model is evaluated on airplane, bench, cabinet, car, display, lamp, loudspeaker, rifle, sofa, table, telephone, and watercraft.Source: C. More Extreme Generalization · p. 12
  • Table 6 and Figure 11 compare the chair-only model against the all-category model for those 12 categories.Source: C. More Extreme Generalization · p. 12

D. Inference Timings

The section reports inference timings for convex test-time optimization on ShapeNet and ScanNet, alongside the evaluation settings and a comparison of noise filtering with regularization.

  • Inference uses convex test-time optimization, with timings reported for the ShapeNet and ScanNet reconstruction experiments.Source: D. Inference Timings · p. 12
  • With noisy input points, additional noise filtering smooths bumps that remain in the reconstruction with 0.1 kernel-ridge regularization alone.Source: Figure 10 · p. 13
  • Table 7 reports timings for ShapeNet with 1k input points and 2.1 million evaluation points, and ScanNet with 10k input points and 16.9 million evaluation points.Source: Table 7 · p. 13
  • The timing experiments used grids of 128^3 points for ShapeNet and 256^3 points for ScanNet, with 1000 and 8000 input points respectively.Source: D. Inference Timings · p. 13
  • Kernel evaluation was implemented as a single monolithic CUDA kernel and timed on a Quadro GV100 GPU.Source: D. Inference Timings · p. 13

E. Additional ShapeNet Reconstruction Figures

Figure 12 provides additional ShapeNet reconstruction comparisons under the noise setting used in Section 4.1.

F. Additional ShapeNet Generalization Figures

The additional figures cover out-of-category ShapeNet reconstruction and ShapeNet completion, with the reconstruction figure describing generalization from chairs to other categories.

F. Additional ShapeNet Generalization Figures

  • Figure 13 provides additional reconstructions for the out-of-category experiment described in Section 4.3.Source: F. Additional ShapeNet Generalization… · p. 13
  • A model trained only on chairs generalizes to 12 other ShapeNet categories with slightly worse performance than a model trained on all categories.Source: Figure 11 · p. 14

G. Additional Completion Figures

H. Per-Category ShapeNet Results

Tables 8 and 9 provide per-category ShapeNet results for reconstruction and completion, respectively, with accompanying figures showing reconstruction and completion examples.

  • Tables 8 and 9 report per-category results for ShapeNet reconstruction and completion, respectively.Source: H. Per-Category ShapeNet Results · p. 14
  • The reconstruction figure shows ShapeNet test-set reconstructions from 1000 input points and normals with Gaussian noise.Source: Figure 12 · p. 15
  • Figure 13 shows additional ShapeNet completion examples.Source: Figure 13 · p. 16
  • Table 8 reports per-category ShapeNet reconstruction results for the Section 4.1 experiment.Source: Table 8 · p. 17
  • Table 9 reports per-category completion results for the Section 4.2 experiment.Source: Table 9 · p. 18
Loading 2111.13674v1…