Source-linked AI summary
MetaSDF: Meta-learning Signed Distance Functions
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, Gordon Wetzstein
TL;DR
Existing neural implicit shape methods infer shape-specific representations with encoders or optimize latent codes in auto-decoders. MetaSDF formulates shape-space learning as gradient-based meta-learning, matching auto-decoder performance while achieving an order-of-magnitude faster inference and outperforming pooling-based set encoders.
Problem
Generalizing across shapes requires learning function-space priors and inferring shape-specific parameters from partial observations, while existing approaches rely on encoded or optimized low-dimensional latent codes.
Method
MetaSDF applies gradient-based meta-learning algorithms to learn priors over signed distance functions represented by fully connected neural networks.
Results
MetaSDF performs on par with auto-decoder approaches, is an order of magnitude faster at inference, and outperforms pooling-based set-encoder methods.
Takeaways & Limitations
The approach advances understanding of generalization strategies for neural implicit shape representations and makes weaker assumptions about latent-space dimensionality.
Takeaways & Limitations
Out-of-distribution experiments use models trained and tested with dense signed-distance-function samples, a setting in which baseline approaches perform much better.
Abstract
from arXiv · showhide
Neural implicit shape representations are an emerging paradigm that offers many potential benefits over conventional discrete representations, including memory efficiency at a high spatial resolution. Generalizing across shapes with such neural implicit representations amounts to learning priors over the respective function space and enables geometry reconstruction from partial or noisy observations. Existing generalization methods rely on conditioning a neural network on a low-dimensional latent code that is either regressed by an encoder or jointly optimized in the auto-decoder framework. Here, we formalize learning of a shape space as a meta-learning problem and leverage gradient-based meta-learning algorithms to solve this task. We demonstrate that this approach performs on par with auto-decoder based approaches while being an order of magnitude faster at test-time inference. We further demonstrate that the proposed gradient-based method outperforms encoder-decoder based methods that leverage pooling-based set encoders.
1 Introduction
Neural implicit representations learn priors over function spaces to generalize across shapes from partial observations. Existing approaches trade fast encoder inference for auto-decoder flexibility but incur costly test-time optimization, motivating MetaSDF’s gradient-based alternative.
- Neural implicit shape representations model each shape as a function, such as a signed distance function, enabling learned priors over neural implicit shape spaces.
- Existing methods represent shapes with low-dimensional latent codes that condition a shared neural function through concatenation or hypernetworks.
- Encoder-based methods infer latent codes quickly but require regular-grid observations, whereas auto-decoders support flexible observations through per-shape latent optimization.
- Several seconds of test-time optimization may be required to infer an auto-decoder embedding even for a single ShapeNet object.
- MetaSDF frames shape-space learning as meta-learning and is reported to match auto-decoders while being an order of magnitude faster at inference.
2 Related Work
Related work learns priors for neural implicit geometry through latent shape spaces, using encoders, auto-decoders, and set-based neural processes. Meta-learning instead emphasizes rapid adaptation to unseen tasks from limited context observations.
- Learning shape spaces: Prior work learns priors over neural implicit geometry by assuming a low-dimensional latent shape space decoded into functions.
- Learning shape spaces: These approaches use auto-decoders or convolutional encoders, with latent embeddings decoded through concatenation-based conditioning or hypernetworks.
- Meta-Learning: Meta-learning trains learners to adapt quickly to unseen tasks from few context observations, often by learning an optimizer or update rule.
- Meta-Learning: Conditional and attentive neural processes encode context observations with permutation-invariant set encoders and condition a decoder on the resulting embedding.
3 Meta-learning Signed Distance Functions
The paper represents shapes as signed distance functions and casts shape generalization as task-specific neural-network specialization. MetaSDF adapts a shared initialization with gradient steps on context observations, avoiding the conventional low-dimensional latent-code bottleneck.
- Shape representation: Each shape is represented by K spatial point samples paired with signed distances from its ground-truth SDF.
- Shape representation: A neural network Φ approximates each shape’s SDF, whose surface is represented implicitly by the zero-level set L0(Φ).
- Problem formulation: Shape generalization is formulated as learning a prior over functions, including parameterizing Φi and inferring its parameters from partial observations Xi.
- Existing parameterizations: Existing methods condition shared networks on latent embeddings through concatenation or hypernetworks, then infer embeddings with encoders or optimize them as auto-decoder variables.
- Shape generalization as meta-learning: MetaSDF treats each shape’s SDF as a task and quickly specializes a shared meta-network using limited coordinate–distance context observations.
- Shape generalization as meta-learning: Unlike auto-decoders, MetaSDF performs specialization through learned adaptation rather than stochastic gradient descent until convergence at test time.
- Shape generalization as meta-learning: The method performs k gradient-descent updates from meta-parameters θ, evaluates specialized parameters on target observations, and backpropagates the test loss through those updates.
- Shape generalization as meta-learning: MetaSDF supports different context observation types, outperforms pooling-based set encoders, and optimizes directly in the high-dimensional parameter space θ.
4 Analysis
The analysis evaluates MetaSDF on 2D MNIST signed distance functions and 3D ShapeNet shapes under dense, surface-only, and out-of-distribution conditions. Across these settings, MetaSDF combines fast test-time adaptation with competitive or stronger reconstruction performance.
- 2D MNIST SDFs: MetaSDF is evaluated against concatenation and hypernetwork auto-decoders and a conditional neural process on MNIST SDF reconstruction.The experiments use fully connected ReLU-MLPs; MetaSDF performs five inner-loop update steps.
- Inference speed: 4 seconds versus 50 ms: auto-decoder methods require about four seconds to converge, while MetaSDF infers an unseen MNIST SDF in five gradient steps.The comparison covers hypernetwork and concatenation-based approaches.
- Out-of-distribution generalization: MetaSDF far outperforms alternative approaches on out-of-distribution MNIST samples spanning unseen digit classes, rotations, and triple-digit compositions.It qualitatively reconstructs unseen and rotated digits, but no method accurately reconstructs the zero-level set in the compositionality experiment.
- 3D ShapeNet: For dense ShapeNet SDF observations, MetaSDF performs approximately on par with DeepSDF and is slightly better on out-of-distribution samples while speeding inference by one order of magnitude.The comparison is shown for ShapeNet planes, with additional MetaSDF reconstructions for tables.
- Surface-only inference: With only zero-level-set points, the unchanged MetaSDF model reconstructs 3D geometry and outperforms a matched PointNet-based set encoder, especially on out-of-distribution shapes.Auto-decoder models cannot perform zero-level-set reconstruction without additional heuristics.
5 Discussion
MetaSDF connects neural implicit shape-space learning with gradient-based meta-learning. It matches auto-decoder performance, accelerates inference, outperforms pooling-based set encoders, and retains computational and methodological limitations.
- MetaSDF learns priors over fully connected neural-network SDFs using a meta-learning approach.
- It performs on par with auto-decoder approaches while being an order of magnitude faster at inference and outperforming pooling-based set encoders.
- The approach makes weaker assumptions about latent-space dimensionality than existing approaches.
- The current implementation requires memory-intensive second-order gradients through unrolled inner-loop updates.Implicit-gradient methods are identified as a possible way to make memory complexity independent of inner-loop steps.
- Future work may extend the approach to scene representations combining 3D geometry and appearance through neural rendering.
- The work aims to inspire more powerful priors for implicit neural shape representations by connecting this problem to meta-learning.
Broader Impact
Gradient-based meta-learning is presented as a promising approach for efficiently learning distributions over functions in neural implicit representations.
- Generalizing neural implicit representations requires efficient methods for learning distributions over functions.
- Gradient-based meta-learning is identified as one promising avenue for this problem.
- The approach may contribute to the backbone of emerging neural signal representation strategies.
MetaSDF
The supplied passages contain only a project webpage URL and an authorship footnote, without substantive information about MetaSDF.
- The MetaSDF project webpage is listed as vsitzmann.github.io/metasdf/.
- The passage records that the marked authors contributed equally to the work.
1 Conditioning via concatenation as a special case of Hypernetworks
The paper shows that concatenation-based conditioning is a special case of a hypernetwork: its latent-code-dependent term acts as an additive conditional bias.
- A hypernetwork maps a code vector z to the weights and biases of a layer in a hyponetwork.
- Concatenation conditions a shared feedforward network by appending the latent shape embedding to the target coordinates.
- Splitting the concatenated layer weights separates input-dependent weights Whypo from code-dependent weights Whyper.
- The term Whyper · z + b depends only on the latent code and is additive with the input-dependent linear term Whypo · x.
- Therefore, concatenation is equivalent to a single affine hypernetwork that predicts only the hyponetwork biases.
2 2D Experiments
The 2D experiments compare MetaSDF with conditional neural processes across in-distribution and out-of-distribution signed-distance-function reconstruction settings. Increasing the conditional neural process encoder depth improves performance only slightly and remains substantially behind MetaSDF.
- 2.1 Additional Results: A 9-layer ReLU set encoder improves conditional neural process performance only slightly over a 4-layer ReLU set encoder.The comparison more than doubles the encoder depth while yielding only a small performance increase.
- 2.1 Additional Results: The deeper conditional neural process still lags far behind MetaSDF in the reported 2D experiments.The passage characterizes the gap as substantial after comparing both set-encoder depths with MetaSDF.
- 2D Experiments: The out-of-distribution experiments use dense SDF training samples and dense test-time conditioning to provide a fair comparison with baseline approaches.The passage notes that baseline methods perform much better in the dense setting.
- 2D Experiments: Conditional neural processes train in about 3 hours, whereas MetaSDF trains in about 5 hours.For rotated-MNIST and compositionality experiments, dataset size required 24 hours of training per model and another 24 hours of test-time reconstruction.
3 3D Experiments
The 3D experiments qualitatively compare MetaSDF with DeepSDF and a PointNet encoder on ShapeNet tables, while also examining the geometry represented before specialization. MetaSDF’s meta-initialization differs markedly from DeepSDF’s all-zero latent-code shape and specializes in five gradient-descent steps.
- 3.1.1 “Average airplane” of concatenation-based approach vs. Meta-network initialization: The MetaSDF meta-initialization’s zero-level set bears little resemblance to DeepSDF’s zero-level set for the all-zero mean latent code.The comparison is shown in Figure 1 before specialization.
- 3.1.1 “Average airplane” of concatenation-based approach vs. Meta-network initialization: MetaSDF reaches swift specialization with only five gradient-descent steps.The passage presents this behavior as a property that future work may investigate in relation to the learned initialization.
- 3D Experiments: Figure 2 qualitatively compares MetaSDF with DeepSDF on the ShapeNet V2 tables class.The comparison concerns reconstructed geometry for the tables category.
- 3D Experiments: Figure 3 qualitatively compares MetaSDF with a PointNet-encoder approach on ShapeNet V2 table reconstructions.The comparison uses zero-level-set reconstructions.
3.2 Reproducibility.
The 3D experiments use ShapeNetV2 planes and tables with DeepSDF preprocessing and compare reconstruction methods under specified training and evaluation procedures. MetaSDF has O(nm) complexity in query points and inner-loop steps, but takes longer to train than the other reported 3D approaches.
- 3.2 Reproducibility.: The 3D experiments use the Planes and Tables categories of ShapeNetV2 with DeepSDF’s train-test splits and preprocessing pipeline.Twenty training examples from each object category are held out for validation.
- 3.2 Reproducibility.: MetaSDF uses a composite loss combining L1 SDF-value loss with binary crossentropy on the predicted SDF sign.The sign term is motivated by the importance of accurate near-surface sign prediction for zero-level-set mesh accuracy.
- 3.2 Reproducibility.: Dense MetaSDF uses five inner-loop update steps with per-parameter, per-step learning rates initialized to 5 × 10^-3.The dense model is an eight-layer, 512-unit fully connected ReLU MLP without normalization layers.
- 3.2 Reproducibility.: Zero-level-set MetaSDF uses a seven-layer, 512-unit network to better match PointNet encoder parameter counts.DeepSDF uses an eight-layer, 512-unit auto-decoder with 256-dimensional latent codes, while the PointNet encoder also produces a 256-dimensional latent code.
- 3.2 Reproducibility.: The proposed MetaSDF approach has O(nm) complexity, where n is the number of query points and m is the number of inner gradient-descent steps.DeepSDF and PointNet encoders train for 2000 epochs in approximately 4 days, compared with approximately 6 days for MetaSDF.
- 3.2 Reproducibility.: Random context-point sampling makes reconstructed meshes vary between runs, so each model reconstructs every mesh twice and retains the better result for Chamfer-distance evaluation.This evaluation procedure follows DeepSDF and is conducted once for each model.