Source-linked AI summary
SALD: Sign Agnostic Learning with Derivatives
Matan Atzmon, Yaron Lipman
TL;DR
Learning 3D geometry from raw point clouds, triangle soups, and unoriented meshes remains challenging. SALD extends sign agnostic learning with derivative information to learn implicit neural representations directly from raw data, achieving state-of-the-art results on ShapeNet and D-Faust while retaining a theoretically justified minimal-surface property.
Problem
Learning 3D geometry directly from raw data is challenging, especially for point clouds, triangle soups, and unoriented or inconsistently oriented meshes.
Method
SALD generalizes sign agnostic learning by adding a sign agnostic derivative term to regression on the unsigned distance function of raw input geometry.
Results
SALD achieves state-of-the-art results on ShapeNet and D-Faust, with improved reconstruction details and sharp features.
Takeaways & Limitations
SAL and SALD favor minimal-surface solutions, and SALD’s derivative term improves the approximation power of the resulting signed implicit neural network.
Takeaways & Limitations
SALD can add surface sheets in areas lacking inside/outside information and can miss thin structures.
Abstract
from arXiv · showhide
Learning 3D geometry directly from raw data, such as point clouds, triangle soups, or unoriented meshes is still a challenging task that feeds many downstream computer vision and graphics applications. In this paper, we introduce SALD: a method for learning implicit neural representations of shapes directly from raw data. We generalize sign agnostic learning (SAL) to include derivatives: given an unsigned distance function to the input raw data, we advocate a novel sign agnostic regression loss, incorporating both pointwise values and gradients of the unsigned distance function. Optimizing this loss leads to a signed implicit function solution, the zero level set of which is a high quality and valid manifold approximation to the input 3D data. The motivation behind SALD is that incorporating derivatives in a regression loss leads to a lower sample complexity, and consequently better fitting. In addition, we prove that SAL enjoys a minimal length property in 2D, favoring minimal length solutions. More importantly, we are able to show that this property still holds for SALD, i.e., with derivatives included. We demonstrate the efficacy of SALD for shape space learning on two challenging datasets: ShapeNet that contains inconsistent orientation and non-manifold meshes, and D-Faust that contains raw 3D scans (triangle soups). On both these datasets, we present state-of-the-art results.
1 INTRODUCTION
SALD learns implicit neural representations directly from raw, potentially unoriented 3D data by extending sign agnostic learning with derivatives. The method targets better detail and generalization while retaining a minimal-surface property and achieves strong results on ShapeNet and D-Faust.
- Learning from raw data avoids requiring ground-truth signed distance supervision and accommodates inconsistent normals or missing parts.
- Derivative supervision reduces sample complexity, helping determine neural functions more uniquely and improving accuracy and generalization.A one-hidden-layer example is not uniquely determined by two function samples, but becomes unique when derivative information is added.
- SAL and SALD favor minimal-area zero level sets across missing parts and holes; in 2D, both encourage straight lines connecting neighboring data points.
- SALD achieves superior results against state-of-the-art methods on ShapeNet and D-Faust, balancing scan detail against noise and ghost geometry.
- SALD extends SAL by incorporating derivatives into sign agnostic regression, improving surface approximation and detail.
- SALD trains directly on raw data, including unoriented or inconsistently oriented triangle soups and raw 3D scans.
2 PREVIOUS WORK
Prior neural 3D-shape methods use parametric charts, implicit functions, primitives, or template deformations, each offering distinct representation benefits and constraints.
- Parametric representations: Parametric methods represent surfaces as collections of neural charts and ease surface sampling and direct raw-data training.Their main challenge is producing collectively consistent, low-distortion charts that cover the shape.
- Implicit representations: Implicit methods represent surfaces as zero level sets of functions modeled by grid-based convolutional networks or continuous multilayer perceptrons.
- Primitives: Primitive-based methods compose shapes from parametric primitives, Gaussians, or differentiable binary space-partitioning trees.
- Template fitting: Template-fitting methods learn class-specific shapes by deforming a template, using matching techniques, geometric losses, or autoencoder architectures.
3 METHOD
SALD learns signed implicit representations from unsigned distances by adding sign-agnostic derivative supervision to SAL. The resulting loss improves sharp-feature preservation while retaining a minimal-surface property in the analyzed 2D setting.
- Sign agnostic learning with derivatives: SALD learns an implicit surface as the zero level set of an MLP fitted to unsigned distances from raw geometric input.The method performs sign-agnostic regression to obtain a signed function whose zero level set approximates the input data.
- Sign agnostic learning with derivatives: The SALD loss generalizes SAL by incorporating sign-agnostic comparisons between both function values and input gradients.The derivative similarity is extended from scalar values to vector-valued gradients, with λ controlling the derivative term.
- Qualitative behavior: SALD preserves sharp corners and unsigned-distance isolevels on an L-shaped curve, whereas SAL smooths them.The comparison uses level sets produced by MLPs optimized with the two losses.
- Minimal surface property: For curves with missing segments, both SAL and SALD produce approximately minimal-length connections, while SALD better preserves sharp features elsewhere.The theoretical analysis restricts the losses to curves connecting neighboring data points and examines the missing-part region.
- Minimal surface property: Under uniform distributions in the 2D two-point setting, the straight line is the strict global minimizer for both SAL and SALD losses.The proof analyzes curves connecting x1 and x2 and shows that the straight line minimizes the restricted losses.
4 EXPERIMENTS
SALD is evaluated for raw-data shape-space learning on ShapeNet and D-Faust using established auto-decoder and VAE architectures. It compares favorably with existing methods while retaining strong detail and avoiding some baseline artifacts, but has identifiable failure cases.
- Shape space learning architecture: SALD is incorporated into auto-decoder and modified VAE shape-space architectures for raw 3D data.The decoder uses an implicit representation, while the VAE encoder is PointNet.
- ShapeNet: ShapeNet evaluation uses non-manifold or non-orientable meshes, with comparisons against SAL and DeepSDF across five categories.IGR is omitted because ShapeNet lacks consistently oriented normals required by that method.
- ShapeNet: SALD compares favorably with DeepSDF on ShapeNet reconstruction performance when both use the same auto-decoder.The comparison is reported on held-out test data using quantitative and qualitative results.
- D-Faust: On D-Faust, SALD is roughly comparable to the best baseline across measures while producing details comparable to IGR without its undesired surface sheets.It also preserves the minimal-surface property associated with SAL and generates novel mixed faces and body parts through latent interpolation.
- Failure cases: SALD can add surface sheets that close open areas when inside/outside information is unavailable or ill-defined.Typical examples include lamp bottoms and chair holes.
- Failure cases: Thin structures can be missed, such as an electric cord in a lamp example.
5 CONCLUSIONS
SALD learns implicit neural representations from raw data and improves reconstruction detail through sign-agnostic derivative information. Its theoretical minimal-surface property is established in 2D, while 3D generalization and approximation improvements remain open directions.
- Contributions: SALD generalizes sign agnostic learning by adding derivative data to improve the approximation power of signed implicit neural networks.The paper reports improved reconstruction details and sharp features.
- Theoretical properties: SAL and SALD favor minimal-area zero level sets in regions with missing parts and holes, with a theoretical justification in 2D.The authors show that both losses encourage straight-line solutions connecting neighboring data points in the 2D setting.
- Limitations and future work: SALD still has room for improved implicit-representation approximation, especially because ShapeNet reconstructions retain a gap from input models.The paper identifies greater network expressivity and improved losses as possible ways to address this gap.
- Evaluation: D-Faust quantitative results compare one-sided Chamfer and normal distances among registration meshes, reconstructions, and raw input scans.The Chamfer-distance values are reported after multiplication by 10^2.
- Future work: Applications to generative 3D modeling and learning geometry with 2D supervision are proposed as future research directions.These are presented as possible settings in which SALD could improve current state-of-the-art methods.
A.1 PROOF OF LEMMA 1
The proof establishes a lower bound relating sign-agnostic differences between unit vectors to the sine of their angle. It derives the result by rewriting the minimum norm expression and treating acute and obtuse angle cases separately.
- Lemma statement: For any pair of unit vectors a and b, the minimum of their aligned and sign-reversed distances is at least |sin ∠(a,b)|.The lemma is the central statement proved in this appendix subsection.
- Algebraic reduction: The minimum squared distance is rewritten using the inner product as min {2 + 2⟨a,b⟩, 2 − 2⟨a,b⟩}.This algebraic form supports the subsequent angle-based case analysis.
- Case split: The proof considers separately α ∈ [0, π/2] and α ∈ [π/2, π], where α denotes the angle between a and b.The sign of cos α determines the relevant case.
A.2.1 DATA PREPARATION
SALD samples unsigned-distance values and derivatives from distributions built around the raw 3D data. The value-sampling distribution combines Gaussian neighborhoods and projected points, while derivative samples are uniform on the data.
- Sampling inputs: SALD evaluates its loss using unsigned-distance values and corresponding derivatives sampled from distributions D and D′.The values are sampled at points in D, and gradients at points in D′.
- Value distribution: D2 is obtained by projecting D1 to the surface S.This adds projected samples to the Gaussian-based value distribution.
- Derivative distribution: D′ is uniform on X, where unsigned-distance derivatives form the convex hull of the two possible normal vectors at each data point.Because the loss is sign agnostic, either normal choice can be used.
A.2.2 GRADIENT COMPUTATION
SALD computes the network gradient required by its derivative loss using forward-mode automatic differentiation. The gradient is represented through differentiable network layers tied to the original network parameters.
- Gradient construction: The derivative term ∇_x f(x; θ) is incorporated into the SALD loss through differentiable computation.This enables gradient information to participate in optimization.
- Gradient construction: Forward-mode automatic differentiation is used to compute the gradient of the implicit network.The approach follows the construction described by Gropp et al. (2020).
- Layerwise representation: The gradient network is constructed from layers based on the outputs yℓ of f(x; θ) and learnable parameters θ = (Wℓ, bℓ).The layerwise construction exposes how the derivative computation depends on the original network.
A.2.3 ARCHITECTURE DETAILS
The method uses VAE and auto-decoder shape-space architectures with 256-dimensional latent vectors and a shared implicit decoder design. Training and evaluation use SALD-specific sampling, Chamfer and normal metrics, and Marching Cubes meshing.
- VAE architecture: The VAE encoder combines DeepSets and PointNet layers to map an input point cloud to Gaussian parameters μ, η ∈ R^256 and a sampled latent vector z ∈ R^256.The Gaussian is N(μ, diag exp η).
- Decoder architecture: The decoder concatenates a 3D point x with z, uses eight fully connected layers with a middle skip connection, and outputs a scalar implicit value.The first layer is FC(256 + 3, 512), middle layers are FC(512, 512), and the final layer is Linear(512, 1).
- Auto-decoder architecture: The auto-decoder also uses a 256-dimensional latent vector and the same decoder architecture as the VAE, with latent regularization added to the loss.This architecture is described as similar to Park et al. (2019).
- Training: Training uses ADAM with batch size 64, evaluates SALD on 922 randomly selected points from 500K precomputed samples, and runs for 3000 epochs.The VAE uses a fixed learning rate of 0.0005, while the auto-decoder halves its learning rate every 500 epochs.
- Evaluation: Evaluation measures shape similarity with Chamfer distances and triangle-soup normal similarity, using Marching Cubes on a [512]^3 grid and 30K sampled surface points.The metrics are computed from the meshed learned implicit representation.