Source-linked AI summary
Controlling Neural Level Sets
Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, Yaron Lipman
TL;DR
Directly incorporating neural level sets into differentiable losses is challenging, though they represent classification margins and manifolds. The paper links sampled level-set positions to network parameters and proves exact representation of piecewise linear hypersurfaces as MLP level sets, while reporting applicability across varied learning tasks.
Problem
Standard deep-network hinge losses target output-space rather than input-space margin, although larger margins are sought for generalization and adversarial robustness.
Source: §1 · p. 2Method
The method samples neural level sets and uses a sample network, formed by adding one fixed linear layer, to relate sample positions to parameters for use in loss functions.
Source: §1 · p. 1Results
Takeaways & Limitations
The reported applications span classification margin control and curve and surface reconstruction from point clouds.
Source: §1 · p. 2The generalized Newton method is not guaranteed to reach the zero level set and can land far from its initial projection point.
Source: §2.1 · p. 3
Abstract
from arXiv · showhide
The level sets of neural networks represent fundamental properties such as decision boundaries of classifiers and are used to model non-linear manifold data such as curves and surfaces. Thus, methods for controlling the neural level sets could find many applications in machine learning. In this paper we present a simple and scalable approach to directly control level sets of a deep neural network. Our method consists of two parts: (i) sampling of the neural level sets, and (ii) relating the samples' positions to the network parameters. The latter is achieved by a sample network that is constructed by adding a single fixed linear layer to the original network. In turn, the sample network can be used to incorporate the level set samples into a loss function of interest. We have tested our method on three different learning tasks: improving generalization to unseen data, training networks robust to adversarial attacks, and curve and surface reconstruction from point clouds. For surface reconstruction, we produce high fidelity surfaces directly from raw 3D point clouds. When training small to medium networks to be robust to adversarial attacks we obtain robust accuracy comparable to state-of-the-art methods.
1 Introduction
Neural level sets encode classifier decision boundaries and can represent manifolds, motivating direct control of their positions. The paper links sampled level-set points differentiably to network parameters and applies this framework to classification and point-cloud reconstruction.
1 Introduction
- Sample networks link level-set sample positions differentiably to network parameters, allowing samples to serve as proxies for level sets in general loss functions.They are formed by adding a single linear layer to the original network.Source: §1 · p. 1
- The method controls classifier decision boundaries and reconstructs curves and surfaces from point-cloud data.For reconstruction, the paper targets raw 3D point clouds without requiring a precomputed signed-distance or occupancy function.Source: §1 · p. 2
- For medium-to-small networks, sampled-boundary robust training achieves robust accuracy comparable to state-of-the-art methods.The method measures distances between sampled decision-boundary points and training examples.Source: §1 · p. 2
- For small-data generalization, the paper formulates a geometric soft-margin SVM loss that directly increases input-space margin.This contrasts with standard deep-network hinge losses, which address output-space margin.Source: §1 · p. 2
- Any piecewise-linear hypersurface in R^d can be precisely represented as a neural level set of a suitable MLP.The result includes hypersurfaces that are not necessarily bounded.Source: §1 · p. 2
2 Sample Network
The sample-network method makes neural level sets usable in differentiable losses by sampling points, relating their positions to network parameters, and representing that relation with an efficient network.
2 Sample Network
- At each iteration, the method samples points on the level set, builds a sample network by adding a fixed linear layer, and uses it as a proxy for the set in a loss.The points are sampled from the level set before being incorporated into the loss.Source: §2 · p. 3
2.1 Level set sampling
- Generalized Newton iterations move initial points toward the zero level set using a Moore-Penrose pseudoinverse; for a linear function, the step is an orthogonal projection.The method is not guaranteed to reach the zero level set, and successful zeros can lie far from the initial point.Source: §2.1 · p. 3
2.2 Differentiable sample position
- A sample position is related differentiably to changing parameters by a first-order condition, whose underdetermined system is solved with a minimal-norm pseudoinverse solution.The resulting sample network stores the input-derivative pseudoinverse, requiring O(d) rather than O(m) space per point.Source: §2.2 · p. 4
3 Incorporation of samples in loss functions
The paper incorporates sampled level-set points into losses for geometric margin control, adversarial robustness, and manifold reconstruction, adapting the objective to each task.
3 Incorporation of samples in loss functions
- The sample network lets level-set samples enter losses that directly control the neural level set.The section gives classification and point-cloud reconstruction as applications.Source: §3 · p. 4
3.1 Geometric SVM
- The geometric SVM formulation targets input-space margin using distances to the classifier’s level sets, unlike standard deep-network hinge losses based on output-space margin.For general nonlinear networks, the level sets need not be equidistant, so the margin is defined separately for each data sample; the affine L2 case reduces to regular SVM.Source: §3.1 · p. 5
3.2 Robustness to adversarial perturbations
- For adversarial robustness, the loss pulls the decision boundary toward incorrectly classified samples and pushes it at least ε_j from correctly classified samples.The implementation uses a coordinate selected by the largest gradient component, with a distance measure bounded by the L∞ distance; the same loss applies to both correct and incorrect examples.Source: §3.2 · pp. 5–6
3.3 Manifold reconstruction
- For manifold reconstruction, the loss places selected level sets at prescribed distances from point-cloud samples, while curve reconstruction uses a vector-valued network whose zero set is an intersection of hypersurfaces.For higher-codimension curves, only the zero level set is encouraged to approach the samples because no signed distance function is expected.Source: §3.3 · p. 6
4 Universality
The paper establishes a geometric universality result for ReLU multilayer perceptrons: their level sets can exactly represent watertight piecewise-linear hypersurfaces, including unbounded ones.
5 Experiments
The experiments apply neural level-set control to classification generalization, adversarial robustness, and reconstruction of curves and surfaces from point clouds. Results favor the geometric SVM loss on small-data classification and show competitive robustness to margin-based attacks, while surface reconstruction outperforms the compared method.
5.1 Classification generalization
- The geometric SVM loss outperformed cross-entropy and hinge losses in test accuracy across MNIST, Fashion-MNIST, and CIFAR10 as training-set size varied.The experiments used binary classification formed by merging labels and evaluated on each dataset’s original test set.Source: §5.1 · pp. 6–7, Figure 2 · p. 6
5.2 Robustness to adversarial examples
- Our models were more robust than baselines to Xent-loss PGD attacks, but comparable to baselines on margin-loss attacks.The authors attribute relatively low ResNet-18 robust accuracy to not adapting the method specifically to Batch-Norm layers.Source: §5.2 · p. 7, Table 1 · p. 7
- Black-box attacks were generally less effective than relevant white-box attacks, and the three compared methods had broadly similar black-box robustness.The authors also report that their method performed better when using standard-model black-box attacks.Source: §5.2 · p. 7
5.3 Surface and curve reconstruction
- Our method outperformed the recent method of on FAUST surface reconstruction, using L1 and L2 Chamfer distances between 30K uniformly sampled points from reconstructed surfaces and dataset ground-truth registrations.Training used 10 FAUST raw scans, each with approximately 170K points; our formulation works directly on raw point clouds, unlike implicit methods using signed-distance information and/or occupancy targets.Source: §5.3 · p. 8
- For curve reconstruction, intersecting two scalar level sets recovered curves from noisy point clouds and gracefully completed partial curve data.The experiments used a two-output network and the reconstruction loss with T = {0}.Source: §5.3 · p. 8, Figure 3 · p. 8
6 Conclusions
The paper presents a simple, scalable method for incorporating neural level sets into a general family of loss functions. It reports applicability across different learning settings and identifies several directions for extending the method.
6 Conclusions
- The method incorporates neural network level sets into a general family of loss functions.The conclusion characterizes it as simple and scalable.Source: §6 · p. 8
- The authors report testing the method across a wide range of learning tasks and finding it applicable in different settings.They describe it as particularly easy to use.Source: §6 · p. 8
- Future work includes batch-normalization support, control of intermediate-layer level sets, improved level-set sample coverage, and geometric regularization such as curvature penalties.Batch normalization would require generalizing from points to batches.Source: §6 · p. 8
A Additional Experiments
The additional experiments examine neural level-set sample distribution in two dimensions and the spacing and shape of reconstruction-network level sets.
A.1 Distribution of points on the level set.
- In two dimensions, the sampling experiment compares Chamfer and Hausdorff distances for projected samples initialized by uniform points or normally perturbed level-set samples.The ground-truth dense level-set sampling is tractable in this low-dimensional setting.Source: §A.1 · p. 10, Figure A1 · p. 10
A.2 Level sets of reconstruction networks resemble signed distance function
- Cross-cuts of reconstruction-network level sets show equispaced iso-levels, as encouraged by Equation 13.Source: Figure A2 · p. 10
- The reconstruction-network level sets resemble those of a signed distance function.Source: §A.2 · p. 10
B Implementation details
The implementation details specify the computing framework and summarize the network-layer conventions used in the experiments.
B Implementation details
- Experiments were conducted using PyTorch on an Nvidia Tesla V100 GPU.Source: B Implementation details · p. 11
- The architecture table defines convolution layers by channels, kernel size, and stride, and fully connected layers by output count.Source: Table B1 · p. 11
- Each convolutional and fully connected layer is followed by ReLU, except the final fully connected layer.Source: Table B1 · p. 11, B Implementation details · p. 11
B.1 Parameters of experiments shown in Figure 1
The appendix specifies the training setup for Figure 1 and the generalization experiments, including optimizer, learning-rate, and hyperparameter choices.
B.1 Parameters of experiments shown in Figure 1
- The Figure 1 experiments use a 4-layer MLP trained for 1000 epochs with ADAM at learning rate 0.001 and geometrical SVM loss weight λ = 0.001.The 16 training points lie inside [0, 0.5]2, and projection uses at most 20 iterations.Source: §B.1 · p. 11
B.2 Classification generalization
- Generalization experiments use SGD with Nesterov momentum 0.9 and weight decay 10−4; learning rates were selected from five tested values for cross-entropy and hinge losses.The tested rates were 0.001, 0.005, 0.01, 0.02, and the chosen rates are summarized in Table B2.Source: §B.2 · p. 11, Table B2 · p. 11
B.3 Adversarial robustness
The appendix records the settings for adversarial-robustness and reconstruction experiments, including attack configurations, projection procedures, and data preparation.
Training parameters
- The MNIST and CIFAR10 robustness experiments use ConvNet-4a and ConvNet-4b, respectively, with batch size 128 and false-position projection capped at 40 iterations.Standard models use cross-entropy for 200 epochs; the appendix also includes a CIFAR10 ResNet-18 experiment.Source: Training parameters · p. 12
- Bounded attacks use εattack = 0.3 with PGD-40 and PGD-100 at step size 0.01 on MNIST, and εattack = 8/255 with PGD-20 at step size 0.003 on CIFAR10.Tables B3 and B4 describe black-box comparisons for MNIST PGD40 and CIFAR10 PGD20; their diagonals represent white-box attacks.Source: Training parameters · p. 12, Table B3 · p. 12, Table B4 · p. 12
B.4 Surface Reconstruction
- For Faust surface reconstruction, the model and AtlasNet use FC-2, with ADAM, batch size 10, and a learning rate halved at epochs 500, 1500, and 3500.The reconstruction-loss weight grows from 1 to 5 over 1000 epochs; noisy input points seed projection to S0, while the remaining points sample other level sets.Source: §B.4 · p. 12
- Curve reconstruction uses a cubic-spline ground truth through six randomly sampled points, then forms a 300-point cloud by sampling the curve and adding small Gaussian noise.Farthest Point Sampling selects input points, and the full batch initializes projection to S0; other level sets are not sampled.Source: §B.4 · pp. 12–13
C Proofs
The proofs establish a projection property for the generalized Newton step and show how ReLU networks can represent watertight piecewise-linear hypersurfaces as level sets.
Projection property
- For a full-rank affine map with fewer outputs than inputs, the generalized Newton step is the orthogonal projection onto its zero level set.The proof places the update displacement in the normal space to the level set, making it orthogonal to the tangent space.Source: C Proofs · p. 13
- The parameter derivatives of a projected point lie orthogonal to the neural level set, since the pseudoinverse selects a minimum-norm solution in the normal space.This statement assumes the input Jacobian is full rank, described as the generic case.Source: C Proofs · p. 13
Representing piecewise-linear hypersurfaces
- Any watertight, possibly unbounded piecewise-linear hypersurface can be represented exactly as the level set of a ReLU multilayer perceptron.The proof describes the surface as the boundary of a polytope and uses its face-supporting planes, whose normals point outward.Source: C Proofs · p. 13
- The construction partitions the polytope into contained intersections of closed half-spaces and uses their union to cover its interior, including points on partition boundaries.The resulting piecewise-linear function is positive inside the polytope, negative outside, and has the target surface as its level set; max and min operations are encoded with ReLU layers.Source: C Proofs · pp. 13–14
- The construction’s piecewise-linear function can be encoded as a ReLU MLP by recursively building max and min operations.The proof uses ReLU to implement binary maxima, then recursively extends this to a vector.Source: C Proofs · p. 14