PaperlayerView exact revision on arXiv

Source-linked AI summary

SampleNet: Differentiable Point Cloud Sampling

Itai Lang, Asaf Manor, Shai Avidan

arXiv:1912.03663v2cs.CV

TL;DR

Large point clouds make downstream processing costly, while FPS ignores task objectives and prior task-specific sampling uses nondifferentiable matching. SampleNet introduces differentiable soft projection, representing sampled points as weighted averages of input neighbors during training. Across classification, reconstruction, and registration, it consistently outperforms learned and non-learned alternatives, although reconstruction includes an airplane-class failure case.

  • Problem

    Point-cloud tasks need smaller inputs for efficiency, but FPS is task agnostic and prior task-specific sampling cannot backpropagate through nearest-neighbor matching.

  • Method

    SampleNet uses differentiable soft projection, expressing each output point as a weighted average of nearby input points whose weights approximate nearest-neighbor selection.

  • Results

    SampleNet consistently outperforms competing learned and non-learned sampling alternatives across classification, reconstruction, and registration.

  • Takeaways & Limitations

    The method supports task-optimized sampling for multiple point-cloud applications, including registration where samples must remain consistent across two inputs.

  • Takeaways & Limitations

    For airplane reconstruction with 64 points, FPS achieves NRE 1.31 versus 1.39 for SampleNet and 1.41 for S-NET.

Abstract

from arXiv · show

There is a growing number of tasks that work directly on point clouds. As the size of the point cloud grows, so do the computational demands of these tasks. A possible solution is to sample the point cloud first. Classic sampling approaches, such as farthest point sampling (FPS), do not consider the downstream task. A recent work showed that learning a task-specific sampling can improve results significantly. However, the proposed technique did not deal with the non-differentiability of the sampling operation and offered a workaround instead. We introduce a novel differentiable relaxation for point cloud sampling that approximates sampled points as a mixture of points in the primary input cloud. Our approximation scheme leads to consistently good results on classification and geometry reconstruction applications. We also show that the proposed sampling method can be used as a front to a point cloud registration network. This is a challenging task since sampling must be consistent across two different point clouds for a shared downstream task. In all cases, our approach outperforms existing non-learned and learned sampling alternatives. Our code is publicly available at https://github.com/itailang/SampleNet.

1. Introduction

Point-cloud sampling reduces computational and communication costs, but task-agnostic methods overlook downstream objectives and prior learned methods mishandle nondifferentiable matching. SampleNet introduces differentiable soft projection and applies it across several tasks.

  • Motivation: Large point clouds increase computational demands, motivating sampling before classification, registration, or reconstruction.Sampling preserves the point-cloud representation and can improve computational efficiency and reduce communication costs.
  • Existing approaches: FPS maximizes geometric coverage but is task agnostic and ignores the subsequent processing of sampled points.A prior learned method instead simplified point clouds and matched simplified points to input nearest neighbors afterward.
  • Existing approaches: Prior task-specific sampling improved application performance over FPS and random sampling, but nondifferentiable matching prevented gradient propagation and created a train–inference gap.Matching was not included during training, compromising performance with sampled points relative to the simplified set.
  • SampleNet: SampleNet replaces nearest-neighbor matching during training with soft projection, representing each simplified point as a weighted average of nearby input points.The weights are optimized during training to approximate nearest-neighbor selection used at inference.
  • SampleNet: Soft projection changes optimization from ambient-space regression to localized classification over input-neighborhood weights, with temperature annealing during training.Projected points use weight coordinates relative to local neighborhoods rather than absolute free-space coordinates.
  • Applications: SampleNet supports classification, registration, and reconstruction, with experiments reporting consistent outperformance of prior learned and non-learned alternatives.Registration additionally requires consistent samples across two point clouds for a shared downstream task.

2. Related Work

Point-cloud learning moved from regular but costly representations toward direct processing of irregular point sets, while sampling methods increasingly target downstream tasks. Related work highlights nondifferentiable nearest-neighbor selection and train–inference mismatch as central issues.

  • Deep learning on point clouds: Regular 2D-view and voxel representations enabled neural processing of 3D data but incur high computational load and quantization errors.Point clouds are irregular and sparse, motivating direct point-set processing.
  • Deep learning on point clouds: PointNet demonstrated direct raw-point-cloud processing using per-point MLPs, global pooling, and fully connected classification layers.The architecture maps point coordinates to features, aggregates them, and predicts the object class.
  • Deep learning on point clouds: Point-cloud learning now spans classification, segmentation, detection, reconstruction, completion, registration, generation, and adversarial attacks.The cited applications illustrate the breadth of downstream tasks that can consume point clouds.
  • Nearest-neighbor selection: Nearest-neighbor selection is widely used for information fusion, but its discrete rule is nondifferentiable in neural networks.A stochastic relaxation represents candidate neighbors with a categorical distribution whose limit is 1-NN.
  • Nearest-neighbor selection: Deterministic KNN relaxation uses temperature-scaled neighbor weights for feature-space averaging, which SampleNet adapts to approximate point-cloud sampling.Unlike the prior formulation, SampleNet promotes a small temperature during training to approximate nearest-neighbor selection.
  • Sampling methods: FPS is widely used for point-cloud pooling but ignores later processing and may produce suboptimal task performance.Alternative subsampling methods include critical-point selection and task-oriented simplification.
  • Sampling methods: SampleNet addresses the train–inference performance gap in task-oriented simplification by approximating sampling differentiably during training.Its approximation is based on relaxed nearest-neighbor selection.

3. Method

SampleNet trains a task-specific sampler by simplifying an input point cloud, softly projecting the simplified points onto local input neighborhoods, and feeding the projected set to a fixed task network. At inference, projection weights select actual input points, with progressive sampling and duplicate completion supported.

  • SampleNet pipeline: A frozen task network evaluates projected points R while SampleNet transforms complete input P into a smaller set Q and then R.The training pipeline uses three loss terms for task preservation, simplification, and projection.
  • Soft projection: Each simplified point q is softly projected onto its k nearest neighbors in P using distance- and temperature-dependent weights to produce r.The projected point is a weighted average of original input points.
  • Soft projection: As t →0, the neighbor-weight distribution converges to a Kronecker delta at the nearest neighbor, approximating discrete sampling.A projection loss promotes small temperature values during training.
  • Task optimization: Feeding R rather than Q trains SampleNet to sample the input cloud rather than merely simplify it.Each projected point estimates selection of a point from P for the downstream task.
  • Progressive sampling: The method extends to progressive sampling, where nested control-size subsets use corresponding simplification and task losses.The progressive formulation orders simplified points for variable sample sizes.
  • Inference: At inference, each output point selects the input point with the highest projection weight; duplicates are removed and FPS completes the set to m points.The resulting sampled point cloud R* is evaluated by the task network.

4. Results

SampleNet improves task-specific point-cloud sampling across classification, registration, and reconstruction by using differentiable soft projection and task-aware optimization. Its results remain strong at aggressive sampling ratios, while performance depends on the temperature schedule and varies by application and shape class.

  • Classification: SampleNet directly trains for sampled points and outperforms FPS and S-NET by a large margin in classification.FPS is task agnostic, while S-NET trains on simplified points but uses sampled points at inference.
  • Classification: 80.1% accuracy at sampling ratio 32 is a 20% improvement over S-NET and only 9% below complete-input accuracy.Ratio 32 corresponds to approximately 3% of the original points.
  • Classification: Softly projected points closely match sampled-point accuracy, while simplified-point accuracy changes relative to sampled points across sampling ratios.For ratios up to 16, simplified points perform considerably worse; at higher ratios, they perform better.
  • Classification: During training, projection weights become more concentrated on nearby neighbors, with the first and last neighbor converging to 0.43 and 0.03.The distribution does not become a delta function, because the task objective does not require exact nearest-neighbor selection.
  • Classification: A decaying temperature profile is required for SampleNet-Progressive, whereas a constant temperature causes substantial degradation, worse than FPS.Linear rectified, exponential, and learned profiles achieve similar performance, with a slight advantage for the exponential profile.
  • Efficiency: At ratio 32 before PointNet, SampleNet saves about 90% inference time, uses 6% additional memory, and causes less than a 10% accuracy drop.This illustrates the method’s time, space, and performance trade-off.
  • Registration: In registration, SampleNet maintains low MRE as sampling becomes more aggressive, achieving 5.94° versus 13.46° for FPS at ratio 32.SampleNet learns semantically similar points across source and template clouds, enabling registration with sampled sets.
  • Reconstruction: For reconstruction, SampleNet reduces NRE relative to S-NET from 1.57 to 1.33 at ratio 32, a 24% reduction.Its advantage increases at higher sampling ratios, and it reconstructs shapes with fewer artifacts or lost details than FPS or S-NET.

5. Conclusions

SampleNet addresses non-differentiable point-cloud sampling with a differentiable soft projection that represents outputs as weighted averages of input points. It generally improves downstream tasks, while a reconstruction failure case favors FPS.

  • SampleNet uses soft projection to represent output points as weighted averages of input points during training.At inference, projection weights approximate nearest-neighbor sampling, replacing ambient-space regression with localized classification problems.
  • SampleNet better preserves input shape and achieves reconstruction similar to using the complete input point cloud.The comparison uses complete inputs and 64-point samples from SampleNet, S-NET, and FPS.
  • In one reconstruction failure example, uniform FPS sampling is preferred over SampleNet.
  • SampleNet consistently outperforms competing learned and non-learned sampling alternatives across classification, reconstruction, and registration.Registration additionally requires consistent sampling across two point clouds.

Supplementary

The supplementary material provides additional results, ablations, mathematical details of soft projection, and experimental settings.

  • The supplementary sections cover additional results, an ablation study, soft projection mathematics, and experimental settings.The settings include network architecture and hyperparameters.

A. Additional results

Additional experiments show that SampleNet transfers to retrieval, improves progressive sampling, and substantially reduces computation with modest memory and accuracy costs.

  • A.1. Point cloud retrieval: SampleNet improves retrieval precision over S-NET across the recall range and approaches complete-input performance using 32 points.The retrieval evaluation uses PointNet shape descriptors on ModelNet40.
  • A.1. Point cloud retrieval: With 32 points, SampleNet is within 4% AUC of complete-input retrieval performance.
  • A.2. Progressive sampling: SampleNet-Progressive outperforms FPS and ProgressiveNet for classification, though gains are smaller than when training separately per sample size.The progressive model must be optimal for all control sizes concurrently.
  • A.2. Progressive sampling: As the sampling ratio increases, SampleNet’s reconstruction improvement over alternatives becomes more dominant.The normalized reconstruction error is compared with FPS and ProgressiveNet on nested sampled sets.
  • A.3. Computation load and memory space: Reducing sampled points lowers memory increase and raises computation reduction, with a mild decrease in classification accuracy.
  • A.3. Computation load and memory space: SampleNet with 32 points adds about 6% memory while reducing computational load by almost 90%.The chained system uses 48M MACs and 3.72M parameters, compared with 440M MACs for PointNet on 1024 points.

A.4. Sampling consistency for registration task

For registration, SampleNet is evaluated on sampling consistency and across multiple ModelNet40 shape categories. It is substantially more consistent and achieves the best registration results against FPS and random sampling.

  • A.4. Sampling consistency for registration task: Sampling consistency is defined as the Chamfer distance between rotated template samples and source samples.The metric measures whether a sampler selects similar points from the two point clouds.
  • A.4. Sampling consistency for registration task: SampleNet sampling is substantially more consistent than random sampling and FPS on ModelNet40’s Car category.The authors state that this behavior can explain its registration success.
  • A.5. Registration for different shape categories: SampleNet achieves the best registration results across Table, Sofa, Toilet, Chair, Laptop, Airplane, and Guitar categories.
  • A.5. Registration for different shape categories: The registration evaluation uses different shape categories from ModelNet40.

B.1. Neighborhood size

The neighborhood size k controls the local context used for soft projection, and k = 7 provides the best classification accuracy among the tested settings. Deviating from this value decreases accuracy.

  • B.1. Neighborhood size: k defines the number of input-cloud neighbors used to softly project each query point.It controls the local context in which the sampler searches for an optimal point.
  • B.1. Neighborhood size: k = 7 is the baseline for evaluating neighborhood-size effects in SampleNet-Progressive classification.The comparison includes k ∈ {2, 4, 12, 16}, with the baseline accuracy difference set to 0.
  • B.1. Neighborhood size: Smaller or larger neighborhood sizes reduce classification accuracy relative to k = 7.The authors identify k = 7 as a sweet spot for the local exploration region.
  • B.1. Neighborhood size: Additional cross-entropy or entropy losses drive projection weights toward a delta function but hinder local exploration.The resulting nearest-neighbor locking does not improve task performance and instead worsens it.

D.2. SampleNet architecture

SampleNet uses pointwise convolution, global symmetric pooling, and fully connected layers, with task-specific architectures and final-layer widths determined by the sample size.

  • D.2. SampleNet architecture: SampleNet combines per-point convolution layers, symmetric global pooling, and fully connected layers.The architecture is specified separately for the paper’s different applications.
  • D.2. SampleNet architecture: The final fully connected layer has m outputs, where m is the requested sample size.For progressive sampling, m is 1024 for classification and 2048 for reconstruction.
  • D.2. SampleNet architecture: Batch normalization and ReLU are used throughout the convolutional layers, with task-specific fully connected-layer configurations.Classification and registration apply batch normalization and ReLU to non-final fully connected layers; reconstruction uses ReLU in its first two fully connected layers without batch normalization.

D.3. SampleNet optimization

SampleNet is optimized with task-specific hyperparameters and Adam, while progressive classification uses scheduled learning-rate decay and extended training.

  • D.3. SampleNet optimization: Progressive classification uses γ = 0.5 and δ = 1/30, with other optimization values taken from the hyperparameter table.Adam is used with momentum 0.9.
  • D.3. SampleNet optimization: The classification learning rate decays by 0.7 every 60 epochs.This schedule is part of the progressive-sampling optimization setup.
  • D.3. SampleNet optimization: The temperature coefficient starts at 1, is learned during training, and is clipped to maintain numerical stability.The minimum is 0.1 for registration and 0.01 for reconstruction.
  • D.3. SampleNet optimization: Training time ranges from 1.5 to 7 hours for classification, 1 to 2.5 hours for registration, and 4 to 30 hours for reconstruction.Progressive classification takes about 11 hours, while progressive reconstruction requires about 2.5 days.

D.4. Losses and evaluation metric for registration

Registration uses PCRNet with Chamfer and rotation supervision, while SampleNet is frozen and applied consistently to source and template point clouds. Performance is measured by mean rotation error.

  • D.4. Losses and evaluation metric for registration: PCRNet is trained with Chamfer distance between registered source and template clouds plus a supervised ground-truth rotation loss.The combined registration loss is Lcd(S, T) + Lrm(Rpred, Rgt).
  • D.4. Losses and evaluation metric for registration: Mean rotation error is computed from predicted and ground-truth rotations and reported in degrees over the test set.The rotation error is averaged after converting radians to degrees.
  • D.4. Losses and evaluation metric for registration: SampleNet-Progressive reconstructions emphasize shape outlines, unlike ProgressiveNet’s interior selections and FPS’s uniform spread.The associated figure compares input clouds, samples, and reconstructions across progressive sampling methods.
Loading 1912.03663v2…