PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds

Lei Li, Siyu Zhu, Hongbo Fu, Ping Tan, Chiew-Lan Tai

arXiv:2003.05855v2cs.CV

TL;DR

Local 3D descriptors must handle noisy, incomplete geometry, while existing multi-view methods rely on hand-crafted viewpoints and limited feature fusion. The paper integrates differentiable rendering with optimizable viewpoints and attentive soft-view pooling, and reports superior performance on 3D registration benchmarks, including under stricter evaluation conditions.

  • Problem

    Noisy, incomplete 3D data and detached viewpoint selection make it difficult to learn informative multi-view local descriptors.

  • Method

    An end-to-end network uses differentiable point-cloud rendering with trainable viewpoints and soft-view pooling to attentively fuse features across views.

  • Results

    The method outperforms competing local descriptors on 3D registration benchmarks, achieving 86.9% average recall at τ2 = 0.2 and 82.1% under rotated evaluation.

  • Takeaways & Limitations

    Jointly optimized viewpoints and attentive cross-view fusion provide a strong local descriptor that generalizes to outdoor scenes.

  • Takeaways & Limitations

    Equal-weight view fusion can reduce descriptor discriminativeness, and reported runtime comparisons depend on hardware and implementation conditions.

Abstract

from arXiv · show

In this work, we propose an end-to-end framework to learn local multi-view descriptors for 3D point clouds. To adopt a similar multi-view representation, existing studies use hand-crafted viewpoints for rendering in a preprocessing stage, which is detached from the subsequent descriptor learning stage. In our framework, we integrate the multi-view rendering into neural networks by using a differentiable renderer, which allows the viewpoints to be optimizable parameters for capturing more informative local context of interest points. To obtain discriminative descriptors, we also design a soft-view pooling module to attentively fuse convolutional features across views. Extensive experiments on existing 3D registration benchmarks show that our method outperforms existing local descriptors both quantitatively and qualitatively.

1. Introduction

The paper addresses the difficulty of learning informative local descriptors from noisy, incomplete 3D data by making multi-view rendering and feature fusion trainable within one network. It optimizes rendering viewpoints and uses attentive soft-view pooling to improve descriptor learning.

  • Motivation: Robust local descriptors are important for correspondence, registration, segmentation, and retrieval because consumer RGB-D scans are noisy and incomplete.The paper motivates descriptor learning particularly for scene alignment and reconstruction from voluminous scanned data.
  • Motivation: Hand-crafted viewpoints are selected before learning and may not produce informative renderings for neural networks.Existing approaches use randomly sampled or heuristically chosen viewpoints, leaving data-driven viewpoint selection unresolved.
  • Motivation: Max-view pooling can overlook subtle cross-view details because it retains only the strongest response at each feature-map position.This creates a need for a fusion operation that integrates information from multiple views rather than selecting one response.
  • Proposed framework: The framework jointly learns local multi-view descriptors through in-network rendering, per-view feature extraction, and cross-view fusion.Its three stages are multi-view rendering, feature extraction in each rendered view, and feature fusion across views.
  • Proposed framework: Differentiable rendering makes viewpoints optimizable during training, allowing them to be learned jointly with later descriptor-learning stages.The approach replaces detached preprocessing with an in-network renderer whose viewpoint parameters can be optimized.
  • Proposed framework: Soft-view pooling adaptively aggregates responses from all views with attentive weights, improving supervision flow into each input view.Unlike max-view pooling, the module uses a sub-network to estimate weights for cross-view feature integration.

2. Related Work

Prior 3D local descriptors use point patches, voxel grids, or multi-view images, with learned methods complementing older histogram-based designs. Multi-view representations offer rich geometric information, but viewpoint selection and feature fusion remain central design issues.

  • Hand-crafted 3D local descriptors: Hand-crafted descriptors mostly encode local geometry with histograms of low-level geometric properties.Examples include Spin Image, 3D Shape Contexts, PFH, FPFH, SHOT, and Unique Shape Context.
  • Learned 3D local descriptors: Learned 3D local descriptors generally use point cloud patches, voxel grids, or multi-view images as input representations.These categories organize recent work according to how local 3D geometry is represented.
  • Learned 3D local descriptors: Voxel-grid methods such as 3DMatch [66] and 3DSmoothNet [12] provide structured representations, with smoothed density voxelization reducing noise and boundary effects.3DSmoothNet [12] achieved state-of-the-art performance on the 3DMatch benchmark [66] against the cited PointNet-based approaches.
  • Multi-view descriptors: Multi-view images have outperformed voxel grids in 3D shape recognition and retrieval, motivating their extension to local descriptor learning.Their advantage is attributed to delivering rich information about 3D geometry.
  • Multi-view descriptors: Max-view pooling is computationally efficient and view-order invariant but can overlook subtle details during feature fusion.Fuseption [67] avoids this issue with residual learning but is not view-order invariant and grows in parameter count with the number of views.
  • Differentiable rendering: Conventional rendering is nondifferentiable with respect to projected coordinates and view-dependent depths, preventing supervision from flowing from 2D images back to 3D shape space.This limits direct integration of the conventional graphics pipeline into end-to-end neural networks.

3. Methodology

The framework jointly learns multi-view descriptors by rendering point-cloud neighborhoods with optimizable viewpoints, extracting per-view CNN features, and attentively fusing them. Differentiable rendering and soft-view pooling address viewpoint selection and gradient-flow limitations in conventional multi-view pipelines.

  • 3. Methodology: The network renders local point-cloud neighborhoods, extracts features from each view, and fuses them into descriptors through three jointly trained stages.It uses SoftRas for rendering, a lightweight 2D CNN for per-view features, and a final embedding layer with l2 normalization.
  • 3.1. Multi-view Rendering: Optimizable viewpoints replace detached hand-crafted selection, using spherical angles, distance, and upright orientation constrained to the point-normal hemisphere.Each rendered view is augmented with four 90-degree in-plane rotations, producing 4n patches.
  • 3.1. Multi-view Rendering: The differentiable point renderer models each point as a projected sphere and combines depth-dependent pixel contributions, enabling gradients through rendering.Its linear aggregation approximates rasterization and visibility while using view-dependent depth when point colors are unavailable.
  • 3.1. Multi-view Rendering: A hard-forward soft-backward scheme preserves conventional rasterized outputs while retaining differentiability to reduce blurry depth-discontinuity artifacts.The paper reports that this approximation works well in experiments.
  • 3.3. Multi-view Fusion: Soft-view pooling adaptively weights all views at each feature location, addressing max pooling’s one-view gradient bottleneck and average pooling’s equal-weight limitation.A sub-network predicts weight maps, softmax normalizes them across views, and the resulting fusion remains view-order invariant.

4. Experiments

On 3DMatch and related robustness tests, the proposed descriptor consistently outperforms competing methods, especially under stricter matching conditions, rotations, sparse points, and outdoor-scene generalization. Ablations show that optimizable viewpoints and soft-view pooling are key design choices, while performance saturates at d = 32 and n = 8.

  • 4.2. Evaluation Results: 86.9% average recall at τ2 = 0.2, exceeding 3DSmoothNet’s 72.9% and LMVCNN’s 81.0% on 3DMatch.At τ2 = 0.05, the method also achieves 97.5%, outperforming all competing descriptors, although results are nearly saturated.
  • 4.2. Evaluation Results: 96.9% and 82.1% average recall at τ2 = 0.05 and 0.2, respectively, surpassing 3DSmoothNet and LMVCNN on randomly rotated fragments.The results indicate that the descriptor handles rotation well.
  • 4.2. Evaluation Results: The descriptor maintains superior performance under 50% or 25% point-density sampling, indicating robustness to sparse point clouds.Sphere-based rendering is identified as supporting performance across different point densities, similarly to LMVCNN and 3DSmoothNet.
  • 4.3. Generalization to Outdoor Scenes: 79.9% average recall on the ETH outdoor benchmark is comparable to 3DSmoothNet’s 79.0% and exceeds LMVCNN’s 39.7% and SHOT’s 61.1%.The method trained on 3DMatch therefore generalizes well to noisy, low-resolution outdoor scenes.
  • 4.4. Ablation Study: Average recall becomes saturated for descriptor dimensions d ≥ 32 and viewpoint numbers n ≥ 8, motivating the choices d = 32 and n = 8.The ablation evaluates both descriptor dimensionality and the number of rendered viewpoints.
  • 4.4. Ablation Study: Optimizable viewpoints outperform random, clustered, orbited, and non-augmented alternatives, especially on ETH, while soft-view pooling mainly improves outdoor generalization over max-view pooling.On 3DMatch, soft-view pooling improves only slightly over max-view pooling; noisy outdoor renderings make selecting a single strongest response more difficult.

5. Conclusion

The paper presents an end-to-end framework that jointly learns optimizable viewpoints and attentively fuses multi-view features for local 3D point-cloud descriptors, achieving superior performance and outdoor-scene generalization.

  • 5. Conclusion: The framework performs in-network multi-view rendering with jointly trainable viewpoints and attentively integrates convolutional features through soft-view pooling.These components form the paper’s end-to-end local multi-view descriptor architecture.
  • 5. Conclusion: Experiments demonstrate superior descriptor performance and generalization to outdoor scenes.The conclusion reports these outcomes without specifying numerical results here.
  • 5. Conclusion: Future work includes accelerating differentiable point-cloud rendering and extending the framework to 3D object detection and recognition.

6. Supplementary Material

The supplementary material details the CNN backbone, learned viewpoint behavior, fusion comparisons, descriptor visualizations, and runtime differences relative to 3DSmoothNet.

  • 6. Supplementary Material: The CNN processes 64×64 single-channel depth patches and outputs 8×8 feature maps with 128 channels for each view.Table 7 documents the backbone configuration used for per-view feature extraction.
  • 6. Supplementary Material: Learned viewpoints concentrate their distances and distribute viewing directions more broadly than the clustering-based handcrafted rule.The comparison suggests reduced emphasis on broader global context and less concentration on dominant directions.
  • 6. Supplementary Material: Soft-view pooling and NetVLAD produce stronger pre-fusion responses than max-view pooling and Fuseption, while fusion changes the resulting feature-map responses.Figure 10 visualizes rendered inputs, per-view feature maps, and fused maps across these alternatives.
  • 6. Supplementary Material: PCA visualizations show that both methods produce geometry-aware descriptors, while the proposed method captures more geometric changes and supports additional registration results.The highlighted wall, pillow, and floor regions illustrate the reported geometric sensitivity.
  • 6. Supplementary Material: 3DSmoothNet input-preparation time differs substantially between experiments and the authors’ report, likely because of hardware and CPU-versus-GPU processing differences.The reported times are 39.4 ms versus 4.2 ms for input preparation, while inference is 0.2 ms versus 0.3 ms.
Loading 2003.05855v2…