PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiresolution Tree Networks for 3D Point Cloud Processing

Matheus Gadelha, Rui Wang, Subhransu Maji

arXiv:1807.03520v2cs.CVcs.GRcs.LG

TL;DR

Efficient processing of 3D point clouds remains challenging because voxel and multiview representations have scalability or geometric limitations. MRTNet addresses this with locality-preserving multiresolution point orderings, 1D convolutions, and multigrid processing, supporting recognition, generation, and inference tasks. The authors report stronger benchmark performance, faster classification inference than PointNet, and higher-quality unsupervised generation, subject to rotation sensitivity and generation difficulty at 4K points.

  • Problem

    Existing voxel and multiview representations make efficient point-cloud processing difficult because they scale poorly, model surface details inefficiently, or struggle with complex interiors.

  • Method

    MRTNet represents shapes as locality-preserving ordered point lists at multiple resolutions and processes them with 1D convolutions, pooling, and multigrid encoder-decoder architectures.

  • Results

    MRTNet outperforms existing methods on ShapeNet image-to-shape inference, reaches 86.4% ModelNet40 classification accuracy using MR-VAE features, and runs classification forward passes in 8.0ms versus PointNet’s 25.3ms.

  • Takeaways & Limitations

    The multiresolution tree representation supports a single point-cloud framework for shape classification, direct generation, image-to-shape inference, and unsupervised shape learning.

  • Takeaways & Limitations

    Kd-tree ordering is not rotation-invariant, and generating high-quality random-noise samples with 4K points remains difficult for vanilla and single-resolution decoders.

Abstract

from arXiv · show

We present multiresolution tree-structured networks to process point clouds for 3D shape understanding and generation tasks. Our network represents a 3D shape as a set of locality-preserving 1D ordered list of points at multiple resolutions. This allows efficient feed-forward processing through 1D convolutions, coarse-to-fine analysis through a multi-grid architecture, and it leads to faster convergence and small memory footprint during training. The proposed tree-structured encoders can be used to classify shapes and outperform existing point-based architectures on shape classification benchmarks, while tree-structured decoders can be used for generating point clouds directly and they outperform existing approaches for image-to-shape inference tasks learned using the ShapeNet dataset. Our model also allows unsupervised learning of point-cloud based shapes by using a variational autoencoder, leading to higher-quality generated shapes.

1 Introduction

MRTNet addresses efficient 3D point-cloud processing by representing shapes as locality-preserving ordered points across resolutions. Its encoder and decoder support classification, image-to-shape inference, and unsupervised shape learning.

  • Motivation: Point clouds are increasingly important because sensors directly collect them, while existing grid-based representations can be costly and resolution-limited.Voxel methods scale poorly and model surface details inefficiently; multiview methods struggle with concave or filled interiors because of self-occlusion.
  • Approach: MRTNet represents 3D shapes as locality-preserving 1D ordered point lists at multiple resolutions for direct point-cloud processing.The ordering can be obtained with space-partitioning trees such as kd-trees or rp-trees.
  • Applications: MRTNet supports shape classification, image-to-shape inference, and unsupervised shape learning through tree-structured encoders and decoders.The decoder directly outputs point clouds and can be combined with image-based encoders.
  • Approach: The network applies 1D convolutions and pooling to the ordered representation while using a multi-grid architecture to address distortion of 3D structure.Maintaining representations across resolutions supports coarse-to-fine processing and improves communication between scales.
  • Results: On ShapeNet single-image shape inference, MRTNet outperforms existing voxel-based, view-based, and point-based techniques.For unsupervised learning, its VAE features achieve 86.4% accuracy on ModelNet40, exceeding features from other unsupervised networks.

2 Related Work

Prior work processes 3D shapes with voxels, multiview images, meshes, graphs, or point clouds, but each representation has important scalability, geometric, or generation limitations. MRTNet builds on multiscale and multigrid ideas to process point clouds efficiently across resolutions.

  • Voxel-based methods: Uniform voxel grids have poor scalability, large memory requirements, high processing cost, and limited ability to model fine surface details.Multiscale and sparse variants reduce some costs but require additional bookkeeping.
  • Multiview methods: Multiview methods enable efficient convolutional processing but are suboptimal for complex interiors because of self-occlusion and require postprocessing to resolve view inconsistencies.They have nevertheless been adapted successfully for classification, segmentation, and image-to-shape generation.
  • Mesh and graph methods: Mesh and graph convolution methods support local correspondence and matching, but some are restricted to manifold surfaces and their performance for shape generation is unclear.These methods generalize convolution beyond regular grids in different ways.
  • Point-cloud methods: PointNet preserves permutation invariance by processing points independently and aggregating them with a symmetric max-pooling function.This approach is effective for shape classification and segmentation.
  • Point-cloud generation: Direct point-cloud generation avoids intermediate view-based depth maps, but scalable generation of point sets remains difficult.Prior approaches use Earth Mover’s Distance or Chamfer distance, while some generative models assume a linear shape basis that produces less detailed surfaces.
  • MRTNet’s position: MRTNet uses multiscale and multigrid principles to build efficient multiresolution representations for 3D point clouds.Communication across resolutions is associated with improved convergence and accuracy in related image-recognition work.

3 Method

MRTNet spatially sorts point clouds into locality-preserving 1D lists at multiple resolutions, enabling multigrid 1D convolutional processing for shape analysis and generation. Its encoder, decoder, and VAE variants support classification, image-to-shape inference, segmentation, and unsupervised shape learning.

  • Spatial sorting: MRTNet centers, normalizes, and spatially sorts fixed-size point clouds into 1D coordinate lists using a space-partitioning tree.KD-trees recursively split points along alternating x, y, and z axes to preserve local neighborhoods in the ordering.
  • Shape tasks: The multiresolution encoder supports ModelNet40 classification, while the decoder generates 4K-point shapes from images using a pretrained VGG-11 encoder.Classification uses a fully connected layer after the 512-D encoding; image-to-shape inference uses ShapeNet renderings and a decoder outputting a 4K point set.
  • Additional applications: MRTNet is also applicable to shape part segmentation, while its encoder-decoder combination enables unsupervised point-cloud learning.The classification, image-inference, and unsupervised models use task-specific combinations of the multiresolution encoder and decoder.
  • Multiresolution convolution: Three point-cloud resolutions are maintained through average pooling and nearest-neighbor upsampling, with scale ratio k between adjacent levels.Pooling uses kernel size and stride k, where k is a power of 2; the model uses k = 4 for generation and inference and k = 8 for classification.
  • Multiresolution convolution: MR-CONV combines pooled, upsampled, and same-resolution features across three scales before applying 1D convolution, batch normalization, and ReLU.This multigrid design combines global and local information while retaining the hierarchical tree structure.
  • Shape tasks: Chamfer distance replaces ordered pointwise L2 reconstruction loss because it is permutation-invariant for unstructured point clouds.The decoder is trained with Chamfer distance, whose order invariance avoids relying on the spatial ordering induced by partitioning.
  • Unsupervised learning: The MR-VAE combines the encoder and decoder, reconstructs point clouds with Chamfer distance, and regularizes latent statistics toward N(0, I).The regularizer matches the first two moments of the encoding distribution and adds Gaussian noise δ sampled from N(0, cI), with c = 0.01.

4 Experimental Results and Discussions

Experiments evaluate MRTNet for classification, image-to-shape inference, and point-cloud generation. Across these settings, multiresolution processing improves accuracy, convergence, spatial coherence, and computational efficiency, while transformation robustness has a rotation-related boundary.

  • Classification: MRTNet achieves the best results among point-based methods using xyz data only and remains competitive with methods using substantially more points or normals.With 4K points, MRTNet is comparable to KDNet at 32K points, which is only 0.1% better while using 8× more points and memory; PointNet++ with 5K points and normals is 0.2% better.
  • Classification: More than 2% higher classification accuracy is obtained than the single-resolution baseline with the same number of parameters.The multiresolution model also converges faster according to the cross-entropy decay comparison.
  • Single-image shape inference: MRTNet outperforms competing methods in 12 of 13 categories on pred!GT error and 6 categories on GT!pred error.It wins by a considerable margin in the mean per category on both metrics and captures thin structures better in the lamp category than the compared multiview method.
  • Point-cloud generation: MRTNet-generated point clouds show stronger spatial coherence than fully connected and single-resolution decoder outputs.Nearby 3D points are more likely to remain nearby in the 1D output list, supporting synthesis of detailed point clouds with coherent geometric structures.
  • Single-image shape inference: MRTNet captures object structure and geometric details from single images, including features such as office-chair wheels.The qualitative comparison includes input images, ground-truth shapes, MRTNet, Fan et al., and Choy et al.
  • Unsupervised learning: MR-VAE embeddings support high-quality reconstruction and achieve 86.34% accuracy on ModelNet40 without fine-tuning beyond a learned linear classifier.The embeddings are learned from unlabeled ShapeNet objects and outperform similar unsupervised features.
  • Robustness: The method is invariant to translations and uniform scaling, but kd-tree representations are not inherently rotation invariant.The authors report that unaligned training data was sufficient to account for rotations in ModelNet40 classification.
  • Computation: MRTNet classification runs in 8.0ms versus 25.3ms for PointNet on a TITAN GTX1080 with batch size 8.Building the kd-tree takes O(N log N), and is described as faster than rendering multiple views or voxelizing a shape.

5 Conclusion

MRTNet introduces multiresolution tree networks for point-cloud processing across shape classification, generation, and inference. Its multiscale representation supports coarse-to-fine analysis, faster training convergence, and higher-quality shape generation, while future work targets additional point attributes and spatio-temporal data.

  • 5 Conclusion: MRTNet represents shapes as locality-preserving 1D ordered point lists at multiple resolutions for efficient convolution and pooling.The representation uses space-partitioning trees and supports processing through 1D operations.
  • 5 Conclusion: The multiresolution representation improves information flow across scales, enabling coarse-to-fine analysis, faster convergence, and higher-quality shape generation.
  • 5 Conclusion: MRTNet is designed for shape classification, generation, and inference tasks using point clouds.
  • 5 Conclusion: Future work will incorporate point attributes such as normals and color and extend the method to spatio-temporal shape analysis.

Supplemental Material – Multiresolution Tree Networks for 3D Point Cloud Processing

This supplemental section contains author and institutional information rather than technical findings or experimental results.

  • Supplemental Material – Multiresolution Tree Networks for 3D Point Cloud Processing: The paper lists Matheus Gadelha, Rui Wang, and Subhransu Maji as its authors.
  • Supplemental Material – Multiresolution Tree Networks for 3D Point Cloud Processing: The authors are affiliated with the College of Information and Computer Sciences at the University of Massachusetts Amherst.

1 Shape segmentation model

The segmentation model adapts MRTNet for point-part labeling by adding U-Net-style skip connections and producing per-point scores over 50 possible parts. On ShapeNet, it is competitive, nearly surpasses KDNet by 2% in mean per-category mIoU, but trails PointNet.

  • 1 Shape segmentation model: The segmentation model adds skip connections between matching-resolution encoder and decoder tensors and outputs 50 part-classification scores per point.The ShapeNet segmentation task covers 50 possible parts across 16 object categories.
  • 1 Shape segmentation model: RPtree spatially sorts the input using shared random splitting axes sampled uniformly from the unit sphere.The experiments report a slight segmentation improvement over KDtree with this ordering.
  • 1 Shape segmentation model: The ShapeNet part-segmentation benchmark contains 16,881 labeled shapes across 16 categories and 50 total parts.Each object contains 2 to 6 parts, and varying point counts are handled by duplicating existing points.
  • 1 Shape segmentation model: The segmentation model is trained with cross-entropy using Adam at learning rate 10^-3, halving the learning rate every five epochs.
  • 1 Shape segmentation model: Figure 1 presents qualitative segmentation results with different colors indicating different segmentation parts.
  • 1 Shape segmentation model: Nearly 2%: MRTNet outperforms KDNet in mean per-category mIoU, but trails PointNet and is described as competitive with state-of-the-art methods.Table 1 reports mean intersection over union scores for methods using 3D position without normal information.
  • 1 Shape segmentation model: The single-resolution counterpart lowers performance by about 1% relative to the multiresolution U-Net.The authors attribute the smaller segmentation gap to skip connections already conveying information across scales.

2 Image-to-Shape Inference: Additional Results

Additional image-to-shape results show how predicted point clouds can be converted into triangle meshes and provide full ablations across 13 shape categories. The reconstructed meshes are reasonably faithful to input images, and MRTNet consistently outperforms both reported baselines.

  • 2 Image-to-Shape Inference: Additional Results: MRTNet point-cloud outputs can be converted into triangle meshes using point-centered cubes followed by Poisson Surface Reconstruction.The procedure generates meshes without normal estimation and may smooth some geometric details.
  • 2 Image-to-Shape Inference: Additional Results: Each image-to-shape example contains an input photo, a generated 4K-point cloud, and a wireframe rendering of the reconstructed mesh.
  • 2 Image-to-Shape Inference: Additional Results: The reconstructed meshes are reasonably faithful to the input images, although some geometric details are necessarily smoothed.
  • 2 Image-to-Shape Inference: Additional Results: Across all 13 shape categories, MRTNet is consistently better than both the single-resolution and fully connected baselines in the full ablation study.

3 Unsupervised Shape Generation (MR-VAE): Additional Results

Additional results evaluate MR-VAE through random shape generation and latent-space visualization, including comparisons with fully connected and single-resolution decoders.

  • Comparison with FC Decoder: MR-VAE generates higher-quality random samples than the fully connected decoder baseline.The comparison uses chairs sampled from the encoding rather than reconstructed inputs.
  • Decoder Limitations: Random-noise generation with 4K points is difficult for vanilla decoders, which produce many misplaced points.Single-resolution decoders also produce low-quality samples.
  • Decoder Comparisons: PointNet and KD-net are not usable as decoders because their architectures respectively ignore point ordering or condition processing on spatial splits.These architectural properties prevent their direct use for decoding.
  • Latent-Space Visualization: MR-VAE learns latent representations in which similar ShapeNet shapes tend to appear near one another after t-SNE projection.The visualization uses 1,000 randomly selected ShapeNet shapes.
Loading 1807.03520v2…