PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiresolution Tree Networks for 3D Point Cloud Processing

Matheus Gadelha, Rui Wang, Subhransu Maji

arXiv:1807.03520v2cs.CVcs.GRcs.LG

TL;DR

Point clouds are increasingly collected directly by sensors, yet common grid-based shape representations can scale poorly or struggle with concave and filled interiors. MRTNet processes point clouds through a multiresolution tree representation, with reported gains in image-to-shape inference and higher-quality shape generation.

  • Problem

    Point clouds are increasingly collected directly by sensors, yet common grid-based shape representations can scale poorly or struggle with concave and filled interiors.

    Source: §1 · p. 1

  • Method

    MRTNet represents point clouds as locality-preserving ordered point lists at multiple resolutions, enabling 1D convolution and coarse-to-fine analysis.

    Source: §5 · p. 14

  • Results

    MRTNet outperforms voxel-, view-, and point-based methods on single-image ShapeNet inference; VAE features achieve 86.4% accuracy on ModelNet40, exceeding other unsupervised networks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The representation supports coarse-to-fine analysis, faster training convergence, and higher-quality shape generation.

    Source: §5 · p. 14

    Kd-trees are not rotation-invariant, although unaligned training data sufficed to account for rotations in the classification task.

    Source: §4.4 · p. 14

Abstract

from arXiv · show

We present multiresolution tree-structured networks to process point clouds for 3D shape understanding and generation tasks. Our network represents a 3D shape as a set of locality-preserving 1D ordered list of points at multiple resolutions. This allows efficient feed-forward processing through 1D convolutions, coarse-to-fine analysis through a multi-grid architecture, and it leads to faster convergence and small memory footprint during training. The proposed tree-structured encoders can be used to classify shapes and outperform existing point-based architectures on shape classification benchmarks, while tree-structured decoders can be used for generating point clouds directly and they outperform existing approaches for image-to-shape inference tasks learned using the ShapeNet dataset. Our model also allows unsupervised learning of point-cloud based shapes by using a variational autoencoder, leading to higher-quality generated shapes.

1 Introduction

The paper introduces a multiresolution tree network that recognizes and generates 3D shapes directly as point clouds. It uses locality-preserving point orderings at multiple resolutions for efficient processing.

1 Introduction

  • Direct point-cloud processing is increasingly desirable, while voxel grids scale poorly, view-based methods struggle with concave or filled interiors, and view collections require geometric-consistency reasoning.Point clouds are lightweight shape representations and are collected directly by sensors in robotic and remote-sensing applications.Source: §1 · p. 1
  • The network achieves 91.7% accuracy on ModelNet40 and outperforms existing point-based networks using point-position information [24] [32].It also outperforms similar graph networks that do not maintain multiresolution representations.Source: §1 · p. 2
  • The network orders points at multiple resolutions using space-partitioning trees and applies 1D convolutions and pooling within a multi-grid architecture [22].Cross-resolution influence addresses distortions from 1D ordering, preserves locality, supports coarse-to-fine learning, and enables faster training convergence.Source: §1 · p. 2
  • For ShapeNet single-image shape inference, the method outperforms voxel-based [9], view-based [26], and point-based [12] techniques.Its decoder directly generates point clouds, supports order-invariant losses such as Chamfer distance, and preserves overall structure and fine details.Source: §1 · p. 2
  • VAE encoder features achieve 86.4% accuracy on ModelNet40, outperforming features from other unsupervised networks.The VAE is trained on unlabeled ShapeNet data.Source: §1 · p. 2

2 Related Work

Prior 3D-shape methods use voxels, rendered views, surfaces or graphs, and point sets, with limitations in scalability, geometric detail, or generation. This work presents multiscale tree processing as an efficient representation for point-cloud analysis and direct generation.

2 Related Work

  • Voxel methods [9] scale poorly and often use low-resolution grids; multiscale and sparse variants [41] [42] [38] [16] add bookkeeping, while voxel processing remains costly and weak for surface detail and normals.Source: §2 · pp. 2–3
  • Multiview methods [39] [31] [28] [20] leverage image convolutions and pretrained networks for exterior-surface tasks, but self-occlusion limits complex interiors and generated depth or normal views require difficult inconsistency and outlier resolution [28] [26].Source: §2 · p. 3
  • Mesh CNNs [29] show success in correspondence and matching but may require manifold surfaces and have unclear generation performance; graph convolution [35] handles varying graph size and connectivity.Source: §2 · p. 3
  • PointNet [32] uses pointwise processing and symmetric max pooling for permutation-invariant classification and segmentation, while KD-net [24] is scalable and memory efficient but has not been shown as a shape-generation decoder.Source: §2 · p. 3
  • Our multiscale 1D-deconvolution decoder is shallower than Fan et al. [12]’s two-branch design, improves generated-shape quality, and contrasts with ’s linear-basis GAN that produces less detailed surfaces.Permutation-invariant losses and latent-variable regularization enable VAE-like shape sampling from Gaussian noise [23]; improved scalability permits more points and more accurate geometric detail.Source: §2 · pp. 3–4
  • The tree network draws on multiscale [18] [27], multigrid [22], and dilated or atrous filters [46] [8], using multigrid-like representations to process 3D point clouds efficiently.Source: §2 · p. 4

3 Method

MRTNet represents point clouds as spatially sorted 1D lists and processes them with 1D convolutions at multiple resolutions. The encoder and decoder support classification, image-to-shape inference, and unsupervised shape learning.

Spatial sorting

  • The network sorts points recursively by alternating split axes; probabilistic KD-trees [24] and RP-trees are alternatives, and the split directions are discarded after ordering.The probabilistic KD-tree chooses axes with probability proportional to the point subset's span along each axis.Source: §3 · p. 5
  • MRTNet uses a hierarchical spatial order to approximate local 3D neighborhoods with 1D operations, while multiple resolutions help preserve locality at different scales.Long-range relationships are not well preserved by any single ordering.Source: §3 · p. 5

Multiresolution convolution

  • Three resolutions exchange pooled or upsampled features through concatenation, then apply stride-2 1D convolution, batch normalization, and ReLU.Average pooling uses kernel and stride k, a power of 2; upsampling uses nearest neighbors. Classification uses k = 8, while inference and generation use k = 4.Source: §3 · pp. 5–6

Shape classification model

  • For ModelNet40 classification, the multiresolution encoder feeds a fully connected layer that outputs 40 class scores.The model uses 1K sampled points per object and probabilistic KD-tree ordering.Source: §3 · pp. 4, 6

Single-image shape inference

  • For single-image shape inference, a pretrained VGG-11 encoder [36] feeds the multiresolution decoder, which predicts a 4K-point cloud using permutation-invariant Chamfer distance.The passage reports that L2 loss based on the induced point ordering works poorly.Source: §3 · pp. 6–7

Unsupervised learning of point clouds

  • MR-VAE combines the multiresolution encoder and decoder, reconstructs point clouds with Chamfer distance, and regularizes latent codes by matching their first two moments to N(0, I) for more stable training.The encoder produces a 512-dimensional code.Source: §3 · p. 7

Shape part segmentation

  • MRTNet can also be applied to shape part segmentation.The section refers readers to the supplemental material for details.Source: §3 · p. 7

4 Experimental Results and Discussions

Experiments report strong results for MRTNet on point-cloud classification and single-image shape inference, and show that MR-VAE produces useful unsupervised representations and shape reconstructions. The discussions also describe its runtime and transformation robustness.

4.1 Shape classification

  • MRTNet is the best point-based method using xyz data alone, leads the 1K-point group, and is only 0.1% behind KDNet using eight times as many points at 32K.PointNet++ [32] with 5K points and normals is 0.2% better than MRTNet.Source: §4.1 · p. 8, Table 1 · p. 8
  • The multiresolution classifier is more than 2% better than a same-parameter single-resolution baseline and converges faster; MRTNet also gains greater benefits with less overhead than KDNet’s axis-conditioned convolutions.The single-resolution baseline is 1.3% less accurate than KDNet at 1K points, while axis conditioning entails extra bookkeeping and at least three times more parameters.Source: §4.1 · pp. 8–9, Table 1 · p. 8, Figure 3 · p. 8

4.2 Single-image shape inference

  • MRTNet outperforms competing methods in 12 of 13 categories on pred!GT error and 6 on GT!pred, with a considerable mean-per-category margin on both metrics.The lamp-category GT!pred error is more than two times smaller than [26]’s, indicating better capture of thin structures; pred!GT measures prediction-to-ground-truth distance, while GT!pred measures coverage.Source: §4.2 · pp. 9–10, Table 2 · p. 9
  • MRTNet’s decoder produces spatially coherent point lists that support detailed geometric structures, and its outputs capture object structure and details from real photos despite training on rendered images.The real-photo inputs had their backgrounds manually removed; examples include office-chair wheels.Source: Figure 4 · p. 10, Figure 5 · p. 10, §4.2 · pp. 11–12

4.3 Unsupervised Learning of Point Clouds

  • MR-VAE generates higher-quality chair shapes than its single-resolution baseline and reconstructs high-quality shapes across ShapeNet categories using an 80%/20% training/test split.The results indicate potential for inference tasks such as shape completion or point-cloud reconstruction.Source: §4.3 · pp. 12–13, Figure 7 · p. 12, Figure 8 · p. 12
  • MR-VAE’s generated points show consistent ordering across shapes, and its decoder produces plausible chair interpolations from linearly interpolated encodings.Points from selected index ranges fall in approximately corresponding regions across generated chairs.Source: §4.3 · p. 13, Figure 9 · p. 13, Figure 10 · p. 13
  • MR-VAE features achieve 86.34% ModelNet40 accuracy with a single linear layer and no fine-tuning, outperforming similar unsupervised autoencoder features.The features come from the encoder’s first three layers; only the linear classifier is learned.Source: Table 1 · p. 8, §4.3 · pp. 13–14

4.4 Discussions

  • On a TITAN GTX1080 with batch size 8, MRTNet’s classification forward pass takes 8.0ms versus PointNet’s 25.3ms; the method is translation- and uniform-scale-invariant, while kd-trees are not rotation-invariant.The authors report that unaligned ModelNet40 training data was sufficient to account for rotations in classification; building the kd-tree takes O(N log N) time.Source: §4.4 · p. 14

5 Conclusion

MRTNet represents point clouds as locality-preserving ordered point lists at multiple resolutions, enabling efficient 1D operations and coarse-to-fine analysis. The authors identify additional point attributes and spatio-temporal shape analysis as future directions.

5 Conclusion

  • MRTNet’s multiresolution point-list representation supports efficient 1D convolution and pooling, improves information flow across scales, and enables coarse-to-fine analysis, faster training convergence, and higher-quality shape generation.The network is applicable to shape classification, generation, and inference tasks.Source: §5 · p. 14
  • Future work would incorporate attributes such as normals and color and extend MRTNet to spatio-temporal analysis of animated shapes or other changing 3D point data.The proposed attribute extensions aim to improve recognition accuracy and allow the generator to produce those attributes.Source: §5 · p. 14

1 Shape segmentation model

The section adapts MRTNet’s multiresolution encoder–decoder for pointwise part labeling and reports competitive ShapeNet segmentation results. It also examines the effects of multiresolution processing and skip connections.

Shape segmentation model

  • The segmentation network follows MR-VAE, adds UNet-style skip connections between matching encoder and decoder tensors, and outputs 50 part-class scores per point.The 50 possible parts span 16 ShapeNet object categories.Source: §1 · p. 18
  • Spatial sorting uses shared precomputed random splitting axes sampled over the unit sphere, yielding consistent point order and a slight improvement over KDtree in segmentation experiments.The axes replace the xyz splitting axes of vanilla KDtree and are intended to preserve neighboring-point relationships for dense classification.Source: §1 · p. 18
  • Evaluation uses 16,881 ShapeNet shapes across 16 categories and 50 parts, duplicates points to 4,096 per shape, and ignores labels absent from each category.Each object has 2 to 6 parts, and the point-cloud dataset requires no additional sampling.Source: §1 · p. 18
  • Training minimizes cross-entropy with Adam at learning rate 10^-3 and β = 0.9, while using anisotropic scaling augmentation and averaging predictions across 16 scaled inputs at inference.The learning rate is halved every 5 epochs; the first three convolutional layers use 32 filters and the next three use 64. Augmentation scales are sampled uniformly from [0.8, 1.2].Source: §1 · pp. 18, 20
  • The model achieves competitive ShapeNet part-segmentation results, outperforming KDNet by nearly 2% in mean per category mIoU but trailing PointNet.The reported comparisons use 3D position information without normals.Source: Table 1 · p. 19, §1 · p. 20
  • Removing skip connections reduced accuracy from 79.82% to 76.14%; single resolution and increasing kernel size from 2 to 8 each caused about a 1% drop.The authors attribute the smaller single-resolution drop to multiresolution features already providing benefits similar to UNet skip connections; the kernel result suggests multiresolution adds more than receptive-field size alone.Source: §1 · p. 20

2 Image-to-Shape Inference: Additional Results

The section extends MRTNet image-to-shape inference results by reporting ablations across 13 shape categories and showing how its point-cloud outputs can be converted into triangle meshes.

Reconstructed meshes

  • For mesh outputs, 4K-point MRTNet predictions are converted with Poisson Surface Reconstruction, yielding reasonably faithful meshes despite smoothing some geometric details.The method places a 0.0253 cube centered at each point and reconstructs the mesh without normal estimation.Source: §2 · pp. 20–21, Figure 2 · p. 21

Full ablation studies

  • Across all 13 shape categories, MRTNet is consistently better than the single-resolution and fully connected baselines in the full image-to-shape ablation study.The section refers readers to Table 2 for the full results.Source: §2 · p. 21

3 Unsupervised Shape Generation (MR-VAE): Additional Results

Additional MR-VAE results show higher-quality generated samples than a fully connected baseline and latent encodings in which similar shapes tend to cluster together.

Comparison with FC Decoder

  • MR-VAE generates higher-quality randomly sampled shapes than the fully connected (FC) baseline, whose samples are much lower quality.Solutions like use FC layers as an additional branch to a much deeper convolutional architecture, rather than as standalone decoders.Source: Figure 3 · p. 22, §3 · p. 22
  • Random-noise generation is harder than reconstruction; for 4K-point shapes, vanilla decoders tend to produce misplaced points, and single-resolution decoders also yield low-quality samples.The passage contrasts random-noise samples with reconstructions, which it describes as easy to generate at high quality.Source: §3 · p. 22
  • PointNet and Kd-net cannot serve as decoders because PointNet ignores point ordering, while Kd-net conditions processing on the splits.Source: §3 · p. 22

Visualization of MR-VAE encodings using t-SNE

  • Similar shapes tend to cluster together in MR-VAE’s t-SNE visualization, indicating that the model learns latent shape representations.The visualization projects encodings of 1,000 randomly selected ShapeNet shapes into 2D using t-SNE.Source: §3 · pp. 22–23, Figure 4 · p. 23

Related papers

Loading 1807.03520v2…