PaperlayerView exact revision on arXiv

Source-linked AI summary

FoldingNet: Point Cloud Auto-encoder via Deep Grid Deformation

Yaoqing Yang, Chen Feng, Yiru Shen, Dong Tian

arXiv:1712.07262v2cs.CV

TL;DR

Point clouds have irregular, sparse structure that makes conventional deep-learning representations challenging. FoldingNet combines a graph-based encoder with a decoder that folds a 2D grid into a point-cloud surface. Its ShapeNet-to-ModelNet40 transfer classification accuracy is 88.4%, and its decoder is theoretically universal, though complex surfaces can retain genus-wise distortions.

  • Problem

    Point clouds have irregular, sparse structure that does not meet conventional CNN assumptions, while voxelization can sacrifice accuracy or add redundancy.

    Source: §1 · p. 1

  • Method

    FoldingNet combines graph-based local neighborhood pooling with a decoder that concatenates codewords to 2D grid points and folds the grid into a point cloud.

    Source: §1 · pp. 1–2, §1.1 · p. 3

  • Results

    88.4% transfer classification accuracy is reported on ModelNet40 after ShapeNet training, close to the state-of-the-art supervised result [41].

    Source: §1 · p. 2

  • Takeaways & Limitations

    The folding decoder avoids costly 3D voxelization and is theoretically universal for point-cloud reconstruction.

    Source: §1 · p. 2

    Complex surfaces may retain genus-wise distortions, and finer details may require more input samples and more complex encoder/decoder networks.

    Source: §1.1 · p. 3

Abstract

from arXiv · show

Recent deep networks that directly handle points in a point set, e.g., PointNet, have been state-of-the-art for supervised learning tasks on point clouds such as classification and segmentation. In this work, a novel end-to-end deep auto-encoder is proposed to address unsupervised learning challenges on point clouds. On the encoder side, a graph-based enhancement is enforced to promote local structures on top of PointNet. Then, a novel folding-based decoder deforms a canonical 2D grid onto the underlying 3D object surface of a point cloud, achieving low reconstruction errors even for objects with delicate structures. The proposed decoder only uses about 7% parameters of a decoder with fully-connected neural networks, yet leads to a more discriminative representation that achieves higher linear SVM classification accuracy than the benchmark. In addition, the proposed decoder structure is shown, in theory, to be a generic architecture that is able to reconstruct an arbitrary point cloud from a 2D grid. Our code is available at http://www.merl.com/research/license#FoldingNet

1. Introduction

FoldingNet addresses unsupervised point-cloud learning with graph-based local feature aggregation and a decoder that folds a 2D grid into a reconstructed point cloud. Its transfer-classification result and the decoder’s theoretical universality motivate the approach, while complex surfaces can still show genus-wise distortions.

1. Introduction

  • Point clouds’ irregular, sparse structure complicates traditional CNN processing, while voxelization sacrifices accuracy or adds redundancy and processing cost.CNNs rely on neighboring samples at fixed spatial orientations and distances, which point clouds typically do not provide.Source: §1 · p. 1
  • FoldingNet encodes point clouds into codewords and folds a 2D grid twice to reconstruct the input surface.The decoder concatenates the codeword with grid points and applies perceptrons; the paper theoretically establishes that folding can represent arbitrary point clouds.Source: §1 · pp. 1–2
  • 88.4% transfer classification accuracy on ModelNet40 follows ShapeNet training, and FoldingNet outperforms [1] on ModelNet40 transfer classification.The reported accuracy is close to the state-of-the-art supervised result [41].Source: §1 · p. 2, §1.1 · p. 3

1.1. Related works

  • The graph encoder uses K-nearest-neighbor neighborhoods and repeated local max-pooling to capture local signatures, unlike global max-pooling alone.The authors describe this graph-based layer as simpler and computationally efficient compared with graph-based convolution networks.Source: §1.1 · p. 3
  • Complex surfaces may retain genus-wise distortions, such as missing winglets or holes; finer detail may require more input samples and more complex networks.The paper notes that the 2D-grid mapping can generate cuts and surfaces not topologically equivalent to a 2D grid, but reconstruction can remain imperfect.Source: §1.1 · p. 3

1.2. Preliminaries and Notation

  • The preliminaries use S for a point set, θ for the codeword, bold lowercase letters for vectors, and bold uppercase letters for matrices.An m-by-n matrix has m rows and n columns.Source: §1.2 · p. 4

2. FoldingNet Auto-encoder on Point Clouds

FoldingNet encodes a point cloud into a codeword and reconstructs it with a decoder that folds a 2D grid. Its graph-based encoder aggregates local neighborhood information, while successive folding operations produce more elaborate shapes.

2. FoldingNet Auto-encoder on Point Clouds

  • The encoder maps each input point cloud to a codeword, and the decoder reconstructs point positions evaluated with an extended Chamfer distance.The reconstructed point count need not equal the input point count; the experiments use a codeword length of 512.Source: §2 · p. 4

2.1. Graph-based Encoder Architecture

  • The graph-based encoder concatenates each point's 3D position and 3-by-3 local covariance, then uses two graph layers to aggregate neighborhood features.The K-NN graph is built from input positions with K = 16; successive local max-pooling layers propagate topology information into larger areas.Source: §2.1 · p. 4

2.2. Folding-based Decoder Architecture

  • The decoder concatenates replicated codewords with points on a fixed 2D grid and applies two successive 3-layer perceptrons to reconstruct the cloud.The first folding maps the grid into 3D space and the second folds within 3D, allowing their composition to produce elaborate surface shapes.Source: §2.2 · pp. 4–5
  • Each folding operation applies a point-wise MLP to grid coordinates concatenated with the codeword, which guides the resulting 2D-to-3D mapping.The codeword is described as storing the force needed to make the folding operation more diverse.Source: §2.2 · pp. 4–5

3. Theoretical Analysis

The theoretical analysis establishes permutation invariance for the encoder and an existence result for the folding-based decoder. The universality construction guarantees arbitrary point-cloud reconstruction in theory, but differs from the trained FoldingNet decoder used in practice.

3. Theoretical Analysis

  • The encoder is permutation invariant: permuting the input point-cloud rows leaves the codeword unchanged.Source: §3 · p. 5
  • A 2-layer perceptron using the folding operation can reconstruct an arbitrary point cloud from a 2D grid for a suitable codeword.The proof sketch constructs such a perceptron explicitly.Source: §3 · p. 5
  • The universality proof is existence-based and does not describe the trained decoder's behavior; its construction uses 3m hidden units and is lossless, unlike FoldingNet's lossy reconstruction.The authors state that the theorem guarantees the decoding operation's legitimacy, while suitable parameterization and training lead to better performance.Source: §3 · p. 5

4. Experimental Results

Across the experiments, FoldingNet’s codewords support transfer classification and classification with few labels, while its decoder reconstructs point clouds through progressively deformed 2D manifolds. Visualizations also examine interpolation, clustering, and the folding decoder’s comparison with a fully connected decoder.

4.1. Visualization of the Training Process

  • During training, random 2D manifolds are warped, cut, squeezed, stretched, or attached into point-cloud surfaces.Table 2 illustrates the progression across training iterations.Source: §4.1 · p. 5, Table 2 · p. 6, §4.3 · p. 6

4.2. Point Cloud Interpolation

  • The experiments present both intra-class and inter-class point-cloud interpolations using a single auto-encoder for all shape categories.Table 3 assigns its first three rows to intra-class interpolation and its last three to inter-class interpolation.Source: §4.2 · p. 5, Table 3 · p. 6

4.3. Illustration of Point Cloud Clustering

  • Most ModelNet10 classes are separable in the codeword visualization, except dresser/nightstand and desk/table, pairs that are also difficult for humans to distinguish.The T-SNE embedding uses perplexity 50.Source: §4.3 · pp. 5–6, Figure 2 · p. 7

4.4. Transfer Classification Accuracy

  • FoldingNet outperforms the compared methods on MN40 transfer classification, while the [1] auto-encoder performs slightly better on MN10, where the comparison is inconclusive.The auto-encoder is trained on ShapeNet and codewords are evaluated with a linear SVM on ModelNet; Table 5 presents classification accuracy comparisons.Source: §4.4 · p. 7, Table 5 · p. 7

4.5. Semi-supervised Learning: What Happens when Labeled Data are Rare

  • With 1% of ModelNet training data (98 labeled examples), linear SVM test accuracy exceeds 55%; with 20%, it is close to 85%.The auto-encoder is trained on ShapeNet, and the linear SVM test set always includes all ModelNet test data.Source: §4.5 · pp. 7–8

4.6. Effectiveness of the Folding-Based Decoder

  • The folding decoder almost always has higher classification accuracy and lower reconstruction loss than the fully connected decoder, using about 7% as many parameters.The reported parameter counts are 1.05 × 10^6 for folding and 1.52 × 10^7 for the fully connected decoder.Source: §4.6 · p. 8, Figure 5 · p. 8

6. Supplementary: Proof of Theorem 3.1

The proof establishes that the encoder’s codeword is unchanged by permuting the input point order. Its pointwise and graph layers preserve that permutation structure, which global max-pooling then removes.

6. Supplementary: Proof of Theorem 3.1

  • The first encoder stage applies a shared pointwise function, so permuting input rows permutes its output rows in the same way.The proof states f1(PL) = Pf1(L).Source: §6 · p. 11
  • The graph stage is likewise row-equivariant: permuting input rows permutes the rows of Amax(X).The proof shows Amax(PX) = PAmax(X).Source: §6 · p. 11
  • Permuting input rows leaves the encoder’s codeword unchanged after graph-layer outputs are globally max-pooled.The graph layers’ outputs reorder with the input, while global max-pooling is invariant to row order.Source: §6 · p. 11

7. Supplementary: Proof of Theorem 3.2

The proof constructs a two-layer perceptron and codeword that reproduce target points from grid points under stated spacing and boundedness assumptions. This establishes the theorem’s existence claim by an explicit construction.

7. Supplementary: Proof of Theorem 3.2

  • The construction uses the vectorized target point-cloud matrix as the codeword and concatenates it with each grid point’s 2D coordinates.Each perceptron input contains the grid position and the target point coordinates stored in the codeword.Source: §7 · p. 11
  • The construction assumes grid points are spaced by at least 2δ on an integer-coordinate lattice bounded by M, with target coordinates satisfying |sij| ≤ 1.The grid coordinates are written as [(2βi + 1)δ, (2γi + 1)δ], with |βi| and |γi| smaller than M.Source: §7 · p. 11
  • A two-layer perceptron is constructed to map each concatenated grid-point input to its corresponding target point in 3D.Its hidden layer has 3m neurons, and its output layer produces three scalars.Source: §7 · pp. 11–12
  • The proof’s activation can be implemented with ReLU functions, which the authors also use in their experiments.The proof describes a nonlinear activation followed by a linear combination at the output.Source: §7 · p. 11

8. Supplementary: Decoder Variations

Decoder variations show that the regular 2D grid performs better than a 1D grid on both classification and reconstruction, while increasing folding stages or using a 3D grid adds little. Random sampling performs slightly worse.

8. Supplementary: Decoder Variations

  • A 2D grid outperforms a 1D grid on classification and reconstruction, while a 3D grid brings only marginal improvement.The authors relate the limited gain from a 3D grid to the datasets’ intrinsic dimensionality of 2, because their points are sampled from object surfaces.Source: §8 · p. 12
  • Increasing the number of folding operations does not significantly improve performance.The comparison varies the number of folding operations under the Section 4.6 experimental settings.Source: §8 · p. 12, Table 6 · p. 12
  • Uniformly random grid sampling gives slightly worse results, which the authors attribute to local density variation.The comparison distinguishes uniformly random from regularly spaced grids.Source: Table 6 · p. 12, §8 · p. 13

9. Supplementary: Folding by Deconvolution

The comparison finds worse reconstruction from the deconvolution implementation, despite comparable classification accuracy. The authors conjecture that its stronger smoothness constraint explains the reconstruction difference.

9. Supplementary: Folding by Deconvolution

  • Deconvolution produces worse reconstruction than FoldingNet, while classification accuracy is comparable.Table 7 compares two implementations of the folding operation.Source: §9 · p. 13, Table 7 · p. 13
  • The authors conjecture that deconvolution’s stronger smoothness constraint on the reconstructed surface accounts for its worse reconstruction.They contrast that constraint with FoldingNet’s implicit constraint from point-wise MLPs on a grid.Source: §9 · p. 13
  • FoldingNet’s operation maps each 2D grid point to 3D independently, whereas deconvolution uses grid structure to fuse information.The paper describes the folding operation as a per-point 2D-to-3D function.Source: §9 · p. 13

10. Supplementary: Robustness of the graph-based encoder

With 5% of ModelNet40 points randomly shifted within each cloud’s bounding box, FoldingNet’s graph-based encoder retains classification accuracy while removing its graph-based max-pooling layers degrades performance.

10. Supplementary: Robustness of the graph-based encoder

  • The noise experiment compares FoldingNet with an auto-encoder using the same folding-based decoder but without graph-based max-pooling layers.Five percent of points in each ModelNet40 cloud are shifted to other positions within the original bounding box.Source: §10 · p. 13
  • Removing graph-based max-pooling layers degrades performance by approximately 2% under random noise, while FoldingNet’s classification accuracy changes little.The comparison uses ModelNet40 clouds with 5% of points randomly shifted within each cloud’s original bounding box.Source: Figure 6 · p. 13, §10 · p. 13
  • The results support the authors’ conclusion that the graph-based encoder makes FoldingNet more robust to random noise.Source: §10 · p. 13

11. Supplementary: More Details on the Linear SVM Experiment on ModelNet10

On MN10, the linear SVM classification accuracy was 94.4%; the supplementary table lists the incorrectly classified models and their point-cloud representations.

11. Supplementary: More Details on the Linear SVM Experiment on ModelNet10

  • 94.4% classification accuracy was obtained on the MN10 dataset in Section 4.4.Source: §11 · p. 13
  • The authors note that many wrongly classified pairs are difficult for humans to distinguish as well.Source: §11 · p. 13
  • The table lists incorrectly classified models with their point clouds; “table →desk” denotes a table-labeled cloud classified as desk by the linear SVM.Source: §11 · p. 13
Loading 1712.07262v2…