PaperlayerView exact revision on arXiv

Source-linked AI summary

PointMixer: MLP-Mixer for Point Cloud Understanding

Jaesung Choe, Chunghyun Park, Francois Rameau, Jaesik Park, In So Kweon

arXiv:2111.11187v5cs.CV

TL;DR

Point clouds are sparse, unordered, and irregular, while MLP-like approaches have not been fully explored for point-cloud understanding. PointMixer uses channel-mixing MLPs and softmax-based mixing across point sets; experiments report compelling performance among Transformer-based studies across 3D tasks.

  • Problem

    Point clouds are sparse, unordered, and irregular, and MLP-like architectures have not yet been fully explored for point-cloud understanding.

    Source: §1 · pp. 1–2

  • Method

    PointMixer uses channel-mixing MLPs and softmax-based interactions to mix point features within and between point sets, including hierarchical sets.

    Source: §1 · p. 2, §3.3 · pp. 7–8

  • Results

    Extensive experiments report compelling or superior performance among Transformer-based studies across point-cloud tasks.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    The paper concludes that dense token-wise calculation is not essential for point-cloud understanding and presents symmetric encoder-decoder blocks for propagating responses through hierarchical point sets.

    Source: §5 · p. 14

    Inverse mapping may collect a variable number of points.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

MLP-Mixer has newly appeared as a new challenger against the realm of CNNs and transformer. Despite its simplicity compared to transformer, the concept of channel-mixing MLPs and token-mixing MLPs achieves noticeable performance in visual recognition tasks. Unlike images, point clouds are inherently sparse, unordered and irregular, which limits the direct use of MLP-Mixer for point cloud understanding. In this paper, we propose PointMixer, a universal point set operator that facilitates information sharing among unstructured 3D points. By simply replacing token-mixing MLPs with a softmax function, PointMixer can "mix" features within/between point sets. By doing so, PointMixer can be broadly used in the network as inter-set mixing, intra-set mixing, and pyramid mixing. Extensive experiments show the competitive or superior performance of PointMixer in semantic segmentation, classification, and point reconstruction against transformer-based methods.

1 Introduction

Point clouds are sparse, unordered, and irregular, and MLP-like architectures had not been fully explored for them. PointMixer extends MLP-like processing to mix point features across multiple sets.

1 Introduction

  • Sparse, unordered, and irregular point clouds make standard neural network architectures hardly applicable.Existing approaches include sparse convolution, graph neural, and point convolution networks.Source: §1 · p. 1
  • MLP-like architectures for point clouds had not yet been fully explored despite growing interest in MLP-Mixer.Prior work raised questions about MLP locality and whether self-attention is necessary.Source: §1 · p. 2
  • PointMixer mixes point features within grouped points, between point sets, and across hierarchical sets using channel-mixing MLPs.The paper introduces inter-set and hierarchical-set mixing beyond prior studies focused only on intra-set mixing [56].Source: §1 · p. 2
  • Extensive experiments report compelling performance among Transformer-based studies across various 3D tasks.The listed contributions include semantic segmentation, classification, and point reconstruction.Source: §1 · p. 2

2 Related Work

Prior point-cloud methods address irregular geometry through voxel- or point-based processing, while MLP-like architectures and Transformers offer other design choices. PointMixer unifies local aggregation and sampling operations in a symmetric operator for multiple point-cloud tasks.

2 Related Work

  • Voxel-based methods regularize point clouds into grids for efficient neighbor search, but voxelization loses geometric detail and can lead to inaccurate predictions.This motivates active study of point-based methods.Source: §2 · p. 3
  • PointNet++ uses kNN and farthest point sampling for locality and hierarchy, but its operator captures intra-set responses and its encoder-decoder is asymmetric.Its upsampling recomputes kNN and uses trilinear interpolation; the cited Transformer-based pyramid architecture has the same asymmetry.Source: §2 · p. 3
  • PointMixer unifies local feature aggregation, downsampling, and upsampling in a symmetric, learnable set operator for 3D perception tasks.The paper reports use for shape classification, semantic segmentation, and point reconstruction.Source: §2 · pp. 3–4
  • MLP-like vision work explored locality and token interactions, while MetaFormer emphasized general architecture over dense token-wise operators.MetaFormer replaces token-mixing operators with average pooling.Source: §2 · p. 4

3 Method

PointMixer adapts MLP-Mixer to unordered point sets by replacing token-mixing MLPs with a permutation-invariant set operator. It supports intra-set, inter-set, and hierarchical mixing, including symmetric downsampling and upsampling.

3.1 Preliminary

  • kNN maps each query point to its K closest neighbors; PointNet++ aggregates local set responses, while Point Transformer uses vector-subtraction attention.PointNet++ also uses Farthest Point Sampling to build feature hierarchy.Source: §3.1 · p. 5

3.2 MLP-Mixer as Point Set Operator

  • Vanilla MLP-Mixer is permutation-variant and processes fixed-size token sets, limiting its direct fit to irregular, unordered point clouds.PointMixer is designed to be permutation-invariant and to support learnable upsampling.Source: §3.2 · p. 6

3.3 PointMixer: Universal Point Set Operator

  • PointMixer scores neighboring features using channel-mixing and relative-position MLPs, then applies spatially normalized Softmax weights in a symmetric aggregation.The aggregation differs from average pooling in MLP-Mixer and max pooling in PointNet++.Source: §3.3 · pp. 6–7
  • The operator supports intra-set mixing, inter-set mixing through inverse neighborhood mappings, and hierarchical-set mixing across point-set levels.Inverse mappings can gather point responses from neighboring sets; hierarchical mixing applies to transition down and up layers.Source: §3.3 · pp. 7–8, Figure 3 · p. 8
  • PointMixer reuses the downsampling index map for symmetric upsampling, avoiding another kNN search and replacing trilinear interpolation with score-based response propagation.Conventional nearest-neighbor mappings are asymmetric because nearest-neighbor relations need not be reciprocal.Source: §3.3 · pp. 8–9

3.4 Network Architecture

  • The network uses transition-down, transition-up, and Mixer blocks in a deep pyramid that downsamples points for global features and upsamples for dense prediction.Classification uses fully connected and global-pooling layers; segmentation and reconstruction use MLP headers without pooling.Source: §3.4 · p. 9, Figure 5 · p. 10

4 Experiments

PointMixer performs strongly across semantic segmentation, point reconstruction, and shape classification, while ablations examine its mixing types, positional encoding, token-mixing MLPs, and symmetric architecture.

4.1 Semantic Segmentation

  • PointMixer achieves state-of-the-art performance on S3DIS Area 5 with 6.5M parameters, versus 7.8M for Point Transformer.The comparison is reported for semantic segmentation on the S3DIS Area 5 test dataset.Source: §4.1 · p. 10, Table 2 · p. 11

4.2 Point Cloud Reconstruction

  • PointMixer generalizes from synthetic ShapeNetPart training to unmet ScanNet real scenes and ICL-NUIM synthetic scenes, comparing favorably with Transformer- and MLP-based methods.The authors report that its performance gap over previous studies is larger on ScanNet and ICL-NUIM.Source: §4.2 · p. 11, Table 3 · p. 12

4.3 Shape Classification

  • PointMixer achieves the highest ModelNet40 mAcc, exceeding other methods by 0.8 mAcc with 1024 points and no normals.It uses 3.9M parameters, compared with 5.3M for Point Transformer and 15.2M for KPConv.Source: Table 4 · p. 12, §4.3 · p. 12

4.4 Ablation Study

  • Combining hierarchical-set and inter-set mixing improves results by 2.2 mIoU in segmentation and 0.02 Chamfer distance in reconstruction.The ablation also reports a 2.3-point mIoU and 0.01 Chamfer-distance degradation for the asymmetric architecture relative to the symmetric one; inter-set mixing further increases the gap.Source: §4.4 · pp. 13–14
  • Removing positional encoding causes a large performance drop, which the authors interpret as evidence that relative position encoding is necessary for 3D points.The positional-encoding ablation evaluates the PointMixer layer across its different uses.Source: §4.4 · p. 13
  • Replacing Softmax with token-mixing MLPs yields similar accuracy, but takes three days to train versus one day and uses 18% more parameters.The authors use this result to argue that dense token-wise interaction is not essential.Source: §4.4 · p. 13

5 Conclusion

PointMixer shares point responses across intra-set, inter-set, and hierarchical sets in a symmetric encoder-decoder architecture. Experiments report compelling or superior performance against Transformer-based studies.

5 Conclusion

  • PointMixer mixes point responses within, between, and across hierarchical point sets, regardless of set cardinality.The paper presents it as a universal point set operator.Source: §5 · p. 14
  • A symmetric encoder-decoder propagates information through hierarchical point sets.The conclusion contrasts this design with heuristic alternatives by emphasizing the symmetric architecture.Source: §5 · p. 14
  • Experiments validate compelling or superior performance relative to Transformer-based studies, while the authors argue dense token-wise calculation is not essential for point-cloud processing.They emphasize sharing information across varied point sets instead.Source: §5 · p. 14

Supplementary Material

The supplementary material covers efficiency, receptive fields, token-mixing MLP cardinality, and task-specific training details.

Supplementary Material

A Efficiency analysis

The efficiency analysis compares point-set layers on ModelNet40 under matched downsampling, without inter-set mixing. PointMixer uses less GPU memory than Point Transformer but is slower than PointNet++ because both PointMixer and Point Transformer use Softmax.

A Efficiency analysis

  • The comparison measures mAcc, OA, average inference time per object, and peak GPU memory on ModelNet40.PointNet++ and Point Transformer are reimplemented with the same residual-block count and training configuration as PointMixer.Source: A Efficiency analysis · p. 15
  • PointMixer uses 8MB less GPU memory than Point Transformer, while both are slower than PointNet++ because they use Softmax.The comparison measures inference on ModelNet40 and aligns downsampling layers while omitting inter-set mixing.Source: Table 6 · p. 15, A Efficiency analysis · p. 15
  • With intra-set mixing, PointMixer exceeds PointNet++ by 0.3 mAcc and 0.1 OA.The authors attribute the result to score-based aggregation embedding local responses more effectively than PointNet++ pooling.Source: A Efficiency analysis · p. 15

B Point receptive field analysis

PointMixer expands receptive fields by mixing features across neighboring sets and by reusing downsampling neighborhoods during upsampling. These choices reach larger contexts than the compared methods, without additional upsampling computation.

B Point receptive field analysis

  • Intra-set mixing is limited to k closest neighbors, while combining it with inter-set mixing propagates responses to variable numbers of neighbors in adjacent sets.The passage says this combined use is consistently supported by Fig. 8.Source: B Point receptive field analysis · pp. 16–17
  • Reusing the downsampling kNN graph lets PointMixer’s upsampling reach larger receptive fields without additional computational cost.PointNet++ and Point Transformer typically interpolate three nearest neighbors, versus k = 16 in downsampling.Source: B Point receptive field analysis · p. 17
  • PointMixer has larger receptive fields than the compared methods, which the authors identify as a possible reason for its superior dense-prediction performance.The passage presents this as a possible explanation, not a demonstrated causal result.Source: B Point receptive field analysis · p. 17

C Limited cardinality issues in token-mixing MLPs

Token-mixing MLPs require a fixed number of input points, which prevents them from operating on inter-set mixing when set cardinality varies.

C Limited cardinality issues in token-mixing MLPs

  • Unlike image pixels with systematically eight adjacent pixels, a point can have an arbitrary number of neighbors.Shared MLPs can handle variable input lengths, but vanilla MLP-Mixer token-mixing MLPs limit the number of points processed.Source: C Limited cardinality issues in… · p. 17
  • Channel-mixing MLPs require predefined channel dimensionality, whereas token-mixing MLPs switch the channel and spatial axes.That axis switch makes the number of input tokens predefined.Source: C Limited cardinality issues in… · p. 17
  • Because token-mixing MLPs require a predefined number of input tokens, they cannot perform inter-set mixing when point-set cardinality varies.Source: C Limited cardinality issues in… · p. 17

D Training details

The paper reports separate training configurations for semantic segmentation, point-cloud reconstruction, and object classification.

D Training details

  • Semantic segmentation used batches of 40K points, a learning rate initially set to 0.1, and 60 training epochs.The learning rate was reduced tenfold at epochs 40 and 50; training took 44 hours on two NVIDIA 1080-Ti GPUs.Source: D Training details · p. 17
  • Point-cloud reconstruction used batch size 4 and the semantic-segmentation training conditions, with the output dimensionality changed to 3 for [x, y, z].Source: D Training details · pp. 17–18
  • Object classification used batch size 32, 300 epochs, cosine-annealing learning-rate decay, and MLP headers with dropout ratio 0.5.The same SGD optimizer as in semantic segmentation was used.Source: D Training details · p. 18

E Ablation study (rebuttal)

The rebuttal ablations report that adding inter- and hierarchical-set mixing improves existing backbones, and that the softmax mixing scheme compares favorably with max and vector-attention alternatives. The authors frame these operations as complementary set mixing and a route to a symmetric network.

E Ablation study (rebuttal)

  • The ablation’s stated aims are to improve expressiveness through complementary intra-, inter-, and hierarchical-set operations, develop a universal operator, and design a symmetric network.Source: E Ablation study (rebuttal) · p. 18
  • Inter- and hierarchical-set mixing consistently improves PointNet++ and Point Transformer regardless of block design; softmax also appears more effective than max or vector-attention.Table 7 covers S3DIS semantic-segmentation results; all methods were trained for 30 epochs.Source: E Ablation study (rebuttal) · p. 18, Table 7 · p. 19
  • The authors align PointMixer’s direction with MLP-Mixer variants that modify or remove token-mixing MLPs.The cited examples include CycleMLP’s cyclic sampling and AS-MLP’s Axial Shift operations.Source: E Ablation study (rebuttal) · pp. 18–19
  • Point Transformer results on S3DIS and ModelNet40 were produced by the authors’ own training with its official code because pretrained weights were unavailable.Source: E Ablation study (rebuttal) · p. 18
  • PointMixer’s softmax is presented as a projection across token dimensions, while its use of the bidirectional kNN structure extends mixing beyond the directional intra-set operation.The authors support this design choice with permutation-invariant-operator analysis and experiments.Source: E Ablation study (rebuttal) · p. 19
Loading 2111.11187v5…