PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking and Analyzing Point Cloud Classification under Corruptions

Jiawei Ren, Liang Pan, Ziwei Liu

arXiv:2202.03377v3cs.CV

TL;DR

Point-cloud corruption robustness lacks a standard, comprehensive benchmark despite its importance in real-world and safety-critical applications. The paper builds ModelNet-C, benchmarks representative classifiers, and proposes robustness techniques; its results show improving clean-data performance alongside increasing or variable corruption vulnerability.

  • Problem

    Point-cloud classifiers lack a standard, comprehensive way to evaluate robustness to corruptions that arise in complex scenes and from sensor inaccuracy.

    Source: §1 · pp. 1–2

  • Method

    The paper creates ModelNet-C from seven atomic corruptions at five severity levels, benchmarks 14 methods, and proposes RPC and WOLFMix.

    Source: §1 · p. 2

  • Results

    Clean ModelNet40 performance improves over time while corruption robustness varies substantially, and RPC achieves the least mCE on ModelNet-C with comparable clean overall accuracy.

    Source: §1 · p. 2, §4 · p. 5

  • Takeaways & Limitations

    WOLFMix improves robustness over existing augmentation techniques, but augmentation gains differ across architectures, making architecture–augmentation synergy a research consideration.

    Source: §1 · p. 3

    The atomic corruptions do not seamlessly simulate real-world corruptions, and the study excludes noisy translation and random permutation.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

3D perception, especially point cloud classification, has achieved substantial progress. However, in real-world deployment, point cloud corruptions are inevitable due to the scene complexity, sensor inaccuracy, and processing imprecision. In this work, we aim to rigorously benchmark and analyze point cloud classification under corruptions. To conduct a systematic investigation, we first provide a taxonomy of common 3D corruptions and identify the atomic corruptions. Then, we perform a comprehensive evaluation on a wide range of representative point cloud models to understand their robustness and generalizability. Our benchmark results show that although point cloud classification performance improves over time, the state-of-the-art methods are on the verge of being less robust. Based on the obtained observations, we propose several effective techniques to enhance point cloud classifier robustness. We hope our comprehensive benchmark, in-depth analysis, and proposed techniques could spark future research in robust 3D perception.

1. Introduction

The paper addresses the lack of a standard, comprehensive benchmark for point-cloud robustness under natural corruptions. It introduces ModelNet-C, benchmarks representative methods, and proposes RPC and WOLFMix to improve robustness.

1. Introduction

  • Existing robustness evaluations use different corruption selections and protocols, while sim-to-real tests mix natural corruptions with domain gaps and adversarial tests do not measure natural-corruption robustness.These approaches leave point-cloud classification without a standard, comprehensive corruption benchmark.Source: §1 · pp. 1–2
  • Clean ModelNet40 performance improves over time, but state-of-the-art methods show substantial corruption-robustness variation and may become less robust.Figure 1 contrasts chronological overall accuracy with mean Corruption Error (mCE).Source: Figure 1 · p. 1, §1 · p. 2
  • ModelNet-C applies seven atomic corruptions to ModelNet40 validation data, uses five severity levels and mCE, and benchmarks 14 methods.The benchmark includes nine architectures, three augmentations, and two pretrains; adversarial attacks are excluded because the focus is natural-world robustness.Source: §1 · p. 2
  • RPC achieves the least mCE on ModelNet-C while matching state-of-the-art overall accuracy on clean ModelNet40.The paper presents RPC as a robust point-cloud classifier architecture.Source: §1 · p. 2
  • WOLFMix combines deformation-based and mix-based augmentation, achieves the best robustness among existing augmentation techniques, and transfers unevenly across architectures.The authors identify architecture–augmentation synergy as a future design consideration.Source: §1 · p. 3

2. Related Works

Related work spans diverse point-cloud architectures, augmentation and pretraining methods, specialized robustness approaches, and robustness benchmarks for image classification.

Point Cloud Classification

  • Point-cloud classifiers include MLP, convolutional, graph, and transformer designs, alongside mix-based, deformation-based, and auto-augmentations and self-supervised pretraining.The cited pretraining tasks include occlusion reconstruction and mask inpainting, which provide better classification performance than random initialization.Source: §2 · p. 3

Robustness in Point Cloud

  • Robustness methods include invariant features, adaptive sampling, and adversarial defenses, while reported trade-offs include below-state-of-the-art clean performance and fixed point counts.Prior work also found basic augmentations generalize poorly to unseen corruptions, whereas mixing and local deformation can improve robustness.Source: §2 · p. 3

Robustness Benchmarks in Image Classification

  • Image robustness benchmarks cover simulated corruptions, natural distribution shifts, varied viewpoints and backgrounds, and natural adversarial or abstract renditions.Examples include ImageNet-C, ObjectNet, ImageNetV2, ImageNet-A, and ImageNet-R.Source: §2 · p. 3

3. Corruptions Taxonomy and Test Suite

The section abstracts real-world point cloud corruption into seven atomic operations and builds ModelNet-C as a controlled test suite. It defines DGCNN-normalized corruption metrics and a consistent evaluation protocol.

3.1. Corruptions Taxonomy

  • Real-world corruptions span object-, sensor-, and processing-level sources, abstracted into seven atomic operations for controllable analysis despite imperfect simulation.The taxonomy excludes noisy translation and random permutation because point-cloud normalization and permutation invariance are basic properties of recent classifiers.Source: §3.1 · pp. 3–4

3.2. ModelNet-C: A Robustness Test Suite

  • ModelNet-C corrupts ModelNet40’s test set with seven atomic corruptions at five severities while keeping training corruption-free; rotations are restricted to small angles.The benchmark uses the ModelNet40 training set, and arbitrary SO(3) rotation robustness is outside this work’s scope.Source: §3.2 · p. 4

3.3. Evaluation Metrics

  • mCE averages corruption errors normalized to DGCNN, while RmCE measures clean-to-corrupted performance drop relative to DGCNN.The benchmark averages over seven corruption types; RmCE uses clean-test accuracy in its comparison.Source: §3.3 · pp. 4–5

3.4. Evaluation Protocol

  • The protocol uses DGCNN settings, fixed training-time point sampling, no inference voting, and the clean-test-best model; the benchmark corruptions are barred during training.Training uses anisotropic scaling and translation augmentations, with training and test scaling ranges not overlapping.Source: §3.4 · p. 5

4. Systematic Benchmarking

The benchmark covers representative architecture, pretraining, and augmentation methods. Clean-accuracy progress does not ensure stable corruption robustness, and recent augmentations substantially improve robustness.

Implementation Details

  • The benchmark evaluates 14 methods: nine architectures, two pretraining approaches, and three augmentation methods.The architectures include PointNet, PointNet++, DGCNN, RSCNN, SimpleView, GDANet, CurveNet, PAConv, and PCT.Source: §4 · p. 5

Architecture Design

  • Table 2 summarizes architecture design choices, while Figure 5 schematizes recurring components without faithfully detailing every architecture.The schematic shows local operations, advanced grouping, and featurization before classification, or a multi-view-image route through CNN backbones.Source: Table 2 · p. 5, Figure 5 · p. 5

Main Results

  • Architecture performance improves and saturates around 0.94, but mCE varies widely; pretraining has mixed overall effects, while recent augmentations substantially improve robustness.The benchmark reports mCE for architectures, pretraining, and augmentations in Tables 3–5, respectively.Source: §4 · p. 5

5. Comprehensive Analysis

The analysis compares architecture choices, self-supervised pretraining, and augmentations for corruption robustness. It finds that their effects vary by corruption and model, with trade-offs across choices.

5.1. Architecture Design

  • Local aggregations improve robustness overall; k-NN generally outperforms ball-query, especially for local point drops, while ball-query better handles added global outliers.Without local aggregation, PointNet has the highest mCE, but is strongest on jitter and global point drops.Source: §5.1 · p. 6
  • Frequency and curve grouping both improve robustness, but add training and testing time; their effects also reflect a trade-off across corruption frequencies.The analysis interprets locally grouped features as low-frequency and curve-grouped features as high-frequency.Source: §5.1 · p. 6
  • Adaptive-kernel methods are more corruption-sensitive than conventional operators, while self-attention improves robustness in several areas, particularly global point drops.The authors speculate that self-attention’s global perspective helps it capture non-local relations.Source: §5.1 · p. 6
  • Projecting 3D points into 2D has mixed effects: it reduces jitter and local-addition effects but is vulnerable to global point drops.The authors conclude that 3D cues are more straightforward and preferable for robust point-cloud classification.Source: §5.1 · p. 7

5.2. Self-supervised Pretraining

  • Self-supervised pretraining transfers robustness to specific corruptions: Point-BERT helps when local parts are missing, while OcCo improves rotation, jitter, and global-addition robustness.The pretraining tasks use block-wise point masking for Point-BERT and viewpoint-based self-occlusion masking with rotations for OcCo.Source: §5.2 · p. 7

5.3. Augmentation Method

  • Mixing and deformation augmentations improve corruption robustness, but mix strategies can increase vulnerability to scaling; PointWOLF consistently improves results across evaluated corruptions.The evaluation excludes training on the benchmark corruptions to preserve the out-of-distribution setting.Source: §5.3 · p. 7, Table 5 · p. 8

6. Boosting Corruption Robustness

The section proposes RPC, a robustness-focused architecture, and WOLFMix, an augmentation combining deformation and mixing. RPC has the best reported mCE among SoTA methods, while augmentation gains vary across architectures.

RPC: A Robust Point Cloud Classifier

  • RPC achieves the best mCE among SoTA methods, validating the architecture choices and providing a strong baseline for robustness research.RPC combines 3D representation, k-NN, frequency grouping, and self-attention.Source: §6 · p. 8

WOLFMix: A Strong Augmentation Strategy

  • WOLFMix deforms objects before rigidly mixing them and mixing their ground-truth labels, yielding substantial robustness gains over PointWOLF or RSMix alone.Figure 7 illustrates the sequence of local deformation followed by rigid mixing.Source: §6 · p. 8, Figure 7 · p. 8

Synergy between Architecture and Augmentation

  • WOLFMix’s robustness gains vary by architecture; GDANet with WOLFMix achieves the best reported result, 0.571 mCE.RPC has lower standalone mCE, but its WOLFMix gain is smaller than those for DGCNN, PCT, and GDANet; the authors suggest studying architecture–augmentation synergy.Source: §6 · p. 8

7. Conclusion

The conclusion presents ModelNet-C as a comprehensive corruption test suite and reports benchmark analysis leading to RPC and WOLFMix as robustness strategies. The accompanying appendix passages specify several corruption constructions and parameter settings.

7. Conclusion

  • The paper establishes ModelNet-C, benchmarks representative classifiers, and proposes RPC and WOLFMix to improve corruption robustness.The authors report that clean-data state-of-the-art methods are becoming less robust to random real-world corruptions.Source: §7 · p. 9

A.1. Jitter

  • Jitter adds Gaussian noise independently to each coordinate, with σ values from 0.01 to 0.05 across five severity levels.Source: §A.1 · p. 13

A.2. Scale

  • Scale corruption independently samples each axis scale from U(1/S, S), with S from 1.6 to 2.0, then renormalizes the cloud to a unit sphere.Source: §A.2 · p. 13

A.3. Rotate

  • Rotation samples each X-Y-Z Euler angle from −θ to θ across five levels, but does not guarantee uniform SO(3) sampling.The passage describes this sampling as sufficient to cover a range of rotation variations.Source: §A.3 · p. 13

A.4. Drop Global; A.5. Drop Local

  • Global dropping shuffles points and removes a fraction of the 1,024 points, while local dropping removes one or more nearest-neighbor clusters.Global-drop fractions range from 0.25 to 0.75; local-drop totals range from 100 to 500 points.Source: §A.4 · p. 13, §A.5 · p. 13

A.6. Add Global; A.7. Add Local

  • Global addition samples points uniformly inside a unit sphere, while local addition forms normally distributed clusters around selected local centers.Both additions use five levels, with 10–50 global points or 100–500 local points.Source: §A.6 · p. 13, §A.7 · p. 13

B.1.1. DETAILED ARCHITECTURE

The appendix points to a detailed RPC architecture and specifies selected hyperparameters for its local operation and frequency grouping.

B.1.1. DETAILED ARCHITECTURE

  • Figure 9 presents the detailed architecture of RPC, designed from the benchmark conclusions and using existing point-cloud classifier building blocks.Source: §B.1.1 · p. 13, Figure 9 · p. 15

B.1.2. HYPER-PARAMETERS

  • RPC uses 30 neighbors for k-NN; frequency grouping follows GDANet defaults and uses 256 points per frequency component.Source: §B.1.2 · p. 15

B.1.3. TRAINING

The training setup specifies optimization and augmentation settings, including separate schedules for the base training and WOLFMix experiments. The additional evaluation reports a strong correlation between ModelNet-C mCE and ScanObjectNN OA, unlike ModelNet40 OA.

B.1.3. TRAINING

  • Training runs for 250 epochs with batch size 32, using momentum-0.9 SGD and cosine annealing from 1e-2 to 1e-4.Source: §B.1.3 · p. 15

B.2. WOLFMix

  • WOLFMix uses PointWOLF deformation and RSMix mixing settings, with 4 farthest-point-sampled anchors, RSMix probability 0.5, k-NN 20, and 500 training epochs; AugTune is omitted.The deformation settings include bandwidth 0.5, maximum rotation 10 degrees, scaling 3, and translation 0.25; the maximum number of point modifications is 512.Source: §B.2 · p. 15

C.1. Correlation between ModelNet-C mCE and ScanObjectNN OA

  • ModelNet-C mCE strongly correlates with ScanObjectNN OA, while ModelNet40 OA has nearly no correlation.The reported ScanObjectNN OA is lower than Uy et al. (2019), whose training used random rotation and per-point jitter; this work follows the DGCNN protocol.Source: §C.1 · p. 15

C.2. More results of Wolfmix

The additional results show different robustness outcomes across augmentation and architecture choices. PointASNL performs well on noisy jittering and overall mCE, while Vector Neurons excel on rotational corruption but underperform on other corruption types.

C.2. More results of Wolfmix

  • Unequal WolfMix benefits across RSCNN, SimpleView, and PointNet++ motivate further study of architecture–augmentation synergy.Source: §C.2 · p. 15

C.3. Evaluation on specific techniques proposed for enhancing robustness

  • PointASNL requires a fixed number of points, so the authors evaluated it with additional manual effort; it performs strongly on noisy jittering and has competitive overall mCE.Source: Table 8 · p. 16, §C.3 · p. 16

C.4. Evaluation on works designed for rotation robustness

  • Vector Neurons show impressive robustness to rotational corruption but underperform on other corruption types; arbitrary SO(3) rotation robustness is outside the benchmark scope.Source: §C.4 · p. 16

D. Full Results

The full-results section presents OA and Relative mCE results in separate tables and specifies conventions for reading their entries.

D. Full Results

Related papers

Loading 2202.03377v3…