PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions

Jiachen Sun, Qingzhao Zhang, Bhavya Kailkhura, Zhiding Yu, Chaowei Xiao, Z. Morley Mao

arXiv:2201.12296v1cs.LGcs.AIcs.CV

TL;DR

Point-cloud recognition research has largely assessed clean-data performance or adversarial robustness, leaving common corruption robustness underexplored despite safety-critical applications. After evaluating data augmentation and test-time adaptation methods, the paper combines PointCutMix-R and TENT, achieving the lowest average error rate of 15.4%. Error rate is 3× larger on ModelNet40-C than on ModelNet40 for state-of-the-art point-cloud recognition models.

  • Problem

    Point-cloud recognition research has largely assessed clean-data performance or adversarial robustness, leaving common corruption robustness underexplored despite safety-critical applications.

    Source: §1 · p. 1

  • Method

    After evaluating data augmentation and test-time adaptation methods, the paper combines PointCutMix-R and TENT, achieving the lowest average error rate of 15.4%.

    Source: §1 · p. 2

  • Results

    Error rate is 3× larger on ModelNet40-C than on ModelNet40 for state-of-the-art point-cloud recognition models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis identifies architecture and augmentation effects across corruptions, with Transformer architectures appearing promising and test-time adaptation showing potential as a robustness strategy.

    Source: §6 · p. 8

    The benchmark covers point-cloud recognition; extending it to segmentation and object detection is left for future work.

    Source: §6 · p. 8

Abstract

from arXiv · show

Deep neural networks on 3D point cloud data have been widely used in the real world, especially in safety-critical applications. However, their robustness against corruptions is less studied. In this paper, we present ModelNet40-C, the first comprehensive benchmark on 3D point cloud corruption robustness, consisting of 15 common and realistic corruptions. Our evaluation shows a significant gap between the performances on ModelNet40 and ModelNet40-C for state-of-the-art (SOTA) models. To reduce the gap, we propose a simple but effective method by combining PointCutMix-R and TENT after evaluating a wide range of augmentation and test-time adaptation strategies. We identify a number of critical insights for future studies on corruption robustness in point cloud recognition. For instance, we unveil that Transformer-based architectures with proper training recipes achieve the strongest robustness. We hope our in-depth analysis will motivate the development of robust training strategies or architecture designs in the 3D point cloud domain. Our codebase and dataset are included in https://github.com/jiachens/ModelNet40-C

1. Introduction

Clean-data benchmarks offer only a limited view of point-cloud recognition, while common corruption robustness remains important for safety-critical applications. The paper introduces ModelNet40-C, evaluates models and mitigation strategies, and reports substantial performance degradation on corrupted inputs.

1. Introduction

  • Error rate is 3× larger on ModelNet40-C than on ModelNet40, with evaluation also finding nearly 3× gaps across six representative models.The paper reports substantial room to improve robustness against common corruptions.Source: §1 · p. 2
  • ModelNet40-C provides 15 corruption types at five severity levels, grouped into density, noise, and transformation categories.The benchmark was designed to cover real-world point-cloud distortions while preserving original semantics.Source: §1 · pp. 1–2
  • Combining PointCutMix-R with TENT achieves an average error rate of 15.4% after evaluation of augmentation and test-time adaptation strategies.The study examines 3,180 configurations across feasible architecture and methodology combinations.Source: §1 · p. 2
  • Occlusion, rotation, and background noise pose significant challenges for most point-cloud recognition models.These are identified as especially challenging corruption patterns in the paper's summary of findings.Source: §1 · p. 2
  • Architecture vulnerabilities and the benefits of data augmentation vary with corruption type and align with the corresponding design choices.The paper presents these patterns as insights for future corruption-robustness research.Source: §1 · p. 2
  • Batch normalization and TENT test-time adaptations improve corruption robustness, particularly for occlusions and rotations.The reported benefit is especially pronounced for these hard corruptions.Source: §1 · p. 2
  • The authors aim for the benchmark and analysis to inform robust architectures and training- and test-time strategies on ModelNet40-C.They position the benchmark as a resource for developing corruption-resistant point-cloud models.Source: §1 · p. 2

2. Related Work

Prior point-cloud robustness work has largely focused on adversarial examples, leaving common corruption robustness without a systematic study. The benchmark groups its 15 corruptions into density, noise, and transformation patterns, designed to represent varied real-world distortions while preserving point-cloud semantics.

2. Related Work

  • Prior 3D point-cloud robustness research largely addressed adversarial threats, while common corruption robustness received little attention.The benchmark design is motivated by the gap and by the need to preserve semantics, reflect realistic corruption, and cover diverse patterns.Source: §2 · p. 3, §3 · p. 3

3.1. Density Corruption Patterns

  • Five density corruptions model missing or uneven points through ray-traced occlusion, local-density changes, and cluster removal.The set comprises Occlusion, LiDAR, Local Density Inc, Local Density Dec, and Cutout; LiDAR also adds vertically line-styled patterns.Source: §3.1 · p. 3

3.2. Noise Corruption Patterns

  • Five noise corruptions cover distributional perturbations, deterministic point changes, nearby added points, and random background points.The set comprises Uniform, Gaussian, Impulse, Upsampling, and Background; the paper notes that point-cloud noise has a larger manipulation space because point counts are adjustable.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Transformation Corruptions Patterns

  • Transformation corruptions include mild rotations of at most 15°, xy-plane shear, and nonlinear deformations using FFD and RBF variants.Translation and scale are excluded because normalization can restore them; the RBF-based set includes RBF and inverse RBF.Source: §3.3 · p. 4

4. ModelNet40-C Robustness Benchmark

ModelNet40-C uses ModelNet40 as a base for evaluating point-cloud recognition under common corruptions, with multiple severities and corruption-specific metrics. It is intended as a test-time robustness benchmark, not for training-time optimization.

4. ModelNet40-C Robustness Benchmark

  • ModelNet40-C is constructed from ModelNet40, a clean CAD-derived dataset of 12,308 point clouds across 40 classes, to study performance under common test-time corruptions.The designed corruptions are also described as applicable to datasets such as ShapeNet.Source: §4 · p. 4
  • Each corruption has five severity levels; five view angles are used for LiDAR and occlusion because their corruption severity is difficult to quantify.The benchmark is 75 times larger than the ModelNet40 validation set and is intended for testing, not training.Source: §4 · pp. 4–5
  • Benchmarking uses error rate (ER) and class-wise mean error rate (mER), with ERclean denoting clean-set error and ERs,c denoting error for corruption c at severity s.Table 1 reports ERclean across models and training strategies.Source: §4 · p. 5, Table 1 · p. 5
  • ModelNet40-C is intended to assess general robustness across real-world scenarios; similar corruptions may be used for augmentation if explicitly disclosed.The benchmark itself should be used at test time rather than for training-time optimization.Source: §4 · p. 5

5. Experiments

Experiments benchmark six architectures on ModelNet40-C and evaluate data augmentation and test-time adaptation. Corruptions degrade recognition, while combining PointCutMix-R and TENT yields the strongest overall robustness among the evaluated mitigation methods.

5.1. Comparison Among Corruption Types

  • ERc increased 17.6%–89.7% over ERclean across corruptions; Occlusion and LiDAR averaged 55.0% and 71.6% error, respectively.Even small rotations (≤15°) produced error rates of 18.1%–36.8%; Impulse and Background errors averaged 33.2% and 48.0%.Source: §5.1 · p. 6

5.2. Comparison Among Model Architectures

  • Architecture robustness is corruption-specific: PointNet handles density well, while PointNet++ leads overall standard-training ERcor at 23.6%.PointNet++ and RSCNN resist Background corruption, while PCT reports ER of 13.5% on transformation corruptions.Source: §5.2 · pp. 6–7

5.3. Data Augmentation Strategies

  • No single augmentation covers every corruption: PointCutMix-R is strongest on noise, PointMixup on transformations, and RSMix on density.Their mixing patterns respectively resemble noise, smooth transformations, and isolated density changes.Source: §5.3 · p. 7
  • Adversarial training does not outperform other strategies overall, though point-shifting attacks improve robustness on noise corruptions.The authors call for methods that improve both adversarial and corruption robustness.Source: §5.3 · p. 8
  • Data augmentation generally improves corruption robustness, and PointCutMix-R achieves the best overall augmentation result (ERcorrup=18.7%), especially at high severity.The paper reports that augmentation improves overall corruption robustness from 19.6% to 28.4%.Source: §5.3 · pp. 7–8

5.4. Test-Time Adaptation Methods

  • Test-time adaptation is less effective overall than augmentation, but TENT beats the best augmentation on Occlusion, LiDAR, and Rotation errors by 6.7%, 1.9%, and 7.9%.The authors attribute the weaker overall adaptation results to 3D corruptions changing both point counts and positions, creating a large distribution shift.Source: §5.4 · p. 8
  • 14.7% improvement in corruption robustness came from combining PointCutMix-R with TENT, the best training- and test-time methods by overall ERcor.The combination is evaluated in Table 5.Source: §5.4 · p. 8, Table 5 · p. 8

6. Discussion and Conclusion

The paper concludes that ModelNet40-C reveals substantial corruption-related degradation and offers insights into architecture and training choices for point-cloud robustness. It also identifies test-time adaptation as a promising strategy and points to segmentation and detection as future benchmark extensions.

6. Discussion and Conclusion

  • The authors plan to extend the benchmark to point-cloud segmentation and object detection.They report that performance discrepancies across models and corruptions are much larger than for 2D architectures.Source: §6 · p. 8
  • The authors present ModelNet40-C as a comprehensive corruption-robustness benchmark and report massive performance degradation for six representative models.They aim for the benchmark to benefit future research on robust 3D point-cloud models.Source: §6 · p. 8
  • Transformer architectures appear promising for improving robustness in 3D vision tasks.The conclusion also reports insights about how architecture and data-augmentation designs affect robustness across corruptions.Source: §6 · p. 8
  • The study identifies test-time adaptation as a potential robustness strategy for point-cloud recognition.Source: §6 · p. 8

A. ModelNet40-C

ModelNet40-C corruption generation uses distinct procedures for occlusion and LiDAR, density and noise changes, geometric transformations, and deformation-based corruptions.

A. ModelNet40-C

  • Occlusion and LiDAR corruptions use five viewing angles; LiDAR additionally renders points in a vertically multi-line pattern.The viewing angles are 0°, 72°, 144°, 216°, and 288°, with elevation angles randomly sampled from 30°–60°.Source: A. ModelNet40-C · p. 12, Figure 5 · p. 12
  • Local Density Inc and Dec alter the density around sampled anchor points, while Cutout removes each anchor’s full kNN subset.Cutout uses k = 50 to simulate scanning-device sensor limitations.Source: A. ModelNet40-C · p. 12
  • Gaussian and Uniform noise vary by severity, while Background, Impulse, and Upsampling noise add points or perturbations using specified sampling procedures.Impulse and Upsampling perturbations are bounded by ℓ∞ = 0.05.Source: A. ModelNet40-C · p. 12
  • Rotation and Shear use relatively small transformations because larger ones affect human perception of object class.Source: A. ModelNet40-C · p. 12
  • FFD, RBF, and Inv RBF deformations use 125 control points, with deformation distance set by severity and directions randomized in 3D.Source: A. ModelNet40-C · p. 12

B.1. Data Augmentation Setups

The experimental setup specifies label mixing for mixing-based augmentations and a point-shifting adversarial-training procedure.

B.1. Data Augmentation Setups

  • Mixing-based augmentation uses λ to weight two samples and their virtual label vector.Source: §B.1 · p. 12
  • The experiments set λ = 0.5, with the mixed label formed from the two point clouds’ label vectors.The cited prior work is reported as support for this setting: (Zhang et al., 2021).Source: §B.1 · pp. 12–13
  • Adversarial training uses point shifting in the inner maximization, with ϵ = 0.05, α = 0.01, and seven PGD steps.Source: §B.1 · p. 13

B.2. Evaluation Results

The evaluation reports results across 3,180 data points, using confusion matrices and comparisons of augmentation and test-time adaptation strategies. Figures also present class-wise mean error rates for both strategy families.

B.2. Evaluation Results

Loading 2201.12296v1…