Source-linked AI summary
Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data
Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, Sai-Kit Yeung
TL;DR
The paper asks whether high CAD-benchmark accuracy reflects solved point-cloud classification when objects appear in realistic scenes. It introduces ScanObjectNN, benchmarks synthetic and real-world classification, and proposes a background-aware network with joint classification and segmentation. The benchmark finds real-world classification difficult, while the proposed network improves results on synthetic and real-world datasets.
Problem
Point-cloud classification remains insufficiently tested under real-world conditions, where scans contain background clutter and partial objects despite high CAD-benchmark accuracy.
Method
The paper introduces ScanObjectNN, benchmarks existing methods across synthetic and real-world data, and proposes a network jointly learning classification and segmentation.
Results
Real-world classification is challenging: benchmarked state-of-the-art methods reach up to 78.5% accuracy on the hardest PB T50 RS variant.
Takeaways & Limitations
Real-world point-cloud classification requires addressing domain generalization, partial observations, and background clutter beyond clean CAD benchmarks.
Takeaways & Limitations
The BGA method requires object masks and background in the data and has a failure case on background-free ModelNet40 objects.
Abstract
from arXiv · showhide
Deep learning techniques for point cloud data have demonstrated great potentials in solving classical problems in 3D computer vision such as 3D object classification and segmentation. Several recent 3D object classification methods have reported state-of-the-art performance on CAD model datasets such as ModelNet40 with high accuracy (~92%). Despite such impressive results, in this paper, we argue that object classification is still a challenging task when objects are framed with real-world settings. To prove this, we introduce ScanObjectNN, a new real-world point cloud object dataset based on scanned indoor scene data. From our comprehensive benchmark, we show that our dataset poses great challenges to existing point cloud classification techniques as objects from real-world scans are often cluttered with background and/or are partial due to occlusions. We identify three key open problems for point cloud object classification, and propose new point cloud classification neural networks that achieve state-of-the-art performance on classifying objects with cluttered background. Our dataset and code are publicly available in our project page https://hkust-vgd.github.io/scanobjectnn/.
1. Introduction
Point-cloud classification appears near saturation on synthetic CAD data but remains challenging in real-world scenes. The paper addresses this gap with ScanObjectNN, a benchmark, and a network that jointly learns classification and segmentation.
- Synthetic-data accuracy is approaching saturation, motivating a reassessment of whether 3D object classification is solved.
- The authors benchmark classification methods on synthetic ModelNet40 and introduce ScanObjectNN from scanned indoor scenes for real-world evaluation.ScanObjectNN is derived from SceneNN and ScanNet scene meshes.
- Real-world classification remains challenging because objects may contain cluttered background, be partial, or differ across synthetic and real-world domains.The benchmark identifies poor cross-domain generalization, occlusion and reconstruction-related partiality, and background handling as open issues.
- The proposed network jointly learns classification and segmentation to classify objects observed with background.
- The paper contributes a new real-world object dataset for point-cloud classification.
- It also provides a comprehensive benchmark spanning synthetic and real-world point-cloud data.
2. Related Works
Prior point-cloud classification research relies heavily on clean CAD benchmarks, where methods perform similarly at high accuracy. Real-world datasets instead introduce partiality, clutter, and limited scale that better expose practical challenges.
- Object Classification on Point Clouds: Point-cloud classification methods have adapted image-based multi-view approaches and 3D voxel convolutions, but point-cloud convolutions are nontrivial.The difficulty stems from the absence of a well-defined point ordering.
- Object Classification on Point Clouds: 92% accuracy is reported for point-cloud methods on ModelNet40, with leading methods separated by less than 1%.
- Object Datasets: ModelNet40 contains CAD models from 40 common categories and is a standard benchmark for point-cloud object classification.
- Object Datasets: ModelNet40 assumes objects are clean, complete, and free of background noise, conditions that are not often held in practice.
- Object Datasets: Real-world scans commonly contain partial objects and cluttered background from reconstruction imperfections and scene placement.
- Object Datasets: Existing real-world object-scan datasets were often too small or controlled for training large classification networks.The Sydney urban objects dataset, for example, contains 631 objects across 26 categories and is mainly used for evaluation.
3. Benchmark Data
The benchmark pairs synthetic ModelNet40 data with ScanObjectNN, a real-world dataset designed to test classification under noise, clutter, partiality, and non-uniform sampling. It enriches real-world objects through background attachment and bounding-box perturbations that create progressively harder variants.
- Real-world challenges: Real-world scans differ from CAD models through background noise, non-uniform density, incomplete reconstructions, and occlusions.These properties can leave objects partial and include background elements within object regions.
- ScanObjectNN construction: 2902 objects are categorized into 15 categories in the resulting real-world dataset.
- Dataset variants: OBJ ONLY contains ground-truth segmented objects and tests robustness to deformed geometry and non-uniform surface density.
- Dataset variants: OBJ BG attaches background points extracted from ground-truth axis-aligned object bounding boxes.The resulting regions can include distracting scene elements, such as supporting surfaces.
- Dataset variants: Four perturbation variants progressively increase difficulty by translating, rotating, and scaling ground-truth bounding boxes before geometry extraction.The variants are PB T25, PB T25 R, PB T50 R, and PB T50 RS; invalid samples are discarded unless at least 50% of original object points remain.
4. Benchmark on ScanObjectNN
The ScanObjectNN benchmark shows that point-cloud classification becomes substantially harder under real-world conditions, especially background clutter, partial observations, and domain shift between CAD models and scans. Training and testing on real-world data improves performance relative to cross-domain evaluation, while the dataset also supports part segmentation and motivates further work on bridging CAD–scan differences.
- Benchmark setup: Six representative point-cloud classifiers are benchmarked on ScanObjectNN, including 3DmFV, PointNet, SpiderCNN, PointNet++, DGCNN, and PointCNN.The benchmark compares established methods under real-world object-classification conditions.
- Training on ModelNet40: Less than 50% accuracy is achieved by all methods trained on ModelNet40 and tested on ScanObjectNN, indicating poor transfer from clean CAD geometry to noisy, incomplete scans.Real-world objects contain partial surfaces, low-frequency noise, and inaccurate boundaries unlike typical CAD models.
- Background effects: Background lowers classification performance for nearly all methods, because foreground and background points can distract existing models.On the hardest perturbation, every existing model loses overall accuracy when background is present.
- Perturbation effects: Larger translation and rotation perturbations reduce accuracy substantially, while scale causes a smaller additional degradation as objects become more partial and cluttered.Confusion matrices on PB T50 RS show no major ambiguity among object classes despite the difficult setting.
- Generalization and future directions: Training on ScanObjectNN generalizes better to ModelNet40 than training on ModelNet40 generalizes to ScanObjectNN, although classes commonly adjacent to walls remain difficult.The study identifies domain generalization, background separation, and partial-object handling as open problems; part annotations may help future partial-object classification.
5. Background-aware Classification Network
The paper introduces background-aware point-cloud classifiers that jointly learn object classification and foreground-background segmentation. These models improve classification and mask prediction on cluttered real-world scans, while exposing limitations and cross-domain differences.
- Motivation: Existing classifiers process the entire point cloud without distinguishing foreground from background, limiting their handling of real scans.The paper attributes this issue to network design and the simplicity of datasets such as ModelNet40.
- Method: A segmentation-guided branch predicts an object mask separating foreground from background and is trained jointly with classification.The masks are available from the source scene instance-segmentation datasets.
- Network Architecture: BGA-PN++ extends PointNet++ with classification and segmentation branches, using classification features to guide mask prediction.Its joint loss is Ltotal = Lclass + λLseg, with λ = 0.5.
- Network Architecture: BGA-DGCNN applies the same joint classification-segmentation design to DGCNN, indicating that the background-aware concept transfers across architectures.The classification branch supplies input to the segmentation branch.
- Evaluation: 52.6% accuracy was achieved by BGA-PN++ on ModelNet40, while BGA-DGCNN achieved 56.5% under training on PB T50 RS.Both BGA models outperformed their vanilla counterparts on the reported evaluations, with BGA-PN++ best on PB T50 RS.
- Evaluation: 77.6% and 71.0% segmentation accuracy were obtained by BGA-PN++ on PB T50 RS and ModelNet40, respectively; BGA-DGCNN obtained 78.5% and 74.3%.The predicted masks were reported to remove background fairly accurately.
- Discussion: DGCNN-based models generalized better between real and CAD data, suggesting further DGCNN investigation could help bridge the domain gap.This pattern was observed for both real-to-CAD and synthetic-to-real cases.
- Discussion and Limitation: BGA requires object masks and background in the data, and it failed on a background-free ModelNet40 object.The limitation is illustrated as a failure case in Figure 8-(a).
6. Conclusion
The paper revisits point-cloud classification by introducing a real-world dataset with practical object-scanning challenges and benchmarking existing methods. It also proposes a background-aware network and reports advances on synthetic and real-world datasets.
- Conclusion: The paper introduces a real-world dataset containing ∼15,000 objects in 15 categories, with background occurrence, object partiality, and deformation variants.These properties create more practical challenges than current datasets.
- Conclusion: The authors benchmark existing methods, identify open problems, and propose a point-cloud network for classifying objects with background.The study compares performance across the new real-world dataset and synthetic data.
- Conclusion: Experimental results show advances of the proposed method on both synthetic and real-world object datasets.