Source-linked AI summary
Deep Hough Voting for 3D Object Detection in Point Clouds
Charles R. Qi, Or Litany, Kaiming He, Leonidas J. Guibas
TL;DR
3D detection systems often convert irregular point clouds into grids or depend on 2D image proposals, while sparse surface samples make object centers difficult to regress directly. VoteNet addresses this with an end-to-end deep Hough-voting detector that generates and aggregates votes near object centers, achieving strong geometry-only results on ScanNet and SUN RGB-D.
Problem
Sparse point clouds sample object surfaces, so object centers may be far from observed points and difficult to regress directly; existing methods often rely on grids or 2D detectors.
Method
VoteNet directly processes point clouds with PointNet++ features and learned Hough voting that generates votes near object centers for box proposals.
Results
VoteNet significantly outperforms prior methods on SUN RGB-D and ScanNet using geometry only, including methods using RGB and geometry or multi-view RGB images.
Takeaways & Limitations
The voting scheme supports effective context aggregation, with the largest improvements when object centers are far from object surfaces.
Takeaways & Limitations
Without RGB information, detecting very thin objects such as doors, windows, and pictures is almost impossible.
Abstract
from arXiv · showhide
Current 3D object detection methods are heavily influenced by 2D detectors. In order to leverage architectures in 2D detectors, they often convert 3D point clouds to regular grids (i.e., to voxel grids or to bird's eye view images), or rely on detection in 2D images to propose 3D boxes. Few works have attempted to directly detect objects in point clouds. In this work, we return to first principles to construct a 3D detection pipeline for point cloud data and as generic as possible. However, due to the sparse nature of the data -- samples from 2D manifolds in 3D space -- we face a major challenge when directly predicting bounding box parameters from scene points: a 3D object centroid can be far from any surface point thus hard to regress accurately in one step. To address the challenge, we propose VoteNet, an end-to-end 3D object detection network based on a synergy of deep point set networks and Hough voting. Our model achieves state-of-the-art 3D detection on two large datasets of real 3D scans, ScanNet and SUN RGB-D with a simple design, compact model size and high efficiency. Remarkably, VoteNet outperforms previous methods by using purely geometric information without relying on color images.
1. Introduction
VoteNet directly detects oriented 3D boxes and semantic classes from point clouds, avoiding 2D-detector dependencies and regular-grid conversion. It uses learned Hough voting to address the difficulty of locating object centers in sparse surface samples, achieving strong results on SUN RGB-D and ScanNet.
- 3D object detection estimates oriented bounding boxes and semantic classes from scene point clouds.
- Regular-grid and 2D-detector pipelines incur computation, geometric-detail, or dependency costs when processing irregular point clouds.
- VoteNet directly processes raw point clouds without 2D detectors, using PointNet++ features and an end-to-end Hough-voting architecture.
- Voting generates points near object centers so they can be grouped and aggregated into box proposals despite sparse surface observations.
- On SUN RGB-D and ScanNet, VoteNet using geometry only significantly outperforms prior methods using RGB and geometry or multi-view RGB images.
- The work analyzes voting’s importance for 3D point-cloud detection.
2. Related Work
Prior 3D detection methods commonly project or voxelize point clouds, use 2D proposals, or apply point-cloud networks in limited detection settings. VoteNet combines point-cloud processing with a voting-based proposal architecture.
- Hough voting has been applied to object detection, 3D point-cloud detection, pose estimation, and image or medical-image analysis.
- Deep point-cloud networks have shown strong results in classification and segmentation, while relatively few methods directly propose and detect 3D objects from point clouds.
- VoteNet’s architecture processes points into votes, groups votes into clusters, and produces classified, non-maximum-suppressed proposals.
3. Deep Hough Voting
Deep Hough voting adapts classical voting to sparse point clouds and integrates its stages into an end-to-end trainable network. Learned features, vote generation, and vote aggregation replace hand-crafted or separated components.
- Classical Hough voting uses interest-point patches and a codebook of offsets to generate votes toward object centers.
- Voting suits sparse point sets because it avoids requiring proposals near empty-space object centers and accumulates partial evidence bottom-up.
- Integrating traditional voting’s separated modules into point-cloud networks is the central open problem addressed by VoteNet.
- VoteNet learns interest-point selection, vote generation, vote features, and vote aggregation with trainable point-cloud layers.
- The resulting network is a single end-to-end trainable architecture combining existing-point processing for votes with virtual-point processing for object proposals and classification.
4. VoteNet Architecture
VoteNet transforms point-cloud seeds into votes near object centers, then clusters and aggregates those votes to generate and classify 3D box proposals in one end-to-end pipeline.
- Point cloud feature learning: The network processes point clouds through a backbone that outputs M seed points with coordinates and enriched feature vectors.Each seed generates one vote.
- Learning to vote: A shared MLP predicts coordinate and feature offsets for each seed, producing votes whose locations and features are shifted from the seed representation.The coordinate offset is explicitly supervised toward the displacement between the seed and its object’s bounding-box center.
- Learning to vote: Votes from the same object become closer together than the original surface seeds, creating meeting points that make contextual aggregation easier.Votes are no longer grounded on object surfaces.
- Vote clustering: The model samples K votes by farthest-point sampling and forms spatial clusters by grouping neighboring votes within radius r.This simple clustering strategy is integrated into the end-to-end pipeline.
- Proposal and classification: A shared PointNet independently processes each cluster, max-pools vote features, and predicts objectness, box parameters, and semantic classes.The proposal stage uses normalized local vote coordinates and supervises objectness, box estimation, and class prediction for positive proposals.
- Inference: At inference, VoteNet processes an entire scene in one forward pass and applies 3D non-maximum suppression with an IoU threshold of 0.25.Evaluation uses mean average precision.
5. Experiments
Experiments compare VoteNet with prior detectors and controlled baselines on SUN RGB-D and ScanNetV2, then analyze voting, aggregation, efficiency, and qualitative behavior. VoteNet achieves strong geometric-only performance, with voting particularly helping when object centers are far from observed surfaces.
- 5. Experiments: Evaluation covers SUN RGB-D and ScanNetV2, including comparisons with prior 3D detectors and analyses of voting, aggregation, efficiency, and qualitative results.SUN RGB-D uses 3D boxes for 10 common categories, while ScanNetV2 provides reconstructed indoor scenes and 18 object categories.
- 5.2. Analysis Experiments: Voting improves mAP by approximately 5 points on SUN RGB-D and more than 13 points on ScanNet over the direct BoxNet baseline.BoxNet shares VoteNet’s backbone but proposes boxes directly from seed points rather than vote clusters.
- 5.2. Analysis Experiments: Voting provides broader coverage of scene locations that can generate accurate proposals, effectively increasing detection context compared with direct seed-point proposals.The ScanNet visualization overlays seed points or votes associated with good boxes and shows denser VoteNet coverage.
- 5.2. Analysis Experiments: Voting helps more when object points lie farther from amodal box centers, linking its gains to the sparse surface sampling of point clouds.Category-level analysis compares VoteNet–BoxNet mAP gains with normalized distances between object surfaces and box centers.
- 5.2. Analysis Experiments: Learned PointNet max aggregation outperforms manual local vote-feature pooling, while performance peaks at an aggregation radius of approximately 0.2.The aggregation study attributes the advantage of learned aggregation to clutter votes from non-object seeds.
- 5.2. Analysis Experiments: VoteNet is more than 4× smaller than F-PointNet and more than 20× faster than 3D-SIS under the reported comparison settings.The speed comparison uses sequential processing for VoteNet and offline batch-mode averaged timing for 3D-SIS on ScanNetV2.
- 5.3. Qualitative Results and Discussion: Qualitative results show robust detections in cluttered, partial, and artifact-affected scenes, but thin objects such as doors, windows, and pictures are commonly missed without RGB.The method can sometimes hallucinate amodal boxes from partial observations, though very large, extremely partial objects remain difficult.
6. Conclusion
VoteNet is a simple 3D object detector inspired by Hough voting that learns object-center votes and aggregates them into high-quality proposals. Using only point clouds, it improves over prior methods using depth and color images, while future work considers RGB integration and downstream applications.
- VoteNet learns to vote directly to object centroids from point clouds.
- The network aggregates vote features and local geometry to generate high-quality object proposals.
- Using only 3D point clouds, VoteNet showed significant improvements over previous methods using depth and colored images.
- Future work includes incorporating RGB images, applying the detector to 3D instance segmentation, and exploring 6D pose estimation and template-based detection.
A. Appendix
The appendix provides additional architectural, loss-function, analytical, per-category, and visualization details.
- The appendix adds network architecture and loss-function details.
- It also reports more analysis experiments, per-category ScanNet results, and visualizations.
A.1. Details on Architectures and Loss Functions
The appendix specifies VoteNet’s PointNet++ backbone, voting and proposal modules, oriented-box outputs, training losses, and the Chamfer-based center-regression design. It also contrasts VoteNet with a direct-proposal BoxNet baseline.
- VoteNet combines a PointNet++ backbone, a voting module, and a proposal module.
- The backbone uses set abstraction and feature-propagation layers to subsample and upsample point features.
- The voting module transforms seed features into XYZ and feature offsets for votes.
- The proposal module predicts objectness, center, heading, size, and semantic-class outputs for oriented 3D boxes.
- VoteNet is trained end-to-end with voting, objectness, box-estimation, and semantic-classification losses.
- Chamfer loss for center regression encourages proposals near ground-truth objects and votes from nearby non-object seeds.
- BoxNet shares VoteNet’s backbone but directly proposes boxes from seed features without vote regression.
A.2. More Analysis Experiments
Analysis experiments examine proposal count, voting context and multiplicity, cluster sampling, and height features. VoteNet maintains strong recall with many proposals, performs best with FP2 voting context, and benefits consistently from height features.
- Proposal count: With 10 proposals, VoteNet achieves about 45% AP; with 100 proposals, AP exceeds 57%.
- Proposal count: With 1,000 proposals, recall reaches about 78.7% for joint proposals and about 87.7% for per-class proposals.
- Seed context: Voting from SA2 achieves 51.2% mAP, FP2 performs best, and the larger-context FP3 layer adds no further improvement.
- Votes per seed: One vote per seed achieves the best results in the vote-count ablation.
- Proposal sampling: Different proposal-sampling strategies produce similar performance, indicating robustness to cluster-center selection.
- Height feature: Adding point height consistently improves detection performance on both SUN RGB-D and ScanNet.
A.3. ScanNet Per-class Evaluation
On ScanNetV2, VoteNet performs especially well on objects whose geometry supports recognition, but struggles with texture- and color-dependent objects.
- VoteNet reports per-class average precision across 18 ScanNetV2 classes at 0.25 and 0.5 box IoU thresholds.Its strongest performance is especially evident for mAP@0.25.
- Purely geometric data supports strong detection of beds, chairs, tables, and desks, while pictures remain difficult because texture and color are important cues.
A.4. Visualization of Votes
VoteNet’s predicted votes generally point from object seeds toward object centers, while clutter votes depend on proximity and local structure.
- Seed points on objects such as beds and sofas vote toward their object centers.
- Clutter points vote toward nearby object centers or nowhere when the clutter area lacks structure, such as on a wall.