PaperlayerView exact revision on arXiv

Source-linked AI summary

H3DNet: 3D Object Detection Using Hybrid Geometric Primitives

Zaiwei Zhang, Bo Sun, Haitao Yang, Qixing Huang

arXiv:2006.05682v3cs.CV

TL;DR

3D object detection from colorless point clouds must recover oriented boxes and labels despite irregular inputs and varying scene object counts. H3DNet predicts hybrid geometric primitives, optimizes proposals through a parametric distance function, and then classifies and refines them. It achieves state-of-the-art results on ScanNet and SUN RGB-D, including 67.2% mAP (0.25) on ScanNet.

  • Problem

    Detecting labeled, oriented 3D bounding boxes from colorless point clouds is challenging because inputs are irregular and scenes contain varying numbers of objects.

  • Method

    H3DNet predicts BB centers, face centers, and edge centers, converts them into proposals using a parametric distance function, then classifies and refines the proposals.

  • Results

    67.2% mAP (0.25) on ScanNet was an 8.5% relative improvement over state-of-the-art methods using only 3D point positions, with state-of-the-art results on ScanNet and SUN RGB-D.

  • Takeaways & Limitations

    The hybrid and overcomplete primitive representation supplies diverse constraints and features that improve 3D object detection and reduce the influence of individual primitive prediction errors.

Abstract

from arXiv · show

We introduce H3DNet, which takes a colorless 3D point cloud as input and outputs a collection of oriented object bounding boxes (or BB) and their semantic labels. The critical idea of H3DNet is to predict a hybrid set of geometric primitives, i.e., BB centers, BB face centers, and BB edge centers. We show how to convert the predicted geometric primitives into object proposals by defining a distance function between an object and the geometric primitives. This distance function enables continuous optimization of object proposals, and its local minimums provide high-fidelity object proposals. H3DNet then utilizes a matching and refinement module to classify object proposals into detected objects and fine-tune the geometric parameters of the detected objects. The hybrid set of geometric primitives not only provides more accurate signals for object detection than using a single type of geometric primitives, but it also provides an overcomplete set of constraints on the resulting 3D layout. Therefore, H3DNet can tolerate outliers in predicted geometric primitives. Our model achieves state-of-the-art 3D detection results on two large datasets with real 3D scans, ScanNet and SUN RGB-D.

1 Introduction

H3DNet detects labeled, oriented 3D bounding boxes from colorless point clouds using hybrid, overcomplete geometric primitives. It continuously generates proposals from these primitives, then classifies and refines them, achieving strong results on ScanNet and SUN RGB-D.

  • Motivation: H3DNet targets oriented 3D bounding boxes and semantic labels from colorless 3D point clouds.The task is challenging because point clouds are irregular and scenes contain varying numbers of objects.
  • Core idea: The model predicts BB centers, BB face centers, and BB edge centers as a hybrid, overcomplete geometric representation.Different primitive types focus on different regions and provide complementary constraints and features.
  • Proposal generation: A parametric distance function converts predicted primitives into object proposals whose local minimums support continuous BB optimization.This produces high-quality proposals from imprecise initial proposals.
  • Matching and refinement: H3DNet classifies proposals, predicts geometric-parameter offsets, and assigns semantic labels while aggregating features from associated primitives.The refinement module is designed to handle a varying number of geometric primitives.
  • Results: 67.2% mAP (0.25) on ScanNet represented an 8.5% relative improvement over state-of-the-art methods using only 3D point positions.On SUN RGB-D, H3DNet achieved 60.1% mAP (0.25), a 2.4% relative improvement over the same methods.

2 Related Works

H3DNet builds on point-cloud object detection and jointly predicts multiple geometric primitives. Its overcomplete primitive constraints extend prior multi-feature prediction ideas to 3D bounding-box detection.

  • 3D detection and point-based learning: Prior 3D detection methods commonly generate candidate objects and classify them, while point-based networks support several 3D vision tasks.H3DNet is most closely related to point-cloud object detection methods such as VoteNet.
  • Multi-primitive prediction: HybridPose showed that jointly predicting keypoints, connecting edges, and symmetric correspondences can improve feature prediction accuracies.H3DNet applies a related joint-prediction rationale to BB centers, face centers, and edge centers.
  • Overcomplete constraints: H3DNet extends overcomplete-constraint regression by using multiple BB primitive types to improve the generalization behavior of primitive predictions.The representation supplies diverse geometric constraints rather than relying on a single primitive type.

3 Approach

H3DNet predicts an overcomplete hybrid set of geometric primitives, converts them into object proposals through a distance function, and then classifies and refines those proposals. Its modules integrate complementary primitive features and tolerate erroneous predictions during detection.

  • Approach Overview: H3DNet processes point clouds through geometric primitive prediction, proposal generation, and object refinement.The network is trained end-to-end.
  • Geometric Primitive Module: The geometric primitive module predicts BB centers, BB face centers, and BB edge centers, each with a latent feature for later modules.The three prediction branches operate on dense pointwise descriptors.
  • Geometric Primitive Module: Different primitive types address different scene conditions: centers suit complete solid objects, faces suit planar or partial objects, and edges suit irregularly sampled objects with complete boundaries.Face centers are useful for rich planar patches, while edge centers help when point clouds contain large holes but object edges remain complete.
  • Approach Overview: Because only a subset of predicted primitives is sufficient for detection and different primitive errors are largely uncorrelated, detected objects can tolerate erroneous predictions.The hybrid representation combines complementary signals rather than relying on a single primitive type.
  • Geometric Primitive Module: H3DNet uses up-to 19 positional constraints for one BB by combining one center, six face centers, and twelve edge centers.This overcomplete representation provides flexibility for generating, classifying, and refining proposals.
  • Proposal Module: A parametric distance function matches predicted primitives to object primitives, and its local minimums become continuously optimized object proposals.The function incorporates multiple primitive types and makes proposals less sensitive to initial conditions.
  • Classification and Refinement Module: The final module classifies proposals, predicts bounding-box offsets and semantic labels, and aggregates features from matched center, face, and edge primitives.The aggregation combines potentially complementary semantic and geometric information after primitive groups have been matched.

4 Experimental Results

H3DNet is evaluated against state-of-the-art 3D detectors on ScanNet V2 and SUN RGB-D, with analyses of primitive choices, proposal refinement, and descriptor towers. It achieves stronger detection performance, especially for accurate localization and objects with rich geometric structure.

  • Quantitative comparison: The evaluation compares H3DNet with state-of-the-art 3D object detection methods on ScanNet V2 and SUN RGB-D using mAP@0.25 and mAP@0.5.The datasets use the validation protocols and semantic classes aligned with VoteNet.
  • Quantitative comparison: 67.2% mAP@0.25 on ScanNet V2 is 8.5% better than the top-performing baseline, while mAP@0.5 improves by 14.6%.On SUN RGB-D, H3DNet gains 2.4% and 6.1% in mAP at thresholds 0.25 and 0.5, respectively.
  • Quantitative comparison: H3DNet produces larger gains at mAP@0.5 than at mAP@0.25 on both datasets, indicating more accurate predictions than baseline approaches.The paper attributes the improvements to overcomplete geometric primitives and associated features for proposal generation and refinement.
  • Per-category analysis: H3DNet improves thin-object detection in ScanNet, including door, window, picture, curtain, and shower-curtain categories.Reported gains are 13.7%, 13.8%, 10.8%, 10.1%, and 18.2%, respectively.
  • Ablation study: Combining BB centers, face centers, and edge centers outperforms using a single primitive group because different object structures favor different primitives.Edge primitives benefit objects such as windows, while face primitives benefit objects such as shower curtains.
  • Ablation study: Proposal refinement improves performance, while results remain stable across truncation thresholds; adding descriptor towers helps but yields quickly diminishing gains.Replacing VoteNet’s descriptor tower with four H3DNet towers alone produces modest or no gains, supporting the importance of hybrid primitives and proposal matching and refinement.

5 Conclusions and Future Work

The paper introduces H3DNet as a 3D object detector that predicts and fits labeled, oriented bounding boxes using hybrid geometric primitives. Future work extends this framework to additional scene-understanding tasks and additional primitives.

  • Conclusion: H3DNet takes a 3D scene as input and outputs labeled, oriented bounding boxes by fitting objects to an overcomplete set of geometric primitives.The primitives include BB centers, BB face centers, and BB edge centers with associated features.
  • Future work: The authors propose applying H3DNet to instance segmentation and CAD model reconstruction in future work.
  • Future work: Another future direction is integrating additional geometric primitives, such as BB corners, for 3D object detection.

B Proof of Proposition 1

The supplemental material proves Proposition 1 by differentiating the relevant equations with respect to Θ. Applying the chain rule yields the equation stated in the main paper.

  • Proof: The supplemental material presents a proof of Proposition 1 from the main paper.
  • Proof: The derivation applies the chain rule together with equation (3) from the main paper.
  • Proof: The proof computes derivatives of both sides with respect to Θ.
  • Proof: The resulting derivation leads to equation (4) in the main paper.

C.1 Network architecture details

H3DNet uses multiple network modules to predict geometric primitives, generate proposals, and classify and refine detected objects. The architecture is trained end-to-end with multitask objectives and processes 40000 sampled points per scene.

  • Architecture: H3DNet comprises geometric primitive, proposal generation, and classification-and-refinement modules trained end-to-end.The modules progress from primitive prediction to proposal generation and final object refinement.
  • Geometric primitive module: The geometric primitive module uses multiple PointNet++-based backbone towers to extract per-point features and predict BB centers, face centers, and edge centers.Four backbones sample 1024 points from input point clouds containing 40000 points.
  • Proposal generation module: The proposal module generates initial object proposals and associates them with an overcomplete set of geometric primitives using local minimums of a distance function.Predicted flags select geometric primitives before association.
  • Classification and refinement module: The classification-and-refinement module predicts object existence, semantic labels, and offsets for BB center, size, and orientation.It combines proposal features with object-center features from the primitive module.
  • Training: Training uses a multitask loss with geometric primitive, proposal, distance-function, classification, and object-refinement objectives.The refinement objectives include semantic-label cross-entropy and L2 regression for BB center, size, and orientation.
  • Implementation: The implementation uses PyTorch and Adam with batch size 8 for 360 epochs.Learning-rate schedules differ between ScanNet and SUN RGB-D.

D.1 Dataset Statistics

H3DNet relies on face and edge annotations, but visibility and scan quality vary across datasets. SUN RGB-D has fewer edge labels and less accurate box annotations than ScanNet.

  • Each 3D bounding box has up to 6 boundary faces and 12 boundary edges, though occlusion and irregular shapes can hide some.
  • SUN RGB-D contains significantly fewer edge labels per object than ScanNet.
  • ScanNet provides dense face and edge labels across objects, with averages of 2.37 faces and 6.25 edges per object.
  • SUN RGB-D box labels are less accurate than ScanNet labels, and lacking per-point instance labels makes dense annotation generation more difficult.

D.2 Qualitative Results

H3DNet generally detects geometric primitives reasonably well, but edge predictions are sparse on SUN RGB-D because training labels are limited.

  • Qualitative examples show that H3DNet performs reasonably well at detecting geometric primitives.
  • Edge predictions on SUN RGB-D are sparse because the training data lacks sufficient edge labels.

D.3 Quantitative Analysis

Quantitative analyses support hybrid geometric primitives: different primitives suit different categories, their errors are often weakly correlated, and aggregation can reduce variance and improve accuracy.

  • Different geometric primitives are suitable for different object categories, while prediction errors across primitives are mostly uncorrelated.
  • The truncated L2 loss can prune outlier predictions when aggregating diverse geometric primitives.
  • Edge-center predictions are more accurate for most ScanNet categories, whereas face centers perform better for categories such as windows and curtains.
  • Prediction errors are measured by the norm of each 3D error vector, and bias is generally smaller than variance across ScanNet and SUN RGB-D.
  • SUN RGB-D face-center errors are slightly correlated, but other geometric primitive errors remain uncorrelated.
  • With suitable trade-off parameters, aggregating primitive predictions reduces variance; because bias is smaller than variance, accuracy is expected to improve.

E.1 More quantitative results

H3DNet outperforms baseline approaches on per-category ScanNet and SUN RGB-D detection, with especially large gains for thin ScanNet objects.

  • 14.9%, 24.0%, 25.7%, and 27.0% gains are reported for ScanNet Window, Counter, Curtain, and Shower-curtain categories at 3D IoU threshold 0.5.

E.2 More qualitative results

The paper presents additional qualitative 3D object-detection results on ScanNet V2 and SUN RGB-D V1, including thin-object comparisons and failure cases. Its method detects more positive thin objects and is more accurate than top-performing baseline approaches in the shown comparisons.

  • Figures 16 and 17 provide additional qualitative 3D object-detection examples on ScanNet V2 and SUN RGB-D V1.
  • On thin objects, the method is more accurate and detects more positive examples than the top-performing baseline approach.Figure 14 marks the thin objects with red arrows.
  • Figure 15 presents failure cases from the method, with their failure patterns summarized in the caption.
Loading 2006.05682v3…