PaperlayerView exact revision on arXiv

Source-linked AI summary

ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes

Charles R. Qi, Xinlei Chen, Or Litany, Leonidas J. Guibas

arXiv:2001.10692v1cs.CV

TL;DR

Point clouds enable 3D detection but are sparse and lack color, leaving open how images can assist point-cloud-based detection. ImVoteNet fuses lifted 2D geometric and semantic/texture cues with 3D votes through a multi-tower VoteNet architecture, improving SUN RGB-D performance by 5.7 mAP over VoteNet.

  • Problem

    Point-cloud 3D detection lacks color and is limited by sparsity and sensor noise, while effectively using complementary image information remains an open problem.

  • Method

    ImVoteNet fuses image-based geometric and semantic/texture cues with point-cloud features by lifting 2D votes to 3D and training separate image, point, and joint towers.

  • Results

    +5.7 mAP over geometry-only VoteNet on SUN RGB-D, with state-of-the-art 3D object detection performance.

  • Takeaways & Limitations

    Image cues can be integrated into voting-based 3D detection and are especially useful when point clouds are sparse or unfavorably distributed.

  • Takeaways & Limitations

    The evaluation excludes ScanNet because its multiple 2D views per reconstructed scene require additional multi-view feature merging.

Abstract

from arXiv · show

3D object detection has seen quick progress thanks to advances in deep learning on point clouds. A few recent works have even shown state-of-the-art performance with just point clouds input (e.g. VoteNet). However, point cloud data have inherent limitations. They are sparse, lack color information and often suffer from sensor noise. Images, on the other hand, have high resolution and rich texture. Thus they can complement the 3D geometry provided by point clouds. Yet how to effectively use image information to assist point cloud based detection is still an open question. In this work, we build on top of VoteNet and propose a 3D detection architecture called ImVoteNet specialized for RGB-D scenes. ImVoteNet is based on fusing 2D votes in images and 3D votes in point clouds. Compared to prior work on multi-modal detection, we explicitly extract both geometric and semantic features from the 2D images. We leverage camera parameters to lift these features to 3D. To improve the synergy of 2D-3D feature fusion, we also propose a multi-tower training scheme. We validate our model on the challenging SUN RGB-D dataset, advancing state-of-the-art results by 5.7 mAP. We also provide rich ablation studies to analyze the contribution of each design choice.

1. Introduction

ImVoteNet addresses the open problem of using images effectively in point-cloud 3D detection by combining 2D geometric and semantic cues with 3D voting. Built on VoteNet, it uses lifted image votes and multi-tower training, achieving state-of-the-art results on SUN RGB-D.

  • Motivation: Point clouds provide strong 3D detection results but remain sparse, lack color, and can suffer from sensor noise.These limitations motivate incorporating complementary RGB information.
  • Motivation: RGB images offer higher resolution, rich texture, and coverage of blind regions that complement point-cloud geometry.Images also lack absolute 3D measurements, motivating geometric fusion rather than image-only detection.
  • Motivation: Effective image use remains open because directly appending RGB values to sparse points loses dense image-domain patterns.Prior approaches include 2D proposals that constrain 3D search and intermediate feature concatenation, each with stated drawbacks.
  • Approach: ImVoteNet combines 2D and 3D voting while preserving proposals from the full point cloud and leveraging geometric, semantic, and texture cues from images.Image features are lifted into 3D and concatenated with point-cloud features before generating unrestricted 3D Hough votes.
  • Approach: A multi-tower architecture separately trains point-only, image-only, and joint-feature towers to balance information from both modalities.The towers use separate voting, proposal, and loss parameters, while testing uses the main joint tower.
  • Results: +5.7 mAP over geometry-only VoteNet establishes state-of-the-art performance for ImVoteNet on SUN RGB-D.The paper also reports extensive ablations analyzing individual components.

2. Related Work

Prior RGB-D detection methods use 2D-driven proposals, 3D-driven processing, or feature concatenation. ImVoteNet instead lifts explicit 2D geometric cues as pseudo-3D votes and uses multi-tower training to balance modalities.

  • RGB-D detection: RGB-D detection methods combine depth and color through 2D-driven, 3D-driven, or feature-concatenation strategies.2D-driven methods guide 3D search from image detections, while 3D-driven methods use image features after generating 3D proposals.
  • ImVoteNet: ImVoteNet’s pipeline contains separate image-detection and point-cloud branches before lifting image votes and cues to 3D seed points.The joint tower then uses concatenated features for 3D voting and box proposals, alongside image-only and point-only towers.
  • ImVoteNet: Unlike prior concatenation-based fusion, ImVoteNet explicitly lifts 2D detector geometry into 3D as pseudo-3D votes.It also uses a multi-tower architecture instead of training only on concatenated features.
  • Multimodal fusion: Gradient blending was developed for multimodal learning to improve robustness to overfitting and different convergence rates, and ImVoteNet adopts it.The broader multimodal literature applies such fusion to vision-language and video-sound tasks.

3. ImVoteNet Architecture

ImVoteNet extends VoteNet for RGB-D scenes by combining 2D image votes with 3D point-cloud voting. It lifts geometric, semantic, and texture cues into 3D and uses multi-tower training to fuse the modalities.

  • Architecture: ImVoteNet builds on VoteNet by passing geometric and semantic/texture cues from RGB images into 3D voting and proposal generation.The architecture is designed for RGB-D scenes and uses 2D detectors to create image votes before lifting image information into 3D.
  • Deep Hough Voting: VoteNet extracts seed features from a subsampled point cloud, generates 3D votes toward object centers, and clusters votes for proposal generation and classification.Each vote contains 3D coordinates supervised toward an object center and a learned feature vector for detection.
  • Image Votes from 2D Detection: Image votes connect pixels to detected 2D box centers and augment each connection with semantic and texture features from the source image.Pixels inside multiple boxes receive multiple votes, while pixels outside detected boxes are padded with zeros.
  • Geometric Cues: Given camera parameters, a 2D box-center vote becomes a 3D ray that narrows the possible object-center location from a volume to a one-dimensional search space.The method associates each surface point with its projection and uses the projected 2D vote to derive a pseudo 3D vote.
  • Geometric Cues: The pseudo 3D vote assumes similar depths for a surface point and object center, then passes the ray direction to compensate for approximation error.The paper states this assumption is reasonable for most objects that are not too close to the camera.
  • Feature Fusion: RGB features provide complementary semantic and texture cues: region features help distinguish geometrically similar classes, while pixel features preserve dense image detail.The implementation can use lightweight region representations, including one-hot class vectors with confidence scores.

4. Experiments

Experiments on SUN RGB-D compare ImVoteNet with prior methods and analyze its image-fusion components, multi-tower training, and sparse-point robustness. ImVoteNet improves substantially over VoteNet and benefits especially from image cues when geometry is limited.

  • 4.1. Comparing with State-of-the-art Methods: On SUN RGB-D, ImVoteNet outperforms previous methods by 5.7 mAP over VoteNet, with especially large gains for occluded or geometrically similar categories.Improvements include +12.5 AP for bookshelves, +11.6 AP for dressers, and +7.7 AP for nightstands.
  • 4.2. Qualitative Results and Discussion: Qualitative examples show image cues recovering cluttered bookshelves, depth-sparse black sofas, and distant objects missed by VoteNet or ground-truth annotations.The examples attribute gains to 2D localization, semantics, and compensation for depth-sensor limitations.
  • 4.3. Analysis Experiments: Ablations show that removing all 2D geometric cues drops performance by 2.2 points, while removing ray angle drops it by 1.2 points.The ray angle provides a corrective cue to pseudo 3D votes.
  • 4.3. Analysis Experiments: The one-hot class score is the strongest tested semantic cue, whereas unreduced 1024-dimensional RoI features perform worst and 64-dimensional reductions remain inferior.Raw RGB features are effective, while per-pixel CNN texture features hurt performance, likely because of over-fitting.
  • 4.3. Analysis Experiments: Image-only tower features can outperform several previous methods, and balanced multi-tower gradient blending is evaluated against single-tower and tower-dominated settings.The analysis varies tower weights, including single-tower training, 0.8-dominant weights, and a more balanced default.
  • 4.4. Detection with Sparse Point Clouds: With fewer point-cloud samples, geometry-only detection drops quickly, while image cues improve performance most for non-uniform samples near ORB key points.The sparse-point experiments compare random uniform sampling with ORB-key-point sampling.

5. Conclusion

IMVOTENET integrates image-derived geometric and semantic/texture information into VoteNet’s 3D voting pipeline, improving detection especially when point clouds are sparse or unfavorably distributed.

  • IMVOTENET integrates geometric and semantic/texture information from images into a voting-based 3D detection pipeline.It lifts 2D geometric information into 3D using camera parameters and pixel depth.
  • IMVOTENET significantly boosts 3D object detection performance through multi-modal training with gradient blending.The reported gains are especially pronounced for sparse or unfavorably distributed point clouds.

A. Overview

The supplementary material expands the IMVOTENET architecture and visualizes sparse point clouds.

  • The supplement details the IMVOTENET architecture, including the point cloud network, 2D detector, 2D votes, and image-vote lifting.It organizes these details in Section B.
  • The supplement also provides visualizations of sparse point clouds.These visualizations appear in Section C.

B. Details on IMVOTENET Architecture

This section describes the IMVOTENET architecture, covering the point cloud network, training procedure, 2D detection and votes, and lifting 2D votes with camera parameters.

  • Section B.1 describes the point cloud deep network and training procedure.
  • Section B.2 provides details on the 2D detector and 2D votes.
  • Section B.3 explains how 2D votes are lifted using general camera parameters.

B.1. Point Cloud Network

The point cloud network samples and encodes SUN RGB-D depth points, fuses lifted image-vote features before voting, and trains the resulting detector with a staged procedure.

  • The backbone takes 20k-point SUN RGB-D depth-image clouds, with each point represented by XYZ coordinates and height.The floor height is estimated from the 1% percentile of point heights, and points are augmented by subsampling and horizontal flips.
  • The PointNet++ backbone uses four set-abstraction and two feature-propagation layers, producing 1024 seed points with 256-dimensional features.The input has shape N×4 with N=20k, and the output has shape K×(3 + C) with K=1024 and C=256.
  • Before voting, the network fuses lifted image votes with seed points, duplicating seeds that fall into overlapping 2D detection boxes.The fused image-vote features include lifted geometric cues, semantic cues, and texture information.
  • The voting layer maps seed features through an MLP with output sizes 256, 256, and 259.The final layer predicts XYZ vote offsets and feature residuals relative to the 256-dimensional seed feature.
  • The proposal module aggregates generated votes to produce objectness, center, heading, and box-size predictions.Its output dimension is K′×(5+2NH+4NS+NC), where K′ is the number of duplicated seed points.
  • The 2D detector is pretrained, after which the point cloud network is trained with Adam for 140 epochs and learning-rate reductions after 80 and 120 epochs.The initial learning rate is 0.001, with each scheduled reduction multiplying it by 0.1.

B.2. 2D Detector and 2D Cues

The model uses a Faster R-CNN detector and combines detected 2D boxes with semantic and texture cues for image-assisted 3D detection. Training uses detector outputs rather than ground-truth boxes, with additional experiments on alternative region and pixel features.

  • 2D detector training: Faster R-CNN is fine-tuned on SUN RGB-D and achieves 58.5 mAP at overlap 0.5 on the validation set.The detector uses a ResNet-50 backbone with FPN and is pretrained on COCO.
  • 2D boxes: At inference, the system aggregates the top 100 detection boxes across classes and removes boxes with confidence below 0.1.The input image is resized to a shorter side of 600 for 2D box detection.
  • 2D boxes: Training with ground-truth 2D boxes hurts performance, so detector boxes are used instead and randomly dropped with probability 0.5.The passage attributes this to differing train-test detection statistics and aims to reduce overfitting from training-set pretraining.
  • Alternative semantic cues: The default semantic cue is a one-hot classification score vector for each 2D box region, while dense RoI features provide alternative semantic representations.Reported alternatives include 1024-dimensional features from the final fully connected layer before classification and regression.
  • Alternative texture cues: Raw RGB values normalized to [-1,1] serve as the default texture cue, with FPN feature maps explored as spatially preserving alternatives.FPN maps combine top-down and lateral connections and differ in stride and spatial size.

B.3. Image Votes Lifting

Image votes are lifted into the upright 3D coordinate system before fusion with point-cloud features. Because the upright-coordinate network cannot infer camera-axis depth displacement, the pseudo vote is transformed to satisfy the coordinate constraints.

  • Image vote lifting with camera extrinsic: The image-vote lifting process transforms a 2D image vote into a 3D pseudo vote and then applies the camera extrinsic rotation used for the upright point cloud.The transformed pseudo vote is the version used in feature fusion.
  • Image vote lifting with camera extrinsic: In upright coordinates, the point-cloud network estimates displacement along Z_upright but cannot estimate displacement along Z_camera because camera-to-upright rotation angles are unknown.This limitation motivates recalculating the pseudo vote.
  • Image vote lifting with camera extrinsic: FPN features for image-vote lifting are computed from images without the shorter-side-600 resizing used for 2D box detection.
  • Image vote lifting with camera extrinsic: The method computes a new pseudo vote PC′′ whose endpoint lies on the ray OC and whose vote vector is perpendicular to OZ_upright.The construction first transforms P and C′ into upright coordinates before computing C′′.

C. Visualization of Sparse Points

The visualization compares projected point distributions under random and ORB key-point-based sampling at different point-cloud sizes. It illustrates how sampling changes scene coverage and density on SUN RGB-D images.

  • Sparse point cloud visualization: With 20k randomly sampled points, the first example has dense and uniform coverage across the entire scene.The passage introduces subsequent 5k- and 1k-point subsampling for comparison.
  • Sparse point cloud visualization: The visualization projects sampled 3D points onto RGB images to compare their density and distribution across three SUN RGB-D examples.
  • Sparse point cloud visualization: Table 5 compares random sampling at 20k, 5k, and 1k points with ORB key-point-based sampling at 5k and 1k points.ORB sampling retains 3D points projected near detected RGB-image key points.
Loading 2001.10692v1…