PaperlayerView exact revision on arXiv

Source-linked AI summary

Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, Dong Xu

arXiv:2104.06114v2cs.CV

TL;DR

Voting-based point-cloud detection can suffer partial object-surface coverage and clutter outliers. BRNet back-traces representative points from vote centers and revisits nearby seeds; it outperforms prior methods on ScanNet V2 and SUN RGB-D.

  • Problem

    Voting-based 3D detection can receive votes with partial object-surface coverage and clutter-background outliers, while point clouds are irregular, sparse, and orderless.

    Source: §1 · p. 1

  • Method

    BRNet generatively back-traces representative points from vote centers, revisits nearby seed points, and fuses their features with vote-center features to refine proposals.

    Source: §1 · p. 2

  • Results

    BRNet outperforms baseline methods on ScanNet V2 and SUN RGB-D, with more than 7.5% and 4.7% improvement in mAP@0.50, respectively.

    Source: §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The paper reports that BRNet brings negligible increases in model size and execution time compared with VoteNet.

    Source: §5 · p. 8

Abstract

from arXiv · show

3D object detection in point clouds is a challenging vision task that benefits various applications for understanding the 3D visual world. Lots of recent research focuses on how to exploit end-to-end trainable Hough voting for generating object proposals. However, the current voting strategy can only receive partial votes from the surfaces of potential objects together with severe outlier votes from the cluttered backgrounds, which hampers full utilization of the information from the input point clouds. Inspired by the back-tracing strategy in the conventional Hough voting methods, in this work, we introduce a new 3D object detection method, named as Back-tracing Representative Points Network (BRNet), which generatively back-traces the representative points from the vote centers and also revisits complementary seed points around these generated points, so as to better capture the fine local structural features surrounding the potential objects from the raw point clouds. Therefore, this bottom-up and then top-down strategy in our BRNet enforces mutual consistency between the predicted vote centers and the raw surface points and thus achieves more reliable and flexible object localization and class prediction results. Our BRNet is simple but effective, which significantly outperforms the state-of-the-art methods on two large-scale point cloud datasets, ScanNet V2 (+7.5% in terms of mAP@0.50) and SUN RGB-D (+4.7% in terms of mAP@0.50), while it is still lightweight and efficient. Code will be available at https://github.com/cheng052/BRNet.

1. Introduction

Voting-based 3D detection faces sparse, irregular point clouds and vote clusters whose corresponding seeds may miss object surfaces or include background clutter. BRNet responds by generatively back-tracing representative points and revisiting nearby seeds, then reports state-of-the-art results on two benchmarks.

1. Introduction

  • Irregular, sparse, orderless point clouds make reliable point-based 3D object detection difficult.The task predicts amodal 3D bounding boxes and associated semantic labels in real scenes.Source: §1 · p. 1
  • VoteNet clusters point-wise votes to form proposals, but their seed points can cover only part of an object or include clutter outliers.These seed points can make proposal features less informative for bounding-box prediction and object detection.Source: §1 · pp. 1–2
  • BRNet generatively back-traces representative points from vote-cluster centers and revisits nearby seed points for complementary local structural clues.The paper presents this bottom-up and then top-down interaction as a way to associate seed and vote features.Source: §1 · p. 2, Figure 2 · p. 2
  • Its learnable modules generate class-agnostic representative points, aggregate revisited seed features, and fuse them with vote-center features to refine proposals.The proposed bounding-box regression depends on the representatives’ spatial distribution, which the paper says improves robustness to shape variations.Source: §1 · p. 2
  • 50.9% mAP@0.50 on ScanNet V2 and 43.7% mAP@0.50 on SUN RGB-D are BRNet’s reported state-of-the-art results.The paper identifies these as its results on the two benchmark datasets.Source: §1 · p. 2

2. Related Works

Prior 3D detection work moved from regularized or template-based inputs toward raw-point networks and voting-based proposals. BRNet draws on anchor-free localization and back-tracing ideas while addressing vote-clustering weaknesses with generative representative points.

2. Related Works

  • Earlier 3D detectors used regular grids, RGB-driven proposals, segmentation hypotheses, shape templates, contextual potentials, sliding shapes, or oriented gradients.The cited approaches include multi-view images, voxel grids [30], RGB-driven proposals, segmentation hypotheses, shape templates, contextual potentials, sliding shapes [33, 32], and COG.Source: §2 · p. 2
  • PointRCNN introduced a two-stage detector, while VoteNet applied end-to-end trainable Hough voting to raw point clouds.VoteNet was built on PointNet++ [23].Source: §2 · p. 2
  • VoteNet extensions add context, hierarchical graphs, or feature-FPS sampling, but their vote clustering can retain outliers and overlook inlier seeds.H3DNet adds geometric primitives, yet their centers use less accurate supervision and similar clustering; BRNet instead uses back-traced representative points.Source: §2 · p. 3
  • BRNet borrows anchor-free, class-agnostic offset regression to infer likely shape profiles, unlike VoteNet ’s localization constrained by class-aware statistics.Related 2D approaches regress borders, use iterative dynamic sampling, or regress offsets.Source: §2 · p. 3
  • Earlier methods combined top-down hypotheses with bottom-up voting or MRFs; 3D-MPA directly back-traces vote-associated surface points, unlike BRNet’s generative approach.The related-work discussion describes mutual agreement between top-down and bottom-up strategies as supporting more reliable detection.Source: §2 · p. 3

3. Methodology

BRNet combines vote generation with generatively back-traced representative points and revisited seed features, then refines proposals for object localization and classification. Its design addresses the partial object coverage and background outliers associated with conventional back-tracing of votes.

3.1. Overview

  • BRNet uses vote generation and clustering, representative-point generation, seed-point revisiting, and proposal refinement and classification before standard 3D NMS.The input is a point cloud, and the output is a collection of oriented bounding boxes with category labels.Source: §3.1 · p. 3

3.2. Generating Back-traced Representative Points

  • The representative-point generation module predicts a potential object's orientation and six surface offsets from each vote center, then samples 12 points along the oriented directions.This generative strategy is proposed because conventional back-tracing can yield incomplete object coverage and background outliers.Source: §3.2 · pp. 3–4

3.3. Revisiting Seed Points

  • For each back-traced point, BRNet revisits nearby seed points and aggregates their features to capture local object-level structure from the raw point cloud.The resulting seed features are combined into a feature for each vote center.Source: §3.3 · pp. 4–5

3.4. Proposal Refinement and Classification

  • BRNet fuses revisited seed features with vote-center features to predict residual corrections to representative-point offsets and orientation, as well as objectness and semantic-class scores.The refined representative points are min-max clipped to obtain bounding-box sizes.Source: §3.4 · p. 5

3.5. The Learning Objective

  • The framework's objective combines vote regression, objectness, semantic classification, representative-point, and proposal-refinement losses.Proposal refinement is supervised on positive vote clusters.Source: §3.5 · p. 5

4. Experiments

Experiments on SUN RGB-D and ScanNet V2 compare BRNet with prior detectors and test its components. BRNet improves mAP@0.50 on both datasets, while ablations examine class-agnostic regression, back-tracing and revisiting, and efficiency.

4.1. Setups and Implementation Details

  • The evaluation uses SUN RGB-D single-view scans and ScanNet V2 reconstructed scenes, with 20,000 and 40,000 input points, respectively.Both datasets use the VoteNet training/validation split and are evaluated with mAP@0.25 and mAP@0.50.Source: §4.1 · p. 5

4.2. Comparisons with the State-of-the-art Methods

  • More than 7.5% and 4.7% improvement in mAP@0.50 on the ScanNet V2 and SUN RGB-D validation sets, respectively, over baseline methods.The comparison covers average precision at 3D IoU thresholds of 0.25 and 0.50.Source: §4.2 · p. 6, Table 1 · p. 6
  • BRNet performs best on 12 of 18 ScanNet classes at mAP@0.50 and outperforms H3DNet despite using one PointNet++ backbone instead of four.It also performs better on categories with irregular sizes or shapes, including chairs, sofas, tables, counters, and desks.Source: §4.2 · p. 6, Table 2 · p. 6
  • Qualitative comparisons show more accurate boxes and orientations, fewer false positives, and more detected objects for BRNet than the baselines.The ScanNet V2 and SUN RGB-D visualizations compare BRNet with VoteNet, MLCVNet, and H3DNet where available.Source: §4.2 · p. 6, Figure 4 · p. 7, Figure 6 · p. 8

4.3. Ablation Study and Discussions

  • VoteNet with class-agnostic regression significantly outperforms VoteNet, with SUN RGB-D mAP@0.50 gains positively related to intra-category size variance.The comparison is presented as VoteNet+CA-Reg; Figure 5 plots relative accuracy gains and normalized size variance by category.Source: §4.3 · p. 7, Figure 5 · p. 7, Table 3 · p. 7
  • The complete back-tracing, revisiting, and refinement method gains about 10% mAP on ScanNet and about 6% on SUN RGB-D at mAP@0.50.Revisited seed points cover object surfaces more compactly than corresponding vote seed points, which may be partial and include outliers.Source: §4.3 · p. 7, Figure 7 · p. 8
  • Fusing votes with their corresponding seed-point features yields non-trivial gains on both datasets, especially on ScanNet V2 at mAP@0.50.Source: §4.3 · pp. 7–8
  • BRNet is 3× faster than H3DNet on both datasets and has a model size around 4× smaller, with only a marginal increase over VoteNet.The comparison was run on the same NVIDIA GeForce RTX 2080 Ti GPU configuration.Source: §4.3 · p. 8, Table 5 · p. 8

5. Conclusion

BRNet improves voting-based 3D detection by back-tracing representative points and revisiting nearby seed points to extract object-surface features. Ablations and evaluation results support the effectiveness of these operations, with negligible increases in model size and execution time compared with VoteNet.

5. Conclusion

  • BRNet generatively back-traces representative points without class-specific sampling and revisits nearby seed points to extract fine object-surface features for high-quality proposals.The method also refines proposals through back-tracing, revisiting, and refinement operations.Source: §5 · p. 8
  • Ablation studies support the importance and effectiveness of back-tracing, seed-point revisiting, and proposal refinement.Source: §5 · p. 8
  • BRNet outperforms existing methods while adding negligible model size and execution time compared with VoteNet.Source: §5 · p. 8

A. Supplementary

The supplementary material provides further quantitative and qualitative results, followed by implementation details.

A. Supplementary

A.1. More Quantitative Results

The supplementary evaluates BRNet at multiple IoU thresholds and reports comparisons with H3DNet, including category-specific gains on SUN RGB-D.

A.1. More Quantitative Results

  • −1.1%, 2.8%, and 3.7% are BRNet’s respective mAP@0.25, mAP@0.50, and mAP@0.75 increases over H3DNet on ScanNetV2.The comparison uses four PointNet++ backbones for both methods, but BRNet uses 20,000 input points and H3DNet uses 40,000.Source: §A.1 · p. 11, Table S1 · p. 11
  • BRNet performs best on mAP@0.50 and mAP@0.75 among the evaluated methods, with mAP@0.75 requiring more than 90% coverage in each bounding-box dimension.The supplementary evaluates VoteNet, HGNet, MLCVNet, H3DNet, and BRNet at mAP@0.25, mAP@0.50, and mAP@0.75.Source: §A.1 · p. 11
  • The SUN RGB-D comparison table reports results at multiple IoU thresholds and notes that H3DNet uses 40,000 input points while other methods use 20,000.Source: Table S2 · p. 11
  • 7.9%, 10.9%, 6.7%, 7.6%, and 6.3% are BRNet’s respective gains over H3DNet for Bathtub, Bed, Dresser, Nightstand, and Sofa on SUN RGB-D.Source: §A.1 · p. 11

A.2. More Qualitative Results

Qualitative comparisons report compact, high-quality BRNet boxes on ScanNet V2 and SUN RGB-D, while also identifying false positives and missed smooth-wall objects as failure cases.

A.2. More Qualitative Results

  • BRNet produces high-quality, compact predicted boxes compared with VoteNet, MLCVNet, and H3DNet on ScanNet V2 and SUN RGB-D.Source: §A.2 · p. 11
  • BRNet can produce false positives on hollow floors and miss objects on smooth walls, especially windows and pictures.The authors say these cases are also common and difficult for the reference methods.Source: §A.2 · p. 11
  • The authors identify reducing false positives in sparse point regions and improving robustness in cluttered backgrounds as future directions.Source: §A.2 · p. 11

A.3. Implementation Details

Implementation follows VoteNet for vote generation and clustering, then generates representative points, revisits nearby seed points, and refines proposals. Evaluation tables specify validation datasets, IoU thresholds, and comparison conditions.

A.3. Implementation Details

  • BRNet uses VoteNet’s network architecture and vote regression loss for vote generation and clustering.The four-module pipeline continues with representative-point generation, seed-point revisiting, proposal refinement and classification, and 3D NMS.Source: §A.3 · p. 11
  • 12 representative points per proposal are generated by sampling two points on each skewed direction from six predicted surface-distance offsets.The generation MLP predicts six offsets from the vote point to the object surface and heading-bin estimates.Source: §A.3 · p. 11
  • Seed features within 0.2 m of each representative point are aggregated, ordered into a 384-dimensional proposal feature, and projected to 128 dimensions.The 128-dimensional revisited-seed feature is concatenated with a 128-dimensional vote-cluster feature for proposal refinement and classification.Source: §A.3 · pp. 11–12
  • The ScanNet V2 and SUN RGB-D validation tables report AP at 3D IoU 0.25; SUN RGB-D uses V1 data, and H3DNet uses four backbones.H3DNet also subsamples 40,000 SUN RGB-D points per scene, versus 20,000 for other methods; its ScanNet per-category results likewise use four backbones.Source: Table S3 · p. 12, Table S4 · p. 12
Loading 2104.06114v2…