PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Sparse Detection Networks for 3D Single-shot Object Detection

JunYoung Gwak, Christopher Choy, Silvio Savarese

arXiv:2006.12356v1cs.CV

TL;DR

Sparse 3D observations sample object surfaces rather than the centers needed for bounding-box anchors. GSDN uses a sparse fully-convolutional network with a generative decoder to expand support for anchors, outperforming the previous state-of-the-art method by 4.2 mAP@0.25 while being 3.78× faster.

  • Problem

    Sparse 3D data is costly to process, and surface samples do not directly observe the object centers where bounding-box anchors are located.

    Source: §1 · p. 2

  • Method

    GSDN uses sparse tensors throughout a fully-convolutional detector, with a decoder that repeatedly expands support using transposed convolutions and pruning.

    Source: §4 · p. 5, §4.2 · p. 6

  • Results

    4.2 mAP@0.25 improvement over the previous state-of-the-art method, while being 3.78× faster.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The sparse fully-convolutional design scales to scenes orders of magnitude larger without cropping them into smaller windows.

    Source: §1 · pp. 2–3

    Qualitative results indicate difficulty detecting thin structures such as bookcases and boards.

    Source: §5.1 · p. 11

Abstract

from arXiv · show

3D object detection has been widely studied due to its potential applicability to many promising areas such as robotics and augmented reality. Yet, the sparse nature of the 3D data poses unique challenges to this task. Most notably, the observable surface of the 3D point clouds is disjoint from the center of the instance to ground the bounding box prediction on. To this end, we propose Generative Sparse Detection Network (GSDN), a fully-convolutional single-shot sparse detection network that efficiently generates the support for object proposals. The key component of our model is a generative sparse tensor decoder, which uses a series of transposed convolutions and pruning layers to expand the support of sparse tensors while discarding unlikely object centers to maintain minimal runtime and memory footprint. GSDN can process unprecedentedly large-scale inputs with a single fully-convolutional feed-forward pass, thus does not require the heuristic post-processing stage that stitches results from sliding windows as other previous methods have. We validate our approach on three 3D indoor datasets including the large-scale 3D indoor reconstruction dataset where our method outperforms the state-of-the-art methods by a relative improvement of 7.14% while being 3.78 times faster than the best prior work.

1 Introduction

Real-time 3D detection matters for robotics and mixed-reality applications, but sparse surface observations do not directly reveal object-center anchor locations. GSDN uses sparse fully-convolutional processing and a generative decoder to address these challenges and scale to large scenes.

1 Introduction

  • 3D data is sparse, incurs cubic processing complexity, and samples object surfaces rather than the centers used for bounding-box anchors.These characteristics make the anchor support disjoint from the observed point support.Source: §1 · p. 2
  • Dense representations require O(N^3) memory and cropping, while point-based MLPs limit point counts or rely on cropping that reduces context.GSDN instead uses a hierarchical sparse tensor encoder to process large scenes fully convolutionally without reducing point density or receptive field through window cropping.Source: §1 · p. 2
  • 78M points across a 13,984 m^3, 53-room building are processed in one feed-forward pass using 5G of GPU memory.The example is the entire S3DIS building 5.Source: Figure 2 · p. 3
  • The generative sparse tensor decoder expands input support to cover bounding-box anchors, then prunes unlikely object centers to limit runtime and memory.This addresses the gap between observed object surfaces and unobserved centers, where anchors are located.Source: §1 · p. 3
  • GSDN outperforms state-of-the-art methods on two large-scale indoor datasets and is faster than the best prior work.The method also demonstrates scalability on 3D scenes orders of magnitude larger.Source: §1 · p. 3

2 Related Work

Prior 3D indoor detectors include sliding-window classifiers, clustering-based approaches, and bounding-box proposal methods. GSDN is distinguished by maintaining spatial sparsity while generating anchors for detection.

2 Related Work

  • Indoor detection approaches include sliding-window classification, clustering-based methods, and bounding-box proposal methods.Examples of clustering methods predict object centers per point and cluster center votes [17].Source: §2 · pp. 3–4
  • Bounding-box proposal methods adapt 2D rectangular proposals to 3D; GSDN maintains sparsity across input and anchor spaces.The paper contrasts its anchor-generating process with prior work focused on object shapes and sparse generative methods that subdivide voxels.Source: §2 · p. 4
  • Sparse tensor networks process spatially sparse feature maps, unlike compressed networks whose weights are sparse but feature maps remain dense.The paper adopts spatially sparse networks to scale detection to greater depth and extremely large scenes.Source: §2 · p. 4

3 Preliminaries

The preliminaries establish sparse-tensor notation and representation, then describe how 3D surfaces are discretized for use as network input.

3 Preliminaries

  • The preliminaries introduce sparse-tensor operations and notation conventions for scalars, constants, vectors, matrices, tensors, and sets.The stated conventions distinguish variable scalars from constants and use different forms for vectors, matrices, tensors, and sets.Source: §3 · pp. 4–5

3.1 Sparse Tensor

  • A sparse tensor is mostly zero and can be represented by its nonzero coordinates and corresponding features.For 3D data, the coordinates and features can be represented as matrices in COO format.Source: §3.1 · p. 5

3.2 Sparse Tensor for 3D Data Representation

  • Point clouds and meshes become sparse tensors by discretizing their surface coordinates; voxel size is a hyperparameter that affects network performance.The discretized coordinates represent vertices or points on the 3D surface.Source: §3.2 · p. 5

4 Generative Sparse Detection Networks

GSDN processes sparse tensors throughout, pairing hierarchical feature extraction with a decoder that expands support for object proposals.

4 Generative Sparse Detection Networks

  • The network uses sparse tensors throughout, with a hierarchical encoder extracting features and a generative decoder proposing new supports.The encoder's features capture object geometry and identity; the decoder proposes supports based on those features.Source: §4 · p. 5

4.1 Hierarchical Sparse Tensor Encoder

  • Residual blocks and strided convolutions downsample the input into hierarchical feature maps while increasing receptive field size.The maps are cached for use by the decoder.Source: §4.1 · p. 6

4.2 Generative Sparse Tensor Decoder

  • Transposed convolutions expand sparse support, then pruning removes coordinates unlikely to contain anchors to limit memory and computation.Kernels larger than 2 expand support beyond simple upsampling; skip connections combine decoder features with cached encoder maps to recover fine details.Source: §4.2 · p. 6, §4.2.1 · p. 7, §4.2.2 · pp. 7–8

4.3 Multi-scale Bounding Box Anchor Prediction

  • Predictions at multiple decoder stages capture object scale variation, with anchors doubling in size at each successive level.Each anchor predicts an object likelihood, six offsets, and semantic class scores.Source: §4.3 · p. 8

4.4 Summary of GSDN Feed Forward

  • The feed-forward pass builds hierarchical feature maps, then iteratively predicts anchors, prunes voxels, and upsamples sparse tensors.Upsampled tensors are added to corresponding encoder feature maps.Source: §4.4 · p. 8

4.5 Losses

  • Balanced cross entropy trains sparsity and anchor predictions; semantic classification and box regression use cross entropy and Huber losses in a weighted sum.Anchors are positive above 3D IoU 0.35 and negative below 0.2; the intermediate range is not assigned anchor loss.Source: §4.5 · pp. 8–9

4.6 Prediction post-processing

  • Non-maximum suppression filters overlapping predictions, which are merged using a score-weighted average of their bounding boxes.The network is trained to overestimate the number of anchors by labeling anchors with 3D IoU > 0.35 as positive.Source: §4.6 · p. 9

5 Experiments

Experiments evaluate GSDN on ScanNet, S3DIS, and Gibson, covering detection performance, efficiency, and behavior on large scenes. Results include benchmark gains, whole-scene processing, and tests beyond the training scale and setting.

5.1 Object detection performance analysis

  • 4.2% mAP@0.25 and 1.3% mAP@0.5 gains over two-stage baselines accompany GSDN’s single-shot result; it also beats prior state of the art on most semantic classes.The ScanNet v2 validation tables report overall and class-wise detection results.Source: Table 1 · p. 10, Table 2 · p. 10, §5.1 · p. 11
  • On S3DIS, GSDN processes whole scenes, unlike methods that crop scenes into 1m×1m floor areas and heuristically merge results, restricting receptive fields and slowing pre/post-processing.The method takes the whole scene as input rather than using those crops and merges.Source: §5.1 · p. 11
  • Thin structures such as bookcases and boards remain difficult; the authors suggest more extreme-shaped anchors as a possible remedy.The qualitative results identify these structures as detection challenges.Source: §5.1 · p. 11

5.2 Speed and Memory Analysis

  • 0.12 seconds versus 0.45 seconds per ScanNetV2 scene, GSDN is 3.78× faster than Qi et al. and maintains low memory use as network depth grows.The memory comparison is against a dense object detector on ScanNetV2 validation.Source: §5.2 · pp. 12–13, Figure 8 · p. 13, Figure 9 · p. 14
  • Runtime grows linearly with point count and sublinearly with floor area; unlike Qi et al., GSDN maintains constant point density as scenes grow.Qi et al. sample a constant number of points regardless of input size, so their point density changes with scene size.Source: §5.2 · pp. 12–13, Figure 8 · p. 13, Figure 9 · p. 14

5.3 Scalability and generalization of GSDN on extremely large inputs

  • On S3DIS building 5, GSDN processed 78M points across 53 rooms in 20 seconds, using 5G GPU memory to detect 573 object instances.The reported time includes data preprocessing and post-processing.Source: §5.3 · p. 14
  • Trained on single-floor ScanNet scans, GSDN generalized to Gibson’s three-story Uvalda building without ad-hoc preprocessing or post-processing.The building covers 173m2 of floor area; processing took 2.2 seconds and used 1.8G GPU memory to detect 129 instances.Source: §5.3 · p. 14

6 Conclusion

GSDN maintains sparsity while generating object centers, enabling fully-convolutional detection on large point clouds without cropping scenes into smaller windows. It exceeds the previous state-of-the-art on mAP@0.25 and processes a 17-room building in one feed-forward pass.

6 Conclusion

  • 4.2 mAP@0.25 higher than the previous state-of-the-art, while GSDN is 3.78× faster than the best prior work.The method maintains sparsity by generating object centers with its generative sparse tensor decoder.Source: §6 · p. 14
  • GSDN processes large-scale point clouds without cropping scenes into smaller windows, taking advantage of the full receptive field.The network maintains sparsity throughout and generates object centers using its generative sparse tensor decoder.Source: §6 · p. 14
  • GSDN processes a 17-room building with 1.4M points in a single fully-convolutional feed-forward pass.The demonstrated scene is Uvalda, a Gibson environment.Source: Figure 10 · p. 15

S.1 Controlled experiments and analysis

The controlled experiments use a shared ScanNet training and validation setup, varying one control variable at a time.

S.1 Controlled experiments and analysis

  • All controlled experiments use the same architecture, ScanNet training and validation data, and 60k training iterations, changing one control variable at a time.The 60k-iteration models perform below the 120k-iteration models reported in the main paper.Source: §S.1 · p. 18

S.1.1 Balanced cross entropy loss

Balanced cross entropy is evaluated for sparsity and anchor prediction under heavy label imbalance. It improves performance, especially sparsity prediction.

S.1.1 Balanced cross entropy loss

  • Balanced cross entropy improves GSDN performance, especially sparsity prediction, compared with regular cross entropy for sparsity and anchor prediction.Table 4 presents the comparison of balanced cross entropy (BCE) and cross entropy (CE) losses.Source: §S.1.1 · p. 18, Table 4 · p. 18
  • The decoder adds cubically growing coordinates from surface voxels, most of which must be pruned, motivating balanced loss for sparsity and anchor prediction.The authors identify heavy class imbalance in both sparsity and anchor labels.Source: §S.1.1 · p. 18

S.1.2 Sparsity pruning confidence

The pruning confidence threshold affects proposal counts, recall, average precision, and decoder runtime. Lower thresholds generally perform better, while higher thresholds modestly reduce runtime.

S.1.2 Sparsity pruning confidence

  • Three models trained with τ = {0.1, 0.3, 0.5} were tested at τ = {0.1, 0.2, . . . , 0.9}, using the closest-trained threshold when needed.The analysis covers average recall, mAP@0.25, and decoder runtime on ScanNet.Source: §S.1.2 · pp. 18–19
  • Lower pruning confidences yield more proposals and higher average recall; average precision follows a similar trend, while higher confidence runs the decoder marginally faster.The authors attribute the runtime trend to generating fewer proposals at higher pruning confidences.Source: Figure 11 · p. 19, §S.1.2 · p. 19

S.1.3 Encoder backbone models

Across backbone, anchor-ratio, and dataset experiments, GSDN’s results vary with encoder depth and anchor coverage. The reported tests also show ScanNet performance against a two-stage detector, S3DIS class-level recall challenges, and transfer to Gibson environments.

S.1.3 Encoder backbone models

  • GSDN with a deeper encoder performs better, while the encoder dominates runtime and decoder and post-processing times stay almost constant across backbones.The runtime breakdown is shown in Figure 12.Source: §S.1.3 · p. 19, Table 5 · p. 19, Figure 12 · p. 20

S.1.4 Anchor ratios

  • More anchors marginally improve mAP@0.25 but significantly improve mAP@0.5, which requires approximately 80% overlap on each axis.The authors attribute the stricter-threshold gain to better capture of varied ground-truth boxes.Source: §S.1.4 · pp. 19–20, Table 6 · p. 20

S.2.1 Experiments on the ScanNet dataset

  • On the ScanNet v2 validation set, GSDN outperforms the two-stage detector of Qi et al. despite being single-shot.Table 7 reports class-wise mAP@0.5 results; Figures 13 and 14 show qualitative results.Source: §S.2.1 · p. 20, Table 7 · p. 21, Figure 13 · p. 21

S.2.2 Stanford Large-Scale 3D Indoor Spaces Dataset

  • On S3DIS building 5, classes with extreme bounding-box ratios, such as board and bookcase, tend to have very low recall.Figure 16 visualizes per-class precision/recall curves, and Figure 15 shows additional qualitative results.Source: Figure 15 · p. 23, §S.2.2 · p. 24, Figure 16 · p. 24

S.2.3 Gibson environment

  • A model trained on single-story ScanNet rooms generalizes to multi-story, multi-room Gibson scenes without sliding-window cropping and stitching.Across 572 Gibson V2 environments, runtime and GPU memory grow linearly with input points and sublinearly with point-cloud volume (Figure 17); Figure 18 shows qualitative results.Source: §S.2.3 · p. 24, Figure 17 · p. 25, Figure 18 · p. 25
Loading 2006.12356v1…