PaperlayerView exact revision on arXiv

Source-linked AI summary

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Qiangeng Xu, Yiqi Zhong, Ulrich Neumann

arXiv:2112.02205v1cs.CVcs.AIcs.LGcs.RO

TL;DR

LiDAR point clouds can miss most parts of objects because of occlusion and signal miss, making complete object shapes unavailable for 3D detection. BtcDet learns shape priors, estimates occupancy in affected regions, and integrates it into proposal generation and refinement; experiments on KITTI and Waymo show it surpasses published state-of-the-art methods by remarkable margins.

  • Problem

    Occlusion and signal miss leave point clouds with only partial object shapes, raising questions about the causes and detection impact of this shape miss.

  • Method

    BtcDet learns underlying shape priors, estimates complete-shape occupancy in regions affected by occlusion and signal miss, and uses it for proposal generation and refinement.

  • Results

    BtcDet surpasses all published state-of-the-art methods by remarkable margins on the KITTI and Waymo Open Dataset experiments.

  • Takeaways & Limitations

    Learning occluded shapes benefits both region proposal generation and proposal refinement within the detection pipeline.

  • Takeaways & Limitations

    The shape approximation assumes a limited set of shape prototypes and rough symmetry, with mirroring used for cars and cyclists.

Abstract

from arXiv · show

Advances in LiDAR sensors provide rich 3D data that supports 3D scene understanding. However, due to occlusion and signal miss, LiDAR point clouds are in practice 2.5D as they cover only partial underlying shapes, which poses a fundamental challenge to 3D perception. To tackle the challenge, we present a novel LiDAR-based 3D object detection model, dubbed Behind the Curtain Detector (BtcDet), which learns the object shape priors and estimates the complete object shapes that are partially occluded (curtained) in point clouds. BtcDet first identifies the regions that are affected by occlusion and signal miss. In these regions, our model predicts the probability of occupancy that indicates if a region contains object shapes. Integrated with this probability map, BtcDet can generate high-quality 3D proposals. Finally, the probability of occupancy is also integrated into a proposal refinement module to generate the final bounding boxes. Extensive experiments on the KITTI Dataset and the Waymo Open Dataset demonstrate the effectiveness of BtcDet. Particularly, for the 3D detection of both cars and cyclists on the KITTI benchmark, BtcDet surpasses all of the published state-of-the-art methods by remarkable margins. Code is released (https://github.com/Xharlie/BtcDet}{https://github.com/Xharlie/BtcDet).

1 Introduction

LiDAR point clouds are effectively 2.5D because occlusion and signal miss remove parts of object shapes, complicating 3D detection. BtcDet addresses this by learning shape priors and estimating missing occupancy, with recovery experiments showing large benefits for heavily occluded objects.

  • LiDAR frames are technically 2.5D because returning laser beams leave shapes behind occluders missing from the point cloud.
  • Shape miss makes highly occluded objects harder to detect, motivating recovery of missing object shapes before or during detection.The study evaluates this by filling missing points from similar objects and retraining PV-RCNN under controlled recovery scenarios.
  • Shape miss arises from external occlusion, signal miss, and self-occlusion in LiDAR point clouds.External occlusion blocks beams, signal miss prevents returns from certain materials or angles, and self-occlusion hides far-side object parts.
  • APs > 99% were achieved across almost all occlusion levels when external, signal, and self-occlusion shape miss were all recovered.Without recovery, higher occlusion levels were more difficult; recovering shapes reduced performance gaps between occlusion levels.
  • BtcDet estimates occupancy for complete object shapes in regions affected by occlusion and signal miss, integrating it into proposal generation and refinement.The method targets test-time shape recovery without requiring bounding-box labels.

2 Related Work

Prior LiDAR detectors extract voxel- or point-based features and supervise bounding boxes, while shape-aware methods preserve structure or complete point clouds without explicitly modeling occlusion's effect on shapes. BtcDet targets this gap by learning occluded shapes in point-cloud data.

  • Voxel- and point-based LiDAR detectors extract features or generate proposals but supervise bounding boxes without explicitly considering object shapes.
  • BtcDet identifies occlusion and signal-miss regions, estimates shape occupancy probability, and feeds that information into proposal generation and refinement.
  • Shape-aware detectors learn part locations, structural information, or point-cloud completions, but overlook how occlusion affects object shapes.
  • Prior occlusion-handling work mainly addresses 2D images, while point-cloud visibility methods do not learn its influence on object shapes.BtcDet is presented as the first 3D object detector that learns occluded shapes in point-cloud data.

3 Behind the Curtain Detector

BtcDet learns complete object-shape occupancy in regions affected by occlusion and signal miss, then integrates those estimates into proposal generation and refinement. It approximates complete shapes from similar objects, predicts occupancy probabilities, and uses them to improve 3D detection features and bounding-box regression.

  • Motivation: Existing detectors supervise boxes or observed-shape statistics but do not explicitly model complete object shapes S = Sob ∪ Soc.BtcDet instead predicts occupancy for complete shapes in regions affected by occlusion and signal miss.
  • Shape Approximation: Complete-shape targets are approximated by combining an object’s points with missing-region points borrowed from the top three similar source objects.The approximation relies on limited shape prototypes and approximate symmetry, including mirroring car and cyclist points across the box middle plane; borrowed points receive δ < 1 weighting.
  • Region Identification: BtcDet identifies occlusion and signal-miss regions with spherical voxelization, including empty voxels behind observed points and voxels projecting to signal-miss boundaries.Spherical coordinates allow occluded regions behind LiDAR points to be formed accurately.
  • Occupancy Learning: A shape occupancy network predicts P(OS) for spherical voxels in ROC ∪ RSM, using binary labels for voxels containing or not containing the approximated complete shape.Occupancy is used instead of point generation because rasterization avoids inconsistent borrowed-point density and point-generation plausibility issues.
  • Detection Integration: The predicted occupancy map is transformed into Cartesian coordinates and fused with backbone features for region proposals and proposal refinement.For refinement, occupancy helps regression avoid irrelevant regions and indicates unobserved shapes in empty regions with high occupancy probability.
  • Detection Integration: The RPN generates 3D proposals containing center, box size, heading, and confidence, while the refinement module produces final bounding-box predictions from local geometric features.These local features combine occupancy with multi-scale backbone features pooled onto grids covering each proposal box.

4 Experiments

BtcDet is evaluated on KITTI and Waymo using benchmark comparisons and ablations of shape occupancy and feature integration. It outperforms published detectors across the reported datasets, while ablations identify effective shape representations and integration choices.

  • KITTI results: 2.05% 3D R11 AP separates BtcDet from other models on moderate KITTI validation cars, making it the first detector above 86% on this metric.The comparison covers cars, pedestrians, and cyclists across all object classes and difficulty levels.
  • KITTI results: BtcDet surpasses published leaderboard-associated methods on KITTI car and cyclist detection, including methods using LiDAR alone and methods combining LiDAR with RGB.The KITTI test comparison was reported as of May 4, 2021.
  • Waymo results: 2.99% and 3.51% are BtcDet’s reported improvements over other detectors on Waymo LEVEL 1 and LEVEL 2 3D mAP, respectively.The model outperforms state-of-the-art detectors across all distance ranges and difficulty levels; larger gains occur for LEVEL 2 objects with fewer points.
  • Shape features: 2.86 AP is the improvement contributed by the selected shape features over baseline BtcDet1 in the shape-feature ablation.Binary occupancy maps provide limited improvement, Cartesian probability prediction misrepresents spherical regions, and hard occupancy removes likelihood information.
  • Integration strategies: 0.8 final prediction AP comes from occupancy integration for proposal refinement alone, while backbone integration alone improves proposal-box AP by 1.2 and final-box AP by 2.0.The former primarily affects refinement, whereas integration with Ψ primarily affects proposal generation.
  • Integration strategies: Integrating P(OS)⊥ with Ψ’s first two layers is best; adding it to Ψ’s third layer causes regression because the features differ in level.The occupancy and feature integrations each improve performance independently, and together help BtcDet surpass state-of-the-art models.

5 Conclusion and Future Work

BtcDet addresses shape miss from occlusion and signal miss by learning shape priors and estimating complete object-shape occupancy, improving detection across evaluated datasets. The authors report strong benchmark performance, while identifying model efficiency as future work.

  • 5 Conclusion and Future Work: BtcDet targets object shapes affected by occlusion and signal miss, using learned shape priors to estimate complete object-shape occupancy.The method integrates occupancy estimation into both region proposal generation and proposal refinement.
  • 5 Conclusion and Future Work: Ablation studies examine learned features and feature-fusion choices, including which Ψ layers are fused with P(OS)⊥ and whether P(OS)⊥ enters fgeo.The studies evaluate these choices on KITTI moderate car objects using proposal and final-box 3D APR11.
  • 5 Conclusion and Future Work: BtcDet surpasses all published state-of-the-art methods by remarkable margins on the KITTI and Waymo Open datasets.The conclusion attributes the result to integrating occupancy estimation into the detection pipeline.
  • 5 Conclusion and Future Work: Improving model efficiency remains an open direction, particularly through faster occlusion identification and shape learning.The authors present this as a promising future direction after demonstrating benefits from learning occluded shapes.

A Data and Code License

The experiments use the KITTI and Waymo Open datasets, which are licensed for academic research, while the code is released under Apache License 2.0.

  • A Data and Code License: Experiments use the KITTI Dataset and the Waymo Open Dataset, both described as well-known and licensed for academic research.These datasets support the paper’s experimental evaluation.
  • A Data and Code License: The code is licensed under Apache License 2.0 and is stated to be released.The passage specifies the software license rather than dataset licensing terms.
  • A Data and Code License: The heuristic H(A, B) evaluates whether source object B can provide points for shape-miss regions of target object A; lower scores are better.It combines point coverage, bounding-box-size similarity, and extra voxels contributed by B.

C.1 Region Proposal Network (RPN)

The RPN uses anchor-based 3D detection with class-specific average-object anchors, focal classification, box regression, direction classification, and IoU-based assignment rules. Proposal refinement uses separate confidence and regression branches with IoU-weighted targets.

  • C.1 Region Proposal Network (RPN): The RPN sets per-class anchors to average 3D object sizes and uses orientations of 0° and 90° to generate region proposals.It adopts the box encoding introduced in prior anchor-based 3D detectors.
  • C.1 Region Proposal Network (RPN): The box encodings represent centers, dimensions, and yaw angles, with subscripts distinguishing encoded values, anchors, ground truth, residues, and proposals.The orientation loss is paired with a direction classifier because opposite directions are otherwise indistinguishable.
  • C.1 Region Proposal Network (RPN): Car and vehicle anchors are foreground above 0.6 IoU and background below 0.45, while pedestrian and cyclist thresholds are 0.5 and 0.35.Anchors between the thresholds are ignored during training.
  • C.1 Region Proposal Network (RPN): The RPN uses focal loss with α = 0.25 and γ = 2, plus SmoothL1 box regression and direction classification losses.Regression applies to foreground anchors, and direction classification predicts the angle bin.
  • C.1 Region Proposal Network (RPN): Proposal refinement has separate confidence and box-regression branches and uses a 3D IoU-weighted confidence target for each RoI.Regression loss is applied only to positive proposals with IoU ≥ 0.55.

C.3 Total Loss

BtcDet combines the shape loss with the RPN and proposal-refinement losses, weighting the shape loss by 0.3. A grid search identified this weighting as producing the best results.

  • C.3 Total Loss: The total loss is Ltotal = 0.3Lshape + Lrpn + Lpr.It combines the shape, region-proposal, and proposal-refinement losses.
  • C.3 Total Loss: A grid search found that the shape-loss weight 0.3 helps BtcDet achieve the best results.The passage reports the selected weighting factor without specifying a separate metric.

D Network Architecture

BtcDet combines sparse 3D feature extraction, occupancy probabilities, bird’s-eye-view proposal generation, and shifted local-grid refinement in a compact detection architecture.

  • Shape occupancy network: The shape occupancy network uses five spherical sparse-convolution layers, including two down-sampling and two up-sampling stages.Output feature dimensions are 16, 32, 64, 32, and 32.
  • Detection feature backbone: The detection backbone voxelizes points in Cartesian coordinates and concatenates transformed occupancy probabilities as two additional channels.The channels represent occupancy probability and whether occupancy probability exists in a voxel.
  • Region proposal network: Bird’s-eye-view features feed thinner 2D convolutional networks that predict residues for two anchors per class per grid.The intermediate feature dimension is 128 rather than 256 in the cited prior networks.
  • Proposal refinement network: Proposal refinement aggregates pooled local geometric features from grids shifted around each proposal center.The network uses 3 × 3 × 3 shifted local grids, with λ = 0.25 reported as best.

E Occupancy Estimation of Complete Object Shapes

BtcDet’s occupancy estimation predicts complete-shape occupancy in affected voxels and supplies probabilities used by downstream detection components.

  • Evaluation: Occupancy estimation is evaluated over voxels in the union of occlusion and signal-miss regions using precision, recall, F1 score, accuracy, and object coverage.Positive voxels satisfy P(OS) > threshold, with thresholds of 0.3, 0.5, and 0.7.
  • Evaluation: Accuracy exceeds 99% under all thresholds, while relatively high object coverage indicates occupancy estimates are sufficiently faithful for downstream networks.The paper attributes the high accuracy to extreme class imbalance.
  • Architecture: The shape occupancy network is implemented as a lightweight spherical sparse 3D convolutional architecture.The architecture is shown in Figure 6 alongside the sparse-convolution layer types used.
  • Architecture: The detection feature backbone uses regular and sub-manifold sparse 3D convolutions to process the detection features.Its architecture is shown in Figure 7.

F More Comparison Results on the KITTI Test Set

The KITTI test-set analysis examines comparisons, occupancy visualization, shape assembly, and pedestrian limitations, while supplementary figures illustrate the model’s occupancy predictions.

  • Proposal refinement: The proposal-refinement architecture uses sparse 3D convolutions, fully connected layers, and nearby-point aggregation.The stride numbers correspond to the Z, Y, and X axes.
  • Occupancy evaluation: The occupancy-estimation evaluation uses KITTI train-to-validation results and measures voxels in the union of occlusion and signal-miss regions.Object coverage counts bounding boxes containing at least one voxel predicted positive.
  • Pedestrian results: Pedestrian detection on KITTI’s test split is weaker than on the validation split because pedestrian shape estimation is sensitive to sparse observed points.The paper identifies differences in pedestrian point-number distributions as a source of inaccurate occupancy estimation.
  • Shape assembly: Cyclist and pedestrian shape-assembly figures show complete-shape approximation by borrowing points from selected source objects.Cyclists are mirrored during assembly, whereas pedestrians are not because they are less likely to be symmetric.
  • Occupancy visualization: Waymo vehicle visualizations encode higher predicted occupancy probability with greater opacity on spherical voxels.Figures provide both zoomed-in and full-scene views.
  • KITTI test-set comparison: BtcDet’s KITTI test-set comparison reports 3D AP at 40 recall thresholds, with mAP averaged across easy, moderate, and hard objects.Table 7 covers all three object classes evaluated on the KITTI server.
Loading 2112.02205v1…