Source-linked AI summary
What You See is What You Get: Exploiting Visibility for 3D Object Detection
Peiyun Hu, Jason Ziglar, David Held, Deva Ramanan
TL;DR
LiDAR sweeps are 2.5D measurements whose occlusions encode visibility and freespace that point representations can discard. The paper reconstructs visibility through raycasting and adds it as a stream to a 3D detector. On NuScenes, the approach improves overall mAP over PointPillars by 4.5%.
Problem
LiDAR sweeps encode visibility and freespace through occlusion, but normalized point representations can discard this information.
Source: §1 · p. 2Method
The approach uses 3D raycasting to create voxel visibility and adds the visibility representation as an input stream to a 3D detector.
Source: §3.2 · p. 4, §3.3 · p. 5Results
4.5% overall mAP improvement over PointPillars on the NuScenes test set.
Source: §4 · p. 6Takeaways & Limitations
The results support visibility as a useful representation cue for 3D object detection on NuScenes.
Source: §4 · pp. 6, 8The model underperforms official PointPillars on motorcycles; the authors hypothesize coarser xy-resolution or lack of ImageNet/KITTI pretraining may contribute.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Recent advances in 3D sensing have created unique challenges for computer vision. One fundamental challenge is finding a good representation for 3D sensor data. Most popular representations (such as PointNet) are proposed in the context of processing truly 3D data (e.g. points sampled from mesh models), ignoring the fact that 3D sensored data such as a LiDAR sweep is in fact 2.5D. We argue that representing 2.5D data as collections of (x, y, z) points fundamentally destroys hidden information about freespace. In this paper, we demonstrate such knowledge can be efficiently recovered through 3D raycasting and readily incorporated into batch-based gradient learning. We describe a simple approach to augmenting voxel-based networks with visibility: we add a voxelized visibility map as an additional input stream. In addition, we show that visibility can be combined with two crucial modifications common to state-of-the-art 3D detectors: synthetic data augmentation of virtual objects and temporal aggregation of LiDAR sweeps over multiple time frames. On the NuScenes 3D detection benchmark, we show that, by adding an additional stream for visibility input, we can significantly improve the overall detection accuracy of a state-of-the-art 3D detector.
1. Introduction
The introduction argues that LiDAR is better understood as 2.5D data: occlusion leaves visibility and freespace information that point-based representations can discard. It proposes recovering visibility through raycasting and adding it to deep 3D detection, including alongside object augmentation and temporal aggregation.
1. Introduction
- LiDAR readings suffer occlusion: once a scene element is measured at a depth, visibility occludes all elements behind it along the line of sight, so such data may be better characterized as 2.5D.Source: §1 · p. 2
- Normalized (x, y, z) point representations can destroy visibility information implicit in measured LiDAR data.The paper argues that many point-cloud representations were designed for truly 3D data, such as points sampled from mesh models.Source: §1 · p. 2
- Range images encode visibility implicitly, but the paper notes that they do not appear to match voxel-based or BEV grids in state-of-the-art 3D object understanding accuracy.The authors propose augmenting voxel representations with visibility while retaining convolutional reasoning along depth.Source: §1 · p. 2
- The paper proposes efficient on-the-fly raycasting to compute voxel visibility and incorporate it into batch-based gradient learning.Visibility is added as an input stream, with early and late fusion explored.Source: §1 · p. 2
- Visibility is also combined with synthetic virtual-object augmentation and temporal aggregation of LiDAR sweeps.The authors report using visibility cues to better place virtual objects and characterize multi-frame visibility reasoning as akin to online occupancy mapping.Source: §1 · p. 2
2. Related Work
Related work spans learned point representations and visibility reasoning for mapping, alongside multiple LiDAR detector representations. The paper connects these lines to visibility-aware detection and to established object-augmentation and temporal-aggregation techniques.
2.1. 3D Representations
- Point-representation research ranges from hand-crafted descriptors to learned encoders, but measured LiDAR point clouds also carry visibility information beyond the points themselves.The cited learned methods include PointNet, PointNet++[23], Kd-networks, PointCNN, Edge-Conv, and PointConv.Source: §2.1 · p. 2
- Visibility reasoning is established in robotic occupancy mapping but remains less explored for object detection, where prior examples include occupancy-based detection with known surface models.Octomap is cited as a general-purpose 3D occupancy mapping system, while uses a probabilistic framework based on occupancy maps.Source: §2.1 · p. 2, §2.2 · p. 3
2.2. LiDAR-based 3D Object Detection
- LiDAR detectors use range images, BEV maps, raw points, or voxels; the general framework places class-shaped 3D anchors over a ground plane and predicts confidence and box adjustments from BEV feature maps.The caption describes a convolutional network that produces oriented 3D boxes from those predictions.Source: §2.2 · p. 2, Figure 2 · p. 3
- Object augmentation copy-pastes point clouds between scenes, but can insert objects in ways that violate scene visibility.The paper proposes using visibility reasoning to maintain correct visibility during cross-scene augmentation.Source: §2.2 · p. 3
- Temporal aggregation combines LiDAR sweeps, with timestamps supplied as per-point features in one established approach; this paper explores temporal aggregation over visibility representations.Prior approaches cited include recurrent models, 4D ConvNets [6], and sweep aggregation in SECOND.Source: §2.2 · p. 3
3. Exploit Visibility for 3D Object Detection
The section integrates raycast-derived visibility into a 3D detection framework, including object augmentation and temporal aggregation. It adds visibility as a second PointPillars stream and explores early and late fusion.
3.1. A General Framework for 3D Detection
- The framework converts point clouds to multi-channel BEV representations so standard 2D convolutional architectures can process them.The section presents object augmentation and temporal aggregation as innovations used by contemporary 3D detectors.Source: §3.1 · p. 3, §3 · p. 3
- +9.1% average improvement on augmented classes is reported for vanilla PointPillars with object augmentation.The strategy copy-pastes virtual objects of rarely seen classes into LiDAR scenes.Source: §3.1 · p. 3
- +8.6% overall mAP is reported for vanilla PointPillars with temporal aggregation.The method combines motion-compensated points from multiple sweeps and adds a relative-timestamp channel.Source: §3.1 · p. 4
3.2. Compute Visibility through Raycasting
- Raycasting marks voxels along each sensor-to-return segment as free, the return voxel as occupied, and remaining voxels as unknown.Visibility along the vertical dimension is represented as separate channels in a 2D feature map; sparse voxel traversal is used for efficiency.Source: §3.2 · p. 4
- Raycasting rectifies augmented scenes by either culling occluded virtual objects or drilling away original points that occlude them.Culling may excessively reduce the number of augmented objects.Source: §3.2 · p. 4, Figure 3 · p. 5
- For multi-frame visibility, Bayesian filtering turns spatial-temporal observations into a 3D posterior occupancy probability.The section notes that directly integrating a 4D volume into the detector would be too expensive.Source: §3.2 · p. 5
3.3. Approach: A Two-stream Network
- The two-stream PointPillars approach combines point-cloud and visibility representations, with early or late fusion alternatives.Early fusion joins visibility with pillar features before a backbone; late fusion uses separate backbones and concatenates their outputs.Source: §3.3 · pp. 5–6, Figure 5 · p. 6
4. Experiments
Experiments on NuScenes show that the two-stream detector improves test-set mAP over PointPillars, with larger gains for heavily occluded objects. Validation ablations examine visibility, fusion, object augmentation, and temporal aggregation.
Setup and baselines
- The benchmark uses NuScenes scenes and evaluates average mAP across classes and distance thresholds; the reproduced PointPillars baseline reaches 31.5% validation mAP.The reproduced baseline is 2 percentage points above the official PointPillars mAP of 29.5%, and uses no ImageNet/KITTI pretraining.Source: §4 · p. 6, Table 2 · p. 8
Improvement at different levels of visibility
- The largest visibility gains occur for heavily occluded cars, while fully visible cars improve least; fully visible pedestrians improve by 3.2%.On the validation set, visibility improves overall mAP by 4%.Source: §4 · p. 6
Main results
- 4.5% higher overall mAP than PointPillars on the NuScenes test set; the approach underperforms on motorcycles.It also outperforms PointPillars on cars (10.7%), pedestrians (5.3%), trucks (7.4%), buses (18.4%), and trailers (16.7%).Source: §4 · p. 6, Table 1 · p. 7
Qualitative results
- Qualitative test-set examples show detections of vehicles, motorcycles, and pedestrians, alongside a rare-class failure that labels construction vehicles as trucks and a bus.The front-camera images provide context and are not model inputs.Source: Figure 6 · p. 7
Ablation studies
- Replacing early with late fusion drops overall mAP by 1.4%; replacing drilling with culling drops bus mAP by 11.4% and trailer mAP by 4.9%.The authors report that culling can remove severely occluded augmented trucks and trailers.Source: §4 · p. 7, Table 3 · p. 8
- Removing temporal aggregation drops overall mAP by 9.4%, while removing visibility drops it by 1.4%; removing object augmentation reduces mAP on augmented classes.Removing object augmentation drops truck mAP by 2.5%, bus by 13.7%, and trailer by 7.9%; adding object augmentation and temporal aggregation to vanilla PointPillars improves mAP by 9.1% on augmented classes and 8.6% overall, respectively.Source: §4 · pp. 7–8, Table 3 · p. 8
Runtime
- Visibility computation takes 24.4±3.5 ms per 32-beam LiDAR point cloud on one Intel i9-9980XE CPU thread.The implementation runs in C++ as part of parallel PyTorch data loading.Source: §4 · p. 8
A. Additional Results
Additional results provide a visualization guide and assess whether reported gains persist across random-seed runs.
Additional results
- The supplementary video includes a visual guide for interpreting its detection visualizations.The guide points readers to Figure A.Source: A. Additional Results · p. 9
Performance over multiple runs
- Across five additional training runs with different random seeds, standard deviations were much smaller than the reported improvements, which the authors say suggests gains are not due to a lucky run.Seeds changed both network initialization and training-data sampling order; accuracy was evaluated on the NuScenes validation set.Source: A. Additional Results · p. 9
B. Additional Experimental Details
The supplementary details specify point filtering, pillar construction, network components, and training choices for the baseline and two-stream models.
Pre-processing
- Pillar features use points within x=[−50, 50], y=[−50, 50], and z=[−5, 3], while raycasting includes points outside that range.Pillars are 0.25 × 0.25 × 8, resampled to 60 points each; raycasting voxels are 0.25 × 0.25 × 0.25.Source: B. Additional Experimental Details · p. 9
Network structure
- The network combines a pillar feature network, a convolutional encoder-decoder backbone, and detection heads for large and small object classes.The pillar network forms 64-dimensional features; the early-fusion backbone input has 64 pillar-feature and 32 visibility channels, and separate heads predict anchor classification and box regression.Source: B. Additional Experimental Details · p. 9
Loss function and training
- Classification uses focal loss with α=0.25 and γ=2.0, regression uses smooth L1 loss with σ=3.0, and the combined loss weights classification by 2.Models train for 20 epochs with Adam and a two-phase one-cycle learning-rate schedule.Source: B. Additional Experimental Details · p. 9
Visualization guide
- Figure A uses transparent solid cuboids for ground truth, wireframe boxes for predictions, and colors for object classes.The figure highlights the algorithm's mistakes as well as detected cars, trucks, and trailers.Source: B. Additional Experimental Details · p. 10
Multiple-run results
- Table 4 reports 3D detection mAP over multiple NuScenes validation runs.Source: Table 4 · p. 10