Source-linked AI summary
Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots
Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, Alan Yuille
TL;DR
LiDAR 3D detection must handle irregular, sparse point clouds, where object-level aggregation can be unreliable. The paper proposes Object as Hotspots (OHS), an anchor-free detector that selects informative interior voxels and balances hotspot assignments. It reports robust performance on objects with few points, including first-place KITTI cyclist and pedestrian results and state-of-the-art NuScenes performance.
Problem
LiDAR point clouds are irregular and sparse, with occlusion, self-occlusion, reflection, and bad weather limiting captured points.
Method
Object as Hotspots (OHS) represents objects as compositions of interior non-empty voxels and spatial relations, using an anchor-free head with imbalance-aware hotspot assignment.
Results
The method shows robust performance on objects with few points, ranks first on KITTI cyclist and pedestrian detection, and achieves state-of-the-art performance on NuScenes.
Takeaways & Limitations
Hotspots provide a sparse-object representation intended to preserve essential semantic and localization information from limited LiDAR points.
Takeaways & Limitations
The hotspot design assumes hotspots should be distinguishable, category-shared, and minimal for robustness to sparse observations.
Abstract
from arXiv · showhide
Accurate 3D object detection in LiDAR based point clouds suffers from the challenges of data sparsity and irregularities. Existing methods strive to organize the points regularly, e.g. voxelize, pass them through a designed 2D/3D neural network, and then define object-level anchors that predict offsets of 3D bounding boxes using collective evidences from all the points on the objects of interest. Contrary to the state-of-the-art anchor-based methods, based on the very nature of data sparsity, we observe that even points on an individual object part are informative about semantic information of the object. We thus argue in this paper for an approach opposite to existing methods using object-level anchors. Inspired by compositional models, which represent an object as parts and their spatial relations, we propose to represent an object as composition of its interior non-empty voxels, termed hotspots, and the spatial relations of hotspots. This gives rise to the representation of Object as Hotspots (OHS). Based on OHS, we further propose an anchor-free detection head with a novel ground truth assignment strategy that deals with inter-object point-sparsity imbalance to prevent the network from biasing towards objects with more points. Experimental results show that our proposed method works remarkably well on objects with a small number of points. Notably, our approach ranked 1st on KITTI 3D Detection Benchmark for cyclist and pedestrian detection, and achieved state-of-the-art performance on NuScenes 3D Detection Benchmark.
1. Introduction
LiDAR 3D detection is difficult because point clouds are irregular and sparse, motivating a representation that uses informative object parts rather than object-level anchors. The paper proposes Object as Hotspots with an anchor-free head and imbalance-aware assignment.
- Motivation: LiDAR point clouds are irregular and sparse because points lie on surfaces and sensing conditions can limit captured points.Voxelization and 2D/3D CNN processing regularize the input, but sparsity remains a central challenge.
- Imbalance handling: The hotspot assignment strategy selects limited, discriminative features and balances positive examples across objects with different point counts.It is designed to prevent bias toward objects with more points.
- Detection head: The method uses an anchor-free detection head, whose regression targets are difficult to design without human-defined anchor sizes.The introduction identifies regression target imbalance as an additional challenge for anchor-free detection.
- Proposed representation: Object as Hotspots (OHS) compositionally represents objects using interior non-empty voxels and their spatial relations.The representation is inspired by compositional part-based models and selects a small subset of spots as hotspots.
- Results: The approach performs robustly on objects with very few points and reports new state-of-the-art results on NuScenes and KITTI cyclist and pedestrian detection.It also achieves 25 FPS on KITTI.
2. Related Work
Related work contrasts anchor-free point-based detection with anchor-based normalization and other anchorless instance methods. The section highlights scalability, partial-observation, and regression-target challenges.
- Anchor-free detection: Anchor-free RGB detectors represent objects using points such as centers, corners, extrema, or dense interior locations.These approaches motivate representing 3D objects through informative point locations rather than object-level anchors.
- Anchorless 3D methods: SGPN and 3D-BoNet provide anchorless indoor-scene approaches, but the cited discussion states both fail when only partial point clouds are observed.SGPN also has a similarity matrix whose size grows quadratically with the number of points.
- Regression target imbalance: Bounding-box centers and sizes have different scales, causing scale variance in regression targets and gradients during anchor-free training.Small target values tend to produce smaller gradients and less training impact.
- Anchor normalization: Anchor-based detectors use anchor locations and sizes as normalization factors, while anchor-free detectors lack the normalizing effect of varied anchor sizes.Hand-designed anchor sizes and aspect ratios are used to capture multimodal bounding-box-size distributions.
3. Object as Hotspots
OHS represents each object through a limited subset of informative non-empty interior voxels called hotspots. Training assigns these hotspots from ground truth, while selection favors center-near spots and suppresses point-count imbalance.
- 3.1. Hotspot Definition: Spots are non-empty voxels containing points and overlapping objects; only a subset becomes hotspots for training and inference.Hotspots aggregate minimal, discriminative features for foreground/background and inter-class classification.
- 3.1. Hotspot Definition: Hotspots are intended to capture distinguishable, category-shared, and minimal object parts that remain useful when few LiDAR points are available.These properties define the desired robustness of the representation to sparse observations.
- 3.2. Hotspot Selection & Assignment: The backbone feature map assigns hotspots through neurons mapped to super voxels in input point-cloud space.The voxel corresponding to a feature-map neuron is denoted Vn.
- 3.2. Hotspot Selection & Assignment: A voxel is a spot when it is non-empty and inside the ground-truth box, and hotspots are chosen as the spots nearest the object center.Center-near locations are selected because they are considered more reliable across view angles and provide more accurate localization.
- 3.2. Hotspot Selection & Assignment: At most M nearest spots are selected per object, with M determined adaptively by bounding-box volume; if fewer spots exist, all are assigned.The adaptive limit further suppresses hotspot counts for large objects with richer features.
4. HotSpot Network
HotSpotNet voxelizes LiDAR point clouds, extracts 3D features, and predicts objects from selected hotspots using classification, box regression, and spatial-relation supervision. During inference, it uses hotspot classification and box regression, followed by confidence filtering and NMS.
- Architecture: HotSpotNet combines a 3D feature extractor with an Object-as-Hotspots head containing classification, box regression, and spatial-relation subnets.The input is voxelized and processed by a 3D CNN before hotspot assignment and prediction.
- Hotspot classification: The classification subnet predicts K category heatmaps, labels hotspots as ones, ignores non-hotspots inside ground-truth boxes, and applies focal loss.Classification loss is averaged over hotspots and non-hotspots outside ground-truth boxes.
- Box regression: For each hotspot, box regression predicts [dx, dy, z, log(l), log(w), log(h), cos(r), sin(r)] to represent the object’s centroid, dimensions, height, and orientation.The regression operates only on hotspots and uses Smooth L1 loss; soft argmin is used for dx, dy, and z.
- Hotspot assignment: Only selected non-empty voxels inside objects are assigned as positive hotspots, rather than densely labeling all interior locations.The selected hotspots supervise hotspot classification on the backbone’s output feature map.
- Spatial relation and inference: Spatial relations are encoded by assigning each hotspot to a BEV quadrant relative to the object center and orientation, with a weighted sum combining the three losses.The spatial-relation encoder is trained with quadrant classification but does not contribute during inference.
5. Experiments
The experiments evaluate HotSpotNet on KITTI and NuScenes using established train/validation or benchmark settings, specified voxelization and backbones, and dataset-specific training and augmentation procedures. Implementation details include backbone choices, voxel sizes, optimization schedules, proposal filtering, and rotated NMS.
- Datasets and evaluation: The evaluation covers KITTI 3D detection and NuScenes detection, with ablations conducted on KITTI’s common train/validation split.KITTI provides 7,481 training and 7,518 testing point clouds; NuScenes contains 1,000 scenes and 40,000 annotated frames.
- Backbones and voxelization: KITTI experiments use the SECOND backbone, point-cloud range [0, 70, 4], [−40, 40], [−3, 1], and voxel size (0.025, 0.025, 0.05)m.At most five points are randomly sampled per voxel, with voxel features obtained by averaging point features.
- Backbones and voxelization: NuScenes experiments use CBGS as baseline and report implementations with ResNet and PointPillars backbones using backbone-specific voxel sizes.The point-cloud range is [−50.4, 50.4], [−50.4, 50.4], [−5, 3].
- Training and inference: KITTI training uses AdamW, a one-cycle learning-rate policy, batch size 8, and 150 epochs; testing retains 100 proposals after confidence filtering and applies rotated NMS.The confidence threshold is 0.3 and the rotated-NMS IoU threshold is 0.01.
- Training and inference: NuScenes training uses a maximum learning rate of 0.001, batch size 48, and 20 epochs, while testing retains 80 proposals with confidence threshold 0.1 and rotated-NMS IoU threshold 0.02.NuScenes augmentation follows CBGS with additional x-axis flipping and attached ground-truth objects.
5.3. Experiment results on KITTI benchmark
On KITTI, HotSpotNet outperforms published LiDAR-only one-stage detectors across cars, cyclists, and pedestrians at all difficulty levels, while remaining as fast as its SECOND baseline. It ranked first for cyclist and pedestrian detection at submission time and was particularly effective for objects with few points.
- KITTI benchmark results: HotSpotNet outperforms all published LiDAR-based one-stage detectors on KITTI cars, cyclists, and pedestrians across all difficulty levels.The comparison is reported on the KITTI test dataset using the 3D object detection benchmark.
- KITTI benchmark results: HotSpotNet ranked 1st among published methods on the KITTI test set for cyclist and pedestrian detection at submission time.The paper reports this ranking specifically for cyclist and pedestrian detection.
- KITTI benchmark results: HotSpotNet shows advantages on objects with a small number of points, supporting the paper’s hotspot-based object representation.The method also beats some classic two-stage detectors for car detection, including methods that fuse LiDAR and RGB.
- Efficiency: 25FPS inference on KITTI achieves a significant performance gain while maintaining the speed of the SECOND baseline on a Titan V100.The reported inference speed is measured on KITTI with a Titan V100.
5.4. Experiment results on NuScenes dataset
On NuScenes, HotSpotNet outperforms the CBGS baseline on validation mAP and all detectors on the test benchmark using a single model.
- HotSpotNet outperforms CBGS by 1.8 and 3.2 mAP with PointPillars and ResNet backbones, respectively.
- The NuScenes evaluation reports mAP on validation and test sets, with NDS additionally combining mAP and multiple precision attributes.
- HotSpotNet outperforms all detectors on the NuScenes 3D Detection benchmark using a single model.
5.5. Analysis
The analysis examines hotspot-count selection and sparse-object detection, showing that limited hotspot assignment improves balance while preserving strong performance.
- Analysis goal: The analysis evaluates whether hotspot selection prevents bias toward objects with more points without compromising their detection performance.
- Hotspot count: C = 64 is selected to balance performance across cars, cyclists, and pedestrians while preventing over-fitting on one class.Car performance is generally insensitive to C except at C = 32, cyclist performance peaks at C = 128, and pedestrian performance changes little for C ≤64.
- Sparse-object robustness: HotSpotNet consistently improves detection over SECOND across objects with different point counts, with larger relative gains as point counts decrease.This indicates greater robustness to sparse objects within the evaluated KITTI validation comparison.
5.6. Ablation Studies
Ablations show that balanced hotspot assignment, spatial relation encoding, and soft argmin each address distinct challenges in anchor-free detection.
- Effect of different target assignment strategies: The target-assignment ablation compares dense, all-spot, and limited-hotspot supervision while holding other settings constant.
- Effect of different target assignment strategies: C = 64 hotspot assignment outperforms dense and C = inf strategies by a large margin for cyclist and pedestrian detection, while car performance barely changes.Dense assigns empty and non-empty voxels; C = inf assigns all non-empty spots; C = 64 limits hotspots adaptively.
- Effect of soft argmin: Soft argmin improves performance over raw regression values, particularly for small objects such as cyclists and pedestrians.It avoids regressing absolute values with different scales.
- Sparse-object robustness: Recall is evaluated for objects grouped by their number of LiDAR points on KITTI validation.
- Effects of spatial relation encodings: Quadrant spatial encoding improves performance, whereas two-part and eight-direction encodings do not help regression.The authors attribute the benefit to encoding coarse hotspot locations that assist accurate localization.
5.7. Qualitative Visualization
Qualitative visualization traces detections back to fired voxels and reveals a recurring concentration of hotspots at a car’s front corner.
- Ablation context: The study also reports quadrant spatial encoding and soft argmin as evaluated design choices supporting hotspot localization and regression.
- Qualitative visualization: Fired hotspots in the visualized car examples consistently sit at the front corner, identified as a distinctive part for car detection and representation.The visualization marks fired hotspots in red and compares original BEV point clouds with detected-car point clouds.
6. Conclusion
The paper combines Object-as-Hotspots with an anchor-free detection head and target assignment strategy to address inter-object point-sparsity imbalance. Experiments show the approach is effective and robust for sparse point clouds, with hotspots visualized on detected cars.
- Object-as-Hotspots and an anchor-free detection head address inter-object point-sparsity imbalance.
- Soft argmin and carefully designed regression targets address regression target imbalance and boost accurate 3D localization.
- The approach is effective and robust to sparse point clouds.
- Figure 6 visualizes active hotspots in red on cars from the original LiDAR point clouds.