Source-linked AI summary
HVPR: Hybrid Voxel-Point Representation for Single-stage 3D Object Detection
Jongyoun Noh, Sanghoon Lee, Bumsub Ham
TL;DR
3D object detection must balance voxel features’ efficiency with point features’ structural accuracy when predicting boxes from sparse point clouds. HVPR combines them in a memory-augmented single-stage network, and KITTI experiments report a favorable speed–accuracy compromise.
Problem
Voxel features are efficient but lose fine-grained 3D structure, whereas point features preserve structure at high computational cost.
Method
HVPR integrates voxel and point features in a single-stage pseudo-image representation, using memory prototypes to reduce point-feature extraction cost.
Results
The model achieves the best results among voxel-based pseudo-image methods and a favorable compromise between detection accuracy and speed on KITTI.
Takeaways & Limitations
Hybrid representations and AMFM provide a practical way to combine structural detail, efficient inference, and scale-aware detection features.
Takeaways & Limitations
The model misses heavily occluded objects and objects captured with little or no point-cloud data.
Abstract
from arXiv · showhide
We address the problem of 3D object detection, that is, estimating 3D object bounding boxes from point clouds. 3D object detection methods exploit either voxel-based or point-based features to represent 3D objects in a scene. Voxel-based features are efficient to extract, while they fail to preserve fine-grained 3D structures of objects. Point-based features, on the other hand, represent the 3D structures more accurately, but extracting these features is computationally expensive. We introduce in this paper a novel single-stage 3D detection method having the merit of both voxel-based and point-based features. To this end, we propose a new convolutional neural network (CNN) architecture, dubbed HVPR, that integrates both features into a single 3D representation effectively and efficiently. Specifically, we augment the point-based features with a memory module to reduce the computational cost. We then aggregate the features in the memory, semantically similar to each voxel-based one, to obtain a hybrid 3D representation in a form of a pseudo image, allowing to localize 3D objects in a single stage efficiently. We also propose an Attentive Multi-scale Feature Module (AMFM) that extracts scale-aware features considering the sparse and irregular patterns of point clouds. Experimental results on the KITTI dataset demonstrate the effectiveness and efficiency of our approach, achieving a better compromise in terms of speed and accuracy.
1. Introduction
3D object detection uses sparse LiDAR point clouds to predict object boxes, but voxel and point representations trade efficiency against fine-grained structure. HVPR combines both through a memory-augmented, single-stage architecture.
- LiDAR point clouds provide accurate object depth but are sparse and vary in density with sensor distance.
- Voxel-based methods extract compact 3D features efficiently but lose fine-grained object structure during voxelization and downsampling.
- Point-based methods preserve more discriminative 3D structure, but processing large-scale point clouds requires substantial computation.
- HVPR integrates voxel-based and point-based features with a two-stream encoder and memory module for efficient single-stage 3D detection.The memory stores point-based features during training, avoiding the point-feature encoder at test time.
- HVPR aggregates memory features semantically similar to voxel features into a pseudo image, helping represent small or occluded objects from sparse point clouds.
2. Related work
Prior LiDAR detectors use voxel or point representations, while multi-stage methods combine them across separate stages. HVPR instead jointly weaves both representations into a pseudo image within a single-stage framework.
- Multi-sensor based 3D detection: Multi-sensor methods combine RGB and depth information because the modalities provide complementary semantic and structural cues.
- LiDAR based 3D detection: Voxel-based approaches convert irregular point clouds into ordered grids such as BEV images or voxels to obtain compact representations.
- LiDAR based 3D detection: PV-RCNN and Fast Point R-CNN use voxel features for proposals and point features for refinement, but separate the representations across stages.
- HVPR: HVPR uses a pseudo-image representation like PointPillars while adding point-based features efficiently to boost detection performance and retain fast runtime.
- HVPR: HVPR jointly exploits voxel-based and point-based representations in a single stage and uses 2D convolutions for efficient 3D detection.
3. Approach
HVPR combines voxel- and point-based features into hybrid pseudo-image representations, using memory to reduce point-feature extraction costs. Its AMFM refines multi-scale features with spatial attention, while a single-stage detection head predicts 3D boxes and classes.
- HVPR Network: HVPR uses voxel and point streams, aggregating point features similar to each voxel to form hybrid voxel-point pseudo-images.The concatenated features are scattered to corresponding voxel locations, producing a representation of size H × W × 2C.
- Voxel-based feature: Voxel features are extracted by voxelizing the x-y plane and applying a tiny PointNet with max pooling over points within each voxel.The resulting voxel feature map has size C × N, where N is the number of voxels.
- Point-based feature: Point-based features are extracted directly from raw point clouds with PointNet++, using set-abstraction and feature-propagation layers.The method gradually downsamples points and propagates features back to recover features for the initial points.
- Voxel-memory representation: A memory module stores point-feature prototypes and uses voxel features as queries to aggregate matching memory items during inference.The voxel-memory pseudo-image replaces the voxel-point pseudo-image at test time, avoiding point-feature extraction while retaining reported quality.
- Backbone Network with AMFM: AMFM refines multi-scale feature maps with spatial attention and a skip connection, using voxel density and position-related scale information.The backbone concatenates the resulting scale-aware maps before predicting 3D object bounding boxes.
- Detection Head and Loss: The detection head uses two fully connected layers for 3D box regression and classification, trained with regression, direction, classification, and memory-update losses.Boxes are parameterized by center coordinates, dimensions, and heading angle; focal loss handles classification.
4. Experiments
Experiments on KITTI evaluate HVPR against state-of-the-art methods and analyze its components, memory settings, runtime, and qualitative detections. The results indicate strong pseudo-image performance, efficient inference, and improved handling of small or occluded objects, with limitations for heavy occlusion.
- Dataset and evaluation: KITTI evaluation covers car and pedestrian detection across easy, moderate, and hard splits, using official test-server mAP scores.The dataset contains 7,481 training and 7,518 test samples, with difficulty determined by object size, occlusion, and truncation.
- Comparison with state of the art: HVPR achieves the best results among voxel-based methods using pseudo images for the car class.The comparison reports mAP on the KITTI test dataset and includes point-based and other voxel-based methods for context.
- Comparison with state of the art: HVPR is fastest among voxel-based approaches except PointPillars, while retaining competitive detection performance.The comparison explicitly evaluates both mAP and runtime for pseudo-image, voxel-3D, and point-based methods.
- Ablation study: Adding point-based features increases mAP by 1.31, 1.53, and 1.94 for easy, moderate, and hard car splits, respectively.The larger gains on moderate and hard splits support complementary voxel- and point-based representations for sparse, small, or occluded objects.
- Qualitative results: Qualitative results show good localization of small and moderately occluded objects, but weaker localization for heavily occluded objects.The figure compares predicted and ground-truth 3D boxes and also shows projected 2D boxes.
- Memory analysis: The model’s mAP is robust to the total number of memory items, while retrieving more than 20 nearest items lowers performance.Performance increases up to K = 20, whereas including less similar items is not helpful.
5. Conclusion
The conclusion presents HVPR as a single-stage detector that combines voxel- and point-based features through hybrid representations, memory augmentation, and scale-aware processing. On KITTI, it achieves the best results among pseudo-image voxel methods and a favorable speed–accuracy compromise.
- Conclusion: HVPR integrates voxel- and point-based features into hybrid 3D scene representations formatted as pseudo images.The architecture combines these representations for single-stage 3D object detection.
- Conclusion: A memory module makes point-feature augmentation efficient, while AMFM supplies scale-aware representations for 3D detection.The conclusion attributes significant performance gains to the scale-aware feature module.
- Conclusion: On KITTI, HVPR achieves the best results among voxel-based methods using pseudo images and a better accuracy–speed compromise than other 3D detectors.This is the paper’s stated overall experimental conclusion.
Supplementary Material HVPR: Hybrid Voxel-Point Representation for Single-stage 3D Object Detection
The supplementary material identifies the authors’ institutional affiliation. No experimental or methodological content is supplied in these passages.
- Author information: The paper lists Jongyoun Noh, Sanghoon Lee, and Bumsub Ham as its authors.The author line is followed by the Yonsei University affiliation.
- Scope: These passages provide bibliographic information rather than details about HVPR’s architecture, training, or evaluation.
- Author information: The authors are affiliated with the School of Electrical and Electronic Engineering at Yonsei University.
1. Implementation details
The implementation uses PointNet++ point features, a multiscale convolutional backbone with AMFM, fully connected detection layers, and class-specific training settings. Runtime analysis reports a 27.7-millisecond average for the full car model.
- Point-based features: Point-based features are extracted with PointNet++ after subsampling 16,384 input points to 4,096 and 1,024 points.Two set-abstraction layers perform the downsampling, followed by feature propagation producing 64×1 features per input point.
- Backbone network: The backbone has three convolutional blocks with stride-2 downsampling and output channels of 128, 256, and 512.Each block contains convolution, BatchNorm, and ReLU layers and extracts a different feature-map scale.
- AMFM: AMFM uses an MLP, spatial attention, and transposed convolution to produce aligned scale-aware features.The MLP hidden layers have 16 and 32 channels, and upsampling matches the largest feature map’s resolution and channel size.
- Detection head: The detection head uses two fully connected layers with 384 channels to localize and classify objects.
- Training settings: Pedestrian training uses 200 epochs, a 2e-4 learning rate, 1e-4 weight decay, and batch size 1 per GPU.The learning rate is decayed by 0.8 every 15 epochs, and the pedestrian setting uses K = 10 prototypes.
2. More results
HVPR combines efficient pseudo-image processing with scale-aware features to improve localization of difficult 3D objects while maintaining fast runtime. The model remains limited on heavily occluded objects and objects with little or no point-cloud coverage.
- Runtime analysis: 27.7 milliseconds is the average runtime of the full car-class model on an Nvidia 2080Ti GPU.Most computation is spent in the backbone and post-processing.
- Runtime analysis: 4.7 milliseconds is sufficient to estimate the pseudo image.
- AMFM: AMFM scale-aware features activate more strongly on decisive regions than multi-scale features.
- AMFM: High-resolution features attend to small or distant objects, while low-resolution features attend more to regions near the sensor containing large objects.This indicates that AMFM considers complex scale variations for object localization.
- Qualitative results: The model localizes small or occluded objects with sparse point clouds, but misses heavily occluded objects and objects captured with little or no point clouds.