Source-linked AI summary
PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
Danfei Xu, Dragomir Anguelov, Ashesh Jain
TL;DR
3D detection remains challenging because monocular depth is inaccurate and existing multisensor systems often rely on multistage or sensor-specific assumptions. PointFusion independently processes images and raw point clouds, then fuses them through spatially anchored box hypotheses. Across KITTI and SUN-RGBD, it performs on par with or better than state-of-the-art methods using the same architecture and hyperparameters.
Problem
3D detection is difficult because monocular methods have low accuracy, while existing multisensor algorithms often use multistage pipelines or domain-specific sensor assumptions.
Method
PointFusion uses a CNN for image crops, PointNet for raw point clouds, and a dense fusion network that predicts multiple box hypotheses relative to input points and selects among them.
Results
PointFusion achieves comparable or better 3D detections than state-of-the-art methods on both the KITTI and SUN-RGBD datasets.
Takeaways & Limitations
The same architecture and hyperparameters support competitive results across outdoor lidar-camera and indoor RGB-D datasets.
Takeaways & Limitations
Achieving reasonable performance requires non-trivial effort when selecting among architectures.
Abstract
from arXiv · showhide
We present PointFusion, a generic 3D object detection method that leverages both image and 3D point cloud information. Unlike existing methods that either use multi-stage pipelines or hold sensor and dataset-specific assumptions, PointFusion is conceptually simple and application-agnostic. The image data and the raw point cloud data are independently processed by a CNN and a PointNet architecture, respectively. The resulting outputs are then combined by a novel fusion network, which predicts multiple 3D box hypotheses and their confidences, using the input 3D points as spatial anchors. We evaluate PointFusion on two distinctive datasets: the KITTI dataset that features driving scenes captured with a lidar-camera setup, and the SUN-RGBD dataset that captures indoor environments with RGB-D cameras. Our model is the first one that is able to perform better or on-par with the state-of-the-art on these diverse datasets without any dataset-specific model tuning.
1. Introduction
PointFusion addresses 3D detection’s difficulty with heterogeneous sensors by combining image and raw point-cloud processing in a generic early-fusion architecture. It is evaluated across outdoor lidar-camera and indoor RGB-D datasets, achieving competitive results without dataset-specific tuning.
- 3D object detection estimates each object’s 6 DoF pose and 3D bounding-box dimensions, but monocular methods remain inaccurate, especially for long-range depth.
- Existing systems often process sensor modalities separately and fuse decisions through multistage pipelines or expert-designed tracking systems.
- PointFusion is designed to be agnostic to the placement, type, and number of 3D sensors, supporting varied robotics applications.
- PointFusion combines an off-the-shelf CNN for RGB image crops, PointNet for raw point clouds, and a fusion network for 3D box prediction without lossy preprocessing.
- The dense fusion network predicts each 3D box corner relative to every input point and learns to select the best hypothesis using a scoring function.
- On KITTI and SUN-RGBD, combining PointFusion with an off-the-shelf 2D detector produces comparable or better detections than respective state-of-the-art methods.
2. Related Work
Related work includes geometry-based 6-DoF pose estimation and image–lidar fusion methods. PointFusion is illustrated as a heterogeneous architecture that preserves native image and point-cloud representations rather than imposing restrictive spatial assumptions.
- Geometry-based methods recover 6-DoF object pose using 2D–3D keypoint matching or alignment of reconstructed models with known CAD models.
- PointFusion instead processes raw point clouds with PointNet and images with a CNN before dense fusion predicts spatially anchored box hypotheses.
- MV3D fuses features from top-down lidar, front-lidar, and image views after generating object proposals in the top-down lidar view.
- MV3D assumes a single lidar and that objects lie on the same spatial plane and can be localized solely from a top-down point-cloud view.
3. PointFusion
PointFusion regresses 3D bounding boxes by combining CNN image features with PointNet features from raw point clouds. Its dense fusion network uses input points as spatial anchors and scoring functions to select among box hypotheses.
- Fusion Network: The model offers global and dense fusion variants, with dense fusion using input 3D points as spatial anchors for predicting corner offsets.This replaces direct regression of absolute corner locations with point-relative offset prediction.
- Architecture: PointFusion combines a CNN for image crops, a PointNet variant for raw point clouds, and a fusion network for 3D bounding-box prediction.The architecture preserves both modalities in their native representations and removes batch normalization from the PointNet variant.
- Point Cloud Network: Camera geometry supplies a canonical rotation Rc for the point cloud inside each region of interest, addressing spatial biases tied to the 2D box location.The paper states that this replaces reliance on a Spatial Transformer Network that did not fully correct those biases.
- Fusion Network: The global fusion formulation faces regression-target variance because object distances can range from 1m to over 100m in autonomous driving.Dense spatial anchors are introduced to address this scale and location variability.
- Fusion Network: 3D boxes are parameterized by their 8 corners, enabling spatial offset prediction and facilitating comparison with existing methods.The corner representation is described as generalizable to shapes represented by reference points.
- Dense Fusion Prediction Scoring: The dense fusion loss combines offset regression with scoring so the network emphasizes points close to or likely to produce good box hypotheses.Supervised scoring predicts whether each point lies inside the target box, while unsupervised scoring learns which point is likely to yield the best prediction.
4. Experiments
Experiments test PointFusion across outdoor KITTI and indoor SUN-RGBD settings, compare architectural variants, and evaluate point-count sensitivity. Results show strong benchmark performance, benefits from heterogeneous fusion and dense prediction, and qualitative gains over lidar-only inputs.
- Experimental setup: PointFusion is evaluated on KITTI driving scenes and SUN-RGBD indoor scenes using AP3D-based 3D detection evaluation.KITTI includes cars, pedestrians, and cyclists; SUN-RGBD contains up to 700 labeled object categories and uses a 0.25 IoU threshold for reported 3D average precision.
- Model variants: Six variants compare dense and global prediction, image use, scoring functions, and heterogeneous versus homogeneous architectures.The final model uses dense prediction with unsupervised scoring; dense-no-im and global-no-im remove image input, while rgb-d replaces PointNet with a depth-image CNN.
- KITTI results: The final model achieves the best KITTI performance among the authors’ variants, while rgb-d performs worst, supporting the heterogeneous design.The comparison covers car-only KITTI validation models and includes 3DOP, VeloFCN, and MV3D baselines.
- KITTI results: On KITTI, final exceeds MV3D by 3% AP3D on easy examples and is 1.5% lower on moderate examples; all-class training gains 6% and 0.5%, respectively.MV3D remains ahead on hard, substantially occluded examples by 6% and 3% AP3D for the two PointFusion models discussed.
- Ablations: Dense prediction substantially outperforms direct global corner regression, indicating that input points are effective spatial anchors.The comparison is made between dense versus global variants with matching image-use conditions.
- Qualitative and sensitivity analysis: Image appearance features improve estimation of object dimensions and orientation, while the model remains robust to variations in the number of input points.Qualitative comparisons show fusion correcting orientation and spatial-extent errors made by the lidar-only model, although extremely cluttered scenes can still cause failures.
- SUN-RGBD results: On SUN-RGBD, final outperforms the rgb-d baseline by 6% mAP, beats DSS by 3% mAP while running 15 times faster, and outperforms COG on 6 of 10 categories.PointFusion uses only points projecting to a 2D detection box, whereas DSS and COG use slower proposal or exhaustive-search procedures.
5. Conclusions and Future Work
PointFusion estimates 3D bounding boxes by combining image and raw point-cloud representations through heterogeneous processing and dense fusion. The same architecture and hyperparameters perform on par with or better than methods using dataset- and sensor-specific assumptions across two distinct datasets.
- PointFusion processes images with a CNN and raw point clouds with PointNet, avoiding lossy input preprocessing such as quantization or projection.
- Its dense fusion network predicts multiple 3D box hypotheses relative to input 3D points and learns to select the best hypothesis.
- The method combines image and point-cloud representations through heterogeneous network architectures.
- With the same architecture and hyperparameters, PointFusion performs on par with or better than methods holding dataset- and sensor-specific assumptions on KITTI and SUN-RGBD.
6. Supplementary
The supplementary material specifies evaluation metrics, baseline implementation details, and additional methodological choices for 3D box regression and KITTI comparisons.
- The rgbd baseline: On KITTI, the rgbd baseline performs worst because most predicted boxes have less than 0.5 IoU with ground truth due to depth-prediction errors.Its performance gap is reduced on SUN-RGBD because denser depth maps are available, although comparable CNN-based performance is non-trivial to achieve.
- The rgbd baseline: The rgbd baseline uses a five-channel CNN input containing RGB, depth, and a binary depth-measurement mask.KITTI depth is obtained by projecting lidar onto the image plane, while SUN-RGBD uses the depth image.
- Box regression: Predicting 3D corner locations directly was ineffective, so the model regresses corner pixel locations and depths before recovering the full 3D box using camera geometry.
- KITTI results: The supplementary tables report KITTI car-category AP3D results and all-category KITTI AP3D results.