ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection

Motivation

• RGB images lack explicit scene geometry and absolute scale, making monocular 3D object detection ill-posed.Invisible areas also prevent scene geometry from being unambiguously derived from a single RGB image.Source: §1 · p. 1
• Existing RGB-based methods commonly separate monocular prediction from later multi-view aggregation.This late aggregation may fail to exploit spatial information effectively.Source: §1 · p. 1

Approach

• ImVoxelNet constructs a voxel representation to accumulate information from multiple inputs.The resulting unified approach uses the same meta-architecture for indoor and outdoor detection, switching only between domain-specific heads.Source: §1 · p. 1

Contributions

• The method is trained end-to-end for multi-view 3D object detection from posed RGB images only.This is presented as the first formulation of this task according to the authors.Source: §1 · p. 1
• ImVoxelNet is a fully convolutional 3D detector that works in both monocular and multi-view settings.Monocular inputs are treated as a special case of multi-view inputs.Source: §1 · pp. 1–2

Results

• Domain-specific heads yield state-of-the-art results across indoor and outdoor datasets.The reported contribution covers both scene types rather than a single benchmark domain.Source: §1 · p. 2

Multi-view scene understanding

• SLAM, Structure-from-Motion, and Multi-View Stereo use multiple images to estimate poses, geometry, or point clouds.These tasks illustrate established multi-view scene-understanding settings.Source: §2.1 · p. 2
• Recent methods also apply multi-view RGB-D or RGB inputs to 3D instance segmentation, semantic segmentation, and TSDF reconstruction.These examples broaden multi-view processing beyond geometric reconstruction alone.Source: §2.1 · p. 2

Point-cloud-based methods

• Point-cloud detectors often voxelize scenes, but outdoor systems commonly project them to BEV to reduce exhaustive 3D-convolution computation.Indoor detectors instead use deep Hough voting because single-center proposals can be irrelevant for non-convex objects.Source: §2.2 · p. 2

Stereo-based methods

• Stereo methods use exactly two images with fixed, precisely known relative transformation, unlike multi-view methods accepting arbitrary inputs and poses.This difference enables stereo reconstruction through left-right optical-flow estimation.Source: §2.2 · p. 2

Monocular-based methods

• Monocular detectors estimate 3D structure through anchors, discretized orientations, decomposed subtasks, or lifted 2D detections.Some methods additionally use external data such as CAD models or mesh annotations.Source: §2.2 · p. 2

Indoor detection

• Indoor monocular 3D detection remains less explored, with SUN RGB-D described as the only existing benchmark.Related methods may jointly estimate camera poses, room layouts, or object meshes.Source: §2.2 · p. 2

Multi-view detection

• Existing nuScenes methods predict independently per monocular image and aggregate outputs only afterward.The paper identifies this late-stage aggregation as potentially ineffective for exploiting spatial information.Source: §2.2 · p. 2

Research gap

• The paper states that existing methods do not formulate multi-image RGB 3D detection as an end-to-end optimization problem.This gap motivates ImVoxelNet’s unified treatment of posed RGB inputs.Source: §2.2 · p. 2

3.1. 3D Volume Construction

• ImVoxelNet extracts 2D image features, projects them into a 3D voxel volume, averages valid multi-view features, and predicts parameterized 3D bounding boxes.The box tuple is (x, y, z, w, h, l, θ), representing center, dimensions, and rotation around the z-axis.Source: §3 · p. 3, §3.1 · pp. 3–4
• The method accepts an arbitrary-sized set of RGB inputs with camera poses, including monocular inputs as a special case.The number of views can vary across inputs during training and inference.Source: §3 · p. 3, §3.1 · p. 3
• A pinhole camera model maps 2D feature-map coordinates to 3D voxel coordinates within dataset-specific spatial limits and a fixed voxel size.A binary frustum mask identifies valid voxels before feature projection and aggregation.Source: §3.1 · pp. 3–4

3.3.2 Indoor Head

• Indoor detection uses dense multi-scale 3D features and a novel FCOS-inspired 3D head with classification, location, and centerness branches.The head replaces 2D convolutions with 3D convolutions and samples up to 27 candidate locations per object.Source: §3.3.2 · pp. 4–5
• Indoor training combines focal classification loss, centerness cross-entropy, and rotated 3D IoU location loss.The indoor loss is normalized by the number of positive 3D locations.Source: §3.3.2 · p. 5

Datasets

• Experiments cover monocular KITTI and SUN RGB-D benchmarks, and multi-view ScanNet and nuScenes benchmarks.KITTI and SUN RGB-D evaluate monocular inputs, while ScanNet and nuScenes use multi-view formulations.Source: §4.1 · p. 5
• KITTI uses car-category evaluation, with moderate objects on the test set serving as the main assessment protocol.The reported validation and test evaluations follow the benchmark’s established difficulty and category conventions.Source: §4.1 · p. 5

Outdoor results

• ImVoxelNet achieves the best moderate AP on KITTI’s test split and surpasses previous state-of-the-art by 6% AP3D and 4% APBEV for easy objects.The method is superior on almost all reported metrics across both test and validation splits.Source: §4.3 · p. 6
• On nuScenes, ImVoxelNet outperforms MonoDIS by more than 1% mean AP and produces almost twice as many highly accurate estimates under AP@0.5.The method uses multi-view inputs during training, unlike methods that only apply multi-view images at inference.Source: §4.3 · p. 6

Indoor results

• On SUN RGB-D, ImVoxelNet surpasses previous methods by more than 18% mAP and also outperforms Total3DUnderstanding in layout and camera pose estimation.The benchmark includes 37 NYU-37 object categories and evaluates room layout and camera pose alongside detection.Source: §4.3 · p. 7, Table 4 · p. 8
• On ScanNet, ImVoxelNet remains competitive without point clouds and outperforms point-cloud-based 3D-SIS.Increasing the number of test images per scene improves results, although processing the voxel volume is the most time-consuming pipeline component.Source: §4.3 · pp. 7–8, Table 5 · p. 8

• ImVoxelNet formulates multi-view RGB-based 3D object detection as an end-to-end optimization problem.Source: §5 · p. 8
• The method accepts posed multi-view inputs with an arbitrary number of views during training and inference, including monocular inputs as a special case.Source: §5 · p. 8
• ImVoxelNet achieves state-of-the-art outdoor car-detection results on the monocular KITTI and multi-view nuScenes benchmarks.Source: §5 · p. 8
• The method surpasses existing 3D object-detection methods on SUN RGB-D and sets a new benchmark for indoor multi-view detection on ScanNet.Source: §5 · p. 8
• ImVoxelNet works on both indoor and outdoor data, giving the method a general-purpose scope.Source: §5 · p. 8

• ImVoxelNet is compared with PerspectiveNet using a protocol with 30 SUN RGB-D object categories and matched category training.Source: A. More results on SUN RGB-D · p. 11
• Ten categories are shared across the 30-category protocol and benchmarks using 10 categories, allowing metrics to be merged on the same subset.Source: A. More results on SUN RGB-D · p. 11
• The point-cloud comparison on SUN RGB-D uses 10 categories and mAP@0.25 as the main metric.Source: A. More results on SUN RGB-D · p. 11
• The point-cloud comparison favors point-cloud methods because they access more complete data, while ImVoxelNet provides a monocular RGB baseline.Source: A. More results on SUN RGB-D · p. 11
• For the all-category comparison with Total3DUnderstanding, camera pose is assumed unknown and ImVoxelNet optimizes L_indoor + L_extra.Source: A. More results on SUN RGB-D · p. 11

• Bounding-box colors encode object categories, with the color mapping kept consistent within each dataset.Source: B. Visualization · p. 11
• Table 8 reports AP@0.25 scores for 10 SUN RGB-D categories, and all methods except ImVoxelNet use point clouds.Source: Table 8 · p. 12
• Tables 9 and 10 report AP@0.15 scores for SUN RGB-D evaluations over 10 of 30 categories and all 30 categories, respectively.Source: Table 9 · p. 12, Table 10 · p. 12
• Table 11 reports AP@0.15 scores for all 37 SUN RGB-D object categories.Source: Table 11 · p. 12
• The figures visualize detections on monocular SUN RGB-D and KITTI validation images and multi-view ScanNet validation scenes.Source: Figure 6 · p. 13, B. Visualization · p. 14, Figure 8 · p. 15
• SUN RGB-D failure examples show precise detections for small sinks, garbage bins, and recycle bins but poor rotation estimates for large cabinets.Source: Figure 10 · p. 16