PaperlayerView exact revision on arXiv

Source-linked AI summary

ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection

Danila Rukhovich, Anna Vorontsova, Anton Konushin

arXiv:2106.01178v3cs.CV

TL;DR

RGB images make 3D object detection difficult because they lack explicit geometry and scale, especially from monocular views. ImVoxelNet addresses this with a fully convolutional voxel-based detector accepting posed monocular or arbitrary-sized multi-view inputs, achieving state-of-the-art results across indoor and outdoor benchmarks.

  • Problem

    RGB images lack explicit scene geometry and absolute scale, while existing methods do not formulate multi-image RGB 3D detection as an end-to-end optimization problem.

  • Method

    ImVoxelNet extracts image features, projects them into a 3D voxel volume, aggregates views, and applies 3D convolutional neck and detection heads.

  • Results

    ImVoxelNet achieves state-of-the-art results across indoor and outdoor datasets, including KITTI, nuScenes, SUN RGB-D, and ScanNet.

  • Takeaways & Limitations

    A single fully convolutional framework supports posed monocular inputs and multi-view inputs with an arbitrary number of views.

Abstract

from arXiv · show

In this paper, we introduce the task of multi-view RGB-based 3D object detection as an end-to-end optimization problem. To address this problem, we propose ImVoxelNet, a novel fully convolutional method of 3D object detection based on monocular or multi-view RGB images. The number of monocular images in each multi-view input can variate during training and inference; actually, this number might be unique for each multi-view input. ImVoxelNet successfully handles both indoor and outdoor scenes, which makes it general-purpose. Specifically, it achieves state-of-the-art results in car detection on KITTI (monocular) and nuScenes (multi-view) benchmarks among all methods that accept RGB images. Moreover, it surpasses existing RGB-based 3D object detection methods on the SUN RGB-D dataset. On ScanNet, ImVoxelNet sets a new benchmark for multi-view 3D object detection. The source code and the trained models are available at https://github.com/saic-vul/imvoxelnet.

1. Introduction

ImVoxelNet formulates posed RGB-based 3D object detection as an end-to-end task spanning monocular and multi-view inputs. Its voxel-based architecture supports indoor and outdoor scenes while retaining a common meta-architecture.

  • Motivation: RGB images lack explicit scene geometry and absolute scale, making monocular 3D object detection ill-posed.Invisible areas also prevent scene geometry from being unambiguously derived from a single RGB image.
  • Motivation: Existing RGB-based methods commonly separate monocular prediction from later multi-view aggregation.This late aggregation may fail to exploit spatial information effectively.
  • Approach: ImVoxelNet constructs a voxel representation to accumulate information from multiple inputs.The resulting unified approach uses the same meta-architecture for indoor and outdoor detection, switching only between domain-specific heads.
  • Contributions: The method is trained end-to-end for multi-view 3D object detection from posed RGB images only.This is presented as the first formulation of this task according to the authors.
  • Contributions: ImVoxelNet is a fully convolutional 3D detector that works in both monocular and multi-view settings.Monocular inputs are treated as a special case of multi-view inputs.
  • Results: Domain-specific heads yield state-of-the-art results across indoor and outdoor datasets.The reported contribution covers both scene types rather than a single benchmark domain.

2. Related Works

Prior work treats multi-view scene understanding through reconstruction, segmentation, or late aggregation, while RGB-based 3D detection lacks an end-to-end multi-image formulation. ImVoxelNet addresses this gap with a shared voxel-based detection framework.

  • Multi-view scene understanding: SLAM, Structure-from-Motion, and Multi-View Stereo use multiple images to estimate poses, geometry, or point clouds.These tasks illustrate established multi-view scene-understanding settings.
  • Multi-view scene understanding: Recent methods also apply multi-view RGB-D or RGB inputs to 3D instance segmentation, semantic segmentation, and TSDF reconstruction.These examples broaden multi-view processing beyond geometric reconstruction alone.
  • Point-cloud-based methods: Point-cloud detectors often voxelize scenes, but outdoor systems commonly project them to BEV to reduce exhaustive 3D-convolution computation.Indoor detectors instead use deep Hough voting because single-center proposals can be irrelevant for non-convex objects.
  • Stereo-based methods: Stereo methods use exactly two images with fixed, precisely known relative transformation, unlike multi-view methods accepting arbitrary inputs and poses.This difference enables stereo reconstruction through left-right optical-flow estimation.
  • Monocular-based methods: Monocular detectors estimate 3D structure through anchors, discretized orientations, decomposed subtasks, or lifted 2D detections.Some methods additionally use external data such as CAD models or mesh annotations.
  • Indoor detection: Indoor monocular 3D detection remains less explored, with SUN RGB-D described as the only existing benchmark.Related methods may jointly estimate camera poses, room layouts, or object meshes.
  • Multi-view detection: Existing nuScenes methods predict independently per monocular image and aggregate outputs only afterward.The paper identifies this late-stage aggregation as potentially ineffective for exploiting spatial information.
  • Research gap: The paper states that existing methods do not formulate multi-image RGB 3D detection as an end-to-end optimization problem.This gap motivates ImVoxelNet’s unified treatment of posed RGB inputs.

3. Proposed Method

ImVoxelNet projects features from an arbitrary-sized set of posed RGB images into a 3D voxel volume, then applies 3D or BEV detection heads. Its architecture adapts dense voxel processing and FCOS-style prediction to indoor and outdoor 3D detection.

  • 3.1. 3D Volume Construction: ImVoxelNet extracts 2D image features, projects them into a 3D voxel volume, averages valid multi-view features, and predicts parameterized 3D bounding boxes.The box tuple is (x, y, z, w, h, l, θ), representing center, dimensions, and rotation around the z-axis.
  • 3.1. 3D Volume Construction: The method accepts an arbitrary-sized set of RGB inputs with camera poses, including monocular inputs as a special case.The number of views can vary across inputs during training and inference.
  • 3.1. 3D Volume Construction: A pinhole camera model maps 2D feature-map coordinates to 3D voxel coordinates within dataset-specific spatial limits and a fixed voxel size.A binary frustum mask identifies valid voxels before feature projection and aggregation.
  • 3.3.2 Indoor Head: Indoor detection uses dense multi-scale 3D features and a novel FCOS-inspired 3D head with classification, location, and centerness branches.The head replaces 2D convolutions with 3D convolutions and samples up to 27 candidate locations per object.
  • 3.3.2 Indoor Head: Indoor training combines focal classification loss, centerness cross-entropy, and rotated 3D IoU location loss.The indoor loss is normalized by the number of positive 3D locations.

4. Experiments

Experiments evaluate ImVoxelNet across indoor and outdoor RGB-based 3D detection benchmarks in monocular and multi-view settings. The method achieves strong results on KITTI, nuScenes, SUN RGB-D, and ScanNet while using dataset-specific implementations and evaluation protocols.

  • Datasets: Experiments cover monocular KITTI and SUN RGB-D benchmarks, and multi-view ScanNet and nuScenes benchmarks.KITTI and SUN RGB-D evaluate monocular inputs, while ScanNet and nuScenes use multi-view formulations.
  • Datasets: KITTI uses car-category evaluation, with moderate objects on the test set serving as the main assessment protocol.The reported validation and test evaluations follow the benchmark’s established difficulty and category conventions.
  • Outdoor results: ImVoxelNet achieves the best moderate AP on KITTI’s test split and surpasses previous state-of-the-art by 6% AP3D and 4% APBEV for easy objects.The method is superior on almost all reported metrics across both test and validation splits.
  • Outdoor results: On nuScenes, ImVoxelNet outperforms MonoDIS by more than 1% mean AP and produces almost twice as many highly accurate estimates under AP@0.5.The method uses multi-view inputs during training, unlike methods that only apply multi-view images at inference.
  • Indoor results: On SUN RGB-D, ImVoxelNet surpasses previous methods by more than 18% mAP and also outperforms Total3DUnderstanding in layout and camera pose estimation.The benchmark includes 37 NYU-37 object categories and evaluates room layout and camera pose alongside detection.
  • Indoor results: On ScanNet, ImVoxelNet remains competitive without point clouds and outperforms point-cloud-based 3D-SIS.Increasing the number of test images per scene improves results, although processing the voxel volume is the most time-consuming pipeline component.

5. Conclusion

ImVoxelNet formulates multi-view RGB-based 3D object detection as end-to-end optimization and handles posed monocular or multi-view inputs with arbitrary view counts. It achieves strong results across outdoor and indoor benchmarks, supporting its general-purpose scope.

  • ImVoxelNet formulates multi-view RGB-based 3D object detection as an end-to-end optimization problem.
  • The method accepts posed multi-view inputs with an arbitrary number of views during training and inference, including monocular inputs as a special case.
  • ImVoxelNet achieves state-of-the-art outdoor car-detection results on the monocular KITTI and multi-view nuScenes benchmarks.
  • The method surpasses existing 3D object-detection methods on SUN RGB-D and sets a new benchmark for indoor multi-view detection on ScanNet.
  • ImVoxelNet works on both indoor and outdoor data, giving the method a general-purpose scope.

A. More results on SUN RGB-D

Additional SUN RGB-D evaluations compare ImVoxelNet under protocols involving different category sets, camera-pose assumptions, and point-cloud baselines. These comparisons establish RGB-only performance against methods with more complete data and under matched evaluation settings.

  • ImVoxelNet is compared with PerspectiveNet using a protocol with 30 SUN RGB-D object categories and matched category training.
  • Ten categories are shared across the 30-category protocol and benchmarks using 10 categories, allowing metrics to be merged on the same subset.
  • The point-cloud comparison on SUN RGB-D uses 10 categories and mAP@0.25 as the main metric.
  • The point-cloud comparison favors point-cloud methods because they access more complete data, while ImVoxelNet provides a monocular RGB baseline.
  • For the all-category comparison with Total3DUnderstanding, camera pose is assumed unknown and ImVoxelNet optimizes L_indoor + L_extra.

B. Visualization

The visualizations show detections across monocular KITTI and SUN RGB-D images and multi-view ScanNet inputs, with dataset-consistent colors marking object categories. SUN RGB-D failure examples highlight accurate small-object detections but poor rotation estimates for large cabinets.

  • Bounding-box colors encode object categories, with the color mapping kept consistent within each dataset.
  • Table 8 reports AP@0.25 scores for 10 SUN RGB-D categories, and all methods except ImVoxelNet use point clouds.
  • Tables 9 and 10 report AP@0.15 scores for SUN RGB-D evaluations over 10 of 30 categories and all 30 categories, respectively.
  • Table 11 reports AP@0.15 scores for all 37 SUN RGB-D object categories.
  • The figures visualize detections on monocular SUN RGB-D and KITTI validation images and multi-view ScanNet validation scenes.
  • SUN RGB-D failure examples show precise detections for small sinks, garbage bins, and recycle bins but poor rotation estimates for large cabinets.
Loading 2106.01178v3…