PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild

Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, Georgia Gkioxari

arXiv:2207.10660v2cs.CV

TL;DR

Monocular 3D detection has been constrained by small, domain-specific benchmarks and specialized methods. The paper introduces the large, diverse OMNI3D benchmark and unified Cube R-CNN detector, which outperforms prior approaches across indoor, urban, and mixed benchmarks while improving single-dataset performance.

  • Problem

    Existing 3D detection benchmarks are small, cover few categories, and focus on individual domains, unlike much larger 2D benchmarks.

  • Method

    The paper combines existing datasets into OMNI3D and extends Faster R-CNN with a 3D cuboid head, IoUness prediction, and virtual depth in Cube R-CNN.

  • Results

    Cube R-CNN outperforms prior methods by 4.1% AP on SUN RGB-D, 9.5% on KITTI, and 7.9% on OMNI3D.

  • Takeaways & Limitations

    OMNI3D improves single-dataset performance by up to 5.3% AP on urban and 3.8% on indoor benchmarks.

  • Takeaways & Limitations

    The method remains limited in localizing far-away objects and uncommon object types or contexts, and assumes known camera intrinsics.

Abstract

from arXiv · show

Recognizing scenes and objects in 3D from a single image is a longstanding goal of computer vision with applications in robotics and AR/VR. For 2D recognition, large datasets and scalable solutions have led to unprecedented advances. In 3D, existing benchmarks are small in size and approaches specialize in few object categories and specific domains, e.g. urban driving scenes. Motivated by the success of 2D recognition, we revisit the task of 3D object detection by introducing a large benchmark, called Omni3D. Omni3D re-purposes and combines existing datasets resulting in 234k images annotated with more than 3 million instances and 98 categories. 3D detection at such scale is challenging due to variations in camera intrinsics and the rich diversity of scene and object types. We propose a model, called Cube R-CNN, designed to generalize across camera and scene types with a unified approach. We show that Cube R-CNN outperforms prior works on the larger Omni3D and existing benchmarks. Finally, we prove that Omni3D is a powerful dataset for 3D object recognition and show that it improves single-dataset performance and can accelerate learning on new smaller datasets via pre-training.

1. Introduction

The paper addresses the limited scale, category coverage, and domain specificity of monocular 3D detection by introducing OMNI3D and Cube R-CNN. OMNI3D unifies diverse data, while Cube R-CNN provides a unified detector that outperforms prior methods across indoor, urban, and mixed settings.

  • 3D object detection estimates each object’s 3D location and extent as a tight oriented 3D bounding box from a single image.
  • Existing benchmarks are small and domain-specific, with KITTI containing 7k images and SUN RGB-D 10k, compared with COCO’s 20× larger scale.Existing methods also rely on domain-specific assumptions such as ground-plane placement, yaw-only rotation, or limited depth ranges.
  • 234k images, 3 million annotated objects, and 98 categories make OMNI3D a large, diverse benchmark assembled from publicly released datasets.The benchmark is 20× larger than popular existing 3D benchmarks, and its fast 3D-box IoU algorithm is 450× faster than previous solutions.
  • Cube R-CNN detects 2D objects and predicts their 3D locations, sizes, and rotations end-to-end across domains and many categories.Virtual depth addresses scale-depth ambiguity caused by varying focal lengths and supports image-rescaling augmentation.
  • 4.1% AP on indoor SUN RGB-D, 9.5% on urban KITTI, and 7.9% on OMNI3D are the reported gains over ImVoxelNet, GUPNet, and PGD respectively.

2. Related Work

Prior work develops monocular 3D detection largely within separate urban and indoor domains, while datasets likewise differ in task, scene, and annotation properties. The paper positions Cube R-CNN and OMNI3D as unified alternatives drawing on 2D and 3D detection advances.

  • 2D Object Detection: 2D detection includes two-stage methods that refine region proposals and single-stage methods that predict regions directly from backbone features.
  • 3D Object Detection: Urban monocular 3D detection centers heavily on cars and includes center-based, anchor-based, and anchorless detector families.
  • 3D Object Detection: Indoor 3D methods address room layouts, oriented object boxes, voxel shapes, and joint box-and-mesh prediction, often using domain-specific frameworks.
  • 3D Datasets: Existing 3D datasets target different applications and carry distinct biases in object and scene types, focal lengths, and coordinate systems.The paper unifies representative indoor, outdoor, and general datasets.
  • 3D Datasets: OMNI3D analysis measures object-center distributions, normalized depth, relative 2D scale, and category frequency.

3. The OMNI3D Benchmark

OMNI3D bridges the scale and diversity gap between 3D and 2D detection benchmarks by combining urban, indoor, and general datasets. Its spatial, depth, scale, and category statistics show broad variation and substantial challenges for 3D recognition.

  • 7k images and 3 categories characterize KITTI, while SUN RGB-D has 10k images, illustrating the small scale and narrow coverage of primary 3D benchmarks.
  • 234k images, 3 million labeled 3D boxes, and 98 categories comprise OMNI3D across urban, indoor, and general scenes.The source datasets include nuScenes, KITTI, SUN RGB-D, ARKitScenes, Hypersim, and Objectron.
  • 175k/19k/39k images form the train/validation/test splits, with indoor and outdoor subsets defined separately and Objectron reserved for full OMNI3D.
  • Dataset Analysis: OMNI3D has center-biased object locations and predominantly small objects, with outdoor objects noticeably smaller than objects in the full dataset.Relative object size is defined as the square root of object area divided by image area.
  • Dataset Analysis: 88% of instances fall within the [0, 20m] visualization depth range, although OMNI3D extends to 300m and is more depth-diverse than SUN RGB-D and KITTI.
  • Dataset Analysis: The 98-category distribution is long-tailed, making low-shot recognition important for both 2D and 3D performance.
  • Cube R-CNN processes an RGB image, detects objects in 2D, predicts 3D cuboids, and trains by comparing cuboid corners with 3D ground truth using Chamfer distance.

4. Method

Cube R-CNN extends Faster R-CNN with a unified 3D detection head that predicts cuboids for detected 2D objects across domains. Its design addresses incomplete labeling, camera-intrinsic variation, and coupled 3D prediction errors.

  • Cube R-CNN extends Faster R-CNN with a 3D detection head that predicts a cuboid for each detected 2D object.The framework is designed without domain- or object-specific strategies.
  • IoUness: The RPN replaces binary objectness with foreground IoUness regression because OMNI3D does not guarantee that all instances are labeled.Regions with IoU above 0.05 are used, and the loss is weighted by ground-truth IoU to prioritize candidates near true objects.
  • Cube Head: The cube head predicts 13 category-specific parameters describing projected center, depth, dimensions, allocentric rotation, and 3D uncertainty.It processes 7×7 pooled feature maps through two fully connected layers with 1024 hidden dimensions.
  • 3D box construction: The predicted 3D center combines the projected image-plane center with depth, while box dimensions are recovered from log-normalized category-specific size parameters.The center uses known focal lengths and principal point; dimensions use pre-computed category-specific means.
  • 3D box construction: The cuboid is formed by scaling an axis-aligned unit cube, rotating it with a 6D pose representation, and translating it to the predicted 3D center.The model derives a 3 × 3 rotation matrix and uses eight cuboid corners.
  • Training objective: The training objective combines 2D detection losses with chamfer-based 3D losses, including disentangled variable-group losses and uncertainty weighting.Predicted uncertainty can trade a penalty to reduce the 3D loss for uncertain examples and helps improve inference ratings.
  • Virtual depth: Virtual depth rescales metric depth using known camera intrinsics so effective image height and focal length remain consistent across images.This reduces scale-depth ambiguity and enables scale augmentations during training, although the method assumes known camera intrinsics.

5. Experiments

Experiments evaluate Cube R-CNN across Omni3D, existing indoor and urban benchmarks, ablations, cross-dataset transfer, and pre-training. The results support a unified detector and show benefits from virtual depth, uncertainty modeling, diverse training data, and OMNI3D pre-training.

  • Benchmark comparisons: Cube R-CNN is evaluated as a unified detector across Omni3D, KITTI, SUN RGB-D, and mixed indoor–urban settings.The experiments compare against domain-specific methods and assess both benchmark performance and cross-dataset generalization.
  • Ablations: +6% AP3D: virtual depth improves performance most on full Omni3D, where camera intrinsics vary most.Virtual depth addresses focal-length variation across mixed datasets.
  • Ablations: +6%: 3D uncertainty improves performance across all Omni3D subsets by informing prediction confidence and loss weighting.Using uncertainty only during training still improves AP3D by +1.6% at inference.
  • Ablations: +3.1% AP3D: entangled 3D loss outperforms disentangled losses, while IoUness improves Omni3D AP3D by +1.1%.The ablation compares alternative loss designs and RPN scoring choices.
  • Benchmark comparisons: Cube R-CNN outperforms competitive methods on OMNI3DOUT and OMNI3D, with virtual-camera extensions also improving competing approaches.PGD gains +4.2% on OMNI3D from the virtual-camera extension.
  • Benchmark impact: OMNI3D training improves single-dataset performance and enables rapid adaptation: 5% of a target dataset reaches more than 70% of upper-bound performance after pre-training.ARKit improves by +2.6% and KITTI by +5.3% in the reported cross-dataset comparison.

6. Conclusion

The paper concludes that OMNI3D and Cube R-CNN support general 3D object detection across diverse scenes and categories. It also identifies limitations for far-away objects, uncommon contexts, and settings without known camera intrinsics.

  • Conclusion: OMNI3D is a large, diverse benchmark, and Cube R-CNN is a general-purpose detector for image-based 3D object detection.The conclusion presents the dataset and model as the paper’s central contributions.
  • Limitations: The method remains limited when localizing far-away objects or handling uncommon object types and contexts.These limitations are identified from the paper’s extensive analysis.
  • Limitations: Accurate real-world 3D predictions assume known camera intrinsics, though future self-calibration could reduce this limitation.The stated scope boundary concerns camera information required by the method.
  • Dataset analysis: Object-center and depth distributions vary across Omni3D subsets and other 3D and 2D datasets.Figure 7 compares normalized image-coordinate and depth distributions across these datasets.

A1. Dataset Details

The dataset details describe how OMNI3D is assembled from existing sources and how its splits and category statistics are reported.

  • Dataset construction: OMNI3D reuses official dataset splits where available and otherwise derives validation and test sets using sequence-based partitioning.When no public test set exists, the validation set is used as test data; remaining training data may be split 10:1.
  • Dataset statistics: Table 6 reports category counts and train, validation, and test image totals for each dataset split.It distinguishes all categories from those with at least 1000 positive instances used in the paper’s AP3D metrics.

A2. Model Details

The model-details section focuses on Cube R-CNN’s allocentric 3D-box rotation and its virtual-depth derivation.

  • Model details: Cube R-CNN’s model details explain allocentric 3D bounding-box rotation and the derivation of virtual depth.These components are treated as the section’s key technical details.

A2.1. 3D Box Rotation

Cube R-CNN represents 3D box rotation with a continuous 6D parameter mapped to a rotation matrix, then converts allocentric predictions into egocentric form using camera geometry. Allocentric rotation remains visually consistent across spatial locations, unlike egocentric rotation.

  • 3D Box Rotation: Camera intrinsics and the predicted 2D center define a ray used to convert allocentric rotation into the final egocentric rotation.The conversion forms an axis-angle matrix M and computes R = M · Ra.
  • 3D Box Rotation: Allocentric rotation stays visually consistent under location changes, whereas identical egocentric rotations can appear different at different spatial locations.This alignment makes allocentric rotation more compatible with visual 2D evidence.

A2.2. Virtual Depth

Virtual depth transforms predictions into an invariant camera space so varying image resolutions and focal lengths produce consistent depth representations. The transformation preserves corresponding pixel locations between original and virtual images.

  • Virtual Depth: Virtual depth helps Cube R-CNN handle varying input resolutions and camera intrinsics.The authors also report that it benefits competing approaches as a general-purpose feature.
  • Virtual Depth: The transformation keeps effective image size and focal length consistent in an invariant camera space.Camera systems with nearly identical visual evidence therefore map to the same virtual depth.
  • Virtual Depth: The virtual 3D point (X, Y, Zv) projects to a virtual image location corresponding to the same pixel as (x, y) in the original image.The vertical coordinates are related through the original and virtual image heights.
  • Virtual Depth: The projection derivation substitutes the original and virtual image-height relationships to convert between metric and virtual depth.The displayed derivation uses focal lengths and principal-point coordinates in the projection equations.

A2.3. Training Details and Efficiency

The appendix describes training and evaluation choices for Cube R-CNN, including compute requirements, fast 3D IoU, and comparisons under the nuScenes protocol. It also documents performance behavior across categories, datasets, and qualitative in-the-wild settings.

  • Training Details and Efficiency: Cube R-CNN training on smaller subsets uses linearly adjusted learning rates, batch sizes, and iterations to reach 128 epochs.Training takes 14–26 hours on V100 GPUs in multi-node settings and uses approximately 1.6 GB per image.
  • Training Details and Efficiency: 3D IoU declines much faster than 2D IoU when a predicted box is translated along depth, motivating relaxed AP3D IoU thresholds.The example translates a box by up to one unit length while keeping rotation and dimensions fixed.
  • Training Details and Efficiency: The fast IoU3D algorithm computes the intersection polyhedron of two oriented boxes by collecting planar intersection units and measuring their volume.It iterates over box sides, removes duplicates, and computes the resulting shape’s volume.
  • Training Details and Efficiency: The nuScenes comparison uses a single front camera and removes velocity and attribute metrics from NDS, while Cube R-CNN remains competitive but slightly trails FCOS3D and PGD on center-distance mAP.Those baselines use nuScenes-specific tuning and test-time augmentation.
  • Training Details and Efficiency: On all 98 categories, AP3D decreases from 23.3 to 14.1 and AP2D from 27.6 to 17.3 compared with the 50-category setting.The authors attribute the long-tail decline to fewer positive examples for learning.
  • Training Details and Efficiency: Qualitative tests show generalization to COCO and headset-mounted video, but unknown camera intrinsics make predicted 3D localization accurate only up to a scaling factor.The COCO visualization assumes f = 2 · H, px = W/2, and py = H/2.
Loading 2207.10660v2…