Cubify Anything: Scaling Indoor 3D Object Detection

1. Introduction

• Noisy depth and pose, low-resolution 3D inputs, and room-object-focused labels leave smaller objects and image-level alignment under-addressed.Reprojecting boxes annotated on noisy representations also produces misalignment, inconsistency, and jitter.Source: §1 · p. 2
• CA-1M extends ARKitScenes LiDAR and handheld RGB-D captures with FARO-labeled 9-DOF scene boxes, projected to every handheld frame to produce spatially realistic, pixel-perfect 2D and 3D boxes.The dataset contains over 439K unique objects across 3,500 handheld captures and nearly 13M training frames with at least one ground-truth instance.Source: §1 · p. 2
• CuTR predicts 2D and 3D boxes from RGB with optional depth, without lifting inputs into 3D or adding machinery beyond ordinary 2D detection architectures.The model is designed to scale to CA-1M's millions of images and pixel-perfect 3D boxes.Source: §1 · pp. 2–3
• With sufficient training data, CuTR can outperform point-based architectures on CA-1M and smaller, entangled datasets such as SUN RGB-D.The paper positions this comparison as a test of whether 3D inductive biases remain preferable in a large, accurate, diverse data regime.Source: §1 · pp. 2–3

2. Related Work

• Existing indoor datasets have limitations in size, object diversity, annotation-source accuracy, or explicit 3D box availability compared with CA-1M.SUN RGB-D emphasizes large room-defining objects, while ScanNet V2 and ScanNet++ do not provide explicit oriented 3D box labels.Source: §2 · p. 3
• Point-based methods operate directly on point clouds, while voxel-based methods discretize 3D space and can use sparse convolutions for efficiency.Some voxel-based variants can operate without depth by backprojecting rays through a dense voxel grid.Source: §2 · p. 3
• Image-based methods predict 3D boxes from images, with monocular approaches relying on learned priors; this paper focuses on single-image detection with optional commodity-LiDAR depth.The single-image setting avoids scan aggregation and aligns the task more closely with 2D detection.Source: §2 · p. 3
• Omni-detection work targets inter-dataset generalization through dataset aggregation, whereas CA-1M targets intradataset completeness and accuracy.The stated aim is supervision for detecting any object in a dataset.Source: §2 · p. 3

3. CA-1M Dataset

• CA-1M has over 439K objects across 3.5K videos, compared with ARKitScenes' 56K objects across 5K videos, and targets exhaustive class-agnostic labeling.Unlike ARKitScenes, CA-1M labels on FARO scans and renders annotations to each frame for accurate, pixel-perfect alignment.Source: §3 · p. 4, Table 1 · p. 4

3.2. Rendering CA-1M

• Per-frame ground truth is rendered from world-space boxes, then filtered and cut to reflect the frame's visibility and occlusion.The pipeline uses projected boxes and a coarse instance mask to determine frame-specific boxes.Source: Figure 4 · p. 4, §3.2 · p. 5

3.1. Annotation of CA-1M

• Annotators use registered RGB frames and automatically selected supporting views to label objects even when FARO scans omit regions.The tool also supports initializing an object with a 2D box from a bootstrapped CuTR model.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. Rendering CA-1M

• CA-1M shares stationary laser scanners' missing-region limitation, but explicit 3D boxes and multi-view image support allow annotation of objects absent from the FARO mesh.ScanNet++ labels mesh instances without explicit 3D boxes, leaving a mismatch when objects appear in RGB captures but not the scan.Source: §3.1 · p. 4, Figure 5 · p. 5

4. Cubify Transformer

• CuTR uses a ViT backbone for RGB and RGB-D variants, with MultiMAE providing RGB-depth encoding for the latter and Depth-Anything initializing the RGB-only variant.RGB and depth can use asymmetric resolutions, reflecting the higher resolution generally available from RGB sensors.Source: §4 · p. 5
• CuTR predicts yaw but leaves non-zero pitch and roll prediction to future work, using gravity-aligned boxes for training and evaluation.The model uses Chamfer loss on predicted box corners and Hungarian matching over 2D box predictions.Source: §4 · p. 5
• Built on Plain DETR, CuTR uses global cross-attention modulated by query box estimates and predicts 2D boxes, 3D boxes, and class scores.The 3D box predictor outputs projected center coordinates and depth, dimensions, and orientation, then backprojects using camera intrinsics.Source: §4 · p. 5, Figure 6 · p. 6
• The RGB and RGB-D variants use MLP and vanilla Transformer operations, enabling acceleration on Metal, Apple Neural Engine, and ONNX without custom sparse-convolution or voxelization kernels.The paper contrasts this accessibility with operations requiring custom accelerator kernels.Source: §4 · p. 6

5. Experiments

• The suite compares CuTR with FCAF3D and TR3D on SUN RGB-D, ScanNet++, and CA-1M, reporting AP/AR at 3D IoU 0.25 and 0.50, with evaluation capped at 100 detections.Source: §5 · p. 6

5.1. SUN RGB-D

• On traditional SUN RGB-D, point-based methods generally outperform CuTR, but on the more diverse 38-class Omni3D split, CuTR improves AP25 over FCAF and TR3D variants.The traditional split covers 10 classes and large objects; the Omni3D split generally includes more small objects.Source: §5.1 · p. 6, Table 2 · p. 7

5.2. CA-1M

• Nearly 62% of objects are recalled by CuTR on CA-1M at 25% IoU, 10% more than the best point-based methods; RGB-only CuTR recalls nearly 35%, almost 13% more than the best RGB-only method.CuTR also maintains relatively excellent precision levels.Source: §5.2 · p. 7, Table 2 · p. 7

5.2.1. Explaining (some of) the gap

• With FARO ground-truth depth, point-based methods improve substantially and CuTR’s lead narrows, though CuTR still leads except in AP50.The authors suggest point-based methods may be less capable of handling noisy depth signals because backprojection and voxelization are relatively hard operations.Source: Table 3 · p. 7, §5.2.1 · p. 8

5.3. CA-1M as Pretraining: SUN RGB-D Revisited

• CA-1M pretraining gives CuTR over 13 points in AR25 and 12 in AR50, enabling it to significantly outperform point-based methods on Omni3D SUN RGB-D.CA-1M also provides significant pretraining benefit over ARKitScenes and ScanNet++.Source: §5.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5.4. Qualitative Results

• CuTR qualitatively detects more objects, especially small ones, handles transparency, and aligns better with images, while FCAF misses or groups small and thin objects and produces false positives and rotation errors.Figure 7 shows detection and reprojected views; FCAF uses NMS while CuTR has no post-processing.Source: Figure 7 · p. 7, §5.4 · p. 8

6. Conclusion

• CA-1M and CuTR challenge the idea that point-based 3D architectures are better suited to indoor 3D object detection.The conclusion reports that image-based CuTR improves both recall and precision on CA-1M, with its exhaustive annotations derived from highly accurate laser scans.Source: §6 · p. 8

7.1. Additional Dataset Samples

• The appendix includes additional panoramic projections of sample annotations.Source: §7.1 · p. 10

7.2. Rendering CA-1M

• The per-frame ground-truth pipeline transforms boxes into camera space, clips them to the frustum and scene, evaluates visibility, and removes boxes cut too significantly.It uses per-frame intrinsics, distortion parameters, and poses registered to the laser scans; the process is parallelized across GPUs and nodes.Source: §7.2 · p. 10

7.2.1. Importance of Handling Distortion

• Box rendering must account for distortion in handheld captures to maintain pixel alignment, a nontrivial requirement for 3D boxes.Figure 8 contrasts distortion-aware rendering with a pinhole-only model.Source: §7.2.1 · p. 10

8.1. Additional Training Details

• CuTR is trained with settings similar to ViTDet, while Cube R-CNN uses Omni3D training settings; point-based methods use each work’s default MMDetection3D settings.Source: §8.1 · p. 10
• Distortion-aware box rendering is important on CA-1M for pixel accuracy and alignment, particularly for small, distant objects.Figure 8 contrasts distortion-aware rendering with a pinhole model.Source: Figure 8 · p. 10
• Image-based models train across 8×4 V100 nodes with batch size 64 and apply random resizing, cropping, image-depth ratio changes, and horizontal flips.Depth maps are augmented to preserve the image-to-depth ratio after image resizing and cropping.Source: §8.1 · p. 10

9.1. ScanNet++ versus CA-1M

• CA-1M explicitly labels every object, while ScanNet++ derives boxes from mesh instance labels that can miss objects absent from the FARO scan.ScanNet++ has 230 captures versus 3,500 in CA-1M, and directly registers 8% of its frames; CA-1M registers nearly every frame.Source: §9.1 · pp. 10–11

9.2. Experiments on ScanNet++

• ScanNet++ boxes are derived from instance segmentation using a wall normal to set scene rotation, which can rarely violate a Manhattan assumption.Per-frame ground truth renders modal instance segmentation, and experiments treat the dataset as class-agnostic.Source: §9.2 · p. 11
• CuTR’s RGB and RGB-D variants outperform existing methods across the range of ScanNet++ evaluation thresholds.Table 6 evaluates class-agnostic precision and recall.Source: §9.2 · p. 11

• No specific ARKitScenes results are reported; per-frame ARKitScenes pretraining is used for experiments in Section 5.Source: §10 · p. 11
• Table 6 reports class-agnostic precision and recall on ScanNet++ using ARKit depth and axis-aligned boxes derived from instance segmentation.Source: Table 6 · p. 11
• Per-frame ground truth follows the CA-1M protocol, with original ARKitScenes poses and 3D box annotations.Source: §10 · p. 11

Model Size

• CuTR scales gracefully to ViT-S and ViT-T backbones, retaining performance competitive with point-based methods even at ViT-T size.Table 7 presents the smaller-backbone results on CA-1M.Source: §11 · p. 11, Table 7 · p. 11

Depth (Modality) Fusion

• MultiMAE initialization shows stronger 3D accuracy than DINOv2 or Depth-Anything initialization, despite similar 2D box performance.The alternatives learn depth fusion during training, whereas MultiMAE is pre-trained with depth fusion.Source: §11 · pp. 11–12, Table 8 · p. 12

Pixel for point

• At 256×192 RGB and depth, CuTR matches point-based methods; at 512×384, it has minimal losses versus the 1024×768 evaluation setting.Both 2D and 3D recall suffer at lower RGB resolutions, while CuTR remains competitive with point-based methods.Source: Table 9 · p. 12, §11 · p. 12
• Resolution-related performance drops are largest for medium and far boxes (2–4 m and 4–5 m), potentially reflecting increased depth uncertainty and smaller image size.Table 10 breaks down Table 9 results by near, medium, and far distances.Source: §11 · p. 12, Table 10 · p. 12