PaperlayerView exact revision on arXiv

Source-linked AI summary

WildDet3D: Scaling Promptable 3D Detection in the Wild

Weikai Huang, Jieyu Zhang, Sijun Li, Taoyang Jia, Jiafei Duan, Yunqian Cheng, Jaemin Cho, Matthew Wallingford, Rustin Soraki, Chris Dongjoo Kim, Shuo Liu, Donovan Clay, Taira Anderson, Winson Han, Ali Farhadi, Bharath Hariharan, Zhongzheng Ren, Ranjay Krishna

arXiv:2604.08626v2cs.CV

TL;DR

Monocular 3D detectors must recover object geometry while generalizing beyond closed-set categories, supporting varied prompts, and using available geometric cues. WildDet3D addresses these gaps with a unified geometry-aware detector and WildDet3D-Data, a human-verified in-the-wild dataset. It reports state-of-the-art results across benchmarks and zero-shot transfer, with further gains from depth cues, while distant or heavily occluded objects remain limited by single-image depth ambiguity.

  • Problem

    Existing monocular 3D methods lack open-world generality, unified support for multiple prompt types and additional geometric cues, while datasets cover narrow categories in controlled environments.

  • Method

    WildDet3D unifies text, point, and box prompts in a geometry-aware monocular detector, alongside WildDet3D-Data with human-verified 3D annotations across diverse real-world categories.

  • Results

    WildDet3D achieves state-of-the-art results across open-world, standard, and zero-shot settings, including 34.2 AP3D text and 36.4 AP3D oracle on Omni3D.

  • Takeaways & Limitations

    The model functions as a general-purpose 3D perception module across diverse platforms and applications, including iPhone, Meta Quest 3, robotic manipulation, and VLM-based spatial reasoning.

  • Takeaways & Limitations

    Performance remains limited for distant or heavily occluded objects because single-image input cannot fully resolve metric depth without additional cues.

Abstract

from arXiv · show

Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).

1 Introduction

WildDet3D targets open-world monocular 3D detection by unifying diverse prompts and optional geometric cues, while WildDet3D-Data broadens training coverage through human-verified 3D annotations. Together, they support strong performance across open-world, standard, and zero-shot settings, with additional gains when depth is available.

  • Motivation: Monocular 3D detection must recover object position, extent, and pose from a single RGB image for spatially grounded applications.The paper motivates this capability for robotics, embodied AI, autonomous driving, and AR/VR.
  • Motivation: Existing methods lack the generality needed for open-world use, including long-tailed categories, unseen objects, multiple prompt modalities, and additional depth signals.The stated prompt modalities include language, points or regions, and bounding boxes.
  • Geometry and deployment: Optional depth retains visual semantics for open-vocabulary recognition while reducing metric scale ambiguity when geometric signals are available.The architecture also includes auxiliary depth estimation and 2D detection components that provide complementary supervision.
  • Model: WildDet3D unifies text, 2D point, and 2D box prompts in a geometry-aware architecture that predicts 3D boxes from a single image and can use partial depth.Its design combines open-vocabulary visual recognition with monocular geometry estimation.
  • Data: WildDet3D-Data contains over 1M images across 13.5K categories, using candidate 3D boxes generated from 2D annotations and retained after human verification.The dataset spans diverse real-world scenes to expand vocabulary and scene diversity.
  • Results: 22.6/24.8 AP3D is achieved on WildDet3D-Bench with text/box prompts, while Omni3D reaches 34.2/36.4 AP3D and zero-shot transfer reaches 40.3/48.9 ODS on Argoverse 2/ScanNet.The reported pairs preserve the paper’s task and prompt distinctions: text then box for WildDet3D-Bench and Omni3D, and Argoverse 2 then ScanNet for zero-shot ODS.
  • Deployment: WildDet3D is demonstrated in an interactive web demo, on-device iPhone inference, AR/VR systems, vision-language integration, and robotic manipulation.These applications are presented as plug-and-play deployments across platforms and upstream systems.

2 WildDet3D

WildDet3D is a geometry-aware monocular 3D detector that unifies multiple prompt modalities and incorporates optional depth and camera cues. Its dual-vision architecture separates semantic and geometric encoding before fusion, then predicts prompt-conditioned 3D boxes.

  • Dual-vision encoder: The model combines a segmentation-pretrained image encoder with a pluggable RGBD encoder for semantically rich and metrically informed features.The RGBD branch accepts optional partial or full depth and can generate depth features from RGB when external depth is unavailable.
  • Dual-vision encoder: Depth fusion injects projected depth latents into visual features through a residual module while preserving pretrained visual features at initialization.LayerNorm and a 1×1 convolution transform the depth latents; zero initialization makes the fusion initially an identity mapping.
  • Promptable detection: WildDet3D accepts text, point, box, and exemplar prompts within one promptable detector.Text selects category instances; point and box prompts select individual objects, while exemplar prompts retrieve visually similar objects.
  • Geometry-aware decoding: Camera-ray and depth cross-attention modules enrich decoder hidden states with geometric information before 3D box prediction.Camera intrinsics produce ray directions encoded with spherical harmonics, while projected depth latents are fused into the query embedding space.

3 WildDet3D-Data

WildDet3D-Data scales open-vocabulary 3D detection data by lifting diverse 2D annotations into candidate 3D boxes, filtering them, and selecting qualified annotations. The resulting dataset combines over 1M images, 3.7M valid annotations, and 13.5K categories with human and VLM-based quality control.

  • Dataset overview: 13.5K categories span over 1M images and 3.7M valid 3D annotations, expanding category coverage 138× over Omni3D’s 98 categories.The dataset covers 22 scene categories and diverse real-world settings.
  • Construction pipeline: The pipeline generates candidate 3D boxes from existing 2D annotations, applies geometric and semantic filters, and selects candidates through human or VLM verification.Candidate generation uses multiple complementary models before filtering and selection.
  • Annotation sources: Human annotation covers ∼103K images, while VLM filtering adds ∼896K images with automatically verified annotations.Human ratings classify annotations as good_fit, acceptable, or unacceptable; unacceptable annotations are ignored.
  • Category coverage: 826 of 881 validation categories, or 99.9%, have at least one training annotation, and approximately 820 have at least three.This indicates broad training coverage of validation categories.
  • Candidate quality: SAM-3D contributes ∼55% of selected boxes, RANSAC-PCA ∼28%, and LabelAny3D ∼17%, reflecting complementary candidate-generation strengths.Human evaluation also finds substantial quality differences: RANSAC-PCA has a 12.5% rejection rate, versus 42.9% for DetAny3D.
  • Pipeline validation: VLM scores correlate perfectly monotonically with human rejection rates, but score 10 still has a 16.7% rejection rate, so human verification remains necessary.The VLM top-2 candidates cover 73.4% of human selections, motivating VLM pre-filtering followed by human review.

4 Experiments

WildDet3D is evaluated across in-the-wild, standard, zero-shot, and depth-assisted settings using text and oracle box prompts. It achieves strong benchmark and transfer performance, with especially large gains from additional depth and joint 2D–3D prediction.

  • Depth assistance: 19.0 AP is gained by adding ground-truth depth to the full text-prompt model, increasing performance from 22.6 to 41.6 AP.The Omni3D-only model improves from 6.8 to 20.7 AP, a +13.9 gain.
  • Prompt modes: Box prompts outperform text prompts without depth: 24.8 versus 22.6 AP for the full model, indicating a 2D detection bottleneck.The same pattern appears for Omni3D-only training: 8.4 versus 6.8 AP.
  • Omni3D: 34.2 AP with text prompts and 36.4 AP with oracle box prompts on Omni3D surpass prior methods despite training for only 12 epochs.Sparse depth raises oracle performance to 45.8 AP, while gains are especially pronounced on indoor datasets.
  • Zero-shot transfer: 40.3 ODS on Argoverse 2 and 48.9 ODS on ScanNet outperform 3D-MOOD Swin-B by +16.5 and +17.4 ODS, respectively.The model also reports stronger detection AP and orientation estimation on both datasets.
  • Stereo4D: 27.7 AP with real depth on Stereo4D is a 2.8× improvement over OVMono3D-LIFT’s 9.9 AP.Without depth, WildDet3D reaches 7.5 AP, competitive with DetAny3D’s 7.1 AP.
  • Ablations: Removing the shared 2D head collapses AP from 30.2 to 11.1, showing that joint 2D and 3D prediction is the most critical architectural choice.Indoor datasets are particularly affected, including SUNRGBD, where AP falls from 33.9 to 5.1.
  • Depth ablation: Sparse depth adds +9.4 AP in oracle mode and +7.4 AP in text mode without architectural changes.The gains are particularly large on indoor datasets equipped with RGB-D sensors.

5 Applications

WildDet3D is demonstrated as a general-purpose 3D perception module across mobile, AR, robotics, and vision-language applications. Its box-prompt interface also lets grounded language models lift queried 2D regions into 3D.

  • Deployment scenarios: WildDet3D is deployed across on-device mobile inference, AR headsets, robotic manipulation, and VLM-based spatial reasoning.These scenarios span iPhone, Meta Quest 3, robotic grasping, and language-guided localization.
  • Deployment scenarios: The iPhone application supports text and box prompts, streams RGB and LiDAR depth to cloud inference, and renders detected 3D boxes as AR overlays.ARKit provides depth capture and world tracking for scene-anchored visualization.
  • Deployment scenarios: WildDet3D integrates with Meta Quest 3 by combining passthrough images, calibrated intrinsics, and tracked 6-DoF poses to render 3D detection overlays.The system exposes queried objects in the user’s augmented environment.
  • Deployment scenarios: For robotics, predicted 3D box centers and dimensions are transformed into the robot frame and consumed for grasp-pose generation.The demonstrated pipeline uses a Franka Emika Panda arm and an IK-based interpolation planner.
  • Vision-language integration: WildDet3D-Agent combines VLM reasoning with box-prompted 3D detection: a VLM returns a relevant 2D box, which WildDet3D lifts into a 3D cuboid.The figure compares WildDet3D-Agent’s 3D outputs with VST and Qwen3-VL direct 3D localization.

6 Related work

Related work spans monocular 3D detection, open-vocabulary promptable perception, and scalable 3D annotation. Existing efforts establish useful benchmarks and lifting pipelines, while automatic annotations remain noisy and coverage remains limited.

  • Monocular 3D object detection: Monocular 3D detection recovers 3D boxes from one RGB image but is ill-posed because of scale ambiguity, occlusion, and missing geometric cues.Earlier work largely targeted closed-set, domain-specific driving and indoor benchmarks, while Omni3D pursued cross-dataset unification.
  • Open-vocabulary and promptable perception: Open-vocabulary detectors use language supervision to detect beyond fixed taxonomies, while promptable segmentation systems move perception toward interactive interfaces.The cited line of work includes GLIP, OWL-ViT, Grounding DINO, SEEM, and SAM 3.
  • 3D annotation pipelines and open-world data: Large-scale 3D annotation is costly because it requires metric structure, camera parameters, and geometric verification.Omni3D provides a valuable multi-dataset benchmark but does not eliminate the annotation bottleneck.
  • 3D annotation pipelines and open-world data: Lifting 2D cues into 3D and combining reconstruction with model- and human-in-the-loop annotation can provide supervision at scale, but automatic annotations remain noisy.Noise is especially relevant to object scale, rotation, and extent.

7 Limitations

WildDet3D remains constrained by camera calibration, monocular depth ambiguity, rotation estimation, computation, rare categories, and prototype-level deployment boundaries.

  • Geometry and perception: Predicted camera intrinsics are less accurate than ground-truth calibration, degrading absolute depth and physical dimensions when metadata is unavailable.Closing this calibration gap remains an open challenge for in-the-wild deployment.
  • Geometry and perception: Single-image depth ambiguity limits performance on distant or heavily occluded objects despite learned depth priors.The gains from sparse depth input expose this unresolved monocular bottleneck.
  • Geometry and perception: Rotation prediction is the weakest component for near-symmetric objects or scenes with limited visible surface area.Round tables and square boxes provide inherently ambiguous orientation signals.
  • Efficiency: The dual-backbone design increases memory and compute requirements, making the full model unsuitable for real-time on-device inference without distillation or quantization.The authors consider the cost acceptable for server-side deployment.
  • Data coverage: Rare categories lag behind frequent ones, and few training examples produce high variance in 3D prediction quality.The long-tailed dataset distribution only partially addresses this limitation.
  • Deployment boundaries: The demonstrated iPhone, AR, robotics, and VLM systems are research prototypes rather than production-ready or safety-critical systems.Predictions may contain incorrect depth, dimensions, or missed detections, with no guaranteed error bounds.

8 Conclusion

The paper presents WildDet3D and WildDet3D-Data as complementary model and data advances for open-world monocular 3D detection. The system achieves strong benchmark and zero-shot results while supporting deployment across diverse platforms.

  • Contributions: WildDet3D unifies text, point, and box prompts in one geometry-aware monocular 3D detector.Its architecture includes dual vision encoders, optional-depth fusion, prompt integration, and a 3D detection head.
  • Contributions: WildDet3D-Data contains 1M images and 13.5K categories with human-verified 3D annotations for open-world supervision.Its multi-model candidate generation and human/VLM selection expand category coverage by 138× over Omni3D.
  • Results: 34.2 AP3D text and 36.4 AP3D oracle are achieved on Omni3D, alongside zero-shot ODS of 40.3 on Argoverse 2 and 48.9 on ScanNet.The reported Omni3D results use 6–10× fewer training epochs than prior methods.
  • Results and applications: WildDet3D shows strong open-world transfer across 700+ in-the-wild categories and practical deployment on iPhone, Meta Quest 3, robotic manipulation, and VLM spatial reasoning.These demonstrations support its use as a general-purpose 3D perception module across platforms and applications.

B.1 Three-stage training pipeline.

The three-stage pipeline uses standardized distributed training, mixes diverse datasets in Stage 2, and applies mask-guided point/box training in Stage 3 while selectively freezing backbone parameters.

  • Optimization: All three stages use AdamW with learning rate 10^-4, weight decay 10^-4, 32 GPUs, and total batch size 128.The learning rate decays by 0.1× at two stage-specific fractional epoch milestones.
  • Stage 2: Stage 2 mixes seven datasets, with Omni3D at 40%, FoundationPose and WildDet3D-Data at 20% each, and the remaining sources at smaller proportions.The remaining sampling proportions are CA-1M 10%, Waymo 5%, and 3EED-det and 3EED-ref 2.5% each.
  • Stage 3: Stage 3 combines Omni3D at 90% with 10% human-annotated WildDet3D-Data for mask-guided training.Point and box prompts, including points sampled inside SAM 2 masks, provide geometric inputs for tighter 3D localization.
  • Parameter freezing: The SAM3 backbone’s first 28 transformer blocks and the depth encoder’s first 21 blocks remain frozen, while the final 3 depth blocks adapt and the detection head trains from scratch.This configuration is maintained across all stages.

C Evaluation details

Evaluation combines benchmark-specific matching rules, long-tail and depth stratification, federated handling of incomplete annotations, and the composite ODS metric for zero-shot transfer.

  • Overview: The evaluation section defines the metrics and protocols used across all benchmarks.
  • Matching protocols: Omni3D uses oriented 3D bounding-box IoU matching, averages AP over 10 thresholds from 0.05 to 0.50, and reports near, medium, and far depth strata.Maximum detections per image are capped at 100.
  • Matching protocols: WildDet3D-Bench and Stereo4D use center-distance matching with thresholds proportional to object spatial extent and average AP over 11 thresholds from 0.50 to 1.00.Depth results use the same near, medium, and far splits.
  • Zero-shot metric: ODS combines AP, translation error, orientation error, and scale error as ODS = 3 ⋅AP + (1 −mATE) + (1 −mAOE) + (1 −mASE).AP contributes half of the total score through weight 3/6.
  • WildDet3D-Bench: WildDet3D-Bench reports rare, common, and frequent category AP using LVIS-style frequency splits for categories with fewer than 5, 5–20, and more than 20 images.The benchmark contains 700+ evaluation categories.
  • WildDet3D-Bench: Federated evaluation treats predictions overlapping 2D-annotated objects without valid 3D boxes as neutral rather than false positives.This addresses the benchmark’s non-exhaustive annotations.
  • Post-processing: Test-time post-processing applies per-category NMS at 2D IoU 0.6 and discards predictions with 2D objectness below 0.05.These operations occur before evaluation.

D Dataset details

WildDet3D-Data combines automated candidate generation and geometric optimization with human screening, balanced sampling, category-size filtering, and documented annotator and pipeline biases.

  • Annotation: Human annotators screen candidate 3D boxes through a 10-task qualification batch assessing rejection of bad annotations and best-candidate selection.The screening takes approximately 5 minutes and offers a $1.50 reward.
  • Candidate generation: Candidate boxes are aligned to scene depth by coarse translation search followed by local optimization of the 3D center.The process uses inclusion, tightness, and 2D projection losses while keeping dimensions and rotation fixed.
  • Annotation: The annotation interface presents candidate columns with 2D crops and orthographic point-cloud views, then records candidate quality ratings.Annotators select among good_fit, acceptable, unacceptable (3D), and unacceptable (2D).
  • Sampling: Validation and test sampling targets 50% near, 25% mid, 20% far, and 5% super-far objects, with source proportions of COCO 20%, LVIS 40%, and Objects365 40%.Categories with fewer than 3 sampled images are marked rare_category and excluded from evaluation.
  • Filtering: Small-object candidates must satisfy VLM, category, and model criteria, with an additional projected-IoU requirement for candidates scoring exactly 10.Per-category selection is capped at 1,500 annotations.
  • Filtering: GPT-4.1-mini estimates category-specific physical box-size ranges and flags flat or elongated categories to support size filtering.The model returns six axis-range fields and Boolean is_flat and is_elongated flags.
  • Annotators: The annotation pool includes 1,786 Prolific annotators, predominantly from the United States, United Kingdom, and Canada.The reported pool is gender-balanced and has a mean age of 42.1.
  • Limitations: The annotator pool’s geographic and ethnic skew may influence judgments about plausible 3D boxes for culture-specific categories or unfamiliar scenes.The pipeline’s LLM size filters and VLM heuristics may also inherit training-data biases affecting retained annotations.

E More WildDet3D-Data examples

WildDet3D-Data examples visualize annotated 3D boxes both in the input image and in reconstructed point clouds, with category labels shown alongside image overlays.

  • Qualitative examples: Additional qualitative examples are provided in Figure 12.
  • Qualitative examples: Each qualitative example overlays 3D bounding boxes and category labels on the input image, then renders the corresponding boxes in the reconstructed point cloud.

F Additional qualitative results

The paper provides additional qualitative comparisons for box-prompted and text-prompted 3D detection across representative scenes.

  • Box-prompted comparison: Figure 13 compares WildDet3D, OVMono3D, and DetAny3D on identical scenes using 2D bounding-box prompts.The blocks show prompts, ground-truth 3D boxes, model predictions, 2D overlays, and corresponding 3D boxes.
  • Text-prompted comparison: Figure 14 compares WildDet3D with 3D-MOOD on identical scenes using text categories only.Each block places WildDet3D above 3D-MOOD for visual comparison.
Loading 2604.08626v2…