PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniAct3D: Leveraging Foundation Geometry and Evidence-Grounded Reasoning for Panoramic 3D Detection

Runtong Wu, Fei Teng, Di Wen, Guoqiang Zhao, Kunyu Peng, Kailun Yang

arXiv:2610.03015v1cs.CVcs.AIcs.RO

TL;DR

Perspective-trained VFM detectors are challenging to transfer to ERP, whose geometry and visual cues are organized differently. OmniAct3D adapts them with geometry alignment, evidence-grounded reasoning, and appearance-guided heading estimation; it reports gains on two ERP benchmarks and strong cross-configuration VARC performance.

  • Problem

    Perspective-trained VFM detectors are difficult to transfer to ERP because its geometry and visual information differ, making object-relevant cues difficult to model, localize, and preserve.

    Source: §I · p. 1

  • Method

    OmniAct3D adapts perspective-trained VFM detectors to ERP while preserving their priors, using ERGA-RAY, VARC, and AGHE for geometry alignment, evidence-grounded reasoning, and heading estimation.

    Source: §I · p. 2

  • Results

    OmniAct3D improves the previous best detector's NDS by 2.96 points on Spheriverse and the unadapted VFM baseline's mAP by 24.87 points on PanoMMOcc.

    Source: §I · p. 2

  • Takeaways & Limitations

    With target-specific geometry adaptation, VARC retains 95% and 98% of same-configuration mAP on Spheriverse and PanoMMOcc, respectively, indicating reusable object-level 3D reasoning across sensing configurations.

    Source: §I · p. 2

Abstract

from arXiv · show

Accurate 3D detection is essential for mobile embodied agents, while Vision Foundation Models (VFMs) offer transferable visual and geometric priors. Yet existing VFM-based 3D detectors rely on narrow-view monocular images or discrete perspective views, limiting coherent surround perception; equirectangular projection (ERP) instead encodes a continuous 360 scene in a single image. Direct transfer remains difficult because ERP organizes geometry and visual information differently, making object-relevant cues hard to model, localize, and preserve. We propose OmniAct3D, a framework that adapts perspective-trained VFM detectors to ERP while preserving transferable VFM priors. To resolve geometric mismatch, the ERP-Ray Geometry Adapter (ERGA-Ray) models spherical viewing rays and periodic spatial structure. To localize evidence in scene-wide context, the Visual-Action Reasoning Chain (VARC) grounds each hypothesis in relevant panoramic evidence and converts it into a structured geometric action. To recover local cues lost under fixed token budgets, the Appearance-Guided Heading Expert (AGHE) re-encodes object regions at higher resolution for heading estimation. Experiments show that OmniAct3D improves over the previous best 3D detector by 2.96 NDS points on Spheriverse and over the unadapted VFM baseline by 24.87 mAP points on PanoMMOcc. With target-specific geometry adaptation, VARC retains 95--98% of the same-configuration mAP, indicating reusable object-level 3D reasoning across sensing configurations. The source code will be made publicly available at https://github.com/FeiT-FeiTeng/OmniAct3D.

I. INTRODUCTION

OmniAct3D adapts perspective-trained VFM detectors to ERP while preserving transferable priors, addressing challenges in geometry alignment, evidence localization, and local appearance. It reports improved benchmark results and strong cross-configuration VARC performance.

  • Perspective-trained VFM detectors are difficult to transfer to ERP because its geometry and visual information are organized differently, making object cues hard to model, localize, and preserve.The challenges include spherical rays, latitude-dependent distortion and horizontal periodicity, scene-wide evidence localization, and loss of local appearance cues under fixed token budgets.Source: §I · pp. 1–2
  • ERGA-RAY models spherical viewing rays and ERP-aware spatial interactions to align pretrained geometry with ERP's spherical and periodic structure.Source: §I · p. 2
  • VARC grounds each 3D hypothesis in relevant panoramic evidence and converts it into a structured geometric action, while AGHE re-encodes object regions at higher resolution for heading estimation.Source: §I · p. 2
  • 2.96 NDS points over the previous best detector on Spheriverse and 24.87 mAP points over the unadapted VFM baseline on PanoMMOcc; VARC retains 95–98% of same-configuration mAP with target-specific geometry adaptation.The reported retention spans Spheriverse and PanoMMOcc.Source: §I · p. 2
  • OMNIACT3D adapts perspective-trained VFM detectors to ERP while preserving transferable VFM priors.Source: §I · p. 2

II. RELATED WORK

Prior work covers perspective-view 3D detection, ERP perception, and transferable foundation-model priors. This paper studies panoramic object-level detection and defines its target as 3D states inferred directly from an ERP image.

A. Vision-Based 3D Object Detection

  • Perspective-view 3D detection includes monocular methods, multi-view aggregation, and temporal approaches, while this work studies object-level detection from a panoramic representation.Monocular methods use geometric constraints, depth estimation, or object-centric representations,,; multi-view approaches use projection, query-based aggregation, or BEV representations,,, and temporal methods exploit cross-frame information.Source: A. Vision-Based 3D Object Detection · p. 2

B. Omnidirectional Vision and ERP Perception

  • ERP provides continuous horizontal 360° coverage in one image, but latitude-dependent distortion, horizontal periodicity, and spherical rays distinguish its geometry from perspective imaging.Prior ERP work addresses these properties through spherical convolutions and distortion sampling,, projection fusion, and panoramic attention.Source: B. Omnidirectional Vision and ERP… · p. 2

C. Foundation Models for Panoramic 3D Perception

  • Vision and geometry foundation models provide transferable semantic, depth, and 3D-structure priors, but adapting VFM-based detection from perspective imagery to ERP remains challenging.The paper uses “VFM-based 3D detector” for a detector built on pretrained models,,, [37].Source: C. Foundation Models for Panoramic 3D… · p. 2

III. METHODOLOGY

  • The method directly infers each object's 3D center, dimensions, orientation, category, and detection confidence from a full ERP image.Source: §III · p. 2

A. Overview of OmniAct3D

OmniAct3D keeps its VFM-based detector frozen while adapting geometry for ERP and refining its box predictions through evidence-grounded reasoning and appearance-guided heading estimation.

A. Overview of OmniAct3D

  • The detector extracts visual features and pretrained geometry memory, adapts the memory for ERP, generates box hypotheses, retrieves evidence for a geometric action, then refines heading.The VFM detector remains frozen; ERGA-RAY, VARC, and AGHE are trainable components.Source: A. Overview of OmniAct3D · pp. 2–3, Figure 2 · p. 3

B. ERP-Ray Geometry Adapter

  • ERGA-RAY conditions pretrained geometry on ERP spherical viewing rays and angular cues, then applies spatial mixing with circular horizontal and non-periodic vertical interactions.The latitude-dependent distortion cue includes cos ϕ, and ray-conditioned tokens modulate a low-dimensional projection of foundation tokens.Source: B. ERP-Ray Geometry Adapter · pp. 3–4
  • ERGA-RAY adds a gated residual to pretrained geometry memory, preserving transferable foundation priors while introducing geometry for panoramic 3D detection.The residual is predicted from the stacked ray-conditioned tokens.Source: B. ERP-Ray Geometry Adapter · p. 4

C. Visual-Action Reasoning Chain

VARC grounds each 3D box hypothesis in object-relevant panoramic evidence, uses that evidence to predict a structured geometric action, and converts the updated state back into a 3D box.

Hypothesis-grounded panoramic evidence

  • VARC samples hypothesis-grounded panoramic evidence from 13 object-centric support points: the box center, eight corners, and four side-face centers.Query-conditioned offsets adapt the support template, and a projection descriptor captures angular extent, seam crossing, and visible support-point fraction.Source: C. Visual-Action Reasoning Chain · p. 4

Evidence-driven geometric action

  • The reasoner combines evidence with object query, spherical state, quality cues, and projection descriptors to predict a gated action over range, position, dimensions, and yaw.The horizontal-range action uses a discrete direction and continuous magnitude; longitude, latitude, dimensions, and observation-relative yaw use bounded actions.Source: C. Visual-Action Reasoning Chain · p. 4

Hypothesis representation

  • WrapClamp handles periodic longitude and observation-relative yaw, clamps latitude to the calibrated vertical field of view, and preserves log-parameterized range and dimensions when forming the resulting box.The inverse parameterization yields the updated 3D box B′.Source: C. Visual-Action Reasoning Chain · p. 4

D. Appearance-Guided Heading Expert

AGHE restores local appearance detail for directed heading estimation by re-encoding projected object regions at higher resolution rather than rerunning the full detector.

D. Appearance-Guided Heading Expert

  • AGHE re-encodes projected object regions at higher object-centric resolution to estimate directed heading from fine-grained local appearance cues.It crops each projected box region with contextual margins and uses the pretrained DINOv2 encoder.Source: D. Appearance-Guided Heading Expert · p. 4
  • The heading expert combines local appearance with category and orientation-free geometry, then predicts periodic observation-relative heading and converts it to camera-frame direction.The geometric encoding includes log width, length, and height.Source: D. Appearance-Guided Heading Expert · p. 5
  • AGHE updates only box orientation while preserving center, dimensions, category, and detection score, without rerunning SAM3, the depth decoder, or detection head.Crops from each ERP image are batched for encoding.Source: D. Appearance-Guided Heading Expert · pp. 4–5

E. Training Objectives

The training objectives supervise ERGA-RAY’s geometry adaptation, VARC’s geometric actions, and AGHE’s observation-relative heading prediction.

E. Training Objectives

  • ERGA-RAY’s multi-target objective regresses seam-aware image-center residuals, log-range, and log-dimensions while preserving orientation, pretrained regression, and confidence predictions and constraining adaptation residuals.The objective also includes VARC supervision.Source: E. Training Objectives · p. 5
  • VARC’s objective supervises the resulting spherical box state, identity actions for accurate hypotheses, action-gate activation, and radial direction, ranking, and magnitude.Source: E. Training Objectives · p. 5
  • AGHE is supervised on the sine-cosine encoding of periodic observation-relative heading with a cosine loss.Source: E. Training Objectives · p. 5

IV. EXPERIMENTS

The experiments evaluate panoramic 3D detection on two benchmarks, analyze three components, and test VARC transfer across sensing configurations.

IV. EXPERIMENTS

  • Experiments assess OMNIACT3D on two panoramic 3D detection benchmarks, analyze ERGA-RAY, VARC, and AGHE, and test VARC transfer across sensing configurations.Source: §IV · p. 5

A. Experimental Setup

The setup constructs ERP detection benchmarks from two occupancy datasets, compares diverse detector families under a common protocol, and specifies training and evaluation conditions.

A. Experimental Setup

  • Spheriverse and PanoMMOcc provide LiDAR, 3D boxes, and calibration; the boxes are transformed to panoramic-camera coordinates to form ERP detection benchmarks.Source: A. Experimental Setup · p. 5
  • Evaluation reports mAP, NDS, mATE, mASE, and mAOE; because attribute and velocity annotations are unavailable, NDS uses three true-positive metrics.Source: A. Experimental Setup · p. 5
  • Comparisons span BEV, query-based, configuration-aware, monocular, and VFM detectors; all ablations use ERP, and methods receive minimal dataset-specific adaptations unless noted.WildDet3D uses square preprocessing, while 3D-MOOD and DetAny3D decompose panoramas into perspective views.Source: A. Experimental Setup · p. 5
  • Figure 3 presents qualitative comparisons on Spheriverse and PanoMMOcc, with columns for ground truth, OmniAct3D without VARC, full OmniAct3D, and representative baselines.Source: Figure 3 · p. 6

B. Main Results

OmniAct3D outperforms prior detectors on both panoramic benchmarks; the analysis also examines component effects and reasoning reuse.

B. Main Results

  • 1.98 mAP and 2.96 NDS points on Spheriverse, and 1.39 mAP and 2.19 NDS points on PanoMMOcc, are OmniAct3D’s gains over the previous best.It also achieves the lowest mASE and mAOE on Spheriverse.Source: B. Main Results · p. 6

V. ANALYSIS AND ABLATION STUDIES

  • The component ablation examines ERGA-RAY, VARC, and AGHE, while the broader analysis includes ERP geometry adaptation and reasoning reuse.Source: §V · p. 6, Table II · p. 6

A. Component Ablation Analysis

The ablations distinguish VARC’s contribution to object localization from AGHE’s contribution to heading estimation. ERP geometry adaptation and evidence-grounded reasoning are also evaluated across methods, backbones, and difficult targets.

A. Component Ablation Analysis

  • Adding VARC to ERGA-RAY raises mAP by 6.96 points on Spheriverse and 15.44 points on PanoMMOcc; AGHE improves heading estimation without changing mAP.The component ablation attributes localization gains mainly to VARC and heading gains to AGHE.Source: A. Component Ablation Analysis · p. 6

B. ERP Geometry Adaptation

  • ERGA-RAY improves mAP/NDS over the strongest alternatives by 21.5%/9.1% on Spheriverse and 16.7%/11.1% on PanoMMOcc, and also improves DAP’s mAP on both datasets.With DAP, the reported mAP changes are 0.0683 to 0.0986 on Spheriverse and 0.1854 to 0.2097 on PanoMMOcc.Source: B. ERP Geometry Adaptation · p. 6, Table III · p. 7

C. Hypothesis-Evidence-Action Reasoning

D. Fine-Grained Heading Recovery

AGHE improves heading estimation without changing object localization. VARC retains most same-configuration mAP after transfer between the two datasets using target-specific ERGA-RAY; the conclusion identifies temporal panoramic observations and more complex embodied perception as future directions.

D. Fine-Grained Heading Recovery

  • AGHE leaves mAP unchanged while reducing mAOE from 1.5585 to 0.8455 on Spheriverse and from 2.0781 to 0.7902 on PanoMMOcc.The resulting NDS is 0.1825 on Spheriverse and 0.4032 on PanoMMOcc.Source: D. Fine-Grained Heading Recovery · p. 7

E. Cross-Configuration Reasoning Transfer

  • Without updating VARC parameters, transfer with target-specific ERGA-RAY retains 95% of same-configuration mAP on Spheriverse and 98% on PanoMMOcc.The reported values are 0.1800 vs. 0.1887 on Spheriverse and 0.5803 vs. 0.5897 on PanoMMOcc.Source: E. Cross-Configuration Reasoning… · p. 8, Table VII · p. 8

VI. CONCLUSION

  • OmniAct3D achieves the best mAP and NDS on two ERP benchmarks; future work will extend evidence-grounded reasoning to temporal panoramic observations and more complex embodied perception tasks.Source: §VI · p. 8
Loading 2610.03015v1…