AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks

1 Introduction

• Existing occupancy methods face difficulty balancing accuracy and latency, adapting to heterogeneous robot sensors, and generalizing across embodied tasks.Holistic occupancy represents both geometry and semantic categories, supporting embodied agents' perception of their environments.Source: §1 · pp. 1–2
• An adaptive geometry-guided dual-branch encoder handles varying RGB-view counts and unifies estimated depth, depth-camera, or LiDAR cues into calibrated point clouds.The LiDAR cloud is calibrated to the RGB camera coordinates, while depth maps are lifted into point clouds.Source: §1 · pp. 2–3
• Progressive query learning supports budget-adaptive inference by varying query counts or using intermediate decoder outputs.These controls are intended to balance accuracy and latency across computational budgets and environmental complexity.Source: §1 · pp. 2–3
• 65.29 IoU and 59.67 mIoU on Occ-ScanNet outperform the strongest prior method by 2.15 and 3.48 points, respectively.Efficiency, adaptability, and real-world deployment studies further support its practicality as a spatial-perception module.Source: §1 · pp. 2–3
• A containment loss keeps predicted points within valid occupied regions, reducing surface-floating artifacts and improving spatial consistency around boundaries.This design targets geometric fidelity in the point-based pipeline.Source: §1 · pp. 2–3
• Efficiency, adaptability, and real-world deployment studies demonstrate AdaOcc's practical applicability as a spatial-perception module for embodied tasks.The paper positions the framework for downstream embodied systems in addition to benchmark evaluation.Source: §1 · p. 3

Semantic occupancy prediction

• Occupancy research progresses from semantic scene completion to RGB-based occupancy and then toward compact representations as scenes and view counts grow.Dense voxel and view-transformation methods provide regular 3D supervision but incur high memory and computation costs.Source: §2 · p. 3
• Many established occupancy backbones tie input, view, and inference-budget assumptions to fixed evaluation protocols.This limits adaptability across protocols even as the field develops sparse, Gaussian, and query-based formulations.Source: §2 · p. 3

Occupancy for embodied perception

• Embodied occupancy work addresses deployable robot perception, but most pipelines still use fixed sensing inputs, view configurations, and prediction budgets.Adapting these pipelines across platforms and tasks may require redesign or retraining.Source: §2 · p. 3

3.1 Problem Setup and Method Overview

• AdaOcc predicts sparse occupied semantic points in a bounded agent-centric 3D region, then voxelizes them for standard occupancy evaluation.Source: §3.1 · pp. 3–4

3.2 Adaptive Geometry-Guided Dual-Branch Encoding

• RGB observations form the primary branch, while optional depth or LiDAR geometry is aligned into a shared point-set input and encoded as sparse geometric features.Source: §3.2 · p. 4

3.3 Progressive Point Query Decoding

• Geometry-seeded queries sample and fuse image and geometric features, then refine their coordinates across decoder layers.A growing query curriculum, intermediate decoder outputs, and view masking support different compute budgets and numbers of retained views.Source: §3.3 · p. 5

3.4 Containment-Guided Point Optimization

• Containment-Guided Point Optimization penalizes predictions in supervised empty space according to their distance from occupied voxel boxes.The loss encourages points inside valid occupied regions while leaving valid occupied predictions and unsupervised space unpenalized.Source: §3.4 · p. 6

3.5 Training Objective

• Each decoder layer receives semantic classification, point-set reconstruction, and containment supervision, with the containment term applied to selected late layers.Source: §3.5 · p. 6

4.1 Datasets and Evaluation Metrics

• Occ-ScanNet evaluates semantic occupancy over 11 occupied categories, while the mini split supports efficient comparisons and ablations and the full split evaluates the final model.The benchmark reports occupied IoU, mIoU across occupied classes, and per-class IoU.Source: §4.1 · p. 6, Figure 4 · p. 7

4.3 Main Results

• 59.67 mIoU and 65.29 IoU on full Occ-ScanNet beat GPOcc-VGGT’s 56.19 mIoU and 63.14 IoU by 3.48 mIoU and 2.15 IoU, respectively.The comparison uses RGB-estimated depth for AdaOcc, while GPOcc-VGGT uses the stronger VGGT geometry prior.Source: Table 1 · p. 7, §4.3 · p. 8

4.2 Experimental Setup

• Benchmark experiments use RGB-estimated depth for geometry encoding and query initialization, without ground-truth depth, point clouds, or occupancy inputs at inference.The default image encoder is C-RADIOv3-B; AdaOcc† substitutes SplatSSC’s EfficientNet encoder for a controlled comparison.Source: §4.2 · pp. 7–8

4.5 Efficiency and Adaptability

• On TartanGround, the same model reconstructs coherently from depth-map or LiDAR geometry and jointly processes varying numbers of RGB views in one forward pass.Figure 5 presents this heterogeneous-input and multi-view adaptability analysis.Source: Figure 5 · p. 8, §4.5 · p. 9

4.4 Ablation Study

• Removing either containment-guided optimization or progressive training causes clear performance drops; the EfficientNet variant remains competitive.The ablation study attributes gains to complementary spatial grounding and budget-aware training, and indicates gains are not solely due to the image backbone.Source: Table 2 · p. 8, §4.4 · p. 9
• Moderate loss-related hyperparameter changes have limited effects, whereas removing progressive training costs 3.03 IoU and 4.11 mIoU on Occ-ScanNet-mini.Changing the offset margin to zero lowers IoU and mIoU by 1.13 and 1.27 points, while varying the C.P.O. weight schedule or decoder decay has smaller reported effects.Source: Table 3 · p. 9, §4.4 · p. 9

4.5 Efficiency and Adaptability

• On Occ-ScanNet-mini, 500-query AdaOcc† reaches 57.97 mIoU and 65.16 IoU at 64.4 ms, versus SplatSSC’s 48.46 mIoU and 60.86 IoU at 85.1 ms.The same trained AdaOcc model can also produce intermediate predictions by selecting decoder depth, without retraining.Source: Table 4 · p. 9, §4.5 · p. 9

4.6 Embodied Applications

• Real-world demonstrations use semantic occupancy for target-aware navigation, object and receptacle localization in manipulation, and navigation followed by placement in mobile manipulation.These demonstrations suggest AdaOcc can serve as a practical perception module beyond offline benchmark evaluation.Source: §4.6 · p. 10

5 Conclusions

• AdaOcc combines varied-view RGB with depth- or LiDAR-derived geometry, adjusts inference through query counts or decoder outputs, and uses containment loss for spatial consistency.Experiments demonstrate state-of-the-art Occ-ScanNet performance, efficiency-adaptability trade-offs, and deployment in a real-robot navigation system; future work targets small and thin objects in cluttered scenes.Source: §5 · p. 10

A.1 Open-Vocabulary Semantic Occupancy Prediction

• Open-vocabulary occupancy represents each occupied voxel with a text-aligned feature, enabling arbitrary language queries rather than closed-set voxel labels.The variant uses a CLIP-aligned feature branch, with queries producing occupancy scores and semantic features mapped into the CLIP text embedding space.Source: §A.1 · p. 15
• Semantically related queries, including “bed” and “mattress,” activate consistent object regions in the reconstructed scenes.Figure 7 visualizes top-activated occupied points in red and remaining points in gray.Source: §A.1 · p. 15, Figure 7 · p. 16

A.2 Flexible Multi-View and Geometric Input

• Training randomly samples one to four camera views and pairs selected RGB views with either depth-derived points or LiDAR points; the occupancy decoder and prediction heads remain unchanged.The same model can be evaluated with one to four views without retraining.Source: §A.2 · pp. 16–17
• With ground-truth depth, IoU on the fixed four-view union rises from 17.24 for one view to 67.55 for four views, while within-view-region quality changes only slightly.The model is trained once with randomly sampled view subsets and evaluated without retraining on 1,379 test samples from 12 scenes.Source: Table 6 · p. 17, §A.2 · p. 17

• AdaOcc improves RayIoU by 3.8 over other SOTA methods on Occ3D-nuScenes, and reaches 50.9 RayIoU with LiDAR and surrounding RGB inputs.Source: §A.3 · p. 17

A.4 Hyperparameter Sensitivity

• A positive C.P.O. margin trades recall (87.88 → 83.41) for precision (70.50 → 75.32); removing progressive training costs 3.03 IoU and 4.11 mIoU.The default m = 0.04 gives the best IoU, mIoU, and F1; tested loss-weight and decay-factor variations have only mild effects.Source: §A.4 · p. 17, Table 8 · p. 18, Table 9 · p. 19

A.5 Depth Estimator Sensitivity

• Replacing DAv2 with MoGe-2 reduces IoU and mIoU by 3.06 and 2.59 points, while ground-truth depth improves them by a further 4.11 and 4.32 points.The authors locate remaining headroom in external depth estimation, with errors around reflective or low-texture surfaces, thin structures, and occluded boundaries.Source: §A.5 · p. 18, Table 10 · p. 19

B.1 Object Navigation with Go2

• For Go2 navigation, AdaOcc's semantic PLY map is converted to a conservative 2.5D costmap, then A* plans clearance-aware paths and robot-friendly waypoints.Floor points become free space, collision-height non-floor points become inflated obstacles, and unknown cells are non-traversable; open-semantic activation supplies the target.Source: §B.1 · p. 19

B.2 Closed-Loop Navigation on FloorPlan-R2R

• The FloorPlan-R2R agent acts from egocentric RGB-D under language instructions, while the shared downstream backend makes occupancy representation the compared component.FP-Nav additionally uses privileged floorplan alignment, whereas AdaOcc's pipeline estimates pose and scale online; occupancy serves grounding, target projection, and collision-free planning.Source: §B.2 · pp. 19–20, Table 11 · p. 20
• AdaOcc reduces final goal distance by 4.02 m and improves OSR, SR, and SPL by 23.86, 14.73, and 1.04 points on FloorPlan-R2R.OSR measures reaching the success region, SR additionally requires a correct final stop, and SPL weights SR by path efficiency.Source: §B.2 · p. 20

B.3 Pick and Place with ARX X5

• On ARX X5, AdaOcc predicts point sets for two bottles and a basket, whose geometry supports executable bottle grasps and basket placement.GraspNet generates grasp poses from each bottle point cloud, and the basket point-cloud center supplies the placement target.Source: §B.3 · p. 20

B.4 Mobile Manipulation with ARX X5 on Go2

• The mobile manipulation demonstration connects long-range bottle localization and Go2 navigation with local bottle grasping and basket placement in the real world.After navigation, AdaOcc predicts local bottle and basket point sets for grasp generation and placement targeting.Source: §B.4 · p. 21

C Standard Deviation of Experiments

• Across five seeds, mIoU has a 0.20 standard deviation and IoU a 0.10 standard deviation; the observed ranges are 58.26–58.74 mIoU and 65.24–65.49 IoU.Source: C Standard Deviation of Experiments · pp. 21–22, Table 13 · p. 22

D.1 Implementation Details

• Training runs for 200 epochs, increasing the active query budget from 100 to 500 across five stages; evaluation uses the full 500-query budget.The configuration also combines the final and auxiliary decoder-layer losses, applying containment loss on layers d4 and d5 with a weight ramped from 0.1 to 0.3 between epochs 20 and 60.Source: §D.1 · pp. 22, 24
• RGB-D pseudo-points are transformed from camera to LiDAR and occupancy frames, with image projections adjusted for resize and crop transforms.The Occ-ScanNet configuration uses the front camera, resizes/crops the 1296 × 968 image to 960 × 720, and retains only current-frame points in the reported single-frame setup.Source: §D.1 · pp. 22–23
• Raw Occ-ScanNet supervision uses a 60 × 60 × 36 grid at 0.08 m, converts occupied labels to point targets, and excludes unknown voxels.The raw tensor uses [y, x, z] order, while target points use Cartesian order; the voxel origin denotes the center of voxel (0, 0, 0).Source: §D.1 · p. 23
• Training matches predicted queries bidirectionally to occupied voxel centers, using density-aware Chamfer-style point loss and focal-loss semantic supervision.Containment loss penalizes predictions in known empty regions and pulls them toward nearby occupied voxel boxes.Source: §D.1 · p. 24
• Predictions are mapped into the raw grid, filtered and aggregated by voxel, then evaluated on the 11 occupied classes using IoU and mIoU.Evaluation uses the official 60 × 60 × 36 grid at 0.08 m, excludes unknown labels, and ignores points outside the grid; points below 0.4 confidence or more than 1.0 m from their query center are discarded.Source: §D.1 · pp. 23–24

D.2 Experimental Settings on Different Datasets

• Experimental settings across datasets vary in RGB and LiDAR inputs, open-vocabulary support, depth sources, and available view counts.The appendix summarizes those setting combinations in Table 15.Source: §D.2 · p. 25, Table 15 · p. 25

D.3 Compute Resources

• The Occ-ScanNet-mini RADIO run used eight H20 GPUs for about 64 GPU-hours of training and approximately 11.0 GB per GPU in its final 500-query stage.Validation latency was approximately 0.34 seconds per sample per GPU, including data loading and metric computation.Source: §D.3 · p. 25, Table 16 · p. 25

E Limitations

• Small and thin structures may be underrepresented, especially when they occupy few voxels or are weakly observed; RGB-based depth estimation adds latency and has identified failure cases.The evaluation is limited to indoor benchmarks in single-frame or direct-aggregation settings, leaving broader indoor evaluation and explicit temporal memory open.Source: E Limitations · pp. 25–26

F Broader Impacts

• The authors identify potential benefits for indoor navigation, assistive robotics, and other embodied tasks, including access to 3D perception on compute-limited platforms.These are stated as potential impacts of more accurate spatial representations under varying sensing and computational conditions.Source: F Broader Impacts · p. 26
• Incorrect occupancy or semantic predictions may affect downstream planning, while collecting or storing raw indoor sensor data may raise privacy concerns.The paper says AdaOcc does not address these deployment-level issues and recommends task-specific safety checks, privacy-preserving handling, and validation under target conditions.Source: F Broader Impacts · p. 26

G Assets and Licenses

• The work uses existing datasets, pretrained models, and baseline implementations under their original licenses and terms, and does not redistribute them.The named assets include Occ-ScanNet, Occ-ScanNet-mini, TartanGround, Depth Anything v2, EfficientNet, RADIO/C-RADIO, OPUS, and SplatSSC.Source: G Assets and Licenses · p. 26