Source-linked AI summary
CrowdOcc: Monocular Semantic Scene Completion for Quadruped Robots in Crowded Indoor Environments
Feiyang Chen, Jincheng Hu, Yiduo Chen, Jihao Li, Yue Liang, Bingzhao Gao, Yanjun Huang, Yuanjian Zhang
TL;DR
Monocular SSC for quadruped robots is underexplored in crowded indoor scenes, where human occlusion challenges geometry and occupancy prediction. CrowdOcc provides an RGB-D dataset and a framework combining surface-normal geometry cues with sparse human-centric interaction. It reports state-of-the-art SSC performance on scene-disjoint test scenes.
Problem
Crowded indoor quadruped SSC has limited dataset coverage, while human occlusion challenges static geometry and partial observations make human occupancy difficult to recover reliably.
Source: §I · p. 1Method
CrowdOcc pairs an RGB-D dataset with NGSGF, which adds surface-normal cues to depth-based lifting, and HCSI, which models human-centric 3D interactions.
Source: §I · pp. 1–2Results
15.80 IoU, 11.40 mIoU, and 46.23 Human IoU; the method achieves state-of-the-art SSC performance on three test scenes disjoint from training and validation.
Source: B. Quantitative Comparisons · p. 5Takeaways & Limitations
Evaluation on unseen indoor scenes demonstrates strong cross-scene generalization under the scene-disjoint protocol.
Source: §I · p. 2
Abstract
from arXiv · showhide
Monocular semantic scene completion (SSC) for quadruped robots remains underexplored in real crowded indoor environments, where human-scene occlusion disrupts static geometry and human occupancy predictions are often incomplete or spatially misplaced. We present CrowdOcc, an RGB-D dataset and monocular SSC framework for this setting. CrowdOcc contains 25.1K frames from 11 indoor scenes, with semantic occupancy annotations constructed through static dynamic decoupling. Our framework combines: (i) Normal Guided Scene Geometry Fusion (NGSGF) to complement depth-aware lifting with surface-normal cues for occlusion robust geometry; and (ii) Human-Centric Sparse Interaction (HCSI) to selectively model human-human and local human scene relations in 3D. Our method achieves state-of-the-art SSC performance on CrowdOcc's scene-disjoint test set, reaching 15.80 IoU, 11.40 mIoU, and 46.23 Human IoU, demonstrating generalization to unseen indoor scenes.
I. INTRODUCTION
Crowded indoor monocular SSC for quadruped robots is underexplored, with occlusion challenging static geometry and human occupancy. CrowdOcc pairs a dataset with geometry- and human-focused modules and reports state-of-the-art performance on unseen scenes.
I. INTRODUCTION
- Existing SSC datasets provide limited coverage of low-viewpoint observations with dense human presence and dynamic human occupancy.Most are built around static indoor scenes or autonomous-driving settings with different viewpoints, dynamics, and occupancy distributions.Source: §I · p. 1
- Human-scene occlusion weakens depth-based recovery of static structures, while partial observations and foreground-background ambiguity yield incomplete or misplaced human occupancy.These difficulties are more pronounced when test scenes differ from training scenes and scene-specific geometric priors do not transfer reliably.Source: §I · p. 1
- CrowdOcc contains 25.1K frames from 11 independent indoor scenes, with 8 occupied semantic categories and 0.08 m voxels on a 50×60×96 grid.It is an RGB-D dataset for monocular SSC from a quadruped viewpoint.Source: §I · p. 2
- NGSGF combines depth-based lifting with surface-normal cues, while HCSI uses sparse human-centric 3D interaction to refine human geometry and spatial placement.The modules target static-scene geometry under human occlusion and human occupancy, respectively.Source: §I · p. 2
- CrowdOcc achieves state-of-the-art SSC performance on test scenes excluded from training and validation, demonstrating strong cross-scene generalization.The introduction identifies the test scenes as unseen indoor environments.Source: §I · pp. 1–2
II. RELATED WORK
Prior monocular SSC methods develop image-to-voxel lifting and 3D representations, but depth remains the dominant geometric cue. CrowdOcc adds surface-normal cues to address limitations of depth under occlusion.
A. Monocular Semantic Scene Completion
- MonoScene lifts 2D features into a 3D volume for volumetric context reasoning, while VoxFormer, NDC-Scene, and ISO develop alternative lifting strategies.VoxFormer uses depth-guided sparse voxel queries, NDC-Scene uses normalized device coordinates, and ISO incorporates monocular depth for indoor SSC.Source: A. Monocular Semantic Scene Completion · p. 2
- Despite later advances in lifting and 3D representation, depth remains the dominant geometric cue in monocular SSC.Under heavy occlusion, depth alone provides limited information about local surface orientation and structural continuity.Source: A. Monocular Semantic Scene Completion · p. 2
- NGSGF complements depth-based lifting with surface-normal cues to improve static-scene geometry in crowded environments.The approach addresses the limited orientation and continuity information available from depth alone under heavy occlusion.Source: A. Monocular Semantic Scene Completion · p. 2
B. Human-Centric and Embodied Occupancy Perception
CrowdOcc focuses on human-centric 3D reasoning and constructs annotations by separating static scenes from dynamic humans. Its quadruped dataset supports cross-scene evaluation with single-image inference.
B. Human-Centric and Embodied Occupancy Perception
- CrowdOcc targets completeness and spatial placement of human occupancy through targeted human-centric 3D reasoning in crowded indoor scenes.Prior work includes embodied and legged-robot occupancy as well as human-aware spatial-relation modeling.Source: B. Human-Centric and Embodied… · p. 2
C. SSC Datasets and Occupancy Construction
- CrowdOcc reconstructs static scenes from human-free sequences and derives human occupancy from synchronized current-frame observations, avoiding temporal accumulation of non-rigid human geometry.The dataset-generation pipeline aligns human observations with the static scene and refines labels during voxelization.Source: C. SSC Datasets and Occupancy… · p. 3, Figure 2 · p. 3
III. CROWDOCC DATASET
- The dataset contains 25,140 RGB-D frames from 11 indoor scenes, with three test scenes disjoint from training and validation and 50 × 60 × 96 occupancy grids at 0.08 m resolution.LiDAR and IMU are used only for offline annotation construction; inference takes a single RGB image.Source: §III · p. 3
A. Static Semantic Scene Annotation
CrowdOcc builds static semantic annotations from human-free observations and constructs dynamic human occupancy separately from synchronized current-frame data. The branches are aligned and refined to form the final annotations.
A. Static Semantic Scene Annotation
- Human-free observations avoid introducing transient human geometry, while registered point clouds from multiple frames are aggregated into a dense static scene.Multi-frame aggregation increases surface coverage and recovers structures partially occluded or sparsely observed in individual views.Source: A. Static Semantic Scene Annotation · p. 3
- Grounded SAM 2, generates semantic masks that are projected into 3D, integrated across frames, and manually refined into consistent scene labels.This produces semantic labels on the reconstructed static scene.Source: A. Static Semantic Scene Annotation · p. 3
B. Dynamic Human Occupancy Construction
- Synchronized current-frame human masks and point clouds yield human observations that are transformed into the reconstructed static scene's coordinate system.This avoids temporal accumulation of non-rigid human geometry.Source: B. Dynamic Human Occupancy Construction · p. 4
- During voxelization, synchronized depth refines human-label placement, while uncertain or geometrically invalid voxels are excluded from supervision.Regions with insufficient 3D support are marked uncertain rather than assigned unsupported occupancy or free-space labels.Source: B. Dynamic Human Occupancy Construction · p. 4
IV. METHODOLOGY
Crowded-scene occlusion disrupts static geometry, while human occupancy may be incomplete or misplaced. CrowdOcc addresses these problems with normal-guided geometry fusion before 3D lifting and human-centric refinement after lifting.
IV. METHODOLOGY
- NGSGF complements depth-based lifting with surface-normal cues to address discontinuous static geometry under human-scene occlusion.The method is motivated by discontinuities on walls and floors, attributed to limited surface-orientation constraints in depth-based lifting.Source: §IV · p. 4
- The architecture extracts multi-scale image features and estimates depth and normals, enhances features with NGSGF, then lifts and processes them before HCSI and occupancy decoding.A frozen MoGe-2 estimates depth and normals; D-FLoSP uses the depth representation for lifting.Source: §IV · p. 4
A. Normal-Guided Scene Geometry Fusion
- A masked Normal Encoder processes the normal map and validity mask into geometry features that interact with image features through queries, keys, and values.Image features form queries, while normal geometry features form keys and values; learnable projections and local windows avoid global cross-attention costs.Source: A. Normal-Guided Scene Geometry Fusion · p. 4
- Window-wise interaction outputs are merged into a normal-guided residual and fused with the original image representation using a learnable scale.The normal-guided residual is added to the original representation through the learnable coefficient γs.Source: A. Normal-Guided Scene Geometry Fusion · p. 4
B. Human-Centric Sparse Interaction
HCSI selectively refines human-related 3D features by routing likely human voxels, modeling relations among candidates, and incorporating nearby scene context.
B. Human-Centric Sparse Interaction
- A Human-Routing module predicts voxel-wise human responses and uses local 3D NMS followed by Top-K selection to form sparse candidates.The same prediction-based routing operates during training and inference; human occupancy labels supervise routing scores only.Source: B. Human-Centric Sparse Interaction · p. 4
- Selected candidate features interact through self-attention with positional encodings of their 3D voxel locations.The selected features then cross-attend to local 3D neighborhoods sampled from the dense scene representation.Source: B. Human-Centric Sparse Interaction · p. 4
- Refined features are written back only at selected candidate locations, leaving zero residuals elsewhere.The pipeline then decodes the enhanced representation for semantic scene completion.Source: B. Human-Centric Sparse Interaction · pp. 4–5, Figure 3 · p. 5
C. Training Loss
Training combines the retained scene-completion and auxiliary-depth objectives with human-occupancy supervision for the Human-Routing module.
C. Training Loss
- Human candidates are supervised against human occupancy labels with jointly applied binary cross-entropy and Dice losses, excluding invalid voxels.These human-routing losses are added to the retained scene-completion and auxiliary-depth objectives.Source: C. Training Loss · p. 5
V. EXPERIMENTS
The experiments use a frozen monocular depth-and-normal model, train for 10 epochs, and evaluate methods under a shared CrowdOcc protocol.
V. EXPERIMENTS
- All methods follow the same CrowdOcc split, taxonomy, and evaluation protocol unless otherwise stated.The model is implemented in PyTorch with CUDA acceleration and trained on four NVIDIA GeForce RTX 4090-D GPUs.Source: §V · p. 5
- A frozen MoGe-2 ViT-L provides monocular depth and surface normals, and training runs for 10 epochs with AdamW.The total batch size is 4 and weight decay is 1 × 10−4.Source: §V · p. 5
- The main network and HCSI use a 1 × 10−4 learning rate, while DepthNet uses 5×10−6 and the normal branch follows a separate warmup and cosine decay.The normal-branch learning rate warms from 1 × 10−5 to 3 × 10−5 during the first epoch, then decays to 5×10−6; λh is 1.0.Source: §V · p. 5
A. Evaluation Metrics
Evaluation reports binary scene completion, semantic completion, and human occupancy; on three held-out scenes, CrowdOcc achieves the strongest reported overall and human-occupancy results.
A. Evaluation Metrics
- Binary scene completion reports Precision, Recall, and IoU; semantic completion reports occupied-class IoUs and their mean across eight categories.Human IoU is additionally highlighted, and invalid or ignored voxels are excluded from completion metrics.Source: A. Evaluation Metrics · p. 5
B. Quantitative Comparisons
- 21.12 floor IoU, 11.80 wall IoU, 46.23 human IoU, and 4.64 door IoU are the best class-wise results, extending gains beyond human occupancy.Wall IoU rises from ISO's 9.88 to 11.80, while floor IoU rises from NDC-Scene's 20.66 to 21.12; these gains align with the complementary geometry and human-occupancy roles of NGSGF and HCSI.Source: B. Quantitative Comparisons · pp. 5–6
- 15.80 IoU and 11.40 mIoU exceed the strongest competing results by +1.31 and +0.68, while 46.23 Human IoU exceeds NDC-Scene and ISO by +2.11 and +2.44 points.The test set contains three physical scenes disjoint from training and validation; the method also reaches 37.02 recall versus 34.00 for NDC-Scene and 31.32 for ISO, with competitive precision.Source: B. Quantitative Comparisons · p. 5, Table II · p. 6
C. Ablations
The cumulative ablation reaches the full model's strongest results; qualitative comparisons show more coherent structure and correctly located human occupancy under heavy occlusion.
C. Ablations
- 14.89/11.25 IoU/mIoU follows the MoGe-2 depth-prior change from Q0's 14.49/10.72, and adding HCSI raises Human IoU from 42.88 to 45.77 and IoU to 15.34.The cumulative sequence starts from ISO Q0, then introduces MoGe-2 Depth, HCSI, and NGSGF.Source: C. Ablations · p. 6
D. Qualitative Comparisons
- Under dense human-scene occlusion, CrowdOcc maintains coherent wall and floor geometry while preserving human occupancy at the correct spatial locations.The comparisons describe fragmented geometry for EmbodiedOcc, discontinuous surfaces for MonoScene, and background-misplaced foreground occupancy for ISO.Source: D. Qualitative Comparisons · p. 6
C. Ablations
- 46.23 Human IoU, 15.80 IoU, and 11.40 mIoU are reached by the full Q3 model, which gains +2.44, +1.31, and +0.68 on those metrics over Q0.The ablation table reports successive component additions on the CrowdOcc test set.Source: Table III · p. 6, C. Ablations · p. 6
VI. CONCLUSIONS
- CrowdOcc is an RGB-D dataset for monocular SSC on quadruped robots in real crowded indoor environments, paired with NGSGF and HCSI for geometry and human-occupancy reasoning.Experiments on scenes disjoint from training and validation establish state-of-the-art SSC performance on CrowdOcc and strong cross-scene generalization.Source: §VI · p. 6