PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Semantic Scene Completion: a Survey

Luis Roldao, Raoul de Charette, Anne Verroust-Blondet

arXiv:2103.07466v3cs.CV

TL;DR

SSC addresses the challenge of inferring complete scene geometry and semantics from sparse, partial observations, where missing regions and imperfect ground truth leave substantial ambiguity. This survey critically analyzes SSC methods and datasets, including their design choices and benchmark performance. It reports that the best methods remain below 30% mIoU on the challenging SemanticKITTI benchmark.

  • Problem

    SSC must infer geometry and semantics from sparse, partial observations, with ambiguous missing regions and weak supervision from incomplete ground truth.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Method

    The paper presents a comprehensive critical review of SSC methods and datasets, analyzing approaches, design choices, and performance.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    The best methods still perform little below 30% mIoU on the challenging SemanticKITTI benchmark, while the literature shows no consensus on input encoding or architectures.

    Source: §4 · p. 6

  • Takeaways & Limitations

    The survey highlights missing research areas and aims to help readers navigate the emerging SSC field.

    Source: §1 · p. 2

    Point-based SSC remains underexplored: SPC-Net is the only method described as solely point-based, and generative use faces fixed-size output and point-generation constraints.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Semantic Scene Completion (SSC) aims to jointly estimate the complete geometry and semantics of a scene, assuming partial sparse input. In the last years following the multiplication of large-scale 3D datasets, SSC has gained significant momentum in the research community because it holds unresolved challenges. Specifically, SSC lies in the ambiguous completion of large unobserved areas and the weak supervision signal of the ground truth. This led to a substantially increasing number of papers on the matter. This survey aims to identify, compare and analyze the techniques providing a critical analysis of the SSC literature on both methods and datasets. Throughout the paper, we provide an in-depth analysis of the existing works covering all choices made by the authors while highlighting the remaining avenues of research. SSC performance of the SoA on the most popular datasets is also evaluated and analyzed.

1 Introduction

Semantic Scene Completion jointly infers scene geometry and semantics from partial observations, which are sparse, noisy, and incomplete. This survey critically reviews SSC methods and datasets to help readers navigate the field and identify research gaps.

1 Introduction

  • Sparse sensing, limited field of view, and measurement noise leave large scene regions unobserved, making computer-based 3D understanding difficult.The task matters for applications such as robotic navigation and augmented reality, where geometry and semantics support interaction with the real world.Source: §1 · p. 1
  • SSC jointly infers complete scene geometry and semantics from partial observations, extending scene completion beyond geometry alone.Unlike object-level completion or multi-view scene reconstruction, it relies on learned priors to resolve ambiguities across the scene.Source: §1 · pp. 1–2
  • The survey offers a comprehensive critical review of SSC methods and datasets, highlighting missing areas for future research.It also covers input encodings, architectures, design choices, training strategies, performance, and discussion.Source: §1 · p. 2

2 Problem statement

SSC maps an incomplete 3D scene representation to a dense, semantically labeled estimate of the real scene, with learned priors addressing ambiguity in missing regions. Related work spans geometry completion, semantic segmentation, and instance-level completion, but SSC targets the whole scene.

2 Problem statement

  • SSC is a function that maps an incomplete 3D scene representation to a dense semantic scene that approximates the real scene.Because large regions may be missing through sparse sensing or occlusion, the task is most often learned from sparse-input and dense-scene pairs.Source: §2 · p. 2
  • RGB-D or stereo cameras provide dense visible surfaces, whereas LiDAR is sparser and leaves a larger unknown volume.For cameras, completion is limited to occluded regions; LiDAR point returns cover only an infinitesimal portion of space, with density decreasing farther away.Source: §2 · p. 2

2.1 Related tasks

  • Completion progressed from interpolation and energy minimization to model-based inference for occluded parts of objects and entire scenes.Symmetry and prior 3D models are among the approaches described.Source: §2.1 · p. 2
  • Semantic segmentation can operate on predefined points, while semantic instance completion targets detected objects and cannot fill scene background such as walls or floors.Instance methods commonly use CAD models or a completion head; Song et al. first jointly addressed semantic segmentation and scene completion.Source: §2.1 · p. 3

3 Datasets and representations

The survey describes SSC-ready datasets, their construction and annotation challenges, and the trade-offs among common 3D scene representations. Dataset coverage is predominantly indoor and stationary, while representation choice affects method design, memory, and computation.

3.1 Datasets

  • 14 datasets meet the survey’s SSC-ready criteria, but only half have been used for SSC; indoor stationary datasets predominate over outdoor ones.The four most popular datasets are highlighted in the survey.Source: §3.1 · p. 3
  • Synthetic datasets offer virtually free labels, while real datasets require costly recording, annotation, and processing.Indoor datasets commonly use RGB-D or stereo sensors; outdoor datasets often use LiDAR and must account for dynamic agents.Source: §3.1 · p. 3
  • Real-world ground truth is an imperfect approximation, affected by occlusion, distance-related sensor sparsity, motion traces, and labeling errors.Outdoor scenes are aggregated and labeled from sparse sequential data, while indoor RGB-D ground truth commonly uses SfM or visual SLAM; semantics are highly imbalanced in common datasets.Source: §3.1 · p. 4, Figure 4 · p. 4, Figure 3 · p. 4
  • NYUv2 has 1,449 RGB-D and annotated-volume pairs; NYUCAD uses rendered depth maps for simplified pairs at the cost of geometric detail.SemanticKITTI is the most popular large-scale outdoor dataset and the only one described here as providing real-world sparse/dense semantic point-cloud pairs.Source: §3.1 · pp. 4–5

3.2 Scene representations

  • Voxel grids provide adjacent-cell neighborhoods suited to 3D CNNs but incur memory and computation costs by representing occupied and free space.Point clouds are memory efficient but omit free space and geometrical structure; implicit surfaces encode signed distance fields, while meshes are explicit polygon surfaces.Source: §3.2 · pp. 5–6, Figure 5 · p. 5

4 Semantic Scene Completion

SSC methods jointly infer geometry and semantics for unseen regions, but the survey finds wide methodological variety and little consensus. It reviews methods and datasets while identifying remaining research avenues.

4 Semantic Scene Completion

  • Best methods still perform little below 30% mIoU on SemanticKITTI, leaving a significant margin for improvement.The survey identifies SemanticKITTI as its most challenging benchmark.Source: §4 · p. 6

4.1 Input Encoding

  • f-TSDF moves strong gradients toward the surface, but its benefit over TSDF is disputed and all TSDF variants except p-TSDF hinder real-time use.The survey reports experiments in which f-TSDF may slightly harm performance or bring negligible improvement.Source: §4.1 · p. 7
  • Common inputs include depth and range images, occupancy grids, TSDFs, and point clouds, with 2D encodings processed cheaply by 2D CNNs.Some works also combine depth or range with RGB or other modalities for semantic cues.Source: §4.1 · pp. 6–8

4.2 Architecture choices

  • Architectures span volume, view-volume, point-based, and hybrid networks; point-based SSC remains rare, while hybrid designs combine different modalities or representations.Volume networks commonly use 3D CNNs, while view-volume approaches combine 2D and 3D processing.Source: §4.2 · pp. 8–9

4.3.1 Contextual awareness

  • Multi-scale context is gathered through skip connections, dilated convolutions, ASPP, and cascaded aggregation; dilation suits dense networks after input-manifold dilation.The survey also describes global pooling and Primal-Dual multi-scale aggregation.Source: §4.3.1 · pp. 9–10

4.3.2 Position awareness

  • Position-aware approaches emphasize geometrically informative boundaries through weighted loss, contour supervision, or edge and sketch cues.The 3D sketch embedding is reported to be resolution-insensitive, benefiting partial low-resolution observations.Source: §4.3.2 · p. 10

4.3.3 Fusion strategies

  • Fusion combines RGB and geometry or differently encoded geometric inputs; the survey groups strategies as early input, middle-feature, or late/output fusion.Bird's-eye-view fusion is also used for outdoor SSC.Source: §4.3.3 · p. 10

F DNN SSC

The survey compares multimodal fusion strategies, lightweight designs, and refinement methods for SSC. It finds middle fusion to be the best general practice, while late fusion helps in some specific settings.

F DNN SSC

  • Early fusion concatenates inputs, whereas middle fusion learns modality-specific features and combines them at one or multiple stages.The survey reports that multi-stage fusion improves information flow between modalities; selective schemes include RefineNet and Gated Recurrent Fusion.Source: F DNN SSC · p. 11
  • Middle fusion appears to be the best general SSC practice; selective schemes yield at least a 20% performance boost over simple fusion, while late fusion helps in specific settings.S3CNet's late-fusion ablation reports an overall 20% boost, evaluated only on outdoor scenes.Source: F DNN SSC · p. 11

4.3.4 Lightweight designs

  • Sparse tensors and compact representations address voxel-grid costs, while custom and decomposed convolutions exploit spatial redundancy to reduce computation or parameters.Sparse convolutions can be less suitable for completion because dense decoding is still needed; Kd-Tree and Octree implementations have limited uptake in SSC.Source: §4.3.4 · pp. 11–12

4.3.5 Refinement

  • Refinement is uncommon in SSC, but reported approaches include CRF output refinement, spatial propagation, and fusion of predictions from different viewpoints.VD-CRF achieves little over 4% gain; the survey describes refinement as an encouraging topic to explore.Source: §4.3.5 · p. 12

4.4 Training

The survey reviews SSC loss functions and training strategies, including their supervision signals, trade-offs, and reported results. It covers geometric, semantic, and consistency losses alongside end-to-end, staged, and adversarial training.

4.4.1 Losses

  • SSC losses fall into geometric, semantic, and consistency categories, while sparse ground truth restricts geometric and semantic losses to known locations.Class balancing is often used because SSC is highly imbalanced.Source: §4.4.1 · pp. 12–13
  • Cross-Entropy is the preferred semantic loss; using C + 1 classes includes free space and enforces reconstruction, unlike C-class semantics alone.The C-class formulation therefore requires an additional geometric loss.Source: §4.4.1 · p. 13
  • 1–3% performance gains are reported for PALNet’s Position Aware loss, which weights cross-entropy using local semantic entropy.Its Local Geometric Anisotropy term scales with nearby semantic entropy and lowers in locally smooth semantic regions.Source: §4.4.1 · p. 13
  • Consistency losses provide self-supervised semantic signals available throughout the scene, though support-point strategies may suit repetitive outdoor patterns better than cluttered indoor scenes.The cited approaches use Kullback-Leibler or Jensen-Shannon divergence for completion or spatial semantic consistency.Source: §4.4.1 · p. 13

4.4.2 Training strategies

  • End-to-end single-scale training predominates, while alternatives supervise multiple resolutions, use auxiliary semantics, or train coarse-to-fine stages.Coarse-to-fine training feeds earlier-stage predictions into later stages; progressive refinement and adversarial learning are other strategies.Source: §4.4.2 · pp. 13–14
  • 10%-15% boosts on several datasets are reported for adversarial training by and.Discriminators add supervision by distinguishing ground-truth from predicted scenes; also discriminates latent depth or semantic features.Source: §4.4.2 · p. 14

4.5 Evaluation

The survey compares SSC metrics, benchmark performance, and network efficiency, finding that real-dataset results remain limited and vary by metric and evaluation setting.

4.5.1 Metrics

  • mIoU measures semantic-class IoU without free space, while geometric IoU, precision, and recall evaluate binary free/occupied scenes; real indoor and outdoor protocols differ.For real indoor data, geometry is evaluated on occluded input regions, whereas real outdoor metrics commonly cover the entire known space; indoor/outdoor comparisons are further complicated by resolution differences.Source: §4.5.1 · pp. 14–15, §4.5.2 · p. 15

4.5.2 Performance

  • Without pretraining, best mIoU is 41.1% on NYUv2 and 29.5% on SemanticKITTI, illustrating the challenge of real-dataset completion.Table 3 reports the popular-dataset comparison; indoor results commonly use 60 × 36 × 60 grids for historical reasons.Source: Table 3 · p. 16
  • S3CNet leads SemanticKITTI mIoU by +5.7% over other methods, while SISNet leads NYUv2 at 52.4% mIoU and 78.2% IoU; metric rankings differ.S3CNet's SemanticKITTI performance combines 2D BEV and 3D f-TSDF features; the survey reports a 10−35% synthetic-to-real indoor mIoU gap.Source: §4.5.2 · pp. 15–16

4.5.3 Network efficiency

  • Network parameters and FLOPs vary by 1:144 and 1:1260, respectively; Chen et al. and LMSCNet are the lightest indoor and outdoor networks.Lower operation counts may contribute to faster inference, though not necessarily; sparse convolutions are used to reduce memory overhead.Source: §4.5.3 · p. 18, Table 8 · p. 18

5 Discussion

The discussion highlights persistent real-world challenges and reviews practices and data avenues that may support further SSC research.

Best practices for SSC

  • Contextual feature aggregation, geometric cues, auxiliary tasks, and position-aware losses are among the practices the survey identifies as beneficial for SSC.The discussion also recommends combining sparse with dense convolutions and evaluating on both indoor and outdoor datasets.Source: §5 · p. 18

Supervision bias

  • Free and occupied space have a 9:1 imbalance in NYUv2 and SemanticKITTI; random undersampling to 2:1 reportedly improves completion by +4% IoU.Semantic class imbalance also persists, especially outdoors, where underrepresented classes such as pedestrians and motorcycles remain difficult to predict.Source: §5 · p. 18

Object motion

  • Rigid registration of sequential frames creates temporal tubes around moving objects; methods accounting for only a few future scans yield marginal completion improvement.The survey notes that methods tend to predict vehicles as stationary on SemanticKITTI, where parked vehicles are abundant.Source: §5 · p. 19

Datasets extendable for SSC

  • Existing 3D semantic datasets can be extended for SSC, often with little processing, while some—such as nuScenes—require more complex processing.The survey also suggests autonomous-driving simulators such as CARLA and SYNTHIA for synthetic dataset generation.Source: §5 · p. 21, Table 9 · p. 21

6 Conclusion

The paper concludes with a comprehensive survey and critical analysis of contemporary 3D Semantic Scene Completion methods.

6 Conclusion

  • The survey reviews major SSC approaches and design choices and compares their performance on popular datasets.The authors intend the survey to provide insights and help readers navigate the field.Source: §6 · p. 21

Related papers

Loading 2103.07466v3…