PaperlayerView exact revision on arXiv

Source-linked AI summary

SSC-Priors: Exploring Semantic and Visibility Priors to Boost Lidar Semantic Scene Completion

Tetiana Martyniuk, Jonathan Seele, Alexandre Boulch, Gilles Puy, Renaud Marlet, Raoul de Charette

arXiv:2609.17413v1cs.CV

TL;DR

Lidar SSC must recover geometry and semantics from sparse single scans, while prior methods often bind auxiliary cues to complex architectures. This paper injects semantic pseudo-labels and ray-based visibility directly as replaceable inputs, finding systematic gains across architectures and datasets, including up to +9.6 mIoU pts and +3.0 IoU pts in reported conclusion results. The resulting prior-augmented older models become competitive with state-of-the-art systems, with one case surpassing them.

  • Problem

    Existing lidar SSC methods commonly entangle semantic and visibility priors with specific architectures, limiting independent assessment and replaceability.

  • Method

    The paper supplies frozen segmenter pseudo-labels and ray-cast visibility as additional voxel-level inputs to existing SSC networks, with minimal input-layer adaptation and oracle analyses.

  • Results

    +9.6 mIoU pts semantic gain and up to +3.0 IoU pts geometric completion boost are reported, while prior augmentation makes older lightweight models competitive with state-of-the-art systems across tested settings.

  • Takeaways & Limitations

    Semantic prior quality is a dominant driver of semantic SSC quality, while visibility cues add consistent completion gains and the approach remains usable with different segmenters and lidar densities.

  • Takeaways & Limitations

    Visibility priors can be imperfect because lidar noise and registration inconsistencies may conflict with free spaces represented in accumulated SSC ground truth.

Abstract

from arXiv · show

This paper investigates easy strategies to boost the performance of existing networks for lidar semantic scene completion (SSC) without requiring complex architectural redesigns. The fact is that, over the last years, SSC methods have mostly pursued architectural innovations, making the models heavier and more complex, e.g., by jointly training a point cloud semantic segmentation branch. In this work, we take a step back and explore two priors used as simple ingredients (possibly noisy) to improve existing approaches: semantic pseudo-labels and sensor visibility information. Concretely, we provide both kinds of information directly as additional inputs to a given SSC network, requiring only a minimal adaptation of the original architecture. We first demonstrate that endowing input point clouds with semantic pseudo-labels from off-the-shelf segmenters significantly improves the performance of existing SSC models. In fact, by evaluating these models against an oracle, we establish that high-quality semantic priors are a primary driver of semantic gains (mIoU), and that the SSC model can be trained just once with ground-truth semantics and then exploited without retraining using any segmenter. Furthermore, we equip the input lidar point cloud with visibility information that distinguishes between empty spaces (between the lidar and a scanned point) and unknown spaces (outside of lines of sight), providing a secondary performance boost across the tested architectures. We study the design space of data for representing visibility information and bound the remaining headroom with a ground-truth oracle on the free-space labels. On SemanticKITTI, these enhancements make older models competitive with state-of-the-art systems across four architectures, in one case even outperforming them. On the SSCBench-nuScenes benchmark, both priors also transfer with the sparser 32-beam sensor.

1 Introduction

Lidar SSC must infer geometry and semantics from sparse single scans, while existing methods often entangle auxiliary priors with increasingly complex architectures. This paper instead injects semantic and visibility priors as replaceable input data, showing that they boost established networks with minimal architectural change.

  • 1 Introduction: Lidar SSC is difficult because a single scan provides accurate 3D geometry but sparse measurements and no photometric information.SSC jointly estimates scene geometry and semantics in a voxel grid, including regions beyond direct sensor observations.
  • 1 Introduction: Existing methods commonly entangle semantic or visibility priors with jointly trained branches, making each prior’s independent contribution difficult to isolate.This coupling also limits replaceability because changing the prior source can require changes to the completion system.
  • 1 Introduction: The proposed setting supplies semantic and visibility cues as additional voxel-level inputs while otherwise training the SSC model under the original conditions.Semantic labels come from a frozen off-the-shelf point-cloud segmenter and visibility labels distinguish ray-traversed empty voxels from unknown voxels.
  • 1 Introduction: Ground-truth oracle experiments bound each prior’s potential and separate residual errors attributable to semantic information from those attributable to geometry.The oracle is diagnostic rather than deployable.
  • 1 Introduction: Semantic and visibility priors systematically boost four existing SSC architectures across two datasets, making older models competitive with current state-of-the-art systems and surpassing them in one case.The approach requires only widening the input layer and is evaluated with semantic and visibility cues supplied directly to the completion networks.

2 Related work

Related work spans sensor-specific SSC architectures, coupled semantic priors, and ray-based visibility reasoning. The paper distinguishes its approach through decoupled, input-level priors for single-frame lidar SSC, including frozen segmenter predictions and explicit empty-space visibility.

  • Lidar SSC: Lidar SSC methods range from lightweight BEV completion networks to computationally intensive diffusion models, reflecting increasingly large and intricate architectures.Earlier lidar approaches used voxelized TSDFs and 3D CNNs, while later methods adopted BEV, latent diffusion, point-space diffusion, and sparse octree representations.
  • 2.2 Priors in SSC: Semantic priors are often introduced through jointly trained segmentation branches or coupled representations, tying completion performance to specific architectures and training procedures.Replacing or upgrading a jointly trained segmenter can require retraining the completion network.
  • 2.2 Priors in SSC: The paper’s novelty is decoupled input-level prior injection for single-frame lidar SSC, requiring minor input-layer adaptation without test-time adaptation.It identifies this as the first study of explicit empty-space visibility priors decoupled from the completion architecture in this setting.
  • Visibility priors: Unlike prior visibility work requiring multiple frames and per-sequence test-time optimization, this study uses single frames and precomputed visibility consumed at the SSC input.Camera-based visibility methods are related but address a different sensing modality.
  • 2.2 Priors in SSC: The proposed semantic prior uses predictions from a frozen off-the-shelf segmenter and shows that an SSC network trained with ground-truth semantic input can generalize to inference-time pseudo-labels without retraining.This decouples completion from the particular semantic teacher and allows stronger segmenters to be adopted later.
  • Visibility priors: Visibility priors exploit the sensor line of sight to distinguish traversed free space from occluded or unknown regions, building on ray-casting and related occupancy reasoning.The paper evaluates efficient visibility representations as input signals to SSC networks.

3 Study of individual priors

The study isolates semantic and visibility priors by adding them directly to existing SSC inputs, showing complementary gains with minimal architectural changes.

  • 3.1 Integration of raw priors in a lidar SSC architecture: The priors enlarge the input state representation only, distinguishing semantic classes and ray-traversed empty voxels from unknown space without auxiliary branches.The augmented grid uses semantic labels for occupied voxels and visibility labels for empty space, while unknown voxels remain distinct.
  • 3.3 Semantic prior: Semantic completion mIoU broadly rises with input-segmentation mIoU across segmenters, and pseudo-labels recover a large but incomplete fraction of the oracle headroom.The best off-the-shelf segmenter reaches 70.3% input mIoU, leaving substantial semantic headroom relative to perfect labels.
  • 3.4 Visibility prior: Visibility estimates remain imperfect because lidar noise and registration inconsistencies can conflict with free-space labels in accumulated SSC ground truth.These issues constrain the reliability of single-scan visibility supervision.
  • 3.4 Visibility prior: Visibility markers consistently improve completion; dense uniform sampling reaches 57.5% IoU on LMSCNet-SS and yields +1.9 and +3.0 IoU pts over prior-free models.From 10 markers per ray onward, marker placement matters little, while the visibility oracle leaves additional geometric headroom.
  • 3.4 Visibility prior: Visibility cues also provide mild semantic gains of up to +1.6 and +1.1 mIoU pts, while oracle visibility adds +3.1 and +3.6 mIoU pts over vanilla models.The two priors are chiefly complementary because semantic and occupancy coupling remains weak.

4 Prior combination

The paper injects semantic and visibility priors directly into existing SSC backbones as additional voxel-level inputs, requiring only backbone-specific input adaptation and no new auxiliary branch.

  • 4 Prior combination: The combined prior tensor assigns each voxel a semantic class for occupied locations and an empty or unknown visibility state otherwise.The resulting state set contains C + 2 categories.
  • 4 Prior combination: The recipe modifies only each backbone’s existing input pathway, adding no auxiliary network or jointly trained branch.The implementation widens or adapts the input representation while preserving downstream components.
  • 4 Prior combination: Semantic and visibility priors enter different backbones through architecture-specific pathways, including input embeddings, BEV completion modules, and point-segmentation branches.JS3C-Net receives visibility at completion input but semantic labels through its jointly trained point-wise branch.

4.2 Experimental setup

The experiments cover SemanticKITTI and SSCBench-nuScenes, with standardized class schemes, retrained baselines, selected semantic and visibility priors, and oracle variants. The setup also excludes a refinement method because its implementation leaks ground-truth occupancy.

  • Dataset and evaluation: SSCBench-nuScenes uses a sparser 32-beam lidar than SemanticKITTI’s 64-beam sensor and provides 500 training plus 199 validation sequences.The validation split comprises sequences 500–698; the reported 200-scene count is inconsistent with the released split.
  • Dataset and evaluation: The SSCBench-nuScenes experiments use a 12-class taxonomy for both supervision and semantic priors, remapping WaffleIron’s 16-class pseudo-labels before voxelization.This keeps the input semantic prior and output targets in a single class space.
  • Dataset and evaluation: Baselines are retrained under the paper’s 12-class scheme, with retrained LMSCNet-SS reaching 37.6 IoU versus 21.1 for the reported LMSCNet baseline.The released checkpoint follows a different 10-class scheme, so its reported 12-class results cannot be recovered directly.
  • Prior configuration: Experiments use WaffleIron with and without test-time augmentation, dense free-space markers spaced at the 20 cm voxel size, and oracle semantics and visibility.The chosen visibility prior is dense uniform sampling along lidar lines of sight.
  • Evaluation constraints: The generic SemCity refinement is omitted because its official implementation uses non-annotated unobserved ground-truth voxels and leaks surface locations.This leakage would expose information about surfaces that should be reconstructed.

4.3 Quantitative results

Semantic and visibility priors improve existing SSC architectures across SemanticKITTI and transfer to the sparser SSCBench-nuScenes sensor. The strongest semantic comparison places the combined recipe ahead of reproducible single-frame baselines, while oracle results reveal additional headroom.

  • SemanticKITTI prior augmentation: Both priors yield consistent gains across all four SemanticKITTI architectures, with semantic improvements ranging from +0.9 to +9.5 mIoU points.The largest semantic gains occur for SemCity-AE and LMSCNet-SS, while SSA-SC and JS3C-Net gain less.
  • SemanticKITTI prior augmentation: Semantic gains are concentrated where architectures lack strong internal semantic pathways, whereas geometric gains are larger for models that are weaker at completion.For example, LMSCNet-SS improves by +9.5 mIoU and +3.1% IoU, while JS3C-Net improves by +0.9 mIoU and +0.4% IoU.
  • Prior interaction: The two priors are mostly complementary: adding semantics changes geometry by at most +0.4 IoU, while adding visibility changes LMSCNet-SS semantics by +2.2 mIoU and barely affects SemCity-AE.The combination therefore provides gains along both completion and semantic axes rather than simply duplicating one prior’s effect.
  • Oracle headroom: Oracle semantics add +6.2 to +11.0 mIoU points, while oracle visibility further adds +4.1 to +6.3 IoU points and up to +8.4 mIoU points.These results show substantial remaining semantic and geometric headroom and continued complementarity between the priors.
  • Comparison to reproducible SSC baselines: 28.9% mIoU makes prior-augmented SSA-SC the best fully reproducible single-frame method, ahead of DPS2CNet at 26.7% mIoU.The comparison excludes methods using non-comparable panoramic or multi-frame test-time protocols.
  • Runtime: Prior augmentation adds latency mainly through semantic pseudo-label inference, while the extra SSC input channels cost less than 5 ms per frame.Free-space sampling and voxelization add approximately 6 ms and 9 ms per frame, respectively.
  • SSCBench-nuScenes transfer: On SSCBench-nuScenes, semantic priors add +11.1 to +11.7 mIoU points and the combined recipe adds +11.1 to +12.2 mIoU plus +1.5 to +2.0 IoU points.Visibility alone remains positive but smaller than on SemanticKITTI, adding +0.7 and +1.2 IoU points with the two evaluated methods.

4.4 Qualitative results

Across SemanticKITTI and the sparser SSCBench-nuScenes scans, input semantic and visibility priors produce cleaner, more structured SSC outputs than prior-free baselines.

  • 4.4 Qualitative results: Input priors make SemCity-AE and LMSCNet-SS predictions cleaner and more structured, with roads connected, vegetation better aligned, and thin structures more reliably restored.The comparison covers two validation scenes per dataset and contrasts each prior-augmented model with its prior-free counterpart and ground truth.
  • 4.4 Qualitative results: The same qualitative improvement holds on 32-beam SSCBench-nuScenes scans, showing transfer across lidar densities.The appendix reports corresponding quantitative per-class confirmation.

4.5 Segmenter-agnostic training

The completion network can be trained once with ground-truth semantics and then used with different segmenters at inference, with negligible performance cost from not retraining.

  • 4.5 Segmenter-agnostic training: Not retraining changes performance by −0.5 to +0.2 occupancy IoU points and −0.2 to +1.0 semantic mIoU points across segmenters.The comparison uses the full recipe with semantic cues and dense visibility markers, contrasting GT-trained and segmenter-trained completion networks.
  • 4.5 Segmenter-agnostic training: The semantic prior is segmenter-agnostic because the completion head can be trained once on ground-truth semantics and later receive arbitrary off-the-shelf segmenter labels without retraining.This decouples the completion network from a specific segmenter, unlike the rigid coupling in jointly trained multi-task methods.
  • 4.5 Segmenter-agnostic training: The authors hypothesize that overfitting makes training-time pseudo-labels resemble ground-truth semantics, explaining why GT-trained models tolerate segmenter substitution at inference.This is presented as a hypothesis rather than an established mechanism.
  • 4.5 Segmenter-agnostic training: The decoupling does not extend to visibility: training with oracle visibility degrades performance when testing with dense, noisy free-space markers.Thus, semantic and visibility priors differ in their robustness to a train–test cue mismatch.

5 Conclusion

The paper concludes that input-level semantic and visibility priors improve existing lidar SSC models with minimal architectural change, while semantic quality remains the main source of headroom.

  • 5 Conclusion: The prior augmentation systematically improves completion and semantics, bringing older lightweight models to recent state-of-the-art levels among reproducible single-frame SSC methods.The approach is reported across four SSC methods, four segmenters, and two datasets with different lidar sensors.
  • 5 Conclusion: The method applies broadly with no architectural cost beyond widening the input layer, and supports varied off-the-shelf semantic and visibility cues.The priors are supplied at input level rather than through jointly trained branches.
  • 5 Conclusion: Segmenter-agnostic training lets a model trained once with ground-truth point labels later use any segmenter, while prior-augmented performance grows with prior accuracy.The conclusion also reports that visibility benefits from lidar density whereas the semantic prior does not directly depend on it.
  • 5 Conclusion: Semantic gains reach up to 12 mIoU points, whereas completion improves by no more than 3 IoU points, identifying semantics as the dominant bottleneck.Oracle input semantics leaves substantial additional headroom above prior-augmented baselines.
  • 5 Conclusion: The authors identify future directions including multiple frames and learned visibility priors to reduce free-space marking noise.These directions accompany further exploration of oracle headroom and geometry–semantics coupling.

Appendix A Conference paper extension

The journal version extends the conference study with broader experiments, visibility analysis, architectural interpretation, oracle headroom analysis, and segmenter-agnostic training.

  • Appendix A Conference paper extension: It systematically studies visibility-prior design and discusses why visibility has less impact on SSCBench-nuScenes, attributing this to fewer and noisier visibility cues.The visibility analysis is part of the expanded evaluation and design-space study.
  • Appendix A Conference paper extension: The journal version adds experiments on SSCBench-nuScenes with a different lidar sensor, confirming the SemanticKITTI results.The added benchmark uses a sparser 32-beam sensor.
  • Appendix A Conference paper extension: The extension adds an architectural account of uneven gains, deeper analysis of combining priors, and improved discussion of oracle headroom for semantics and visibility.These additions analyze how benefits vary across SSC methods and how much improvement ideal priors could leave.
  • Appendix A Conference paper extension: A new segmenter-agnostic training analysis shows that the completion network need not be retrained when the semantic segmenter changes.This extends the prior study beyond the conference version’s SemanticKITTI-only experiments.
  • Appendix A Conference paper extension: The journal version also adds experiments showing that GT visibility training degrades performance with actual noisy visibility cues, plus inference-time analysis and expanded related work.These additions include an inference-time study and a substantially expanded related-work discussion.

Appendix B Inference time profiling

Inference profiling shows that visibility-prior preprocessing is inexpensive relative to SSC networks and semantic segmenters, with voxelization dominating preprocessing cost.

  • The end-to-end latency measurements agree with the sum of independently measured ray-sampling and voxelization stages within 1.8%, confirming that these stages account for essentially all preprocessing runtime.The discrepancy is at most 0.07 ms across methods.
  • Voxelization dominates visibility-prior preprocessing, while ray sampling takes at most 1.2 ms for sparse variants and becomes substantial mainly for dense variants.Marker-free voxelization costs 3.2 ms/frame; the reported end-to-end timings include ray sampling and voxelization.
  • 33–665 ms/frame is required by SSC networks, while semantic segmenters take 155–1871 ms/frame, making prior preprocessing comparatively inexpensive.These timings are measured on an NVIDIA A100 GPU; the SSC range spans SSA-SC to JS3C-Net, and the segmenter range spans WaffleIron to WaffleIron with TTA.
  • The dense visibility prior is the default because it provides the best or near-best accuracy, has no sample-count hyperparameter, and costs only 2.3 ms/frame more than 100-random.With both priors, its runtime share is at most about 7% in the cited SSA-SC setting and below 1% with WaffleIron-TTA.

Appendix C Per-class results

Per-class analysis shows that semantic-prior gains concentrate on classes that lightweight baselines predict poorly without semantic input.

  • Truck IoU rises from 3.7 to 30.9 for LMSCNet-SS and from 6.9 to 46.0 for SemCity-AE when semantic priors are added on SemanticKITTI.The appendix reports per-class IoU on SemanticKITTI and SSCBench-nuScenes for the prior combinations studied in the main tables.

Appendix D Visibility oracle training

Visibility-oracle training does not transfer cleanly to dense deployed visibility priors: the mismatch sharply reduces occupancy recall while preserving or improving precision.

  • 21.7 and 11.9 IoU points are lost when LMSCNet-SS and SemCity-AE switch from oracle-trained to dense visibility at inference without retraining.The semantic prior is held fixed at WaffleIron in this comparison.
  • Occupancy recall falls from 0.66 to 0.38 on LMSCNet-SS and from 0.76 to 0.53 on SemCity-AE, while occupancy precision rises from 0.82 to 0.87 and from 0.70 to 0.75.The authors interpret this as predicting too little occupancy rather than placing occupancy incorrectly.
  • The oracle visibility prior is therefore a performance ceiling, not a training-time substitute for the deployed prior.The cited comparison establishes a train/test mismatch specific to visibility priors.
Loading 2609.17413v1…