PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Semantic Scene Completion

Shi Chen, Weifeng Ge

arXiv:2608.26737v1cs.CVcs.LGcs.RO

TL;DR

Outdoor LiDAR SSC must infer dense semantic volumes from scans covering about 1% of the target volume amid severe class imbalance. GSSC uses one discrete-diffusion framework for paired data synthesis, noise-conditioned completion, and frozen-base refinement, reaching 38.8% hidden-test mIoU in one step without test-time augmentation. Its scope remains bounded because the refiner cannot exceed its source and its correction latency does not yet pay for itself within a one-second window.

  • Problem

    Outdoor LiDAR SSC must infer dense semantic voxel grids from scans observing about 1% of the target volume under severe class imbalance.

    Source: §I · p. 1

  • Method

    GSSC uses one discrete-diffusion formulation for PS3 paired synthesis, SGSC noise-conditioned completion, and S2D2 refinement from a frozen prediction.

    Source: §II · p. 3, D. Structured Source Discrete… · p. 9

  • Results

    38.8% hidden-test mIoU is reached with one correction step and no test-time augmentation, +2.1 pp over SCPNet’s published 36.7%.

    Source: §I · p. 1, A. Main Results · p. 11

  • Takeaways & Limitations

    S2D2 improves SGSC and every external SSC base tested without base retraining or test-time adaptation.

    Source: §V · p. 16

    The refiner cannot exceed its source, and its correction pass does not pay for its latency inside a one-second window.

    Source: §V · p. 17

Abstract

from arXiv · show

Outdoor LiDAR semantic scene completion (SSC) recovers a dense semantic voxel grid from a scan observing 1% of the target volume, under class imbalance beyond 7,000x. We recast SSC as generative semantic scene completion (GSSC): a single discrete-diffusion formulation in three roles. First, paired sparse-dense scene synthesis (PS$^3$) generates matched sparse LiDAR observations with their dense semantic completions, addressing the long tail at its source and yielding the PS$^3$-SemanticKITTI corpus we train on alongside SemanticKITTI. Second, semantic-guided generative scene completion (SGSC) generates the scene from noise with multinomial discrete diffusion, conditioned on the sparse scan through a bird's-eye-view semantic map and a sparse 3D feature stream. Third, the same framework instead refines an existing completion in one flow-matching step: structured source discrete diffusion (S$^2$D$^2$). S$^2$D$^2$ improves the mIoU of SGSC's own output and every external SSC base tested, without base retraining or test-time adaptation. On the strongest base, one step without test-time augmentation reaches 38.8% mIoU on the SemanticKITTI hidden test. To our knowledge that is the best causal, single-sweep, single-sample result on that leaderboard, +2.1 pp over the previous best published score under the same restriction. Four correction steps with eight-view test-time augmentation reach 39.2%, outside that restriction.

I. INTRODUCTION

Outdoor LiDAR semantic scene completion must infer dense structure and semantics from extremely sparse, long-tailed observations. GSSC addresses these constraints with one discrete-diffusion framework for data synthesis, completion, and frozen-base refinement.

Motivation

  • One LiDAR sweep observes about 1% of the target volume, leaving occluded, thin, and rare structures to be inferred.The paper identifies rare classes, thin structures, ghost trails, boundary errors, and occlusion as recurring failure modes.Source: §I · p. 1

Approach

  • GSSC uses one discrete-diffusion formulation in three roles: PS3 paired synthesis, SGSC base-free completion, and S2D2 frozen-base refinement.PS3 supplies matched sparse–dense pairs, SGSC starts from categorical noise, and S2D2 transports a frozen prediction toward ground truth.Source: §I · pp. 1–2

Results

  • 38.8% mIoU is reached on the SemanticKITTI hidden test with one sampling step and no test-time augmentation, +2.1 pp over SCPNet’s published 36.7%.The result is reported under a causal, single-sweep, single-sample restriction.Source: §I · p. 1

Motivation

  • Vegetation accounts for 26.7% of labelled points versus 0.0037% for motorcyclists, and class frequency correlates with published per-class IoU.The reported Pearson correlation is r=0.80.Source: §I · p. 1, Figure 1 · p. 2

Results

  • Four correction steps with eight-view test-time augmentation reach 39.2%, but this result lies outside the single-sweep, single-sample restriction.The restriction excludes multi-sweep entries, test-time adaptation, and ensembling.Source: §I · p. 2

Data and limitations

  • PS3-SemanticKITTI adds rare-class-amplified synthetic sparse–dense pairs and improves a voxel-grid-native base when trained with real data.The paper also identifies thin structures as barely moving and reports that occlusion is not isolated experimentally.Source: §I · p. 2

II. RELATED WORK

Prior work includes discriminative voxel and point-based completion, scene generation, and structured-source diffusion. The paper distinguishes GSSC by learning complete labelled scenes, generating matched observations, and correcting frozen predictions directly in the scored label space.

Existing SSC approaches

  • Outdoor SSC methods primarily change representation, supervision, input features, or completion architecture, while generative approaches model complete scenes.Examples include voxel labelling, implicit fields, BEV–voxel fusion, and camera-based refinement.Source: §II · p. 3

Generative scene data

  • PS3 differs from prior paired scene generation by supplying sparse–dense supervision with amplified rare classes.Its purpose is to generate scenes together with the observations that would have produced them.Source: §II · p. 3

Generative completion

  • SGSC conditions categorical diffusion on BEV semantics and sparse 3D features rather than only the sparse cloud used by related from-noise work.The paper evaluates the value of each conditioning stream under an oracle BEV setting.Source: §II · p. 3

Structured-source diffusion

  • S2D2 starts from a frozen network’s prediction and transports it deterministically toward ground truth, unlike stochastic structured-source diffusion baselines.The method operates directly in the discrete label space scored for SSC.Source: §II · p. 3

III. METHODOLOGY

The methodology represents semantic voxel grids categorically and applies multinomial discrete diffusion to generate plausible complete scenes. GSSC instantiates this formulation across PS3 data construction, SGSC completion, and S2D2 refinement.

Problem formulation

  • SSC predicts a categorical semantic voxel grid Y from sparse LiDAR observations X, using 20 labels on a 256 × 256 × 32 SemanticKITTI grid.The 20 labels comprise 19 semantic classes plus one empty/unlabelled class.Source: §III · p. 4

Framework

  • GSSC learns a distribution of plausible complete scenes rather than a single deterministic completion.Its stages use uniform noise or a frozen prediction as sources with distinct conditioning designs.Source: §III · p. 4

GSSC stages

  • PS3 builds paired sparse–dense training data, SGSC completes from noise under LiDAR-derived conditioning, and S2D2 refines a frozen prediction.The three stages are presented as successive components of the shared framework.Source: §III · p. 4

Discrete diffusion

  • Multinomial diffusion keeps each voxel’s label with probability 1−β_t or uniformly resamples it with probability β_t.The closed-form marginal mixes the clean label toward the uniform simplex point, with x_T approximately uniform noise.Source: A. Preliminaries · p. 4
  • The reverse process estimates the clean categorical distribution with a simplex-valued denoiser because the clean scene is unavailable at inference.The reverse posterior remains categorical and is available in closed form for t>1.Source: A. Preliminaries · p. 4

B. Paired Sparse–Dense Scene Synthesis (PS3)

PS3 creates reusable sparse–dense training pairs offline by generating complete scenes, balancing rare classes, and rendering realistic LiDAR observations.

Data-pair construction

Scene generation

Distribution filtering

Rare-class augmentation

Observation rendering

D. Structured Source Discrete Diffusion (S2D2)

S2D2 refines a frozen completion by transporting its one-hot prediction toward the one-hot ground truth on the per-voxel simplex, enabling short deployment schedules.

Structured source

  • S2D2 replaces the usual noise prior with a frozen base prediction and uses the ground truth as the fixed target for discrete flow matching.The straight path between endpoints enables a single deployment step.Source: D. Structured Source Discrete… · p. 7

Source conditioning

  • The structured source is the base’s decoded label map, while conditioning combines a base-derived BEV semantic map with raw-scan occupancy.The scan reinjects geometric evidence without requiring a completed-scene input.Source: D. Structured Source Discrete… · pp. 7–9

Forward process

  • The deterministic interpolant is xt = ᾱt x0 + (1 − ᾱt) xsrc, remaining on the simplex while anchoring the terminal state to the base prediction.The implementation uses ᾱT ≈ 5.6 × 10^-3 rather than an exactly terminating schedule.Source: D. Structured Source Discrete… · p. 8

Denoiser

Training objective

  • The variational objective trains on categorical draws around the path and uses the source-conditioned posterior rather than plain velocity regression.The source-conditioned reverse shares the SGSC parameterisation while replacing the uniform prior with xsrc.Source: D. Structured Source Discrete… · pp. 8–9

Scope boundary

  • The shipped schedule does not exactly terminate at the structured source, so the deterministic-path likelihood identity is not claimed for deployment.Its effect is bounded by the 2ᾱT term.Source: D. Structured Source Discrete… · p. 9

E. Why the Correction Works

S2D2’s correction works because the structured path preserves the base prediction and makes errors telescope rather than amplify, while one-step sufficiency remains empirical.

Error localization

  • Where the base prediction is correct, the structured path is pinned to the ground truth; where it is wrong, training contrasts the target with the source.Replacing this objective with timestep-uniform cross-entropy collapses training.Source: E. Why the Correction Works · p. 10

Objective design

  • The added loss terms vanish when the endpoint estimate is correct, preserving the ground-truth endpoint as a minimizer.They provide gradients where the KL term is flat because the base is already correct.Source: E. Why the Correction Works · p. 10

One-step deployment

  • At N=1, the bound rules out error amplification but does not bound the network residual, so one-step sufficiency is empirical.The single evaluation starts from the source state used during training.Source: E. Why the Correction Works · p. 10

IV. EXPERIMENTS

Experiments evaluate PS3 augmentation, SGSC, and S2D2 on SemanticKITTI, with additional frozen-checkpoint zero-shot evaluation and explicit split-aware baseline reporting.

Datasets

  • SemanticKITTI contains 19,130 training frames, 4,071 validation frames, and 3,901 hidden-test frames on a 256×256×32 grid.Its distribution is severely skewed, with empty voxels comprising approximately 95% of the ground truth.Source: §IV · p. 10
  • PS3 adds 32,039 synthetic pairs to SemanticKITTI, expanding training from 19,130 to 51,169 frames, while two further datasets are used only for zero-shot evaluation.The synthetic pool derives from the real training split and adds no target labels or fine-tuning in the further domains.Source: §IV · p. 10

Training

  • SGSC and S2D2 share one approximately 35M-parameter denoiser architecture and training recipe on NVIDIA H100 GPUs.The recipe uses AdamW, learning rate 10^-4, effective batch 4, and T=100.Source: §IV · p. 10

Evaluation protocol

  • Published baselines are reported at their own leaderboard results, whereas the authors’ per-class and ablation evidence uses validation sequence 08.Table I includes an explicit evaluation split column and should not be read across splits.Source: §IV · p. 11, Table I · p. 11
  • S3CNet leads every single-sweep vulnerable-road-user class despite not being the authors’ method.Its completion IoU is 45.6 versus 56.1–68.5 for the other listed test rows.Source: §IV · p. 11, Table I · p. 11

A. Main Results

S²D² reaches 38.8% mIoU on the SemanticKITTI hidden test with one correction step over a frozen SCPNet base, while validation gains concentrate in recoverable, underrepresented classes. The method improves most classes but includes an explicitly discounted motorcyclist result and qualitative tail-class improvements.

Headline result

  • 38.8% mIoU is reached on the hidden test with one N=1 correction step over a frozen SCPNet base, +2.1 pp over its published score.The comparison uses the causal, single-sweep, single-sample predicate; the base was not re-submitted.Source: A. Main Results · p. 11

Validation result

  • 38.54% validation mIoU is achieved by the released checkpoint, compared with 38.05% for the from-scratch retrain.Table II reports these values for SemanticKITTI validation sequence 08 with N=1.Source: Table II · p. 11

Class-wise analysis

  • Eighteen of nineteen classes improve, with the largest gains for bicyclist and truck (+5.3 each), road (+4.6), and traffic-sign (+4.2).The gain is class-selective and concentrates where the frozen base is weakest; motorcycle decreases by 0.1 point.Source: A. Main Results · p. 11
  • Motorcyclist’s reported +8.3 pp validation gain is discounted because it falls to +0.3 on retrain and to +1.3 against the published test row.The authors attribute the validation figure to a seed-and-split artefact and reduce the associated VRU-IoU gain from +4.9 to +2.5 on retrain.Source: A. Main Results · p. 11

Qualitative analysis

  • One correction step leaves bulk classes nearly unchanged while roughly doubling rare-class IoU and reducing phantom voxels by about 6.6× on the motorcyclist scene.The qualitative comparison reports under-1% IoU drift for road, sidewalk, building, and vegetation, with the comparison drawn from the N=4, +D4-TTA configuration.Source: A. Main Results · p. 12, Figure 6 · p. 12

B. S2D2 as a Base-Agnostic Refinement Operator

S²D² acts as a single-pass refinement operator trained separately on each frozen base, improving all three tested bases without retraining them. Its benefit is strongest for voxel-grid-native bases and can trade bulk structural accuracy for semantic gains on weaker or mismatched architectures.

Operator design and transfer

  • A single N=1 pass improves LMSCNet by +1.8, JS3C-Net by +1.6, and SCPNet by +2.36 pp validation mIoU.Each denoiser is trained on its own base’s decoded output, and no base is retrained or architecturally modified.Source: B. S2D2 as a Base-Agnostic Refinement… · p. 12, Table III · p. 12
  • The controlled comparison is per-base uplift, because each base uses its own denoiser, pool, step budget, and LiDAR stream.The three frozen bases span a lightweight 2D–3D hybrid, a point-voxel coupled network, and a distillation-trained state-of-the-art model.Source: B. S2D2 as a Base-Agnostic Refinement… · p. 12

Implementation caveat

  • The SCPNet port reaches 36.17% validation mIoU under spconv 2.3, 1.0 pp below the paper’s 37.2%, so test comparisons quote SCPNet’s published 36.7%.The release depends on unavailable spconv 1.0; kernel-shape patches replicate its shared-indice-key behavior, but the port was not re-submitted.Source: B. S2D2 as a Base-Agnostic Refinement… · p. 12

Transfer boundary

  • JS3C-Net gains 1.6 points overall but loses on person and bicyclist, leaving VRU-IoU at 2.4 from 3.8 and scoping the benefit to voxel-grid-native bases.The synthetic pool also costs this point-voxel architecture −3.8 pp under the internal evaluator.Source: B. S2D2 as a Base-Agnostic Refinement… · p. 13

Precision–recall trade-off

  • On weaker bases, correction can prune false positives without improving completion IoU and can reduce bulk-class IoU, including building losses on both LMSCNet and JS3C-Net.The SCPNet base improves completion IoU from 49.9 to 52.7, whereas LMSCNet remains at 46.7 and JS3C-Net loses 2.2 points.Source: B. S2D2 as a Base-Agnostic Refinement… · p. 13

C. Ablations

The ablations isolate how conditioning, synthetic data, loss design, and diffusion-step choices affect completion and refinement. Conditioning quality and the interaction between KL and class-balanced surrogates are especially consequential, while one deployment step preserves performance.

Conditioning

  • Oracle BEV conditioning preserves stronger SGSC performance than predicted BEV conditioning, whose road-user semantics collapse while bulk geometry remains intact.The BEV stream alone reaches 34.08%, while adding per-voxel features reaches an oracle ceiling of 42.30%; deployed SGSC reaches 30.5%.Source: C. Ablations · p. 13, Figure 8 · p. 14

Conditioning and data

  • +1.1 pp remains after removing the base-derived BEV stream, versus ∼+1.7 pp after removing the 32K synthetic pool.The measured uplift from the full recipe is +2.5 pp, making the BEV stream the dominant component and synthetic data the second.Source: C. Ablations · pp. 13–14

Loss design

  • 10.3% results when KL is replaced by clean-target cross-entropy, while KL alone reaches 36.1% against the base’s 36.17%.The +2.36 pp headline therefore depends on the interaction between KL posterior anchoring and class-balanced Lovász-softmax and focal surrogates.Source: C. Ablations · p. 14

Step budget

  • 38.65% is the peak across one to 100 correction steps, with deployed N=1 only 0.11 pp below that peak.The sweep supports one-step deployment as a 100× speedup without measurable loss under the stated single-seed band.Source: C. Ablations · p. 14

Training and deployment

  • EMA weights are required, with their removal costing −3.5 pp; a single-frame retrain reaches 38.4%, within 0.1 pp of the released checkpoint.The headline uses one deployment sweep, although training accumulated several sweeps.Source: C. Ablations · p. 14

D. Transfer and Secondary Tasks

The refiner transfers zero-shot beyond SemanticKITTI and also improves 2D BEV perception, while synthetic data reshapes rare-class frequency and rare-class gains can be large but uneven across scenes.

Transfer

  • 54.9% completion IoU is reached on SemanticPOSS after refinement, versus 31.8% for the frozen SemanticKITTI checkpoint.On this different-sensor dataset, mIoU also rises from 1.0 to 6.5% without target labels or fine-tuning.Source: D. Transfer and Secondary Tasks · p. 14
  • 19.5 completion IoU on SSCBench-KITTI360 exceeds the frozen checkpoint’s 18.1 over 16 shared classes.Its mIoU change from 5.8 to 6.2 lies inside the assumed single-seed band, so completion IoU carries the transfer claim.Source: D. Transfer and Secondary Tasks · p. 14

Secondary task

  • +9.1 pp over 2D S3CNet’s 27.0% is achieved by applying the same residual-transport machinery to 2D BEV perception.The resulting 36.1% BEV mIoU includes 34.8 from parameter-free projection of the frozen 3D base and +1.3 from S2D2.Source: D. Transfer and Secondary Tasks · p. 14

Long-tail data

  • 3.2×–3,907× gains in pooled rare-class voxel frequency result from adding synthetic data to real training scenes.The synthetic pool’s own frequency relative to real-only spans 4.6×–6,239×.Source: Figure 9 · p. 15

Qualitative gains

  • +40.1 IoU points is the largest illustrated D4-TTA-vs-base gain among the six rare-class validation scenes.The figure compares SCPNet with S2D2 using 32K multi-frame training and D4 test-time augmentation under shared framing.Source: Figure 10 · p. 15, D. Transfer and Secondary Tasks · p. 15

E. Where the Method Loses

The method’s failures are bounded by source and guidance quality, rare-class gains do not reliably reproduce, vulnerable-road-user performance remains behind a prior method, and latency can outweigh accuracy gains.

Source-bounded failures

  • The method cannot exceed its guidance or source, and its evaluated failure mode is erasure rather than hallucination.The transfer claim is scoped to voxel-grid-native sources; the official metric excludes never-observed volume.Source: E. Where the Method Loses · pp. 15–16
  • 0.7% person IoU remains after S2D2 erases pedestrians recovered by the JS3C-Net base.Among frames where the base recovers the class, one step lowers person IoU on 74.7% and bicyclist IoU on 92.5% of frames.Source: E. Where the Method Loses · p. 15, Figure 12 · p. 16

Reproducibility

  • +0.3 pp is the reproduced motorcyclist gain after retraining, versus +8.3 pp in the original validation result.Against the published test row, the gain is +1.3 pp, so the rarest class is not claimed.Source: E. Where the Method Loses · p. 16

Safety metric

  • 32.6 VRU-IoU for S3CNet exceeds the method’s 21.6 among single-sweep rows.The paper does not lead on safety, and a 2021 method remains ahead on the introduced tail-sensitive metric.Source: E. Where the Method Loses · p. 16

Latency

  • 3.23 FPS end-to-end is slower than the frozen base’s 4.95 FPS on an idle H100.At a one-second window, DW-VRU-IoU moves from 54.9 to 49.7 on test, so the accuracy gain does not yet pay for latency.Source: E. Where the Method Loses · p. 16

V. CONCLUSION

GSSC uses one discrete-diffusion framework for synthesis, completion, and refinement of outdoor LiDAR semantic scenes. S2D2 reaches 38.8% hidden-test mIoU in one step without test-time augmentation, while generality remains limited in scope and latency is unresolved.

Conclusion

  • GSSC unifies PS3 data synthesis, SGSC completion from noise, and S2D2 refinement of frozen base predictions within one discrete-diffusion formulation.The framework supplies training data, completes scenes, and corrects deployed predictions without replacing the discriminative pipeline.Source: §V · p. 16
  • 38.8% hidden-test mIoU is achieved by S2D2 in one step without test-time augmentation, reported as the best causal, single-sweep, single-sample result to date.The checkpoint also transfers zero-shot to SemanticPOSS and, on completion IoU, to SSCBench-KITTI360.Source: §V · p. 16

Limitations

  • The refiner can erase rare classes on weaker sources, its correction pass does not repay latency within a one-second window, and generality is shown only for 3D completion and BEV segmentation.Whether it extends to outputs that are per-element categorical simplexes remains untested.Source: §V · p. 17

Related papers

Loading 2608.26737v1…