PaperlayerView exact revision on arXiv

Source-linked AI summary

SplatLabel: Pseudo-Labelling through 4D Gaussian Splatting

Nitya Nanvani, Andras Palffy, Holger Caesar

arXiv:2609.29836v1cs.CVcs.RO

TL;DR

Automated 3D pseudo-labelling faces costly annotation and challenges in handling dynamic scenes, inconsistent semantics, and varying recall. SplatLabel combines explicit temporal modeling with semantic and geometric priors in a 4D Gaussian pipeline, and reports stronger performance than baselines on SemanticKITTI across recall levels and occupancy metrics.

Abstract

from arXiv · show

While 2D Vision Foundation Models offer a pathway to automate 3D semantic pseudo-labelling, translating these priors into robust 3D representations typically requires complex heuristics or multi-model ensembles. We introduce SplatLabel, an automated pipeline that leverages a 4D Gaussian representation to extract LiDAR segmentation with predictive confidence, as well as semantic occupancy grids at arbitrary voxel resolutions. At its core, SplatLabel handles dynamic environments through an explicit temporal manifold that models the trajectories and lifespans of individual 3D primitives. This allows the system to accurately track moving actors and strictly define when objects appear and disappear, completely eliminating the need for pre-annotated 3D bounding boxes. To robustly support this dynamic tracking, the representation is grounded by structural and semantic priors: we guide scene geometry in unobserved regions by integrating 360-degree LiDAR via virtual depth maps, and rather than relying on domain-specific prompt engineering, we directly distill continuous soft probabilities from 2D models to inherently resolve semantic ambiguities over time and space. Finally, to accurately reflect the real-world trade-off between precision and recall, we reframe pseudo-label evaluation as a selective classification task using a generalized risk-recall metric. Experiments on SemanticKITTI demonstrate that SplatLabel consistently outperforms state-of-the-art baselines across multiple recall levels, establishing a highly robust framework for both 3D LiDAR segmentation and occupancy prediction.

I. INTRODUCTION

Automated 3D pseudo-labelling must address dynamic scenes, inconsistent semantics, and evaluation across different recall levels. SplatLabel uses a 4D Gaussian representation with explicit temporal tracking and spatiotemporal semantic and geometric priors.

I. INTRODUCTION

  • Existing 3D pseudo-labelling pipelines require pre-annotated boxes for dynamic scenes,, complex heuristics or ensembles for consistent semantics and 360-degree LiDAR,,,, and assess accuracy only on successfully labeled points, ignoring recall trade-offs.Source: §I · p. 1
  • SplatLabel parameterizes motion and temporal existence in individual Gaussians, tracking actors against static backgrounds without pre-annotated 3D bounding boxes.Its Dual Deformation Kernel and Gated Deformation Mask support object-centric tracking across time.Source: §I · pp. 1–2
  • The pipeline distills continuous soft semantics and uses 360-degree LiDAR priors to ground geometry, resolving semantic ambiguity through spatiotemporal consensus.This approach avoids domain-specific prompt engineering and guides scene geometry in unobserved regions.Source: §I · p. 2
  • The paper reformulates pseudo-label evaluation as selective classification with class-averaged AUGRC to capture risk across recall and class imbalance.The stated aim is to reveal trade-offs that standard baselines overlook.Source: §I · p. 2

A. Spatiotemporal Semantic Distillation in 3DGS

  • SplatLabel directly distills continuous soft logits into 4D Gaussians, aiming to converge on a unified semantic identity across time and space.The passage frames this as an alternative to complex prompts or multi-model ensembles used to address temporal inconsistency and label flickering.Source: A. Spatiotemporal Semantic… · p. 2

B. LiDAR-Guided Geometric Optimization

This block covers LiDAR-guided geometry, dynamic scene modeling, and evaluation of pseudo-label reliability. SplatLabel uses virtual depth maps and a bounding-box-free temporal representation, and evaluates performance across recall with a class-averaged risk-coverage metric.

B. LiDAR-Guided Geometric Optimization

  • Virtual Depth Maps use 360-degree LiDAR in unobserved regions without spherical rasterization, while a geometry-guided sampler spawns primitives where structure is inconsistent.Prior approaches commonly discard measurements outside the visual frustum or use LiDAR only as post-rendering supervision.Source: B. LiDAR-Guided Geometric Optimization · p. 3

C. Dynamic Scene Modeling and the Temporal Manifold

  • SplatLabel combines an explicit Dual Deformation Kernel with a temporal manifold to separate moving actors from static backgrounds while preserving semantic identities without 3D boxes.This addresses the box requirement of explicit object-centric tracking and semantic changes over time in implicit methods.Source: C. Dynamic Scene Modeling and the… · p. 3

D. Selective Classification for Pseudo-Label Evaluation

  • SplatLabel’s performance-recall profile consistently dominates the single-operating-point baselines LeAP and UniLiPs, with tight variance across five random seeds.The figure reports mean mIoU and standard deviation.Source: Figure 3 · p. 3
  • Class-averaged AUGRC evaluates selective risk across recall while addressing how standard metrics can mask minority-class rejection under dataset imbalance.The formulation is presented as a holistic, bias-aware measure of label reliability.Source: D. Selective Classification for… · p. 3

III. METHODOLOGY

The method takes continuous LiDAR point clouds and 2D images as input to generate 3D semantic pseudo-labels and volumetric occupancy grids. It centers the representation on explicit temporal tracking and regulates Gaussian capacity using multimodal priors.

A. Problem Formulation and Overview

  • The task is to generate dense 3D semantic pseudo-labels and volumetric occupancy grids from continuous LiDAR point clouds and 2D images.The central challenge is tracking moving actors while keeping the static environment stable over time.Source: A. Problem Formulation and Overview · p. 3
  • An explicit temporal manifold makes motion and lifespan intrinsic to individual Gaussian primitives, defining when and where objects appear and separating actors from static backgrounds.This formulation aims for a globally consistent 4D reconstruction.Source: A. Problem Formulation and Overview · p. 3

B. Spatiotemporal Distillation and Geometric Priors

  • To regulate primitive capacity under semantic and depth supervision, the method replaces fragile photometric densification heuristics with MCMC state transitions via SGLD and incorporates multimodal priors.The passage identifies altered loss scales as the reason standard densification heuristics fail.Source: B. Spatiotemporal Distillation and… · pp. 3–4

1) Continuous Semantic Distillation:

SplatLabel grounds its 4D Gaussian representation with continuous semantic distillation, omnidirectional LiDAR geometry, and explicit temporal modeling. Its trajectory model combines polynomial and Fourier components to represent different motion patterns.

1) Continuous Semantic Distillation:

  • Raw SAM 3 logits queried with base dataset labels provide dense continuous soft labels, whose sequence-level gradients smooth frame-to-frame flickering and resolve categorical ambiguity.The approach bypasses domain-specific prompt manipulation and backpropagates semantic loss into Gaussian parameters.Source: 1) Continuous Semantic Distillation · p. 4

2) Omnidirectional Geometric Supervision:

  • Virtual Depth Maps project full LiDAR point clouds into artificial cameras placed in unobserved regions, and LiDAR cues guide primitive spawning where geometry is inconsistent.This preserves use of the efficient pinhole rasterizer while integrating 360-degree geometry.Source: 2) Omnidirectional Geometric Supervision · p. 4

C. Temporal Manifold Parameterization

  • The temporal manifold makes motion and temporal existence intrinsic to each Gaussian primitive, unifying spatial tracking and temporal boundaries.It is designed to model dynamic environments without decoupling actors from global environmental deformations.Source: C. Temporal Manifold Parameterization · p. 4

1) Dual Deformation Kernel:

  • The Dual Deformation Kernel combines an n-degree polynomial framework with an n-degree Fourier noise network to model smooth paths and high-frequency variation.The passage describes the polynomial component as capturing smooth linear paths and the Fourier component as capturing complex, high-frequency variations.Source: 1) Dual Deformation Kernel · p. 4

2) Gated Deformation Mask:

SplatLabel uses a gated deformation mask to separate static geometry from moving actors, then couples that mask to temporal lifespans and opacity activation. These mechanisms preserve static primitives while constraining when dynamic primitives exist.

2) Gated Deformation Mask:

  • The deformation mask clamps background deformation to zero and prioritizes respawning along dynamic trajectories when moving primitives have low opacity.Scaling respawn probabilities with the motion mask helps reconstruct moving actors.Source: 2) Gated Deformation Mask · p. 4

3) Deformation-Coupled Lifespan:

  • Static primitives retain a sequence-wide lifespan, while dynamic primitives are bounded by localized lifespans and culled below a temporal threshold.Primitives below the minimum threshold are respawned.Source: 3) Deformation-Coupled Lifespan · p. 4

4) Temporal Existence Activation:

  • Temporal existence activation multiplies spatial opacity, progressing from smooth temporal gradients to strict boundaries as sharpness increases.The activation is evaluated around each primitive’s temporal center within its effective lifespan.Source: 4) Temporal Existence Activation · p. 4

5) Trajectory-Preserving Temporal Decoupling:

Temporal decoupling lets SplatLabel shift a primitive’s existence window without disrupting its learned trajectory. Regularization encourages binary deformation masks and smooth motion unless structural gradients support complexity.

5) Trajectory-Preserving Temporal Decoupling:

  • Temporal exploration and cloning can shift existence windows while analytic position compensation preserves established kinematics on global sequence time.Cloning splits a dynamic primitive’s lifespan into sequential segments to represent piecewise motion.Source: 5) Trajectory-Preserving Temporal… · p. 4

D. Deformation and Trajectory Regularization

  • Table I reports per-class LiDAR segmentation IoU on SemanticKITTI at the baseline methods’ specific recall levels.Best results are highlighted in bold.Source: Table I · p. 5
  • Mask sparsity and self-entropy penalties drive deformation values toward 0 or 1, while trajectory-coefficient L1 regularization favors smooth paths.Complex motion is favored when supported by structural gradients.Source: D. Deformation and Trajectory… · pp. 4–5

E. Unified Downstream Extraction

SplatLabel extracts LiDAR segmentation and semantic occupancy from its continuous representation, then evaluates pseudo-label quality across recall using a class-averaged risk-coverage metric.

E. Unified Downstream Extraction

  • A voxel-based filter uses accumulated LiDAR points to remove unconstrained floaters before continuous Gaussians are converted into discrete outputs.This grounds extraction in physical scene geometry.Source: E. Unified Downstream Extraction · p. 5
  • Inverse-distance voting gives LiDAR points a local semantic consensus and Shannon-entropy uncertainty, while occupancy labels use filtered Gaussians and semantic consensus at eight voxel vertices.The occupancy procedure supports arbitrary target voxel resolutions and addresses boundary ambiguity.Source: E. Unified Downstream Extraction · p. 5

F. Pseudo-Labelling as Selective Classification

  • Treating unassigned predictions as rejected enables risk-coverage comparisons across pipelines with different recall levels.Generalized Risk measures the joint probability that an accepted pseudo-label is incorrect.Source: F. Pseudo-Labelling as Selective… · p. 5
  • Class-averaged AUGRC evaluates pseudo-label risk across recall while accounting for class imbalance that can make global selective metrics appear optimistic.Lower AUGRC indicates better performance, and the metric is bounded between 0 and 0.5.Source: F. Pseudo-Labelling as Selective… · p. 5

IV. EXPERIMENTS

On SemanticKITTI, SplatLabel is evaluated for LiDAR segmentation and semantic occupancy. It outperforms the compared segmentation baselines at their recall levels and occupancy baselines under the stated protocol.

A. Experimental Setup

  • The evaluation uses SemanticKITTI’s Sequence 08 validation set, with zero-shot SAM 3 semantic features and independently trained approximately 20-second scenes stitched at inference.The setup uses NVIDIA A40 GPUs and directly prompts dataset class names rather than using complex prompt engineering.Source: A. Experimental Setup · p. 5

B. LiDAR Segmentation and Recall Evaluation

  • The segmentation evaluation uses 11 common merged classes and six KITTI-360 categories.The classes include car, road, vegetation, and terrain; the categories include flat, construction, nature, human, and vehicle.Source: B. LiDAR Segmentation and Recall… · pp. 5–6
  • SplatLabel maintains a superior segmentation performance profile across recall levels and outperforms baseline operating points, while occupancy results exceed zero-shot transferred baselines and AutoOcc.The segmentation baselines label 65.3% of points for LeAP and an average of 47.32% for UniLiPs; occupancy comparisons use the stated cross-dataset protocol.Source: B. LiDAR Segmentation and Recall… · p. 6, Table II · p. 6

C. Semantic Occupancy Prediction

SplatLabel’s occupancy predictions are reported to outperform prior work on geometric and semantic IoU, while its explicit kinematics prevent temporal smearing. The spatial evaluation compares AUGRC across visibility subsets, where lower values indicate a more favorable accuracy-coverage trade-off.

C. Semantic Occupancy Prediction

  • Explicit kinematic parameterization prevents temporal smearing artifacts in aggregated ground truth data.Occupancy evaluation covers 18 semantic classes and excludes the ambiguous “other flat/ground” class.Source: C. Semantic Occupancy Prediction · p. 6
  • SplatLabel demonstrates superior Geometric and Semantic IoU, with five-seed standard deviations of 0.066 and 0.041 percentage points, respectively.The reported variances are described as much smaller than the margins over previous baselines.Source: C. Semantic Occupancy Prediction · p. 6

D. Spatial Domain and Generalized Risk Evaluation

  • Spatial reliability is assessed with AUGRC across all points (100% recall), camera-frustum points (92.0%), and unoccluded points (62.3%); lower values indicate a better accuracy-coverage trade-off.Unoccluded status uses a 0.5-meter tolerance when comparing true LiDAR depth with rendered depth.Source: D. Spatial Domain and Generalized Risk… · p. 6

E. Computational Resources

The pipeline is parallelized over continuous scene splits, with distinct training and per-frame inference costs for segmentation and occupancy prediction. Ablations report that semantic and geometric priors, as well as coupled spatiotemporal kinematics, support performance.

E. Computational Resources

  • Training takes 15.28 GPU hours for LiDAR segmentation and 33.67 GPU hours for occupancy prediction; inference averages 0.28 and 4.62 seconds per frame, respectively, on one NVIDIA A40.The dataset is divided into independent, approximately 20-second continuous scene splits; experiments use four NVIDIA A40 GPUs concurrently.Source: E. Computational Resources · pp. 6–7

1) Semantic and Geometric Priors:

  • Replacing continuous soft labels with hard assignments degrades performance, removing Virtual Depth Maps sharply lowers Geometric IoU, and bypassing Geometry Filtering harms all metrics.The authors attribute soft-label benefits to resolving categorical ambiguity through spatiotemporal consensus; Virtual Depth Maps integrate 360-degree LiDAR into unobserved regions.Source: 1) Semantic and Geometric Priors · p. 7

2) Spatiotemporal Kinematics:

  • Disabling the Gated Deformation Mask introduces geometric jitter, while decoupling lifespan from the deformation mask causes a consistent drop across metrics.The authors state that retaining a full sequence-wide footprint for static primitives is vital to global scene stability.Source: 2) Spatiotemporal Kinematics · p. 7

VI. CONCLUSION

SplatLabel uses an explicit temporal manifold to track dynamic actors while preserving static backgrounds without pre-annotated 3D boxes. The authors report strong SemanticKITTI results, note input and resource constraints, and propose downstream validation as future work.

VI. CONCLUSION

  • SplatLabel combines an explicit temporal manifold, continuous soft semantics, and 360-degree LiDAR priors for annotation-free 3D pseudo-labelling and occupancy prediction.The temporal representation parameterizes motion, lifespans, and physical existence boundaries as primitive properties and eliminates the need for pre-annotated 3D boxes.Source: §VI · p. 7

A. Limitations

  • The approach requires high VRAM and precise camera-LiDAR synchronization, and robustness degrades in adverse lighting or weather when sensor inputs are compromised.Source: A. Limitations · pp. 7–8

B. Future Work

  • Future work will train downstream models on generated labels and benchmark them against models trained on human annotations.The authors also plan to use predictive confidence scores to dynamically weight downstream training losses.Source: B. Future Work · p. 8

Related papers

Loading 2609.29836v1…