PaperlayerView exact revision on arXiv

Source-linked AI summary

Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models

Jihun Kim, Hyun-Kurl Jang, Hyemin Yang, Jinnyeong Yang, Hyeokjun Kweon, Kuk-Jin Yoon

arXiv:2608.25418v1cs.CV

TL;DR

4D LiDAR segmentation is constrained by costly, difficult-to-scale temporal annotation of sparse point clouds. LiDAR-SAM2 uses SAM2 video masks, geometric projection, spatio-temporal aggregation, and LiDAR-aware adaptation to generate supervision without manual LiDAR labels. On SemanticKITTI, its labels and downstream models approach full human-annotation and ground-truth performance from minimal prompting.

  • Problem

    Temporally consistent labeling of sparse LiDAR sequences is costly and difficult to scale, while different tasks and domains require fresh dense annotation.

  • Method

    LiDAR-SAM2 converts SAM2 video masks into LiDAR supervision through multi-view projection and spatio-temporal aggregation, then adapts SAM2 to LiDAR with a tailored interface and two-stage objective.

  • Results

    Models trained on automatically generated labels approach full ground-truth supervision, while aggregation raises recall to 79.58% with 85.93% precision on SemanticKITTI.

  • Takeaways & Limitations

    LiDAR-SAM2 substantially reduces annotation cost by providing interactive, spatiotemporally consistent 4D LiDAR segmentation without manual LiDAR labels.

  • Takeaways & Limitations

    The pipeline assumes synchronized RGB images at each timestep and known geometric calibration between LiDAR and camera.

Abstract

from arXiv · show

Progress in 4D LiDAR segmentation is bottlenecked by data. Assigning temporally consistent labels across sparse point cloud sequences is costly and hard to scale, and every new task or domain tends to demand fresh dense annotation. This motivates a simple question of whether high-quality LiDAR training data can be produced automatically, without any human labeling. To this end, we introduce LiDAR-SAM2, a framework that turns a 2D video foundation model, SAM2, into a scalable source of supervision for the 4D LiDAR domain. On the data side, it automatically generates temporally coherent LiDAR-level labels from SAM2 video masks through multi-view projection and spatio-temporal aggregation. On the modeling side, a tailored modality interface and a two-stage learning objective adapt SAM2's video segmentation kernel to spatio-temporal LiDAR structure, so that a single click per object yields a consistent mask track across the sequence. Trained with no human LiDAR annotation, LiDAR-SAM2 produces semantic and panoptic labels on SemanticKITTI that approach the quality of full human annotation from only a few points, and models trained on these labels approach the performance of full ground-truth supervision. This positions LiDAR-SAM2 as a scalable labeling tool that substantially reduces the annotation burden for 3D and 4D scene understanding.

1 Introduction

LiDAR-SAM2 addresses the annotation bottleneck in 4D LiDAR segmentation by using SAM2 to generate scalable supervision and adapt video segmentation to LiDAR structure. Its automatically generated labels support interactive, temporally consistent segmentation and approach full-supervision quality.

  • 4D LiDAR annotation is difficult because sparse, irregular points require labor-intensive temporally consistent labeling across sequences.
  • LiDAR-SAM2 uses a 2D video foundation model as a scalable source of LiDAR supervision instead of fixed human-annotated data.
  • The framework automatically converts SAM2 video masks into dense, temporally coherent point-wise labels through multi-view projection and spatio-temporal aggregation.
  • A tailored modality interface and two-stage learning objective adapt SAM2’s video segmentation kernel to 4D LiDAR geometry and temporal structure.
  • Models trained on LiDAR-SAM2 labels approach models trained on full manual annotation, while a single initial click per object produces consistent mask tracks across sequences.
  • The results position automatic foundation-model supervision as a practical path toward large-scale 3D scene understanding without manual annotation.

2 Related Works

Related work extends 2D vision foundation models to 3D through lifting, distillation, or 3D-aware adapters, but each route has limitations. LiDAR-SAM2 advances adapter-based transfer with a geometry-preserving interface for promptable LiDAR-only inference.

  • 4D LiDAR methods address semantic, instance, panoptic, and moving-object segmentation with differing label granularity and temporal-consistency requirements.
  • Distillation can erode large-scale 2D foundation-model advantages such as broad category coverage and open-set generalization.
  • LiDAR-SAM2 adapts SAM2 through a geometry-preserving interface that enables promptable, LiDAR-only inference.

3 Pseudo-Label Generation using SAM2

LiDAR-SAM2 generates 4D pseudo-labels by propagating SAM2 masks across synchronized videos, projecting them into LiDAR, and aggregating tracks across views and time. Temporal aggregation substantially increases label coverage while retaining high precision.

  • View-wise mask proposals: SAM2 produces temporally consistent 2D mask trajectories by propagating initial mask proposals through each synchronized video.
  • View-wise mask proposals: Each 2D mask is transferred to the LiDAR domain through calibrated LiDAR-camera projection, producing a 3D binary mask for each proposal and timestep.
  • Spatio-temporal 4D mask aggregation: Multi-view spatial aggregation merges view-specific tracks that share LiDAR points and correspond to the same physical object.
  • Spatio-temporal 4D mask aggregation: Temporal aggregation propagates masks using ego-motion and fuses current and propagated masks after voxelizing the point cloud, expanding coverage beyond camera-visible regions.
  • Spatio-temporal 4D mask aggregation: 73.84 segments, 79.58% recall, and 85.93% precision follow aggregation, versus 22.12 segments, 13.59% recall, and 91.90% precision without it.

4 LiDAR-SAM2

LiDAR-SAM2 adapts SAM2 to interactive 4D LiDAR segmentation through a modality interface and two-stage training. Range-view alignment precedes temporal propagation, with refinement restoring point-space detail.

  • 4.1 Overview: The framework’s objective is to predict a 4D mask track from a LiDAR sequence and user-selected point prompts, with per-point segmentation scores over time.Both training stages are supervised by pseudo-labels.
  • 4.2 Learning Cross-Modal Alignment (Stage 1): Stage 1 aligns LiDAR-derived range-view features with SAM2 frame by frame using pseudo-label prompts and an interactive segmentation loss.The trainable components are the LiDAR encoder, lightweight MLP, and SAM2 image encoder with LoRA; other SAM2 components remain unchanged.
  • 4.2 Learning Cross-Modal Alignment (Stage 1): LiDAR-SAM2 represents LiDAR scans in range view, mapping 3D points to image-plane coordinates while preserving geometric structure.A geometric encoder extracts per-point features, which an MLP converts to three channels before rasterization into a dense range-view image.
  • 4.3 Learning Temporal Propagation (Stage 2): Stage 2 learns temporal propagation and object consistency across LiDAR sequences using SAM2 features, temporal modules, and interactive and refinement losses.Stage 1 modules are frozen while the prompt encoder, memory attention, mask decoder, and added temporal components are trained.
  • 4.3 Learning Temporal Propagation (Stage 2): Inverse projection converts range-view predictions into 4D point-wise mask tracks, after which a lightweight 3D decoder refines predictions over temporal windows.The refinement stage restores geometric detail, corrects projection-induced artifacts, and reinstates surface continuity in 3D space.

5 Experiments

Experiments evaluate LiDAR-SAM2 as an interactive labeling tool on SemanticKITTI, measuring semantic and 4D panoptic supervision and its downstream utility against sparse prompts and full ground truth. The generated labels produce strong semantic and panoptic performance while reducing manual annotation requirements.

  • Evaluation Setup: Experiments use SemanticKITTI’s synchronized 64-beam LiDAR and two forward-facing RGB cameras, with training on the training set and evaluation on validation data.Semantic segmentation is measured by 19-class mIoU, while 4D panoptic segmentation uses LSTQ and its component metrics.
  • Evaluation Setup: LiDAR-SAM2 is evaluated by initializing each instance with point prompts on the first frame and extending labels across 4-frame LiDAR sweeps.The interaction protocol places an initial positive click near each object center and adds corrective clicks based on error regions.
  • Semantic Segmentation: Semantic labels from LiDAR-SAM2 substantially outperform naive SAM2 labels, roughly doubling mIoU and approaching full-ground-truth performance for both evaluated backbones.The comparison includes point prompts, RGB-projected SAM2 labels, LiDAR-SAM2 labels, and full human ground truth.
  • Panoptic Segmentation: Panoptic models trained with LiDAR-SAM2 labels achieve strong LSTQ, association, classification, stuff-IoU, and thing-IoU performance, recovering much of full-ground-truth performance.This evaluation uses 4D-PLS, 4D-StOP, and Mask4Former and requires temporally consistent instance identities.
  • Labeling Quality: Direct inspection finds clean object boundaries and temporally consistent identities in both instance-level and semantic-level labels.These qualitative results, together with downstream performance, support LiDAR-SAM2’s use as a labeling tool for 3D and 4D LiDAR segmentation.

6 Conclusion

LiDAR-SAM2 is an interactive 4D LiDAR segmentation framework trained without manual LiDAR labels. By distilling SAM2’s temporal priors into a LiDAR-only model, it produces spatiotemporally consistent segmentation with minimal prompting and substantially reduces annotation cost.

  • 6 Conclusion: LiDAR-SAM2 requires no manual LiDAR labels and uses synchronized video–LiDAR sequences to train a LiDAR-only interactive segmentation model.Its tailored LiDAR-aware pipeline and specialized objectives adapt SAM2’s video segmentation kernel to LiDAR data.
  • 6 Conclusion: The framework produces spatiotemporally consistent interactive segmentation with strong performance from only minimal prompting.The conclusion frames this capability as substantially reducing annotation cost.
Loading 2608.25418v1…