PaperlayerView exact revision on arXiv

Source-linked AI summary

Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

Cigdem Kokenoz, Amir Salarpour, Alkim Domeke, Christopher Salas, Pedram MohajerAnsari, Long Cheng, Mert D. Pesé, Bing Li

arXiv:2610.00855v1cs.CVcs.RO

TL;DR

Supervised LiDAR segmentation requires point-wise annotation and is tied to closed taxonomies, while prior open-vocabulary point-transformer work had not addressed outdoor LiDAR. Lang3DSeg trains a point transformer from scratch and corrects projection errors through offline curation. It reports the highest annotation-free mIoU on nuScenes and SemanticKITTI without a vision-language model at inference.

  • Problem

    Supervised LiDAR segmentation depends on dense point-wise annotation and closed taxonomies, while open-vocabulary point-transformer segmentation had not been established for sparse outdoor LiDAR.

    Source: §I · p. 1

  • Method

    Lang3DSeg trains a PointTransformerV3 from scratch on open-vocabulary supervision and uses offline curation with priority-ordered masks and instance-wise depth truncation to correct projection errors.

    Source: §I · pp. 1–2, A. Offline Target Curation · p. 3

  • Results

    52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI are the highest reported annotation-free results on both benchmarks.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    A single-sweep 3D network can deliver annotation-free open-vocabulary segmentation without running vision-language models at inference.

    Source: §I · p. 2, §VI · p. 8

    Single-sweep accuracy trails sequence-consolidating methods for motorcycle, traffic cone and pedestrian, including pedestrians at 30 m with too few returns to separate from background structure.

    Source: §V · p. 7

Abstract

from arXiv · show

Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it into a 3D network. These methods rely almost exclusively on voxel-based sparse convolutions, and point transformers have so far been limited to indoor environments, where 3D data is dense and bounded. We present Lang3DSeg, which establishes a point transformer as the backbone for annotation-free open-vocabulary segmentation of outdoor 3D LiDAR, and is trained from scratch without geometric pre-training. This training paradigm necessitates addressing the inherent noise in 2D-to-3D label projections; specifically, naive projection often suffers from depth ambiguity, where points behind an object are erroneously assigned its semantic label. We therefore composite masks using an explicit class-priority rule and truncate each projected instance at the first gap in its depth distribution, correcting the projection error directly rather than averaging it over registered sequences. Lang3DSeg achieves 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI, the highest among published annotation-free methods on both benchmarks. Every 3D semantic segmentation is on a single LiDAR sweep, and inference operates in real-time without running vision-language models.

I. INTRODUCTION

Lang3DSeg addresses annotation and taxonomy limits in outdoor LiDAR segmentation by training a point transformer from scratch and correcting projection errors offline. It reports the highest annotation-free mIoU on both driving benchmarks.

I. INTRODUCTION

  • Supervised point transformers require dense point-wise labels and remain tied to closed taxonomies, while outdoor LiDAR is sparse and spans long distances.Prior point-transformer open-vocabulary work was limited to dense, bounded indoor scenes.Source: §I · p. 1
  • Lang3DSeg trains a PointTransformerV3 from scratch without geometric pre-training and pairs it with offline curation that corrects projection errors directly.The approach avoids compensating for errors after they enter training.Source: §I · pp. 1–2
  • Occlusion bleed is a dominant systematic projection error: points behind an object can receive its label from similar viewpoints, so temporal voting may reinforce the mistake.A per-pixel test is insufficient when the occluded point lands on another pixel inside the object mask.Source: §I · p. 2
  • Priority-ordered mask rasterization and instance-wise depth truncation address projection errors without registration or cross-frame voting.The paper identifies occlusion bleed as concentrated in foreground categories important to class-averaged accuracy.Source: §I · p. 2
  • 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI are the highest reported annotation-free results on both benchmarks.The system segments a single LiDAR sweep without a vision-language model at inference.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

The related work is organized around where vision-language foundation models sit relative to inference, the 3D backbone, and how methods handle projection noise.

II. RELATED WORK

  • The review distinguishes methods by whether the foundation model remains involved at inference or sits elsewhere in the pipeline.This is one of three organizing dimensions for the related work.Source: §II · p. 2
  • The review compares the 3D backbones adopted by open-vocabulary 3D scene-understanding methods.Backbone choice is a separate organizing dimension from foundation-model placement.Source: §II · p. 2
  • The review also groups methods by how they handle projection noise.This completes the three stated dimensions of organization.Source: §II · p. 2

A. Cross-Modal Distillation and Open-Vocabulary Frameworks

Prior work spans test-time vision-language inference and offline distillation, while open-vocabulary LiDAR backbones have largely voxelized points. Lang3DSeg places a point transformer in the outdoor setting.

A. Cross-Modal Distillation and Open-Vocabulary Frameworks

  • PointCLIP and OpenScene use image-plane projections and test-time CLIP features, while OpenMask3D [13] refines masks with SAM before extracting features.Running vision-language models across several cameras for each sweep is described as unsuitable for embedded platforms.Source: A. Cross-Modal Distillation and… · p. 2
  • Offline approaches include direct point-text alignment with ULIP, sequence correspondence learning with SLidR, Seal and CLIP2Scene, and pseudo-label distillation with OV3D, GGSD and AFOV.AFOV uses tri-modal contrastive pre-training, and LOSC initializes from ScaLR.Source: A. Cross-Modal Distillation and… · p. 2

B. Backbones for Open-Vocabulary LiDAR Segmentation

  • Open-vocabulary LiDAR methods predominantly voxelize sweeps or project them onto dense grids, using backbones such as SPVCNN, MinkowskiNet, SparseConvNet and WaffleIron.The cited methods include CLIP2Scene, AdaCo, CNS, OpenScene, Better Call SAL, 3D-AVS, OV3D, GGSD, AFOV and LOSC.Source: B. Backbones for Open-Vocabulary LiDAR… · p. 2
  • Point transformers attend over serialized neighborhoods of raw points, and PointTransformerV3 scales to LiDAR sweep point counts.Prior point-transformer open-vocabulary evaluation described here used dense, bounded indoor scenes; Lang3DSeg is presented as the first outdoor LiDAR distillation approach, trained from scratch.Source: B. Backbones for Open-Vocabulary LiDAR… · p. 2

C. Noise Mitigation and Geometric Consistency

Lang3DSeg treats much projection noise as systematic geometry error, filtering labels at projection time rather than averaging them across sequences. Its target-curation pipeline also uses per-sweep geometry to resolve residual terrain ambiguity.

C. Noise Mitigation and Geometric Consistency

  • Prior noise-mitigation approaches use per-frame DBSCAN outlier removal, temporal voting, adaptive label correction, or approximate-flat label propagation.The cited discussion describes temporal voting as effective against per-frame flicker.Source: C. Noise Mitigation and Geometric… · p. 2
  • Temporal voting can reinforce identical behind-car errors; Lang3DSeg instead cuts projected instances at their first depth gap and resolves residual ground ambiguity with a per-sweep plane model.The method uses the depth available at projection time rather than relying on similar viewpoints to correct the label.Source: C. Noise Mitigation and Geometric… · p. 2

III. METHODOLOGY

  • Because a from-scratch backbone has no prior to discount projection errors, Lang3DSeg curates targets offline using information from one sweep.The framework then trains PointTransformerV3 on those curated targets.Source: §III · pp. 2–3, Figure 2 · p. 3
  • A per-sweep RANSAC ground model separates flat terrain from vegetation, and unlabeled points are filled only when neighboring labels agree.The framework caption describes both steps as part of offline target curation.Source: Figure 2 · p. 3

A. Offline Target Curation

  • SAM 3 receives separate natural-language phrase queries per class, while fixed class priority protects small foreground masks from broad road masks and mask score breaks within-class ties.Priority is higher for thin and small foreground categories and lower for road surface.Source: A. Offline Target Curation · p. 3, Table I · p. 4
  • For foreground instances, points are depth-sorted and retained through the first gap greater than 2.5 m, preserving continuous objects such as buses and trailers.The depth-gap test is not applied to background regions, which can legitimately extend in depth.Source: A. Offline Target Curation · p. 3

THAT COVERS LARGE CONTIGUOUS REGIONS, AND THE ROAD SURFACE

The target-curation rules use ground proximity and neighborhood geometry to distinguish terrain from vegetation, then conservatively fill unlabeled points.

Ground-plane purification

  • Vegetation points are reassigned to terrain only when they lie near the fitted ground and their neighborhood has limited vertical span.The tests use an elevation threshold of 0.35 m and a vertical-span threshold of 0.5 m within a 1.0 m radius; the span test protects tree trunks.Source: THAT COVERS LARGE CONTIGUOUS REGIONS,… · p. 4
  • Within 15 m, local planarity is additionally required, using planarity above 0.35, surface variation below 0.20, and a ground-aligned normal.The normal condition is |v3,z| > 0.80.Source: THAT COVERS LARGE CONTIGUOUS REGIONS,… · p. 4
  • Beyond the near field, ring spacing makes covariance-based planarity unreliable, so the range-independent elevation test remains the primary signal.The text gives approximate consecutive-ring spacing of 1.3 m at 10 m range and 5 m at 20 m for the described sensor.Source: THAT COVERS LARGE CONTIGUOUS REGIONS,… · p. 4

Label completion

B. Network

Lang3DSeg uses a PointTransformerV3 backbone trained from scratch on single-sweep point inputs, retaining vertical position in the sensor frame.

Backbone

  • PointTransformerV3 is initialized from scratch without geometric pre-training or supervised initialization, and consumes one LiDAR sweep.Its encoder widths are (32, 64, 128, 256, 512), with depths (2, 2, 2, 6, 2) and stride 2 between stages.Source: B. Network · p. 4
  • The decoder widths are (64, 64, 128, 256), and points are serialized on a 0.05 m grid using z-order and Hilbert curves.Source: B. Network · p. 4

Input representation

  • Coordinates are centered in x and y but not z, and each point feature contains x′, y′, z, and normalized return intensity.The passage states that height helps separate target classes and that centering z introduces a train–test shift under differing sampling.Source: B. Network · p. 4

Backbone

  • The network predicts dense N-by-K logits and 512-dimensional point embeddings for each sweep.Source: B. Network · p. 4

C. Objective

The objective combines weighted semantic losses with a light cosine-alignment term that places point embeddings in CLIP text space.

Semantic terms

  • Weighted cross-entropy and Lovász-Softmax are combined, with Lovász-Softmax directly optimizing an IoU surrogate.Source: C. Objective · p. 4
  • Class weights use inverse square-root frequency, are normalized to unit mean, and are clipped to [0.2, 8].The frequencies are measured on curated targets; the passage says inverse frequency without the square root destabilizes training with Lovász-Softmax.Source: C. Objective · p. 4

Cosine alignment

  • A light cosine-alignment term places point embeddings in CLIP text space using frozen text anchors for the assigned classes.The anchors enter training through this term but are absent from the forward path, allowing inference without a text encoder.Source: C. Objective · p. 4

Total objective

IV. EXPERIMENTS AND EVALUATION

The experiments evaluate Lang3DSeg on nuScenes and SemanticKITTI using their stated training and validation splits, with mean IoU as the metric. Training uses full sweeps and specified augmentations; inference averages outputs over four vertical rotations, though latency excludes that averaging.

A. Experimental Setup

  • nuScenes uses 700 training scenes and 150 validation scenes over 16 categories; SemanticKITTI trains on sequences 00–07, 09, and 10 and validates on sequence 08 over 19 categories.No human annotations are used for curation or training; ground truth is used only to compute metrics.Source: A. Experimental Setup · p. 5
  • Evaluation reports mean Intersection-over-Union, excludes ignore-labeled points, and maps SemanticKITTI moving categories to static counterparts under the single-scan protocol.Source: A. Experimental Setup · p. 5
  • Training uses full unsampled sweeps, AdamW, a one-cycle schedule, and geometric augmentations; inference averages softmax outputs across four vertical rotations, while latency excludes this averaging.Effective batch sizes are 12 on nuScenes and 16 on SemanticKITTI; latency is measured without rotation averaging.Source: A. Experimental Setup · p. 5

B. Benchmark Comparison

Lang3DSeg posts the highest published annotation-free mIoU on both benchmarks. Ablations identify occlusion-depth curation as a clear label-quality contributor, while other design choices have mixed or negative effects.

B. Benchmark Comparison

  • 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI sequence 08 are the highest reported annotation-free figures on both benchmarks.The comparison is against published annotation-free results.Source: B. Benchmark Comparison · p. 5
  • No controlled backbone substitution separates the architecture’s contribution from that of curated targets.Source: B. Benchmark Comparison · p. 5
  • A full nuScenes sweep takes about 52.6 ms and SemanticKITTI takes about 88 ms; inference uses no image data, and the method uses one training run without self-supervised initialization.The nuScenes latency exceeds 19 frames per second at batch size one without test-time augmentation.Source: B. Benchmark Comparison · p. 5

C. Ablation Studies

  • The occlusion depth test raises retained-label quality by 2.3 points, including car by 9.1, pedestrian by 7.2, and barrier by 6.1, but trailer loses 5.9.Manmade absorbs most removed error because walls and buildings often lie behind foreground objects.Source: C. Ablation Studies · p. 5
  • Ground-plane refinement changes labeled mIoU by −0.1; label completion trades 1.6 points of per-point accuracy for 3.7 points of coverage and gains 3.3 when unlabeled points count as errors.Source: C. Ablation Studies · p. 5
  • A predominantly indoor self-supervised checkpoint gives no improvement over scratch training, while routing text anchors through cross-attention costs 1.6 to 1.9 mIoU.The authors suggest pre-training benefits may be domain-specific; the cross-attention branch adds parameters to fit from noisy supervision.Source: C. Ablation Studies · pp. 5–6

D. Qualitative Results

Qualitative results show clean ground-to-structure boundaries but persistent errors on small, distant, and co-planar objects. The limitations section traces further constraints to depth-gap curation, sparse single-sweep observations, camera coverage, and fixed-class inference.

D. Qualitative Results

  • Ground-to-structure boundaries are clean, but errors cluster on small or thin distant objects and co-planar ground boundaries where one sweep provides insufficient evidence.Ground-plane refinement produces clean boundaries without changing the aggregate metric.Source: D. Qualitative Results · p. 6

V. LIMITATIONS AND FUTURE WORK

  • A depth-gap threshold can reject a long vehicle’s own sparse returns: trailer IoU falls 5.9 points while car IoU gains 9.1.Class-dependent thresholds would reintroduce circularity into curation.Source: §V · p. 6
  • Single-sweep sparsity limits motorcycle, traffic-cone, and pedestrian accuracy versus sequence consolidation; SemanticKITTI’s single forward camera supervises only the frontal frustum.A pedestrian at 30 m may return too few points to distinguish it from background structure, and novel objects in camera blind spots remain out of reach.Source: §V · p. 7

CHARGES UNLABELED POINTS AS ERRORS AND THEREFORE REFLECTS

  • Co-planar flat surfaces lack a geometric cue for separation, and flush islands remain unresolved; nuScenes curation costs about 270 NVIDIA H200 GPU-hours.The deployed network has fixed-class logits, and the paper does not evaluate scoring novel categories against CLIP text embeddings.Source: CHARGES UNLABELED POINTS AS ERRORS AND… · p. 7
  • Variants use a single seed, making differences of about one point uninformative; hand-written prompt ensembles must be rewritten for a new taxonomy.Proposed future work includes extent-adaptive depth thresholds, language-model-generated prompts, and selective multi-sweep use for sparse instances.Source: CHARGES UNLABELED POINTS AS ERRORS AND… · p. 8

VI. CONCLUSION

Lang3DSeg uses a point transformer for annotation-free, open-vocabulary outdoor LiDAR segmentation. It reports leading results on two driving benchmarks with single-sweep inference and no vision-language model at deployment.

VI. CONCLUSION

  • 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI sequence 08 set new state-of-the-art results without self-supervised initialization, sequence consolidation, or iterative retraining.Every 3D operation uses a single sweep; the deployed network processes a full 360° scan in about 50 ms using coordinates and intensity.Source: §VI · p. 8
  • Lang3DSeg establishes that a PointTransformerV3 backbone can train from scratch on open-vocabulary supervision for outdoor LiDAR segmentation.The framework is annotation-free and keeps vision-language foundation models offline.Source: §VI · p. 8
  • The authors present the framework as a practical basis for real-time open-world 3D semantic perception.Source: §VI · p. 8
Loading 2610.00855v1…