Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridge

Samed Doğan; Nico Leuze; Alfred Schöttl

Source-linked AI summary

Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridge

Samed Doğan, Nico Leuze, Alfred Schöttl

arXiv:2609.10322v1Sep 9, 2026cs.CV

TL;DR

Sparse 3D LiDAR lacks the scale and annotation resources available for 2D foundation models. The paper trains a LiDAR-conditioned diffusion model on 2D pseudo-labels and probes its projected U-Net features without coordinates, recovering structured 3D information and layered cross-modal organization. The results support diffusion as a mechanism for transferring 2D priors into 3D, while evaluation remains limited in scope and output quality.

Problem
Large-scale, well-aligned 3D data and annotations remain limited relative to 2D vision, constraining the transfer of foundation-model priors to LiDAR.
Method
A LiDAR-conditioned diffusion model learns multiple prompted representations from fixed pseudo-labels generated by off-the-shelf 2D foundation models and is evaluated through coordinate-free point-cloud probes.
Results
The learned features encode non-trivial 3D structure and exhibit modality-specific streams that converge near a shared bottleneck before decoder re-specialization.
Takeaways & Limitations
Diffusion can act as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.
Takeaways & Limitations
Experiments are conducted on a single dataset, and output quality is not competitive with task-specialized 3D networks.

Abstract

from arXiv · show

Transferring the rich priors of large 2D foundation models to sparse 3D LiDAR remains challenging, as training native 3D foundation models at comparable scale is limited by data and annotation scarcity. We introduce a LiDAR-conditioned diffusion model trained on pseudo-labels from off-the-shelf 2D foundation models. The model supports multiple output modalities, including depth, semantic segmentation and instance prediction, selectable via a textual task prompt. Because the model is conditioned on LiDAR, both its outputs and its intermediate UNet features can be projected back onto the input point cloud, enabling analysis of a 3D representation learned entirely under 2D supervision. We study this representation directly in point-cloud space, explicitly excluding raw spatial coordinates to isolate feature content from projection geometry. Linear probes recover up to ~23% Mean Intersection over Union (MIoU) on 3D semantic classes, compared to ~3.5% for a matched Gaussian-noise control, indicating substantial non-trivial structure. Pairwise cosine similarity across modality-specific feature streams reveals a layered organization. Early encoder layers remain weakly aligned across modalities while individually decodable, intermediate layers converge toward a shared representation, and decoder layers re-specialize toward task-specific outputs. These findings indicate that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck. This positions diffusion as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.

1 Introduction

The paper addresses limited scale and annotation for aligned 3D LiDAR data by transferring 2D foundation-model priors through a LiDAR-conditioned diffusion framework. It probes the resulting features in point-cloud space without raw coordinates and finds structured, layered 3D representations.

Large-scale, well-aligned multi-modal 3D data remains a central bottleneck for LiDAR perception.
The framework generates dense multi-task scene representations from LiDAR using textual prompts and pseudo-labels from 2D foundation models.
The probing protocol evaluates diffusion features directly on point clouds while excluding raw spatial coordinates.
Despite lacking 3D ground-truth labels, the learned features encode structured 3D information across modality-specific streams that converge near a shared bottleneck.

2 Related Work

Prior work established diffusion as a scalable generative framework and showed that intermediate U-Net features from 2D diffusion models can transfer to perceptual tasks. This paper extends that feature-oriented perspective to LiDAR-conditioned, multi-modal 3D representations.

Diffusion models support conditional generation through text and spatial conditioning, including scalable text-to-image systems.
Intermediate U-Net features from pretrained text-to-image diffusion models encode perceptual priors useful for dense and open-vocabulary visual tasks.

3 Preliminaries

The model uses latent diffusion conditioned jointly on projected LiDAR views and a textual prompt. Calibrated LiDAR-camera projection maps 3D points to image pixels and supports inverse backprojection from pixels and depth.

Stable Diffusion 1.5 performs diffusion in the latent space of a pretrained autoencoder.
The conditioning signal combines projected LiDAR depth and intensity with a textual prompt selecting the target representation type.
The projection operator maps calibrated 3D LiDAR points to image pixels, while its inverse maps pixels and depth back into 3D.

4 Methodology

The methodology creates fixed pseudo-labels from 2D foundation models, trains a LiDAR-conditioned diffusion model across multiple representations, and evaluates reprojected U-Net features with coordinate-free linear probes and controls.

Pseudo-label Generation: Offline predictions from Depth Anything v3, Segment Anything 2, and Cityscapes-trained SegFormer provide fixed depth, instance, and semantic pseudo-labels.
Pseudo-label Generation: Modality-specific preprocessing enforces geometric and semantic consistency before diffusion training.
Pseudo-label Generation: Relative pseudo-depth is normalized logarithmically, while metric scale is recovered later through least-squares alignment to LiDAR.
Pseudo-label Generation: Instance masks are represented continuously using a foreground channel and two normalized center-offset channels.
Conditioning Ablations: The model compares channel-level and joint pixel-space dropout regimes, each using p = 0.20.
Probing Methodology: Linear probes use reprojected U-Net features in point-cloud space, excluding xyz coordinates and evaluating on nuScenes lidarseg.
Probing Methodology: Long-probe MIoU is reported at the strongest independently selected U-Net level for each ablation, stage, and feature type.
Probing Methodology: The Gaussian-noise control preserves per-channel mean and standard deviation while removing spatial structure.

5 Experiments

Experiments show that LiDAR-conditioned diffusion features encode task-relevant 3D structure and organize modality streams differently across U-Net depth. Conditioning dropout reshapes this organization, while depth evaluation measures the resulting output-space behavior.

5.1 Diffusion Features Encode 3D Relevant Information: 0.231 MIoU is reached on semantic segmentation, versus 0.035–0.038 for the Gaussian-noise control across ablations and tasks.The gap indicates that probes read task-relevant feature content rather than projection geometry.
5.1 Diffusion Features Encode 3D Relevant Information: Encoder probe quality decreases with depth, while decoder features peak at intermediate levels rather than near the shallowest decoder level.Semantic MIoU drops from 0.229 at encoder level 0 to 0.137 at level 3; decoder MIoU is near chance at L0 and peaks at levels 1–2.
5.3 Feature Restructuring: Joint pixel-space dropout is the only conditioning regime that consistently shifts representation structure, suppressing shallow-encoder probe quality and increasing shallow alignment.It also reduces decoder cosine alignment across every modality pair and level; single-channel dropout remains near zero on both metrics.
5.2 Modality Stream Analysis: The U-Net preserves modality-decomposed streams while locally unifying them near a shared bottleneck and re-specializing them in the decoder.Per-level decodability and cosine similarity together indicate that the modalities do not collapse into a single representation.
5.4 Evaluation: The evaluation reports depth metrics on LiDAR points after least-squares metric alignment, while semantic and instance outputs receive qualitative evaluation.Depth metrics include AbsRel, δ1, and RMSE; the study limits quantitative semantic and instance evaluation because of label-alignment factors.

6 Discussion and Limitations

The study finds structured 3D features and layered cross-modal organization, but its evaluation remains limited in dataset coverage, output quality, and quantitative instance assessment.

The model learns non-trivial 3D structure from 2D pseudo-labels, with features progressing from modality-specific shallow representations through a shared bottleneck to task-specialized decoder representations.
Limitations: Experiments use one dataset and one training run, leaving cross-dataset generalization and sensitivity to initialization untested.
Principal-component visualizations show visually distinct decoder feature structures for the same scene under different task prompts.
Predicted semantic outputs are color mapped in point-cloud space, while instance outputs are represented as processed clusters.
Limitations: Instance predictions are evaluated qualitatively because instance metrics depend strongly on post-processing that is orthogonal to the representational analysis.

A Preliminaries

Diffusion progressively corrupts samples with Gaussian noise and learns a conditioned reverse process, while LiDAR-camera projection and backprojection connect 3D points with image pixels and depths.

The forward diffusion process gradually perturbs data samples into Gaussian noise, while the learned reverse process reconstructs data from noise.
A neural network predicts the added noise from the noisy latent, timestep, and conditioning signal.
Latent diffusion performs the diffusion process in a compressed latent space to reduce computational cost.
LiDAR-Camera Projection: A calibrated rigid transformation maps LiDAR points into camera coordinates, where camera intrinsics project them onto the image plane.
LiDAR-Camera Projection: Given a pixel and depth, inverse projection recovers a 3D point, and composite operators define forward projection and full backprojection.

B Short-Probe Convergence

Short probes test linear separability under a fixed optimization budget: real task-conditioned features quickly separate, whereas matched Gaussian noise remains near chance.

The short probe is trained and evaluated on one sample, so it measures linear separability under a fixed optimization budget rather than generalization.
∼0.36–0.65 mIoU is reached by all three task-conditioned feature streams within 30 steps at the narrowest U-Net levels.
∼0.08 mIoU is reached by the matched Gaussian-noise control, whose lower score reflects feature separability rather than probe capacity.

C Implementation Details

The model uses a fixed two-field textual prompt that combines a task modality with a scene description, varying only the task field across modality-conditioned passes.

Each prompt has the form “Task: <modality>. Scene: <description>.”, with modality chosen from depth, semantic segmentation, or instance.
The same scene description is reused across modalities while only the Task field changes for cross-modal cosine analysis.

C.2 Training Configuration

The model is initialized from Stable Diffusion 1.5, conditioned on sparse LiDAR, and jointly trained across depth, semantic, and instance modalities. Feature probes decode backprojected U-Net activations without using spatial coordinates.

Model training: Stable Diffusion 1.5 is adapted with six-channel inputs combining four VAE-latent channels with LiDAR depth and intensity.The added input weights are zero-initialized, while other weights inherit pretrained values; inputs use 224 × 400 resolution and 28 × 50 latents.
Optimization: The model is optimized with AdamW using a 4×10−4 learning rate, 0.01 weight decay, cosine scheduling, and 1000 warmup steps.Classifier-free guidance dropout and conditioning-projection dropout are both 20%, with a minimum learning rate of 10−5.
Model training: All three modalities are trained jointly, with each batch expanded along the modality axis so one optimizer step sees all three prompts.Training uses an effective batch size of 576 across eight NVIDIA A40 GPUs for 50 nuScenes epochs.
Feature probing: Linear probes map each backprojected per-point feature directly to class logits without pooling, nonlinearities, or spatial coordinates.The probes use cross-entropy against pseudo-label classes; coordinates are excluded by construction.
Feature probing: Probe evaluation uses full-batch AdamW updates on fixed validation chunks, with separate short-range and long-range training schedules.The short-range probe runs 30 steps and the long-range probe 60 steps; evaluation uses a disjoint 10-sample bucket.

D Additional Qualitative Results

Additional validation scenes demonstrate the model’s direct 2D outputs, backprojected 3D predictions, feature visualizations, and metric-aligned depth densification. These results cover all six camera views and multiple output modalities.

2D and 3D outputs: Figure 7 presents depth, semantic, and instance predictions across all six nuScenes camera views under textual task prompts.The views are arranged as front three on top and rear three below.
2D and 3D outputs: Figure 8 compares backprojected semantic and instance predictions with ground truth on an additional nuScenes validation scene.Semantic predictions use the Cityscapes label set, while ground-truth colors represent nuScenes lidarseg classes with a random palette.
Feature visualization: Figure 9 visualizes decoder level 2 features for each task prompt by mapping PCA components to RGB on the backprojected LiDAR point cloud.The visualization uses the same prompt and feature level as the main-text Figure 4.
Depth densification: Figure 10 shows dense depth produced by the model after least-squares alignment to the input LiDAR depth scale.The figure compares the original sparse LiDAR scan with the aligned dense depth output.