PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridge

Samed Doğan, Nico Leuze, Alfred Schöttl

arXiv:2609.10322v1cs.CV

TL;DR

Large-scale aligned 3D data are scarce, and generative models have primarily targeted observation simulation rather than transferable representations. This paper trains a LiDAR-conditioned diffusion model on 2D foundation-model pseudo-labels and finds that its point-cloud features encode 3D structure and converge toward a shared bottleneck.

  • Problem

    Large-scale aligned multi-modal 3D data remain a LiDAR-perception bottleneck, while generative approaches primarily simulate observations rather than produce transferable representations.

    Source: §1 · p. 2

  • Method

    A LiDAR-conditioned diffusion framework uses 2D foundation-model pseudo-labels and textual task prompts to produce multi-modal scene representations for transfer into 3D point clouds.

    Source: §1 · p. 2, §4.1 · p. 3, §3 · p. 3

  • Results

    0.231 MIoU on semantic segmentation versus 0.035 −0.038 for the Gaussian-noise control; features progress from modality-specific streams toward a shared bottleneck.

    Source: §1 · p. 2, §5.1 · p. 6, §5.2 · p. 7

  • Takeaways & Limitations

    The findings support diffusion as a viable mechanism for transferring large-scale 2D priors into 3D representations under 2D supervision alone.

    Source: §6 · p. 8

    Output quality is not competitive with task-specialized 3D networks, as the work focuses on representation analysis rather than task-specific output optimization.

    Source: §6 · p. 9

Abstract

from arXiv · show

Transferring the rich priors of large 2D foundation models to sparse 3D LiDAR remains challenging, as training native 3D foundation models at comparable scale is limited by data and annotation scarcity. We introduce a LiDAR-conditioned diffusion model trained on pseudo-labels from off-the-shelf 2D foundation models. The model supports multiple output modalities, including depth, semantic segmentation and instance prediction, selectable via a textual task prompt. Because the model is conditioned on LiDAR, both its outputs and its intermediate UNet features can be projected back onto the input point cloud, enabling analysis of a 3D representation learned entirely under 2D supervision. We study this representation directly in point-cloud space, explicitly excluding raw spatial coordinates to isolate feature content from projection geometry. Linear probes recover up to ~23% Mean Intersection over Union (MIoU) on 3D semantic classes, compared to ~3.5% for a matched Gaussian-noise control, indicating substantial non-trivial structure. Pairwise cosine similarity across modality-specific feature streams reveals a layered organization. Early encoder layers remain weakly aligned across modalities while individually decodable, intermediate layers converge toward a shared representation, and decoder layers re-specialize toward task-specific outputs. These findings indicate that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck. This positions diffusion as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.

1 Introduction

Because 3D data and foundation-model priors remain limited relative to 2D, the paper uses LiDAR-conditioned diffusion to transfer 2D priors into 3D representations and probes the resulting features in point-cloud space.

1 Introduction

  • ∼23% MIoU versus ∼3.5% for a Gaussian-noise control shows that the learned features encode meaningful 3D semantic structure.Linear probes evaluated 3D semantic classes while excluding raw spatial coordinates.Source: §1 · p. 2
  • Across network depth, modality-conditioned features remain distinct in early layers, converge near a shared bottleneck, and re-specialize toward task-specific outputs.The network organizes modalities along its depth rather than collapsing them.Source: §1 · p. 2
  • A LiDAR-conditioned diffusion framework transfers 2D foundation-model priors into 3D using pseudo-labels and textual prompts to produce depth, semantic segmentation, and instance predictions.The single backbone's outputs and intermediate features can be reprojected into the input point cloud.Source: §1 · p. 2
  • The probing protocol evaluates diffusion features directly in point-cloud space while excluding raw coordinates to separate feature content from projection geometry.This addresses the possibility that projection geometry, rather than learned features, could account for probe performance.Source: §1 · p. 2

2 Related Work

Prior work extends diffusion from high-fidelity generation to conditional and structured prediction, while showing that pretrained diffusion features transfer to visual perception. This work addresses the open question of how such features behave across multiple prompted outputs and in sparse 3D point-cloud space.

2.1 Diffusion Models for Conditional and Structured Generation

  • Diffusion expanded from high-fidelity generation to text- and spatially conditioned synthesis and dense prediction, including depth estimation and segmentation.DDPMs [16] became a dominant generative paradigm; Stable Diffusion [38] and DALL.E 2 [33] demonstrate scalability under large-scale supervision. ControlNet [61], T2I [29], and IP-Adapter [59] add spatial conditioning, while diffusion has also been applied to depth estimation [20] and segmentation [56].Source: §2.1 · p. 2

2.2 Diffusion Features for Perception

  • Diffusion U-Net features transfer perceptual structure to 2D tasks, but prior studies examine single modalities; this work probes prompted depth, semantic, and instance outputs in 3D point-cloud space.Prior work adapts Stable Diffusion features for dense perception [64] and open-vocabulary panoptic segmentation [55], studies semantic structure across layers and timesteps [27], semantic correspondence [46], and train–inference feature mismatch [44]; [18] uses diffusion features for segmentation without task-specific finetuning. The paper analyzes cross-modal feature organization from one LiDAR-conditioned model, which single-modality probing cannot reveal.Source: §2.2 · p. 3

3 Preliminaries

The paper uses Stable Diffusion 1.5 conditioned on projected LiDAR data and a textual task prompt, alongside calibrated LiDAR-camera projection and backprojection notation.

Latent Diffusion Models

  • Stable Diffusion 1.5 [38] denoises pretrained-autoencoder latents conditioned on projected LiDAR depth and intensity plus a textual prompt selecting the target representation.The denoising network is trained to predict noise added to encoded latents.Source: §3 · p. 3

LiDAR-Camera Projection

  • For calibrated LiDAR-camera pairs, Π maps a 3D LiDAR point to pixel coordinates, while Π−1 maps a pixel and depth value back to 3D.The projection uses known intrinsic K and extrinsic TCL.Source: §3 · p. 3

4 Methodology

The method trains a LiDAR-conditioned diffusion model on fixed, multimodal pseudo-labels from pretrained 2D models, then evaluates its U-Net features in point-cloud space without raw coordinates. It also tests how conditioning-channel dropout affects learning and uses a Gaussian-noise control to distinguish feature structure from projection effects.

4.1 Pseudo-label Generation

  • Fixed depth, semantic, and instance pseudo-labels are generated offline with pretrained 2D models, including Depth Anything v3 [24], Segment Anything 2 [37], and a Cityscapes-finetuned SegFormer [53].The targets are concatenated across the batch axis to form a structured multimodal representation.Source: §4.1 · p. 3

4.1.1 Depth

  • Depth targets use relative depth, assign sky pixels the 0.99-quantile value, then apply logarithmic and quantile-based normalization before replication across three channels.Metric scale is recovered post-hoc by least-squares alignment to LiDAR; the quantile bounds are 0.02 and 0.98.Source: §4.1.1 · p. 4

4.1.2 Instance Segmentation

  • Instance masks are represented by a foreground channel and two normalized center-offset channels, then decoded by multiview fusion and 3D DBSCAN clustering.Road-user bounding boxes initialize iterative mask refinement; candidates require fused foreground confidence above 0.8, and DBSCAN uses a 1 m radius and minimum cluster size of 5.Source: §4.1.2 · p. 4

4.1.3 Semantic Segmentation

  • Semantic class labels are encoded as fixed Cityscapes RGB colors and recovered at inference by nearest-neighbor color quantization.Source: §4.1.3 · p. 4

4.2 Conditioning Dropout Ablation

  • Five conditioning regimes compare no dropout with depth, intensity, random per-channel, and joint pixel-space dropout, each using a 0.20 dropout probability.Channel-level regimes drop entire channels; joint pixel-space dropout applies the same per-pixel mask to both channels, so neither modality is ever fully present or absent.Source: §4.2 · p. 4

4.3 Probing Methodology

  • U-Net features are reprojected onto LiDAR points and evaluated with a linear probe excluding xyz coordinates, while a matched Gaussian-noise control tests whether scores reflect feature structure.Probes use 60 full-batch optimization steps over chunks of 10 training samples and are evaluated on a disjoint 10-sample validation bucket; the control preserves per-channel feature means and standard deviations, and its near-chance score is reported as evidence against projection-geometry effects. Figure 1 reports shallow encoder peaks, decoder peaks at L1–L2, and a visible pattern shift only under joint pixel-space dropout.Source: §4.3 · pp. 4–5, Table 1 · p. 5, Figure 1 · p. 5

5 Experiments

The experiments find that LiDAR-conditioned U-Net features encode probe-decodable 3D structure, organize modality streams differently across network depth, and respond distinctly to joint pixel-space dropout. Output analysis focuses quantitative evaluation on depth and uses qualitative evaluation for semantic and instance predictions.

5.1 Diffusion Features Encode 3D Relevant Information

  • 0.231 MIoU on semantic segmentation exceeds the Gaussian-noise control’s 0.035–0.038 across ablations and tasks, showing that the features contain task-relevant 3D information.The strongest probe result is selected independently by ablation, stage, and feature type; concatenating levels underperformed the best single-level probe.Source: §5.1 · pp. 5–6

5.2 Modality Stream Analysis

  • Encoder cross-modal similarity rises with depth, while decoder similarity reaches its minimum at level 2 before a small rebound, indicating local convergence near the bottleneck and decoder re-specialization.Depth–semantic encoder alignment rises from ∼0.19 at level 0 to ∼0.44 at level 3; decoder similarity is ∼0.2–0.28 at level 2.Source: §5.2 · pp. 6–7, Table 2 · p. 6

5.3 Feature Restructuring

  • Depth-, intensity-, and random-channel dropout produce probe-MIoU shifts within ±0.01 and cosine-similarity shifts within ±0.07, whereas joint pixel-space dropout restructures features and slightly reduces output quality.Joint dropout suppresses shallow-encoder probe quality, raises deeper-level probe quality, and reduces decoder cosine alignment across levels; the intensity-channel result is essentially a no-op.Source: §5.3 · pp. 7–8, Figure 3 · p. 7

5.4 Output Quality

  • Quantitative output evaluation is restricted to depth, while semantic and instance predictions are evaluated qualitatively because label alignment and instance post-processing introduce unrelated variance.The study treats depth metrics as a more direct, minimally confounded probe of geometric structure.Source: §5.4 · p. 8

5.4.1 Depth

  • Depth evaluation reports AbsRel, δ1 accuracy, and RMSE on LiDAR points after least-squares metric alignment of affine-invariant predictions.AbsRel measures absolute relative distance, δ1 counts predictions within the 1.25 ratio threshold, and RMSE reflects accuracy in sparse, noisy far-field regions; the protocol follows [35] [36] [60] [8] [58] and [20] [35].Source: §5.4.1 · p. 8, Table 3 · p. 8

5.4.2 Qualitative Results

  • Figure 4 visualizes level-2 decoder features by reducing per-point vectors to three principal components mapped to RGB; this level combines low cross-modal similarity with high probe MIoU.Source: §5.4.2 · p. 8

6 Discussion and Limitations

A single LiDAR-conditioned diffusion model trained on 2D foundation-model pseudo-labels learns non-trivial 3D structure with a layered cross-modal organization. The work extends diffusion-feature perception [64] [46] [27] to a multi-modal, 3D-conditioned setting while acknowledging limits in evaluation and output quality.

Discussion

  • Features progress from modality-specific shallow representations through a shared bottleneck to task-specialized decoder features, with task prompts producing visually distinct decoder structures.Figure 4 visualizes the prompt-dependent decoder feature structures on a representative nuScenes scene.Source: §6 · p. 8, Figure 4 · p. 9

Limitations

  • Experiments use only nuScenes and a single training run, leaving cross-dataset generalization and initialization sensitivity untested; taxonomy mismatch also constrains semantic evaluation.Semantic evaluation is limited by the mismatch between Cityscapes and nuScenes lidarseg taxonomies.Source: §6 · p. 8
  • Instance predictions are qualitative because their metrics depend strongly on orthogonal post-processing; outputs are not competitive with task-specific 3D networks and serve as feature probes.Figure 5 shows predicted semantic outputs and instance clusters in point-cloud space.Source: §6 · p. 9, Figure 5 · p. 9

Future Directions

  • Future work could compare other LiDAR-conditioned diffusion, 3D-aware transformer, and non-diffusion multi-task models, test lightweight 3D detection or panoptic heads, and add modalities such as flow or motion forecasting.These directions would broaden analysis of cross-modal structure and test whether lifted features are operationally useful.Source: §6 · p. 9

A Preliminaries · A.1 Latent Diffusion Models · A.2 LiDAR-Camera Projection

The preliminaries define latent diffusion as noise prediction and reconstruction in a compressed space, then specify how LiDAR points and image pixels are projected between camera and LiDAR frames.

A.1 Latent Diffusion Models

  • Diffusion models gradually add Gaussian noise to data through a forward process and learn a reverse process to reconstruct data from noise.The forward process is a fixed Markov chain.Source: §A.1 · p. 14
  • The forward diffusion process uses a predefined variance schedule, with αt = 1 − βt and cumulative ᾱt.The supplied passage gives the schedule definitions but not the complete forward-process equation.Source: §A.1 · p. 14
  • A conditioned network ϵθ(xt, t, c) is trained to predict noise added to xt.The passage specifies xt as a mixture of the data sample and Gaussian noise, weighted by the cumulative schedule.Source: §A.1 · p. 14
  • Latent diffusion [38] reduces computational cost by encoding inputs into z0, applying diffusion there, and decoding the latent to reconstruct the output.An encoder E maps x0 to z0, and a decoder D reconstructs the output.Source: §A.1 · p. 14

A.2 LiDAR-Camera Projection

  • A LiDAR point is transformed from homogeneous LiDAR coordinates into camera coordinates using the rigid transform TCL.The resulting Euclidean camera point is denoted pC = (X, Y, Z)⊤.Source: §A.2 · p. 14
  • Camera projection maps a 3D point to image coordinates using intrinsic matrix K, with depth λ equal to camera-frame Z.The passages define the projection π from R3 to R2.Source: §A.2 · p. 14
  • Backprojection uses a pixel and its depth to recover a 3D point, which is then transformed back into the LiDAR frame.The passages also name the composite point-to-pixel and pixel-and-depth-to-point mappings Π and Π−1.Source: §A.2 · p. 14

B Short-Probe Convergence

At the two narrowest U-Net levels, task-conditioned features show a large short-probe MIoU gap over matched Gaussian noise. The short probe is a fixed-budget training diagnostic, not a measure of generalization or representational quality.

B Short-Probe Convergence

  • The short probe trains for 30 steps on one sample of six views across all three modalities, logging scores every five steps, and is not a generalization test.It serves as a training-monitoring signal rather than a measure of representational quality.Source: B Short-Probe Convergence · p. 14
  • At encoder level 0 and decoder level 3, task-conditioned features reach ∼0.36–0.65 mIoU within 30 steps versus ∼0.08 for matched Gaussian noise.Both levels have 320 channels; the gap is consistent with feature separability beyond what the probe can synthesize from noise.Source: B Short-Probe Convergence · p. 14, Figure 6 · p. 15

C Implementation Details · C.1 Text Prompt Format

The prompt uses a fixed two-field format combining a task modality with a short scene description. For cross-modal cosine analysis, the scene description stays constant across three modality-conditioned passes while the task field changes.

C.1 Text Prompt Format

  • The textual prompt follows the fixed format “Task: <modality>. Scene: <description>.”The two fields specify the modality and a scene description.Source: §C.1 · p. 15
  • The task modality is selected from depth, semantic_segmentation, and instance.Source: §C.1 · p. 15
  • The scene description is a short caption derived from nuScenes scene metadata.Source: §C.1 · p. 15
  • For a given sample, the same scene description is used across all modalities.Source: §C.1 · p. 15
  • Cross-modal cosine analysis uses three modality-conditioned forward passes for each sample.Source: §C.1 · p. 15
  • Across those passes, only the Task field varies; the Scene field remains unchanged.Source: §C.1 · p. 15

C.2 Training Configuration · C.3 Probing Configuration

The configuration adapts pretrained Stable Diffusion 1.5 to LiDAR conditioning and trains it jointly across three task modalities. Probes assess per-point U-Net features without spatial coordinates, using distinct short- and long-range training protocols.

C.2 Training Configuration

  • The U-Net starts from Stable Diffusion 1.5 [38], with its input convolution expanded from 4 to 6 channels and the added weights zero-initialized [1].Images are processed at 224 × 400 resolution, producing 28 × 50 latents; LiDAR depth and intensity condition the model after max-pooling to latent resolution.Source: §C.2 · p. 15
  • Training uses AdamW [26] with learning rate 4×10−4, weight decay 0.01, cosine scheduling, 1000 warmup steps, and 20% dropout for each guidance and conditioning projection.The schedule has a minimum learning rate of 10−5.Source: §C.2 · p. 15
  • All three task modalities are trained jointly, with each optimizer step seeing all three prompts; training runs for 50 epochs on nuScenes [2].With per-GPU batch size 4, 6 views per sample, 3 modalities, and 8 NVIDIA A40 GPUs, the effective batch size is 576.Source: §C.2 · p. 15

C.3 Probing Configuration

  • Both probes are single linear layers trained with cross-entropy on pseudo-label class assignments, using only per-point backprojected features and excluding coordinates, pooling, and nonlinearities.Probe inputs are U-Net activations with no positional encoding or coordinate concatenation.Source: §C.3 · p. 15
  • The short-range probe trains for 30 steps on one six-view sample, while the long-range probe trains for 60 full-batch steps over a fixed 10-sample validation chunk.Scores are logged every 5 short-range steps; evaluation uses a disjoint 10-sample bucket from the unobserved validation set and mean intersection-over-union over evaluable classes. Both use AdamW with learning rate 10−2 and zero weight decay.Source: §C.3 · p. 15
  • The Gaussian-noise control samples matched-shape feature tensors from per-channel Gaussian distributions parameterized by each U-Net level’s mean and standard deviation.The control distribution is N(µc, σc).Source: §C.3 · p. 16

D Additional Qualitative Results

The appendix adds qualitative results for a further nuScenes validation scene, showing six-view 2D outputs, backprojected 3D predictions, decoder-feature PCA, and metric-aligned dense depth.

D Additional Qualitative Results

  • Figure 7 presents the model’s direct 2D outputs across all six camera views for the additional validation scene.Source: D Additional Qualitative Results · p. 16
  • Figure 8 compares backprojected semantic and instance predictions with ground truth; semantic predictions use Cityscapes labels, while instance correctness is indicated by object separation, not color matching.The ground-truth semantic colors are randomly assigned over nuScenes lidarseg classes because the taxonomies have no canonical color mapping; instance colors are arbitrary and unmatched.Source: D Additional Qualitative Results · p. 16, Figure 8 · p. 17
  • Figure 9 visualizes decoder level 2 features under each task prompt as RGB on the backprojected LiDAR point cloud.It uses the same prompt and feature level as Figure 4 in the main text.Source: D Additional Qualitative Results · p. 16, Figure 9 · p. 17
  • Figure 10 shows dense depth produced from the original sparse LiDAR scan and aligned to metric scale by least-squares regression against input LiDAR depth.Source: D Additional Qualitative Results · p. 16, Figure 10 · p. 17

Related papers

Loading 2609.10322v1…