CRISP: Fixing Flying Pixels in Latent LiDAR Generation via Diffusion Decoding

1 Introduction

• Convolutional VAE decoders smooth sharp depth contours, so interpolated edge depths back-project into floating points that can propagate into downstream perception.LiDM uses flat convolutional kernels to limit this leakage, but its convolutions still cross radial depth discontinuities.Source: §1 · p. 2
• The paper asks whether the decoder itself is a bottleneck in latent LiDAR reconstruction and whether replacing it alone can improve results.The question is motivated by sparse, anisotropic range maps with explicit missing returns.Source: §1 · p. 2
• CRISP is a plug-in pixel-space diffusion decoder with a backbone-agnostic latent adapter, DiT denoiser, and support mask predictor for sparse valid returns.It keeps the encoder and latent generator fixed while replacing the decoder.Source: §1 · p. 2
• 71%/74% reductions in FSVD/FPVD for generic video VAEs and a 71% reduction in FRID for LiDM follow decoder-only replacement under frozen encoders.Inside a frozen LiDM world model, the zero-shot swap improves FSVD by 15.5% and FRID by 4.0%.Source: §1 · p. 2

2 Related Work

• LiDAR generation progressed from simulated assets and structured point maps to range-view and latent generative models, including LiDARGen [70], UltraLiDAR, R2DM [38], and RangeLDM [17].This progression motivates examining the decoder in pipelines built on compressed range-map latents.Source: §2 · pp. 2–3
• Recent LiDAR generators improve latents, conditioning, or pretrained priors while retaining convolutional decoders; replacing only LiDM’s decoder lowers edge CD by 48.8% and FRID by 71.4%.The passage distinguishes LiDM, SG-LDM, R3DPA, and UniDriveDreamer as examples of the prior direction.Source: §2 · p. 3
• Post-hoc 3D residual correction and 4D world generation address geometry beyond decoding, whereas CRISP replaces the inherited or LiDAR-native convolutional decoder itself.The cited approaches are L3DR [31] and LidarDM [71].Source: §2 · p. 3
• Pixel-Perfect Depth [59] diagnoses latent decoding as a source of lost edge sharpness, while CRISP conditions on a frozen latent and adds a support branch instead of using clean RGB guidance.Ablating CRISP’s first two differences degrades results (Table 14).Source: §2 · p. 3
• LiDAR’s sparse, anisotropic range maps and explicit missing returns make decoder smoothing especially damaging after back-projection.The passage contrasts LiDAR with monocular depth and cites [35].Source: §2 · p. 3
• Unlike image diffusion decoders that can sample stochastically, CRISP regresses depth on valid pixels and keeps both encoder and latent generator frozen for a zero-shot LiDM swap.The passage contrasts PiD [32] and ϵ-VAE [65], and notes that RAE [66] instead requires a new decoder and retrained generator.Source: §2 · p. 3

3 Method

• CRISP replaces convolutional VAE decoders with a two-branch pixel-space reconstruction module: a latent-conditioned DiT predicts dense depth, then a support predictor produces the sparse LiDAR output.The mask composes with dense depth to retain valid returns.Source: §3 · p. 3

3.1 Encoding Setup

• The setup encodes a log-normalised range map into a latent tensor using a pretrained encoder.The representation allocates more resolution to near-range structure.Source: §3.1 · p. 4

3.2 Latent Adapter

• A latent adapter projects and tokenizes heterogeneous backbone latents into a common conditioning sequence for the DiT.The channel projection and token-count reduction separate the decoder from any one backbone’s latent geometry.Source: §3.2 · p. 4

3.3 DiT-Powered Pixel Diffusion Decoder

• The DiT denoises a Gaussian noise sample conditioned on latent tokens, fusing those tokens early and again at the network midpoint.Early conditioning helps the denoiser recover global scene layout before refining local geometry.Source: §3.3 · p. 4

3.4 Support Mask Predictor

• A latent-conditioned support branch predicts valid-return support from the frozen latent and dense depth, then sparsifies the decoded range map.Ambiguous structures such as vegetation and thin objects can yield stochastic ray-drop variation.Source: §3.4 · p. 5

3.5 Geometry-Aware Diffusion Objective

• The depth objective combines valid-pixel velocity regression with derivative and gradient regularisers, plus unweighted velocity and low-noise depth terms.Importance weighting emphasizes depth discontinuities while validity-normalised filtering avoids spurious responses at valid/void borders.Source: §3.5 · p. 5

3.6 Support Mask Objective

• The support objective addresses coarse mask structure, sharp boundaries, thin returns, and isolated ray-drop holes with complementary losses and training signals.The passage describes focal BCE, Dice, edge and Laplacian terms, hard-pixel mining, and deep supervision.Source: §3.6 · pp. 5–6

3.7 Full Objective and Encoder Adaptation

• Default training freezes the encoder and trains only decoder branches; encoder-adapted variants add a KL regulariser to keep the encoder posterior near the prior.The passage evaluates adapted SVD and Wan2.1 variants, excluding LiDM because its encoder is LiDAR-native.Source: §3.7 · p. 6

• Experiments span KITTI-360 and SemanticKITTI as KITTI-family benchmarks, plus lower-vertical-resolution nuScenes.Source: §4 · p. 6
• CRISP is tested with LiDM on KITTI-family and SVD/Wan2.1 on nuScenes, including frozen replacement, encoder adaptation, and LiDM world-model decoding.Source: §4 · p. 6
• Evaluation covers FRID/FSVD/FPVD perception, JSD/EMD/MMD distributional fidelity, and CD/F-score@0.5m paired 3D geometry.Source: §4 · p. 6
• Table 1 reports five-run means and standard deviations; FRID is limited to the 64-beam KITTI-family LiDM setting.Source: Table 1 · p. 7
• Boundary analysis splits valid returns by local-depth discontinuity threshold 0.1 into edge and smooth subsets, excluding invalid/empty transitions.Source: §4 · p. 7, Table 2 · p. 7

Plug-and-play world-model decoding

• With fixed world-model latents, zero-shot CRISP lowers FSVD 15.5%, FRID 4.0%, whole-cloud CD 2.2%, and edge CD 17.5%; edge F-score falls 20% and JSD rises marginally.The passage attributes the edge-F-score decrease to latent dynamics optimized with the original decoder; the decoder swap requires no retraining.Source: Table 3 · p. 8, §5 · pp. 8–9

Frozen backbones

• Across frozen backbones, FSVD/FPVD fall 53%/48% on average; generic video VAEs reach 71%/74%, while LiDM FRID drops 71% (2.24→0.64).Whole-cloud CD decreases 34% on average: 46% for generic VAEs and 11% for LiDM.Source: §5 · p. 8

Geometric breakdown

• Edge F-score rises 107% across five settings versus 9% for the whole cloud; edge CD falls in all settings, while smooth-region CD drops 23% on average.The smooth-region decrease has only a negligible Wan ×1 exception.Source: §5 · p. 8

Encoder-adapted backbones

• Wan ×1 achieves the study’s best perceptual scores with FSVD/FPVD reductions of 65%/70%, while SVD ×1 improves geometry but regresses on both perceptual metrics.For SVD ×1, whole-cloud CD falls 18% and edge F-score rises 81%.Source: §5 · p. 8

Qualitative analysis

• Figure 3 shows lost narrow objects, incorrectly filled voids, and flying pixels as target failure modes; CRISP suppresses boundary flying pixels, void infilling, and LiDM depth-gap bridging.The qualitative examples are SVD ×1, Wan ×1, and LiDM, respectively.Source: Figure 3 · p. 9, §5 · p. 9

Capacity and inference cost

• CRISP is 8.1×, 7.0×, and 60.0× larger than the SVD, Wan, and LiDM decoders, yet Wan2.1 throughput rises from 57 to 86 fps.Five-step decoding takes 3.9 ms/frame for SVD and 23 ms/frame for LiDM.Source: §5 · p. 9, Table 4 · p. 9

Ablations and per-component attribution

• Removing early fusion raises whole-cloud CD 15% and FSVD 21%; explicit support masking raises edge F-score 23% and JSD 35%, while the decoder swap supplies 68–98% of SVD ×3 gains.The mask branch accounts for the remaining gain, mainly on JSD and edge F-score.Source: §5 · pp. 9–10, Table 5 · p. 10

Faithfulness and limitations

• Four measurements support that CRISP reads boundary geometry from the latent, but predictor precision is 0.862; residual smooth-surface noise and possible thin-structure flicker remain limitations.The decoder predicts a single depth channel and does not model remission.Source: §5 · p. 10

6 Conclusions

• Edge F-score improves by 107% on average, and edge CD decreases by 17.5% in a zero-shot swap inside a frozen LiDM world model, without retraining the latent generator.The conclusion presents CRISP as a plug-in upgrade path for range-map latent LiDAR systems.Source: §6 · p. 10
• Future work includes remission prediction with multi-channel diffusion, density-aware support calibration, and temporal conditioning for flicker-free rollouts.Source: §6 · p. 10

A Extended Related Work

• Unlike Pixel-Perfect Depth [59], CRISP handles genuine missing returns without clean RGB guidance, while unlike RAE [66], it keeps the encoder and pretrained latent generator frozen.This enables a plug-in decoder swap inside an already-trained LiDM world model.Source: A Extended Related Work · pp. 15–16

B.1 DiT Decoder

• The DiT decoder uses 24 transformer blocks, hidden dimension 1024, and 16 attention heads; its blocks start as identity maps through zero-initialized adaLN modulation projections.Source: §B.1 · p. 16
• The denoiser processes coarse tokens before pixel-shuffling to finer tokens, while latent tokens enter both before the first block and at the midpoint.The coarse-first progression establishes global layout before refining local geometry; early fusion uses a lightweight MLP, while mid-level fusion uses a wider MLP.Source: §B.1 · p. 16

B.2 Support Mask Predictor

• The support predictor combines latent features with a decoded-depth hint before a three-level U-Net predicts valid-return support.The latent branch supplies semantic layout and the depth-hint branch supplies high-frequency geometric structure.Source: §B.2 · p. 16

B.3 Geometry-Aware Diffusion Objective

• The base velocity supervision is validity-masked regression, with ℓ2 used until validation depth metrics plateau and ℓ1 thereafter; every valid pixel contributes equally.Source: §B.3 · p. 17
• Directional first-order terms preserve local slopes, while second-order terms suppress denoising waviness, using only neighborhoods whose involved pixels are valid.The horizontal and vertical operators reflect the range map’s distinct beam-sweep and elevation axes.Source: §B.3 · p. 17
• The low-noise reconstruction term penalizes the denoised depth estimate against the target, with timestep weighting that emphasizes low-noise steps.Source: §B.3 · p. 17
• The boundary-aware velocity loss substitutes importance weights for the validity mask in the base velocity objective.Source: §B.3 · p. 17
• Lgrad is the only term small on both datasets and is dropped from the reduced depth objective; every other surviving term matters on at least one dataset.Source: §B.3 · p. 17, Table 6 · p. 17
• Lx0 matters on nuScenes but less on KITTI-family, while Lder shows the reverse, consistent with sharper radial discontinuities at 64 beams.Dropping either term would silently specialize the objective to one beam count.Source: §B.3 · p. 17, Table 6 · p. 17

B.4 Mask loss weights.

• The mask objective combines six terms with specified weights.Source: §B.4 · p. 18
• Focal BCE and soft Dice are load-bearing on nuScenes but nearly unresolvable on KITTI-family, where support prediction on the LiDAR-native backbone is already near ceiling.Source: §B.4 · p. 18, Table 8 · p. 18
• Focal + Dice + Laplacian retain 96.7% of the full objective’s IoU gain over focal-only supervision and outperform the Focal + Dice + Edge combination.The minimal subset uses three of the six terms.Source: Table 9 · p. 18, §B.4 · p. 18

B.5 Objective Design: Provenance, Balancing, and Compute Cost

• The frozen objective has eleven terms—five depth and six mask terms—while encoder-adapted variants add a KL term; counting fourteen unpacks the four directional derivative components separately.Those components represent two physical axes and two difference orders, not four independently motivated objectives.Source: §B.5 · pp. 18–19
• Table 10 organizes every depth and mask loss term by failure mode, operator provenance, weight, and whether it remains in the reduced objective.Source: §B.5 · p. 18, Table 10 · p. 19
• Per-term weights are set once per training stage by equalizing initial magnitudes and lightly adjusting with a static GradNorm form, without per-backbone or beam-count retuning.Derivative-loss weights rise 8–17× between Stages F1 and F2; matched-reference validation checkpoints improved monotonically.Source: §B.5 · p. 19
• Adding the full eleven-term objective to the minimal two-branch configuration costs 7.81 ms, or 3.0% of a step, with effectively identical peak memory.The seven-term reduced objective saves only 5.36 ms, or 2% of a step, versus the full objective; the mask branch adds 43.03 ms, or 14.6% of decoder latency at inference.Source: §B.5 · p. 19

C.1 Data Preparation

• KITTI-family scans use 64×1024 range images with valid depths of 1–56 m, while nuScenes uses 32×1024 images with valid depths of 2–45 m.KITTI training uses 4,376 25-frame clips augmented fivefold; nuScenes training uses 19,088 clips, with the test set reserved for evaluation.Source: §C.1 · pp. 19–20

C.2 Frozen Backbones.

• For frozen SVD and Wan2.1, CRISP replaces only the decoder and predicts a single-channel range map from the unchanged frozen latent.The input range image is replicated across three channels for encoding, and no other backbone modifications are made.Source: §C.2 · p. 20

C.3 Encoder Adaptation.

• The adapted SVD and Wan2.1 baselines convert their pretrained VAEs to single-channel LiDAR and fine-tune them; Wan’s temporal convolutions remain frozen.SVD uses single-frame processing and averaged input/output weights for its one-channel initialization; updating Wan’s temporal convolutions caused collapse within a handful of iterations.Source: §C.3 · pp. 20–21

C.4 Temporal Compression Bypass.

• Framewise Wan evaluation avoids degradation from joint temporal encoding: Wan ×3/×1 CD rises from 0.770/0.309 at t=0 to 0.806/0.377 at t=5.The comparison uses 25 separate single-frame latents rather than one temporally compressed video tensor.Source: §C.4 · p. 21, Table 12 · p. 21

D Training Protocol

• Frozen-encoder variants train the DiT first, then add the support-mask predictor and switch to L1 with stronger gradient and derivative losses.Encoder-adapted variants train encoder and DiT jointly, add the mask branch, then freeze the encoder for final joint fine-tuning.Source: D Training Protocol · pp. 21–22, Table 13 · p. 22
• Importance weighting emphasizes geometrically complex regions, while encoder learning rates are zero for frozen variants and stop after Stage A3 for adapted variants.Mask gradients are scaled by 0.1 through the shared encoder during Stage A2.Source: D Training Protocol · p. 22
• Models train for up to 100 epochs on 8×A100-80GB GPUs, treating each frame independently without temporal attention, recurrent state, or temporal loss.A full CRISP run takes approximately 36 hours on 8×A100-80GB GPUs, or roughly 288 GPU-hours.Source: D Training Protocol · p. 22
• Evaluation computes point-cloud metrics at native metre scale, and Fréchet metrics use every unique test frame with feature extractors specified by metric.Results average five inference runs differing only in diffusion and Bernoulli support-sampling seeds.Source: D Training Protocol · pp. 22–23
• Across five reconstruction settings, edge F-score seed-to-seed standard deviation is at most 0.0002; in the world-model setting, CRISP and baseline spreads are 0.0014 and 0.0017.The passage attributes the world-model spread increase to sampled latents rather than CRISP’s own sampling.Source: D Training Protocol · p. 23

E Ablations

• Five Euler steps are the chosen operating point: FSVD and FPVD drop most sharply between steps 4 and 5, while 6 steps adds latency for marginal improvement.The step ablation reports 61.2→40.3 FSVD and 55.4→36.9 FPVD between 4 and 5 steps; the figure plots metric groups against sampling steps.Source: E Ablations · p. 23, Figure 4 · p. 24
• Removing early latent fusion raises whole-cloud CD from 0.985 to 1.134 (+15%) and FSVD from 98.6 to 119.6 (+21%) in the SVD ×3 comparison.The decoder starts from pure Gaussian noise, and the authors report that early latent injection establishes scene layout before later blocks.Source: E Ablations · pp. 23–24, Table 14 · p. 25
• Adding the support-mask branch reduces JSD by 35% (0.109→0.071) and improves edge F-score by 23% (0.193→0.238).FPVD is the one reported metric where the no-mask variant is better: 35.993 versus 36.782.Source: E Ablations · p. 24, Table 15 · p. 25
• Adding Dice loss to Focal alone raises mask IoU from 0.758 to 0.827 (+0.069); the full deep-supervision configuration achieves 0.844.The reported 0.844 is marginally below Stage 5’s 0.845, a difference described as within single-run noise.Source: E Ablations · pp. 24–25, Table 15 · p. 25
• Removing latent conditioning drops converged mask IoU from 0.788 to 0.630 (−20%) and F1 from 0.881 to 0.773.The passage says latent semantics help distinguish open voids from sensor shadows or sparse-return regions.Source: E Ablations · p. 25, Table 15 · p. 25
• On LiDM, the same central design choices transfer, but near-ceiling support IoU leaves little room for mask-loss gains; 64-beam geometry is confounded with backbone type.The six-way leave-one-out changes LiDM support IoU by no more than 0.0011, and importance-weighting removal costs 6.0% val L1 on LiDM/KITTI-family versus 25.2% on nuScenes.Source: E Ablations · pp. 25–26, Table 16 · p. 26
• Wan ×3 decoding takes 3.84 ms/frame versus 9.71 ms for the inherited decoder, and its end-to-end pipeline reaches 86 fps versus 57 fps.CRISP adds 23.06 ms/frame on LiDM and 3.92 ms/frame on SVD; both remain real-time capable at 41 and 215 fps, respectively.Source: E Ablations · p. 26, Table 17 · p. 26
• CRISP uses approximately 516M parameters across settings; on LiDM, reducing sampling from five steps to one raises val L1 from 0.0446 to 0.0643.The capacity comparison reports 60× parameters and 22.8× FLOPs for LiDM, while the FSVD gain is comparatively small and the passage notes limited headroom as a confound.Source: E Ablations · p. 26

G Extended Qualitative Results

• In frozen SVD ×3 nuScenes samples, CRISP restores continuous scan-line geometry at vehicle and pole boundaries where the baseline produces scatter.The figure’s inset highlights boundary-bridging artifacts in the baseline reconstruction.Source: G Extended Qualitative Results · p. 27, Figure 5 · p. 28
• In encoder-adapted SVD ×1 samples, CRISP localizes the tree-trunk cluster and sharpens the foreground-background depth contour.The baseline smears points outward from the mid-range object cluster.Source: G Extended Qualitative Results · p. 27, Figure 6 · p. 29
• In frozen Wan ×3 samples, CRISP retains a foreground-background depth gap as a sharp void instead of filling between surfaces.The figure describes the retained gap as consistent with the ground-truth scan.Source: G Extended Qualitative Results · p. 27, Figure 7 · p. 30
• In adapted Wan ×1 samples, CRISP preserves the open void as a sparse valid-return map, while the baseline inserts spurious returns.The figure identifies the void in the ground truth and inset.Source: G Extended Qualitative Results · p. 27, Figure 8 · p. 31
• In frozen LiDM KITTI-family samples, CRISP reproduces the sharp 64-beam depth gap that the baseline bridges between distinct surfaces.The comparison includes the ground-truth scan for the highlighted region.Source: G Extended Qualitative Results · p. 27, Figure 9 · p. 32
• In LiDM world-model samples, both decoders receive the same generated latent, and CRISP produces sharper contours without retraining the generator.The baseline shows grid-aligned blocky artifacts and smeared foreground boundaries; no ground truth is shown.Source: Figure 10 · p. 33

Limitations

• The authors identify single-depth-channel operation, inherited range-map assumptions, smooth-surface noise, thin-structure flicker, and 3.9–23 ms/frame inference overhead as limitations.Distillation is noted as a possible mitigation for inference overhead.Source: NeurIPS Paper Checklist · p. 34

Theory and asset licenses

• The paper presents no formal theoretical results requiring proofs, and reports that licenses for existing assets are provided in Appendix F.The authors state that the datasets and pretrained models used are publicly available and cited.Source: NeurIPS Paper Checklist · pp. 35, 39

Reproducibility and compute

• The paper reports method and training details, and mean and standard deviation over five stochastic sampling runs from fixed checkpoints.Variability reflects diffusion and support-sampling randomness, not retraining or training-seed variability; compute reporting includes 8×A100-80GB training and about 288 GPU-hours per run.Source: NeurIPS Paper Checklist · pp. 35, 37

Open access and assets

• At review, code and checkpoints are absent, although datasets and pretrained backbones are public; code, scripts, and CRISP checkpoints are promised for camera-ready if accepted.The planned asset documentation is to cover training data, intended use, limitations, licenses, dependencies, and reproduction instructions.Source: NeurIPS Paper Checklist · pp. 36, 39

Societal impacts

• The paper identifies improved synthetic LiDAR fidelity for autonomous-driving simulation as a potential benefit and deceptive sensor traces or misleading safety audits as potential harms.It notes that CRISP is a decoder component rather than a standalone broad generative model, while acknowledging that misuse risks remain relevant.Source: NeurIPS Paper Checklist · p. 38

Ethics and safeguards

• The authors consider dedicated release safeguards beyond licensing, documentation, and responsible-use notes inapplicable because CRISP is a domain-specific automotive LiDAR decoder.The work uses publicly licensed automotive datasets and pretrained models; it involves no human subjects, crowdsourcing, or participant studies.Source: NeurIPS Paper Checklist · pp. 38–40

NeurIPS Paper Checklist

• The checklist marks LLM involvement [N/A]: LLMs are not part of the core method, experiments, or scientific claims, and writing or editing assistance alone does not require declaration.The guidance defines [N/A] as indicating that core method development does not involve LLMs as important, original, or non-standard components.Source: NeurIPS Paper Checklist · p. 41