PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Generate Realistic LiDAR Point Clouds

Vlas Zyrianov, Xiyue Zhu, Shenlong Wang

arXiv:2209.03954v2cs.CV

TL;DR

Realistic, scalable LiDAR simulation is desirable, but collecting real-world data for extreme scenarios can be costly, unsafe, or unethical. LiDARGen learns a score-based generative model and applies progressive denoising to generate point clouds, with posterior sampling for conditional generation. Evaluations report superior performance on KITTI-360 and NuScenes and demonstrate conditional densification potential.

  • Problem

    Realistic, scalable LiDAR simulation is desirable, while collecting real-world data for extreme scenarios can be costly, unsafe, or unethical.

    Source: §1 · p. 1

  • Method

    LiDARGen learns a score function for raw LiDAR point clouds and progressively denoises noisy inputs, with posterior sampling for conditional generation.

    Source: §1 · p. 2, §4 · p. 6, §6 · p. 14

  • Results

    LiDARGen reports superior performance over competing methods on KITTI-360 and NuScenes, and a four-subject AB test favored it in 89% of cases.

    Source: §1 · p. 2, §5.3 · p. 12

  • Takeaways & Limitations

    The unconditional model can be applied to conditional generation tasks, including LiDAR densification, through posterior sampling without retraining.

    Source: §1 · p. 2, §6 · p. 14

    Samples have degraded geometric detail at far range and tend to produce less straight walls than real samples; experienced LiDAR perception researchers can still distinguish them.

    Source: §5.5 · p. 14

Abstract

from arXiv · show

We present LiDARGen, a novel, effective, and controllable generative model that produces realistic LiDAR point cloud sensory readings. Our method leverages the powerful score-matching energy-based model and formulates the point cloud generation process as a stochastic denoising process in the equirectangular view. This model allows us to sample diverse and high-quality point cloud samples with guaranteed physical feasibility and controllability. We validate the effectiveness of our method on the challenging KITTI-360 and NuScenes datasets. The quantitative and qualitative results show that our approach produces more realistic samples than other generative models. Furthermore, LiDARGen can sample point clouds conditioned on inputs without retraining. We demonstrate that our proposed generative model could be directly used to densify LiDAR point clouds. Our code is available at: https://www.zyrianov.org/lidargen/

1 Introduction

Realistic, scalable LiDAR simulation is needed for affordable research and safer study of extreme scenarios, but existing approaches struggle to combine realism and scalability. LiDARGen uses score-based generation and posterior sampling to produce point clouds and condition them without retraining.

1 Introduction

  • Realistic, scalable LiDAR simulation is desirable because physical sensors are costly and real-world data collection for extreme scenarios can be expensive, unsafe, or unethical.Simulations can generate long-tail-event data for training and testing at low cost.Source: §1 · p. 1
  • Existing LiDAR data-generation approaches have not jointly achieved realism and scalability, while model-based simulation can be controllable and realistic but requires costly prior scans.Asset-free deep generative approaches had not yet achieved both controllability and realism.Source: §1 · pp. 1–2
  • LiDARGen learns a score function on equirectangular scan views and progressively denoises Gaussian noise with Langevin dynamics to generate physically feasible point clouds.The representation encodes ray angles, reflectance, and depth range.Source: §1 · p. 2
  • Posterior sampling supports conditional generation such as LiDAR densification, combining a learned prior with input conditions without retraining.The generated results are described as realistic and plausible with respect to the controlled input.Source: §1 · p. 2
  • On KITTI-360 and NuScenes, LiDARGen outperforms competing methods on multiple metrics and visual quality, and its densification results show potential for downstream tasks.The comparisons include both quantitative and qualitative evaluation.Source: §1 · p. 2

2 Related Work

The related work spans point-cloud generation, 3D representations, and LiDAR simulation. It highlights gaps in realism, variable-size generation, and scalable controllability that motivate LiDARGen’s sensing-based representation.

2 Related Work

  • The paper connects its work to point-cloud generation, energy-based generative models, and LiDAR simulation.Its overview describes sampling by progressively denoising an equirectangular view with a trained score function and Langevin dynamics.Source: §2 · pp. 2–3

2.1 Point Cloud Generation

  • Prior LiDAR point-cloud generators had limited realism, and many point-cloud methods assumed fixed-size data; LiDARGen instead masks range images to model ray drops and generate variable-size, physically feasible clouds.Some variable-size alternatives require resampling or an expensive autoregressive procedure.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 Deep Learning for Point Clouds

  • 3D learning representations include dense or sparse voxels, direct point-cloud networks, graph methods, and 2D projections such as bird’s-eye, perspective, and equirectangular views.Equirectangular views encode polar coordinates in a panoramic 2.5D image resembling LiDAR imaging.Source: §2.2 · p. 4

2.3 LiDAR Simulation

  • Physical LiDAR simulation uses raycasting but can lack realism without high-quality assets and can miss effects such as atmospheric noise or ray drop.Data-driven simulation and cut-and-paste augmentation use real scans or modeled environments, which can constrain scale, controllability, or physical plausibility.Source: §2.3 · pp. 4–5

3 Background

The background introduces energy-based models and score-based methods for learning distributions without directly computing their intractable normalization terms. It then describes denoising score matching and Langevin dynamics as tools for training and sampling.

3.1 Energy-based Models

  • An energy-based model assigns probability proportional to exp(-Eθ(x)) and normalizes it with Zθ, but evaluating the partition function or its gradient is computationally intractable in high-dimensional spaces.The energy function is real-valued and has learnable parameters.Source: §3.1 · p. 5

3.2 Score-based Energy Models

  • Score matching minimizes an objective whose derivatives with respect to the data avoid dependence on the partition function.The stated objective involves first- and second-order gradients with respect to x.Source: §3.2 · p. 6
  • Score-based models parameterize the gradient of log probability, avoiding the partition function in the score-matching objective; denoising score matching trains this function on noise-perturbed samples.The denoising objective uses a Gaussian perturbation with standard deviation σ.Source: §3.2 · p. 6
  • Langevin dynamics samples from a score-based model by iteratively combining the previous sample, a score-based gradient step, and Gaussian noise.The learning rate is typically annealed; under mild conditions, the process converges to samples from the target distribution as the step size approaches zero and iterations increase.Source: §3.2 · p. 6

4 Method

LiDARGen learns a score function over LiDAR point clouds and samples by progressively denoising in an equirectangular representation. Its generation and conditioning procedures use that representation and score-based sampling to produce physically feasible point clouds and support posterior sampling.

4 Method

  • LiDARGen learns the gradient of the point-cloud log probability and uses Langevin dynamics to denoise an initial random Gaussian point cloud.The equirectangular representation is used to maintain physical feasibility.Source: §4 · pp. 6–7

4.1 LiDAR Generation

  • Cartesian points are converted into spherical coordinates and rasterized as a dense equirectangular image with depth and intensity channels.Depth is normalized and logarithm-mapped so nearby points receive higher geometry resolution.Source: §4.1 · p. 7
  • The score network uses circular convolutions for the image’s connected left and right boundaries and angular coordinates to encode viewpoint-aware geometry.The network is a U-Net that maps a two-channel image to a same-sized score map.Source: §4.1 · p. 8
  • Training uses a noise-conditioned score network with a reweighted multi-scale loss, while sampling anneals noise levels before unprojecting the final range image into 3D points.Sampling starts at the highest pretrained noise level and gradually reduces it; the final Langevin step produces a clean equirectangular range image.Source: §4.1 · pp. 8–9

4.2 Posterior Sampling

  • Posterior sampling combines the pretrained unconditional prior with a condition likelihood gradient, allowing conditional point-cloud generation without training a separate model.The condition and point cloud define a likelihood, and Bayes’ rule gives the posterior score used for sampling.Source: §4.2 · p. 9
  • For densification, a visibility mask identifies observed low-beam rays, and posterior sampling steers generated points toward both realistic structure and consistency with those observations.The target is a high-beam point cloud recovered from a low-beam input.Source: §4.2 · p. 9

5 Experiments

Experiments evaluate LiDARGen on KITTI-360 and nuScenes using quantitative metrics, visual comparisons, and human judgments. The results support realistic generation and input-conditioned densification, while sampling speed and residual artifacts remain limitations.

5.1 Experimental Setup

  • Experiments use KITTI-360 and nuScenes, whose datasets differ in sensor beam count, geography, and scene content, and assess generation with MMD, FRD, and JSD.KITTI-360 has 81,106 scans; nuScenes has 297,737 training sweeps and 52,423 testing and cross-validation sweeps.Source: §5.1 · pp. 10–11

5.2 KITTI-360 Evaluation

  • On KITTI-360, LiDARGen outperforms competing methods in FRD, ranks high in MMD, and is favored in a significant human study; nuScenes judges favor it in 89% of cases.The authors caution that no single metric fully captures sampling quality and recommend considering metrics, human study, and qualitative results together.Source: §5.2 · p. 12, §5.3 · p. 12

5.3 NuScenes Results

  • On nuScenes, LiDARGen is significantly favored over competing methods in 89% of human-study cases, but its BEV MMD is worse than VAE and Projected GAN.The reported BEV MMD values are 2e-3 for LiDARGen, 1.1e-3 for VAE, and 6e-5 for Projected GAN; the authors note a tendency to concentrate point-cloud mass nearer the viewpoint.Source: §5.3 · pp. 12–13

5.4 Posterior Sampling

  • For KITTI-360 densification, LiDARGen’s results are reported as superior to PUNet and interpolation approaches, while posterior samples reflect sparse-input guidance.The qualitative posterior-sampling example uses simulated 4-beam and 16-beam inputs selected from raw 64-beam scans.Source: §5.4 · p. 13, Table 3 · p. 13
  • On densified point clouds, RangeNet++ improves over nearest-neighbor up-sampling without fine-tuning, with per-point accuracy 0.608 versus 0.546 and IoU 0.449 versus 0.394.The reported qualitative examples show cleaner road segmentation and recovery of distant-car details.Source: §5.4 · p. 13, Figure 7 · p. 14

5.5 Discussions

  • Ablations report that circular convolutions and coordinate encoding both improve FRD and MMD.The comparison uses the same score-function model with and without those design components.Source: §5.5 · p. 14
  • Sampling efficiency remains a major limitation: LiDARGen takes approximately 12 minutes to generate a batch of 36 samples.The authors identify diffusion-model acceleration techniques as a future direction for alleviating this issue.Source: §5.5 · p. 14
  • The samples do not yet pass experienced LiDAR researchers’ Turing test and can lose far-range geometric detail or produce less-straight walls.The authors identify hybrid equirectangular-view and bird’s-eye-view networks as a future direction.Source: §5.5 · p. 14

6 Conclusion

The conclusion presents LiDARGen as a score-based LiDAR point-cloud generator that progressively denoises noisy inputs and supports conditional generation through posterior sampling.

6 Conclusion

  • LiDARGen uses a score-based approach to generate LiDAR point clouds.The method samples realistic point clouds by progressively denoising a noisy input.Source: §6 · p. 14
  • Posterior sampling lets the unconditional model handle conditional generation tasks.The paper reports a human study and perceptual-similarity evaluation on KITTI-360 as validation.Source: §6 · p. 14
  • The authors hope the approach will broaden access to realistic LiDAR sensory data and expect to explore 3D environment generation and self-driving applications.Source: §6 · p. 14

Clouds – Supplementary Material

The supplementary material reports additional results and ablations, including NuScenes experiments and posterior-sampling results.

Clouds – Supplementary Material

1 Additional Analysis

The additional analysis examines convolution and coordinate-encoding choices and compares LiDARGen with an adapted ShapeGF baseline.

1 Additional Analysis

  • Circular convolutions fix the horizontal discontinuity caused by ordinary convolutions lacking a receptive field across the range image’s left and right edges.The discontinuity is described as starting from the origin.Source: §1 · p. 20
  • Coordinate encoding produces straighter road layouts that better reflect real-world urban-driving layout distributions.Source: §1 · p. 20
  • ShapeGF showed strong mode collapse on urban driving scenes and could not provide physically feasible results like equirectangular-based approaches.The authors adapted ShapeGF for LiDAR generation and trained it from scratch on KITTI.Source: §1 · p. 21, Figure 1 · p. 21

2 nuScenes

On NuScenes, LiDARGen is favored in a human comparison, while the experiments use a different sensor and settings from KITTI and note a possible noise-parameter issue.

2 nuScenes

  • NuScenes uses 32-beam HDL32E scans from Boston and Singapore, unlike the 64-beam HDL-64E data collected mostly in Karlsruhe for KITTI.The NuScenes sensor has lower vertical resolution but a higher vertical field of view than the KITTI sensor.Source: §2 · p. 21
  • The NuScenes range image is 32×1024, and the encoding is adjusted to normalize the full range to [0, 1].Source: §2 · p. 21
  • The ProjectedGAN and LiDAR VAE are NuScenes baselines, but the GAN model described in Caccia et al. did not converge after hyperparameter tuning and was omitted.Source: §2 · p. 21
  • 89% of human AB-test cases favored LiDARGen over competing algorithms in the NuScenes comparison.The comparison included VAE and ProjectedGAN qualitative results.Source: §2 · pp. 21–22, Figure 3 · p. 23
  • The authors reuse KITTI’s starting and ending noise levels for NuScenes and suggest they may be too large; better-tuned parameters may resolve the issue.The reported noise levels are 150 and 0.01.Source: §2 · p. 23

3 Additional Posterior Sampling Results

Additional posterior samples show diverse, realistic LiDAR point clouds that remain consistent with the sparse input. Multiple plausible outputs from the same input illustrate the probabilistic nature of the approach.

3 Additional Posterior Sampling Results

  • Figs. 5 and 6 show diversified point clouds that are highly realistic and have high fidelity and consistency to the input.Source: §3 · p. 23, Figure 5 · p. 25, Figure 6 · p. 26
  • Fig. 4 shows diverse car shapes and intensity values, as well as wall structure, across multiple samples conditioned on the same sparse input.Source: Figure 4 · p. 24
  • Multiple plausible point clouds from the same sparse input further demonstrate the advantage of treating densification probabilistically.Source: §3 · p. 24

Related papers

Loading 2209.03954v2…