PaperlayerView exact revision on arXiv

Source-linked AI summary

LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance

Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang, Qifeng Li, Junjie Li, Zirui Wang, Yukang Feng, Yu Hong, Shaofeng Zhang, Wenlong Liao, Tao He, Juyong Zhang, Junchi Yan

arXiv:2610.19786v1cs.CVcs.AI

TL;DR

Generating LiDAR from multi-view images could support data synthesis and perception, but existing image-conditioned approaches often lack geometric alignment or rely on monocular inputs. LiFi combines geometric and semantic guidance in a latent diffusion framework, and the authors report high-fidelity generation with some downstream-task value.

  • Problem

    Image-to-LiDAR generation remains under-explored, and existing methods often lack explicit geometric alignment or comprehensive spatial awareness from multi-view inputs.

    Source: §1 · p. 2

  • Method

    LiFi guides latent diffusion with decoupled geometry and semantic branches, using depth and uncertainty-aware geometric encoding alongside cross-view semantic aggregation.

    Source: §1 · p. 2

  • Results

    LiFi generates high-fidelity LiDAR scenes, and experiments report superior performance with geometric consistency and semantic alignment.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The generated LiDAR has certain value for some downstream autonomous-driving tasks.

    Source: §5 · p. 10

    The downstream detection evaluation does not directly demonstrate that generated scenes independently support perception comparable to real LiDAR.

    Source: §1 · p. 3

Abstract

from arXiv · show

Generating LiDAR point clouds from multi-view camera images is a valuable task with applications in controllable data synthesis, cross-modal simulation and various perception tasks. However, most existing methods are not specifically designed for this task, and the quality of their generated LiDAR remains limited. Moreover, these methods typically focus only on semantic cues in images or generate LiDAR from a monocular camera image. In this work, we introduce LiFi (LiDAR From images), a dedicated framework for generating visually and geometrically aligned LiDAR scenes from multi-view camera images. LiFi guides the latent diffusion process through two complementary branches: geometry and semantics. The geometry branch constructs scene features from images through a reverse sampling algorithm and an uncertainty-aware geometric encoder combined with DepthAnything3, whereas the semantic branch provides high-level semantic representations via a cross-view interaction module. We further introduce a dual-branch balanced classifier-free guidance (CFG) strategy, which enhances the model's conditional generation capability while preserving the independent learning and collaborative controllability of the two branches. Extensive experiments demonstrate that LiFi outperforms state-of-the-art methods in generating high-fidelity LiDAR scenes. The code will be publicly available at: https://github.com/szzhou88/LiFi.

1 Introduction

LiFi addresses the challenge of generating geometrically aligned LiDAR from multi-view images with a dedicated, dual-branch framework. The authors report high-fidelity generation and downstream-task value, while noting that detection results do not establish perception parity with real LiDAR.

1 Introduction

  • Existing image-conditioned methods focus on semantic cues or monocular inputs, while image–LiDAR differences make accurate cross-modal correspondence difficult.The paper also notes that acquiring large-scale, high-quality LiDAR data can be expensive.Source: §1 · pp. 1–2
  • The geometry branch aligns depth, confidence, and appearance conditions to the LiDAR scan pattern, while cross-view attention aggregates global semantic context.DepthAnything3 supplies per-pixel depth and uncertainty maps, and an uncertainty-modulated encoder adjusts feature expressiveness to local depth confidence.Source: §1 · p. 2
  • Balanced classifier-free guidance trains with full, single-branch, and unconditional settings to strengthen conditional guidance and collaborative learning.The method is evaluated on nuScenes for generation and downstream detection, and compared with representative baselines on KITTI-360.Source: §1 · pp. 2–3
  • LiFi demonstrates superior performance in generating high-fidelity LiDAR scenes with geometric consistency and semantic alignment.Downstream detection results provide evidence of the framework’s effectiveness, but do not directly show that generated scenes independently support perception comparable to real LiDAR.Source: §1 · pp. 2–3
  • LiFi is presented as the first dedicated framework for generating high-fidelity, geometrically and visually aligned LiDAR scenes from multi-view images.It guides latent diffusion through decoupled geometry and semantic branches.Source: §1 · pp. 2–3

2 Related Work

Prior work spans general generative models, LiDAR-specific generation, and the less-developed camera-to-LiDAR setting. The cited camera-to-LiDAR approaches leave gaps in explicit modality alignment or in support for image-only input.

2 Related Work

  • Generative-model research ranges from GANs and VAEs to autoregressive and diffusion models, with latent diffusion moving the process into latent space for greater efficiency.The passage cites GANs, VAEs [23], diffusion models [47], and latent diffusion models.Source: §2 · p. 3
  • LiDAR generation has used range images, voxelization, score-based and diffusion models, occupancy representations, and autoregressive frameworks.Examples include range-image generation, LiDARGen, UltraLiDAR [52], R2DM, RangeLDM, and UniScene.Source: §2 · p. 3
  • Camera-to-LiDAR research remains limited: LiDM does not investigate modality alignment, while X-Drive [51] is not designed specifically for image-only input and uses additional prompts and box information.HoloDrive [50] and Genesis [10] do not provide a camera-to-LiDAR mode.Source: §2 · p. 3

3 Method

LiFi represents LiDAR scenes as range images and conditions latent diffusion on complementary geometric and semantic information from multi-view cameras. Its method aligns camera-derived visual priors to LiDAR sampling, encodes uncertainty and cross-view context, and balances the two branches during training.

3.1 Preliminaries

  • Range images preserve LiDAR’s angular sampling topology in a regular 2D representation that can be converted back to a 3D point cloud.The latent diffusion model compresses range images with a VAE and performs diffusion in the latent space using a UNet.Source: §3.1 · p. 4

3.2 Overall Framework

  • LiFi extracts multi-view depth, uncertainty, and semantic priors, then uses decoupled geometry and semantic branches to condition LiDAR generation.DepthAnything3 is given as an example of a model that jointly processes views and outputs confidence maps.Source: §3.2 · p. 5

3.3.1 Range-View-Centered Reverse Sampling

  • Range-view-centered reverse sampling projects LiDAR-distributed query points into camera views, avoiding direct projection of dense depth maps onto sparse LiDAR scenes.Valid projected points are sampled for depth, uncertainty, and RGB; out-of-view points receive null values, while overlaps use the nearest-depth criterion.Source: §3.3.1 · pp. 5–6

3.3.2 Geometry-Aware Condition Encoding with Uncertainty Modulation

  • The geometry-aware encoder fuses range-view coordinates, aligned depth, and RGB, using uncertainty to modulate features and suppress unreliable conditions.The branch provides local, structured geometric constraints and integrates depth, appearance, and spatial position.Source: §3.3.2 · p. 6

3.4 Cross-View Semantic Conditioning

  • The geometry-aware branch supplies spatial constraints, while a cross-view semantic encoder uses position-enriched tokens and self-attention to model global relationships across views.Its features enter the UNet Transformer blocks through cross-attention, supporting long-range scene relationships and cross-view consistency.Source: §3.4 · pp. 6–7, Figure 4 · p. 6

3.5 Dual-Branch Balanced Classifier-Free Guidance

  • Dual-branch balanced CFG adds geometry-only and semantics-only condition-dropout cases so the model learns each branch’s effects as well as their joint contribution.This addresses the risk that training favors one branch and underuses the other; at inference, the method uses the standard conditional guidance formulation.Source: §3.5 · p. 7

4 Experiments

LiFi is evaluated for LiDAR generation, camera-to-LiDAR generation, temporal consistency, and downstream detection, with ablations of its components and settings. The reported results show improvements in generation metrics and camera-to-LiDAR fidelity and consistency, while consecutive camera inputs yield sequences with good temporal consistency.

4.1 Experimental Setups

  • Experiments use nuScenes and KITTI-360, comparing LiFi with LiDAR-generation and camera-to-LiDAR baselines; training uses 200 epochs and inference uses 50 denoising steps.Training uses four NVIDIA H200 GPUs and a learning rate of 1 × 10^-4.Source: §4.1 · p. 8

4.2 LiDAR Data Generation

  • On nuScenes, LiFi achieves clear improvements across multiple LiDAR-generation metrics, including MMD, JSD, and two perception-based Fréchet distances.Table 1 reports the main quantitative results; its caption specifies MMD in 10^-5 and JSD in 10^-2.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Camera-to-LiDAR Generation

  • LiFi achieves substantially better point-level fidelity and cross-modal consistency than existing camera-to-LiDAR baselines on nuScenes.Table 3 also reports MMD and JSD, alongside CD, EMD, DAS, and inference time.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 Temporal Consistency of Continuous Scene Generation

  • Consecutive camera inputs produce LiDAR sequences with good temporal consistency, evaluated using CTC and TTCE despite no explicit temporal-generation mechanism.Table 4 reports the temporal-consistency results.Source: §4.4 · p. 9, Table 4 · p. 10

4.5 Downstream Task: 3D Object Detection

  • Generated LiDAR point clouds have certain value in downstream autonomous-driving detection tasks, where the study compares camera-only and camera-LiDAR fusion baselines.The detector is retrained using generated LiDAR data; results are reported in Table 5.Source: §4.5 · p. 10, Table 5 · p. 10

4.6 Ablation Study

  • Ablations examine the two branches, geometric-branch inputs and uncertainty modulation, balanced CFG and guidance scale, plus views, range-image resolution, and denoising steps.Tables 6 and 7 report results for the branch, condition, and CFG ablations; additional setting results are in the supplement.Source: §4.6 · p. 10, Table 6 · p. 10

5 Conclusion

The paper concludes that LiFi generates realistic, high-fidelity LiDAR scenes from multi-view camera images and that its outputs have some downstream autonomous-driving value. It identifies information loss from range-image representation and the lack of explicit temporal modeling as limitations, with implementation details and supplementary material provided in the appendix.

5 Conclusion

  • LiFi generates realistic LiDAR scenes from multi-view camera images, with high-fidelity outputs and some value in downstream autonomous-driving tasks.The authors suggest future applications in simulation, planning, and control in complex environments and long-tail scenarios.Source: §5 · p. 10

A Overview of the Appendix

  • The appendix outlines sections on limitations and future work, additional experiments, theoretical analysis, and supplementary results.These are presented as supporting material for the main paper.Source: A Overview of the Appendix · p. 14

B Limitations and Future Work

  • Range-image representation is not entirely lossless and omits some camera-image details; explicit temporal modeling is also absent from the architecture.The authors suggest refining the representation and adding temporal-consistency mechanisms as future work.Source: B Limitations and Future Work · p. 14

C More Implementation Details

  • Implementation details specify 200 training epochs, 176,000 optimization steps, 50 inference denoising steps, and six camera views per nuScenes scene.The implementation uses pretrained DepthAnything3 and DINOv2 encoders, a RangeLDM-initialized VAE, and circular convolution.Source: C More Implementation Details · p. 14

D Detailed Description of Evaluation Metrics

The appendix describes metrics for distribution, point-cloud, temporal, and detection evaluation, then details range-view projection and geometric-conditioning analyses. It also explains how reverse sampling and uncertainty-conditioned modulation address computational, sampling, and optimization concerns.

D Detailed Description of Evaluation Metrics

  • MMD and JSD compare generated and real distributions; the paper computes them from 160 × 100 bird’s-eye-view histograms.JSD measures distribution similarity through information loss relative to a uniform mixture.Source: D Detailed Description of Evaluation… · pp. 14–15
  • FPD compares generated and real point-cloud distributions using global point-cloud features, while FRD computes a Fréchet distance from range-image features.The described feature extractors include PointNet or PointNet++ for FPD and a 2D CNN such as ResNet for FRD.Source: D Detailed Description of Evaluation… · p. 15
  • Temporal evaluation uses TTCE and CTC; CTC is the average Chamfer Distance across adjacent generated-frame pairs.The appendix defines the CTC score over all adjacent frame pairs in a sequence.Source: D Detailed Description of Evaluation… · p. 15
  • CD measures nearest-neighbor distances between point sets, while EMD measures the minimum average transport cost and better reflects global structure and density differences.Both are introduced for camera-to-LiDAR point-cloud fidelity evaluation.Source: D Detailed Description of Evaluation… · p. 15
  • Downstream detection evaluation uses mAP for precision-recall performance and NDS for overall 3D detection quality, aggregating position, size, orientation, velocity, and attribute errors.The appendix also describes AP as area under the precision-recall curve.Source: D Detailed Description of Evaluation… · p. 16

E Standard Range-View Projection and Corrected Version

  • Range-view projection maps LiDAR points into a 2D representation that can be converted back to 3D; the corrected version accounts for each beam’s sensor height and elevation.The standard projection uses spherical coordinates, while the corrected method uses beam-specific parameters estimated by Hough voting.Source: E Standard Range-View Projection and… · p. 16

F.1 Computational Complexity

  • Reverse sampling queries the target LiDAR grid at O(Hr · Wr), avoiding the O(Himg · Wimg) forward-projection cost and directly sampling in the LiDAR scanning pattern.The appendix describes image-domain sampling as depth-dependent and prone to near-region oversampling and distant-region undersampling, which can cause aliasing and ghost artifacts.Source: §F.1 · p. 17, §F.2 · p. 17

F.3 Weighted Estimation and Optimization Dynamics

  • Uncertainty-based FiLM modulation scales and shifts geometric features by local depth confidence, attenuating gradients in high-uncertainty regions during optimization.The stated effect is adaptive suppression of unreliable-region gradient interference, improving training stability.Source: §F.3 · pp. 17–18

G Theoretical Analysis of the Cross-View Semantic Branch

The cross-view semantic branch addresses the ambiguity of matching image features across views by incorporating pose-conditioned information and attention. The balanced CFG strategy counters reliance on whichever conditioning branch is easier to optimize.

G Theoretical Analysis of the Cross-View Semantic Branch

  • Without depth, a pixel corresponds to a set of possible 3D points, making cross-view interaction based only on 2D features theoretically ill-posed.The paper frames this ambiguity as projection uncertainty.Source: G Theoretical Analysis of the… · p. 18
  • Pose-conditioned patch, viewpoint, and camera-extrinsic encodings reduce feature confusion caused by projection uncertainty.The design combines image features with viewpoint and camera-pose information.Source: G Theoretical Analysis of the… · p. 18
  • The attention kernel uses feature similarity and pose information to learn cross-view associations related to 3D geometry, promoting geometric consistency in the latent space.The attention update is described as kernel smoothing that can reduce cross-view feature consistency error during optimization.Source: G Theoretical Analysis of the… · p. 19
  • The geometric branch captures local high-frequency structures, while global attention models long-range dependencies and low-frequency structures.The paper characterizes their collaboration as preserving local detail while enhancing cross-view consistency through global interaction.Source: G Theoretical Analysis of the… · p. 19

H Theoretical Analysis of the Dual-Branch Balanced CFG

  • When geometry provides strong structural constraints, standard joint training may favor it early and weaken reliance on semantic conditioning.The paper describes this as an information shortcut toward the condition that is easier to optimize.Source: H Theoretical Analysis of the… · p. 19
  • Balanced CFG fits unconditional, geometry-conditioned, semantic-conditioned, and jointly conditioned distributions to maintain predictive capability across conditional subspaces.The authors say this strategy prevents reliance on a single condition and mitigates imbalanced information use.Source: H Theoretical Analysis of the… · p. 19

I Discussion on Pre-trained Vision Models and Metric Scale

LiFi’s geometric branch accepts depth predictions and reliability estimates, so its pretrained depth model is replaceable, though independent-view depth scales may need alignment. Generated LiDAR is in the target dataset’s metric coordinate system, and the section also describes adaptation to new sensor configurations and inference-efficiency analysis.

I Discussion on Pre-trained Vision Models and Metric Scale

  • LiFi’s geometric branch requires per-view depth predictions and reliability estimates, not a specific internal depth-model architecture.An alternative model with these outputs can use the same reverse sampling, FiLM modulation, and diffusion pipeline.Source: I Discussion on Pre-trained Vision… · p. 19
  • Independent-view depth models may predict different scales across views, so the authors recommend cross-view scale alignment before reverse sampling.One suggested approach uses camera calibration and overlapping fields of view to align scale and shift.Source: I Discussion on Pre-trained Vision… · p. 20
  • DepthAnything3 is an instantiation choice rather than an irreplaceable component, and multi-view consistency can improve geometric-condition quality without being a strict architectural requirement.The paper names VGGT and MapAnything as possible direct replacements.Source: I Discussion on Pre-trained Vision… · p. 20
  • LiFi’s generated LiDAR is represented in the target dataset’s metric coordinate system, with valid range-image cells corresponding to actual target-sensor distances.The range image can be back-projected using the sensor’s beam angles, azimuth sampling, and calibration parameters to produce a point cloud in meters.Source: I Discussion on Pre-trained Vision… · p. 20
  • For nuScenes and KITTI-360, generation needs no additional global scale fitting; metric scans from other datasets can be converted to the target sensor’s range-image representation for training.The paper specifically names KITTI and Argoverse as examples of metric LiDAR datasets that can provide training supervision.Source: I Discussion on Pre-trained Vision… · p. 20
  • For a new virtual LiDAR configuration, the target sensor settings determine the range-image representation and the camera–metric LiDAR data used to train or fine-tune the model.The settings include beam elevation angles, horizontal angular resolution, and extrinsic parameters.Source: I Discussion on Pre-trained Vision… · p. 20

J Analysis of Computational Efficiency

K Ablation Study on the Number of Input Image Views

The ablation varies the number of input camera images across four settings. The supplied passages identify the comparison and metrics but do not state the results.

K Ablation Study on the Number of Input Image Views

  • The nuScenes ablation compares LiFi using 1, 2, 4, and 6 input camera images.The paper treats the number of input images as an architectural design choice rather than a fairness difference.Source: K Ablation Study on the Number of… · p. 20
  • The supplied Table 8 caption describes inference efficiency, reporting inference time in seconds per frame alongside MMD and JSD.The caption does not specify a number-of-views comparison.Source: Table 8 · p. 21
  • Table 9 evaluates how the number of input images affects LiFi’s generation performance.The caption reports MMD in 10^-5 and JSD in 10^-2.Source: Table 9 · p. 21

L Ablation Study on the Range Image Resolution

The resolution ablation examines how range-image resolution affects generation and inference efficiency. The text explains the representation trade-offs at three resolutions but provides no table values.

L Ablation Study on the Range Image Resolution

M Quantitative Analysis of Failure Cases at Long Range

LiFi’s generation performance gradually degrades with increasing distance. Table 11 evaluates performance across radial distance ranges using MMD and JSD.

M Quantitative Analysis of Failure Cases at Long Range

N More Qualitative Results

The section presents additional qualitative comparisons on nuScenes, including LiDAR generation examples and camera-to-LiDAR results across diverse scenarios.

N More Qualitative Results

Related papers

Loading 2610.19786v1…