Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

• Reconstruction-based simulators preserve geometry and control sensor viewpoints but require lengthy per-scene fitting, while generative methods scale beyond captured scenes but lack long-horizon consistency.A unified simulator must combine both capabilities while producing accurate, temporally coherent observations at interactive speed.Source: §1 · p. 3
• A universal, attention-based range-image tokenizer provides a shared latent space for 32-, 64-, and 128-beam sensors and generalizes to an unseen scanning pattern.Its local spatial and factorized temporal attention handle sparse geometry across sweeps without reducing native temporal resolution.Source: §1 · p. 3
• A bidirectional teacher conditions on LiDAR prefixes, HD-Maps and actor layouts, persistent spatial cache, and optionally text; varying these conditions selects generation, forecasting, or resimulation.A spatial cache built from observed sweeps anchors resimulation to a particular world without per-scene optimization.Source: §1 · pp. 3–4
• Across three evaluation protocols, LidarVerse outperforms task-specific baselines on most reported metrics and operates at or above the native 10-Hz LiDAR rate.The causal checkpoint retains a shared conditioning interface for all three tasks.Source: §1 · p. 4
• A four-step causal student combines a rolling temporal K/V cache with a world-aligned spatial cache to support persistent, temporally coherent long-horizon rollouts.The temporal cache retains recent dynamics, while the spatial cache preserves scene geometry beyond the finite temporal window.Source: §1 · p. 4

2 Related Work

• LidarVerse spans generation, forecasting, and resimulation with one conditional world model, selected by the scene evidence available at inference.Scene-level conditions guide generation, an observed prefix enables forecasting, and a persistent spatial cache anchors resimulation without per-scene optimization.Source: §2 · p. 4

2.1 LiDAR Generation

• Few-step distillation with persistent 3D memory maintains coherent long rollouts while generating at the native sensor rate in real time.Prior generation approaches include single-sweep synthesis, sequence or 3D/4D generation, and autoregressive methods; the cited discussion notes short rollouts and lack of long-horizon consistency.Source: §2.1 · pp. 4–5

2.2 LiDAR Resimulation

• LidarVerse uses a cross-scene generative model for LiDAR resimulation without per-scene optimization, completing content beyond the captured trajectory and rendering native spinning-LiDAR observations in real time.This addresses a literature gap: generalizable reconstruction and generative-prior approaches had remained largely unexplored for LiDAR resimulation.Source: §2.2 · p. 5

2.3 Point-Cloud Forecasting

• LidarVerse forecasts by conditioning its probabilistic LiDAR generator on an observed prefix, rather than using a task-specific forecasting head.Autoregressive sampling can continue beyond the temporal horizon seen during training, unlike most occupancy-based forecasters that use a prescribed future horizon.Source: §2.3 · p. 5

3.1 LiDAR-Native Tokenizer

• A range-image tokenizer preserves spatial and temporal structure, using neighborhood attention, causal temporal attention, and no temporal downsampling to accommodate different sensor patterns.Each sweep is represented on a calibrated angular grid; the tokenizer preserves one latent slice per input sweep.Source: §3.1 · p. 5, §3.1.1 · p. 6, §3.1.2 · p. 7

3.2 Bidirectional Teacher Model

• A bidirectional diffusion teacher learns one conditional distribution from LiDAR prefixes, HD-Maps and actor layouts, persistent spatial cache, and optional text.The conditions encode different scene evidence rather than task identities; text controls environmental attributes while spatial conditions anchor geometry.Source: §3.2 · p. 7, §3.2.2 · pp. 8–9

3.2.3 Condition Sampling

• Independent condition sampling exposes the teacher to diverse combinations of scene evidence without task-specific tokens or objectives.Prefix lengths, HD-Map dropout, PSC modes, source-sweep counts, and text dropout are sampled during training.Source: §3.2.3 · p. 9

3.3 Causal Distillation

• The causal student adapts the teacher to block-causal attention, then uses four shifted rectified-flow SDE steps per four-frame block and self-forcing distribution matching.SF-DMD matches self-rolled-out latents to the real LiDAR latent distribution rather than regressing to paired ground-truth sequences.Source: §3.3.2 · pp. 9–10, §3.3.1 · p. 9, §3.3 · p. 9

3.3.3 Rolling Persistent Spatial Cache

• A rolling world-aligned voxel map complements the finite temporal K/V cache, retaining scene geometry beyond its recent-context window for later blocks.Repeated observations and free-space contradictions govern voxel persistence; only confirmed, non-removed voxels are rendered before the next block.Source: §3.3.3 · pp. 10–11

3.4 Model Inference

• The same causal checkpoint performs LiDAR generation, point-cloud forecasting, and generalizable resimulation by changing its scene evidence and target sensor poses.Generation builds a PSC from generated sweeps; forecasting starts from observed history, while resimulation uses sparse observed sweeps without per-scene optimization.Source: §3.4 · p. 11

4 Experiments

• Evaluation primarily uses Waymo; tokenizer comparisons pair each representation with the same DiT, training recipe, and protocol.The study also examines intensity and cross-sensor and dataset generalization; cross-sensor adaptation uses NVIDIA PAI, Waymo, and nuScenes.Source: §4 · p. 11, §4.1 · pp. 12–13, §4.2 · p. 13

4.1 Implementation Details

• The tokenizer uses spatial attention and factorized temporal attention to compress each sweep 8×8 while preserving temporal resolution.Its decoder predicts range and validity mask; the tokenizer has 64.5M parameters and was trained for 120,000 iterations.Source: §4.1 · pp. 11–12
• The world model trains a bidirectional teacher, then a causal student with teacher forcing and self-forcing; the student generates four-frame blocks in four steps.It maintains a rolling nine-frame K/V cache including one persistent sink frame.Source: §4.1 · p. 12, Table 1 · p. 12

4.2 Tokenizer Performance

• Tokenizer evaluation covers reconstruction geometry, distributional generation quality, feature-space quality, and temporal consistency.The reported measures include CD, range L1, AbsRel, MMD, JSD, FUD, TTCE, CTC, and ICP-Avg.Source: §4.2 · p. 13
• 41% lower CD and 53% lower AbsRel at 256× compression; at lower compression, CD falls 61% and FUD 50% versus the best competing result.The tokenizer also achieves the best FUD and nearly all temporal metrics at 256× compression; the lower-compression comparison is despite LiDM using 32× compression.Source: §4.2 · p. 13, Table 2 · p. 13

HD-Map-Conditioned Generation

• The teacher follows HD-Map conditions across highway, pedestrian-rich, and urban scenes.Figure 4 shows three sequences conditioned only on the HD-Map.Source: §4.3 · p. 14, Figure 4 · p. 14

Unconditional Generation

• From noise, the teacher generates realistic, diverse sequences ranging from high-speed highways to dense urban traffic.Figure 5 sets all conditioning inputs to null.Source: §4.3 · p. 15, Figure 5 · p. 15

First-Frame-Conditioned Generation

• Given one observed frame, generated continuations show plausible agent motion following the observed traffic structure.Figure 6 depicts a vehicle crossing an intersection and a pedestrian in a crosswalk.Source: §4.3 · p. 15, Figure 6 · p. 15

Text-Controlled Weather Generation

• The text-conditioned teacher reproduces typical rainy-weather LiDAR patterns, including increased road-surface ray drop.The weather example uses the text-conditioned NVIDIA PAI extension.Source: §4.3 · p. 16, Figure 7 · p. 16

Bidirectional Teacher Model

• Together, the examples show one bidirectional prior responding to temporal, geometric, and semantic evidence through a common interface.The next evaluation tests whether causal distillation retains this flexibility during real-time, long-horizon rollout.Source: §4.3 · p. 16

4.4 Autoregressive LiDAR Generation

• The two settings evaluate 129-frame rollouts at 10 Hz: map-conditioned generation starts from a generated frame, while action-conditioned generation starts from a ground-truth frame and follows ego actions.Both settings generate 128 additional frames.Source: §4.4 · p. 16
• Long-horizon stability is assessed with FUD over non-overlapping eight-frame windows, balancing distributional estimate reliability with sensitivity to rollout degradation.The comparison includes LiDARCrafter and reimplemented LaGen and Sensor2Sensor under the study's protocol.Source: §4.4 · pp. 16–17
• LidarVerse outperforms all baselines on statistical and perceptual distribution metrics and performs best at longer temporal offsets in HD-Map-conditioned generation.It remains competitive on short-term temporal consistency, while qualitative rollouts remain structurally stable as baselines visibly degrade.Source: §4.4 · pp. 17–18, Table 3 · p. 17
• In action-conditioned generation, LidarVerse substantially outperforms the remaining methods across reported distributional and temporal metrics, with quality degrading more slowly over time.Sensor2Sensor is omitted because it has no native mechanism for injecting ego poses.Source: Table 3 · p. 17, §4.4 · p. 18

4.5 Point-Cloud Forecasting

• Forecasting uses a LiDAR prefix to specify recent scene state and dynamics, while future measurements remain unobserved; the model is evaluated at one-, two-, and three-second marks.Metrics are L1 error, AbsRel, and inner Chamfer distance, with aggregate means across all 30 predicted frames.Source: §4.5 · pp. 18–19
• The forecasting protocol predicts a three-second future from three seconds of point-cloud history sampled at 10 Hz, using 30 history and 30 future frames.The comparison includes naive raycasting, 4D-Occ, and UnO; LidarVerse uses a K/V cache size of 21.Source: §4.5 · p. 19
• LidarVerse outperforms task-specific baselines on all reported metrics across all forecast horizons.Visualizations show plausible future motion and preserved traffic-participant shapes; the shared generative prior completes plausible structure when future viewpoints reveal regions absent from the input history.Source: Figure 10 · p. 18, §4.5 · p. 19, Table 4 · p. 19

4.6 Generalizable LiDAR Resimulation

• Sparse observed sweeps identify the captured world, and the persistent spatial cache projects accumulated geometry into requested sensor viewpoints.The protocol evaluates held-out frames along the recorded trajectory, using frames 0, 4, 9, and 14 as inputs in each 20-frame chunk.Source: §4.6 · p. 19
• The evaluation covers 202 Waymo validation sequences and a top-20 subset ranked by total ego path length, with interpolation, extrapolation, and aggregate range L1 and Chamfer distance reported.It also reports per-scene optimization time and whether rendering is real time.Source: §4.6 · pp. 19–20
• Unlike SplatAD, LiDAR4D, and LiDAR-RT, which optimize a separate scene representation per test sequence, LidarVerse uses one checkpoint across scenes without test-time optimization.Qualitative results show the causal checkpoint preserving captured scene structure in interpolation and extrapolation views.Source: §4.6 · pp. 20–21, Figure 11 · p. 20
• Without test-time optimization, the causal student is competitive with per-scene reconstruction baselines on the full validation cohort and outperforms them on every reported quality metric in the top-20 travel-distance subset.The bidirectional teacher provides a quality reference and achieves the best fidelity across nearly all reported metrics; LiDAR-RT has the lowest interpolation L1 on All-202.Source: §4.6 · pp. 20–21, Table 6 · p. 21

4.7 Intensity Modeling

• Intensity predictions remain competitive with state-of-the-art reconstruction baselines on the top-20 travel-distance resimulation subset.The tokenizer is fine-tuned for intensity reconstruction, followed by fine-tuning the bidirectional teacher on combined range-intensity latents.Source: §4.7 · p. 21, Table 7 · p. 21

4.8 Generalization Across Sensors and Datasets

• The tokenizer is jointly fine-tuned on 32-beam nuScenes, 64-beam Waymo, and 128-beam NVIDIA PAI, while a sensor embedding identifies each dataset's LiDAR configuration.For cross-dataset intensity generation, the same procedure is applied to a tokenizer and DiT trained on range and intensity.Source: §4.8 · p. 21
• Joint training across datasets improves tokenizer reconstruction quality over each specialist, with evaluation on matched 21-frame sequences.The universal tokenizer also reconstructs the unseen PandaSet scanning pattern; Table 8 reports reconstruction metrics in meters, where lower is better.Source: Figure 12 · p. 22, §4.8 · p. 23, Table 8 · p. 23
• The jointly trained bidirectional model generates nuScenes, NVIDIA PAI, and Waymo scanning patterns from the same Waymo HD-Map input.The cross-dataset intensity model produces plausible future intensity profiles, including high-intensity returns from retroreflective surfaces.Source: §4.8 · p. 23, Figure 13 · p. 23, Figure 14 · p. 24

LiDAR-Native Tokenizer Architecture

• Neighborhood attention improves reconstruction across all reported metrics, strengthens long-horizon consistency, and reduces tokenizer compute by 57% versus convolution at matched compression and latent dimensionality.Both tokenizers use the same DiT and training protocol; Figure 15 evaluates 129-frame sequences after training on nine-frame sequences.Source: §4.9 · p. 24, Table 9 · p. 24, Figure 15 · p. 24

Spatial Cache and Temporal K/V Cache

• Removing the spatial cache substantially worsens temporal consistency with little effect on FUD, while larger K/V caches improve consistency but reduce throughput and fidelity.The selected setting is a nine-frame K/V cache with the spatial cache, balancing fidelity, temporal consistency, and real-time throughput.Source: §4.9 · p. 25, Figure 16 · p. 25

Spatial-Cache Confirmation

• Registering every latest generated sweep in the spatial cache (N_min = 1) yields the best temporal-consistency metrics but worse fidelity; N_min = 3 balances both.The nine-frame K/V cache is fixed, and N_min = 3 is used for the main causal-generation results.Source: §4.9 · p. 25, Figure 17 · p. 25

Unified versus Task-Specific Specialists

• The unified model improves generation MMD and ICP Avg., while specialists achieve lower FUD and resimulation L1 and CD, retaining competitive performance across both tasks.The unified model uses one checkpoint for generation and resimulation; the specialist results combine two separately trained models.Source: §4.9 · p. 25

5 Limitations and Future Work

• LidarVerse models only a single spinning LiDAR, and dynamic resimulation still relies on annotated object poses during training and inference, limiting scalability to unlabeled scenes.Extending to solid-state LiDAR and joint generation for multi-LiDAR rigs is future work; annotation-free generalizable resimulation remains an open challenge.Source: §5 · p. 25
• The framework focuses exclusively on LiDAR; jointly generating synchronized observations across modalities such as cameras and LiDAR remains a future direction.Source: §5 · p. 26

6 Conclusion

• LidarVerse uses scene-level conditions for new-world generation, a LiDAR prefix for forecasting, and a persistent spatial cache to resimulate captured geometry without per-scene optimization.A universal tokenizer provides a shared latent space across sensor configurations, and a bidirectional diffusion teacher learns the common conditioning interface.Source: §6 · p. 26
• The four-step causal student combines temporal and spatial caches, runs at or above the native 10-Hz LiDAR rate, and achieves state-of-the-art performance on most reported metrics across three tasks.The tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern; a complete closed-loop system is not evaluated.Source: §6 · p. 26