PaperlayerView exact revision on arXiv

Source-linked AI summary

UniScene: Unified Occupancy-centric Driving Scene Generation

Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, Shuchang Zhou, Li Zhang, Xiaojuan Qi, Hao Zhao, Mu Yang, Wenjun Zeng, Xin Jin

arXiv:2412.05435v2cs.CV

TL;DR

Existing driving-scene generators often produce one data form directly from coarse layouts, leaving a gap for diverse tasks and complex scene distributions. UniScene progressively generates semantic occupancy, video, and LiDAR, and the paper reports gains over current state-of-the-art methods across all three forms, alongside downstream-task improvements.

  • Problem

    Existing methods often generate a single data form directly from coarse layouts, while complex real-world scene distributions and diverse downstream tasks motivate richer representations and multiple sensor formats.

    Source: §1 · pp. 1–2

  • Method

    UniScene first generates semantic occupancy from customized BEV layouts, then uses occupancy-conditioned Gaussian-based rendering for video and prior-guided sparse modeling for LiDAR.

    Source: §1 · p. 2

  • Results

    UniScene surpasses current state-of-the-art methods in occupancy, video, and LiDAR generation and enhances downstream tasks.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Generated multimodal data with annotations improves training for downstream perception tasks, including occupancy prediction, 3D detection, and BEV segmentation.

    Source: §1 · p. 2

    Integrating multiple generation tasks into a comprehensive system is challenging and resource-intensive.

    Source: §5 · p. 8

Abstract

from arXiv · show

Generating high-fidelity, controllable, and annotated training data is critical for autonomous driving. Existing methods typically generate a single data form directly from a coarse scene layout, which not only fails to output rich data forms required for diverse downstream tasks but also struggles to model the direct layout-to-data distribution. In this paper, we introduce UniScene, the first unified framework for generating three key data forms - semantic occupancy, video, and LiDAR - in driving scenes. UniScene employs a progressive generation process that decomposes the complex task of scene generation into two hierarchical steps: (a) first generating semantic occupancy from a customized scene layout as a meta scene representation rich in both semantic and geometric information, and then (b) conditioned on occupancy, generating video and LiDAR data, respectively, with two novel transfer strategies of Gaussian-based Joint Rendering and Prior-guided Sparse Modeling. This occupancy-centric approach reduces the generation burden, especially for intricate scenes, while providing detailed intermediate representations for the subsequent generation stages. Extensive experiments demonstrate that UniScene outperforms previous SOTAs in the occupancy, video, and LiDAR generation, which also indeed benefits downstream driving tasks. Project page: https://arlo0o.github.io/uniscene/

1. Introduction

Existing driving-scene generation often targets one data format directly from coarse layouts, while complex scene distributions motivate a richer intermediate representation. UniScene uses semantic occupancy to guide generation of video and LiDAR as well as occupancy.

1. Introduction

  • Synthetic driving data may reduce the high resource demands of real-world collection and annotation and support downstream BEV segmentation and 3D detection.The introduction positions these tasks as applications of generated data.Source: §1 · p. 1
  • Prior methods often generate a single data format from coarse layouts, a direct strategy that struggles to capture complex real-world scene distributions.The resulting data may be suboptimal for realistic geometry and appearance.Source: §1 · pp. 1–2
  • Semantic occupancy carries both semantic and geometric information, making it a richer intermediate representation than 2D BEV maps for diverse data generation.The paper argues this can support RGB video and LiDAR generation while improving flexibility and accuracy.Source: §1 · p. 2
  • UniScene first generates occupancy from customized BEV layout sequences, then uses it as an intermediate representation to guide video and LiDAR generation.The occupancy sequences are intended to maintain spatial-temporal consistency.Source: §1 · p. 2
  • Gaussian Splatting renders occupancy into semantic and depth maps for video, while prior-guided sparse modeling uses occupancy-based priors for LiDAR generation.These strategies bridge the representation gap between occupancy and the two generated data formats.Source: §1 · p. 2

2. Related Work

Related work includes occupancy prediction and forecasting, but temporal 3D occupancy generation remains limited. Driving-scene generators also commonly focus on one data form, unlike UniScene's occupancy-centered framework for occupancy, video, and LiDAR.

Semantic Occupancy Representation

  • Occupancy research has focused mainly on prediction and forecasting, while temporal 3D occupancy generation remains limited and OccSora's generated quality lags ground truth [59].Earlier examples include MonoScene, TPVFormer, VPD, OccWorld [86], and OccLlama [68].Source: §2 · p. 2

Overall framework

  • UniScene's framework first generates occupancy from BEV layouts, then branches to video through rendered semantic and depth maps and to LiDAR through sparse modeling.The figure describes these as occupancy-based video and LiDAR generation paths.Source: Figure 2 · p. 3

Generation Models in Autonomous Driving

  • Driving-scene generation spans NeRF and Gaussian Splatting novel-view synthesis, diffusion-based video, world models, and LiDAR point-cloud generation, but these methods primarily target one form [79].The paper contrasts this single-form focus with UniScene's generation of occupancy, video, and LiDAR.Source: §2 · p. 3

3. Methodology

UniScene organizes driving-scene generation as an occupancy-centric hierarchy: it generates semantic occupancy from BEV layouts, then uses occupancy to guide video and LiDAR generation. The method adds specialized transfer strategies for video and LiDAR.

3.1. Controllable Semantic Occupancy Generation

  • The Occupancy Diffusion Transformer generates controllable occupancy sequences from BEV layouts, using a continuous latent VAE and temporal modeling for long-term consistency.The VAE compresses occupancy into a continuous latent space, while the diffusion model aligns BEV layouts with noise volumes to control generated sequences.Source: §3.1 · pp. 3–4

3.2. Video: Occupancy as Conditional Guidance

  • Gaussian-based joint rendering converts occupancy into semantic and depth maps that guide multi-view video generation and provide geometric awareness during sampling.The method also warps appearance priors between frames using rendered depth to address variation in dynamic regions.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. LiDAR: Occupancy-based Sparse Modeling

  • For LiDAR, occupancy priors concentrate ray sampling in occupied voxels, while a sparse UNet and heads predict depth, reflection intensity, and ray dropping.The ray-dropping head eliminates noise points in predictions.Source: §3.3 · p. 5, Figure 4 · p. 5

4. Experiments

On NuScenes, UniScene evaluates occupancy reconstruction, generation and forecasting, multi-view video and LiDAR generation, controllable synthesis, and downstream perception training. Its experiments also test components of the occupancy, video, and LiDAR generation models.

4. Experiments

  • The experiments use NuScenes and interpolate 2Hz key-frame occupancy labels to 12Hz.Occupancy evaluation uses the NuScenes-Occupancy key-frame validation set at 2Hz.Source: §4 · p. 5

4.1. Main Results

  • Editing BEV layouts controls geometry in generated occupancy, video, and LiDAR, while text prompts vary video weather and lighting.The examples include removing cars from a layout and prompting for weather or time of day.Source: §4.1 · p. 5, Figure 5 · p. 6
  • 10.96% higher mIoU than OccWorld is reported for occupancy reconstruction with VAE compression at the default compression ratio of 512.The comparison is against prior VQVAE-based discrete compression methods.Source: §4.1 · p. 6, Table 2 · p. 6
  • UniScene’s forecasting variant surpasses OccWorld with fewer reference frames, while video generation outperforms other methods and LiDAR achieves the best reported generation performance.The comparisons are reported for occupancy forecasting, multi-view video generation, and LiDAR generation, respectively.Source: §4.1 · pp. 6–7, Table 3 · p. 6, Table 4 · p. 6
  • UniScene’s multi-modal augmented data improves occupancy prediction, BEV segmentation, and 3D object detection training over the reported baselines.Occupancy prediction is evaluated with camera, LiDAR, and camera-plus-LiDAR settings.Source: §4.1 · p. 7, Table 6 · p. 8, Table 7 · p. 8

4.2. Ablation Studies

  • Adding temporal information to the occupancy VAE decoder reduces MMD by 38.01%, while temporal and spatial DiT attention improve F3D by 10.29% and 39.26%, respectively.The ablations evaluate separate components of the occupancy generation model.Source: §4.2 · p. 7, Table 8 · p. 8
  • Occupancy-based semantic and depth rendering maps improve video generation, and geometric-aware noise reduces FVD from 87.52 to 70.52.The rendering maps also improve cross-view consistency and high-fidelity structure relative to spatial-temporal attention.Source: §4.2 · p. 8, Table 9 · p. 8, Figure 9 · p. 8
  • LiDAR ablations report lower JSD with Sparse UNet and sparse sampling, reduced computational consumption from sparse sampling, and lower MMD and JSD with ray dropping.Reported changes are -25.77% JSD and +1.63% memory for Sparse UNet, -58.94% computational consumption and -4.00% JSD for sparse sampling, and -25.92% MMD and -25.00% JSD for ray dropping.Source: §4.2 · p. 8, Table 10 · p. 8

5. Conclusion and Future Work

UniScene progressively generates semantic occupancy, video, and LiDAR for autonomous driving. The authors report gains over current SOTAs across all three data types and downstream tasks, while identifying lightweight deployment as future work.

5. Conclusion and Future Work

  • UniScene generates semantic occupancy, video, and LiDAR through two hierarchical steps.The framework is designed to produce high-fidelity, controllable, and annotated driving data.Source: §5 · p. 8
  • UniScene surpasses current SOTAs in all three data types and enhances extensive downstream tasks.Source: §5 · p. 8
  • Integrating multiple generation tasks into one comprehensive system is challenging and resource-intensive, motivating future work on lightweight deployment.The authors also identify extending the system to embodied intelligence and robotics as an opportunity.Source: §5 · p. 8

A. Problem Definition and Distinctiveness

UniScene uses controllable BEV layouts to generate temporal semantic occupancy, then conditions video and LiDAR generation on occupancy. The paper presents this hierarchy as a way to address limitations of static or direct layout-to-data generation.

A.1. Controllable Occupancy Generation

  • Controllable generation P(Occ|BEV) uses BEV scene layouts to generate temporal semantic occupancy, addressing prior occupancy methods focused on static scenes.Source: §A.1 · p. 9

A.2. Conditional Video and LiDAR Generation

  • Conditioning video and LiDAR generation on occupancy decomposes direct layout-to-data generation and, the authors report, alleviates learning burden while better capturing underlying distributions.Source: §A.2 · p. 9

B.1. Semantic Occupancy Generation

  • UniScene aims to generate controllable temporal 3D occupancy sequences with high fidelity while maintaining temporal consistency.The cited prior approaches include static or unconditional methods and a temporal method with suboptimal reconstruction and limited precise control.Source: §B.1 · p. 9

B.2. Driving Video Generation

  • UniScene renders semantic and depth maps from semantic occupancy as detailed priors for video, while BEV layouts simplify geometric editing.The related-work discussion notes limitations in multi-view generation, label alignment, compressed occupancy features, or explicit geometry in prior approaches.Source: §B.2 · p. 9

B.3. LiDAR Point Cloud Generation

  • UniScene uses fine-grained semantic-occupancy priors for LiDAR generation; its occupancy model combines a VAE with a DiT to generate denoised latent representations.The cited related work describes representation trade-offs in LiDAR generation, including loss of 3D geometry, fine detail, or structured point information.Source: §B.3 · p. 10, Figure 10 · p. 10

C. More Implementation Details

The BEV editing scheme modifies a scene layout and uses the revised condition to generate updated occupancy, video, and LiDAR. The supplied reconstruction table caption reports occupancy VAE advantages over VQVAE across compression ratios.

C.1. BEV Editing Scheme

  • The BEV editing scheme changes an initial layout to create out-of-distribution data, then uses the updated layout to generate occupancy and condition corresponding video and LiDAR.DDIM inversion maps original occupancy to a noise latent; denoising with the revised BEV layout produces the updated occupancy.Source: §C.1 · p. 10
  • The occupancy VAE achieves significantly higher mIoU and IoU than VQVAE under the same architecture across compression ratios.Source: Table 11 · p. 10

C.2. Occupancy Generation Model

The occupancy model compresses semantic occupancy with a VAE and denoises latent sequences with a DiT conditioned on BEV layouts. Its forecasting variant outperforms cited baselines across multiple horizons, and reported performance improves with greater computation.

C.2. Occupancy Generation Model

  • The Occupancy VAE encodes semantic occupancy into continuous latent features and reconstructs occupancy sequences using a 3D CNN decoder with axial attention.Training uses cross-entropy and Lovász-softmax losses; inference selects the highest-scoring class for each voxel.Source: §C.2 · p. 10
  • F3D performance and mIoU improve steadily as Occupancy DiT computation, measured in FLOPs, increases.Source: Figure 11 · p. 11, §C.2 · p. 11
  • The forecasting variant surpasses OccWorld and OccLLama in mIoU and IoU across 1s, 2s, and 3s horizons.It predicts six future frames using one or two reference frames rather than five.Source: Table 12 · p. 11, §C.2 · p. 11
  • The Occupancy DiT combines noisy occupancy latents with down-sampled BEV layouts, then uses spatial and temporal transformer blocks to denoise latent sequences.BEV conditions are randomly dropped with probability 0.1 during training to support unconditional generation for occupancy editing.Source: §C.2 · p. 11

C.3. Video Generation Model

The video model uses occupancy grids and textual prompts to guide latent video generation, with rendered depth and semantic maps providing camera-aligned conditions. The LiDAR model extracts occupancy features and samples points along rays using spatial priors.

C.3. Video Generation Model

  • The video model combines noisy video latents with occupancy grids and textual prompts, then uses a Diffusion UNet and VAE decoder to generate video.Occupancy grids provide the basis for depth and semantic guidance maps.Source: §C.3 · p. 11, Figure 12 · p. 12
  • Semantic occupancy cells become 3D Gaussian primitives with cell-center positions, cell-sized scales, and semantic labels, which are rendered into viewpoint-aligned depth and semantic maps.Each primitive receives the semantic label of its corresponding occupancy cell.Source: §C.3 · p. 12
  • Two ControlNet-inspired encoder modules condition video generation on depth and semantic maps, each using two ResNet blocks followed by a zero convolution.The zero convolution is used to preserve the original functionality of the diffusion model.Source: §C.3 · p. 12

C.4. LiDAR Generation Model

  • Occupancy features guide LiDAR point sampling along spatially structured rays, while a LiDAR Head predicts point intensity, ray-drop probabilities, and final positions.The head uses weighted summation and MLP modules to refine and aggregate point features.Source: §C.4 · pp. 12–13

D. Datasets

The study uses NuScenes-Occupancy data and extends occupancy annotations to 12 Hz to meet its data requirements and support fair comparison with prior work.

D. Datasets

  • NuScenes-Occupancy provides 34,000 key frames across 850 scenes at 2 Hz, each annotated with one of 17 semantic categories.Following the dataset methodology, the study uses 28,130 frames for training and 6,019 for validation.Source: D. Datasets · p. 13
  • The study extends the 2 Hz data to 12 Hz using interpolated annotations from ASAP and LiDAR data from NuScenes for fair comparison with prior work.The process concatenates scene LiDAR points, reconstructs a mesh with NKSR, and assigns semantic labels to mesh vertices.Source: D. Datasets · p. 13
  • Semantic occupancy data is formed by converting mesh vertices and their assigned semantic labels into occupancy labels.The labels assigned to vertices use NuScenes LiDARseg tags and ASAP 12 Hz annotation information.Source: D. Datasets · p. 13

E. Evaluation Metrics

The paper evaluates generation quality and downstream perception using task-specific metrics. It describes a two-stage training setup and compares against occupancy, video, and LiDAR baselines.

E. Evaluation Metrics

  • Occupancy fidelity uses F3D and MMD, layout-conditioned accuracy uses mIoU, video quality uses FID and FVD, and LiDAR generation uses JSD and MMD.Occupancy forecasting is evaluated with IoU and mIoU.Source: E. Evaluation Metrics · p. 13
  • Downstream evaluation uses mIoU and IoU for semantic occupancy prediction, mIoU for road and vehicle BEV segmentation, and mAP and NDS for 3D detection.CVT and BEVFusion are the respective BEV segmentation and 3D detection baselines.Source: E. Evaluation Metrics · p. 13

F. Model Setup

  • Training proceeds in two stages: occupancy models are trained first, then fixed to generate occupancy grids while video and LiDAR models train jointly with occupancy-based conditions.The second stage runs for 90 epochs; the video model starts from SVD weights, while the LiDAR model is trained from scratch.Source: F. Model Setup · pp. 13–14

G. Comparison Baselines

  • Comparisons include OccLlama, OccWorld, and OccSora for occupancy; multiple driving-scene methods for video; and Open3D and LiDARDM for LiDAR.The video comparison uses a multi-view Vista variant for fairness; Open3D and LiDARDM do not generate intensity values.Source: G. Comparison Baselines · p. 14

H. More Visualization Results

The visualizations show occupancy generation in unseen settings and transfer to other datasets, alongside configurable LiDAR patterns and controllable, coherent scene generation.

H. More Visualization Results

  • Occupancy generation maintains structural and semantic consistency in unseen scenarios, and the model transfers to Waymo and nuPlan layouts with conditional reference images.The Waymo back view is null because it is unavailable.Source: H. More Visualization Results · p. 14, Figure 19 · p. 16
  • The geometric-aware noise prior improves video generation quality, producing high-fidelity, consistent moving cars across frames.The prior injects dense appearance priors and explicit geometric awareness during video sampling.Source: H. More Visualization Results · p. 16
  • The ray-based LiDAR heads generate 16-, 32-, and 64-beam patterns without retraining while preserving scene geometry.The ground-truth beam configuration is 32 beams.Source: Figure 16 · p. 15, H. More Visualization Results · p. 17
  • Visualizations show temporally and spatially consistent large-scale scenes, videos with varied weather and time of day, and outputs reflecting edited scene geometry.The paper also shows joint generation of semantic occupancy, LiDAR, and multi-view video with temporal consistency.Source: Figure 17 · p. 15, Figure 18 · p. 16, H. More Visualization Results · p. 17

Related papers

Loading 2412.05435v2…