PaperlayerView exact revision on arXiv

Source-linked AI summary

WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation

Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, Li Zhang

arXiv:2312.02934v4cs.CV

TL;DR

Multi-camera driving videos are needed for autonomous-driving datasets, but rendering methods face diversity and lighting challenges, while diffusion methods can lack dense scene information and consistency. WoVoGen predicts a future 4D world volume from vehicle controls and uses it to guide multi-camera video generation; the paper reports image and video quality results and identifies scene editing as a supported task.

  • Problem

    Multi-camera driving datasets are costly to label, while existing rendering and diffusion approaches face limits in diversity, real-world conditions, or dense scene representation and consistency.

    Source: §1 · p. 2

  • Method

    WoVoGen predicts a future 4D world volume from a reference scene and vehicle controls, then uses it to guide diffusion-based multi-camera video generation.

    Source: §1 · pp. 2–3

  • Results

    WoVoGen reports FID 27.6, lower than DriveGan and DriveDreamer, which are limited to single-view image generation, and FVD 417.7, described as superior to other works.

    Source: Multi-camera single-frame image… · p. 13, Multi-camera video generation · p. 13

  • Takeaways & Limitations

    The authors present WoVoGen as producing control-driven, consistent street-view videos while also supporting scene editing.

    Source: §1 · p. 3, §5 · p. 14

Abstract

from arXiv · show

Generating multi-camera street-view videos is critical for augmenting autonomous driving datasets, addressing the urgent demand for extensive and varied data. Due to the limitations in diversity and challenges in handling lighting conditions, traditional rendering-based methods are increasingly being supplanted by diffusion-based methods. However, a significant challenge in diffusion-based methods is ensuring that the generated sensor data preserve both intra-world consistency and inter-sensor coherence. To address these challenges, we combine an additional explicit world volume and propose the World Volume-aware Multi-camera Driving Scene Generator (WoVoGen). This system is specifically designed to leverage 4D world volume as a foundational element for video generation. Our model operates in two distinct phases: (i) envisioning the future 4D temporal world volume based on vehicle control sequences, and (ii) generating multi-camera videos, informed by this envisioned 4D temporal world volume and sensor interconnectivity. The incorporation of the 4D world volume empowers WoVoGen not only to generate high-quality street-view videos in response to vehicle control inputs but also to facilitate scene editing tasks.

1 Introduction

WoVoGen addresses the need for diverse, consistent multi-camera driving videos by conditioning diffusion on an explicit 4D world volume. It first predicts a future world volume from a reference scene and vehicle controls, then uses that volume to guide multi-camera video generation and scene editing.

1 Introduction

  • Diffusion methods using bounding boxes or HD maps provide sparse scene control, motivating a dense world representation for more complete driving-scene depiction.Rendering-based methods provide explicit 3D or 4D structure but tend to have limited diversity and require substantial effort to match real-world lighting and weather.Source: §1 · p. 2
  • WoVoGen represents the scene as a dense 4D voxel volume over time, height, length, and width, including occupancy, HD maps, background details, and road attributes.Each voxel also receives a class label.Source: §1 · pp. 2–3
  • The two-phase system first envisions a future world volume from a reference scene and vehicle controls, then generates multi-camera footage guided by that volume.The contributions also include scene editing alongside control-driven street-view video generation.Source: §1 · pp. 2–3
  • A temporal latent series is generated from encoded world volumes using a temporally refined UNet conditioned on vehicle controls, then decoded into a future 4D volume.CLIP and a 3D sparse CNN turn the volume into camera-specific image features; volume labels guide pixel characteristics, while concatenated surround-view images and temporal Transformer blocks support sensor and temporal consistency.Source: §1 · p. 3
  • WoVoGen uses explicit world volume to guide diffusion generation toward intra-world and inter-sensor consistency.The paper presents control-driven street-view video generation and scene editing as contributions of the system.Source: §1 · p. 3

2 Related work

Prior driving-scene synthesis uses either structured rendering or diffusion models with sparse controls, while related diffusion work also addresses text, layout, video, and novel-view generation. WoVoGen combines an explicit dense world volume with future-volume prediction and multi-camera video synthesis.

2 Related work

  • Diffusion research includes text- and layout-guided image generation, video generation emphasizing temporal consistency, and novel-view generation focused on diverse perspectives.These developments motivate spatial consistency in generated outputs.Source: §2 · pp. 3–4
  • WoVoGen's world-model branch predicts future volumes from past volumes and ego-car actions, and its synthesis branch uses those volumes to produce future videos.The framework caption describes deriving world features, sampling and aggregating image features, then applying panoptic diffusion.Source: Figure 2 · p. 4
  • Rendering-based driving-scene methods reconstruct world structures before rendering, while diffusion-based methods use sparse conditions such as HD maps and bounding boxes.Rendering can yield high-quality synthetic results but often lacks robust control; some diffusion approaches focus on single frames, while others adapt temporal modules for video coherence.Source: §2 · pp. 4–5
  • WoVoGen uses a self-organized dense world volume to generate geometrically consistent multi-view images and predicts future volumes without ground-truth conditions for every video frame.Its temporal world volumes represent driving environments and correlate with critical scene details in generated content.Source: §2 · p. 5

3 Method

WoVoGen combines a future-world-volume model with a world-volume-aware generator to produce multi-camera driving scenes. It encodes scene structure, vehicle actions, and camera-specific features to support coherent generation and editing.

3.2 Overall architecture

  • WoVoGen uses a future-world-volume branch conditioned on initial volumes and driving actions, then generates multi-camera videos from temporal world volumes.The world model predicts future volumes; the generation branch uses them to produce video outputs.Source: §3.2 · p. 6

3.3 World volume

  • The scene representation combines 3D semantic occupancy with road-map information in a voxel world volume.Road maps are constrained to the zero-height voxel plane and zero-padded along height.Source: §3.3 · p. 6

3.4 World model

  • Past world volumes and ego-vehicle actions condition diffusion of future latent volumes, with action tokens injected through cross-attention.The latent encoder and decoder remain frozen during world-model training; consecutive volumes are jointly modeled to enhance temporal consistency.Source: §3.4 · p. 7

3.1 Preliminary

  • The world-volume branch uses a latent diffusion model, while its temporal Transformer adds temporal attention to spatial Transformer blocks.The preliminary latent diffusion model encodes and decodes data through an autoencoder before modeling latent-space distributions.Source: §3.1 · p. 5, Figure 3 · p. 5, §3.4 · p. 7

3.5 World volume-aware 2D feature

  • CLIP features of world-volume classes are projected into each camera frustum, then depth-weighted and summed into world-volume-aware 2D image features.The camera frustum grid is defined by camera intrinsics and extrinsics.Source: §3.5 · p. 8

3.6 World volume-aware diffusion generation

  • Multi-view features are aggregated into a panoptic latent for diffusion, while text prompts guide scene style and object-specific cross-attention guides placement.Object guidance uses category masks projected from the initial world volume and category-specific CLIP features.Source: §3.6 · pp. 9–10

3.7 Video generation

  • Temporal attention is added for multi-frame generation; prompts control weather and location, while world-volume edits support object changes and camera-perspective rotation.The temporal attention block is trained while other blocks are frozen; editing can alter camera extrinsics.Source: §3.7 · pp. 10–11, Figure 5 · p. 10, Figure 6 · p. 11

4 Experiments

Experiments evaluate WoVoGen on nuScenes using six camera views, then assess world-volume prediction, image and video generation, editing, and consistency. Results cover both qualitative capabilities and quantitative comparisons and ablations.

4.1 Experimental setup

  • Experiments use nuScenes training and validation videos across six camera views, with images resized to 256 × 448 and six consecutive frames selected for training.Training uses eight NVIDIA A6000 GPUs; evaluation includes FID and FVD.Source: §4.1 · p. 12

4.2 Results

  • WoVoGen predicts future occupancy and HD maps while maintaining consistency across generated frames.The results describe both single-frame prediction and longer-term scene evolution.Source: 4D World volume generation · p. 12
  • Generated multi-camera images are consistent, correspond to occupancy guides, and respond to text prompts for rainy conditions and Singapore Onenorth.These are qualitative results shown in Figure 5.Source: Multi-camera single-frame image… · p. 13
  • Editing the world volume supports adding, deleting, or moving objects, as well as generating a scene under a changed camera setting.Camera extrinsics are changed when projecting the volume to the camera volume.Source: Multi-camera single-frame image… · p. 13
  • WoVoGen achieves the lowest FID and FVD scores among compared methods while supporting multi-view and multi-frame generation.Reported values include FID 27.6 and FVD 417.7; the cited image-generation comparison notes DriveGan and DriveDreamer are single-view methods.Source: Table 1 · p. 13, Multi-camera single-frame image… · p. 13, Multi-camera video generation · p. 13
  • Generated video maintains temporal coherence in object and background appearance throughout the sequence.The paper reports FVD of 417.7 for video quality.Source: Multi-camera video generation · p. 13

4.3 Ablation studies

  • World volume substantially improves generation over object guidance alone, and adding object guidance improves object localization.The study compares object guidance alone, world volume alone, and their combination.Source: Effectiveness of world volume and… · p. 14, Table 2 · p. 14
  • Up to 80.2% of keypoints match across successive frames, while 90.0% match between adjacent cameras under KPM evaluation.KPM uses a pretrained matching model to calculate average keypoint matches.Source: Consistency · p. 14, Table 2 · p. 14

5 Conclusion

WoVoGen uses a 4D world volume to generate multi-camera driving videos from vehicle controls while supporting scene editing.

5 Conclusion

  • WoVoGen integrates time with spatial data in a 4D world volume to generate vehicle-control-conditioned multi-camera videos and support complex scene editing.The conclusion presents the system as a tool for autonomous-driving technology development.Source: §5 · p. 14

A.1 Masks for object guidance

Object-guidance masks are built by projecting occupancy voxels onto the camera plane and marking image regions around projected points.

A.1 Masks for object guidance

  • Occupancy points are projected onto the camera imaging plane using camera intrinsics and extrinsics.The passage identifies class-labeled occupancy voxels as the projected points.Source: §A.1 · p. 18
  • Projected points mark square-shaped regions as occupied in an initially zero-valued camera-sized mask.The projection coordinates are normalized in x and y.Source: §A.1 · p. 18
  • The mask-region radius δ is inversely proportional to projected depth, with d set to 375.The paper illustrates generated mask samples in Figure 8.Source: §A.1 · p. 18

A.2 Zero-shot generation

Zero-shot examples test WoVoGen with conditions outside the dataset, including unseen weather, location, and object-color guidance. The examples show the model generating scenes that reflect these changes.

A.2 Zero-shot generation

  • The paper presents these examples as zero-shot generation outcomes.Source: §A.2 · p. 18
  • WoVoGen generates scenes with “heavy snow,” a weather condition absent from nuScenes, using the concept from Stable Diffusion pretraining.The authors describe this as combining pretrained concepts with fine-tuned street-view images.Source: §A.2 · p. 18
  • Changing the location prompt to Shanghai produces images with foggy-like weather and Shanghai-style highway architecture.The authors attribute this adaptation to concepts inherited from Stable Diffusion pretraining.Source: §A.2 · pp. 18–19
  • Replacing “car” with “purple car” changes all cars in the scene to purple, although purple cars are rare in the dataset.Source: §A.2 · p. 19

A.3 Downstream tasks

The downstream experiment adds generated data to nuScenes training and evaluates vehicle-class 3D object detection against BEVDet. The authors report improved mAP and reduced orientation error.

A.3 Downstream tasks

  • The experiment supplements nuScenes with generated training data and tests 3D object detection using BEVDet as the baseline.The evaluation focuses on vehicles because the world-volume classes include cars, trucks, and buses.Source: §A.3 · p. 19
  • In 3D object detection, adding generated data significantly improves mAP, with orientation error decreasing from 17.9 to 15.7; evaluation covers vehicle classes: cars, trucks, and buses.Source: §A.3 · p. 19, Table 3 · p. 19

A.4 Ablation studies

The ablation compares video consistency with and without temporal finetuning. The finetuned model better preserves object and background appearances across frames.

A.4 Ablation studies

  • Figure 7 presents examples of out-of-domain generation.Source: Figure 7 · p. 20
  • Temporal finetuning maintains more consistent object and background appearances across frames than the single-frame model.The comparison is illustrated using front- and back-camera results after removing temporal finetuning.Source: §A.4 · pp. 19–20

A.5 Video results

The appendix provides a video-results file and additional examples of video generation and single-frame editing. It also describes the world volume’s edit-friendly features as supporting varied, potentially rare generation outcomes.

A.5 Video results

  • The paper provides video results in the file 10410_video.mp4.Source: §A.5 · p. 21

A.6 Additional single-frame image generation results

  • The appendix presents additional single-frame samples of editing and controlling capabilities.Figure 10 contains these examples.Source: §A.6 · p. 21, Figure 10 · p. 22
  • The world volume’s Lego-style edit-friendly features support many possible edits and controls, including outcomes that may be rare in the real world.Source: §A.6 · p. 21
  • Figure 8 organizes the original images and per-class object masks by six camera views.The top row is front left, front, and front right; the bottom row is back right, back, and back left.Source: Figure 8 · p. 21
  • Additional figures show videos generated from generated world volumes.Figures 11–14 provide further examples.Source: Figure 11 · p. 23, Figure 12 · p. 24, Figure 13 · p. 25

Related papers

Loading 2312.02934v4…