PaperlayerView exact revision on arXiv

Source-linked AI summary

MagicDrive: Street View Generation with Diverse 3D Geometry Control

Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, Qiang Xu

arXiv:2310.02601v7cs.CVcs.AI

TL;DR

Street-view synthesis needs more precise 3D control than prevailing 2D and BEV-based methods provide for perception-oriented data generation. MAGICDRIVE separately encodes geometric conditions and uses cross-view attention, producing realistic, controllable multi-camera views that support BEV segmentation and 3D object detection augmentation, while remaining limited on unseen weather and fully realistic dark night scenes.

  • Problem

    Existing street-view generation methods provide limited 3D control, with BEV or 2D projections losing geometry such as height, depth, and occlusion needed for perception tasks.

  • Method

    MAGICDRIVE separately encodes camera poses, road maps, and 3D bounding boxes with text conditions, and adds cross-view attention for multi-camera consistency.

  • Results

    MAGICDRIVE generates realistic, controllable multi-camera street views and supports augmentation for BEV segmentation and 3D object detection.

  • Takeaways & Limitations

    The framework enables novel street-view synthesis with scene-, background-, and foreground-level controls while maintaining high realism and cross-camera consistency.

  • Takeaways & Limitations

    MAGICDRIVE produces night views that are not as dark as real images and cannot generate unseen weathers for nuScenes.

Abstract

from arXiv · show

Recent advancements in diffusion models have significantly enhanced the data synthesis with 2D control. Yet, precise 3D control in street view generation, crucial for 3D perception tasks, remains elusive. Specifically, utilizing Bird's-Eye View (BEV) as the primary condition often leads to challenges in geometry control (e.g., height), affecting the representation of object shapes, occlusion patterns, and road surface elevations, all of which are essential to perception data synthesis, especially for 3D object detection tasks. In this paper, we introduce MagicDrive, a novel street view generation framework, offering diverse 3D geometry controls including camera poses, road maps, and 3D bounding boxes, together with textual descriptions, achieved through tailored encoding strategies. Besides, our design incorporates a cross-view attention module, ensuring consistency across multiple camera views. With MagicDrive, we achieve high-fidelity street-view image & video synthesis that captures nuanced 3D geometry and various scene descriptions, enhancing tasks like BEV segmentation and 3D object detection.

2 RELATED WORK

Street-view generation has progressed from 2D layout conditioning toward 3D geometry, but preserving depth, height, occlusion, and cross-view consistency remains challenging.

  • Diffusion-based street-view methods commonly condition generation on 2D bounding boxes or semantic segmentation layouts.
  • BEVGen introduced BEV conditioning for roads and vehicles, but omitting height limits its use for 3D object detection.
  • BEVControl adds height through height lifting, while other approaches project 3D boxes into camera views.
  • Projecting 3D information into 2D can lose essential depth and occlusion geometry.
  • Multi-camera generation requires viewpoint consistency, but prior indoor-scene methods rely on view continuity that may not hold in street views.

3 PRELIMINARY

The paper formulates street-view generation around ego-centric geometric scene descriptions, camera poses, text, and noise, while situating MagicDrive within conditional latent diffusion. MagicDrive independently encodes road maps, object boxes, and camera parameters for geometry-guided synthesis with descriptive conditions.

  • Problem Formulation: A driving scene is represented in the LiDAR ego-car coordinate system using a BEV road map, 3D object boxes with classes, and descriptive text.The map contains semantic classes, boxes encode object geometry and categories, and text can describe weather or time of day.
  • Problem Formulation: Given scene description S and camera pose P, the generator synthesizes a realistic image I from Gaussian noise.The camera pose includes intrinsics, rotation, and translation, and the output image has height, width, and three color channels.
  • Conditional Diffusion Models: Diffusion models generate data by iteratively denoising Gaussian noise and train the denoising network with a mean-square error objective.The conditional formulation incorporates optional conditions, diffusion time steps, and additive Gaussian noise.
  • Conditional Diffusion Models: Latent diffusion performs the diffusion process in the latent space of a pretrained VQ-VAE and can use image-describing text as a condition.The encoder maps the input image into a latent representation before conditional noise prediction.
  • MagicDrive Overview: MagicDrive independently encodes road maps, object boxes, and camera parameters to guide realistic street-view synthesis with geometric and descriptive conditions.The framework also accommodates scene guidance such as weather descriptions.

4 STREET VIEW GENERATION WITH 3D INFORMATION

MAGICDRIVE encodes scene-level, object-level, and road-map conditions for controllable street-view synthesis, while cross-view attention promotes consistency across camera perspectives.

  • Scene and condition encoding: MAGICDRIVE generates street-view images conditioned on scene annotations and camera pose for each view.Its inputs include diverse 3D geometric information rather than relying only on a projected BEV representation.
  • Scene and condition encoding: Scene-level encoding combines camera pose and text embeddings, using Fourier embedding and an MLP for pose parameters.The camera pose includes intrinsic and extrinsic parameters, while text is encoded with a pretrained CLIP text encoder.
  • Geometric condition encoding: 3D bounding boxes are encoded by combining class-label embeddings with Fourier-embedded corner-point positions into hidden vectors.Visible boxes are filtered per view, and invisible boxes are added for augmentation.
  • Geometric condition encoding: Road maps use an additive encoder branch suited to grid-like data, avoiding explicit BEV-to-FPV view transformation.The design relies on available 3D cues, including object-box height and camera pose, to support this encoding.
  • Cross-view attention: Cross-view attention lets each target view access adjacent left and right views, then aggregates their information through a skip connection.It is inserted after the UNet cross-attention module; stacked UNet blocks can propagate information from longer-range views.
  • Training and augmentation: Training randomly adds 10% invisible boxes and uses unique noise for different views to support geometric transformation and cross-view attention learning.The training objective is adapted to the multi-condition setting, with classifier-free guidance dropping condition groups during training.

5 EXPERIMENTS

Experiments evaluate MAGICDRIVE on nuScenes for realism, controllability, multi-level control, and video extension. The framework outperforms baselines in generation fidelity and supports perception-data augmentation.

  • Evaluation setup: MAGICDRIVE is evaluated on nuScenes using FID for realism and CVT and BEVFusion for BEV segmentation and 3D object detection controllability.The setup uses 700 training scenes and 150 validation scenes, with BEVGen and BEVControl as generation baselines.
  • Main results: MAGICDRIVE surpasses baseline methods in generation fidelity and provides accurate alignment with 3D annotations.At 224×400 resolution it equals or exceeds baselines on BEV segmentation, while higher-resolution results improve vehicle mIoU; cropping harms road mIoU.
  • Perception-data augmentation: MAGICDRIVE synthetic data improves both BEV segmentation and 3D object detection training, including CAM-only and CAM+LiDAR BEVFusion settings.The synthetic images integrate with LiDAR inputs, indicating fidelity sufficient for multimodal detection augmentation.
  • Qualitative comparison: Compared with BEVControl, MAGICDRIVE produces higher-quality images with more accurate object positioning and stronger background and foreground consistency.The authors attribute this performance primarily to the bounding-box encoder and cross-view attention module.
  • Multi-level controls: MAGICDRIVE supports scene-, background-, and foreground-level controls while maintaining multi-camera consistency and high realism.Controls include time of day and weather, BEV-map and view changes, and object orientation or deletion.
  • Video generation: MAGICDRIVE extends to video generation by adding temporal attention and training on 7-frame clips, with interpolated annotations enabling 16-frame generation at 12Hz.Only the first and last frames receive bounding-box controls in the 7-frame setup.

6 ABLATION STUDY

Ablations examine separate box encoding, classifier-free guidance, and video-oriented design choices. Results show that object-specific encoding improves controllability, while CFG introduces condition-specific trade-offs.

  • Bounding-box encoding: Separate bounding-box encoding improves vehicle mIoU compared with using a BEV map that jointly represents road and object semantics.The visible-object filter further improves performance by reducing irrelevant box inputs.
  • Classifier-free guidance: Increasing CFG from 1.5 to 4.0 degrades FID because of notable changes in image contrast and sharpness.The effect differs across conditions and metrics rather than uniformly improving generation.
  • Video extension: Video generation modifies self-attention to ST-Attn and adds temporal attention within each transformer block.The model is fine-tuned on 7-frame clips, with bounding boxes provided only for keyframes.
  • Classifier-free guidance: At CFG=2.5, vehicle mIoU is highest, whereas road mIoU decreases as CFG increases when the same map is used conditionally and unconditionally.Using M = {0} for unconditional inference raises road mIoU but slightly weakens vehicle guidance.

7 CONCLUSION

MAGICDRIVE combines separated geometric encodings with cross-view attention to generate realistic, controllable multi-camera street views. It also supports augmentation for perception tasks, but remains limited on very dark and unseen-weather scenes.

  • Conclusion: MAGICDRIVE encodes multiple geometric controls to generate high-quality multi-camera street views faithful to 3D annotations.The framework uses separation encoding for geometric information and cross-view attention for camera consistency.
  • Conclusion: The framework’s multiple controls improve generalizability for generating novel street views and support data augmentation for BEV segmentation and 3D object detection.These capabilities are presented within the scope of perception-model training and simulation-oriented synthesis.
  • Limitations: MAGICDRIVE night views are not as dark as real images, and the model cannot generate unseen weather conditions in nuScenes.The authors identify cross-domain generalization as a direction for future work.
  • Object filtering: Object filtering retains a box when at least one box corner lies in front of the camera in its coordinate system.The filter ignores each camera’s FOV for simplicity and is used to facilitate bootstrap learning.

B MORE EXPERIMENTAL DETAILS

The implementation covers semantic classes for objects and road maps and specifies the optimization configuration used to train newly added parameters.

  • Semantic classes: MAGICDRIVE represents ten object categories, including cars, buses, trucks, pedestrians, barriers, and traffic cones.These semantics are selected to support common nuScenes perception-model settings.
  • Semantic classes: The road-map condition contains eight semantic categories, including drivable area, pedestrian crossing, walkway, stop line, and car parking area.The passage lists these categories as part of the generation semantics.
  • Optimization: Newly added parameters are optimized with AdamW at a constant learning rate of 8e−5 and batch size 24.Training uses a 3000-iteration linear warm-up and sets γs = 0.2.

C ABLATION ON NUMBER OF ATTENDING VIEWS

The ablation shows a trade-off between per-view image quality and multi-camera consistency as the number of attended views changes. MAGICDRIVE outperforms BEVGen in qualitative object image quality.

  • Ablation on number of attending views: Attending to one view yields superior FID but compromises mIoU and multi-camera consistency.Neighboring-view information simplifies optimization for the attended view, but one view cannot guarantee consistency, especially for foreground objects.
  • Ablation on number of attending views: Attending to all views deteriorates performance across all metrics, potentially because excessive information interferes in cross-attention.The authors select two attended views as a balance between consistency and computational efficiency.
  • Ablation on number of attending views: Figure 11 confirms that attending to only one view produces worse multi-camera consistency.
  • Comparison with BEVGen: MAGICDRIVE generates higher-quality images than BEVGen, particularly for objects.The improvement is attributed to the diffusion model and customized condition injection strategy.

E MORE RESULTS WITH CONTROL FROM DIFFERENT CONDITIONS

MAGICDRIVE provides scene-level and background-level control, and generated camera views reflect changes in time of day and BEV map conditions.

  • More results with control from different conditions: MAGICDRIVE reflects scene-level time-of-day changes and background-level BEV map alterations in generated camera views.
  • More results with control from different conditions: Figure 13 showcases scene-level control using time of day and background-level control using BEV map alterations.

F MORE EXPERIMENTS WITH 3D OBJECT DETECTION

Additional experiments evaluate MAGICDRIVE-generated images as augmentation for BEVFusion 3D object detection. The paper identifies reduced-epoch training as a way to mitigate performance saturation and states that generated views currently require real-data augmentation.

  • More experiments with 3D object detection: MAGICDRIVE generates 272×736 images for BEVFusion augmentation on the nuScenes validation set.
  • More experiments with 3D object detection: Reducing training epochs mitigates saturation and allows more varied synthetic data to enhance BEVFusion perceptual capacity in CAM-Only and CAM+LiDAR settings.Performance appears to plateau by 12× epochs with the lightweight Swin-T backbone; the passage gives 2× settings of 20 and 6 epochs for CAM-Only and CAM+LiDAR.
  • More experiments with 3D object detection: MAGICDRIVE-generated street views currently serve only as augmented samples trained together with real data.Training detectors solely with generated data is identified as future work.

H DETAILED ANALYSIS ON 3D OBJECT DETECTION WITH SYNTHETIC DATA

Detailed analyses examine per-class 3D detection effects, BEV-map fidelity, camera-parameter generalization, and additional qualitative generations. Improvements vary by object class, while camera generalization remains limited by fixed nuScenes poses.

  • 3D object detection: Per-class detection improvements are significant for large objects and less diverse classes, marginal for cars, and significant for pedestrians, motorcycles, and bicycles.The authors attribute gains for smaller vulnerable classes partly to high-quality synthesis near the ego car.
  • 3D object detection: Overall mAP improvement is accompanied by higher AP across all classes, indicating benefits for perception-model training.
  • BEV segmentation: The 272×736 resolution does not outperform 224×400 for BEV segmentation, supporting preservation of the original aspect ratio during generation training.The passage links better map control to avoiding cropping on each side.
  • Generalization of camera parameters: MAGICDRIVE encodes raw camera intrinsic and extrinsic parameters, but generalization is limited because nuScenes fixes camera poses across scenes.Front and back views differ for identical 3D coordinates because camera positions are asymmetric and fields of view differ.
  • Qualitative generations: Figures 15–18 present corner-case and nuScenes validation-set generations from MAGICDRIVE.
Loading 2310.02601v7…