Source-linked AI summary
Street-View Image Generation from a Bird's-Eye View Layout
Alexander Swerdlow, Runsheng Xu, Bolei Zhou
TL;DR
BEV-conditioned generation of realistic street-view images is less explored than predictive BEV perception, despite its relevance to representing traffic scenarios. BEVGen generates spatially consistent views from BEV layouts, and the authors report improved synthesis quality and semantic consistency over baselines.
Problem
Generating realistic street-view images from BEV layouts is a less-explored counterpart to predictive BEV perception, with the task requiring images to correspond to the HD map and remain consistent across views.
Source: §1 · pp. 1–2Method
BEVGen is an autoregressive generative model that uses camera-aware spatial embeddings and a novel attention bias and decoding scheme to generate a set of spatially consistent images from a BEV layout.
Source: §1 · p. 2Results
BEVGen substantially improves image-synthesis quality and semantic consistency over baselines; on nuScenes, its FID is 25.54 versus the baseline’s 138.30.
Source: §1 · p. 2, §4.3 · p. 5Takeaways & Limitations
The authors show BEVGen’s application to data augmentation and simulated BEV generation.
Source: §6 · p. 8The authors identify improving small-object synthesis as future work.
Source: §4.3 · p. 6
Abstract
from arXiv · showhide
Bird's-Eye View (BEV) Perception has received increasing attention in recent years as it provides a concise and unified spatial representation across views and benefits a diverse set of downstream driving applications. At the same time, data-driven simulation for autonomous driving has been a focal point of recent research but with few approaches that are both fully data-driven and controllable. Instead of using perception data from real-life scenarios, an ideal model for simulation would generate realistic street-view images that align with a given HD map and traffic layout, a task that is critical for visualizing complex traffic scenarios and developing robust perception models for autonomous driving. In this paper, we propose BEVGen, a conditional generative model that synthesizes a set of realistic and spatially consistent surrounding images that match the BEV layout of a traffic scenario. BEVGen incorporates a novel cross-view transformation with spatial attention design which learns the relationship between cameras and map views to ensure their consistency. We evaluate the proposed model on the challenging NuScenes and Argoverse 2 datasets. After training, BEVGen can accurately render road and lane lines, as well as generate traffic scenes with diverse different weather conditions and times of day.
1. Introduction
BEVGen addresses the underexplored task of generating street-view images from a bird’s-eye view (BEV) layout. It is designed to preserve map correspondence and consistency across views while supporting controllable scene generation.
1. Introduction
- BEV-conditioned street-view generation is far less explored than predictive BEV perception, which maps street-view inputs to semantic BEV layouts.A BEV layout concisely represents a traffic scenario at the semantic level.Source: §1 · p. 1
- A controllable generative model offers a simpler alternative to simulator- or mesh-based synthetic data generation and can produce diverse images from editable BEV layouts for corner-case stress tests.The authors also describe synthetic training data and visualization of safety-critical situations as applications.Source: §1 · p. 1
- Generated views must remain consistent across overlapping camera fields of view and match the HD map’s roads, lane lines, and vehicle locations.Unlike semantic-mask translation, BEV generation must infer image layout to account for occlusions and objects’ relative heights.Source: §1 · p. 2
- BEVGen uses camera-aware spatial embeddings and a novel attention bias and decoding scheme to generate spatially consistent images, with substantial gains over baselines in image quality and semantic consistency.The model can also render scenes from out-of-domain BEV maps, including maps provided by a driving simulator or edited by a user.Source: §1 · p. 2
2. Related Work
Prior driving-scene simulation includes learned generative approaches, but the cited work does not address multi-view generation with arbitrary actor and object placement. BEV generation also differs from image-generation and novel-view-synthesis tasks because it must infer spatial constraints from a BEV layout and remain consistent with an HD map.
2. Related Work
- Prior driving-simulation work moved from handcrafted game-engine assets toward learned generative models [7], while demonstrated data-driven approaches lacked multi-view generation and arbitrary actor and object positioning.Source: §2 · p. 2
- Image generation conditioned on semantic masks or bounding boxes uses spatial constraints, whereas BEV generation must infer them from another perspective and requires 3D scene understanding.Source: §2 · p. 2
- Unlike novel view synthesis, BEV generation has no source image and must maintain consistency both across views and with an HD map.Source: §2 · p. 2
3. Method
BEVGen generates multi-view street images from a semantic BEV layout and camera parameters, using discrete representations and spatially informed attention to connect map and camera tokens.
3. Method
- BEVGen encodes a semantic BEV layout and camera parameters to generate n images, modeling scenes with two VQ-VAEs and an autoregressive transformer.The BEV and source images are encoded into discrete representations before transformer processing and image decoding.Source: §3 · p. 2, Figure 2 · p. 3
3.1. Model Structure
- Image VQ-VAE representations use codebook tokens, while the BEV is encoded similarly and its decoder is discarded after first-stage training.The paper notes diffusion could improve synthesis quality but would require an order of magnitude more training data and computational resources.Source: §3.1 · p. 3
- The authors leave diffusion models for future work because training them requires an order of magnitude more data and computational resources.They state such models could potentially improve image synthesis quality.Source: §3.1 · p. 3
- A transformer predicts image tokens from prior image tokens, discretized BEV features, and camera parameters, using causal image attention and global attention to BEV tokens.The authors choose global attention to aid cross-view consistency.Source: §3.1 · p. 3
3.2. Spatial Embeddings
- Spatial embeddings use camera intrinsics and extrinsics to align image tokens with BEV tokens, mapping camera directions and BEV positions into the ego frame.The embeddings provide the transformer with aligned camera and BEV representations.Source: §3.2 · p. 4
3.3. Camera Bias
- The attention bias combines direction-vector cosine similarity with a learnable pairwise term for image-image and image-BEV token relationships.For image-BEV pairs, the BEV direction uses ego-centered world coordinates, while the image direction uses the image-center row.Source: §3.3 · p. 4
4. Experiments
Experiments on nuScenes and Argoverse 2 evaluate BEVGen’s image quality, BEV-layout fidelity, cross-view consistency, and design choices.
4.1. Datasets
- Evaluation uses nuScenes and Argoverse 2 multi-view camera data with projected 3D boxes; nuScenes training is expanded to roughly 260k instances by resampling.The BEV layouts cover 80m × 80m; camera images are resized to 224×400 on nuScenes and cropped to 256×256 on Argoverse 2.Source: §4.1 · pp. 4–5
4.2. Training Details
- The GPT-like transformer uses 16 heads and 24 layers; the sparse model attends to all BEV tokens, nearby image regions, and a 96-token sliding window.Sparse attention has 35% mask density; except in the ablation study, the sparse model is fine-tuned from the full-attention model for 10 epochs.Source: §4.2 · p. 5
4.3. Results
- On nuScenes validation across all six views, BEVGen achieves FID 25.54 versus baseline 138.30 and reference upper-bound FID 9.37; on Argoverse 2, it achieves FID 25.51.Source: Table 1 · p. 5, §4.3 · p. 5
- Generated scenes span varied locations, weather, and times of day; the same BEV layout can produce day and night views, and vehicles remain coherent across views.The Argoverse 2 comparison shows vehicles at the same location as in the corresponding real views, including when visible in multiple views.Source: §4.3 · p. 5, Figure 3 · p. 6
- The sparse-mask variant scores 28.67 FID, while running about 48% faster at inference and 40% faster in training.The reference upper-bound comparison largely removes the effect of the first stage, which is not the paper’s focus.Source: §4.3 · p. 5
- BEVGen beats its baseline by 4.4 road-class and 1.45 vehicle-class mIoU points, but vehicle synthesis remains limited, especially for small objects.The segmentation network’s validation mIoU is 66.31 for road and 27.51 for vehicles.Source: §4.3 · p. 6
- The View Consistency Score is 5.65 for synthesized versus 12.86 for real nuScenes images, and 13.00 versus 42.90 on Argoverse 2.VSC counts correspondences in overlapping regions but does not measure whether viewpoints have the proper perspective shift.Source: §4.3 · p. 7, Figure 5 · p. 7
4.4. Ablation Study
- Ablations show steady FID decreases and VSC increases as camera bias and spatial embeddings are added, using three front-facing nuScenes views.The study compares center-out decoding, camera bias, camera bias with spatial embeddings, and a sparse-mask model trained from scratch.Source: §4.4 · p. 7
5. Applications
BEVGen is applied to procedural MetaDrive scenarios, edited safety-critical scenes, and synthetic-data augmentation for BEV perception. The reported augmentation improves both segmentation and detection on nuScenes.
Image generation from simulated BEV
- BEVGen synthesizes images from procedural MetaDrive BEV layouts, demonstrating a possible bridge to sim2real; long-tail scenario generation itself is outside the work's scope.Figure 6 shows images generated from MetaDrive layouts.Source: §5 · p. 7, Figure 6 · p. 8
Safety-critical scenario editing
- Editing existing scenes by translating cars generates safety-critical scenarios, as illustrated with two scenes in Figure 7.The authors describe editing as supporting permutation testing and construction of difficult scenarios for system observation.Source: §5 · p. 7, Figure 7 · p. 8
Data augmentation for BEV segmentation and detection
- BEVFormer detection gains are 1.6 mAP and 1.3 NDS for Tiny, and 2.9 mAP and 2.7 NDS for Small; road and vehicle segmentation mIoU each improve by 0.6.These are validation results on nuScenes using synthetic-data augmentation.Source: Table 3 · p. 8, §5 · p. 8
6. Conclusion
The paper introduces BEV generation and BEVGen, a model trained on real-world driving datasets to generate spatially consistent multi-view images from a BEV layout. It also demonstrates applications in data augmentation and simulated BEV generation.
Conclusion
- The paper introduces BEV generation as a task and proposes the generative model BEVGen.The model is trained on real-world driving datasets.Source: §6 · p. 8
- After training, BEVGen generates spatially consistent multi-view images from a given BEV layout.Source: §6 · p. 8
- The paper demonstrates BEVGen's application in data augmentation and simulated BEV generation.Source: §6 · p. 8