Source-linked AI summary
Panacea: Panoramic and Controllable Video Generation for Autonomous Driving
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, Xiangyu Zhang
TL;DR
Autonomous-driving research needs large, diverse, annotated video data, but acquiring and annotating such datasets is difficult. Panacea generates controllable panoramic multi-view driving videos from BEV layouts and text using decomposed 4D attention and a two-stage pipeline. Evaluations on nuScenes report high-quality synthesis and improved BEV-perception training utility, while image-to-video elevation shows a trade-off between NDS and mAP.
Problem
High-quality, large-scale annotated driving videos are difficult and sometimes risky to acquire across varied weather, environments, and lighting conditions.
Method
Panacea uses a two-stage generation framework with decomposed 4D attention, ControlNet-based BEV conditioning, and text control to create coherent panoramic multi-view driving videos.
Results
Panacea produces high-quality driving-scene videos and synthesized instances that substantially enhance state-of-the-art BEV perception models.
Takeaways & Limitations
The synthesized videos can enrich existing video datasets and potentially elevate image-only datasets to video datasets for video-based perception techniques.
Takeaways & Limitations
When elevating image-only data, NDS increased by 5.8 points but mAP decreased by 2.3 points, which the authors associate with generated-sample quality and domain mismatch.
Abstract
from arXiv · showhide
The field of autonomous driving increasingly demands high-quality annotated training data. In this paper, we propose Panacea, an innovative approach to generate panoramic and controllable videos in driving scenarios, capable of yielding an unlimited numbers of diverse, annotated samples pivotal for autonomous driving advancements. Panacea addresses two critical challenges: 'Consistency' and 'Controllability.' Consistency ensures temporal and cross-view coherence, while Controllability ensures the alignment of generated content with corresponding annotations. Our approach integrates a novel 4D attention and a two-stage generation pipeline to maintain coherence, supplemented by the ControlNet framework for meticulous control by the Bird's-Eye-View (BEV) layouts. Extensive qualitative and quantitative evaluations of Panacea on the nuScenes dataset prove its effectiveness in generating high-quality multi-view driving-scene videos. This work notably propels the field of autonomous driving by effectively augmenting the training dataset used for advanced BEV perception techniques.
1. Introduction
Panacea targets the difficulty of obtaining large, diverse, annotated driving videos by generating controllable multi-view video from BEV sequences and text. Its two-stage design and decomposed 4D attention address spatial, cross-view, and temporal consistency, with evaluations reporting high-quality synthesis and gains for BEV perception training.
- High-quality, large-scale annotated video datasets are difficult and sometimes risky to acquire across varied weather, environments, and lighting conditions.
- Panacea generates synthetic multi-view driving videos conditioned on BEV layout sequences and descriptive text prompts.The BEV sequences provide 3D bounding boxes and road maps for controllable generation.
- The approach treats controllability and consistency as core criteria: inputs govern generated content, while videos remain coherent across frames and views.
- Panacea uses a two-stage system, decomposed 4D attention, and ControlNet to produce realistic, controllable panoramic driving videos.The first stage creates multi-view images, while the second expands them temporally into video sequences.
- Comprehensive evaluations report high-quality driving-scene videos and substantial enhancement of state-of-the-art BEV perception models using synthesized instances.The authors plan to release the synthesized instances as the Gen-nuScenes dataset.
2. Related Work
Prior work established diffusion-based image and video generation, BEV-conditioned image synthesis, and video-based BEV perception. Panacea differs by generating controllable multi-view driving videos paired with BEV layout sequences.
- Diffusion research progressed from latent-space image generation and additional control signals to temporal video-generation models.
- Panacea focuses on controllable multi-view videos in driving contexts, unlike previous work centered on other video-generation settings.
- BEV representations support downstream autonomous-driving tasks including multi-object tracking, motion prediction, and planning.
- Earlier autonomous-driving generators used BEV layouts mainly to augment single-frame perception with synthetic single- or multi-view images.
- The work presents the first exploration described here of generating multi-view videos paired with BEV layout sequences for video-based BEV perception.
3. Method
Panacea upgrades latent diffusion for controllable multi-view video generation using decomposed 4D attention, a two-stage pipeline, and BEV- and text-based controls. The design separates spatial, view, and temporal processing to improve feasible coherent synthesis.
- Decomposed 4D Attention: The attention design separates intra-view spatial processing, cross-view interactions, and cross-frame temporal processing.Cross-view attention emphasizes adjacent views, while cross-frame attention provides temporal awareness.
- Generating High-Quality Multi-View Videos: Panacea builds on latent diffusion and uses a decomposed 4D attention-based UNet to jointly generate multi-view video sequences.The sequence is arranged by concatenating views across width, with dimensions H × (W × V) × T × C.
- Decomposed 4D Attention: The method is designed to avoid the excessive memory and computation required by exhaustive 4D attention over all views and frames.
- Two-Stage Generation: A two-stage training and inference pipeline first generates multi-view images, then expands them into videos conditioned on the generated initial frames.Training uses ground-truth images as the second-stage condition, while inference uses first-stage generated frames.
- Controllable Generation: Panacea integrates coarse text control and fine-grained BEV layout control for diverse scenes and annotation-aligned synthesis.BEV layouts are converted into 19-channel controls containing depth, bounding boxes, road maps, and camera-pose embeddings.
4. Experiment
Panacea is evaluated on nuScenes for generation quality, BEV controllability, synthetic-data augmentation, image-to-video elevation, and consistency mechanisms. Results show strong video quality and alignment, while ablations identify cross-view attention, temporal attention, and the two-stage pipeline as important contributors.
- 4.1. Datasets and Evaluation Metrics: Panacea is evaluated on the nuScenes dataset using FID, FVD, VMS, and perception-based controllability measures.nuScenes contains 1,000 scenes with six overlapping camera views; FID measures image quality, FVD captures image quality and temporal consistency, and VMS measures cross-view consistency.
- 4.3.1 Quantitative Analysis: 139 FVD and 16.96 FID demonstrate Panacea’s superior generation quality against video- and image-based baselines.The comparison uses the complete validation set without post-processing sample selection.
- 4.3.1 Quantitative Analysis: 68% relative performance indicates strong alignment between generated videos and their conditioned BEV sequences.The perception-based evaluation uses a pretrained StreamPETR model and compares generated-data performance with real-data perception scores.
- 4.3.1 Quantitative Analysis: 36.1% NDS from Gen-nuScenes alone reaches 77% relative performance, while combining synthetic and real data reaches 49.2 NDS, 2.3 points above real-only training.Gen-nuScenes is used as an auxiliary synthetic training resource, and Figure 4 examines augmentation across 25%, 50%, 75%, and 100% real-data ratios.
- 4.3.1 Quantitative Analysis: Image-only data elevation increases NDS by 5.8 points but decreases mAP by 2.3 points.The authors hypothesize that lower generated-sample quality and domain mismatch contribute to the mAP degradation.
5. Conclusion
Panacea generates manipulable panoramic driving videos using decomposed 4D attention, two-stage training, and diverse control signals. Experiments show high-quality, well-annotated videos useful for BEV perception and potentially real-world driving simulation.
- Panacea generates manipulable panoramic driving videos with decomposed 4D attention for temporal and cross-view consistency.The framework targets realistic multi-view images and videos while preserving coherence across views and time.
- A two-stage training strategy further enhances generation quality, while diverse control signals support precise annotations.
- Panacea’s high-quality, well-annotated videos are intended for BEV perception and may support real-world driving simulations.
A. More Implementation Details
The implementation details describe decomposed 4D attention, camera-pose conditioning, and StreamPETR training settings. Camera pose is encoded as a 3D direction-vector control signal to support multi-view consistency.
- Decomposed 4D Attention: Decomposed 4D attention inserts text context into intra-view, inter-view, and cross-frame blocks through cross attention.
- Construction of Camera Pose: Camera pose is encoded into a 3D direction vector and added to control signals to facilitate consistency across multi-view images.
- Construction of Camera Pose: The camera-ray construction uses pixel coordinates, depth, and homogeneous coordinates, with points assigned depths d1 = 1 and d2 = 2.
- Training Details of StreamPETR: StreamPETR training uses 512×256 resolution, batch size 16, and learning rate 4e-4; the single-frame baseline disables query propagation.
B. More Experimental Results
Additional results compare BEV-control designs and provide structural illustrations for decomposed 4D attention and pseudo-color camera-pose images. ControlNet remains the default because the simpler concatenation alternative performs worse on FVD and FID.
- BEV Sequence Control: Using BEV-feature concatenation instead of ControlNet produces FVD and FID values 87 and 3.19 higher, respectively.The comparison motivates retaining ControlNet as the default design for BEV control.
- Decomposed 4D Attention: Figure 8 illustrates the structure of the decomposed 4D attention module.
- Camera Pose: Figure 9 visualizes the camera pose as a pseudo-color image.
C. More Visualization Results
The visualization results show multi-view videos with eight frames and six views per video, alongside examples where attributes are varied under controllable generation.
- Multi-View Video Generation: Validation visualizations contain 8 frames and 6 views per generated video, showing temporal and cross-view consistency.
- Attribute Controllable Video Generation: Attribute-controlled visualizations present the same case with varying attributes.
D. More Related Works
Driving-scene synthesis methods include generative models and NeRF-based approaches, with Panacea visualizations covering multi-view videos, frames, and synthetic-data applications.
- Driving-Scene Synthesis Methods: Driving-scene synthesis methods comprise GAN- or diffusion-based generative methods and NeRF-based methods such as Unisim, Neuralsim, and Mars.NeRF-based methods can achieve high generation quality but reconstruct pre-existing training scenes, limiting diversity.
- Panacea Visualizations: Panacea includes visualizations of generated multi-view driving videos.
- Evaluation Materials: The paper reports a per-class comparison involving data augmentation with synthetic data, including construction vehicles.
- Panacea Visualizations: Panacea also presents multi-view frames generated under different attribute controls.
E. Gen-nuScenes Dataset
Gen-nuScenes is a synthetic training dataset created to enhance Stream-PETR training with 139,440 videos of eight frames each.
- Dataset Purpose: Gen-nuScenes is a newly synthesized training dataset designed to enhance Stream-PETR.
- Dataset Scale: The dataset contains 139440 videos.
- Video Format: Each Gen-nuScenes video contains 8 frames.
F. Limitations
The authors identify unresolved limitations in video quality, consistency, inference efficiency, and spatial resolution, especially for longer videos.
- Video Quality: Panacea’s generated-video quality still leaves room for improvement.
- Consistency: Temporal and view consistency are not perfect because correlating all views and frames is challenging, particularly for long sequences.
- Efficiency: Panacea has relatively high inference computational cost, motivating future efficiency improvements.
- Spatial Resolution: The method currently uses relatively low spatial resolution because of time and resource limitations.