Source-linked AI summary
MIVIFI: Bridging Perspective and Fisheye Domains for Training Multi-View Fisheye Image Generation Models
Matthias Neuwirth-Trapp, Begüm Altunbas, Jiayi Wang, Yan Xia, Maarten Bieshaar, Xinyu Huang, Daniel Cremers
TL;DR
The paper addresses scarce, distortion-heavy multi-view fisheye data for controllable autonomous-driving image synthesis. It introduces SyntheOcc-FE and ERP-based MIVIFI, whose experiments show high-quality generation and cross-domain support for diverse scene editing, while sequence consistency remains outside the current setup.
Problem
Multi-view fisheye generation is underexplored because fisheye datasets are scarce and severe lens distortion complicates transferring perspective-domain generative methods.
Method
The paper introduces SyntheOcc-FE for fisheye-only generation and MIVIFI, which uses shared Equirectangular Projection space to train with fisheye and perspective images.
Results
Experiments demonstrate high-quality multi-view fisheye generation, with SyntheOcc-FE emphasizing geometric fidelity and MIVIFI emphasizing generation diversity and semantic manipulation.
Takeaways & Limitations
MIVIFI provides a practical cross-domain strategy for extending limited fisheye datasets and generating rare edge cases such as nighttime and rainy scenes.
Takeaways & Limitations
Consistent sequence editing is not supported in the current setup and would require conditioning on previous frames.
Abstract
from arXiv · showhide
Achieving 360° coverage is critical for the visual perception systems of autonomous vehicles. Fisheye cameras offer a cost-effective solution by enabling full surround coverage with as few as two sensors. However, existing multi-view fisheye datasets are limited, and synthesizing rare corner cases typically requires computationally expensive 3D simulations, hindering the training. While generative models have achieved significant success in standard perspective imagery, their application to wide-angle distortion remains unexplored. In this work, we formally introduce the novel problem of multi-view fisheye image generation conditioned on volumetric semantic representations and present two distinct methods. We first propose SyntheOcc-FE, which adapts the SyntheOcc architecture to fisheye data. While effective, this method is constrained by the scarcity of fisheye datasets, which limits its generalization. To overcome these limitations, we propose our second method, MIVIFI (multi-view fisheye), which leverages cross-domain learning with Equirectangular Projections. By bridging the gap between dataset domains using KITTI-360 fisheye images alongside nuScenes multi-view standard images, our approach enables high-fidelity manipulation of scene content. This framework enables the structural modification of semantic occupancy inputs to introduce or eliminate specific actors and facilitates the rendering of diverse meteorological conditions and illumination scenarios absent in the limited fisheye datasets. Quantitative and qualitative experiments demonstrate that our methods achieve robust photorealistic multi-view fisheye image generation and highlight the specific advantages of our cross-domain strategy for handling data scarcity.
I. INTRODUCTION
The paper introduces controllable multi-view fisheye image generation conditioned on 3D semantic occupancy, addressing scarce fisheye data and the domain gap with perspective imagery. It presents SyntheOcc-FE and MIVIFI, with MIVIFI using ERP-based cross-domain training for diverse scene editing and environmental conditions.
- Fisheye cameras provide expansive 360° coverage with limited hardware, but richly annotated multi-view fisheye datasets remain scarce compared with perspective datasets.
- Existing occupancy-conditioned generative models offer spatial scene control, but multi-view fisheye synthesis remains difficult because of severe lens distortion and a perspective–fisheye domain gap.
- Experiments report high-quality generation from both methods, with SyntheOcc-FE emphasizing geometric fidelity and MIVIFI emphasizing diversity and semantic manipulation.
- The paper formulates controllable multi-view fisheye image generation conditioned on 3D semantic occupancy grids.
- SyntheOcc-FE adapts latent diffusion to fisheye data, enforcing fisheye geometry and wide-angle consistency, while its limited training diversity constrains long-tail synthesis.
- MIVIFI uses shared Equirectangular Projection space to train jointly on fisheye and perspective images, transferring weather and lighting attributes while supporting structural scene edits.
II. RELATED WORK
Prior generative methods advanced stable, diverse, and spatially controlled image synthesis, including multi-view generation for autonomous driving. However, these approaches primarily target perspective cameras and do not model fisheye-specific nonlinear distortion.
- GANs often suffered training instability and limited mode coverage, while diffusion models enabled stable and diverse generation through iterative latent-space denoising.
- ControlNet and related 3D-aware methods inject spatial or semantic structure to guide synthesis and maintain geometric consistency across multiple perspective cameras.
- Existing autonomous-driving generation methods are designed for perspective cameras and do not account for fisheye lenses’ severe nonlinear distortion and wide field of view.
C. Fisheye Image Generation
SyntheOcc-FE adapts a diffusion pipeline to fisheye geometry through fisheye-aware MPI conditioning, semantic text enrichment, and sensor-specific post-processing. The method maps occupancy semantics into distorted image coordinates before generating multi-view fisheye images.
- SyntheOcc-FE comprises geometric conditioning, semantic enrichment, and domain-aware fine-tuning stages.
- The method discretizes scenes into N-layer Multi-Plane Images and modifies MPI generation to account explicitly for wide-angle distortion.
- Stable Diffusion components initialized on nuScenes are augmented with vision-language captions, then fine-tuned on fisheye data to align generation with target sensor characteristics.
- Geometric Conditioning via Fisheye MPI: The Mei fisheye model replaces perspective projection, mapping voxel-grid semantic labels onto the distorted image plane through depth-specific rays.
- Network Training and Post-Processing: Text prompts and fisheye MPI jointly condition the diffusion model, while a binary mask can blacken mechanically vignetted regions to enforce sensor realism.
B. Second Method: MIVIFI
MIVIFI bridges fisheye and perspective datasets by representing both in a shared ERP space. This enables comparable training, cross-domain visual transfer, and structurally conditioned fisheye generation.
- Motivation: The method extends a fisheye-focused framework to exploit the semantic diversity of large perspective collections.The shared representation supports high-fidelity fisheye structures while incorporating richer object diversity.
- Unified ERP Representation: MIVIFI adopts ERP as a unified representation for jointly training on fisheye and perspective images.Both domains are projected onto a shared spherical manifold to combine fisheye structural realism with perspective-domain semantic variety.
- Masked Supervision and Out-of-FoV Completion: Masked loss excludes unobserved perspective regions, preventing supervision from reinforcing projection padding artifacts.Fisheye images provide valid signals across the full ERP canvas, whereas perspective projections contain masked regions.
- Masked Supervision and Out-of-FoV Completion: MIVIFI projects occupancy MPIs across the full ERP domain even when perspective RGB is masked, pairing complete structure with incomplete visual observations.This setup encourages consistent RGB completion in unobserved regions through cross-domain transfer from fisheye examples.
A. Datasets
The experiments combine KITTI-360 fisheye data with nuScenes perspective data and evaluate image quality, diversity, and semantic accuracy within valid fisheye regions.
- Datasets: nuScenes broadens scene diversity with six perspective cameras from urban scenes in Boston and Singapore.Its occupancy voxel grids are used at 0.5 m resolution.
- Implementation Details: MIVIFI training balances KITTI-360 and nuScenes by sampling equal numbers of examples from each dataset.The model uses Stable Diffusion v2.1 and an MPI occupancy representation of size 100×100×256.
- Evaluation Metrics: FID, SSIM, LPIPS, and mIoU measure distribution fidelity, perceptual similarity, diversity, and semantic accuracy, respectively.Evaluation is restricted to valid fisheye regions, with mIoU excluding ego-vehicle, lens-vignette, and areas lacking occupancy support.
D. Qualitative Results
Both methods generate high-quality images that follow conditioning, while MIVIFI better supports object insertion and environmental changes requiring broader training diversity.
- General Generation Quality: Both SyntheOcc-FE and MIVIFI generate high-quality images that respect the semantic and spatial conditioning.Small differences from ground truth are expected because generation uses MPI information and abstract textual prompts, supporting dataset diversity.
- Adding New Objects: SyntheOcc-FE inserts cars consistently but fails to render inserted pedestrians because KITTI-360 lacks pedestrian instances.The failure is tied to the limited object diversity of its fisheye training data.
- Adding New Objects: MIVIFI successfully synthesizes inserted pedestrians after incorporating nuScenes during training.The result confirms effective scene manipulation through occupancy-map edits.
- Shifting the Time-of-Day: MIVIFI generates realistic nighttime environments from daytime scenes using text prompts, whereas SyntheOcc-FE fails under limited nighttime data.The comparison attributes MIVIFI’s capability to the broader nuScenes training distribution.
- Modifying the Weather: For heavy-rain adaptation, MIVIFI benefits from broader training diversity while SyntheOcc-FE struggles with plausible precipitation effects.Both methods use text descriptors with a time-agnostic MPI to modify environmental attributes.
E. Quantitative Results
SyntheOcc-FE provides stronger image-quality scores, while MIVIFI favors diversity and broader semantic manipulation. Synthetic data remains viable for downstream detection but does not match real-data training.
- Image Quality: SyntheOcc-FE achieves higher scores across all quality-based metrics, whereas MIVIFI attains higher LPIPS and greater diversity.The table characterizes a trade-off between specialized fisheye quality and cross-domain generalization.
- Semantic Manipulation: MIVIFI successfully synthesizes pedestrians through expanded training data, addressing a class underrepresented in KITTI-360.SyntheOcc-FE inserts cars successfully but fails on pedestrians in the corresponding comparison.
- Downstream Detection: Real training data outperforms both synthetic approaches on downstream 2D detection, although synthetic results remain competitive.SyntheOcc-FE generally performs better, while MIVIFI benefits detection of the pedestrian class.
- Environmental Diversity: MIVIFI produces realistic nighttime transformations from daytime inputs, while SyntheOcc-FE fails because of limited modality coverage.The comparison uses text-prompt adaptation and generated multi-view fisheye outputs.
- Ablation Study: Generic prompts and removing nuScenes-pretrained weights both reduce performance in the architectural analysis.The results support the importance of prompt quality and perspective-domain pretraining.
- Ablation Study: Removing the lens mask increases FID to 18.68, while adding pseudo-depth or cosine-similarity loss also increases FID.These modifications do not improve overall image quality under the reported evaluation.
V. DISCUSSION
MIVIFI broadens training beyond scarce fisheye data, while SyntheOcc-FE remains stronger in specialized visual quality. The discussion also identifies alignment and temporal-consistency constraints and reports useful downstream detection signals.
- Regarding Expandability: MIVIFI can train on any perspective dataset, unlike SyntheOcc-FE, and is extensible to single-view images and other sensors or simulations.This flexibility is presented as a route to improved generalization and image quality.
- Weather Editing: MIVIFI successfully renders rain from sunny fisheye imagery, whereas SyntheOcc-FE cannot generate the weather modification.The comparison uses text-prompt adaptation to rain.
- Regarding Generalization vs. Specialization: SyntheOcc-FE provides higher visual quality through KITTI-360 specialization, whereas MIVIFI gains broader object, time, and weather editing from more diverse training data.The paper characterizes this as a generalization–specialization trade-off, with joint training inducing domain conflicts.
- Regarding Perspective Images: Perspective-image misalignment creates faulty training signals that hinder ERP training and also affects perspective-to-fisheye conversion.Suggested remedies include single-image training with larger masks or improved alignment of overlapping regions.
- Regarding Sequences: Current sequence editing does not preserve frame consistency; incorporating previous-frame conditioning is required, but remains outside this work’s scope.Both methods could be extended with additional conditioning.
- Regarding Downstream Application: Synthetic images provide a useful training signal for 2D object detection, with real- and synthetic-data results comparable despite low overall mAP.The semantic conditioning could also support panoptic segmentation, while better training, generation, or more data may improve downstream results.
VI. CONCLUSION
The paper introduces multi-view fisheye generation with SyntheOcc-FE and MIVIFI, using ERP to incorporate perspective datasets. The approach supports scene editing and rare edge-case generation while reducing reliance on costly 3D simulation.
- The paper introduces SyntheOcc-FE and MIVIFI for multi-view fisheye image generation conditioned on semantic occupancy.
- SyntheOcc-FE uses fisheye data and achieves slightly better visual quality, while MIVIFI uses ERP to train on additional perspective datasets such as nuScenes.
- MIVIFI enables scene editing, extending limited fisheye datasets and generating rare edge cases for training 360° perception models.
- Future work will expand training data, add image conditioning for consistent sequence editing, and emphasize downstream tasks.
Appendix
The appendix illustrates single-view ERP training, sequence adaptation, and downstream detection comparisons for SyntheOcc-FE and MIVIFI. The examples show both successful structural insertion and limitations in pedestrian handling and temporal consistency.
- Single-View ERP: MIVIFI can train from single-view images with a full semantic occupancy map, enabling single-view training and out-of-field-of-view completion.
- Sequence Adaptation: SyntheOcc-FE inserts a car correctly in sequence adaptation but struggles with a person because of limited data.
- Sequence Adaptation: Without visual conditioning, generated sequence images are not consistent across frames.
- Object Detection: Qualitative downstream detection results show that SyntheOcc-FE detects no pedestrians, while MIVIFI performs better with people.