Source-linked AI summary
MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving
Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, Yuxin Huang, Xiaoyu Ye, Zike Yan, Yongliang Shi, Yiyi Liao, Hao Zhao
TL;DR
Autonomous-driving systems need safer and less expensive ways to encounter real-world corner cases, because collecting such data directly is dangerous and costly. MARS introduces an open-source, modular NeRF simulator that separately models foreground instances and background environments, achieving state-of-the-art rendering on public benchmarks. Its main practical limitations are training time, lack of real-time rendering, and artifacts from unmodeled dynamic specular effects.
Problem
Corner-case data is dangerous and expensive to collect, while existing simulators have realism or dynamic-scene limitations.
Method
MARS is an open-source modular NeRF framework with separate background and foreground nodes, supporting multiple representations, samplers, and input modalities.
Results
MARS achieves state-of-the-art rendering performance on public benchmarks, outperforming baseline methods in image reconstruction and novel-view synthesis.
Takeaways & Limitations
Instance-wise modeling enables removing, adding, and editing objects, including rotating, translating, and changing vehicle trajectories.
Takeaways & Limitations
The method requires hours to train, cannot render in real time, and may produce artifacts on glass or other reflective materials because dynamic specular effects are not modeled.
Abstract
from arXiv · showhide
Nowadays, autonomous cars can drive smoothly in ordinary cases, and it is widely recognized that realistic sensor simulation will play a critical role in solving remaining corner cases by simulating them. To this end, we propose an autonomous driving simulator based upon neural radiance fields (NeRFs). Compared with existing works, ours has three notable features: (1) Instance-aware. Our simulator models the foreground instances and background environments separately with independent networks so that the static (e.g., size and appearance) and dynamic (e.g., trajectory) properties of instances can be controlled separately. (2) Modular. Our simulator allows flexible switching between different modern NeRF-related backbones, sampling strategies, input modalities, etc. We expect this modular design to boost academic progress and industrial deployment of NeRF-based autonomous driving simulation. (3) Realistic. Our simulator set new state-of-the-art photo-realism results given the best module selection. Our simulator will be open-sourced while most of our counterparts are not. Project page: https://open-air-sun.github.io/mars/.
1 Introduction
Realistic simulation is proposed as a safer, less expensive way to expose autonomous-driving systems to corner cases. MARS addresses limitations of existing simulators with an open-source, modular NeRF framework that separately models foreground instances and background environments.
- Motivation: Real-world corner-case collection is dangerous and expensive, motivating photorealistic simulation with a small sim-to-real gap.Such simulation could let algorithms experience enormous numbers of corner cases in a simulator.
- Existing limitations: Traditional graphics-based simulators are limited by asset modeling and rendering quality, despite AADS achieving photorealism with curated assets.GeoSim instead uses image enhancement and composition for realistic appearance and flexible asset generation.
- Existing limitations: NeRF-based methods provide realistic rendering from real-world data but struggle with dynamic scenes containing moving objects and static environments.This limitation constrains their practical use for real-world sensor simulation.
- MARS: MARS is an open-source NeRF-based modular framework that decomposes foreground instances and background environments for autonomous-driving simulation.The framework is presented as the first open-source framework of this kind.
- MARS: The unified pipeline supports different NeRF backbones, sampling methods, and multi-modal inputs, while its best module combination achieves state-of-the-art rendering with large margins.This modularity is intended to provide flexibility across simulation configurations.
2 Method
MARS represents outdoor driving scenes compositionally with separate background and foreground neural fields, while exposing interchangeable representations and sampling strategies. It composes node samples through volume rendering and adds mechanisms for realistic, conflict-free sensor simulation.
- 2 Method: The system accepts RGB images, sensor poses, object tracklets, and optional depth maps and semantic masks as training inputs.Object tracklets include 3D bounding boxes, categories, and instance IDs; depth and semantics provide extra supervision when available.
- 2.2 Compositional Rendering: For each ray, MARS intersects visible instance boxes, transforms foreground samples into canonical instance frames, queries all nodes, and volume-renders their sorted samples.The resulting pixel-wise outputs include RGB, depth, and semantic properties.
- 2.1 Scene Representation: MARS unifies MLP-based and grid-based NeRF backbones behind modular interfaces and supports proposal-network, coarse-to-fine, and uniform sampling.This design allows different scene representations and sampling strategies for background and foreground nodes.
- 2.1 Scene Representation: MARS decomposes scenes into one large-scale unbounded background NeRF and multiple independent object-centric foreground NeRFs.The background uses scene warping for far regions, while foreground nodes can use conventional or code-conditioned representations.
- 2.3 Towards Realistic Rendering: The framework supports sky modeling for infinitely distant appearances and applies BCE semantic regularization to reduce inconsistency from naive sky blending.The sky model represents regions that never intersect scene geometry, while the regularizer addresses potential blending inconsistency.
- 2.3 Towards Realistic Rendering: Independent background and foreground sampling can place background samples inside foreground boxes, creating ambiguity and artifacts after instance removal.MARS addresses this by minimizing the density sum of background-truncated samples during rendering.
- 2.4 Optimization: Training minimizes a weighted objective combining photometric color loss with sky, accumulation, and available geometric supervision.Depth supervision uses ray-distribution loss when depth data exists and mono-depth loss otherwise.
3 Experiments
Experiments evaluate MARS on KITTI and V-KITTI through image reconstruction, novel view synthesis, and instance-wise editing. The framework supports modular evaluation settings and outperforms baseline methods in the reported rendering comparisons.
- Photorealistic Rendering: MARS is evaluated on KITTI and V-KITTI for image reconstruction and novel view synthesis.The default setting uses a grid-based background NeRF with proposal sampling and category-level foreground representation with coarse-to-fine sampling.
- Photorealistic Rendering: The evaluation compares MARS with NeRF, NeRF+Time, NSG, PNF, and SUDS.These baselines do not simultaneously meet all three standards listed by the paper.
- Photorealistic Rendering: Novel view synthesis holds out frames under 25%, 50%, and 75% training-data settings.The held-out-frame protocols vary how many frames are excluded from training.
- Photorealistic Rendering: 29.79 PSNR on V-KITTI using 75% training data exceeds the best previously published result of 23.87.The paper reports PSNR, SSIM, and LPIPS for quantitative evaluation.
- Instance-wise Editing: Instance-wise editing demonstrates removal, addition, and trajectory editing of vehicles, including rotating and translating a vehicle.Separate background and foreground nodes support these scene edits.
- Ablation Studies: MARS separately evaluates alternative background and foreground representations, samplers, and supervision signals through its modular design.The experiments use different module combinations and compare background and foreground model choices across ablation settings.
4 Conclusion
The paper concludes that MARS is an open-sourced, modular NeRF framework for photorealistic autonomous driving simulation. Its limitations are training cost, lack of real-time rendering, and artifacts from unmodeled dynamic specular effects.
- Conclusion: MARS combines a background node with multiple foreground nodes to model complex dynamic scenes.The framework supports different scene representations and sampling strategies.
- Conclusion: The framework achieves state-of-the-art rendering performance on public benchmarks while offering flexible simulation design choices.The reported flexibility covers scene representations and sampling strategies.
- Limitations: MARS requires hours to train and cannot render in real time.Improving simulation efficiency is identified as future work.
- Limitations: Dynamic specular effects on glass and other reflective materials are not modeled and may cause artifacts in rendered images.View-dependent effects are identified as future work.
A.1 Neural Simulators
Neural simulators use neural rendering to represent autonomous driving scenarios. NSG decomposes dynamic scenes with learned scene graphs, while PNF models instance-aware objects and jointly learns panoptic segmentation.
- Neural Simulators: Neural rendering has been used to build simulators for autonomous driving scenarios.The passage introduces NSG and PNF as neural-simulator approaches.
- Neural Simulators: NSG decomposes dynamic scenes by encoding object transformations in a learned scene graph.It also uses shared implicit fields to describe similar objects.
- Neural Simulators: PNF models instance-aware objects and jointly learns panoptic segmentation.
A.2 Neural Scene Representations
Neural scene representations support image reconstruction and novel view synthesis, but existing approaches generally struggle with high-quality dynamic-scene reconstruction and flexible module combination.
- Neural Scene Representations: Neural scene representation methods have shown impressive results in image reconstruction and novel view synthesis.NeRF introduced implicit-field scene representation, followed by variants targeting acceleration and rendering quality.
- Neural Scene Representations: Most existing neural scene representation methods fail to reconstruct dynamic scenes with high quality.
- Neural Scene Representations: No existing framework described here flexibly switches modules to combine the distinct strengths of different approaches.NeRFStudio provides a unified framework for many approaches, but the passage identifies a remaining gap for dynamic-scene reconstruction and module combination.
B More on the depth supervision
The method uses depth supervision differently depending on whether dense ground-truth depth maps are available. Dense maps enable ray distribution loss, while unavailable dense maps are replaced with monocular-depth regularization using learnable scale and shift factors.
- Dense depth maps enable a ray distribution loss following DS-NeRF.The passage states this loss is used when dense depth maps are available.
- Without dense ground-truth depth, the method uses monocular depth estimation and a mono-depth loss to regularize geometry.This alternative supervision strategy applies when dense depth maps are unavailable.
- Learnable scale and shift factors provide scale- and shift-invariance during monocular-depth training.The factors w and q are optimized through training.
C Details for foreground nodes
Foreground nodes decouple each object instance’s shape and texture, storing their learned latent codes in object libraries for rendering. The implementation uses separate shape and texture networks with positional viewing-direction inputs.
- Details for foreground nodes: Figure 8 presents the category-level model’s decoupling of shape and texture.
- Details for foreground nodes: Shape and texture for each object instance are learned separately as two latent codes and stored in object libraries.The latent codes are optimized over the test sequence before being stored.
- Details for foreground nodes: During rendering, object-library priors are queried by separate shape and texture networks.The shape network has 8 MLP layers, while the texture network has 4 layers.
- Details for foreground nodes: The texture network combines intermediate features with positional viewing directions and texture latents to obtain instance texture.The viewing direction is denoted by d in the method description.
- Details for foreground nodes: Figure 9 provides qualitative traffic-scene decomposition results.
D Synthetic traffic environment generation
The simulator decomposes backgrounds and foreground objects to support editable traffic scenes. Experiments demonstrate object deletion, insertion, translation, and rotation for creating diverse photorealistic scenarios.
- Synthetic traffic environment generation: Decomposed rendering is demonstrated on KITTI, VKITTI, and DAIR-V2X datasets.
- Synthetic traffic environment generation: Foreground editing can delete or add cars at new positions or orientations, equivalent to translating or rotating target objects.
- Synthetic traffic environment generation: These editing operations create diverse, customizable photorealistic traffic scenarios for testing and evaluating autonomous driving systems.