PaperlayerView exact revision on arXiv

Source-linked AI summary

UniSim: A Neural Closed-Loop Sensor Simulator

Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, Raquel Urtasun

arXiv:2308.01898v1cs.CVcs.RO

TL;DR

Safe autonomy evaluation needs rare, safety-critical scenarios that can be changed and run in closed loop rather than merely replayed from logs. UniSim reconstructs recorded scenes with neural representations to simulate camera and LiDAR observations under new views and configurations, achieving realistic data with lower domain gaps and enabling closed-loop testing of autonomy systems on safety-critical scenarios.

  • Problem

    Closed-loop self-driving evaluation requires realistic sensor data for rare, safety-critical scenarios and altered actor or autonomy trajectories beyond what log replay and real-world testing provide.

  • Method

    UniSim reconstructs static backgrounds and dynamic actors from recorded sensor logs with neural feature fields, then renders composited camera and LiDAR observations for new views and scene configurations.

  • Results

    UniSim realistically simulates camera and LiDAR data, achieves lower domain gaps on downstream autonomy tasks, and enables closed-loop evaluation on novel safety-critical scenarios.

  • Takeaways & Limitations

    UniSim supports measuring autonomy performance in editable, photorealistic safety-critical scenarios without exposing systems to real-world safety hazards.

  • Takeaways & Limitations

    UniSim cannot render lighting or weather variations, actor animation or deformation, and can produce artifacts in previously out-of-view regions.

Abstract

from arXiv · show

Rigorously testing autonomy systems is essential for making safe self-driving vehicles (SDV) a reality. It requires one to generate safety critical scenarios beyond what can be collected safely in the world, as many scenarios happen rarely on public roads. To accurately evaluate performance, we need to test the SDV on these scenarios in closed-loop, where the SDV and other actors interact with each other at each timestep. Previously recorded driving logs provide a rich resource to build these new scenarios from, but for closed loop evaluation, we need to modify the sensor data based on the new scene configuration and the SDV's decisions, as actors might be added or removed and the trajectories of existing actors and the SDV will differ from the original log. In this paper, we present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle and converts it into a realistic closed-loop multi-sensor simulation. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene, and composites them together to simulate LiDAR and camera data at new viewpoints, with actors added or removed and at new placements. To better handle extrapolated views, we incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions. Our experiments show UniSim can simulate realistic sensor data with small domain gap on downstream tasks. With UniSim, we demonstrate closed-loop evaluation of an autonomy system on safety-critical scenarios as if it were in the real world.

1. Introduction

UniSim addresses the need for high-fidelity, editable closed-loop sensor simulation by reconstructing recorded driving scenes and rendering new multi-sensor observations under changed configurations and trajectories. It combines neural scene representations, dynamic-object priors, and feature decoding to support realistic autonomy evaluation and downstream task performance.

  • Motivation: Existing log-replay systems cannot let autonomy systems execute altered actions, while real-world testing is costly and safety-constrained.The motivating scenarios include rare safety-critical events and counterfactual actor or ego-vehicle behaviors.
  • Motivation: UniSim builds an editable digital twin from captured logs, allowing actors to be modified, removed, or added and new autonomy trajectories to be executed.The simulator updates sensor observations according to the autonomy system’s new location and the changed dynamic-actor states.
  • Approach: UniSim reconstructs and renders multi-sensor data from a single recorded log at novel views and scene configurations despite sparse, constrained observations.The system is designed for closed-loop data-driven simulation rather than replaying the original sensor stream.
  • Approach: Multi-resolution voxel neural fields represent static scenes and dynamic agents, while volume-rendered feature maps and a CNN produce camera images and unified LiDAR observations.A learned neural shape prior helps complete unseen regions of dynamic agents, improving extrapolated-view rendering.
  • Results: UniSim realistically simulates camera and LiDAR observations, reduces domain gaps on detection, motion forecasting, and motion planning, and enables closed-loop evaluation of safety-critical scenarios.The authors also report photorealism and demonstrate autonomy evaluation as if scenarios occurred in the real world.

2. Related Work

Prior work spans robotics simulation, novel-view synthesis, and data-driven sensor simulation, but UniSim targets realistic multi-sensor closed-loop autonomy evaluation with stronger support for extrapolated views. Its compositional representation separates static background and dynamic actors for scene editing and rendering.

  • Simulation Environments for Robotics: Robotics simulators traditionally model dynamics, physical forces, and domain-specific physics, but full autonomy testing also requires realistic sensor observations.This motivates extending simulation beyond robot behavior and physical interaction models.
  • Novel View Synthesis: Novel-view synthesis methods reconstruct scenes from observed images using feature warping, aggregation, or neural radiance fields with volume rendering.These approaches generally learn from densely collected views and render nearby unseen viewpoints.
  • Data-driven Sensor Simulation for Self Driving: Data-driven self-driving simulators use LiDAR-based geometry or object insertion, but prior approaches may simulate only LiDAR or lack high-resolution image synthesis.These methods therefore provide incomplete support for multi-sensor autonomy evaluation.
  • UniSim: UniSim separates static background and dynamic actors, queries their neural feature fields independently, and composites volume-rendered descriptors for image decoding.The overview depicts a sparse feature-grid for the static scene and hypernetwork-generated actor representations.
  • UniSim: Unlike approaches focused on limited deviations from the original recording, UniSim is designed for multi-sensor closed-loop evaluation and extrapolated views.This distinguishes its target setting from simulation restricted to trajectories close to the recorded one.

3. Neural Sensor Simulation

UniSim constructs an editable neural scene representation by separating static background from dynamic actors, then composes and renders both camera and LiDAR observations from new configurations and viewpoints.

  • Compositional Scene Representation: UniSim builds a digital twin from camera images, LiDAR point clouds, and relative poses, decomposing the world into static background and moving actors.Separate component models support controllable counterfactual scene generation.
  • Neural Feature Fields: Neural feature fields map 3D points and view directions to implicit geometry and feature descriptors, while multi-resolution grids improve expressiveness and inference speed.Grid features are interpolated, concatenated with view direction, and processed by an MLP head.
  • Background and Actor Models: A sparse background grid uses LiDAR geometry priors to optimize near-surface voxels, reducing the cost of representing scenes spanning hundreds to thousands of meters.The method avoids maintaining a dense, high-resolution voxel grid over the entire driving region.
  • Background and Actor Models: UniSim generalizes actor appearance with shared neural components and jointly optimized latent codes, addressing memory demands and overfitting in dense traffic scenes.The shared MLP predicts geometry and feature descriptors from actor-specific latent representations.
  • Compositional Scene Representation: Separate background and actor representations allow actors to be inserted, removed, repositioned, and composed in world coordinates.Actors are represented in object-centric coordinates and transformed using desired poses before composition.
  • Multi-modal Sensor Simulation: Camera rays are volume-rendered into feature maps that a CNN converts to RGB images, while LiDAR rays produce depth and intensity through analogous rendering and decoding.The CNN upsamples a lower-resolution feature map, reducing ray queries; LiDAR intensity is predicted by an MLP decoder.

4. Experiments

UniSim is evaluated as a multi-sensor simulator for novel views, scene configurations, downstream autonomy tasks, and closed-loop safety-critical scenarios. Across these settings, it improves sensor realism and reduces domain gap relative to existing methods.

  • Experimental Details: The experiments use PandaSet scenes and compare UniSim with state-of-the-art neural and geometry-based sensor simulation baselines.PandaSet contains 103 urban driving scenes with camera images and 360° LiDAR point clouds.
  • Sensor Realism: UniSim outperforms camera-simulation baselines on all reported image-similarity metrics, with larger advantages for extrapolated views.The evaluation includes interpolation and lateral lane-shift extrapolation settings.
  • Ablation: The actor model and CNN decoder improve the neural feature-grid baseline, while perceptual and adversarial losses trade interpolation performance for better lane-shift results.The CNN is especially effective for extrapolation because it reasons over spatial relations and increases model capacity.
  • LiDAR Simulation: UniSim outperforms LiDARsim on hit rate, median per-ray ℓ2 error, and average intensity error, indicating higher accuracy and coverage.The visual comparison also reports less noise and more continuous beam rings closer to real LiDAR.
  • Perception Evaluation: UniSim achieves the smallest detection domain gap across Real2Sim and Sim2Real evaluations in both replay and lane-shift settings.The lane-shift setting produces a larger separation between UniSim and the baselines.
  • Downstream Autonomy: UniSim simulation data improves perception training and yields the smallest domain gap for detection, motion forecasting, and planning among compared methods.UniSim augmentation produces significant gains, whereas baseline augmentation brings marginal gains or harms performance.
  • Closed-loop Simulation: UniSim supports closed-loop evaluation by updating actor states and the SDV viewpoint after each autonomy step before simulating the next sensor observations.This enables new safety-critical scenarios such as an actor cutting into the SDV’s lane or an oncoming actor.

5. Conclusion

The conclusion presents UniSim as a high-fidelity virtual-world tool for autonomy testing from real-world sensor logs. It reports improved realism and lower domain gap, demonstrates closed-loop safety-critical evaluation, and identifies several directions for future scene modeling.

  • Conclusion: UniSim decomposes and reconstructs dynamic actors and static backgrounds to render new multimodal sensor observations and create novel scenarios.The system uses real-world scenarios collected by a mobile platform as the basis for the virtual world.
  • Conclusion: UniSim improves over state-of-the-art methods and generates realistic sensor data with much lower domain gap.
  • Conclusion: The paper demonstrates closed-loop autonomy evaluation on novel safety-critical scenarios without real-world safety hazards or comparable capital intensity.
  • Conclusion: Future work includes explicit modeling and manipulation of scene lighting, weather, and articulated actors.

Appendix

The supplementary material provides implementation and experimental details, additional visualizations and metrics, and an analysis of UniSim’s limitations.

  • The supplement covers UniSim’s implementation details, experiment settings, additional camera and LiDAR visualizations and metrics, and limitations.

A1. UniSim Details

UniSim represents a driving scene with bounded static and dynamic neural feature grids, then renders camera and LiDAR observations while refining actors and completing unseen regions.

  • Scene Representation Details: The static region spans 80 meters behind the first SDV position, 80 meters ahead of the last, 120 meters in width, and 40 meters in height.
  • Scene Representation Details: Dynamic actors use independent multi-resolution feature grids generated from a shared HyperNet, while distant regions use a separate background model.
  • Neural Feature Fields Details: MLPs predict signed-distance geometry and neural feature descriptors; CNN and LiDAR-intensity decoders convert rendered features into sensor outputs.
  • Actor Model Details: UniSim jointly optimizes actor rotations and translations during training to refine potentially inaccurate tracklets.
  • Actor Model Details: A longitudinal vehicle-symmetry prior randomly flips canonical object points and view directions during training to improve actor completion.
  • Actor Behavior Model for Closed-loop Simulation: Counterfactual actor behavior is modeled with heuristics, although intelligent-driver and deep-learning behavior models could also control actors.
  • Run-time and Resources: Training and inference speed are compared for UniSim and baselines using an A5000 GPU, with realism and runtime adjustable through rendering parameters.

A2.1. Free View Synthesis (FVS)

The FVS setup uses LiDAR-derived proxy geometry and selected nearby source frames to warp observations into target images under different viewpoint conditions.

  • FVS constructs static-scene proxy geometry by aggregating dynamic-point-filtered LiDAR across frames and creating triangle surfels.
  • Source-frame selection varies by condition: nearby frames for training, the two nearest frames for interpolation, and wider temporal sampling for lane shifts.

A2.2. Instant-NGP

The Instant-NGP baseline uses hash encoding and accelerated rendering, while the appendix figures illustrate CNN-decoder and actor-shape-prior effects.

  • Instant-NGP uses efficient hash encoding, accelerated ray sampling, and fully fused MLPs for state-of-the-art NeRF rendering.
  • The CNN RGB decoder makes rendered images cleaner and helps learn more details.
  • The actor shape prior helps reconstruct a vehicle’s unobserved side after the SDV changes lanes.

A2.3. Neural Scene Graph (NSG)

The NSG baseline is evaluated under memory-constrained preprocessing that represents only a limited set of dynamic vehicle actors.

  • Only dynamic vehicles are treated as object nodes; static vehicles are incorporated into the background.
  • The preprocessing retains the five dynamic actors most present in each log because larger actor sets exceed memory during data loading.The reported failure occurred on a machine with 128GB of memory, partly because the image data has higher resolution.
  • The implementation uses the GitHub codebase with default model and training parameters.

A2.4. LiDARsim

The supplied material covers LiDAR simulation comparisons, autonomy evaluation procedures, ablations, implementation constraints, and UniSim’s stated limitations. It reports improved LiDAR realism and smaller domain gaps while identifying unresolved rendering and modeling boundaries.

  • Autonomy evaluation: The autonomy evaluation measures detection agreement, prediction average displacement error, and plan consistency between real-image and simulated-image outputs.Prediction uses a 6 second horizon at 10Hz, while planning compares 5 second endpoint displacement.
  • Ablation study: The convolutional RGB decoder improves rendered image quality, while actor shape priors reconstruct unobserved vehicle regions during lane shifts.Tracklet optimization further improves dynamic-actor reconstruction and sharpens details.
  • Scene editing: UniSim can copy static elements such as cones to new placements, enabling edited scene configurations.
  • LiDAR simulation: UniSim produces smoother and more accurate LiDAR sweeps and intensity values than LiDARsim in qualitative log-replay comparisons.LiDARsim produces incomplete beam rings and noisy scans, whereas UniSim’s neural representations better capture underlying geometry.
  • LiDAR simulation: UniSim yields a smaller domain gap than LiDARsim in both Real2Sim and Sim2Real, especially at strict IoU thresholds such as AP@0.7 and AP@0.8.The authors state this can help better evaluate and train autonomy.
  • Limitations: UniSim cannot render lighting variations and does not model animation or actor deformation; it also exhibits artifacts outside previously observed fields of view.The stated future work includes explicit lighting and weather modeling and improved handling of these phenomena.
Loading 2308.01898v1…