PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuRAD: Neural Rendering for Autonomous Driving

Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, Christoffer Petersson

arXiv:2311.15260v3cs.CV

TL;DR

Existing automotive NeRF methods face limits in training efficiency, sensor modeling, and generalization across datasets. NeuRAD addresses these issues with a unified dynamic-scene representation and extensive camera-lidar modeling, achieving state-of-the-art performance across five datasets with one hyperparameter set.

  • Problem

    Existing NeRF-based automotive methods can require long training, limited supervision, or lack generalizability, constraining scalable autonomous-driving simulation.

  • Method

    NeuRAD is an editable neural simulator using unified static-dynamic scene modeling, actor-aware hash encoding, and learned rolling-shutter, beam-divergence, and ray-drop sensor effects.

  • Results

    NeuRAD achieves state-of-the-art performance across five publicly available autonomous-driving datasets using a single set of hyperparameters.

  • Takeaways & Limitations

    NeuRAD provides sensor-realistic editable clones of real-world driving scenarios while jointly handling camera and lidar data in 360° settings.

  • Takeaways & Limitations

    NeuRAD assumes each actor’s dynamics can be described by a single rigid transform, which pedestrians violate and which produces very blurry reconstructions for them.

Abstract

from arXiv · show

Neural radiance fields (NeRFs) have gained popularity in the autonomous driving (AD) community. Recent methods show NeRFs' potential for closed-loop simulation, enabling testing of AD systems, and as an advanced training data augmentation technique. However, existing methods often require long training times, dense semantic supervision, or lack generalizability. This, in turn, hinders the application of NeRFs for AD at scale. In this paper, we propose NeuRAD, a robust novel view synthesis method tailored to dynamic AD data. Our method features simple network design, extensive sensor modeling for both camera and lidar -- including rolling shutter, beam divergence and ray dropping -- and is applicable to multiple datasets out of the box. We verify its performance on five popular AD datasets, achieving state-of-the-art performance across the board. To encourage further development, we will openly release the NeuRAD source code. See https://github.com/georghess/NeuRAD .

1. Introduction

NeRFs offer scalable, sensor-realistic simulation for autonomous driving, but existing automotive methods face long training, limited sensor coverage, and generalization challenges. NeuRAD addresses these gaps with unified sensor and scene modeling, achieving state-of-the-art results across five datasets without dataset-specific tuning.

  • Motivation: NeRF-based digital traffic scenes support scalable safety testing and targeted generation of corner-case training data.Editable scenes can explore harsh braking and aggressive merging without risking physical damage.
  • Limitations of prior work: Existing dynamic automotive NeRF methods often require many hours of training and may depend on semantic segmentation or specialized scene components.Earlier approaches separately model static backgrounds and dynamic actors, with improvements based on semantic supervision or anti-aliased embeddings.
  • Limitations of prior work: Recent hash-grid methods reduce training and inference time but have limited evidence for 360° multicamera settings and simple lidar models that cannot model ray drop.These gaps matter for common automotive sensor configurations and sensor-realistic simulation.
  • Results: NeuRAD achieves state-of-the-art performance across five automotive datasets with no dataset-specific tuning.The evaluation is presented as evidence of the method’s generalizability.
  • NeuRAD: NeuRAD combines unified camera and lidar modeling with a simple network that separates static and dynamic elements through positional embeddings.The method explicitly models rolling shutter, beam divergence, and ray dropping.

2. Related work

Prior work developed faster and more realistic NeRF-based automotive simulators, but their modeling choices and evaluation settings limit broader applicability. NeuRAD builds on neural feature fields, hash-grid encoding, and efficient feature rendering while targeting more general automotive sensor configurations.

  • NeRF foundations: NeRFs learn implicit 3D representations that can render new images, while later methods reduce the original formulation’s training time.Instant-NGP uses multiresolution learnable hash grids instead of frequency-based positional encoding.
  • NeRFs for automotive data: Automotive NeRF methods aim to provide sensor-realistic simulation for testing and verification, addressing limitations of game-engine-based asset creation, real-to-sim transfer, and diversity.This positions neural rendering as an alternative for automotive data generation.
  • Prior automotive methods: UniSim renders PandaSet front-camera and 360° lidar data using separate hash-grid features for the sky, static background, and each dynamic actor.It renders neural features and upsamples them with a CNN to reduce the number of sampled rays per image.
  • Prior automotive methods: UniSim’s efficiency comes with approximations that lead to poor performance outside its evaluation protocol.The cited description links these approximations to limited generalization beyond the evaluated setting.

3. Method

NeuRAD represents dynamic automotive scenes with a unified neural feature field and models camera and lidar measurements through volumetric rendering and learned sensor effects. Its design combines actor-aware hash encoding, scale-aware sampling, and efficient sampling for large scenes.

  • Method goals: NeuRAD learns an editable representation from posed camera images, lidar point clouds, and estimates of moving-actor size and pose.The target is realistic sensor generation under changes to the ego-vehicle pose, actor poses, or both.
  • Scene representation: The neural feature field maps position and view direction to implicit geometry and features, which are aggregated along rays using alpha compositing.The method uses an SDF-based opacity formulation and decodes rendered features for camera and lidar outputs.
  • Sensor modeling: NeuRAD predicts lidar intensity and ray-drop probability from rendered ray features with small MLPs, including laser beams that produce no return.Ray dropping is learned from data because low-level sensor detection behavior is difficult to model fully with physics.
  • Dynamic scene modeling: A single unified neural feature field separates static and dynamic elements through actor-aware hash encoding rather than separate networks.Samples inside actor boxes are transformed into actor coordinates, while a fourth hash-grid dimension indexes actors.
  • Dynamic scene modeling: A single 4D hash grid samples all rigid actors in parallel, matching separate-grid performance while providing significant speedups.Actor coordinates and view directions are transformed into the actor frame, removing the need to encode time after transformation.
  • Multiscale and efficient rendering: NeuRAD downweights hash-grid features according to ray-frustum scale and uses efficient sampling to represent both nearby detail and distant scene structure.The approach models rays as pyramids whose cross-sectional area depends on beam divergence, while avoiding reliance on lidar coverage for all image regions.
  • Multiscale and efficient rendering: The sampling strategy addresses the computational cost of capturing nearby fine detail and distant parallax in large automotive scenes.Prior lidar-dependent pruning can struggle outside the lidar field of view, motivating a broader sampling strategy.
  • Learning objectives: Camera and lidar reconstruction objectives combine pixel and perceptual losses with lidar depth supervision and empty-space regularization.The lidar objectives are designed to support more realistic simulation alongside geometric reconstruction.

4. Experiments

NeuRAD is evaluated across five AD datasets, multiple sensor modalities, novel viewpoints, and ablations. It achieves strong camera and lidar performance, generalizes to shifted views and actors, and shows that sensor modeling affects realism.

  • Datasets and baselines: NeuRAD is evaluated on nuScenes, PandaSet, Argoverse 2, KITTI, and ZOD using the same model and hyperparameters.The study includes hold-out images, sensor poses without ground truth, and ablations of important model components.
  • Camera novel view synthesis: NeuRAD achieves state-of-the-art camera performance across all five datasets, improving or matching prior methods on the reported NVS metrics.It improves previous work across all metrics on PandaSet, matches S-NeRF on nuScenes while training in 1 hour versus 17 hours, and outperforms prior state of the art on KITTI in PSNR and LPIPS.
  • Lidar novel view synthesis: NeuRAD decreases front-camera lidar depth error by an order of magnitude compared with UniSim and produces realistic point clouds in the 360° setting.Lidar evaluation uses median L2 depth error, RMSE intensity error, ray-drop accuracy, and Chamfer distance.
  • Lidar modeling: Ray dropping is important for realistic lidar point clouds because distant road hits disperse and transparent surfaces can transmit beams without returns.Figure 4 highlights these regions as cases where ray-drop effects should be modeled.
  • Novel scenario generation: NeuRAD generalizes to new ego-vehicle viewpoints and shifted actor poses, evaluated with FID because ground-truth images are unavailable.The protocol includes lateral and vertical ego shifts plus actor rotations and lateral translations.
  • Ablations: A single 4D hash grid increases training speed without detrimental quality impact, while omitting rolling-shutter modeling causes qualitative rendering and geometry breakdowns.The ablations average results over 20 sequences evenly split across the five datasets.

5. Conclusions

NeuRAD is a neural simulator for dynamic AD data that jointly models 360° camera and lidar observations while separating static and dynamic scene elements. It achieves state-of-the-art performance across five public datasets, but assumes rigid actors and has limitations in harsh weather.

  • Conclusion: NeuRAD jointly handles 360° lidar and camera data while decomposing scenes into static and dynamic elements for editable driving-scenario clones.It models beam divergence, ray dropping, and rolling shutters for sensor-realistic novel view synthesis.
  • Conclusion: NeuRAD achieves state-of-the-art performance on five publicly available AD datasets using a single set of hyperparameters.The source code is publicly released to support further research.
  • Limitations: NeuRAD assumes actors are rigid and does not support deformations, while several modeling assumptions are invalid in heavy rain or snow.The paper identifies these limitations as targets for future work.

A. Implementation details

NeuRAD is trained jointly with sampled camera and lidar rays using Adam and scheduled learning rates. Its compact MLP-based design uses hash grids and a CNN decoder, with an enlarged NeuRAD-2x variant for larger scenes.

  • Learning: NeuRAD trains all model components jointly for 20,000 iterations with Adam, sampling 16,384 lidar rays and 40,960 camera rays per iteration.Most parameters use a learning rate of 0.01 with a 500-step warmup; selected components use longer warmups and lower rates.
  • Networks: The networks use small MLPs with hidden dimension 32, alongside hash-grid representations and a lidar decoder.The geo MLP has one hidden layer, while the feature MLP and lidar decoder have two hidden layers.
  • Scaling: NeuRAD-2x approximately doubles parameter count by doubling hash-table sizes and also doubles static-hash-grid resolution for long sequences and large scenes.The supplied implementation passage introduces this enlarged configuration but ends before describing all adaptations.

B. Evaluation details

The evaluation compares NeuRAD with prior methods using dataset-specific protocols across multiple autonomous-driving datasets. Additional ablations use a shared protocol, with nuScenes retaining a shortened protocol because pose quality complicates full-sequence evaluation.

  • Evaluation protocols: Evaluation protocols differ by baseline and dataset, including alternating frames, selected sequences, camera rigs, and lidar sensors.PandaSet, nuScenes, and KITTI protocols use different sequence and frame selections, while Argoverse uses all surround cameras and both lidars.
  • Evaluation protocols: NeuRAD is compared with SoTA methods under the protocols specified for each method.
  • Ablation protocol: Ablations use 20 sequences across five datasets, alternating frames for training and evaluation on full sequences and available sensors.The ablation protocol covers four sequences from each dataset, except that nuScenes uses a shortened protocol.
  • Ablation protocol: nuScenes retains S-NeRF’s shortened evaluation protocol because the provided poses are too poor for reliable full-sequence training.Optimizing poses produced qualitatively good results and strong FID scores but poor reconstruction scores because learned and evaluation poses were misaligned.
  • Baselines: UniSim is a suitable baseline because it requires only camera images, lidar point clouds, sensor poses, and tracked 3D bounding boxes.Its code is closed-source, so the authors reimplemented it for comparison.

C.1. Data processing

The data-processing and baseline implementation details specify occupancy handling, neural components, loss terms, and numerical-gradient computation. These choices reproduce the compared sensor-simulation pipeline while balancing runtime and rendering quality.

  • Occupancy and architecture: UniSim uses a 0.5 m occupancy grid initialized from accumulated lidar points, then dilated by a factor of two.Points inside dynamic actors are removed before grid initialization, and dilation accounts for point-cloud sparsity.
  • Occupancy and architecture: The upsampling CNN has four residual blocks with 32 channels and a threefold transposed-convolution upsampling factor.The implementation has 0.7M parameters, fewer than the 1.7M reported for UniSim, while larger CNNs increased runtime.
  • Training choices: The implementation omits adversarial training because it did not produce consistent performance increases.The authors associate its removal with slightly worse lane-shift FID than the original results.
  • Losses: UniSim’s regularization uses one loss to reduce weights far from surfaces and another to encourage the signed-distance function to satisfy the eikonal equation near surfaces.
  • Gradient computation: Numerical gradients replace analytical gradients because tiny-cuda-nn does not support second-order derivatives for the MLPs.The gradient is estimated from four signed-distance queries at offset locations.
  • Losses: The perceptual loss compares VGG features from ground-truth and rendered image patches.The same pix2pixHD formulation is used for NeuRAD and UniSim.

D. Inferring ray drop

NeuRAD infers missing lidar rays by undoing ego-motion compensation, assigning diode indices, and filling absent scan bins. The procedure combines pose interpolation, spherical-coordinate processing, sensor structure, and dataset-specific handling.

  • Effect of ray drop: Including dropped lidar rays during supervision increases sensor-rendering fidelity across all evaluated aspects.The missing-ray inference process is illustrated with PandaSet scans, where missing points are visualized at a large range.
  • Preprocessing: Missing-ray inference begins by transforming lidar points through world coordinates and interpolated sensor poses before returning them to the sensor frame.Rotation interpolation uses quaternions and spherical linear interpolation.
  • Diode assignment: Diode indices exploit spinning-lidar channels, whose fixed elevation angles and repeated beams provide structure for identifying missing rays.Most datasets provide this information directly; PandaSet requires predicting assignments from point elevations.
  • Diode assignment: For PandaSet, clustering uses sensor specifications, elevation histograms, empty-bin boundaries, and iterative boundary insertion to recover diode assignments.The method uses 2,000 histogram bins and inserts boundaries until the required number is reached.
  • Point infilling: Point infilling marks bins containing returns and linearly interpolates azimuth and elevation for remaining bins.

E. Modeling rolling shutter

NeuRAD models rolling-shutter effects by assigning rays individual timestamps and interpolating sensor poses at those times. This improves generated renderings, particularly when vehicle velocity is high, and affects ego-motion-compensated lidar structure.

  • Rendering effect: Modeling rolling shutter improves generated renderings, especially at high velocities.
  • Timestamp assignment: Each lidar ray receives an individual timestamp, taken from raw data or approximated from azimuth and sensor RPM.
  • Timestamp assignment: Camera timestamps are approximated through shutter time, with image rows offset accordingly.
  • Pose interpolation: Sensor poses are linearly interpolated to ray timestamps, shifting ray origins to capture motion during acquisition.The same temporal treatment explains cuts in ego-motion-compensated lidar patterns caused by vehicle travel during one revolution.

F. Simulation gap

NeuRAD is evaluated beyond novel-view synthesis through simulation-gap tests, including object detection, depth estimation, and qualitative image comparison. The results indicate realistic synthesized data and stronger side-camera rendering than UniSim.

  • The simulation-gap evaluation compares detector performance on real and synthesized images using BEVFormer mAP.
  • The evaluation also includes zero-shot relative-depth estimation with DepthAnything.
  • NeuRAD captures more image detail than UniSim across three PandaSet sequences, with the clearest advantage in side-camera views.
  • NeuRAD achieves 32.0 mAP on synthesized images, close to 32.4 on real images and above UniSim’s 30.1.

G. Additional results

Additional experiments show that NeuRAD’s proposal sampling and sensor embeddings improve rendering quality, while pose optimization exposes an evaluation mismatch that reduces standard NVS metrics.

  • NeuRAD’s proposal sampling avoids occupancy-grid failures on surfaces without lidar points and on nearby objects requiring dense samples.
  • Sensor embeddings remedy appearance shifts between cameras, including exposure differences, and produce more accurate reconstructions.
  • nuScenes lacks height, roll, and pitch sensor-pose information, limiting NeuRAD especially on sequences where the ego vehicle does not follow flat terrain.
  • Pose optimization sharpens renderings and improves FID but sharply lowers PSNR, LPIPS, and SSIM because training and validation world frames become inconsistent.
  • NeuRAD disables sensor-pose optimization to avoid an unfair advantage over prior work and leaves inaccurate-pose NVS evaluation for future study.

G.1. Limitations

NeuRAD’s assumptions and rendering choices create identifiable failure cases involving sampling, sensor appearance, pose accuracy, deformable actors, and changing illumination.

  • Occupancy-based sampling can misrepresent regions without lidar occupancy and nearby objects that require extremely dense sampling.
  • Without sensor embeddings, differing sensor appearances cause color and lighting shifts, especially where camera views overlap.
  • Camera optimization can sharpen images while reducing PSNR because learned poses become misaligned with evaluation poses.
  • The rigid-actor assumption causes blurry reconstructions for pedestrians and other deformable actors.
  • Night scenes are difficult because noise, long exposures, motion, lens flare, blooming, and actor-generated illumination conflict with standard scene modeling.
  • NeuRAD assumes radiance is static over time, so changing effects such as brake lights cannot be represented directly.
Loading 2311.15260v3…