PaperlayerView exact revision on arXiv

Source-linked AI summary

E-NeRF: Neural Radiance Fields from a Moving Event Camera

Simon Klenk, Lukas Koestler, Davide Scaramuzza, Daniel Cremers

arXiv:2208.11300v2cs.CVcs.RO

TL;DR

NeRF estimation is commonly studied with ideal images, although motion blur and poor illumination can complicate real-world capture. E-NeRF reconstructs NeRFs from fast-moving event cameras, optionally combining events and frames, and reports strong reconstructions in settings where frame-based approaches fail.

  • Problem

    NeRF estimation has largely assumed ideal illumination and slow camera motion, conditions that may not hold in robotics, where blur and poor illumination can affect navigation, inspection, and visualization.

    Source: §I · pp. 1–2

  • Method

    E-NeRF estimates a NeRF from a fast-moving event camera and can combine the event stream with color images.

    Source: §I · p. 2

  • Results

    E-NeRF outperforms multiple baselines on synthetic and real-world data, better reconstructs high-frequency details such as text, and benefits from events when only a few input views are available.

    Source: §V · p. 7

  • Takeaways & Limitations

    Event-only input can render high-quality frames, while combining grayscale events with color frames can produce a sharp, colored reconstruction using both sensors' strengths.

    Source: §V · p. 7

Abstract

from arXiv · show

Estimating neural radiance fields (NeRFs) from "ideal" images has been extensively studied in the computer vision community. Most approaches assume optimal illumination and slow camera motion. These assumptions are often violated in robotic applications, where images may contain motion blur, and the scene may not have suitable illumination. This can cause significant problems for downstream tasks such as navigation, inspection, or visualization of the scene. To alleviate these problems, we present E-NeRF, the first method which estimates a volumetric scene representation in the form of a NeRF from a fast-moving event camera. Our method can recover NeRFs during very fast motion and in high-dynamic-range conditions where frame-based approaches fail. We show that rendering high-quality frames is possible by only providing an event stream as input. Furthermore, by combining events and frames, we can estimate NeRFs of higher quality than state-of-the-art approaches under severe motion blur. We also show that combining events and frames can overcome failure cases of NeRF estimation in scenarios where only a few input views are available without requiring additional regularization.

I. INTRODUCTION

The section motivates NeRF reconstruction under real-world camera conditions, where motion blur and limited dynamic range can compromise useful visual information. It presents event cameras as a way to estimate NeRFs during fast motion and to combine events with frames in challenging settings.

I. INTRODUCTION

  • NeRF studies have largely used simulated or high-quality real-world images captured under ideal conditions.Such images have good surrounding light and minimal noise.Source: §I · p. 2
  • Fast motion and long exposures cause motion blur, while limited dynamic range can obscure important scene information.Shorter exposures may reduce received light and increase noise.Source: §I · p. 2
  • E-NeRF is presented as the first method to estimate a NeRF from a fast-moving event camera.The introduction situates the method in robotic settings with motion blur or non-ideal illumination.Source: §I · p. 2
  • Combining events with color images targets conditions where frame-based methods fail or need extra regularization, including strong blur and few views.Source: §I · p. 2
  • The authors open-source their code and simulated datasets.Source: §I · p. 2

II. RELATED WORK

Prior work addresses NeRFs under blur or low light, event-based reconstruction, and event-derived NeRFs, but the section distinguishes E-NeRF by its moving-camera setting and sensor fusion.

II. RELATED WORK

  • Deblur-NeRF handles moderate, nonconsistent blur, while RawNeRF handles low light but not motion blur and assumes a static camera during exposure.Source: §II · p. 2
  • Event-based 2D methods reconstruct intensity or fuse events with frames, while event-based 3D methods produce sparse point clouds for navigation rather than novel-view visualization.Source: §II · p. 2
  • EventNeRF uses a static camera and controlled illumination and object motion; Ev-NeRF uses a slowly moving camera without considering motion blur.Source: §II · p. 2
  • E-NeRF focuses on a moving event camera under general illumination, including background events, and evaluates frame-based failure cases.The section also identifies event-and-frame NeRF reconstruction, a no-event loss, and direct use of neighboring events as distinctions from the concurrent works.Source: §II · p. 2

A. Optimizing a Neural Radiance Field

E-NeRF represents a scene with an MLP and renders color predictions along camera rays, optimizing from events rather than image-color least squares. The spiral-sequence comparison reports its best results in both event-only and event-plus-blurred-frame settings.

A. Optimizing a Neural Radiance Field

  • An MLP maps 3D coordinates and viewing direction to color and volume density, which are rendered along camera rays to predict pixel color.Source: A. Optimizing a Neural Radiance Field · pp. 2–3
  • E-NeRF shows the best results in the event-only and combined event-and-blurred-frame settings on the spiral sequences.The comparison uses sharp test views and PSNR, SSIM, and LPIPS; the sharp-image NeRF baseline uses ground-truth frames unavailable in real-world applications.Source: A. Optimizing a Neural Radiance Field · p. 3
  • Unlike vanilla NeRF's least-squares loss on rendered and image colors, E-NeRF uses an event-generation model and a multiresolution hash encoding of input coordinates.Source: A. Optimizing a Neural Radiance Field · p. 3

B. Event Stream Utilization

The event-stream objective uses asynchronous events to form brightness-change supervision across successive event times. It predicts rendered brightness differences and compares them with event-camera measurements, with a linlog mapping applied to rendered intensities.

B. Event Stream Utilization

  • Each event records a pixel, microsecond timestamp, and polarity indicating a logarithmic-brightness increase or decrease by the contrast threshold.Source: B. Event Stream Utilization · p. 3
  • For each event, E-NeRF renders rays at the event time and the previous event time at that pixel, then applies a linlog mapping to predicted intensities.The mapping uses threshold B to determine its linear region; the paper sets B = 20 following v2e.Source: B. Event Stream Utilization · p. 3
  • The event loss minimizes the difference between predicted rendered brightness changes and measured changes, with measured change given by polarity times contrast threshold.Source: B. Event Stream Utilization · p. 3

C. Event Sampling

Event supervision uses pairs of events from the same pixel, typically neighboring events in time. The method also samples intervals without events and can use accumulated event polarity for more widely separated pairs.

C. Event Sampling

  • Event pairs link an event to its successor at the same pixel, while optimization samples many pairs across pixels and times.The method can also use pairs that are not direct temporal neighbors.Source: C. Event Sampling · p. 3
  • For non-neighbor pairs, the measured brightness difference is the sum of polarities of intervening events.This supports event pairs separated by more than one event in time.Source: C. Event Sampling · p. 3
  • No-event samples come from pixels with no events during an interval, and the final loss combines event and weighted no-event losses.When enabled, training samples two thirds event rays and one third no-event rays.Source: C. Event Sampling · p. 3

D. Implementation Details

The implementation uses torch-ngp with hash-based encoding and external motion-capture poses. Training typically takes one to three hours on one NVIDIA A40.

D. Implementation Details

IV. EXPERIMENTS

The evaluation compares E-NeRF with frame-based NeRF, Deblur-NeRF, and an E2VID-to-NeRF baseline in challenging robotic-vision scenarios.

IV. EXPERIMENTS

  • The evaluation compares E-NeRF against frame-based torch-ngp, Deblur-NeRF for motion-blur cases, and E2VID reconstruction followed by torch-ngp.Source: §IV · p. 4
  • Experiments use synthetic ESIM data and public real-world benchmarks.Source: §IV · p. 4
  • The scenarios are selected as settings frequent in robotics and challenging for frame-based and event-based vision.RawNeRF is excluded because public code was unavailable and it requires raw images captured with a static camera during exposure.Source: §IV · p. 4

A. Synthetic Data

On synthetic motion-blur sequences, E-NeRF outperforms the reported baselines, including in event-only and combined event-and-frame settings. The experiments also examine event sampling, no-event losses, and qualitative reconstructions.

A. Synthetic Data

  • The five ESIM sequences are evaluated against sharp holdout frames using PSNR, SSIM, and AlexNet-based LPIPS, after affine brightness adjustment.Brightness is adjusted because absolute brightness is not observable from the event stream.Source: A. Synthetic Data · p. 4
  • E-NeRF produces better results than the baselines on simulated motion-blur data, including compared with E2VID followed by torch-ngp.E-NeRF fuses information in 3D and leverages spatial consistency without training data; frame-based methods cannot overcome strong blur.Source: A. Synthetic Data · p. 4
  • For Shake Carpet 1, event-only E-NeRF recovers sharp details, while combining grayscale events with blurry color frames yields sharp, colored reconstructions.The resulting colors are not photorealistic, but may still be useful for object recognition.Source: Figure 2 · p. 4, A. Synthetic Data · p. 4
  • E-NeRF reconstructs sharp details that can benefit robotics tasks such as license-plate detection, where legible characters matter more than overall photorealism.Source: A. Synthetic Data · p. 4, Figure 2 · p. 4
  • Combining events with frames uses a weighted loss; frames can correct fog-like artifacts in uniform areas and color inferred imperfectly from event data.The method maps predicted event-ray colors to grayscale and does not require a color event camera.Source: A. Synthetic Data · p. 4
  • Changing the loss to a normalized form and increasing average event-pair distance to about 30 ms each cause minor performance drops.The authors attribute the longer-distance sampling drop possibly to repeatedly sampled rays; training is unstable on Shake Moon 1 and 2, so accumulation is capped at five events there.Source: A. Synthetic Data · pp. 4–5
  • E-NeRF shows the best results in the event-only setting and outperforms NeRF trained on blurry frames on the shake sequences.Deblur-NeRF beats NeRF for mild blur, but not for strong blur in Shake Carpet 1 or the spiral sequences.Source: A. Synthetic Data · pp. 4–5
  • Adding the no-event loss also causes a minor metric drop, but helps remove artifacts in uniformly colored areas.The reported configuration uses weight 0.1, a 25 ms no-event interval, and up to one fourth no-event rays per batch; it can improve visual quality on real data when used with event accumulation.Source: A. Synthetic Data · p. 5

B. Real Data

On challenging real-world sequences, event-based methods recover details that frame-based methods lose under low light or fast motion, with E-NeRF often improving detail and geometry over E2VID+torch-ngp. On TUMVIE, combining events and frames helps reconstruct fine text and uniform-color regions, including with few input views.

Real-world EDS results

  • On EDS sequence 00, event-based methods show more detail in dark regions, and E-NeRF has fewer foreground artefacts than E2VID+torch-ngp; on sequence 11, E-NeRF recovers table geometry and sharp lettering.No method achieves photorealistic quality in these scenarios.Source: B. Real Data · p. 5, Figure 4 · p. 6

Discussion

  • The real-data comparison is qualitative because the selected sequences have severely blurred frames or low dynamic range; the event-generation model is also only an approximation in these difficult conditions.The authors say this model limitation narrows E-NeRF’s advantage over E2VID+torch-ngp.Source: B. Real Data · pp. 5, 7

Real-world TUMVIE results

  • On TUMVIE mocap-desk2, E-NeRF reconstructs fine text on books and paper that frame-based torch-ngp cannot, while combining events and frames improves uniformly colored regions.The sequence has mild motion blur and high-quality frame-camera images.Source: B. Real Data · pp. 5, 7
  • With only six input views on mocap-desk2, torch-ngp fails to recover correct geometry, while E2VID+torch-ngp reconstructs uniform-color areas but not text.E-NeRF’s conclusion identifies events as helpful when only a few views are provided.Source: B. Real Data · p. 7

V. CONCLUSION

The paper presents E-NeRF for reconstructing NeRFs from fast-moving event cameras, with results on synthetic and real data showing gains in detail and geometry. Supplementary findings describe the effects of no-event sampling and the trade-off when combining event data with RGB frames.

V. CONCLUSION

  • E-NeRF reconstructs NeRFs from a fast-moving event camera and outperforms multiple baselines on synthetic and real-world data, particularly in high-frequency details such as text.The conclusion also reports that events can help when few input views are available and can be combined with color frames for sharp, colored reconstructions.Source: §V · p. 7

I. SUPPLEMENTARY MATERIAL

  • No-event sampling makes uniform-brightness areas smoother; on the reported ablation, SSIM and LPIPS improve for λnoevs = 1 and λnoevs = 10, while PSNR drops slightly.The authors note most other examples lack large uniform-brightness areas, limiting the observed effect in the ablation.Source: §I · p. 9

B. Details on combining frames and events

  • For event-and-frame combinations, higher λrgb yields more motion-blur artefacts, while lower λrgb reduces those artefacts but makes colors less realistic because events provide grayscale information.The method uses a simple weighted combination and can work with sparse, heavily blurred RGB views.Source: B. Details on combining frames and… · p. 9

C. Additional results for slow camera motion

The additional experiments examine E-NeRF under slow camera motion, encoding choices, runtime, and depth quality. They report broadly comparable training time, encoding-dependent quality and cost, and noisy depth maps despite a relative improvement over baselines.

C. Additional results for slow camera motion

  • Both methods produce reasonable reconstructions, but E-NeRF’s evaluation is disadvantaged by training and evaluation from different camera viewpoints and is sensitive to calibration errors.The methods are evaluated in the left grayscale camera viewpoint, while E-NeRF is trained from the left event camera; the cameras have an approximately 4.49 cm baseline.Source: C. Additional results for slow camera… · p. 9, Table II · p. 10

D. Runtime Analysis

  • Increasing λ_noevs regularizes non-event uniform-brightness areas, making regions such as the landing ground and road network smoother.The no-event locations are determined using T_noevs = 20 milliseconds.Source: Figure 1 · p. 10
  • E-NeRF and torch-ngp require approximately the same training time, with E-NeRF adding slight overhead for event sampling.The authors suggest more efficient data structures could speed event sampling; runtime is not the paper’s focus.Source: D. Runtime Analysis · pp. 9–10

E. Details on the Encoding

  • PSNR fell from 26.65 to 24.99 (-1.66) at L = 14, while L = 24 reached 27.42 (+0.77) versus the default L = 16.Using L = 24 increases computational time by around 2.24×.Source: E. Details on the Encoding · p. 10, Table V · p. 11
  • Nmax = 512 caused a performance drop, whereas Nmax = 73728 brought only a minor improvement.Nmax is scaled per scene by 2048 · B; for Shake Carpet 1, B = 3.Source: E. Details on the Encoding · p. 10

F. Depth results

  • Depth maps remain noisy, although E-NeRF’s are described as less noisy and overall better; explicit depth regularizers are omitted because the work focuses on novel-view quality.The reported depth quality is similar to plain NeRF, with explicit depth handling left as a possible extension.Source: F. Depth results · p. 11, Figure 4 · p. 12

Related papers

Loading 2208.11300v2…