PaperlayerView exact revision on arXiv

Source-linked AI summary

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction

Daniel Gehrig, Michelle Rüegg, Mathias Gehrig, Javier Hidalgo Carrio, Davide Scaramuzza

arXiv:2102.09320v1cs.CV

TL;DR

Combining event streams with intensity frames is difficult because the modalities are asynchronous and traditional RNNs are designed for synchronized, fixed-rate inputs. The paper introduces RAM networks, which update a shared internal state asynchronously and apply it to monocular depth estimation; RAM Net outperforms state-of-the-art event-based methods by up to 30% and the work releases EventScape.

  • Problem

    Combining events and frames for learning-based monocular depth estimation is challenging because they are measured asynchronously and at different rates, beyond classical RNN assumptions.

  • Method

    RAM networks generalize RNNs by maintaining a shared internal state updated asynchronously by either modality and decoded into predictions at any time.

  • Results

    RAM Net outperforms state-of-the-art event-based methods by up to 30% in monocular depth estimation.

  • Takeaways & Limitations

    EventScape provides a CARLA-based multimodal dataset with events, intensity frames, semantic labels, depth maps, and vehicle navigation parameters for further research.

Abstract

from arXiv · show

Event cameras are novel vision sensors that report per-pixel brightness changes as a stream of asynchronous "events". They offer significant advantages compared to standard cameras due to their high temporal resolution, high dynamic range and lack of motion blur. However, events only measure the varying component of the visual signal, which limits their ability to encode scene context. By contrast, standard cameras measure absolute intensity frames, which capture a much richer representation of the scene. Both sensors are thus complementary. However, due to the asynchronous nature of events, combining them with synchronous images remains challenging, especially for learning-based methods. This is because traditional recurrent neural networks (RNNs) are not designed for asynchronous and irregular data from additional sensors. To address this challenge, we introduce Recurrent Asynchronous Multimodal (RAM) networks, which generalize traditional RNNs to handle asynchronous and irregular data from multiple sensors. Inspired by traditional RNNs, RAM networks maintain a hidden state that is updated asynchronously and can be queried at any time to generate a prediction. We apply this novel architecture to monocular depth estimation with events and frames where we show an improvement over state-of-the-art methods by up to 30% in terms of mean absolute depth error. To enable further research on multimodal learning with events, we release EventScape, a new dataset with events, intensity frames, semantic labels, and depth maps recorded in the CARLA simulator.

MULTIMEDIA MATERIAL

The paper introduces RAM networks for combining asynchronous events and frames in monocular depth estimation, and releases EventScape to support multimodal research.

  • Event cameras report asynchronous per-pixel brightness changes, while standard cameras capture absolute brightness; their complementarity motivates fusion algorithms.
  • Fusing events and frames is challenging because the modalities are measured asynchronously and at different rates, while classical RNNs assume fixed-rate synchronized inputs.
  • RAM networks generalize traditional RNNs for asynchronous multimodal data by maintaining an internal state updated by either modality and decoded into predictions at any time.
  • RAM networks are applied to monocular depth estimation from events and frames, targeting temporal contexts unavailable from single-modality inputs.
  • EventScape is a CARLA-based synthetic dataset containing events, intensity frames, semantic labels, depth maps, and vehicle navigation parameters in diverse automotive scenes.The dataset addresses the scarcity of high-quality event-and-frame datasets for learning depth.
  • RAM Net outperforms state-of-the-art event-based methods by up to 30% and is reported to generalize better to unseen asynchronous data than classical RNNs.

II. RELATED WORK

Prior approaches struggle with asynchronous multimodal inputs because conventional fusion methods depend on sensor models or synchronization, while RAM networks retain recurrent modeling with asynchronous updates.

  • Event-frame fusion methods have supported tasks including SLAM, feature tracking, high-dynamic-range reconstruction, and deblurring, but often depend strongly on ideal sensor models.
  • Traditional RNNs struggle when input sampling rates vary or become asynchronous, and padding, copying, or rate conversion can reduce frequency or create temporal misalignment.
  • RAM networks address these limitations by applying data-driven recurrent models to complex tasks, maximizing temporal context, and using asynchronous state updates for each modality.

A. Monocular Depth Estimation with Events

RAM networks extend recurrent monocular depth estimation to asynchronously fuse events and frames, preserving temporal context from both modalities. They encode sensor-specific measurements into features, update a shared latent state as measurements arrive, and decode depth predictions at any timestep.

  • A. Monocular Depth Estimation with Events: RAM networks estimate pixel-wise scene depth from combined event and frame inputs, extending recurrent architectures to asynchronous and irregular multimodal data.The approach targets monocular depth estimation while retaining complementary information from events and frames.
  • A. Monocular Depth Estimation with Events: Each sensor measurement is mapped by a sensor-specific encoder into intermediate features before multimodal fusion.This feature stream may contain measurements from sensors with different formats and variable rates.
  • A. Monocular Depth Estimation with Events: Sensor-specific state operators update a latent state whenever either modality becomes available, preserving asynchronous ordering and variable data rates.The operators are applied sequentially as measurements arrive, rather than requiring synchronized inputs.
  • A. Monocular Depth Estimation with Events: Recursive state updates ensure the latent representation contains information from both modalities regardless of which sensor was observed last.The resulting latent variables can be decoded into task predictions at each timestep.
  • A. Monocular Depth Estimation with Events: RAM networks are Markovian because predictions depend only on the current latent state, which encodes the history of sensor measurements.This connects the architecture to hidden Markov models through the same conditional-independence structure.

B. Network Architecture for monocular depth estimation

For monocular depth estimation, RAM Net adapts a U-Net-inspired architecture by combining multiscale encoder features with recurrent latent states and decoding the result through several levels.

  • B. Network Architecture for monocular depth estimation: RAM Net uses a U-Net-inspired architecture with ConvGRU fusion at each scale between intermediate encoder features and the previous latent state.The lowest-scale latent variable is fed into a residual block followed by three decoder levels.

C. Event Generation Model

Event cameras trigger asynchronous events when per-pixel log-brightness changes exceed a threshold. Events encode location, time, and polarity, and are grouped into fixed-size voxel grids for convolutional processing.

  • C. Event Generation Model: An event is triggered at a pixel when the change in log brightness since its previous event exceeds threshold C.The event records the pixel location, timestamp, and polarity of the brightness change.
  • C. Event Generation Model: Δt_k denotes the elapsed time since the previous event at the same pixel.
  • C. Event Generation Model: Event streams are converted into fixed-size voxel grids with spatial dimensions H × W and B temporal bins over a time window ΔT.This conversion enables the use of existing convolutional neural network architectures.
  • C. Event Generation Model: The experiments use B = 5 temporal bins for every dataset.

E. Depth Representation

Depth targets are represented as normalized logarithmic maps to facilitate learning across large depth ranges, while training combines scale-invariant depth and multiscale gradient losses over valid labels.

  • E. Depth Representation: Metric depth is converted into a normalized log-depth map in [0, 1] to facilitate learning of large depth variations.
  • E. Depth Representation: EventScape uses α = 5.7 and Dmax = 1000 m, while MVSEC uses α = 3.7 and Dmax = 80 m.The closest observed depth is mapped to 0, and monocular depth is estimated up to scale.
  • E. Depth Representation: RAM Net is supervised using sparse ground-truth depth labels and combines scale-invariant loss with multiscale scale-invariant gradient matching loss.EventScape labels come from CARLA, while MVSEC labels are measured with LiDAR; losses are summed over event-and-image sequences.
  • E. Depth Representation: The gradient loss uses Sobel edges at four scales to favor smooth gradient changes and sharp predicted depth discontinuities.Its weight is set to λ = 0.25.
  • E. Depth Representation: Training normalizes inputs, randomly crops them to 224 × 224, applies random horizontal flipping, and standardizes nonzero voxel-grid entries.
  • E. Depth Representation: The network is implemented in PyTorch and optimized with ADAM using dataset-specific learning rates and iteration counts.EventScape uses learning rate 0.0003 for 27’000 iterations, whereas MVSEC uses 0.001 for 7’600 iterations.

IV. EXPERIMENTAL SETUP

The experiments validate RAM Net on synthetic EventScape and real-world MVSEC data against unimodal, multimodal, frame-based, and event-based baselines. RAM Net outperforms state-of-the-art methods by up to 30% in absolute relative depth error.

  • RAM Net is evaluated on synthetic EventScape and real-world MVSEC data against event-only, frame-only, multimodal, and state-of-the-art baselines.
  • EventScape provides events, images, depth maps, segmentation labels, and vehicle navigation parameters for multimodal evaluation.
  • 30% improvement is achieved over state-of-the-art methods in absolute relative depth error.

A. Datasets

The experiments use EventScape to compare modality and temporal-update strategies under asynchronous inputs, with additional evaluation on MVSEC. Baselines isolate multimodality, asynchronous updating, and recurrence.

  • Datasets: EventScape contains events, frames, depth, semantic labels, and vehicle control data, with frames, depth, and labels at 25Hz and control data at 1000Hz.Events are generated from 500Hz CARLA images using an event-camera simulator.
  • Baselines: The recurrent baselines comprise event-only E, frame-only I, and event-and-frame E+I models for isolating multimodality and asynchronous updating.E+I receives the same modalities as RAM Net but uses copied frames, creating temporal mismatch with events.
  • Training setup: Training uses 5Hz images and depth labels with voxel grids at 25Hz, organized as sequences containing 10 frames and 50 voxel grids.
  • Results: Unlike recurrent I and E+I baselines, RAM Net produces separate predictions for event and image inputs at different temporal points.
  • Results: RAM Net outperforms all baselines after frame prediction, while its error increases gradually as event updates continue.At frame index 4, RAM Net reduces absolute relative difference by 49% versus E and 38% versus E+I.

B. Generalization to Different Data Rates

RAM Net retains comparatively low error when image frequency changes and generalizes better than the frame-only baseline from synthetic to real data. Real-world performance improves after fine-tuning, although sky artifacts remain a limitation.

  • Different data rates: When tested with images at 1Hz instead of the trained 5Hz, RAM Net degrades gracefully and retains lower error than E+I and E.Its accuracy reaches a plateau at frame index 20 as image gaps increase.
  • Evaluation protocol: Evaluation on MVSEC aligns irregular event packets with depth labels because ground-truth depth and images are not synchronized.
  • Real-world generalization: RAM Net generalizes better than the frame-only baseline from synthetic data to MVSEC, but synthetic-to-real results remain worse than E2Depth.
  • Real-world generalization: After fine-tuning on MVSEC, both the frame-only baseline and RAM Net improve, and RAM Net outperforms nearly all methods across datasets.The frame-only baseline also demonstrates the importance of image data for depth prediction.
  • Limitations: Sky artifacts arise because sky regions are masked during training and differ between clear training images and overcast test images.The authors suggest more realistic training data or semantic segmentation to improve sky predictions.

D. Timing Results

RAM Net supports high-throughput multimodal processing while maintaining asynchronous prediction capability. On a Quadro RTX 8000, its measured throughput reaches up to 320 Hz.

  • 320 Hz throughput is achieved on a Quadro RTX 8000, with voxel-grid and image processing times of 3.08 ms and 3.14 ms.
  • RAM Net generates high-quality depth predictions during the blind-time between image frames on both EventScape and MVSEC.

VII. APPENDIX

The appendix reports qualitative MVSEC comparisons and modality-ablation analyses, alongside architectural details for RAM Net and its baselines.

  • Qualitative results: RAM Net is qualitatively compared with E2Depth and MegaDepth on MVSEC, using LiDAR depth as ground truth.
  • Input-modality analysis: Removing events causes errors in sky prediction and reduces overall sharpness in the resulting depth maps.
  • Input-modality analysis: Removing frames causes intensity differences to be mistakenly incorporated into predicted depth, including in the building on the left.
  • Input-modality analysis: Image data helps distinguish object borders from the sky, while events provide geometric structure cues missing from image-only predictions.
  • Architecture details: RAM Net uses a non-recurrent encoder with recurrent state-combination blocks, whereas baselines use ConvLSTMs at the encoder stage.
Loading 2102.09320v1…