Source-linked AI summary
Learning Monocular Dense Depth from Events
Javier Hidalgo-Carrió, Daniel Gehrig, Davide Scaramuzza
TL;DR
Event-based monocular depth prediction must produce dense scene depth despite sparse event responses and limitations in existing feed-forward methods. The paper introduces a recurrent network trained with synthetic CARLA data and reports more accurate dense depth maps than existing methods, while evaluation excludes night-driving sequences during training.
Problem
Event-based depth prediction must estimate dense per-pixel scene depth, while existing approaches do not leverage temporal consistency and events are inherently sparse because they respond predominantly to scene edges.
Method
A recurrent network predicts dense per-pixel depth from a monocular event camera, using the DENSE CARLA dataset and supervised scale-invariant multi-scale training losses.
Results
The method produces dense depth maps with more accuracy than existing methods on the MVSEC dataset.
Takeaways & Limitations
Training on synthetic data benefits convergence, depth-map quality through perfect ground truth, and coverage of varied conditions.
Takeaways & Limitations
The training data for the evaluated methods excludes night-driving sequences.
Abstract
from arXiv · showhide
Event cameras are novel sensors that output brightness changes in the form of a stream of asynchronous events instead of intensity frames. Compared to conventional image sensors, they offer significant advantages: high temporal resolution, high dynamic range, no motion blur, and much lower bandwidth. Recently, learning-based approaches have been applied to event-based data, thus unlocking their potential and making significant progress in a variety of tasks, such as monocular depth prediction. Most existing approaches use standard feed-forward architectures to generate network predictions, which do not leverage the temporal consistency presents in the event stream. We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods. In particular, our method generates dense depth predictions using a monocular setup, which has not been shown previously. We pretrain our model using a new dataset containing events and depth maps recorded in the CARLA simulator. We test our method on the Multi Vehicle Stereo Event Camera Dataset (MVSEC). Quantitative experiments show up to 50% improvement in average depth error with respect to previous event-based methods.
1. Introduction
Event-based dense monocular depth is difficult because asynchronous events are sparse, especially in low-contrast regions, while prior methods were generally sparse, semi-dense, or stereo-based. The paper addresses this gap with a recurrent network, a synthetic CARLA dataset, and evaluation on MVSEC.
- Event cameras report asynchronous pixel-level brightness changes rather than fixed-rate intensity images.
- Dense depth estimation is challenging because events respond mainly to edges, leaving low-contrast regions without measurements.
- Prior event-based methods typically produced sparse or semi-dense depth, while dense prediction relied on stereo.
- The paper introduces a recurrent network that predicts dense per-pixel depth from a monocular event camera.
- DENSE provides synthetic events and perfect ground truth, while the method is evaluated on the MVSEC dataset.
- The method shows improved performance with respect to the state of the art on MVSEC.
2. Related Work
Prior depth-estimation methods include classical model-based approaches, image-based deep networks, and event-based stereo or learning systems. The paper positions its approach as dense metric depth prediction from a single event camera using recurrent temporal modeling.
- Image-based monocular depth evolved through supervised CNNs, improved losses, and self-supervised joint pose-and-depth estimation.
- Classical event-based methods used stereo or SLAM to estimate metric depth through jointly optimized pose and maps.
- Classical model-based methods produce sparse point clouds or semi-dense depth, depending on whether they are feature-based or direct.
- Stereo event-camera methods estimate depth by maximizing temporal consistency between paired event streams.
- Learning-based event methods integrate multiple event-stream cues, but earlier systems remained semi-dense or required stereo for dense metric depth.
- This work instead predicts dense metric depth from a single monocular camera by exploiting event-stream temporal consistency with a recurrent convolutional network.
3. Depth Estimation Approach
The approach converts asynchronous event windows into voxel grids and processes them with a recurrent convolutional network to predict dense monocular depth. It uses normalized log-depth outputs, supervised sequence losses, and synthetic-to-real training to address limited dense real ground truth.
- 3.2. Network Architecture: The model is a recurrent fully convolutional network with an internal state that predicts normalized log depth maps for each event window.It is based on a UNet architecture with recurrent encoder layers, residual blocks, decoder layers, skip connections, and a final depth prediction layer.
- 3.1. Event Representation: Event streams are processed in non-overlapping windows and converted into tensor-like voxel grids before recurrent depth prediction.The experiments use 50 ms windows, five temporal bins, and normalize non-zero voxel values to zero mean and unit variance.
- 3.3. Depth Map Post-processing: Metric depth is recovered from normalized log depth using the maximum expected depth and a parameter mapping the minimum output to the minimum observed depth.The reported settings are Dmax = 80 meters and α = 3.7, corresponding to a minimum depth of 2 meters.
- 3.4. Training Details: Training minimizes scale-invariant and multi-scale gradient-matching losses at every time step using ground-truth depth sequences.The gradient term encourages smooth depth changes while preserving sharp depth discontinuities, and four scales are used.
- 3.5. Training Data: The network is pretrained on synthetic CARLA events and depth maps, then finetuned with real MVSEC events to obtain metric-scale predictions.The DENSE dataset contains eight sequences split into training, validation, and testing sets, with events and depth maps used for network training.
4. Experiments
Experiments evaluate the recurrent dense monocular depth method on MVSEC, including training-set ablations, qualitative comparisons, and comparisons with image- and event-based methods. Synthetic pretraining and combined synthetic-real training improve results, with reported gains across depth ranges and sequences.
- Training-set ablation: The MVSEC evaluation compares training-set variants using the same network, including synthetic, real, and synthetic-to-real training configurations.Table 2 defines S, R, S∗, S∗→R, and S∗→(S + R) as alternative training sets.
- Training-set ablation: Synthetic data enhance the ablation results, while increasing the amount of training data improves monocular depth prediction potential.The study reports these conclusions from quantitative results and qualitative depth maps.
- Comparison with prior methods: 26.25% at 10m, 25.25% at 20m, and 21.0% at 30m are the average improvements over values reported in.The comparison uses average mean errors at cut-off depths of 10m, 20m, and 30m.
- Comparison with prior methods: Up to 50.0% improvement is reported for the outdoor night3 MVSEC sequence relative to previous methods.The paper also reports that image-based methods have difficulties predicting depth in low-light conditions.
- Qualitative comparison: Qualitative MVSEC comparisons show events, dense predictions, ground truth, DAVIS grayscale frames, and MegaDepth predictions across four test sequences.The figure includes outdoor day1, night1, night2, and night3 samples.
5. Conclusion
The paper presents monocular dense depth estimation from asynchronous events using recurrent convolutional networks and evaluates it on MVSEC and DENSE. It reports that synthetic training benefits convergence and depth quality, while the method produces accurate dense depth maps.
- Contribution: The paper presents the first work on monocular dense depth estimation from events.Its solution uses recurrent convolutional neural networks to infer dense depth from asynchronous event streams.
- Evaluation: The method is evaluated on MVSEC, described as the only currently public dataset containing events, frames, and ground-truth depth.The conclusion identifies MVSEC as the evaluation dataset.
- Synthetic training: Synthetic training helps the network converge faster, improves depth-map quality through perfect ground truth, and captures a larger variety of conditions in simulation.These benefits are stated in the conclusion as reasons synthetic data are useful.
- Conclusion: The methodology produces dense depth maps with more accuracy than existing methods.This is the paper’s concluding comparison to prior methods.
A. Results on Synthetic Data
On the DENSE dataset, the method achieves quantitative and qualitative results, with validation performance on Town06 and Town07 within the range of state-of-the-art image-based methods.
- The method is evaluated quantitatively and qualitatively on the DENSE dataset.
- Validation results are reported on the Town06 and Town07 sequences.
- The metric results fall within the range of state-of-the-art image-based methods on popular datasets such as KITTI.
B. Why using events for Depth prediction?
Event cameras capture salient edges and detailed brightness changes with high temporal-spatial resolution, making them suitable for depth prediction. The paper illustrates event-based depth estimation in HDR driving and compares it with frame-based alternatives.
- Event cameras capture salient edges and detailed brightness changes with high temporal-spatial resolution, which are relevant for estimating depth.
- In an HDR highway scene facing the sun, the method predicts depth directly from events.
- The qualitative comparisons contrast the proposed event-based depth prediction with MegaDepth applied to grayscale or reconstructed frames.