Source-linked AI summary
Recurrent Vision Transformers for Object Detection with Event Cameras
Mathias Gehrig, Davide Scaramuzza
TL;DR
Event-based object detection must efficiently capture both local and global spatial features and information from recent and older events, while frame-based systems face latency-bandwidth trade-offs in dynamic scenes. RVTs revisit backbone macro-design with recurrent, multi-stage processing and achieve state-of-the-art detection performance alongside fast, lightweight inference.
Problem
Event-based object detection must efficiently extract local and global features and retain information from both recent and older events, while frame-based cameras trade resolution against frame rate under fixed bandwidth.
Method
RVTs use a repeated multi-stage hierarchical backbone combining convolutional priors, interleaved local- and global self-attention, and recurrent temporal feature aggregation.
Results
47.2% mAP on the Gen1 detection dataset and 47.4% mAP on the 1 Mpx detection dataset were achieved while training from scratch.
Takeaways & Limitations
The canonical stage design is compatible with existing detection frameworks and supports low-latency event-camera object detection on conventional hardware.
Takeaways & Limitations
The event representation does not leverage the full potential of event-based data, and efficient low-level event processing remains an open problem.
Abstract
from arXiv · showhide
We present Recurrent Vision Transformers (RVTs), a novel backbone for object detection with event cameras. Event cameras provide visual information with sub-millisecond latency at a high-dynamic range and with strong robustness against motion blur. These unique properties offer great potential for low-latency object detection and tracking in time-critical scenarios. Prior work in event-based vision has achieved outstanding detection performance but at the cost of substantial inference time, typically beyond 40 milliseconds. By revisiting the high-level design of recurrent vision backbones, we reduce inference time by a factor of 6 while retaining similar performance. To achieve this, we explore a multi-stage design that utilizes three key concepts in each stage: First, a convolutional prior that can be regarded as a conditional positional embedding. Second, local and dilated global self-attention for spatial feature interaction. Third, recurrent temporal feature aggregation to minimize latency while retaining temporal information. RVTs can be trained from scratch to reach state-of-the-art performance on event-based object detection - achieving an mAP of 47.2% on the Gen1 automotive dataset. At the same time, RVTs offer fast inference (<12 ms on a T4 GPU) and favorable parameter efficiency (5 times fewer than prior art). Our study brings new insights into effective design choices that can be fruitful for research beyond event-based vision.
1. Introduction
Event cameras offer low-latency, high-dynamic-range sensing for time-critical detection, but existing accurate methods are often too slow. RVTs revisit backbone design to improve the accuracy–efficiency trade-off while retaining strong detection performance.
- Motivation: 30 milliseconds can correspond to up to 1 meter of car travel, while an ordinary camera captures only one frame.This motivates low-latency perception in dynamic and safety-critical scenarios.
- Motivation: Frame-based cameras trade resolution against frame rate under fixed bandwidth, potentially missing essential details in highly dynamic scenes.The paper notes that this can be especially consequential in automotive safety-critical settings.
- Motivation: Event cameras provide sub-millisecond visual latency and high dynamic range by encoding brightness changes as asynchronous events.They sacrifice absolute intensity information while recording event time, location, and polarity.
- Research Gap: Prior dense approaches often exceed 40 milliseconds, while sparse or graph-based methods may require specialized hardware or improved detection performance.The central challenge is achieving both high accuracy and efficiency without specialized hardware.
- Approach: RVTs combine convolutional priors, interleaved local/global attention, and recurrent temporal aggregation in a four-stage hierarchical backbone.The design targets efficient spatial and temporal feature processing and is compatible with off-the-shelf detection frameworks.
- Results: 47.2% mAP is achieved on Gen1 and 47.4% mAP on 1 Mpx, while training from scratch and reducing parameters from 100M to 18.5M.Inference time is reduced by up to a factor of 6 compared to prior art.
2. Related Work
Event-camera object detection spans graph, spiking, and dense neural-network approaches. Dense recurrent methods achieve strong performance but remain slow, motivating a backbone redesign that combines event-based recurrence with transformer-style spatial modeling.
- Research Directions: Event-camera object detection is broadly organized into graph neural networks, spiking neural networks, and dense neural networks.These form three emerging research directions in the literature.
- Graph Neural Networks: Graph neural networks dynamically construct spatio-temporal graphs, but must propagate information across large space-time distances.This is particularly relevant for large objects moving slowly relative to the camera.
- Spiking Neural Networks: Spiking neural networks propagate information sparsely through internal neuron states, but nondifferentiable spike generation complicates optimization.Their sparse processing is closely related to recurrent neural networks but uses threshold-triggered spikes.
- Dense Neural Networks: Dense methods convert event streams into tensors compatible with convolutions, and recurrence improves detection by retaining information beyond short temporal windows.Without recurrence, slowly moving objects may generate too few events within the processed window.
- Transformers: Vision transformers provide promising spatial-temporal modeling, but prior video applications are optimized for offline stored video and event-based detection use remained limited.The paper positions RVTs as applying attention-based design to event-camera object detection.
3. Method
RVT processes event tensors through repeated hierarchical stages that combine convolutional spatial priors, local and dilated global attention, and recurrent temporal aggregation. The resulting backbone exposes recurrent features to standard detection frameworks.
- Event Processing: Incoming event streams are converted into tensors, then processed sequentially using previous recurrent-layer states.The recurrent outputs are passed to the detection framework after backbone processing.
- Event Processing: The event representation contains T 2-channel frames for positive and negative events and is flattened into a (2T, H, W) tensor for 2D convolutions.Byte tensors are used to reduce memory and bandwidth.
- Spatial Feature Extraction: Each stage begins with an overlapping convolution that downsamples features and supplies a conditional positional embedding from the 2D pixel grid.This removes the need for absolute or relative positional embeddings.
- Spatial Feature Extraction: Block-SA models local interactions in non-overlapping windows, while Grid-SA performs dilated global feature mixing without full quadratic global attention.The two attention operations provide local and global spatial interaction at lower cost.
- Temporal Feature Extraction: Plain LSTM cells aggregate temporal features separately for each feature, avoiding the higher cost of Conv-LSTM units.A Conv-LSTM with kernel size k × k and stride 1 requires k^2 times the parameters and compute of an original LSTM cell.
- Hierarchical Design: Four RVT blocks form a multi-stage hierarchical backbone, with LSTM states retained across timesteps and hidden states from stages two through four used for detection.The architecture variants share the same parameter set except for channels per stage.
4. Experiments
Experiments evaluate RVT variants, datasets, spatial and temporal design choices, augmentation, benchmark performance, and inference speed. RVTs achieve strong detection accuracy while substantially reducing latency and parameter count relative to prior approaches.
- 4.1. Setup: Experiments use Gen1 and 1 Mpx event-camera datasets, evaluating RVT-B, RVT-S, and RVT-T variants with mAP as the main metric.The Gen1 dataset has 304 × 240 resolution; the 1 Mpx dataset provides higher-resolution driving recordings.
- 4.2.1 Model Components: Multi-axis self-attention outperforms Swin and ConvNext alternatives on both the Gen1 and 1 Mpx datasets.The comparison keeps the LSTM and convolutional downsampling layers identical while exchanging attention and MLP modules.
- 4.2.1 Model Components: Overlapping convolutional kernels improve performance compared with non-overlapping kernels, with a slight increase in parameter count.The study therefore uses overlapping kernels throughout the network stages.
- 4.2.1 Model Components: Plain LSTM cells outperform the evaluated Conv-LSTM variants, supporting temporal aggregation without convolution inside the recurrent cell.The model uses plain LSTMs because temporal and spatial feature aggregation can be separated, reducing computational complexity and parameter count.
- 4.2.1 Model Components: Using recurrence in every stage improves detection, while removing recurrence entirely causes a drastic performance decline.Adding an LSTM to the first stage provides additional improvement, although the mAP increase is small.
- 4.3. Benchmark Comparisons: Recurrent models outperform sparse and dense feed-forward approaches by more than 10 mAP on both evaluated datasets.The comparison includes GNNs, SNNs, and dense feed-forward detectors without recurrent layers.
- 4.3. Benchmark Comparisons: 47.2 mAP on Gen1 and 47.4 mAP on 1 Mpx establish new state-of-the-art results for the RVT base model.The RVT tiny model achieves 4.1 higher mAP on Gen1 than RED while using 5 times fewer parameters.
- 4.3. Benchmark Comparisons: 10.2 ms inference time on Gen1 and 11.9 ms on 1 Mpx make the RVT base model faster than the reported RED and ASTMNet timings.With torch.compile, RVT-T reaches 2.3 ms on Gen1 and 3.5 ms on 1 Mpx.
5. Discussion and Limitations
The paper identifies limitations in its event representation and scope: low-level temporal processing remains unresolved, and using events alone leaves complementary frame information unused.
- Discussion and Limitations: The event representation uses fully connected layers over time, providing only a weak prior on event order.The authors note that temporal convolutions in early layers have produced substantial gains.
- Discussion and Limitations: Efficient low-level processing of event data remains an open research problem not addressed in this work.
- Discussion and Limitations: The approach uses only event streams, so it does not exploit complementary information available from frames.The authors propose multimodal integration on suitable datasets as a promising next step.
- Discussion and Limitations: When events are unavailable, the model can retain information temporarily, but its memory fades and detection performance deteriorates.Low-frame-rate, high-quality images are suggested as a possible source of missing complementary information.
6. Conclusion
The paper concludes that RVTs provide a compact recurrent backbone for event-camera object detection, combining spatial priors, attention, and temporal aggregation in a hierarchical design.
- 6. Conclusion: RVTs repeatedly apply a canonical stage design to form a multi-stage hierarchical neural network.Each stage incorporates convolutional priors, local- and sparse global attention, and recurrent feature aggregation.
- 6. Conclusion: RVTs can be trained from scratch to reach state-of-the-art object-detection performance with event cameras.
- 6. Conclusion: The stage design is directly compatible with existing detection frameworks and supports low-latency detection on conventional hardware.The authors also present it as a basis for future neuromorphic-system designs.
1. Relationship to Temporal Graph Neural Networks (TGNNs)
The RVT backbone can be interpreted as a discrete-time dynamic graph with persistent temporal nodes, while its constrained design targets a better task-performance and inference-speed trade-off.
- 1. Relationship to Temporal Graph Neural Networks (TGNNs): The RVT backbone is an instance of a discrete-time dynamic graph with a fixed number of persistent temporal nodes for a given input resolution.Its hierarchy contains four layers of nodes with temporal history through LSTMs in each stage.
- 1. Relationship to Temporal Graph Neural Networks (TGNNs): The strided convolution can be viewed as creating fewer nodes, followed by local-window or global-dilated-grid edges for self-attention message passing.The resulting node features feed temporal LSTMs that use states from the previous timestamp.
- 1. Relationship to Temporal Graph Neural Networks (TGNNs): The backbone is specifically constrained for event-based vision because arbitrary discrete- or continuous-time dynamic graphs do not necessarily provide the desired trade-off.
2. Data Pipeline
The data pipeline combines BPTT and TBPTT loaders so training benefits from diverse augmentation and initialized temporal states, while mixed loading improves performance with short sequences.
- 2. Data Pipeline: BPTT randomly samples short sequences and supports more aggressive augmentation, but can limit generalization to longer sequences.
- 2. Data Pipeline: Mixed dataloading yields the best results on the 1 Mpx validation set.It enables stable training with sequence length 5 and improved detection performance, whereas longer BPTT sequences increase memory and training time.
- 2. Data Pipeline: TBPTT uses consecutive chunks with detached hidden states from the previous optimization step, improving generalization to longer sequences while restricting some augmentations.
- 2. Data Pipeline: The mixed strategy uses separate BPTT and TBPTT dataloaders to combine sample diversity, full augmentation, and longer-sequence generalization.The two batches are collated before being fed to the model.
- 2. Data Pipeline: The training pipeline applies horizontal flipping and zoom-in or zoom-out augmentation according to sampled probabilities and magnitudes.
3. Additional Model Details
Attention windows are dataset-specific, and the fourth stage uses global or windowed self-attention accordingly.
- The Gen1 dataset uses attention windows of 8 × 10 across all stages.
- The 1 Mpx dataset uses attention windows of 6×10 across all stages.
- The fourth stage applies global self-attention on Gen1 but four windows on 1 Mpx.
4. Additional Experiments
Additional experiments examine residual LSTM connections and cross-dataset generalization from the 1 Mpx dataset to DSEC, including both successful detections and failure cases.
- The additional experiments cover residual LSTM layers and qualitative cross-dataset generalization using a model trained on the 1 Mpx dataset.
- Adding residual connections to LSTM cells worsens results, possibly because they hamper control over current and retained temporal features.
- DSEC differs from the 1 Mpx dataset in geography and event-camera generation, providing a distinct deployment environment for qualitative assessment.
- The model successfully detects cars in mountainous HDR scenes and objects in urban environments on DSEC.
- Cross-dataset failures include mistaking a street pillar for a two-wheeler, indicating errors under distribution shift.
- The model is less confident and accurate on two-wheelers and pedestrians, likely reflecting fewer corresponding labels than car labels in the 1 Mpx dataset.
1 Mpx [5]
The supplied passages list license references for the Prophesee 1MegaPixel Automotive Detection Dataset and DSEC.
- The Prophesee 1MegaPixel Automotive Detection Dataset is associated with stated license terms and conditions.
- The Prophesee dataset license reference is provided as a Prophesee website URL.
- DSEC is associated with the Creative Commons Attribution-ShareAlike 4.0 International public license.