Source-linked AI summary
State Space Models for Event Cameras
Nikola Zubić, Mathias Gehrig, Davide Scaramuzza
TL;DR
Event-camera models trained on one temporal frequency generalize poorly when deployed at higher frequencies. This paper introduces state-space models with learnable timescales and strategies to counter aliasing; it reports smaller high-frequency performance drops than prior methods, alongside faster training.
Problem
Models trained on event representations at one frequency generalize poorly to higher-frequency deployment, while recurrent training methods also incur efficiency costs.
Source: §1 · p. 2Method
Results
SSMs show a 3.76 mAP drop at higher frequencies, compared with 21.25 mAP or more for previous methods, and train up to 33% faster than recurrent vision-transformer approaches.
Source: §1 · p. 2Takeaways & Limitations
The reported results support using SSM-based models for event-camera vision across temporal frequencies, with frequency-selective masking and H2 norm proposed to mitigate aliasing.
Source: §1 · p. 2, §5 · p. 8Whether one method consistently exceeds the other in expressiveness remains open; the described matrices are theoretical tools, not direct practical implementations.
Source: §12.2 · p. 16
Abstract
from arXiv · showhide
Today, state-of-the-art deep neural networks that process event-camera data first convert a temporal window of events into dense, grid-like input representations. As such, they exhibit poor generalizability when deployed at higher inference frequencies (i.e., smaller temporal windows) than the ones they were trained on. We address this challenge by introducing state-space models (SSMs) with learnable timescale parameters to event-based vision. This design adapts to varying frequencies without the need to retrain the network at different frequencies. Additionally, we investigate two strategies to counteract aliasing effects when deploying the model at higher frequencies. We comprehensively evaluate our approach against existing methods based on RNN and Transformer architectures across various benchmarks, including Gen1 and 1 Mpx event camera datasets. Our results demonstrate that SSM-based models train 33% faster and also exhibit minimal performance degradation when tested at higher frequencies than the training input. Traditional RNN and Transformer models exhibit performance drops of more than 20 mAP, with SSMs having a drop of 3.76 mAP, highlighting the effectiveness of SSMs in event-based vision tasks.
1. Introduction
Event-based vision needs models that generalize across inference frequencies while reducing the training cost of recurrent approaches. The paper introduces state-space layers with learnable timescales and anti-aliasing strategies, reporting faster training and smaller high-frequency performance drops.
1. Introduction
- Models trained at a specific event-representation frequency generalize poorly at higher frequencies, a concern for high-speed dynamic visual scenarios.Recurrent memory can improve performance but adds training cost, while recent recurrent vision transformers have longer training cycles.Source: §1 · p. 2
- The paper introduces structured state-space layers in its neural-network framework to address frequency generalization and training efficiency.SSMs can be trained as CNNs and used as RNNs at inference; their continuous-time formulation permits arbitrary inference step sizes through timescale adjustment.Source: §1 · p. 2
- SSM layers train up to 33% faster than recurrent vision-transformer approaches, with a 3.76 mAP drop versus 21.25 mAP or more for previous methods at higher frequencies.The reported comparisons concern object detection.Source: §1 · p. 2
- Frequency-selective masking and the H2 norm are proposed to mitigate aliasing at increased temporal frequencies.Masking encourages smooth learned convolutional kernels; the H2 norm attenuates the frequency response after a chosen frequency.Source: §1 · p. 2
- The introduction reports that the approach outperforms existing methods at higher frequencies by 20 mAP on average and increases training speed by 33%.These are the contribution summary's reported results.Source: §1 · p. 2
2. Related Work
Prior event-camera object-detection work spans dense feed-forward, recurrent, transformer, graph, and spiking approaches, while continuous-time SSM variants offer another sequence-modeling route. This work combines SSMs with attention and addresses aliasing in higher-dimensional visual data.
2.1. Object detection with Event Cameras
- Dense object-detection methods evolved from constrained-window representations to recurrent layers and transformer-RNN hybrids, but prior approaches do not adapt to variable frequencies.Swin-transformer work also had difficulty re-detecting slowly moving objects after long absences; transformer-RNN methods raised training computation demands.Source: §2.1 · p. 3
- Graph networks construct spatio-temporal event graphs, while spiking networks use threshold-triggered spikes; both approaches have distinct information-flow or optimization challenges.Graph sub-sampling risks omitting events, and non-differentiable spike mechanisms make SNNs difficult to optimize.Source: §2.1 · p. 3
2.2. Continuous-time Models
- S4, S4D (designed for easier understanding and implementation, with performance similar to S4), and S5 have not been thoroughly evaluated on complex, high-dimensional visual data with significant temporal resolution.S5 improves efficiency by avoiding frequency-domain computations and utilizing time-domain parallel scans.Source: §2.2 · p. 3
- The paper evaluates S4, S4D, and S5 with attention mechanisms on complex data and proposes two strategies to address aliasing in event-camera object detection.The authors report results on par with state of the art and frame this work as extending continuous-time models to complex sequence modeling.Source: §2.2 · p. 3
3. Method
The method embeds continuous-time SSM variants in a hierarchical event-vision backbone that reuses temporal state and supports inference at varying frequencies. It adds frequency-selective masking and an H2-based penalty to suppress aliasing during higher-frequency deployment.
3.1. Preliminaries
- Event cameras asynchronously register per-pixel log-intensity changes, and each event records its pixel coordinates, timestamp, and polarity.An event is generated when the log-intensity change exceeds threshold C.Source: §3.1 · p. 3
- The method compares S4, S4D, and S5 as continuous-time linear SSM variants, then discretizes the model at fixed step size Δ.The model uses state-transition, input, output, and direct-transmission matrices; discretization can use Euler, bilinear, or zero-order hold.Source: §3.1 · pp. 3–4
3.2. SSM-ViT block
- Reusing SSM states across four hierarchical stages retains temporal information while supporting inference at frequencies different from training.The SSM block computes output and state in parallel using S4, S4D, or S5 variants.Source: §3.2 · p. 4
- The SSM-ViT block combines convolution, local-window self-attention, dilated global attention, and an SSM layer.The convolution captures spatial features; Block-SA focuses on local relations, while Grid-SA uses a broader spatial scope.Source: §3.2 · p. 4
3.3. Low-pass bandlimiting
- Higher-frequency deployment can cause aliasing when the kernel bandwidth exceeds the Nyquist frequency.The method addresses this issue with output masking and an H2-based strategy.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 5
3.3.1 Output Masking
- Frequency-selective masking suppresses state coefficients above a frequency threshold, with α = 0.5 reported as optimal for diagonal S4D and S5 models.The effective frequency is computed per state, and the deployment rate r is halved when inference runs at twice the training frequency.Source: §3.3.1 · p. 4
3.3.2 H2 Norm
- The H2 strategy suppresses response beyond ωmin by minimizing the squared frequency-response magnitude over the selected frequency range.The integral is numerically approximated over N points between ωmin and ωmax.Source: §3.3.2 · p. 5
4. Experiments
Experiments compare SSM-based detectors with existing methods on Gen1 and 1 Mpx, then ablate model variants, SSM placement, and strategies for evaluation at different frequencies.
4.1. Setup
- Experiments use Gen1 and 1 Mpx, train on 50 ms event windows at 20 Hz, and also test generalization on DSEC, which was not in the training corpus.Both ViT-SSM-B and its smaller ViT-SSM-S variant are evaluated; the DSEC visual results are in the appendix.Source: §4 · p. 5, §4.1 · p. 5
4.2. Benchmark comparisons
- On 1 Mpx, S4D scores 46.8 mAP and S5 Base and Small score 47.8 and 46.5; on Gen1, the models are within 0.2 mAP of GET-T.The models secure second-highest task performance on 1 Mpx; Table 1 reports comparisons on both test sets at 20 Hz.Source: §4.2 · p. 6, Table 1 · p. 6
- 3.76 mAP is the average performance drop for the proposed models across training and higher testing frequencies, versus 21.25 for RVT and 24.53 for GET.The drop is averaged across Gen1 and 1 Mpx; Table 2 evaluates RVT, S5, and GET across frequencies.Source: Table 2 · p. 6, §4.2 · p. 6
4.3.1 SSMs: initializations & bandlimiting
- On Gen1 validation, S5-legS reaches 48.48 mAP at α = 0.5 and a 48.27 average, the best overall; higher α encourages smoother kernels.S4D-legS reaches 47.33 mAP at α = 0.5 and the best S4D average of 46.92; initialization and α affect results.Source: Table 3 · p. 7, §4.3.1 · p. 7
4.3.2 SSM Utilization Analysis
- On Gen1, removing SSMs produces the largest detection-performance decrease; adding them from stage four onward or at the first stage improves performance.The study resets SSM states at selected stages to isolate SSM presence while keeping parameter counts consistent.Source: Table 4 · p. 7, §4.3.2 · p. 7
4.3.3 Evaluation at different frequencies
- Across evaluated frequencies, H2 norm and bandlimiting at α = 0.5 perform comparably, with H2 norm having a slight edge at very high frequencies.The authors attribute this marginal edge to H2 norm not explicitly masking columns of the output matrix C.Source: Table 5 · p. 8, §4.3.3 · p. 8
5. Conclusion
The SSM-ViT model improves higher-frequency performance and training speed while using learnable timescales to adapt across inference frequencies. The paper also introduces two strategies to mitigate aliasing.
5. Conclusion
- The approach integrates SSMs with a Vision Transformer for event-based object detection, targeting frequency adaptability and training efficiency.The conclusion characterizes the result as a hybrid SSM-ViT model.Source: §5 · p. 8
- Learnable timescale parameters let the model adapt to different inference frequencies without multiple training cycles.Existing methods are described as requiring retraining for different frequencies.Source: §5 · p. 8
- 20 mAP higher-frequency advantage and 33% increase in training speed are reported for SSM-ViT over existing methods.The paper presents frequency-selective masking and H2 norm adjustments as strategies to counteract aliasing.Source: §5 · p. 8
7. Explanation of S4(D) and S5 models
S4, S4D, and S5 use different state-space structures and computation strategies. S5 replaces S4's bank of SISO models with a MIMO model and parallel scans, while supporting time-varying systems and irregular observations.
7. Explanation of S4(D) and S5 models
- S4 reparameterizes structured state matrices to maintain a hidden state encoding long input history, while S4D uses a fully diagonal state matrix with comparable performance.S4D is described as having a simpler implementation than S4.Source: §7 · p. 9
- S5 replaces S4's multiple SISO SSMs with one MIMO SSM and uses parallel scans, avoiding frequency-domain and convolutional computation.This makes S5 a time-domain recurrent model.Source: §7 · p. 9
- S4 uses a Cauchy-kernel computation for diagonal matrices, while S5 uses a diagonal state matrix for parallel-scan computation and inherits HiPPO initialization schemes.The S5 initialization uses a diagonal approximation to the HiPPO matrix for comparable performance.Source: §7 · p. 9
- S5 matches S4's computational complexity for online generation and offline recurrence, and handles time-varying SSMs and irregularly sampled observations.These are challenges for S4's convolutional implementation.Source: §7 · p. 9
8. Initialization of continuous-time matrices
The section describes continuous-time state-matrix initialization and the input, output, and timescale initializations used in the S5 layer.
8.1. State Matrix Initialization
- HiPPO-LegS initialization supports discretizing continuous-time matrices at different time steps for higher inference frequencies.The matrix is described as decomposing input under an infinitely long, exponentially diminishing measure.Source: §8.1 · p. 9
- The default S5 state matrix is initialized from the normal HiPPO-LegS matrix and decomposed to obtain the initial Λ.The block-diagonal initialization uses HiPPO-N blocks; J = 1 is the default single-block setting.Source: §8.1 · p. 11
8.2. Input, Output, and Feed-through Matrices Initialization
- The input and output matrices ˜B and ˜C are initialized using eigenvectors from diagonalizing the initial state matrix, while each D element is sampled independently from a standard normal distribution.The procedure samples B and C, then sets ˜B = V−1B and ˜C = CV.Source: §8.2 · p. 11
8.3. Timescale Initialization
- Each log ∆ element is initialized uniformly over [log δmin, log δmax), with defaults δmin = 0.001 and δmax = 0.1.The section aligns this initialization with S4 practices.Source: §8.3 · p. 11
9. Background on Parallel Scans for Linear Recurrences
A parallel scan computes cumulative outputs from a sequence under an associative binary operator. For linear recurrences, tuple scans can reduce sequential computation to O(log L) steps.
9. Background on Parallel Scans for Linear Recurrences
- A scan applies a binary associative operator to a sequence and returns its cumulative sequence of prefixes.Associativity permits flexible computation order.Source: §9 · p. 11
- For linear recurrences, the scan initializes tuples as ck = (A, Buk) and combines tuples using an associative operator.The tuple operator uses matrix-matrix and matrix-vector multiplication.Source: §9 · p. 11
- O(log L) sequential steps can compute the linear recurrence with adequate processing power.The section attributes this efficiency to the parallel scan approach.Source: §9 · p. 11
10. DSEC dataset evaluation
The section evaluates a model trained on 1 Mpx data on the DSEC dataset and presents example detections.
10. DSEC dataset evaluation
- The authors test generalizability by running inference on DSEC with a model trained on 1 Mpx.Figure 5 shows example detections, with scene names given in the subcaptions.Source: §10 · p. 11, Figure 5 · p. 11
11. PyTorch implementation of Parallel Scan and S5 model
This section provides PyTorch implementations of the parallel scan operation and S5 layers.
11.1. Parallel Scan Operation
- The parallel scan implementation recursively scans partially reduced tensors and interleaves results to produce the scan.It includes handling for reverse scans and preserves the first element of the original input.Source: §11.1 · p. 12
11.2. S5 model
- The S5 implementation discretizes a continuous-time linear SSM using the bilinear transform before applying it to an input sequence.The discretization uses the state matrix, input matrix, and step sizes to produce discretized state and input matrices.Source: §11.2 · p. 13
- The S5 layer applies the discretized SSM to a batch of input sequences and returns outputs and hidden states.Its parallel scan combines recurrence elements using a binary operator that assumes a diagonal state matrix.Source: §11.2 · p. 13
12. Exploring the S4 and S5 Architectural Link
The section relates S5 dynamics to collections of S4 systems under stated assumptions, then considers relaxed assumptions and timescale parameterization.
12. Exploring the S4 and S5 Architectural Link
- The architectural analysis links S5 states to sums of S4 states through recurrence linearity.The section frames this relationship as a basis for understanding S4 and S5 architectures.Source: §12 · p. 15, §12.2 · p. 15
12.1. Underlying Assumptions
- The equivalence analysis assumes H-dimensional input and output maps, a shared S4 state matrix and timescale, and matching S5 latent size.It also assumes the S5 input matrix concatenates the S4 input vectors.Source: §12.1 · p. 15
12.2. Distinct Output Projections from Equivalent Dynamics
- Under shared state matrices and timescales, S5 outputs are linear combinations of H S4 latent states, but its output projection differs from S4’s.The comparison is conditional and does not establish that either method is consistently more expressive.Source: §12.2 · pp. 15–16
12.3. Relaxing the Assumptions
- With a block-diagonal S5 state matrix, its subsystems can correspond to distinct S4 systems, motivating multiple HiPPO-N initializations across blocks.The JN-dimensional S5 output is a linear combination of latent states from J different S4 systems.Source: §12.3 · p. 16
12.4. Timescale Parameterization
- Empirical evidence favors P distinct timescale parameters, one per state, while the system can also learn a single timescale by equalizing them.Timescale initialization matters, and a single initial parameter may be suboptimal.Source: §12.4 · p. 16
13. SSM-only and SSM-ConvNext models
The section compares SSM-only and CNN-based alternatives with the paper’s S5-ViT-B model. The reported SSM-only scores are lower, while ViT outperforms the CNN structure.
13. SSM-only and SSM-ConvNext models
- 46.10 mAP on Gen1 and 45.74 mAP on 1 Mpx are reported for SSM-only models, below the original S5-ViT-B performance.Table 6 is identified as comparing mAP scores across Gen1 and 1 Mpx base models.Source: §13 · p. 17, Table 6 · p. 17
- The ViT structure achieves better performance than the CNN-based ConvNext structure.The comparison replaces the attention block with a ConvNext block.Source: §13 · p. 17