PaperlayerView exact revision on arXiv

Source-linked AI summary

GET: Group Event Transformer for Event-Based Vision

Yansong Peng, Yueyi Zhang, Zhiwei Xiong, Xiaoyan Sun, Feng Wu

arXiv:2310.02642v1cs.CVcs.AI

TL;DR

Image-based event backbones can underuse event time and polarity. GET groups events by timestamp and polarity and uses transformer modules to communicate and integrate features across spatial and temporal-polarity domains. It reports state-of-the-art results across four classification and two detection datasets.

  • Problem

    Image-based event backbones mainly extract spatial information and do not fully use the temporal and polarity information carried by events.

    Source: §1 · p. 1

  • Method

    GET groups asynchronous events by timestamp and polarity, then uses EDSA and GTA modules to communicate and integrate features across spatial and temporal-polarity domains.

    Source: §1 · p. 2

  • Results

    GET reports state-of-the-art performance across four event-based classification datasets and two object-detection datasets, including 84.8% on Cifar10-DVS and 47.9% and 48.4% mAP on Gen1 and 1Mpx.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported results support GET as an event-based backbone for classification and object detection across the evaluated datasets.

    Source: §5 · p. 8

Abstract

from arXiv · show

Event cameras are a type of novel neuromorphic sen-sor that has been gaining increasing attention. Existing event-based backbones mainly rely on image-based designs to extract spatial information within the image transformed from events, overlooking important event properties like time and polarity. To address this issue, we propose a novel Group-based vision Transformer backbone for Event-based vision, called Group Event Transformer (GET), which de-couples temporal-polarity information from spatial infor-mation throughout the feature extraction process. Specifi-cally, we first propose a new event representation for GET, named Group Token, which groups asynchronous events based on their timestamps and polarities. Then, GET ap-plies the Event Dual Self-Attention block, and Group Token Aggregation module to facilitate effective feature commu-nication and integration in both the spatial and temporal-polarity domains. After that, GET can be integrated with different downstream tasks by connecting it with vari-ous heads. We evaluate our method on four event-based classification datasets (Cifar10-DVS, N-MNIST, N-CARS, and DVS128Gesture) and two event-based object detection datasets (1Mpx and Gen1), and the results demonstrate that GET outperforms other state-of-the-art methods. The code is available at https://github.com/Peterande/GET-Group-Event-Transformer.

1. Introduction

Event-based backbones often rely on image-based feature extraction, which does not fully use events’ temporal and polarity information. GET introduces timestamp- and polarity-based Group Tokens and modules for communicating and integrating features across spatial and temporal-polarity domains, and reports state-of-the-art results across classification and detection datasets.

1. Introduction

  • Event cameras capture per-pixel illumination changes asynchronously and offer high temporal resolution, high dynamic range, and low power consumption.The passage reports >10K fps, >120 dB, and <10 mW, respectively.Source: §1 · p. 1
  • Image-based event backbones mainly extract spatial information and do not fully use event time and polarity, while SNN- and GNN-based alternatives have stated hardware or performance trade-offs.Events are asynchronous arrays containing each illumination change’s location, polarity, and time.Source: §1 · p. 1
  • GET reports state-of-the-art performance on four event-based classification datasets and two object-detection datasets, including 84.8% on Cifar10-DVS and 47.9% and 48.4% mAP on Gen1 and 1Mpx.The other reported classification results are 99.7% on N-MNIST, 96.7% on N-CARS, and 97.9% on DVS128Gesture.Source: §1 · p. 2
  • GET combines Group Tokens, which group asynchronous events by timestamp and polarity, with EDSA and GTA modules for feature communication and integration across spatial and temporal-polarity domains.GTA uses overlapping group convolution to integrate and decouple information in both domains.Source: §1 · p. 2

2. Related Work

Prior event representations and Transformer designs are described as limited by information loss, redundancy, latency, computational cost, or underuse of event properties. Related work also traces self-attention from NLP to vision and discusses local and hierarchical Transformer variants.

2. Related Work

  • Image-like event representations can discard event information, learning-based representations can add redundancy and latency, and graph-based representations have not yet yielded satisfactory performance.Examples include voxel grids, event histograms, and time surfaces.Source: §2 · p. 2
  • Event-based Transformers operate on CNN feature maps or event representations, but their designs often do not fully use event time and polarity.Some methods use global attention on CNN features, which the passage describes as computationally heavy.Source: §2 · p. 2
  • Vision Transformer applied self-attention to images, while later variants use local attention or hierarchical architectures to improve performance and data or computation efficiency.The passage also notes that dual attention has high global-attention costs and that direct spatial-channel fusion can cause interference.Source: §2 · p. 2

3. Method

GET is a three-stage Transformer that embeds asynchronous events as Group Tokens, then uses dual attention and overlapping group convolution to process spatial and temporal-polarity information. Its modules progressively communicate and integrate these features before passing outputs to downstream task heads.

3.1. Main Architecture

  • GET uses three stages: GTE embeds raw events, Stage 1 applies two EDSA blocks, and Stages 2–3 apply GTA followed by two and eight EDSA blocks, respectively.The output feature maps feed heads for tasks such as classification and object detection.Source: §3.1 · pp. 2–3

3.2. Group Token Embedding

  • Group Token Embedding discretizes event times, encodes patch rank and position, combines these with polarity, and embeds the resulting representation into Group Tokens.The representation is further processed by a 3×3 group convolution following an MLP; groups correspond to combinations of time interval and polarity.Source: §3.2 · p. 3

3.3. Event Dual Self-Attention Block

  • EDSA partitions tokens into windows, applies spatial self-attention, then group self-attention to capture temporal-polarity context, while dual residual connections retain features from both domains.The layer outputs one feature map refined by SSA alone and another refined by both SSA and GSA.Source: §3.3 · pp. 3–4

3.4. Group Token Aggregation

  • GTA uses overlapping group convolution and max pooling to integrate and decouple spatial and temporal-polarity information across stages.The convolution doubles channel number and halves group number, giving each group a 3× larger receptive field in both domains; ablations support reliable integration.Source: §3.4 · p. 4

4. Experiments

GET is evaluated on four event-classification datasets and two object-detection datasets, with comparisons to state-of-the-art methods and ablations of its modules. The reported results show strong classification and detection performance, while ablations examine accuracy, efficiency, and group-number choices.

4.1. Experimental Setup

  • The evaluation covers four classification datasets and two large-scale detection datasets, with dataset-specific splits and model configurations.Detection uses 50 ms event intervals; classification uses a 3-stage GET, while detection integrates a 4-stage GET into RVT.Source: §4.1 · pp. 4–5

4.2. Experimental Results

  • GET reports 84.8% on Cifar10-DVS, 99.7% on N-MNIST, 96.7% on N-CARS, and 97.9% on DVS128Gesture, with stated gains over comparison methods.The reported comparisons include 84.8% vs. 74.8% for Cifar10-DVS against PLIF, and 97.9% vs. 97.6% for DVS128Gesture against PLIF using single-pass inference.Source: Table 1 · p. 5, Table 2 · p. 5, §4.2 · p. 6
  • GET achieves 47.9% mAP on Gen1 using a memory mechanism and 48.4% on 1Mpx with ConvLSTM, and is the fastest end-to-end method on both datasets.Source: §4.2 · p. 6

4.3. Ablation Studies

  • Group Token improves top-1 accuracy by 2.7% and mAP by 1.6%, while its implementation reduces runtime by 51%.Converting 108 events into Group Token takes 0.052s; the representation also reports 84.8% versus 80.4% for other representations.Source: §4.3 · p. 7, Table 6 · p. 7
  • EDSA improves top-1 accuracy by 1.6% and mAP by 0.9% over SSA, while dual residual connections yield 84.8% versus 81.9%.GTA reduces parameters by 0.2 M and running time by 1.3 ms, but can slightly reduce performance without EDSA.Source: §4.3 · pp. 7–8, Table 7 · p. 7
  • The best group number differs by dataset: G=12 gives 84.8% on Cifar10-DVS, while G=24 gives 97.9% on DVS128Gesture.The text attributes declining Cifar10-DVS performance at larger G to sparse events per group, and lower performance at small G on DVS128Gesture to action overlap within a group.Source: Table 9 · p. 8, §4.3 · p. 8

4.4. Visualizations

  • GET captures counterclockwise waving in DVS128Gesture, while the two SSA-based classifiers categorize the sample as clockwise waving.On 1Mpx, GET detects more objects than Swin-T v2, especially against intertwined backgrounds, with more accurate bounding boxes.Source: §4.4 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5. Conclusion

GET is an event-based vision backbone designed to use temporal and polarity information while retaining spatial modeling. Its components support communication and integration across these domains, and it achieves superior performance over state-of-the-art methods on four classification and two object-detection datasets.

5. Conclusion

  • GET achieves superior performance over state-of-the-art methods on four event-based classification and two object-detection datasets.Source: §5 · p. 8
  • GET uses temporal and polarity information while preserving the spatial modeling capability of traditional Transformer-based backbones.Source: §5 · p. 8
  • Group Token, EDSA, and GTA enable feature communication and integration in both spatial and temporal-polarity domains.Source: §5 · p. 8

Related papers

Loading 2310.02642v1…