Source-linked AI summary
VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows
Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, Feng Wu
TL;DR
Reliable visible-event tracking requires combining visible cameras' texture information with event cameras' strengths in challenging conditions, but realistic large-scale benchmarks are scarce. The paper introduces the VisEvent benchmark, dual-modality baselines, and a cross-modality transformer; experiments validate the model's effectiveness across real and simulated datasets.
Problem
Realistic large-scale visible-event datasets and baseline methods are lacking, while event cameras alone lack fine-grained texture information needed for tracking.
Method
The paper constructs VisEvent, converts event flows into event images, extends visible trackers to dual-modality baselines, and introduces a cross-modality transformer for feature fusion.
Results
0.627|0.426 is achieved with combined modalities versus 0.460|0.280 using event sensors alone in the reported ablation.
Takeaways & Limitations
VisEvent provides a real-world benchmark and baseline platform for studying visible-event object tracking in challenging scenarios.
Abstract
from arXiv · showhide
Different from visible cameras which record intensity images frame by frame, the biologically inspired event camera produces a stream of asynchronous and sparse events with much lower latency. In practice, visible cameras can better perceive texture details and slow motion, while event cameras can be free from motion blurs and have a larger dynamic range which enables them to work well under fast motion and low illumination. Therefore, the two sensors can cooperate with each other to achieve more reliable object tracking. In this work, we propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios, and it is divided into a training and a testing subset, each of which contains 500 and 320 videos, respectively. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods by extending current single-modality trackers into dual-modality versions. More importantly, we further build a simple but effective tracking algorithm by proposing a cross-modality transformer, to achieve more effective feature fusion between visible and event data. Extensive experiments on the proposed VisEvent dataset, FE108, COESOT, and two simulated datasets (i.e., OTB-DVS and VOT-DVS), validated the effectiveness of our model. The dataset and source code have been released on: \url{https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark}.
I. INTRODUCTION
Visible and event cameras offer complementary sensing properties for object tracking, but realistic large-scale benchmarks and baseline methods remain limited. The paper addresses this gap with VisEvent, dual-modality trackers, and a cross-modality transformer.
- Visible cameras capture color and texture well but struggle under low illumination, fast motion, and background clutter.
- Event cameras provide asynchronous, low-latency sensing suited to motion and poor illumination, but lack slow-motion and fine-grained texture information.
- 820 Visible-Event video pairs form the VisEvent benchmark, split into 500 training and 320 testing pairs.
- The paper extends visible trackers into dual-modality versions using early, middle, and late fusion strategies.
- A cross-modality transformer baseline is proposed to improve feature fusion between visible and event streams.
- More than 35 dual-modality trackers support comparisons across tracking pipelines and fusion strategies.
II. RELATED WORK
Prior work spans visible-camera, event-camera, and combined visible-event tracking. However, existing combined approaches were evaluated mainly on simple videos or simulations, leaving performance on challenging large-scale real data uncertain.
- Visible Camera based Tracking: Deep visible-camera trackers include convolutional, Siamese, attention-based, and Transformer-based approaches.
- Visible Camera based Tracking: Visible-camera trackers remain challenged by low illumination, fast motion, and low-resolution imagery.
- Event-based Tracking: Event-camera tracking research explores redetection, high-speed feature tracking, spiking neural networks, offline-online tracking, graph networks, and cross-modal alignment.
- Tracking by Combining Visible and Event Cameras: Existing combined visible-event studies used several simple real videos or simulation data and lacked suitable baseline comparisons.
III. METHODOLOGY
The method represents asynchronous event streams as synchronous event images and combines them with RGB frames for tracking. Its central design is interactive dual-modal feature fusion using cross-modality transformers.
- A. Motivation and Overview: The tracker jointly exploits RGB frames and event streams through a cross-modality transformer feature-fusion module.
- A. Motivation and Overview: The tracker uses three shared convolutional layers to extract feature representations before dual-modal fusion.
- B. Input Representation: Visible frames and event flows provide synchronous and asynchronous inputs whose representation and adaptive fusion are central tracking challenges.
- B. Input Representation: Each event is represented as {x, y, t, p}, encoding spatial coordinates, timestamp, and brightness-change polarity.
- B. Input Representation: Asynchronous event flows are stacked over a fixed time interval to form synchronous event images for CNN-based processing.
C. Cross-Modality Transformer for Fusion
The Cross-Modality Transformer enables RGB and event features to exchange information before self-attention refines modality-specific global representations. The resulting attended features are classified to select the tracking proposal.
- RGB and event features are fused into a base vector and used to guide cross-attention in both directions.
- Cross-attention aligns the modalities by exchanging information between their feature representations.
- Self-attention is applied separately to the cross-attended event and visible features to strengthen internal global connections.
- Attention weights are computed from processed feature branches with Softmax, then used to form attended event features.
- The framework extracts instance-level RoI features, performs dual-modal attention, and feeds attended features into fully connected layers for proposal classification.
D. Training and Tracking Phase
The tracker is trained with binary classification and instance embedding losses, then adapted online to select the highest-scoring proposal in each subsequent frame.
- Training: The network combines binary cross-entropy and instance embedding losses, L = Lce + Lie, during training.Domain-specific layers learn sequence-specific shared features during training.
- Training: Training introduces domain-specific layers for each video sequence to learn shared features used only during training.
- Tracking: At test time, an online classifier is trained from samples extracted from the first frame.
- Tracking: For each later frame, Gaussian sampling generates proposals around the previous result, and the highest-response proposal becomes the current tracking result.Hard sample mining and online updates further improve tracking.
E. Implementation Details
The implementation trains dual-modality baselines on VisEvent and provides released tracker extensions, while the benchmark emphasizes scale, annotation quality, and modality balance.
- Implementation Details: The extended MDNet/RT-MDNet baseline is trained on VisEvent for 50 epochs using a learning rate of 0.0001 and batch size 8.Training takes about 3 hours; other trackers use their default settings or pretrained models.
- Implementation Details: All extended trackers have been released to help researchers reproduce the experiments.
- Protocols: The benchmark collects 820 video pairs averaging 450 frames each to support data-hungry deep trackers.It also uses dense annotations that are independently checked, according to the construction protocols.
- Protocols: The benchmark includes comprehensive dual-modality baselines by extending visible trackers with varied fusion strategies and adding a cross-modality transformer tracker.
B. Data Collection and Annotation
VisEvent is collected with a synchronized DVS setup that outputs aligned RGB and event streams, then converted, selected, and densely annotated into a challenging benchmark.
- Data Collection: A single DVS sensor simultaneously outputs visible frames and event flows, with hardware alignment eliminating external spatial and temporal registration.Both modalities have resolution 346 × 260.
- Data Collection: The raw ∗.aedat4 output is converted into RGB and event ∗.bmp frames before clips with consistent target objects are selected.
- Annotation: Professional annotators label every frame, and two authors check and further adjust annotations for accuracy.
- Attribute Definition: VisEvent defines 17 attributes covering motion, occlusion, illumination, background activity, and the NOM condition for static targets.NOM denotes NO Motion; other listed challenges include fast motion, motion blur, and low illumination.
- Statistical Analysis: 820 video sequence pairs contain 371,128 RGB frames, with lengths ranging from 18 to 6,246 frames and an average of 450 frames.The distribution supports both short-term and long-term tracking evaluation.
- Statistical Analysis: The dataset includes camera motion, background clutter, scale variation, occlusion, and distractor motion, where tracking remains far from solved.
E. Discussion
Visible-Event tracking complements the strengths and weaknesses of RGB-Thermal and RGB-Depth tracking, while realistic large-scale visible-event benchmarks and baselines remain necessary.
- Discussion: RGB-Thermal and RGB-Depth tracking fuse complementary modalities, but each sensor has practical limitations such as thermal cross-over, motion blur, or restricted operating conditions.Thermal sensing is illumination-independent but can fail when target and background temperatures are similar; depth sensing works in 3D but may only work well locally.
- Discussion: Event-camera tracking still requires better event-flow representations, efficient neural networks, and realistic public datasets with baseline methods.The paper identifies these as unresolved problems that limit development of the research direction.
A. Dataset and Evaluation Metric
The benchmark evaluates visible-event trackers across real and simulated datasets using standard precision and success metrics, with diverse tracker families and fusion strategies as baselines.
- Dataset and Evaluation Metric: VisEvent training uses 500 sequences, while testing covers 320 videos; evaluation also includes FE108, COESOT, OTB-DVS, and VOT-DVS.FE108 has 76 training and 32 testing videos; COESOT has 827 training and 527 testing videos.
- Dataset and Evaluation Metric: Precision Plot measures frames whose center-location error is below 20 pixels, whereas Success Plot measures frames whose bounding-box IoU exceeds a threshold.These metrics assess localization accuracy and overlap quality, respectively.
- Baseline Construction: The benchmark extends representative classification, correlation-filter, and Siamese trackers into visible-event versions for comparison.The listed frameworks include MDNet, KCF, SiamFC, SiamRPN, ATOM, and related trackers.
- Baseline Construction: Early fusion combines inputs before tracking, middle fusion combines modality features, and late fusion combines output responses.Middle-fusion variants include concatenation, addition, 1 × 1 convolution, channel attention, spatial attention, and cross attention.
- Benchmark Results: On VisEvent, deep classification trackers generally outperform correlation-filter trackers, while Siamese trackers are not consistently dominant.MDNet, VITAL, and RT-MDNet exceed CN and KCF in the reported PR|SR comparisons.
D. Ablation Study
Ablation studies show that visible and event modalities provide complementary tracking information, while the cross-modality transformer yields smaller additional gains and challenging scenarios remain central to evaluation.
- Influence of Input Modalities: 0.578|0.421 is obtained by early-fusing PrDIMP18 modalities, exceeding its visible-only 0.554|0.407 and event-only 0.404|0.256 results.The fused setting also performs better on motion-blur and low-illumination attributes.
- Influence of Cross-Modality Transformer: CMT improves MDNet from 0.627|0.426 to 0.632|0.430 and RT-MDNet from 0.560|0.352 to 0.564|0.359.These component-analysis results support the effectiveness of feature-level cross-modality fusion.
- Challenging Scenarios: The proposed tracker reaches top-five or best performance under reported low-illumination and fast-motion attributes.These attributes are included among the challenging VisEvent scenarios.
E. Results of Various Fusion Strategies
The study compares multiple feature-fusion strategies for RGB-Event tracking, including concatenation, addition, convolution, attention, and cross-attention methods.
- Results of Various Fusion Strategies: Feature fusion experiments evaluate concatenation, addition, 1 × 1 convolution, channel attention, spatial attention, and cross-attention modules.The comparison is conducted using MDNet for RGB-Event visual tracking.
- Results of Various Fusion Strategies: Simple concatenation of dual features achieves the best tracking result among the compared fusion methods.The passage reports this outcome without specifying the associated numeric score.
- Results of Various Fusion Strategies: Visualizations compare the proposed tracker with other state-of-the-art trackers on VisEvent.The figure provides qualitative tracking-result comparisons.
F. Efficiency Analysis
CMT improves dual-modality tracking performance while retaining efficiency determined by its baseline tracker. Event images help online trackers handle motion blur, fast motion, and low illumination, but fixed-window representations remain a failure source.
- Efficiency: About 14 FPS is achieved when CMT is integrated into dual-modality RT-MDNet, while attaining the best performance on VisEvent.CMT efficiency mainly depends on the selected baseline tracker.
- Qualitative comparison: Event images help MDNet, VITAL, and CMT handle motion blur, fast motion, and low illumination that challenge PrDiMP50, SuperDiMP, and SiamRPN++.The comparison is illustrated through tracking-result visualizations in Fig. 9.
- Failure analysis: Fixed-time-window event images can cause shape variation and overlapping event streams, producing failures on fast-moving baseball and moving-star sequences.The authors identify alternative event representations such as points or voxels as future directions.