Source-linked AI summary
Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline
Xiao Wang, Shiao Wang, Chuanming Tang, Lin Zhu, Bo Jiang, Yonghong Tian, Jin Tang
TL;DR
Event tracking faces sparse event signals, added inference cost when using RGB-event data, and low-resolution existing datasets. This paper distills multi-modal or multi-view teacher knowledge into an event-only tracker and introduces EventVOT. Experiments across four benchmarks are reported to validate the tracker.
Problem
Event tracking must contend with sparse spatial signals, extra inference cost from RGB-event inputs, and existing datasets limited to 346 × 260 resolution.
Source: §1 · p. 2Method
A Transformer teacher trained on multi-modal or multi-view inputs guides an event-only student through similarity-, feature-, and response-map-based hierarchical distillation.
Source: §1 · p. 2, §3.4 · p. 5Results
Experiments on FE240hz, VisEvent, COESOT, and EventVOT are reported to validate the proposed tracker.
Source: §1 · p. 3Takeaways & Limitations
The student uses only event data at test time and is described as supporting accurate, low-latency, high-speed tracking.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Tracking using bio-inspired event cameras has drawn more and more attention in recent years. Existing works either utilize aligned RGB and event data for accurate tracking or directly learn an event-based tracker. The first category needs more cost for inference and the second one may be easily influenced by noisy events or sparse spatial resolution. In this paper, we propose a novel hierarchical knowledge distillation framework that can fully utilize multi-modal / multi-view information during training to facilitate knowledge transfer, enabling us to achieve high-speed and low-latency visual tracking during testing by using only event signals. Specifically, a teacher Transformer-based multi-modal tracking framework is first trained by feeding the RGB frame and event stream simultaneously. Then, we design a new hierarchical knowledge distillation strategy which includes pairwise similarity, feature representation, and response maps-based knowledge distillation to guide the learning of the student Transformer network. Moreover, since existing event-based tracking datasets are all low-resolution ($346 \times 260$), we propose the first large-scale high-resolution ($1280 \times 720$) dataset named EventVOT. It contains 1141 videos and covers a wide range of categories such as pedestrians, vehicles, UAVs, ping pongs, etc. Extensive experiments on both low-resolution (FE240hz, VisEvent, COESOT), and our newly proposed high-resolution EventVOT dataset fully validated the effectiveness of our proposed method. The dataset, evaluation toolkit, and source code are available on \url{https://github.com/Event-AHU/EventVOT_Benchmark}
1. Introduction
The introduction identifies limitations in event-based tracking and existing low-resolution datasets, then presents hierarchical knowledge distillation and EventVOT as responses. It reports validation across four event-tracking benchmarks.
1. Introduction
- Visual object tracking localizes a target initialized in the first frame, but challenging factors and unevenly distributed difficult frames hinder overall performance.The cited challenges include fast motion, illumination, background distractors, and out-of-view targets.Source: §1 · p. 1
- Event cameras capture motion asynchronously and offer dense temporal resolution, high dynamic range, low energy consumption, and low latency.The introduction notes their use in surveillance, robotics, medical imaging, and sports analysis.Source: §1 · p. 1
- Existing event trackers face sparse spatial signals in slow scenes, while RGB-Event inference adds cost and prior event datasets use 346 × 260 resolution.The introduction frames training-time knowledge transfer with event-only testing as an open question.Source: §1 · p. 2
- EventVOT provides 1,141 videos at 1280 × 720 across diverse targets, including pedestrians, vehicles, UAVs, and ping pongs.The dataset also includes baseline tracker results for future comparisons.Source: §1 · p. 2
- HDETrack transfers multi-modal RGB-Event or multi-view Event Image-Voxel knowledge to an event-only tracker through hierarchical distillation.The teacher uses RGB and event data; the student uses event data, with similarity, feature, and response-map distillation.Source: §1 · p. 2
- Experiments on FE240hz, VisEvent, COESOT, and EventVOT are reported to validate the proposed tracker.The introduction describes evaluations on both existing and newly proposed benchmarks.Source: §1 · pp. 2–3
2. Related Work
Related work covers RGB-based visual tracking, event-camera tracking, and knowledge distillation. The paper distinguishes its approach by transferring multi-modal or multi-view knowledge to event trackers for event-only inference.
2. Related Work
- RGB-based tracking developed from convolutional and Siamese approaches, including SiamFC [1], toward numerous later Siamese trackers such as SiamRPN++ [20] and DiMP [2].The related work describes deep learning as having boosted mainstream RGB tracking.Source: §2 · p. 3
- Transformer trackers span temporal-context models such as TrDiMP [31], attention-based TransT, and one-stream OSTrack [35].Other cited designs use Transformer modules for prediction [23], feature extraction and propagation [14], or joint feature extraction and interaction [4].Source: §2 · p. 3
- Event-based tracking includes RGB-event fusion methods such as AFNet [37] and methods using Transformers with spiking networks, such as STNet.Other cited approaches convert events to frames [5], select key events for graph-based tracking, or fuse modalities through a unified backbone [28].Source: §2 · p. 3
- The paper contrasts event-only testing after multi-modal or multi-view distillation with prior event-tracking approaches.It describes the aim as efficient, low-latency tracking using event data.Source: §2 · pp. 3–4
- Prior tracking distillation includes feature supervision, teacher-student Siamese trackers, and modality-common and modality-specific feature transfer.The paper positions its approach as propagating knowledge from multi-modality or multi-view data to event-tracking networks.Source: §2 · pp. 3–4
3. Methodology
The methodology trains a multimodal or multiview teacher to guide an event-only student tracker through hierarchical knowledge distillation. RGB frames and event streams are represented as template and search inputs, processed by Transformer networks, and used to train the student with tracking and distillation losses.
3.2. Input Representation
- RGB frames supply template and search patches, while event streams are converted into aligned event images or spatial-temporal voxels with corresponding template and search regions.The event images align with RGB exposure time; voxels split the stream across spatial and temporal dimensions.Source: §3.2 · p. 4
3.1. Overview
- The teacher receives RGB and event data (or different event representations), while the student tracks using event data only.The student learns from tracking losses and teacher-based distillation for efficient, low-latency tracking.Source: §3.3 · p. 4, §3.1 · p. 4
3.3. Network Architecture
- The teacher fuses projected template and search tokens from both modalities or event views, while a lightweight student Transformer takes only event input.Search-region tokens feed the tracking head for localization; the student uses hierarchical distillation to address sparse events and cluttered backgrounds.Source: §3.3 · p. 4, Figure 2 · p. 5
3.4. Hierarchical Knowledge Distillation
- Tracking training uses focal, L1, and GIoU losses alongside the knowledge-distillation objectives.The methodology also identifies event-camera dataset comparisons, but the supplied table passage gives no dataset values.Source: §3.4 · pp. 4–5, Table 1 · p. 6
- Similarity-matrix distillation transfers teacher Transformer relations, while feature distillation matches teacher and student token representations.The similarity matrices come from self-attention layers, and feature distillation uses the networks’ token representations.Source: §3.4 · p. 5
- Response-based distillation uses a Gaussian target heatmap and a Gaussian Weighted Focal loss to guide the student’s localization response.The heatmap uses an object-size-adaptive standard deviation; response maps are normalized by temperature before focal-loss distillation.Source: §3.4 · pp. 5–6
4. EventVOT Dataset
EventVOT is a large-scale, high-definition event-tracking dataset designed around diverse targets, environments, and event-camera conditions. Its analysis describes challenge and category distributions, while its benchmark split and retrained trackers support future comparisons.
4. EventVOT Dataset
- The paper proposes a large-scale, high-definition event-based visual-tracking dataset; Fig. 4 presents representative samples, with the first column showing a 3D event point stream and columns 2–4 showing sampled event images.Source: §4 · p. 6, Figure 4 · p. 7
4.1. Criteria for Collection and Annotation
- EventVOT was collected with a 1280 × 720 EVK4-HD camera, spans varied targets, environments, speeds, and lighting, and uses professionally checked annotations over 499 event images per stream.Collection includes day and night settings and 14 challenging factors.Source: §4.1 · p. 6
4.2. Statistical Analysis
- The dataset defines 14 challenging factors across 19 target classes; BC, BOM, and SV are its three most common challenges, and category counts are roughly 50–60 except 96 UAVs.Figure 3 also visualizes annotated bounding-box center points.Source: §4.2 · p. 6, Figure 3 · p. 7
4.3. Benchmarked Trackers
- The benchmark provides retrained Siamese/discriminative and Transformer trackers using an 841/18/282-video training/validation/testing split.The retrained trackers are intended to support future performance comparisons.Source: §4.3 · p. 6
5. Experiment
Across four event-tracking benchmarks, the proposed method improves on its OSTrack baseline, including on EventVOT. The experiments also examine distillation components and settings, and visualize tracker outputs and response maps.
5.1. Dataset and Evaluation Metric
- The evaluation covers FE240hz, VisEvent, and COESOT alongside EventVOT, and reports Precision Rate, Normalized Precision Rate, Success Rate, and tracker speed in FPS.The trackers are retrained on the relevant training subsets for the reported comparisons.Source: §5.1 · pp. 7–8
5.2. Implementation Details
- Training first pre-trains a multimodal teacher for 50 epochs, then trains the student with hierarchical distillation using AdamW.Both stages use learning rate 0.0001, weight decay 0.0001, and batch size 32.Source: §5.2 · p. 8
5.3. Comparison on Public Benchmarks
- On EventVOT, the method reaches 57.0 SR, 57.3 PR, and 66.5 NPR, improving over OSTrack’s 55.4 SR, 56.4 PR, and 65.2 NPR; results also improve on FE240hz, VisEvent, and COESOT.The EventVOT comparison is reported in Table 5; prose reports baseline improvements on the other three datasets.Source: §5.3 · pp. 8–9, Table 5 · p. 9
5.4. Ablation Study
- Similarity-, feature-, and response-based distillation each improve results in both COESOT and EventVOT settings; feature distillation works better on COESOT than EventVOT.The reported best tradeoff weights are [1, 0.7, 0.7], yielding 0.570 SR, 0.573 PR, and 0.665 NPR.Source: §5.4 · pp. 9–10
5.5. Visualization
- Visualizations show trackers performing well in simple scenarios but leaving room for improvement, while HDETrack response maps highlight target regions.Response maps are shown for Car, Goose, Hat, Mask, and Mouse videos.Source: §5.5 · p. 11, Figure 7 · p. 11, Figure 9 · p. 11
6. Conclusion
The paper proposes hierarchical knowledge distillation for event-based tracking, using multimodal or multiview teacher inputs and event data for the student. It also introduces EventVOT as a large-scale, high-resolution benchmark and reports that experiments across multiple datasets validate the distillation strategy.
6. Conclusion
- The framework trains an event-tracking student from a teacher that accepts multimodal or multiview data.The distillation considers similarity-based, feature-based, and response-based knowledge.Source: §6 · pp. 11–12
- EventVOT is a large-scale, high-resolution event-tracking dataset with 1,141 sequences across 19 target categories.The paper also re-trains and evaluates more than 10 recent strong trackers on it as a benchmark for future comparisons.Source: §6 · p. 12
- Experiments on multiple datasets validate the effectiveness of the proposed hierarchical knowledge distillation strategy.Source: §6 · p. 12
- Future work will collect more high-resolution event videos and pre-train an event-based tracker using self-supervised learning.Source: §6 · p. 12