Source-linked AI summary
Object Tracking by Jointly Exploiting Frame and Event Domain
Jiqing Zhang, Xin Yang, Yingkai Fu, Xiaopeng Wei, Baocai Yin, Bo Dong
TL;DR
Single-object tracking remains difficult because frame and event cameras each provide incomplete information, especially in degraded conditions. The paper fuses both domains with attention and adaptive weighting, introduces the FE108 dataset, and reports substantial gains over frame-based trackers. It also identifies a current limitation: the method does not yet exploit events’ high measurement rate.
Problem
Frame cameras struggle with limited frame rates and dynamic range, while event cameras lack absolute-light and texture information, motivating complementary frame-event tracking.
Method
The paper aggregates asynchronous events and fuses frame-event features using a Cross-Domain Feature Integrator with self-attention, cross-attention, and adaptive weighting.
Results
The proposed approach outperforms state-of-the-art frame-based tracking methods by at least 10.4% in success rate and 11.9% in precision rate.
Takeaways & Limitations
The results indicate that leveraging frame-event complementarity boosts tracking robustness in degraded conditions, while FE108 supports further multi-modal learning research.
Takeaways & Limitations
The current method does not leverage the high measurement rate of event cameras.
Abstract
from arXiv · showhide
Inspired by the complementarity between conventional frame-based and bio-inspired event-based cameras, we propose a multi-modal based approach to fuse visual cues from the frame- and event-domain to enhance the single object tracking performance, especially in degraded conditions (e.g., scenes with high dynamic range, low light, and fast-motion objects). The proposed approach can effectively and adaptively combine meaningful information from both domains. Our approach's effectiveness is enforced by a novel designed cross-domain attention schemes, which can effectively enhance features based on self- and cross-domain attention schemes; The adaptiveness is guarded by a specially designed weighting scheme, which can adaptively balance the contribution of the two domains. To exploit event-based visual cues in single-object tracking, we construct a large-scale frame-event-based dataset, which we subsequently employ to train a novel frame-event fusion based model. Extensive experiments show that the proposed approach outperforms state-of-the-art frame-based tracking methods by at least 10.4% and 11.9% in terms of representative success rate and precision rate, respectively. Besides, the effectiveness of each key component of our approach is evidenced by our thorough ablation study.
1. Introduction
Frame and event cameras provide complementary visual information but each has important limitations in degraded conditions. The paper proposes adaptive cross-domain fusion and a frame-event tracking dataset to address these limitations.
- Motivation: Conventional frame cameras provide rich scene representations but have limited frame rates and dynamic range, making degraded-condition tracking difficult.Their limitations are illustrated under high dynamic range and fast-moving-object conditions.
- Motivation: Event cameras offer high temporal resolution and dynamic range but cannot measure absolute light intensity or provide texture cues.The paper reports temporal resolution up to 1MHz and dynamic range up to 140 dB for event cameras.
- Method: The proposed method aggregates asynchronous events into discretized slices that can be processed as conventional frames by CNN-based models.This addresses the mismatch between asynchronous event outputs and synchronous image-processing pipelines.
- Method: A cross-domain feature integrator uses self-attention, cross-domain attention, and adaptive weighting to combine frame- and event-domain cues.The weighting scheme regulates each domain’s contribution according to its reliability.
- Dataset: FE108 contains 108 sequences totaling 1.5 hours, with ground-truth annotations for both domains and annotation frequencies up to 40Hz and 240Hz.The dataset covers challenging scenes and degraded conditions for single-object tracking.
- Results: The approach outperforms state-of-the-art methods by a significant margin, and ablation results support the effectiveness of its attention-based schemes.The introduction reports this outcome without specifying the comparison values in the supplied passage.
2. Related Work
Prior tracking research largely used single-domain frame or event sensors, while multi-modal work emphasized RGB-D and RGB-T combinations. This paper instead focuses on the complementarity of frame and event domains for tracking.
- Single-Domain Object Tracking: Deep-learning-based single-domain tracking methods predominantly use conventional frame-based sensors.Only a few prior studies explored event-based cameras for tracking.
- Single-Domain Object Tracking: Earlier event-based tracking studies addressed tasks such as multi-person or multi-object tracking using event clustering or specialized event representations.Examples include Gaussian Mixture Model clustering, mean-shift clustering, time-image, and synchronous time-surface representations.
- Multi-Domain Object Tracking: Existing event-based tracking approaches did not consider leveraging frame domains, whereas this work targets their complementarity.The paper contrasts its frame-event focus with prior event-only approaches.
- Multi-Domain Object Tracking: Related multi-modal tracking methods commonly combine RGB with depth or thermal inputs to improve tracking performance.The paper positions frame-event fusion as a different multi-domain combination.
3. Methodology
The method converts asynchronous events into CNN-compatible representations and fuses them with frame features in a two-branch tracker. Its CDFI uses attention and adaptive weighting to combine complementary frame texture and event edge cues for target localization and discrimination.
- Event Aggregation: Event streams are aggregated between adjacent frames into n-bin voxel slices, then converted into n 2D frames using the latest event timestamp and polarity per pixel.The bin size is B = (Tj+1 − Tj)/n, and the resulting frames are scaled by 255 for processing.
- Network Architecture: The tracker uses shared-weight reference and test branches, with CDFI receiving a frame and intervening events and producing low- and high-level fused features.The classifier estimates a confidence map, while the bounding-box regressor uses IoU prediction to help locate the target.
- Cross-Domain Feature Integrator: CDFI combines a ResNet18-based frame feature extractor, an event feature extractor, and a cross-domain modulation and selection block.Frame features come from the fourth and fifth ResNet18 blocks, while event features are generated at corresponding low and high levels.
- Cross-Domain Feature Integrator: Cross-domain attention enhances one domain’s features using features from the other, generating enhanced event and frame representations.The cross-attention block is designed to exploit complementary texture, semantic, and edge information across the two sensors.
- Cross-Domain Feature Integrator: An adaptive weighted balance scheme combines feature contributions from the frame and event domains, while event features are fused by weighted summation.The event extractor applies learned weights, 1 × 1 convolution, sigmoid activation, and adaptive average pooling within its feature branches.
- Prediction Heads: The classifier follows DiMP to optimize a target-specific filter, and the bounding-box regressor follows ATOM to estimate IoU for target localization.The classifier uses the optimized filter to distinguish the target object from background distractors.
4. Dataset
FE108 is a large-scale frame-event tracking dataset designed to support multimodal learning across diverse objects, challenging scenes, and motion conditions. It provides high-frequency annotations in both frame and event domains.
- FE108 contains 108 sequences totaling 1.5 hours, with 21 object types across animals, vehicles, and daily goods.
- Frame annotations reach 20/40 Hz, while event-domain annotations reach 240 Hz.
- The dataset covers low-light, high-dynamic-range, and fast-motion scenes with and without motion blur.
- FE108 includes static and dynamic shots containing either single or multiple objects.
- Most annotated bounding-box centers lie near frame centers, while aspect-ratio distributions and event rates are explicitly characterized.
- The event-rate distribution is diverse because events are triggered by motion, indicating broad motion diversity across the 108 scenes.
5. Experiments
Experiments evaluate the proposed tracker against state-of-the-art methods on FE108 and EED, then examine multimodal inputs, fusion design, and adaptive weighting. The results show strong FE108 performance and support the importance of the proposed fusion components.
- 5.1. Comparison with State-of-the-art Trackers: 92.4% RPR and 63.4% RSR on FE108 exceed the runner-up by 11.9% and 10.4%, respectively.
- 5.1. Comparison with State-of-the-art Trackers: The proposed approach achieves the best results under all four FE108 challenging conditions, especially low-light and high-dynamic-range conditions.
- 5.1. Comparison with State-of-the-art Trackers: On EED, the method significantly outperforms other approaches in all conditions except WiB, although limited frames make the comparison less convincing.
- 5.1. Comparison with State-of-the-art Trackers: Combining frame and event inputs improves PrDiMP, DiMP, and ATOM, while the proposed cross-domain feature integrator still outperforms them by a considerable margin.
- 5.2. Ablation Study: Using multimodal inputs performs better than using unimodal inputs, according to the multi-modal input experiments.
- 5.2. Ablation Study: Removing CDMS has the largest performance impact, while removing self-attention has the smallest impact among the tested ablations.
- 5.2. Ablation Study: Removing adaptive weighting degrades performance most within CDMS, and event-domain weights exceed frame-domain weights in seven of eight visual examples.
6. Discussion and Conclusion
The paper introduces a frame-event fusion approach for single-object tracking and reports that combining both domains improves robustness in degraded conditions. It also identifies efficiency trade-offs in event aggregation and limitations related to tracking frequency and dataset coverage.
- Frame-event fusion is proposed to improve single-object tracking robustness in degraded conditions.
- Event aggregation introduces a trade-off between tracking accuracy and efficiency as the number of slices changes.
- Adaptive weighting balances the contributions of frame- and event-domain information.
- The proposed approach does not yet exploit event cameras’ high measurement rate, while the frame rate bounds tracking frequency.
- FE108 contains no sequence with a scenario of no events, limiting coverage of event-sparse conditions.