PaperlayerView exact revision on arXiv

Source-linked AI summary

TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events

Jiaxiong Liu, Zhen Tan, Jinpu Zhang, Yi Zhou, Hui Shen, Xieyuanli Chen, Dewen Hu

arXiv:2603.04989v2cs.CV

TL;DR

TAPFormer addresses the challenge of precise, long-term arbitrary point tracking when frame and event modalities have mismatched sampling and variable reliability. It asynchronously fuses frames with continuous event updates and adaptively weights local cross-modal features, achieving state-of-the-art results across TAP and feature tracking benchmarks. The paper also contributes synthetic and real-world synchronized frame-event datasets, while event tracking remains limited in low-motion or static scenes.

  • Problem

    Arbitrary point tracking requires precise, long-term trajectories, but frame-event fusion can suffer temporal misalignment and degradation when one modality is unreliable.

  • Method

    TAPFormer uses transient asynchronous fusion for continuous event updates between frames and Cross-modal Locally Weighted Fusion to emphasize locally reliable modalities.

  • Results

    TAPFormer consistently achieves state-of-the-art performance across TAP and feature tracking benchmarks, outperforming frame-, event-, and fusion-based baselines.

  • Takeaways & Limitations

    The framework and accompanying datasets provide resources for robust frame-event tracking under diverse illumination and motion conditions.

  • Takeaways & Limitations

    Event-based tracking often fails in low-motion or static scenes because sparse events lack color and texture detail.

Abstract

from arXiv · show

Tracking any point (TAP) is a fundamental yet challenging task in computer vision, requiring high precision and long-term motion reasoning. Recent attempts to combine RGB frames and event streams have shown promise, yet they typically rely on synchronous or non-adaptive fusion, leading to temporal misalignment and severe degradation when one modality fails. We introduce TAPFormer, a transformer-based framework that performs asynchronous temporal-consistent fusion of frames and events for robust and high-frequency arbitrary point tracking. Our key innovation is a Transient Asynchronous Fusion (TAF) mechanism, which explicitly models the temporal evolution between discrete frames through continuous event updates, bridging the gap between low-rate frames and high-rate events. In addition, a Cross-modal Locally Weighted Fusion (CLWF) module adaptively adjusts spatial attention according to modality reliability, yielding stable and discriminative features even under blur or low light. To evaluate our approach under realistic conditions, we construct a novel real-world frame-event TAP dataset under diverse illumination and motion conditions. Our method outperforms existing point trackers, achieving a 28.2% improvement in average pixel error within threshold. Moreover, on standard point tracking benchmarks, our tracker consistently achieves the best performance. Project website: tapformer.github.io

1. Introduction

TAPFormer addresses the complementary weaknesses and temporal mismatch of RGB frames and event streams through asynchronous fusion and adaptive cross-modal attention. It targets robust, high-precision point tracking across diverse conditions and reports leading performance across benchmarks.

  • Task and motivation: TAP estimates motion trajectories for arbitrary video points, requiring accurate long-term tracking across lighting and motion conditions.The task is relevant to applications including augmented reality and autonomous systems.
  • Task and motivation: Frame cameras can degrade under blur, overexposure, limited frame rates, and rapid dynamics, while event streams become sparse and lack texture in slow or static scenes.Event patterns also depend on both scene and camera motion, complicating consistent feature formation.
  • Task and motivation: RGB frames provide detailed spatial structure, whereas events provide fine-grained temporal dynamics, motivating their fusion for stable arbitrary point tracking.The paper explicitly frames the modalities as complementary visual pathways.
  • Challenges: Asynchronous fusion can suffer spatial misalignment, while frame-rate matching through event downsampling sacrifices temporal fidelity.These issues arise from the frequency mismatch between discrete frames and asynchronous events.
  • Proposed approach: TAPFormer uses transient asynchronous fusion to update representations continuously with events between frames and CLWF to emphasize locally reliable modalities.CLWF is designed to maintain spatially coherent features when blur or sparsity degrades one modality.
  • Evaluation and contributions: The work introduces synthetic training data and a real-world synchronized frame-event benchmark, then evaluates both TAP and feature point tracking.The real-world benchmark covers challenging illumination and motion conditions.

2. Related Work

Prior work spans frame-based TAP, event-based tracking, and multimodal fusion, but existing approaches remain limited by long-term drift, sparse events, unreliable modalities, or task-specific constraints. TAPFormer addresses these limitations with asynchronous fusion and locally adaptive cross-modal representations.

  • Frame-based tracking: TAP methods model trajectories across image sequences, while optical flow estimates dense consecutive-frame correspondences but can drift during long-term tracking.TAP-Vid and PointOdyssey provide standardized or large-scale synthetic resources for TAP.
  • Event-based tracking: Early event trackers used geometric or optimization-based methods, while later data-driven methods improved robustness but were limited to predefined feature points.These approaches therefore did not generalize to arbitrary point tracking.
  • Event-based tracking: ETAP adapted trajectory optimization to event-based TAP, and MATE introduced motion augmentation to improve temporal consistency and robustness.Both methods are described as event-domain advances using synthetic or motion-augmented training.
  • Event-based tracking: Event-based tracking remains unreliable in low-motion or static scenes because sparse events provide limited color and texture detail.The passage states that event-only methods are less reliable than frame-based approaches in most scenarios.
  • Frame-event fusion: Frame-event fusion leverages image texture and stable appearance alongside event temporal resolution and dynamic range, but prior methods are often frame-rate constrained or degrade when a modality is unreliable.Existing fusion methods are also not directly applicable to TAP’s pixel-level, long-term requirements.
  • TAPFormer: TAPFormer combines transient event updates between frames with locally weighted cross-modal fusion to select reliable regions and support precise, robust tracking.The design targets motion blur, occlusion, and illumination changes.

3. Method

TAPFormer fuses RGB frames with asynchronous events to build temporally consistent, locally reliability-aware representations for high-frequency point tracking. These representations are decoded into multi-scale features and iteratively optimized to predict point trajectories and visibility.

  • Problem Definition: Event cameras provide asynchronous brightness changes, while TAPFormer combines images and event representations to estimate query-point trajectories and visibility over discrete times.Events are grouped into time bins and transformed into event representations for fusion with image inputs.
  • Transient Asynchronous Fusion: TAF initializes a transient representation from each frame and its exposure-window events, then updates it asynchronously as new event batches arrive.The update integrates fine-grained event cues while preserving spatial consistency from the recent frame.
  • Transient Asynchronous Fusion: TAF maintains temporal fidelity at event-rate updates, enabling smooth tracking beyond standard frame rates in high-speed conditions.The tracker can estimate query-point locations at arbitrary query times, typically operating at 100–200 Hz versus 20–30 Hz for images.
  • Cross-modal Local Weighted Fusion Module: CLWF uses localized cross-attention so event tokens gather neighboring image information and emphasize the more informative modality in each spatial region.A learned locality bias can strengthen spatially adjacent token influence, while residual fusion preserves event-driven temporal dynamics.
  • Cross-modal Local Weighted Fusion Module: Temporal self-attention and skip-connected decoding produce multi-scale fused feature maps combining fine-grained temporal detail with global semantic context.A transformer optimizer then uses these maps and the initial query point to iteratively estimate trajectory and visibility.

4. Dataset Overview

TAPFormer is trained with a high-frame-rate synthetic frame-event dataset and evaluated on a diverse real-world benchmark. The real dataset spans varied illumination, motion, and indoor/outdoor conditions.

  • Training Dataset Generation: FE-FastKub renders 512×512 scenes at 48 FPS over 2 s, providing temporally dense trajectories and realistic event simulation for training.The dataset is generated with the Kubric engine and offers finer temporal granularity than the 12 FPS MoVi dataset.
  • Training Dataset Generation: The synthetic dataset contains 10,953 samples with 96 RGB images, 1024 ground-truth trajectories at 48 FPS, and corresponding v2e event streams.Training samples 12 Hz image inputs while using 48 Hz trajectory supervision.
  • Real Dataset Construction: The real benchmark includes InivTAP sequences from a DAVIS346 camera and DrivTAP driving sequences from synchronized Prophesee event and RGB cameras.The subsets use different resolutions and capture rates, including 20 FPS for InivTAP and 22 FPS RGB capture for DrivTAP.
  • Real Dataset Construction: InivTAP covers fast motion, low light, overexposure, static scenes, dual motion, and indoor and outdoor environments.It contains 8 sequences lasting 5–7.5 s, with an average of 12.8 annotated points per sequence.
  • Real Dataset Construction: The real-world dataset provides 20,450 annotated points across 13 diverse sequences and is presented as a frame-event TAP benchmark for realistic evaluation.Its stated purpose is evaluating frame-event fusion methods under challenging real-world conditions.

5. EXPERIMENTS

Experiments evaluate TAPFormer on arbitrary point tracking and feature tracking across challenging real-world datasets, including comparisons, ablations, and frame-rate sensitivity analyses. TAPFormer consistently achieves strong tracking performance, with fusion improving robustness across modalities and operating conditions.

  • Evaluation setup: Experiments evaluate arbitrary point tracking on InivTAP and DrivTAP, plus feature point tracking on EDS and EC.The evaluation reports TAP-Vid metrics for TAP and Feature Age metrics for feature tracking.
  • Task 1: TAP: TAPFormer achieves the best overall performance on DrivTAP, improving AJ by 261.5% over ETAP, 31.5% over CoTracker3, and 32.6% over FETAP.DrivTAP contains challenging driving sequences with changing illumination, fast traffic, low relative motion, and frequent blur.
  • Task 2: Feature Tracking: TAPFormer consistently improves EFA across the EDS and EC feature-tracking datasets, maintaining smooth trajectories under fast motion and partial occlusions.The method integrates frame and event information for temporally consistent and fine-grained tracking.
  • Ablation Study: The complete ablation model achieves FA 0.823 and EFA 0.704, while the fusion module, multi-scale semantic features, and temporal attention each contribute progressively.Removing the fusion module replaces it with channel-wise concatenation followed by a convolutional layer.
  • Modality analysis: Joint fusion mitigates modality-specific weaknesses: frame-only tracking degrades under fast motion or overexposure, while event-only tracking lacks spatial texture cues.The fused representations also show tighter intra-point clustering and larger inter-point margins than single-modality representations.
  • Frame Rate Sensitivity: As frame rate decreases, TAPFormer loses 6.5% performance compared with a 75.3% drop for the frame-only method.CoTracker3 becomes nearly invalid at 10 Hz, whereas the fusion-based tracker remains stable under low-frame-rate and high-speed conditions.

6. Conclusion

TAPFormer combines transient temporal updates with cross-modal local weighted fusion for robust, high-frequency point tracking. It also contributes synthetic and real-world frame-event datasets for challenging conditions.

  • TAPFormer uses transient feature updates to preserve event-stream temporal dynamics and cross-modal local weighted fusion to integrate complementary frame and event cues.The fused representations are designed to remain stable and discriminative across diverse visual conditions.
  • The work contributes a large-scale synthetic dataset for high-frame-rate supervision and a manually annotated real-world TAP dataset with synchronized frame-event sequences.The real-world data cover challenging illumination and motion conditions.

7. Additional Method Details

The trajectory optimizer refines coordinates and visibility from a three-level fused feature pyramid using local patches, temporal correlations, motion encoding, and a spatio-temporal transformer. Three iterations provide fast convergence while reducing computation relative to prior approaches.

  • Trajectory refinement: The optimizer uses a three-level fused feature pyramid to iteratively refine point coordinates and visibility in temporal windows of size W = 16.It is based on a transformer trajectory optimizer.
  • Local feature sampling: At each iteration, local (2r + 1) × (2r + 1) patches with r = 3 are sampled around predicted coordinates at every pyramid level.
  • Temporal correlation: Temporal correlations across sampled patches are flattened, encoded by an MLP, and concatenated across the three scales into a correlation descriptor.
  • Transformer update: Correlation embeddings, visibility estimates, and relative-motion encoding are passed through a spatio-temporal transformer to predict residual coordinate and visibility updates.
  • Efficiency: Three refinement iterations achieve fast convergence, whereas prior approaches such as CoTracker3 typically require six iterations.The reduced iteration count maintains high accuracy while significantly reducing computational cost.

8. Additional Ablation Experiments

The additional experiments examine event representations, refinement efficiency, and ground-truth correction. Performance saturates near three refinement iterations, supporting the selected accuracy–efficiency trade-off.

  • Event representation: Dense event representations enable standard convolutional processing and aggregate asynchronous events within short temporal intervals.
  • Event representation: Event Image, Voxel Grid, and Time Surfaces are evaluated using five temporal sub-windows under the Stacking Based on Time scheme.They respectively encode event counts, discretized spatio-temporal volumes, and decayed latest-event timestamps.
  • Ground-truth correction: Ground-truth correction smooths two displacement outliers in the peanuts light sequence and improves performance across all sequences.The correction also yields more distinguishable evaluation metrics.
  • Qualitative analysis: Additional visualizations provide qualitative comparisons on the EC and EDS datasets.
  • Iteration number: Performance saturates around the third refinement iteration, while later iterations provide marginal gains and noticeably increase computational cost.The experiments therefore adopt three iterations.

9. Extended Experimental Results

The extended experiments provide detailed evaluations on the InivTAP, EDS, and EC datasets, alongside additional qualitative comparisons. Ground-truth correction is also examined for the peanuts light sequence.

  • Quantitative results: Detailed results are reported for the InivTAP dataset, while full evaluation results are provided for the EDS and EC datasets.
  • Ground-truth correction: Ground-truth correction for the peanuts light sequence smooths two displacement outliers and improves performance across all sequences.The correction produces more distinguishable evaluation metrics.
  • Quantitative results: Table 6 presents performance comparisons across different scenarios and modalities.

10. Application to SLAM

TAPFormer is evaluated for speed-aware SLAM performance using a velocity-weighted success metric and is reported as the best-performing approach among the evaluated methods.

  • Evaluation Metric: The speed-weighted success score weights thresholded relative-velocity accuracy by ground-truth velocity magnitude.This emphasizes accurate estimates during high-speed motion.
  • Evaluation Metric: Sweeping the error threshold ξ from 0 to 1 produces a success-rate curve summarized by the speed-aware AUCv metric.The formulation uses a Dirac indicator for thresholded success.
  • Baselines: TAPFormer is compared with SuperEvent + OKVIS2 and SDEVO, representing multimodal and purely stereo-event SLAM baselines.SuperEvent + OKVIS2 uses stereo events, stereo images, and IMU measurements, whereas SDEVO uses stereo events alone.
  • Results: TAPFormer achieves the best performance among all evaluated methods according to the final AUCv scores.Qualitative tracking results are shown on SLAM sequences, including high-speed UAV and ground-based scenarios.

11. Dataset Details

The paper provides synthetic training data and two manually annotated real-world frame–event TAP test sets designed to cover diverse motion, occlusion, illumination, and imaging conditions.

  • Dataset Construction: The dataset suite combines a large-scale synthetic dataset with two real, manually annotated frame–event TAP test sets.The real test sets are InivTAP and DrivTAP.
  • Synthetic Dataset: FE-FastKub provides high-frame-rate photorealistic RGB images, synthetic event streams, motion blur, and explicit rapid-motion scenarios.It is designed to improve adaptability to fast-moving scenes.
  • Real-World Test Sets: InivTAP and DrivTAP contain manually annotated point trajectories and occlusion labels for evaluating asynchronous frame–event tracking in real scenes.Their collection equipment and annotation interface are documented in Figures 12 and 15.
  • Annotation Pipeline: Annotations are initialized with RAFT optical flow, refined using RGB or event-reconstructed frames, and smoothed by filtering high-frequency noise.Annotators use whichever modality provides clearer visual cues.
  • Trajectory Statistics: More than 60% of tracks span over 90% of sequence duration, 80.04% are never occluded, and most trajectories form one continuous segment.These statistics describe trajectory visibility and continuity in the manually annotated datasets.
  • Motion and Conditions: 16.51% of points move more than 5% of image height between consecutive frames, while 8.1% exceed 10%, alongside nearly static points.The datasets also include overexposure and low-light scenes.
Loading 2603.04989v2…