Source-linked AI summary
TCTrack: Temporal Contexts for Aerial Tracking
Ziang Cao, Ziyuan Huang, Liang Pan, Shiwei Zhang, Ziwei Liu, Changhong Fu
TL;DR
Aerial trackers must remain robust under challenging visual conditions while meeting tight computational constraints, but existing methods do not fully exploit temporal context. TCTrack integrates temporal information into feature extraction and similarity-map refinement, achieving competitive results across four benchmarks and over 27 FPS on an NVIDIA Jetson AGX Xavier. The paper also identifies limits in very long-term temporal modeling and long-time occlusion.
Problem
Existing aerial trackers underuse temporal information across the tracking pipeline, although aerial conditions demand robust and efficient tracking.
Method
TCTrack combines online temporally adaptive convolution for feature extraction with an adaptive temporal transformer for similarity-map refinement.
Results
TCTrack achieves competitive accuracy and precision against 51 state-of-the-art trackers across four aerial benchmarks, runs at 125.6 FPS on PC, and exceeds 27 FPS on NVIDIA Jetson AGX Xavier.
Takeaways & Limitations
Temporal contexts can be explored at both feature and similarity-map levels while retaining deployment-oriented efficiency for aerial tracking.
Takeaways & Limitations
Short-term training leaves very long-term temporal modeling and long-time occlusion insufficiently explored.
Abstract
from arXiv · showhide
Temporal contexts among consecutive frames are far from being fully utilized in existing visual trackers. In this work, we present TCTrack, a comprehensive framework to fully exploit temporal contexts for aerial tracking. The temporal contexts are incorporated at \textbf{two levels}: the extraction of \textbf{features} and the refinement of \textbf{similarity maps}. Specifically, for feature extraction, an online temporally adaptive convolution is proposed to enhance the spatial features using temporal information, which is achieved by dynamically calibrating the convolution weights according to the previous frames. For similarity map refinement, we propose an adaptive temporal transformer, which first effectively encodes temporal knowledge in a memory-efficient way, before the temporal knowledge is decoded for accurate adjustment of the similarity map. TCTrack is effective and efficient: evaluation on four aerial tracking benchmarks shows its impressive performance; real-world UAV tests show its high speed of over 27 FPS on NVIDIA Jetson AGX Xavier.
1. Introduction
Aerial tracking requires robustness to challenging conditions and efficiency under UAV resource limits, while existing trackers underuse temporal information. TCTrack addresses this gap by incorporating temporal context into feature extraction and similarity-map refinement, with strong benchmark and deployment results.
- Aerial tracking must handle motion blur, camera motion, and occlusion while operating under limited onboard computational resources.
- Existing tracking frameworks often process frames independently or use temporal information only in template features, leaving broader temporal dependencies underexploited.
- TCTrack introduces temporal context at two levels: feature extraction through TAdaCNN and similarity-map refinement through AT-Trans.
- TAdaConv dynamically calibrates convolution weights using previous-frame features, while AT-Trans encodes and decodes temporal knowledge to refine similarity maps.
- 125.6 FPS on PC and over 27 FPS on NVIDIA Jetson AGX Xavier accompany competitive accuracy and precision across four aerial tracking benchmarks.
2. Related Work
Prior aerial trackers emphasize efficiency or accuracy but generally neglect temporal context or introduce it at only one tracking stage. Related video-modeling work motivates temporal operations, while TCTrack extends them to two tracking levels.
- DCF-based trackers suit aerial platforms because of their efficiency and low resource requirements but struggle with fast motion and severe appearance variation.
- Siamese-based trackers offer strong accuracy and efficiency, yet prior trackers disregard temporal contexts that could improve robustness during aerial tracking.
- Existing temporal tracking methods typically introduce temporal information at only one level, such as template features, limiting further performance improvement.
- Temporal modelling in videos: Video temporal modeling uses mechanisms including 3D convolutions, temporal convolution, and temporal shift, providing related design precedents.
- TCTrack adapts temporal modeling to tracking by jointly exploring feature extraction and similarity-map refinement.
3. Temporal Contexts for Aerial Tracking
TCTrack incorporates temporal contexts at both feature extraction and similarity-map refinement. TAdaCNN adaptively calibrates convolution parameters from frame history, while AT-Trans encodes and decodes temporal knowledge to refine similarity maps efficiently.
- Framework overview: TCTrack introduces temporal context at two levels: online feature extraction with TAdaCNN and similarity-map refinement with AT-Trans.The framework also includes classification and regression for final prediction.
- Online feature extraction: TAdaConv dynamically adjusts convolution weights and biases for each frame using calibration factors derived from a temporal context queue.The queue stores L frame descriptors, including the current frame, and uses global average-pooled features to generate calibration factors.
- Online feature extraction: TAdaCNN processes one frame at a time and uses past temporal context, with early missing history filled by the first frame descriptor.This online design matches real-world tracking, where future frames are unavailable.
- Similarity refinement with AT-Trans: AT-Trans uses an encoder-decoder structure to encode temporal knowledge and refine the current similarity map through temporal filtering and attention.The decoder extracts valid information from the temporal prior to produce the refined output.
- Similarity refinement with AT-Trans: AT-Trans updates temporal knowledge rather than storing all intermediate information, keeping memory occupancy fixed throughout tracking.Its encoder uses multi-head attention and a temporal filter to support memory-efficient temporal modeling.
- Similarity refinement with AT-Trans: Similarity-map refinement is reported as effective under camera motion, severe motion, and occlusion, while TAdaConv and AT-Trans are presented as first attempts at their respective tracking stages.Figure 5 compares maps before and after refinement; the paper states that the refinement improves robustness and accuracy.
4. Experiments
TCTrack is evaluated against efficient and deeper trackers on four aerial benchmarks, with ablations examining its temporal components and sequence length. The experiments report strong tracking performance alongside high computational efficiency.
- Evaluation setup: TCTrack is evaluated on four aerial benchmarks against 51 existing top trackers, using AlexNet because efficiency is essential for aerial tracking.The benchmarks are UAV123, UAVTrack112 L, UAV123@10fps, and DTB70.
- Benchmark comparison: TCTrack outperforms HiFT and SiamRPN++ on UAV123 in AUC by 3% and 4.3%, respectively, and ranks first on DTB70 with a 5% AUC improvement.DTB70 contains 70 severe-motion scenarios, while UAV123 includes 123 sequences and more than 112K frames.
- Benchmark comparison: TCTrack achieves precision 0.786 and success rate 0.582 on UAVTrack112 L.UAVTrack112 L is used to assess long-term aerial tracking performance and includes over 60k frames.
- Ablation study: AT-Trans improves overall AUC by 9.8% and overall precision by 6.7% over Transformer, while motion-scene improvements reach about 12.0% and 15.1%.In occlusion conditions, consecutive temporal contexts boost the success rate by 11.4%.
5. Real-world Tests
Real-world UAV tests evaluate TCTrack under illumination changes, scale variation, occlusion, motion blur, low resolution, and camera motion. The tracker maintains stable, accurate performance while running above 27 FPS.
- RAM and GPU VRAM usage are 15.29% and 3%, while average GPU and CPU utilization are 46% and 12.43%.
- TCTrack maintains impressive stability and robustness under partial occlusion and low illumination by exploiting consecutive temporal knowledge.
- TCTrack achieves satisfying accuracy under motion blur and object occlusion, and demonstrates strong tracking ability during camera motion.
- Over 27 FPS is maintained during practical UAV tests without TensorRT acceleration, supporting the framework’s feasible deployment ability.
6. Conclusion and Discussion
TCTrack introduces temporal contexts through feature extraction and similarity refinement, with experiments and UAV tests demonstrating effectiveness and efficiency. The authors identify limits in long-term temporal modeling and note future TensorRT and ONNX development, alongside a potential unauthorized-surveillance risk.
- TCTrack explores temporal contexts through two perspectives: feature extraction and similarity refinement.
- AT-Trans and online TAdaCNN are presented as first attempts to exhaustively explore temporal contexts, while online updating avoids unnecessary operations and memory loading.
- Experiments on four benchmarks and real-world UAV tests demonstrate the framework’s effectiveness and efficiency.
- The framework’s potential for very long-term temporal modeling and long-time occlusion remains underexplored because training is short-term.
- TensorRT and ONNX versions are identified as future development directions.
- The framework’s efficiency and effectiveness may facilitate unauthorized surveillance when deployed on UAVs.