PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Real-World Visual Tracking with Temporal Contexts

Ziang Cao, Ziyuan Huang, Liang Pan, Shiwei Zhang, Ziwei Liu, Changhong Fu

arXiv:2308.10330v1cs.CV

TL;DR

Many visual trackers neglect temporal context or use it mainly in templates, and real-world tracking also makes memory and inference time relevant. TCTrack++ incorporates temporal information into features and similarity maps, and the paper reports strong benchmark results and real-time performance in real-world tests.

  • Problem

    Many trackers neglect rich temporal information or integrate it only into templates, while real-world use also raises memory and inference-time concerns.

    Source: §1 · p. 1

  • Method

    TCTrack++ uses attention-based temporally adaptive convolution for feature extraction, an adaptive temporal transformer for similarity-map refinement, and curriculum learning.

    Source: §1 · pp. 1–2

  • Results

    TCTrack++ shows superior effectiveness and efficiency in online and offline evaluations across eight benchmarks, with real-time speed reported in real-world tests.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    Real-world indoor and outdoor tests on NVIDIA Jetson AGX Xavier report promising tracking performance at real-time speed.

    Source: §1 · p. 2

    Temporal information can mislead tracking during motion blur or occlusion.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Visual tracking has made significant improvements in the past few decades. Most existing state-of-the-art trackers 1) merely aim for performance in ideal conditions while overlooking the real-world conditions; 2) adopt the tracking-by-detection paradigm, neglecting rich temporal contexts; 3) only integrate the temporal information into the template, where temporal contexts among consecutive frames are far from being fully utilized. To handle those problems, we propose a two-level framework (TCTrack) that can exploit temporal contexts efficiently. Based on it, we propose a stronger version for real-world visual tracking, i.e., TCTrack++. It boils down to two levels: features and similarity maps. Specifically, for feature extraction, we propose an attention-based temporally adaptive convolution to enhance the spatial features using temporal information, which is achieved by dynamically calibrating the convolution weights. For similarity map refinement, we introduce an adaptive temporal transformer to encode the temporal knowledge efficiently and decode it for the accurate refinement of the similarity map. To further improve the performance, we additionally introduce a curriculum learning strategy. Also, we adopt online evaluation to measure performance in real-world conditions. Exhaustive experiments on 8 wellknown benchmarks demonstrate the superiority of TCTrack++. Real-world tests directly verify that TCTrack++ can be readily used in real-world applications.

1 INTRODUCTION

Existing trackers often neglect temporal information or use it only in templates, while real-world tracking also faces memory and complexity constraints. TCTrack++ uses temporal context at both the feature and similarity-map levels, and the paper reports strong results across offline, online, and real-world evaluations.

1 INTRODUCTION

  • TCTrack++ introduces temporal context at both feature extraction and similarity-map refinement to address limitations in existing trackers.ATT-TAdaConv adjusts feature extraction using temporal context, while AT-Trans refines similarity maps using temporal knowledge.Source: §1 · pp. 1–2
  • A fixed-size calibration vector integrates historical search features without storing historical frames, keeping memory loading and computation low and constant regardless of sequence length.The approach also avoids hyperparameters associated with historical frames.Source: §1 · p. 1
  • AT-Trans combines the previous temporal prior with the current similarity map, then adaptively refines the map using that prior.The temporal prior has fixed size and updates at each frame, supporting memory-efficient operation.Source: §1 · p. 2
  • The paper extends the CVPR 2022 conference version with new architectures, experiments, a survey, and implementation details.The authors identify TCTrack as the earlier version.Source: §1 · p. 2
  • Curriculum learning trains the tracker from easier data with a small temporal range to harder data with a larger range.The strategy is adopted to lower training memory load and improve convergence.Source: §1 · p. 2
  • Evaluations on eight benchmarks include online assessment of real-world tracking performance and offline assessment independent of inference time and latency.The authors also report tests in indoor and outdoor scenes on NVIDIA Jetson AGX Xavier, with real-time speed.Source: §1 · p. 2

2 RELATED WORK

Prior tracking work uses detection-based methods and temporal context in different ways, but many approaches focus on templates or video clips and face memory or hyperparameter costs. TCTrack++ is presented as a two-level, low-memory framework, while online evaluation accounts for inference speed in real-world conditions.

2.1 Tracking by Detection Methods

  • Detection-based trackers can achieve state-of-the-art performance, but the cited work identifies neglected temporal context as a limitation to further performance improvement.The related-work discussion also notes efforts to balance performance and speed on resource-limited aerial platforms.Source: §2.1 · p. 2

2.2 Temporal-Based Tracking Methods

  • Temporal tracking methods commonly update dynamic templates or use consecutive-frame clips, but clip-based approaches require length-related hyperparameters that affect accuracy and speed.The discussion notes that methods using discrete historical frames can lose information from consecutive clips.Source: §2.2 · p. 2
  • TCTrack++ addresses single-level temporal modeling and spatial-only feature extraction with two temporal-context levels and a low-memory structure that avoids growing memory use and hyperparameters.It processes consecutive contexts from the whole tracking scenario online rather than relying on video clips.Source: §2.2 · pp. 2–3

2.3 Temporal Modelling in Videos

  • Temporally adaptive convolutions adjust frame-wise convolution weights according to local and global temporal contexts; this work adapts that idea to online CNN feature extraction for tracking.The passage situates this approach alongside supervised and self-supervised video temporal modeling.Source: §2.3 · p. 3

2.4 Online Evaluations

  • Online evaluation treats inference time as consequential because the tracked object can move during processing, so the paper uses both online and offline evaluation.Offline evaluation assesses tracking performance independently of inference time.Source: §2.4 · p. 3

3 TEMPORAL CONTEXTS FOR REAL-WORLD TRACKING

TCTrack++ uses temporal context at both feature extraction and similarity-map refinement. Its components are ATT-TAdaConv and the encoder-decoder AT-Trans, followed by classification and regression losses.

3 TEMPORAL CONTEXTS FOR REAL-WORLD TRACKING

  • The framework combines online feature extraction with ATT-TAdaConv and similarity-map refinement with AT-Trans.The method section presents these as its two components for incorporating temporal context into the tracking pipeline.Source: §3 · p. 3

3.1 Online Feature Extraction with ATT-TAdaConv

  • ATT-TAdaConv integrates temporal context during feature extraction, adjusting convolution weights and bias for each frame using temporal information.It uses cross-attention to aggregate temporal information adaptively without the video-length hyperparameter L used by Online TAdaConv.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Similarity Refinement with AT-Trans

  • AT-Trans encodes temporal knowledge from consecutive frames and decodes it to refine the current similarity map.Its online updating strategy uses fixed temporal prior knowledge, which the authors describe as memory-efficient compared with approaches that save all intermediate temporal information.Source: §3.2 · pp. 4–5

3.3 Loss Functions

  • The loss design adds Lloc1 to improve regression precision, identifies Lloc2 as an IoU loss, and uses cross-entropy and binary cross-entropy for two classification branches.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments compare trackers across eight public benchmarks and include both latency-aware online and latency-free offline evaluations.

4 EXPERIMENTS

  • Evaluations cover OTB100, GOT-10K, LaSOT, UAV123, UAVTrack112_L, UAV123@10fps, VisDrone2018-test, and DTB70.The comparison includes more than 50 existing trackers, using their official code and corresponding hyperparameters.Source: §4 · p. 6
  • Online evaluation measures real-world tracking performance, while offline evaluation assesses tracking without inference-time latency.For offline comparison, trackers are grouped as lightweight or deep trackers.Source: §4 · p. 6
  • Offline comparisons distinguish SiamRPN++_A with AlexNet from SiamRPN++_Mobv2 with MobileNet_v2.The paper also groups the compared trackers into lightweight and deep categories.Source: §4 · p. 6

4.1 Implementation Details

Implementation choices emphasize low inference latency and train the tracker with temporally lengthened video sequences.

4.1 Implementation Details

  • AlexNet is selected as the backbone because it has the lowest latency among the compared popular lightweight backbones on the resource-limited platform.The paper notes that mobile networks can incur high memory-access cost, and that inference latency increases mismatch with real-world information.Source: §4.1 · p. 6
  • Curriculum training increases video length from 2 to 3 and then 4 across training, with input video lengths transformed from 1 to 4.The video length is 2 for epochs 1–33, varies to 3 for epochs 33–50, and increases to 4 afterward.Source: §4.1 · p. 6
  • Training uses COCO, VID, LaSOT, and GOT-10K images for 100 epochs on two TITAN RTX GPUs, freezing the backbone for the first 10 epochs.The backbone is unfrozen for the last 90 epochs; ATT-TAdaConv replaces the final two convolutional layers.Source: §4.1 · p. 6

4.2 Evaluation Criteria

The evaluation uses OPE precision and success rate, with AUC for ranking and additional GOT-10K metrics.

4.2 Evaluation Criteria

  • OPE precision is defined by center location error between the ground-truth and predicted boxes, while success rate uses IoU.Source: §4.2 · p. 6
  • Trackers are ranked using the area-under-the-curve metric.Source: §4.2 · p. 6
  • GOT-10K evaluation additionally uses average overlap and success rate, with SR_i measuring frames whose IoU exceeds i.Average overlap is the average IoU between predicted boxes and ground truth.Source: §4.2 · p. 6

4.3 Online Evaluation

Latency-aware online evaluations compare trackers on four benchmarks, with TCTrack++ ranking highly across reported benchmark results and challenging-scene attributes.

4.3 Online Evaluation

  • Online evaluation accounts for inference time on the resource-limited NVIDIA Jetson AGX Xavier platform.The evaluations use one hardware platform to control other factors that may affect tracking results.Source: §4.3 · p. 6

4.3.5 Results on GOT-10K

  • TCTrack++ exceeds the second-best trackers by 14.3% in AO, 19.0% in SR0.5, and 13.6% in SR0.75 on GOT-10K.The online comparison covers four benchmarks; some trackers could not run using their official code because of NVIDIA AGX Xavier memory limitations.Source: Table 2 · p. 7, §4.3.5 · p. 8

4.3.1 Evaluation Settings

  • Latency-aware evaluation couples speed and tracking performance by requiring the tracker to provide the latest result at each time point.Because of latency, the available result may correspond to an earlier frame.Source: §4.3.1 · p. 7

4.3.2 Results on OTB100

  • On OTB100, TCTrack++ ranks first in precision (0.720), normalized precision (0.550), and success rate (0.543).The reported improvements over the best deep trackers are 4.7%, 7.2%, and 4.8% in those three metrics, respectively.Source: §4.3.2 · p. 7, Table 2 · p. 7

4.3.3 Results on LaSOT

  • On LaSOT, TCTrack++ improves precision by 13.1% and normalized precision by 11.0% over the best deep tracker.It also surpasses other lightweight trackers and TCTrack by 7.3% and 8.7%, respectively; it ranks first across nine attributes, exceeding the second-best tracker by 6.9%.Source: §4.3.3 · p. 7, Figure 5 · p. 8

4.3.4 Results on UAV123

  • On UAV123, TCTrack++ ranks first in precision (0.731) and success (0.519), maintaining a reported trade-off between performance and speed.Source: §4.3.4 · p. 7

4.3.6 Qualitative Results

  • Qualitative results show TCTrack++ maintaining robustness and accuracy against five other trackers in fast-motion, scale-variation, and occlusion scenes.Source: §4.3.6 · p. 8

4.4 Offline Evaluation

Offline evaluation compares lightweight and deep trackers across five benchmarks. TCTrack++ is reported to perform strongly against both groups, with substantial efficiency reductions relative to named deep baselines.

4.4 Offline Evaluation

  • The offline evaluation follows the standard pipeline across five benchmarks and reports comparisons separately for lightweight and deep trackers.The supplied results describe UAV123@10fps, UAVTrack112, VisDrone2018-test, OTB100, and DTB70.Source: §4.4 · p. 8, §4.4.1 · pp. 8–9, §4.4.2 · p. 9

4.4.1 Comparison with Light-Weight Trackers

  • TCTrack++ achieves the best tracking performance among the compared lightweight trackers on UAV123@10fps and ranks first on three VisDrone2018-test benchmarks.The authors also report a 2% success-rate improvement over TCTrack on UAV123@10fps.Source: §4.4.1 · pp. 8–9

4.4.2 Comparison with Deep SOTA Trackers

  • Against SiamRPN++ on OTB100, TCTrack++ reduces MACs 11×, parameters 8×, and improves speed 4×, despite a 4% offline performance reduction.Against SiamCAR on DTB70, it reduces MACs 11× and parameters 7× while outperforming some deep trackers.Source: §4.4.2 · p. 9

4.5 Ablation Study

The ablations examine temporal-transformer components, the temporal backbone, and training strategy. The authors report improvements over the baseline in offline and online evaluation, including a large GOT-10K gain.

Analysis on AT-Trans

  • Filtering invalid temporal information improves tracking, convolutional initialization raises performance by about 6% in occlusion, and current-similarity-map queries improve it by over 10% in motion scenarios.Without filtering, adding consecutive temporal knowledge confuses the tracker and significantly impedes performance.Source: §4.5 · pp. 9–10
  • AT-Trans improves overall AUC by 9.8% and overall precision by 6.7%; reported gains are 12.0% and 15.1% in motion scenes and 11.4% in occlusion success rate.These comparisons are against Transformer without the encoded temporal knowledge.Source: §4.5 · pp. 10–11

Studies about the ATT-TAdaConv and curriculum learning

  • ATT-TAdaConv gains 17.2% over the baseline on OTB100, with online TAdaConv performance rising as its image range increases.The attention mechanism further boosts the temporal backbone's accuracy.Source: §4.5 · p. 11
  • Curriculum learning increases training-video length step by step, moving from easier to harder temporal contexts; the reported comparisons confirm its effectiveness.The authors state that longer videos make learning temporal contexts harder.Source: §4.5 · pp. 11–12

Ablation study of backbone

  • A deeper ResNet backbone improves offline performance from 60.8 to 66.1, but FPS falls from 27.1 to 10.1 and real-world performance from 0.519 to 0.472; AlexNet is preferred.The authors prioritize AlexNet for real-world tracking performance.Source: §4.5 · p. 12

4.5 Ablation Study

  • TCTrack++ improves over the baseline in offline and online evaluation, with an approximately 134% improvement on GOT-10K.The authors also report better robustness from two-level temporal contexts than from trackers without them.Source: §4.5 · p. 12

5 REAL-WORLD TESTS

The authors test TCTrack++ on an NVIDIA Jetson AGX Xavier across challenging outdoor and indoor scenes. They report real-time speed and tracking performance across low resolution, low illumination, occlusion, camera motion, and out-of-view conditions.

5 REAL-WORLD TESTS

  • Real-world tests use a UAV-collected video set on NVIDIA Jetson AGX Xavier, covering indoor and outdoor scenes, day and dark conditions, and multiple tracking challenges.CLE is measured with a 20-pixel threshold; average speed is 27.1 FPS, with 53% GPU and 13.42% CPU utilization.Source: §5 · p. 12

5.1 Outdoor Tests

  • In outdoor tests, TCTrack++ maintains tracking in low-resolution and very low-illumination scenes, and can relocate targets after severe occlusion.The authors attribute relocation after an object reappears to temporal contexts.Source: §5.1 · p. 12

5.2 Indoor Tests

  • In indoor tests with frequent severe camera movement and deliberately introduced out-of-view scenes, TCTrack++ maintains what the authors describe as superior real-world tracking performance.Source: §5.2 · p. 12

6 CONCLUSION AND DISCUSSION

The conclusion presents TCTrack++ as an efficient framework for temporal-context modeling at feature and similarity-map levels. The authors report superior real-world tracking performance and efficiency across online and offline experiments.

6 CONCLUSION AND DISCUSSION

  • TCTrack++ uses ATT-TAdaConv and AT-Trans to explore temporal contexts at feature and similarity-map levels, respectively.The authors also introduce curriculum learning to further boost tracking performance.Source: §6 · p. 12
  • An online updating strategy avoids unnecessary operations and memory loading.Source: §6 · p. 12
  • Online and offline experiments on eight benchmarks are reported to demonstrate superior real-world tracking performance and efficiency.Source: §6 · pp. 12–13

APPENDIX A FUTURE WORKS AND RESEARCH PROSPECTS

The authors prioritize AlexNet for current online applications while identifying the deeper-backbone/latency trade-off as future work.

APPENDIX A FUTURE WORKS AND RESEARCH PROSPECTS

  • The authors identify balancing deeper-backbone performance with latency as future work, while retaining AlexNet as their priority for current hardware and online applications.They report ResNet50 comparisons but do not provide specific comparison values in this passage.Source: Appendix A · p. 13
  • Across eight benchmarks, the authors report promising tracker speed and accuracy.Source: Appendix A · p. 13
  • With ResNet50, the tracker shows impressive online and offline performance and FPS compared with other trackers.Source: Appendix A · p. 13

Related papers

Loading 2308.10330v1…