Source-linked AI summary
Parallel Tracking and Verifying: A Framework for Real-Time and High Accuracy Visual Tracking
Heng Fan, Haibin Ling
TL;DR
Real-time high-accuracy visual tracking remains scarce because efficient trackers are often less robust while deep approaches can be computationally burdensome. PTAV uses parallel fast tracking and on-demand verification, achieving the best accuracy among real-time trackers across four benchmarks and outperforming many deep learning-based solutions. The framework is flexible and supports future improvement and generalization.
Problem
Real-time high-quality tracking algorithms remain scarce because approaches emphasizing accuracy or speed have different limitations.
Method
PTAV runs a fast tracker and verifier in parallel, with verification requested only periodically and feedback used to correct tracking when needed.
Results
PTAV achieves the best tracking accuracy among real-time trackers across OTB2013, OTB2015, TC128, and UAV20L, outperforming many deep learning-based solutions.
Takeaways & Limitations
PTAV provides a flexible framework with room for improvement and generalization in designing efficient tracking algorithms.
Takeaways & Limitations
The local verification region is constrained to a square centered on the tracking result, with scale controlled by β.
Abstract
from arXiv · showhide
Being intensively studied, visual tracking has seen great recent advances in either speed (e.g., with correlation filters) or accuracy (e.g., with deep features). Real-time and high accuracy tracking algorithms, however, remain scarce. In this paper we study the problem from a new perspective and present a novel parallel tracking and verifying (PTAV) framework, by taking advantage of the ubiquity of multi-thread techniques and borrowing from the success of parallel tracking and mapping in visual SLAM. Our PTAV framework typically consists of two components, a tracker T and a verifier V, working in parallel on two separate threads. The tracker T aims to provide a super real-time tracking inference and is expected to perform well most of the time; by contrast, the verifier V checks the tracking results and corrects T when needed. The key innovation is that, V does not work on every frame but only upon the requests from T; on the other end, T may adjust the tracking according to the feedback from V. With such collaboration, PTAV enjoys both the high efficiency provided by T and the strong discriminative power by V. In our extensive experiments on popular benchmarks including OTB2013, OTB2015, TC128 and UAV20L, PTAV achieves the best tracking accuracy among all real-time trackers, and in fact performs even better than many deep learning based solutions. Moreover, as a general framework, PTAV is very flexible and has great rooms for improvement and generalization.
1. Introduction
Visual tracking has advanced mainly in speed or accuracy, leaving real-time high-quality tracking scarce. PTAV addresses this gap by combining fast tracking with occasional verification in parallel, achieving strong benchmark performance and remaining flexible for extension.
- 1.1. Background: Deep learning-based trackers offer strong discriminative power but often suffer high computational burden and cannot run in real time.
- 1.1. Background: Correlation-filter trackers run efficiently in real time but are usually less robust than deep learning-based approaches.
- 1.2. Motivation: PTAV decomposes tracking into two parallel collaborative tasks: fast tracking and accurate verification.
- 1.2. Motivation: The tracker processes frames continuously, while the verifier checks results only when requested and corrects unreliable tracking when needed.In a typical sequence, verification occurs every 10 frames and correction can use a detection result.
- 1.3. Contribution: PTAV combines a fast correlation-filter tracker, a deep-learning verifier, and coordination between them to support real-time high-accuracy tracking.
- 1.3. Contribution: PTAV achieves the best tracking accuracy among real-time trackers across OTB2013, OTB2015, TC128, and UAV20L, outperforming many deep learning-based solutions.The implementation is described as flexible, with substantial room for future improvement and generalization.
2. Related Work
Prior visual tracking work includes discriminative, generative, deep-feature, and correlation-filter approaches. PTAV differs from earlier verification methods by applying strong verification only on sampled frames rather than every frame.
- Related tracking algorithms: Existing model-free trackers are commonly categorized as discriminative or generative methods.
- Related tracking algorithms: Discriminative methods distinguish the target from changing background, whereas generative methods search for regions similar to the target.
- Related tracking algorithms: Deep features improve object appearance modeling and robustness, but deep Siamese trackers face heavy computation when extracting features in every frame.
- Related tracking algorithms: Correlation filters became popular because of their high computational efficiency, with MOSSE reported to run at hundreds of fps.
- Verification in tracking: Earlier verification approaches, including TLD, validate tracking results per frame to guide learning or detection.
- Verification in tracking: PTAV runs verification only on sampled frames, allowing strong Siamese-network verification without the same real-time efficiency concern.
3. Parallel Tracking and Verifying (PTAV)
PTAV separates tracking and verification into parallel threads: a fast tracker processes every frame, while a verifier checks selected results and corrects unreliable tracking. Its implementation combines fDSST tracking with Siamese-network verification and local-region detection.
- 3.1. Framework: PTAV uses parallel tracker T and verifier V threads to pursue real-time and high-accuracy tracking.
- 3.1. Framework: T processes every frame, sends verification requests intermittently, and uses verifier feedback to adjust tracking or update models.
- 3.1. Framework: V validates requested tracking results asynchronously, detects unreliable targets, and sends correction information back to T.
- 3.2.1 Tracking: The implementation chooses fDSST as the base tracker and uses a Siamese network verifier with region pooling for simultaneous candidate processing.
- 3.2.1 Tracking: The tracker uses correlation-filter operations accelerated by FFT, with scale estimation, PCA feature reduction, and stored intermediate results for fast tracing back.
- 3.2.2 Verifying: When verification fails, detection searches sliding-window candidates in a local region, selects the highest-scoring candidate, and expands or repeats the search if needed.
4. Experiments
PTAV is evaluated across four tracking benchmarks and multiple experimental settings, combining real-time operation with strong accuracy. It performs favorably against state-of-the-art trackers and supports flexible choices of verifier, threading, and base tracker.
- Overall performance: 84.9% DPR and 77.6% OSR on OTB2015, with PTAV running at 27 fps versus HCF at 10 fps.PTAV also improves over SINT from 77.3% to 84.9% DPR and from 70.3% to 77.6% OSR.
- Attribute-based evaluation: PTAV achieves the best DPR on 8 of 11 OTB2015 attributes and the best OSR on all 11 attributes.For fast motion, in-plane rotation, and scale variation, its DPR remains competitive.
- Qualitative evaluation: PTAV locates targets more reliably than competing trackers across qualitative sequences involving occlusion, deformation, rotation, and motion blur.Its verifier can detect short drifts and recover the correct target, while its correlation-filter tracker handles motion blur effectively.
- TC128: 74.1% DPR and 54.4% OSR on TC128 improve over the state-of-the-art by 3.6% and 4.7%, respectively.PTAV runs at 21 frames per second on the 128-video dataset.
- UAV20L: PTAV reaches 62.4% distance precision and 42.3% success overlap on UAV20L, outperforming the second-best approach by 11% and 8%.It runs at 25 frames per second on this dataset.
- Design choices: More frequent verification increases computation, whereas less frequent verification risks delayed correction when target appearance changes quickly.The experiments also compare parallel and single-thread execution and alternative base trackers within PTAV.
5. Conclusion
The paper concludes that PTAV decomposes tracking into parallel tracking and verifying tasks to combine real-time efficiency with high accuracy. Extensive experiments support its performance among real-time trackers, while its flexible design permits further improvement and generalization.
- Conclusion: PTAV decomposes object tracking into two sub-tasks, tracking and verifying, assigned to collaborative parallel threads.The framework allows the two tasks to work together rather than operating as independent parallel components.
- Conclusion: Across four popular benchmarks, PTAV achieves the best known tracking accuracy among real-time tracking algorithms.The paper reports that it also performs better than many deep learning-based solutions.
- Conclusion: PTAV is presented as a flexible framework with room for improvement and generalization, intended to stimulate more efficient tracking algorithms.