Source-linked AI summary
Need for Speed: A Benchmark for Higher Frame Rate Object Tracking
Hamed Kiani Galoogahi, Ashton Fagg, Chen Huang, Deva Ramanan, Simon Lucey
TL;DR
Visual tracking research has focused on approximately 30 FPS, despite widespread 240 FPS cameras and uncertainty about how frame rate should affect tracker choice. The paper introduces the NfS dataset and benchmark to evaluate CF and deep trackers across frame rates, finding that simple CF trackers can outperform complex deep trackers at higher frame rates. This supports explicitly trading capture rate, computation, and accuracy in resource-constrained tracking systems.
Problem
Existing tracking algorithms and datasets largely target approximately 30 FPS, leaving higher-frame-rate tracking and its algorithmic tradeoffs insufficiently explored.
Method
The paper introduces NfS, the first higher-frame-rate general object-tracking dataset and benchmark, and evaluates state-of-the-art CF and deep trackers at 30 FPS and 240 FPS.
Results
At higher frame rates, simple CF trackers trained on hand-crafted features such as HOG outperform complex deep trackers in tracking accuracy and computational efficiency.
Takeaways & Limitations
Capture frame rate should be treated as a system resource alongside hardware and explicitly traded against bandwidth, computational cost, and required tracking accuracy.
Abstract
from arXiv · showhide
In this paper, we propose the first higher frame rate video dataset (called Need for Speed - NfS) and benchmark for visual object tracking. The dataset consists of 100 videos (380K frames) captured with now commonly available higher frame rate (240 FPS) cameras from real world scenarios. All frames are annotated with axis aligned bounding boxes and all sequences are manually labelled with nine visual attributes - such as occlusion, fast motion, background clutter, etc. Our benchmark provides an extensive evaluation of many recent and state-of-the-art trackers on higher frame rate sequences. We ranked each of these trackers according to their tracking accuracy and real-time performance. One of our surprising conclusions is that at higher frame rates, simple trackers such as correlation filters outperform complex methods based on deep networks. This suggests that for practical applications (such as in robotics or embedded vision), one needs to carefully tradeoff bandwidth constraints associated with higher frame rate acquisition, computational costs of real-time analysis, and the required application accuracy. Our dataset and benchmark allows for the first time (to our knowledge) systematic exploration of such issues, and will be made available to allow for further research in this space.
1. Introduction
Visual tracking has largely been designed and evaluated for 30 FPS, despite the spread of 240 FPS cameras. The paper explores whether higher capture rates can support simpler trackers while balancing accuracy and computational cost.
- Motivation: Most tracking algorithms and datasets target approximately 30 FPS, while consumer devices increasingly provide 240 FPS cameras.This creates a mismatch between established evaluation settings and changing real-time acquisition capabilities.
- Tracking Approaches: Correlation filters offer efficient online adaptation and can operate at hundreds of frames per second on embedded devices.Their computational efficiency comes partly from Fourier-domain operations and low-capacity online learning.
- Tracking Approaches: Deep trackers improve robustness to appearance variation but require expensive CNN feature extraction or tracking computation, sometimes demanding high-end GPUs.Training may also require large amounts of data because these methods use high-capacity learners.
- Motivation: Appearance variation between adjacent frames is a central determinant of tracking performance and is reduced by increasing capture frame rate.The paper uses this premise to investigate whether simpler algorithms can replace computationally expensive deep trackers in some settings.
- Contributions: The NfS benchmark evaluates state-of-the-art CF and deep trackers on higher frame rate video and examines accuracy alongside computational efficiency.It is presented as the first benchmark, to the authors’ knowledge, for higher frame rate general object tracking using consumer devices.
- Contributions: At sufficiently high frame rates, CFs using cheap hand-crafted features such as HOG can outperform state-of-the-art deep trackers in accuracy and computational efficiency.The paper frames this as a tradeoff between tracker capacity and capture frame rate, particularly for resource-constrained devices.
2. Related Work
Earlier tracking datasets primarily use low-frame-rate videos, while NfS introduces a 240 FPS alternative. Related trackers fall mainly into correlation-filter and deep-learning categories with different accuracy, adaptation, and computational tradeoffs.
- Tracking Datasets: NfS is the only compared dataset with 240 FPS videos and contains 380K frames, more than twice the size of ALOV300.Table 1 compares NfS with the other object-tracking datasets.
- Tracking Methods: Recent trackers are broadly divided into correlation-filter trackers and deep trackers.The paper reviews both categories before evaluating them on NfS.
- Tracking Methods: Correlation filters gained traction through MOSSE’s approximately 700 FPS speed and online adaptation, with later methods improving accuracy using features such as HOG.KCF improved MOSSE through real-time kernelized learning on HOG features.
- Tracking Methods: Deep-feature CF trackers can achieve stronger results on standard datasets but may run at approximately 0.2 FPS on CPUs because of their complexity.Their motivation is that hand-crafted features may not capture appearance differences between consecutive low-frame-rate frames.
- Tracking Methods: Deep trackers learn target-specific representations, but fine-tuning can be computationally expensive; fixed-model methods such as GOTURN and SFC trade adaptation ability for speed.GOTURN and SFC are reported to run at 100 FPS and 75 FPS, respectively, on GPUs.
3. NfS Dataset
NfS is a 240 FPS tracking dataset combining consumer-device footage with dense bounding-box and attribute annotations. It covers diverse object categories and nine visual tracking conditions.
- Collection: NfS contains 100 higher frame rate videos captured at 240 FPS, including 75 recordings from iPhone or iPad devices and 25 YouTube sequences.The 75 captured videos also include raw IMU and gyroscope data, released for potential future applications but unused in this paper.
- Targets and Annotations: The dataset includes vehicles, people, faces, animals, aircraft, boats, and generic objects such as sport balls, cups, and bags.These categories represent the range of tracking targets covered by the benchmark.
- Targets and Annotations: Every NfS frame is annotated with an axis-aligned bounding box using the VATIC toolbox.The annotations provide per-frame target localization for evaluation.
- Targets and Annotations: All videos receive labels for nine visual attributes: occlusion, illumination variation, scale variation, deformation, fast motion, viewpoint change, out of view, background clutter, and low resolution.Table 2 reports the distribution of coincident attributes across videos.
4. Evaluation
The evaluation compares 15 trackers across higher- and lower-frame-rate tracking scenarios using success-based accuracy measures, with lower-rate videos created by temporal sampling. Results show that higher frame rates substantially benefit correlation-filter trackers, while simple CF trackers can outperform deep trackers and offer strong real-time performance under resource constraints.
- Evaluation setup: 15 trackers are evaluated in three classes: CF trackers with hand-crafted features, CF trackers with deep features, and deep trackers.The evaluation uses the NfS dataset and includes trackers such as BACF, SRDCF, Staple, KCF, HDT, MDNet, SiameseFc, FCNT, GOTURN, and MEEM.
- Evaluation setup: Tracking is evaluated with success plots, ranking trackers by AUC and also reporting success rate at IoU > 0.50 and relative accuracy improvement.Relative improvement compares higher-frame-rate accuracy against lower-frame-rate accuracy, where improved accuracy is their difference.
- Tracking scenarios: 240 FPS tracking uses every frame, whereas 30 FPS tracking is modeled by sampling every 8th frame from the high-frame-rate videos.The sampled setting is evaluated with and without synthesized motion blur in the broader comparison.
- Per-tracker comparison: All CF trackers improve their AUCs by more than 10% at 240 FPS, while FCNT improves by 6% and SFC and MDNet improve the least.The paper attributes the CF improvement to smaller adjacent-frame appearance changes that online per-frame adaptation can learn; offline-trained trackers benefit less.
- Overall comparison: For lower-frame-rate tracking, deep trackers outperform CF trackers, whereas at higher frame rates simple CF trackers with hand-crafted features outperform complex deep trackers.The paper connects this reversal to large adjacent-frame variation at lower rates and smaller variation at higher rates, which favors online CF adaptation.
- Overall comparison: 49.56 is BACF’s higher-frame-rate success rate at IoU > 0.50, exceeding HDT (47.80) and remaining competitive with deep trackers including SFC (47.78), MDNet (47.34), and FCNT (46.94).BACF is a CF tracker with HOG features; SRDCF, Staple, DSST, and SAMF also outperform GOTURN (38.65).
- Runtime comparison: 170.4 FPS makes KCF the only tracker listed as almost real-time for higher-frame-rate videos on CPUs, while GOTURN reaches 155.3 FPS and is almost real-time on GPUs.For lower-frame-rate videos, BACF, Staple, KCF, and CFLB are real-time on CPUs, while GOTURN, FCNT, and SFC are real-time on GPUs.
- Attribute-based evaluation: CF trackers with hand-crafted features outperform deep trackers on six attributes, while MDNet leads on scale variation (61.0), deformation (59.2), and view change (55.9).All trackers show less relative improvement for non-rigid deformation, and CF trackers show much higher relative improvement than deep trackers.