Source-linked AI summary
Learning Policies for Adaptive Tracking with Deep Feature Cascades
Chen Huang, Simon Lucey, Deva Ramanan
TL;DR
Visual tracking requires both high accuracy and real-time efficiency, but deep trackers are costly on limited hardware. The paper introduces EAST, which uses reinforcement learning to adaptively stop a deep feature cascade early on easy frames and continue deeper on difficult ones. It reports near-real-time CPU tracking at 23.2 fps, with about 100× speedup on easy frames almost 50% of the time while maintaining strong benchmark accuracy.
Problem
Visual tracking needs high accuracy under strict time and computational constraints, while deep trackers can be too slow for CPUs and mobile or embedded hardware.
Method
EAST learns a reinforcement-learning agent that decides at each feature layer whether to stop with sufficient localization confidence or continue to deeper layers.
Results
23.2 fps is the near-real-time average speed of EAST on a single CPU, while about 100× speedup occurs for easy frames almost 50% of the time.
Takeaways & Limitations
Adaptive early stopping reduces feed-forward cost by using cheap features for easy frames and heavy invariant features only when challenging frames require them.
Abstract
from arXiv · showhide
Visual object tracking is a fundamental and time-critical vision task. Recent years have seen many shallow tracking methods based on real-time pixel-based correlation filters, as well as deep methods that have top performance but need a high-end GPU. In this paper, we learn to improve the speed of deep trackers without losing accuracy. Our fundamental insight is to take an adaptive approach, where easy frames are processed with cheap features (such as pixel values), while challenging frames are processed with invariant but expensive deep features. We formulate the adaptive tracking problem as a decision-making process, and learn an agent to decide whether to locate objects with high confidence on an early layer, or continue processing subsequent layers of a network. This significantly reduces the feed-forward cost for easy frames with distinct or slow-moving objects. We train the agent offline in a reinforcement learning fashion, and further demonstrate that learning all deep layers (so as to provide good features for adaptive tracking) can lead to near real-time average tracking speed of 23 fps on a single CPU while achieving state-of-the-art performance. Perhaps most tellingly, our approach provides a 100X speedup for almost 50% of the time, indicating the power of an adaptive approach.
1. Introduction
Visual tracking must balance accuracy with strict speed and computational constraints, especially on limited hardware. The paper addresses this by adaptively stopping a deep feature cascade early for easy frames and continuing deeper for challenging ones.
- Motivation: Visual object tracking must handle appearance changes while meeting real-time constraints in time-critical systems and on mobile or embedded hardware.Examples of appearance change include scaling and occlusion.
- Motivation: Trackers face a persistent accuracy–speed trade-off: low-level real-time methods may lack descriptive power, while deep trackers lose real-time capacity.Deep trackers that fine-tune networks online can achieve top performance but remain computationally demanding.
- Motivation: Deep trackers that traverse fixed convolutional networks can process only a couple of frames per second on modern CPUs, smartphones, or tablets.Their major computational burden comes from the network forward pass.
- Approach: Adaptive processing uses deep features for large appearance changes but early layers for visually distinct or barely moving objects.Early stopping can substantially reduce computation when simple features suffice.
- Approach: A learned agent decides at each layer whether its localization confidence is sufficient to stop or whether to continue to the next layer.The paper presents this as a timely decision policy for selecting the tracking layer.
- Results: 10× average speedup over baseline deep tracker [3] accompanies higher accuracy on OTB and VOT benchmarks.The adaptive tracker evaluates templates across multiple network layers, including pixels, using correlation filters for lower layers.
- Results: 23.2 fps is the near-real-time average speed of EAST on a single CPU, reaching about 190 fps almost 50% of the time.The paper describes EAST as a CPU-friendly deep tracker among top benchmark performers.
2. Related Work
Prior tracking work includes fast correlation-filter methods, computationally expensive online-learning deep trackers, and fixed-network deep trackers that improve GPU speed but remain slow on weaker hardware. This paper instead uses reinforcement learning to select an early stopping point in a feature cascade.
- Real-time tracking and correlation filters: Correlation-filter trackers achieve high efficiency by exploiting spatial shifts through the discrete Fourier transform.MOSSE and KCF are reported at 669 fps and 292 fps, respectively, on a single CPU.
- Deep trackers: Online-updated deep trackers improve target-specific modeling but suffer from slow speed because online training and exhaustive search are costly.The passage identifies MDNet as a top-performing example with low speed.
- Deep trackers: Fixed convolutional networks enable GPU real-time tracking, yet such deep trackers may process only a couple of frames per second on modern CPUs and mobile devices.Complete forward passes through deep networks are identified as a major source of slowdown.
- Adaptive cascades: The proposed cascade combines hand-crafted and deep features while learning an agent to select the minimum sequence of layers needed for fast tracking.Unlike attentional cascades, the early layers here are intended to make a strong tracking decision as early as possible.
- Reinforcement learning for tracking: This work is presented as the first use of reinforcement learning to learn an early decision policy for speeding up deep tracking.Earlier visual-tracking RL work addressed temporal target attention or template selection instead.
3. Method
EAST treats tracking as sequential decision-making over a CNN feature cascade, adaptively transforming the bounding box and stopping at the earliest sufficiently confident layer. It combines score-map evidence across layers with reinforcement learning to reduce computation while preserving accurate localization.
- Feature cascade: EAST uses CNN layers as a feature cascade, obtaining object confidence from progressively deeper representations when early layers are insufficient.Easy targets may be tracked with only early layers, whereas ambiguous targets can use deeper features such as C5.
- Siamese network: The fully-convolutional Siamese network compares a template with search-region subregions and produces confidence score maps for localization.The relative position of the maximum score determines frame-to-frame translation, while deeper layers provide the score maps used by the policy.
- State and actions: The action space includes global and local scale transformations, a no-scaling action for ambiguous maps, and a stop action for sufficiently confident localization.The no-scaling action lets the agent postpone a decision, while box changes are applied once per layer.
- State and actions: The agent observes current and accumulated score-map evidence plus recent action history, then either transforms the bounding box or stops with a prediction.Averaging current and preceding score maps combines fine-grained early-layer details with deeper semantic information; the history vector records the past four actions.
- Policy learning: Reinforcement learning rewards accuracy-improving transformations and penalizes stopping with IoU below 0.6, while future rewards are estimated by a deep Q-Network.Training also updates convolutional layers receiving rewards, providing direct learning signals for representations at each layer.
4. Results
EAST adaptively stops at feature layers according to frame difficulty, using cheap early features when sufficient and deeper features for harder cases. This improves the speed–accuracy trade-off across tracking benchmarks, including near-real-time CPU operation.
- Adaptive stopping: EAST learns to stop at early feature layers for over 70% of frames, while hard frames continue through additional layers toward a full forward pass.The policy adapts computation to frame difficulty rather than always using the final feature layer.
- Speed: 10× average CPU speedup and 100× speedup for easy frames result from adding cheap pixel and HOG layers to the cascade.Pixel and HOG features support fast correlation-filter processing, while HOG enables more accurate pruning and is selected more often.
- Speed: 23.2 fps on CPU is achieved by full EAST, despite sacrificing some average GPU speed through greater reliance on CPU computations.The CPU rate is described as near real time and practical for CPU-bound tracking on embedded devices.
- Benchmark comparisons: EAST achieves the highest AUC on OTB-50 while remaining faster than competing trackers, including fast correlation trackers and GPU-based deep trackers.The comparison reports higher accuracy than KCF and faster CPU performance than several deep trackers.
- Benchmark comparisons: EAST outperforms all competitors in overall rank on VOT-14 and is the fastest among top performers on VOT-15 at 21 fps on CPU and 148 fps on GPU.The VOT-14 comparison emphasizes the best accuracy–speed trade-off.
5. Conclusion
The paper proposes adaptive tracking with deep feature cascades, using reinforcement learning to process easy frames cheaply and challenging frames with deeper features. This reduces feed-forward cost while retaining strong tracking performance.
- Reinforcement learning enables an agent to decide when early-layer features suffice and when deeper invariant features are needed.
- Easy frames can receive a 100X reduction in feed-forward cost by stopping before expensive deep layers.
- The learned policy proceeds iteratively through the cascade and is trained end-to-end for time-critical deep-network tasks.
A. Algorithmic Details
The tracker uses a SiamFC-based convolutional cascade and trains its action-value function with deep Q-learning. The Q-function is optimized through a loss defined over future rewards and the discount factor.
- Network architecture: The network follows SiamFC, using convolutional layers C1–C5 with maxpooling in C1 and C2 and ReLU after all layers except C5.
- Deep Q-learning: Deep Q-learning models the optimal action-value function Q(S_l, A_l) with a Bellman equation.
- Deep Q-learning: The Q-function uses future reward Q(S′, A′) and discount factor γ when defining the learning objective.
B. Discussions and Results
EAST adaptively processes easy frames with cheap features and invokes deeper layers only when necessary. Across OTB-50 and VOT-14 evaluations, it combines fast tracking with strong accuracy and robustness rankings.
- Discussions and Results: EAST tracks easy frames with early features and reserves invariant but expensive deep layers for difficult frames.
- Discussions and Results: EAST achieves the highest AUC scores across OPE, TRE, and SRE on OTB-50 while maintaining fast tracking speed.
- Discussions and Results: EAST is one of the fastest trackers and outperforms other top performers in the overall VOT-14 rank.
Template Update
The tracker uses fast correlation filters for pixel and HOG layers and fixed convolutional templates for deep layers. Template-update experiments show limited benefits in lower layers and no difference for the top layer C5.
- Template Update: Pixel and HOG layers use fast correlation filters, whose training exploits shifted patches through the discrete Fourier transform.
- Template Update: The correlation-filter objective uses target response values y_i and regularization parameter λ.
- Template Update: Deep-layer similarity compares template and search-image embeddings using ϕ_l, with an offset value v.
- Template Update: Deep convolutional templates are fixed from the first frame and are not updated during tracking.
- Template Update: Updating lower layers C1–C2 yields marginal AUC gains, whereas updating the top layer C5 produces no difference.