Source-linked AI summary
Tracking as Online Decision-Making: Learning a Policy from Streaming Videos with Reinforcement Learning
James Steven Supancic III, Deva Ramanan
TL;DR
Tracking has limited annotated video data, while online decisions about where to search, when to reinitialize, and whether to update appearance are often hand-designed. The paper learns these decisions as a POMDP policy using reinforcement learning and sparse feedback. It reports training on datasets 100× larger than prior work and an integrated stream for training and evaluation.
Problem
Tracking has limited annotated video data, and crucial online decisions about reinitialization and appearance updates are typically hand-designed.
Source: §1 · p. 1Method
Results
Binary rewards when tracking fails support training on video datasets 100× larger than prior work, with Internet videos also used for integrated streaming evaluation.
Source: §1 · pp. 1–2Takeaways & Limitations
The framework learns policies for different tracking scenarios, including short-term and long-term tracking, and offline heuristics can aid policy learning and regularize Q-learning.
Source: §6.2 · p. 9
Abstract
from arXiv · showhide
We formulate tracking as an online decision-making process, where a tracking agent must follow an object despite ambiguous image frames and a limited computational budget. Crucially, the agent must decide where to look in the upcoming frames, when to reinitialize because it believes the target has been lost, and when to update its appearance model for the tracked object. Such decisions are typically made heuristically. Instead, we propose to learn an optimal decision-making policy by formulating tracking as a partially observable decision-making process (POMDP). We learn policies with deep reinforcement learning algorithms that need supervision (a reward signal) only when the track has gone awry. We demonstrate that sparse rewards allow us to quickly train on massive datasets, several orders of magnitude more than past work. Interestingly, by treating the data source of Internet videos as unlimited streams, we both learn and evaluate our trackers in a single, unified computational stream.
1. Introduction
Tracking must operate with limited annotated video and make online update and reinitialization decisions that are usually hand-designed. The paper proposes learning those decisions with reinforcement learning, using sparse feedback and treating Internet videos as a streaming source for training and evaluation.
1. Introduction
- The motivating challenges are scarce annotated tracking videos and online update or reinitialization decisions that are crucial to performance but typically hand-designed.Tracking datasets have hundreds of videos, unlike image datasets with millions of images.Source: §1 · p. 1
- Binary rewards given when a tracker fails support training on video datasets 100× larger than prior work, with Internet videos also used for integrated streaming evaluation.The stream can evaluate the current tracker and refine it for future use.Source: §1 · pp. 1–2
- The tracker is modeled as an active agent choosing whether to search locally or globally and whether to update its appearance model.These choices direct limited computation and are difficult when image evidence is ambiguous, such as during partial occlusion.Source: §1 · p. 2
- Because the agent receives partial image observations, the paper formulates tracking as a POMDP and maintains beliefs about object location and appearance over time.Actions specify how and when those beliefs are updated; frame-based heuristics provide clues about long-term action effects during policy learning.Source: §1 · p. 2
2. Related Work
Related work addresses tracking benchmarks, interactive annotation, long-term tracking, and computationally efficient attention. The paper distinguishes its approach through open-world streaming evaluation, binary reward feedback, and learned policies for both reinitialization and appearance updates.
Learning-to-track
- Figure 2 illustrates the two-sided update trade-off: updating can help through occlusion, while updating after occlusion can cause drift.The paper recasts these decisions as sequential decision-making learned with reinforcement learning.Source: Figure 2 · p. 2
Tracking datasets
- Open-world evaluation on Internet videos is proposed to make overfitting impossible by design, addressing concerns about aggressive tuning on established benchmarks.The paper also notes that closed-world training can be difficult to generalize from to in-the-wild footage.Source: §2 · p. 2
Interactive tracking
- Prior interactive tracking methods have humans correct major errors with annotations, whereas this approach uses binary rewards instead of bounding boxes.Table 1 contrasts the paper’s binary rewards with other datasets’ axis-aligned or rotated bounding boxes.Source: §2 · p. 2, Table 1 · p. 3
Learning-to-track
- For long-term tracking, the method learns model-update and reinitialization policies rather than relying on hand-designed policies used by trackers such as TLD, ALIAN, and SPL.The paper reports that explicit Q-learning outperforms supervised reductions because it can learn long-term action effects.Source: §2 · pp. 2–3
Real-time tracking through attention
- Unlike selective-search policies that choose between local tracking and global reinitialization, this method also learns when to update appearance while keeping a fixed computational budget.Reinitialization searches a random subset of positions equal in number to those examined during tracking.Source: §2 · p. 3
3. POMDP Tracking
The tracker is modeled as an agent that maintains beliefs about object location and appearance from partial image observations. A learned policy chooses how to update those beliefs, including whether to track locally, reinitialize, or update appearance.
3. POMDP Tracking
- The tracker maintains an appearance distribution θi and a spatial heatmap hi, updating both from the previous belief and current image observation.The agent’s actions determine whether it updates appearance and whether it discards previous heatmaps to reinitialize.Source: §3 · p. 3
- The agent can track near its prior location or globally reinitialize, while separately choosing whether to update its appearance model.The state includes the previous belief and current image observation.Source: §3 · pp. 3–4
- Offline heuristics use ground-truth annotations and future frames to generate per-frame action labels for supervised policy learning.These heuristics cannot be applied at test time because ground truth is unavailable.Source: §3 · p. 4
- Q-functions score candidate actions by expected future reward, and CNNs approximate them for continuous belief states.The reward is 1 for a correct heatmap prediction and 0 otherwise; the discount factor trades off immediate and future rewards.Source: §3 · p. 4
4. Interactive Training and Evaluation
The training procedure gathers state-action-reward-next-state experience from streaming videos and uses Q-learning to update the action policy. Annotators mark failures with binary labels, allowing each video to evaluate the current tracker and train it for future videos.
4. Interactive Training and Evaluation
- Q-learning updates policy parameters by regressing on replayed state-action-reward-next-state samples and their one-step-ahead targets.The training algorithm samples minibatches from experience replay and updates the policy with backpropagation.Source: §4 · pp. 4–5
- Annotators mark incorrect frames at 50-frame strides using a 50% intersection-over-union threshold, providing binary correctness rewards for policy training.The interface averages 1200 frames per minute, compared with 34 frames per minute for bounding-box annotation.Source: §4 · pp. 4–5
- 1200 frames per minute versus 34 for bounding boxes; annotating 10 million frames takes under two days rather than months.The annotation speed and labor comparison are reported for the interactive labeling setup.Source: §4 · p. 5
- Each video is used to evaluate the current tracker and train its policy for future videos.Failure annotations supply rewards, and the resulting experience tuples enter replay memory for policy updates.Source: §4 · p. 5
5. Implementation
The implementation combines tracking and reinitialization modules with learned action-value functions for two binary decisions. Offline heuristics initialize those functions and define when to track or update appearance.
5. Implementation
- TRACK crops and processes a region around the prior heatmap maximum, while REINIT processes a random region when the tracker believes it has lost the target.The paper uses FCNT or CCOT for tracking and focuses on CCOT in its experiments.Source: §5 · p. 5
- CCOT updates appearance with multi-resolution convolutional templates fitted by conjugate gradients; FCNT uses a two-layer template updated through ten gradient-descent iterations.Both approaches use positive patches from the heatmap maximum and negative patches from nearby regions with less than 30% overlap.Source: §5 · p. 6
- Two Q-functions separately score TRACK/REINIT and UPDATE/IGNORE, sharing convolutional layers before independent fully connected output layers.The Q-functions condition on the previous heatmap and predict expected future reward for candidate actions.Source: §5 · p. 6, Figure 5 · p. 6
- Offline heuristics initialize Q-values, assigning the heuristic action discounted goodness and other actions zero; training also minimizes a weighted supervised loss.The TRACK heuristic chooses tracking when the heatmap peak overlaps the ground-truth object; the UPDATE heuristic uses future-frame confidence changes.Source: §5 · p. 6
6. Experiments
The learned tracker performs strongly on short-term benchmarks and held-out long-term Internet videos, while diagnostics show gains from learned action policies and large-scale training.
6.1. Comparative Evaluation
- p-track-short significantly outperforms prior work on OTB-2013 and VOT-2016, with state-of-the-art accuracy and greater robustness on VOT-2016.For VOT-2016, robustness is ranked by failures and accuracy by average overlap with ground truth.Source: §6.1 · p. 6, Figure 7 · p. 7, Figure 9 · p. 7
- Short-term policies improve benchmark performance because long-term policies select IGNORE more often, an action suited to the more frequent occlusions in long videos.The short-term policy is trained using the 40 most dissimilar videos from the other benchmark, with similarity measured by correlation between average ground-truth object images in RGB space.Source: §6.1 · p. 6, Figure 7 · p. 7
- Internet videos include cuts, fast motion, complex illumination, and distractors, while standard benchmarks often contain easy videos and rankings can hinge on a few challenging sequences.The paper identifies cuts as a scalable proxy for occlusions.Source: §6.1 · p. 7, Figure 6 · p. 7, Figure 10 · p. 8
- On a held-out test set of 16 long-term Internet videos, the learned update and reinitialization policy produces a state-of-the-art tracker.Each test video has at least 5,000 frames and was never used for training.Source: Figure 11 · p. 8, §6.1 · p. 8
6.2. System Diagnostics
- After 16,000 Internet videos, average hold-out F1 rises from .08 at initialization to .30, and training appeared to converge after 8,000–12,000 videos.The authors attribute good decision policies to large-scale training made possible through interactive annotation.Source: Figure 13 · p. 9, §6.2 · p. 9
- Offline action heuristics raise F1 from .09 for FCNT to .20 when both tracking and update heuristics are used.The track-only and update-only heuristics reach F1 .13 and .14, respectively.Source: §6.2 · p. 9
- Q-learning refines the heuristics to F1 .30, tripling the default online heuristic baseline from 9% to 30%.Learning appearance updates alone reaches F1 .28; during partial occlusions, the policy accepts some failures to avoid reinitialization costs and risks.Source: §6.2 · p. 9
- The tracker runs at approximately 10 fps on a Tesla K40 GPU, and the conclusion identifies offline heuristics as effective policy-learning tools and Q-learning regularizers.The learned policy balances tracking within a limited region against reinitialization through search.Source: §6.2 · p. 9