PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepTrack: Learning Discriminative Feature Representations Online for Robust Visual Tracking

Hanxi Li, Yi Li, Fatih Porikli

arXiv:1503.00072v1cs.CV

TL;DR

DeepTrack addresses the difficulty of using deep networks for online visual tracking by learning target-specific features with a single CNN trained purely online. It combines structural loss with tracking-tailored stochastic optimization and outperforms state-of-the-art methods on two benchmarks containing over 60 video sequences.

  • Problem

    Deep neural networks are considered impractical for online visual tracking because they require long training times and many training samples.

  • Method

    DeepTrack uses a single CNN with structural loss, temporal sample selection, and iterative online stochastic gradient updates to distinguish target objects from background.

  • Results

    DeepTrack outperforms state-of-the-art methods on two benchmarks containing over 60 video sequences; on CVPR2013, it achieves 83% TP accuracy versus 77% for TGPR and 63% TSR accuracy versus 55% for SCM.

  • Takeaways & Limitations

    The results support online CNN feature learning as an effective approach for robust visual tracking while achieving tracking speeds up to 4fps.

Abstract

from arXiv · show

Deep neural networks, albeit their great success on feature learning in various computer vision tasks, are usually considered as impractical for online visual tracking because they require very long training time and a large number of training samples. In this work, we present an efficient and very robust tracking algorithm using a single Convolutional Neural Network (CNN) for learning effective feature representations of the target object, in a purely online manner. Our contributions are multifold: First, we introduce a novel truncated structural loss function that maintains as many training samples as possible and reduces the risk of tracking error accumulation. Second, we enhance the ordinary Stochastic Gradient Descent approach in CNN training with a robust sample selection mechanism. The sampling mechanism randomly generates positive and negative samples from different temporal distributions, which are generated by taking the temporal relations and label noise into account. Finally, a lazy yet effective updating scheme is designed for CNN training. Equipped with this novel updating algorithm, the CNN model is robust to some long-existing difficulties in visual tracking such as occlusion or incorrect detections, without loss of the effective adaption for significant appearance changes. In the experiment, our CNN tracker outperforms all compared state-of-the-art methods on two recently proposed benchmarks which in total involve over 60 video sequences. The remarkable performance improvement over the existing trackers illustrates the superiority of the feature representations which are learned

I. INTRODUCTION

DeepTrack addresses the difficulty of applying CNNs to online visual tracking by learning target-specific representations purely online. It combines structural loss, tracking-tailored sampling and lazy updates, and reports strong benchmark performance at practical speeds.

  • Motivation: Hand-crafted tracker features can fail under changing conditions, motivating learned representations for robust visual tracking.Normalized cross correlation, for example, may become ineffective when an object moves under shadow.
  • Motivation: CNN adoption for online tracking is constrained by scarce positive samples, overfitting to recent observations, and computationally intensive training.These difficulties contributed to CNNs being treated primarily as offline feature learners for tracking.
  • Approach: DeepTrack uses a four-layer CNN in a tracking-by-detection framework, learned solely from video frames of the target without offline training.The network scores candidate object states in each frame, and the highest-scoring hypothesis becomes the prediction.
  • Approach: The tracker modifies SGD with temporal sampling that uses longer positive than negative temporal ranges and accounts for unavoidable label noise.The mechanism stores observed patches in a training pool and is reported to regularize training for an effective appearance model.
  • Approach: A lazy updating scheme updates the CNN only after significant appearance changes and updates the fusion layer by coordinate descent with a lower learning rate.The authors report that lazy updating increases tracking speed and yields an observable accuracy increase.
  • Results: On two benchmarks containing over 60 videos, DeepTrack outperforms compared state-of-the-art methods and achieves 1.5fps–4fps depending on sequence and settings.The paper also reports that it rarely loses track and that this speed is comparable to many other visual trackers.

II. CNN ARCHITECTURE

The CNN tracker combines multiple image cues through independent channels and a learned fusion layer. Its compact architecture uses normalized image patches, convolutional feature extraction, fully connected scoring, and cue-specific training.

  • B. Network structure: The network processes locally normalized 32 × 32 patches through two convolutional and two fully connected layers.The stated architecture is (32×32) → (10×10×12) → (2×2×18) → (8) → (2).
  • B. Network structure: The fully connected layers map a 72-D representation to an 8-D vector and then to a 2-D confidence vector containing positive and negative scores.The input patch is denoted x_n and the CNN parameters are denoted Ω.
  • A. CNN with multiple image cues: The tracker uses multiple image cues, including normalized grayscale images and gradients, as independent CNN channels to improve generalization across image conditions.For color images, grayscale cues are replaced with H and V channels from HSV representation.
  • A. CNN with multiple image cues: The final fusion layer concatenates responses from three cues and learns a 2-D output used to calculate the CNN score.The fusion design is depicted as a linear mapping R24 → R2.
  • A. CNN with multiple image cues: Cue normalization rescales pixel values to [0, 10], balancing the importance of different image cues.The paper identifies this normalization as crucial for cue balancing.

B. Structural and truncated loss function

The tracker structurizes binary classification by weighting samples according to their overlap with the estimated target state, allowing all sampled patches to contribute to localization. The CNN crops features from candidate motion states and uses these weighted samples to learn object scores.

  • Structural loss: Unlike ordinary binary classification, structural learning uses all sampled patches rather than retaining only high-confidence examples.This provides a larger training set, which is important for CNN training.
  • Structural loss: The structural loss uses candidate motion states and their cropped frame features to train the CNN for object localization.The motion vector determines each candidate’s location and transformation, while φ crops the corresponding features from the current frame.
  • Structural loss: The overlap score Θ compares each candidate region with the estimated target region using intersection and union operations.The region r(y) is defined by motion state y, and Θ is used to construct the sample weight.
  • Structural loss: The weight Δ(y_n, y*) measures the importance of each training patch according to its overlap with the estimated target state.Patches near the object center and patches reasonably far from it can play more significant roles than intermediate patches.
  • Tracking inference: During tracking, the hypothesis with the highest CNN score is selected as the object’s predicted state in the current frame.The current frame supplies the test patches from which candidate states are evaluated.

2) Truncated structural loss:

The truncated structural loss accelerates online CNN training by ignoring low-error samples while preserving the greater sensitivity of tracking to positive-sample errors. Positive and negative samples therefore receive different truncation treatment.

  • Truncated loss: The truncated l2 norm approximates the loss by counting only patches whose prediction errors exceed a threshold.Small-error patches contribute little to back propagation and can be ignored to speed training.
  • Truncated loss: The truncated loss was previously observed to increase training speed, but with reduced prediction accuracy.The present formulation addresses the unequal sensitivity of positive and negative prediction errors.
  • Truncated loss: The tracker treats positive-sample errors as more consequential than negative-sample errors during prediction.Ignoring low-error negative samples is less harmful because the tracker selects among high-scoring candidates, whereas missed positive samples can affect localization.
  • Truncated loss: The truncated positive and negative losses are represented separately from the original l2 loss in Fig. 2.The dashed green curve is the original loss, while the red and blue curves correspond to positive and negative samples.

III. OPTIMIZATION OF CNN FOR TRACKING

CNN optimization is adapted to tracking’s growing, temporally ordered sample pool. The method samples long-term positives and short-term negatives from different temporal distributions to learn a consistent object model.

  • Temporal sampling: The tracker uses SGD with a temporal sampling process tailored to online visual tracking.Tracking requires learning from accumulated past frames while distinguishing the object from the current background.
  • Temporal sampling: Positive samples are drawn from a long-term pool, whereas negative samples are drawn from a short-term pool.This reflects the goal of maintaining object consistency while adapting to the current background.
  • Temporal sampling: The positive and negative sampling probabilities are normalized distributions, with σ = 10 used in the implementation.The mechanism assigns different temporal sampling behavior to the two classes.
  • Temporal sampling: Unlike multiple-lifespan sampling, this method learns one discriminative model from two sampling distributions rather than three separate codebooks.The comparison concerns the organization of sampling and learned representations.

2) Robust Temporal Sampling with Label Noise:

Prediction uncertainty is modeled as label noise because later estimated states can be false positives, while only the initial manually defined state is always reliable. Robust temporal sampling uses prediction quality to reduce selection of contaminated samples.

  • Prediction uncertainty: Only the initial manually defined target state is always reliable; later motion states are estimated and therefore uncertain.This uncertainty makes accurate labels unavailable in every frame.
  • Prediction uncertainty: The previous positive-sample-bag approach could reduce learning effectiveness when distinct samples share one bag.The same issue affects other MIL-based trackers according to the passage.
  • Label noise: The proposed approach casts prediction uncertainty as label noise, allowing some positive and negative labels to be treated as contaminated.An extra variable η represents whether a sampled label is true or false.
  • Label noise: The sampling process jointly considers sample identity, frame time, and label validity.The joint probability space combines n, t′, and η, with conditional label validity estimated from temporal and sample information.
  • Robust sampling: Prediction quality is estimated from the current frame and used to select high-scoring samples for temporal sampling.The method assumes equal conditional probabilities across samples in the same frame when computing this quality.
  • Robust sampling: Widely distributed peaks in a detection heat-map indicate low tracking quality, so samples from occluded or appearance-change frames are rarely selected.The quality measure drops dramatically under these conditions.

3) Iterative Stochastic Gradient Descent (IT-SGD):

IT-SGD trains the multi-cue CNN iteratively across cue-specific filters and the fusion layer, using robust temporal sampling to reduce overfitting. The algorithm selects samples, generates labels, estimates quality, and updates parameters under a bounded training procedure.

  • Iterative optimization: IT-SGD iteratively trains cue-specific filters and the fusion layer rather than optimizing the full multi-cue network simultaneously.The cue filters are updated with SGD, while the fusion layer is updated on dimensions corresponding to the currently trained cue.
  • Inputs and sample preparation: The procedure takes frame images, positive and negative sample pools, the previous CNN model, an estimated object state, learning rates, a loss threshold, and a training-step budget.It selects training samples and generates their labels from the estimated object state before optimization.
  • Robust sampling: Robust temporal sampling stores observed patches and samples training instances from positive and negative pools according to temporal and conditional probabilities.This mechanism is designed to curb overfitting in online tracking.
  • Parameter updates: The fusion layer is updated partially for the current cue with a lower learning rate, while cue-specific parameters are updated using SGD.This coordinate-descent variation continues across cues and the fusion layer until the loss criterion or training budget is reached.
  • Quality estimation: Prediction quality is estimated from CNN scores and high-score particle distributions without using ground-truth information.Widely distributed peaks indicate lower detection quality, so samples from contaminated frames are rarely selected.

B. Lazy Update and the Overall Work Flow

The tracker uses lazy CNN updates to avoid unnecessary computation while retaining adaptation to appearance changes. It updates only when loss indicates a need for retraining and adjusts the fusion layer conservatively.

  • Lazy model updates: Lazy updating avoids training on frames where the current appearance model remains sufficiently discriminative.Training begins only when the first loss L1 exceeds 2ε and continues until the loss falls below ε.
  • Fusion-layer update: The fusion layer is updated in a coordinate-descent manner with a small learning rate because cue contribution ratios are assumed more stable than feature representations.This separates fast feature adaptation from slower adjustment of cue contributions.
  • Overall workflow: The overall workflow combines testing, prediction-quality estimation, sample storage, sample selection, and conditional CNN training.The scheme is intended to accelerate tracking when object appearance changes only intermittently.

IV. EXPERIMENTS

DeepTrack is evaluated on two benchmarks covering more than 60 sequences and varied tracking difficulties. Across CVPR2013 evaluations, it reports stronger average performance and broad robustness across thresholds and attributes.

  • Evaluation setup: The evaluation uses the CVPR2013 and VOT2013 benchmarks, which together contain more than 60 sequences spanning challenging visual conditions.The reported conditions include scale and illumination changes, occlusion, cluttered backgrounds, and motion blur.
  • Fixed-threshold results: 6% higher TP and 8% higher TSR are reported for DeepTrack than the best reported literature baselines, reaching 83% TP and 63% TSR.The corresponding baselines are TGPR for TP at 77% and SCM for TSR at 55%.
  • Robustness across thresholds: Around 90% accuracy is reported when τo < 0.3 and τd > 30, reflecting strong performance under loose localization thresholds.Under tight thresholds, the tracker has a response similar to the other tested trackers.
  • Attribute analysis: DeepTrack outperforms competitors for all 11 listed tracking difficulties except out-of-view.The evaluated attributes include fast motion, background clutter, blur, deformation, illumination variation, rotations, low resolution, occlusion, and scale variation.

C. Comparison results on the VOT2013 benchmark

On VOT2013, DeepTrack ranks first on average for both accuracy and robustness across the benchmark’s experimental settings. Compared with TPGR, it achieves similar accuracy while requiring substantially fewer reinitializations.

  • VOT2013 evaluation: The VOT2013 benchmark evaluates tracking accuracy and robustness on 16 fully annotated sequences in realistic scenes.Accuracy averages overlap ratios over valid frames, while robustness concerns tracking failures.
  • Benchmark results: DeepTrack ranks first on average for both accuracy and robustness and achieves the best robustness scores across all experimental scenarios.It ranks second in accuracy for all reported experimental settings.
  • Interpretation of scores: The VOT2013 scores are rank-based, so accuracy comparisons between trackers with different robustness scores are not directly fair.The paper presents the benchmark as providing another perspective on DeepTrack’s performance.
  • Comparison with TPGR: DeepTrack achieves similar accuracy to TPGR while requiring only about one third as many reinitializations.For Experiment-1, accuracies are 0.62 versus 0.64 with 0.22 versus 0.71 reinitializations per sequence; for Experiment-2, accuracies are 0.59 versus 0.58 with 0.22 versus 0.73 reinitializations.

D. Verification for the structural loss and the robust temporal sampling

Ablation experiments show that the structural loss, temporal sampling, and label uncertainty each contribute to DeepTrack’s performance, with temporal sampling especially important. The truncated loss also improves tracking efficiency while retaining practical speed comparable to other trackers.

  • Ablation results: All three proposed components contribute to DeepTrack’s success, and removing any one consistently lowers the performance curve.The evaluated components are structural loss, temporal sampling, and label uncertainty.
  • Ablation results: Temporal sampling contributes more than the other ablated components in the verification experiments.
  • Ablation results: 10% TP accuracy is gained from the structural loss, while accounting for label noise lifts TP accuracy by 4%.
  • Tracking speed analysis: 37% tracking-efficiency improvement is achieved with the truncated loss compared with DeepTrack without it.The comparison uses average speed scores over sequences and over frames.
  • Tracking speed analysis: DeepTrack tracks objects at around 2.5fps, comparable to TPGR’s around 3fps and faster than typical Sparse Representation methods below 2.5fps.
Loading 1503.00072v1…