PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Tracking Using Learned Hierarchical Features

Li Wang, Ting Liu, Gang Wang, Kap Luk Chan, Qingxiong Yang

arXiv:1511.07940v1cs.CV

TL;DR

Visual tracking needs features robust to complicated motion while adapting to the appearance of a specific target from limited sequence data. The paper learns hierarchical features from auxiliary videos with temporal slowness, then adapts both layers online to the target object. Across integrated tracking methods, the learned features significantly improve baseline tracking performance, especially for complicated motion sequences.

  • Problem

    Visual tracking has limited first-frame supervision, while unrelated-image pretraining lacks temporal invariance and does not explicitly connect offline learning with online tracking.

  • Method

    The method pre-learns generic hierarchical features with a two-layer convolutional network and temporal slowness, then adapts both layers online using target-sequence data.

  • Results

    The learned hierarchical features significantly improve baseline trackers, with especially strong benefits reported for sequences involving complicated motion transformations.

  • Takeaways & Limitations

    The adapted representation is robust to both complicated motion transformations and appearance changes of specific target objects.

Abstract

from arXiv · show

In this paper, we propose an approach to learn hierarchical features for visual object tracking. First, we offline learn features robust to diverse motion patterns from auxiliary video sequences. The hierarchical features are learned via a two-layer convolutional neural network. Embedding the temporal slowness constraint in the stacked architecture makes the learned features robust to complicated motion transformations, which is important for visual object tracking. Then, given a target video sequence, we propose a domain adaptation module to online adapt the pre-learned features according to the specific target object. The adaptation is conducted in both layers of the deep feature learning module so as to include appearance information of the specific target object. As a result, the learned hierarchical features can be robust to both complicated motion transformations and appearance changes of target objects. We integrate our feature learning algorithm into three tracking methods. Experimental results demonstrate that significant improvement can be achieved using our learned hierarchical features, especially on video sequences with complicated motion transformations.

I. INTRODUCTION

The paper targets visual tracking features that must handle complicated motion and target-specific appearance despite limited labeled data. It combines temporally robust hierarchical feature learning with online domain adaptation and reports improved tracking performance.

  • Visual tracking has limited supervision because only the target annotation in the first test frame is available.
  • Prior offline features learned from unrelated images lack temporal invariance, while offline-to-online transfer does not explicitly connect the two stages.
  • The method pre-learns features robust to complicated motion transformations and diverse motion patterns from auxiliary video data.
  • A domain adaptation module updates pre-learned features in both layers to incorporate specific target-object appearance information.

IV. LEARNING FEATURES FOR VIDEO TRACKING

The tracking system learns hierarchical representations for candidate patches and uses them within an ASLA-based tracking pipeline. Feature parameters are updated online while tracking states and exemplars are maintained.

  • The method learns hierarchical features to handle non-rigid deformations and in-plane and out-of-plane rotations that raw pixels or hand-crafted features may not capture.
  • The method is integrated into ASLA, while its feature representation can also replace original representations in other tracking systems.
  • Each tracking iteration applies affine transformations to the previous state, producing candidate states and image patches.
  • Feature representations are extracted from candidate patches under the current feature-learning parameters before posterior probabilities are calculated.
  • The feature-learning parameters and exemplar library are updated every M frames, and the predicted state and updated model are output.
  • The stacked architecture whitens the first-layer output with PCA before using it as input to the second layer.

A. Pre-Learning Generic Features from Auxiliary Videos

The method pre-learns two-layer hierarchical features from auxiliary videos using temporal continuity and reconstruction objectives. The resulting features represent local and more complicated motion patterns for tracking.

  • A. Pre-Learning Generic Features from Auxiliary Videos: Auxiliary-video training learns generic features robust to diverse object motion patterns before tracking begins.The pre-learning is performed offline.
  • A. Pre-Learning Generic Features from Auxiliary Videos: The two-layer model processes 16×16 patches in the first layer and larger 32×32 patches in the second layer.The first-layer mappings are applied to sub-patches, concatenated, whitened, and passed to the second layer.
  • A. Pre-Learning Generic Features from Auxiliary Videos: Each layer learns a feature transformation matrix whose squared responses are pooled into the learned feature representation.The pooling operation sums adjacent intermediate feature dimensions in a non-overlapping manner.
  • A. Pre-Learning Generic Features from Auxiliary Videos: Temporal continuity and auto-encoder reconstruction jointly train features that remain robust to complicated motion transformations.The temporal term links consecutive frames, while the reconstruction term preserves information.
  • A. Pre-Learning Generic Features from Auxiliary Videos: The first layer targets local motion such as translation, while the second layer targets nonlinear warping and out-of-plane rotation.Features from both layers are concatenated as the generic representation.

B. Domain Adaption Module

The domain adaptation module updates generic features using target-object training patches. Adaptation occurs in both layers while regularization preserves robustness to complicated motion.

  • B. Domain Adaption Module: Generic features lack specific target appearance information such as shape and texture, motivating domain adaptation.The module adapts pre-learned features to the target object.
  • B. Domain Adaption Module: Initial target-object training data comes from ASLA tracking results over the first N frames of the target video.These object regions are used to train the adaptation module.
  • B. Domain Adaption Module: The adaptation objective keeps adapted features close to the pre-learned features while incorporating target-object appearance.The trade-off parameter γ controls the adaptation level.
  • B. Domain Adaption Module: The generic features are adapted separately in both layers of the deep feature model.The minimization is performed with respect to each layer's transformation matrix.

C. Optimization and Online Learning

The adaptation objective is optimized with L-BFGS because the parameter space is high-dimensional. Parameters are updated online using recent object-region data to capture appearance changes.

  • C. Optimization and Online Learning: L-BFGS optimizes the adaptation objective with respect to the transformation-matrix parameter set Θ.The objective uses training images of object regions and the pre-learned parameter set.
  • C. Optimization and Online Learning: The search direction is initialized from the negative gradient and refined using stored curvature information.Algorithm 2 specifies the input, update loop, and output for the search direction.
  • C. Optimization and Online Learning: L-BFGS computes search directions from the current gradient and curvature information from the m most recent iterations.The implementation sets m to 5.
  • C. Optimization and Online Learning: L-BFGS updates parameters along the search direction with a step size satisfying Wolfe conditions, reaching convergence after several iterations.The updated curvature information is retained while older information is discarded once the memory limit is exceeded.
  • C. Optimization and Online Learning: Online adaptation learns parameters every M frames from object regions in the M most recent frames to capture target appearance changes.The converged parameters become the old parameters for the next frame group.

D. Implementation Details

The implementation pre-learns generic features from auxiliary tracked-object videos, evaluates them with standard tracking metrics, and compares variants with and without temporal slowness.

  • D. Implementation Details: Generic features are pre-learned from auxiliary video sequences containing tracked objects.The auxiliary data come from Hans van Hateren natural scene videos.
  • D. Implementation Details: The evaluation compares learned-feature tracking with four state-of-the-art trackers using alternative feature representations.The alternatives include raw pixels, HOG, sparse representation, and data-independent features.
  • D. Implementation Details: A variant without the temporal slowness constraint is included to assess the constraint’s contribution.The constrained and unconstrained variants are evaluated on challenging video sequences.
  • D. Implementation Details: The experiments also compare the tracker across multiple state-of-the-art methods and assess generalizability to other tracking methods.The evaluation includes comparisons with 14 state-of-the-art trackers and with DLT.
  • D. Implementation Details: Tracking performance is measured using center location error and overlap rate.Center error measures pixel distance between tracking and ground-truth centers; overlap rate measures region overlap.

A. Evaluation on Our Learned Feature’s Robustness to Complicated Motion Transformations

The learned hierarchical features are evaluated on sequences involving non-rigid deformation, in-plane rotation, and out-of-plane rotation, where baseline trackers often fail and the proposed tracker remains effective.

  • A. Evaluation on Our Learned Feature’s Robustness to Complicated Motion Transformations: The evaluation uses 15 challenging sequences containing non-rigid deformation, in-plane rotation, or out-of-plane rotation.The tracker is compared with methods using raw pixels, HOG, sparse, and data-independent features.
  • A. Evaluation on Our Learned Feature’s Robustness to Complicated Motion Transformations: The qualitative comparisons use colored bounding boxes to distinguish the evaluated trackers’ outputs.The figures present results for non-rigid deformation, in-plane rotation, and out-of-plane rotation sequences.
  • Non-rigid object deformation: On non-rigid deformation sequences, the four baseline trackers fail, whereas the proposed tracker captures targets using deformation-invariant features.Examples include running, acrobatic actions, facial expression changes, surfing, and articulated motion.
  • In-plane rotations: On in-plane rotation sequences, baseline trackers drift away, while the proposed tracker handles these motion transformations effectively.The evaluated sequences include David2, MountainBike, Sylvester, Tiger1, and Tiger2.
  • Out-of-plane rotations: On out-of-plane rotation sequences, the proposed tracker successfully captures targets, unlike baselines whose feature representations lack motion invariance.The rotations significantly change object appearances and produce tracking failures for baseline methods.

B. Evaluation on the Temporal Slowness Constraint and the Adaptation Module in Our Feature Learning Algorithm

Ablation results support both components of the feature-learning method: temporal slowness improves robustness to complicated motion, while adaptation improves performance by adding target-specific appearance information.

  • Temporal slowness constraint: The tracker using the temporal slowness constraint performs better than its variant without the constraint on 15 challenging sequences.This supports the constraint’s role in learning features robust to complicated motion transformations.
  • Adaptation module: The adaptation module improves tracker performance compared with the version without adaptation on eight video sequences.The comparison uses average center location errors and average overlap rates.
  • Adaptation module: Adaptation preserves robustness to complicated motion transformations while incorporating appearance information from specific target objects.This is the stated explanation for the adaptation module’s performance enhancement.

C. Evaluation on Our Tracker’s Capability of Handling Typical Problems in Visual Tracking

On eight sequences involving illumination change, occlusion, and cluttered backgrounds, the learned features are reported as more competitive than four alternative feature representations.

  • C. Evaluation on Our Tracker’s Capability of Handling Typical Problems in Visual Tracking: The evaluation uses eight sequences to test handling of illumination change, occlusion, and cluttered backgrounds.The tracker is quantitatively compared with four baseline trackers.
  • C. Evaluation on Our Tracker’s Capability of Handling Typical Problems in Visual Tracking: The comparison includes raw pixels, HOG, sparse representation, and data-independent features.These representations are used by the four baseline trackers.
  • C. Evaluation on Our Tracker’s Capability of Handling Typical Problems in Visual Tracking: The learned features are reported as more competitive than the other four feature representations for typical tracking problems.The conclusion is based on the quantitative results in Tables III and IV.

D. Comparison with the State-of-the-art Trackers

The proposed tracker is evaluated against state-of-the-art methods and DLT, showing stronger performance across most comparisons.

  • The tracker outperforms 14 state-of-the-art algorithms in most cases across 10 video sequences.
  • The tracker outperforms DLT on 5 of 8 evaluated sequences using average center error.

F. Evaluation on Our Learned Feature’s Generalizability

The learned features generalize across baseline trackers, improving performance for IVT and enabling successful tracking for ℓ1APG on challenging sequences.

  • The proposed tracker is compared with IVT and other trackers using average center error and average overlap rate.
  • The learned features improve IVT performance on most of 12 test sequences, although objects are missed in some sequences.
  • Integrating the learned features enables ℓ1APG to track objects successfully in 6 of 12 challenging sequences.
  • The feature-learning algorithm uses a two-layer convolutional architecture with a temporal slowness constraint and target-specific adaptation.
  • Figures 6 and 7 compare deepIVT with original IVT using average center error and average overlap rates, respectively.
Loading 1511.07940v1…