PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Discriminative Model Prediction for Tracking

Goutam Bhat, Martin Danelljan, Luc Van Gool, Radu Timofte

arXiv:1904.07220v2cs.CV

TL;DR

Visual tracking requires learning a target-specific appearance model online, making end-to-end training difficult, while Siamese prediction ignores background appearance. The paper embeds discriminative model learning in an end-to-end architecture using a few-step optimization process, achieving state-of-the-art results on multiple benchmarks at over 40 FPS.

  • Problem

    Tracking must construct a robust target-specific model during inference, but Siamese methods use only target appearance and therefore have limited target-background discriminability.

  • Method

    The architecture predicts a discriminative target model from an optimization procedure derived from a discriminative loss, with learned loss components and efficient initialization.

  • Results

    State-of-the-art results were achieved on 7 tracking datasets while running at over 40 FPS, including an EAO score of 0.44 for DiMP50 on VOT2018.

  • Takeaways & Limitations

    The approach combines target and background appearance information in end-to-end target-model prediction and achieves strong tracking performance with high processing speed.

  • Takeaways & Limitations

    A simple learned learning-rate approach adapts filter parameters slowly, requiring many iterations that harm efficiency and complicate offline learning.

Abstract

from arXiv · show

The current strive towards end-to-end trainable computer vision systems imposes major challenges for the task of visual tracking. In contrast to most other vision problems, tracking requires the learning of a robust target-specific appearance model online, during the inference stage. To be end-to-end trainable, the online learning of the target model thus needs to be embedded in the tracking architecture itself. Due to the imposed challenges, the popular Siamese paradigm simply predicts a target feature template, while ignoring the background appearance information during inference. Consequently, the predicted model possesses limited target-background discriminability. We develop an end-to-end tracking architecture, capable of fully exploiting both target and background appearance information for target model prediction. Our architecture is derived from a discriminative learning loss by designing a dedicated optimization process that is capable of predicting a powerful model in only a few iterations. Furthermore, our approach is able to learn key aspects of the discriminative loss itself. The proposed tracker sets a new state-of-the-art on 6 tracking benchmarks, achieving an EAO score of 0.440 on VOT2018, while running at over 40 FPS. The code and models are available at https://github.com/visionml/pytracking.

1. Introduction

Visual tracking must construct a target-specific model online during inference, complicating end-to-end learning. The paper addresses Siamese trackers’ limited use of background information with an end-to-end discriminative model-prediction architecture.

  • Problem: Generic tracking estimates an arbitrary target’s state in every video frame from information initially available at test time.Because target-specific information is unavailable during offline training, the target model must be constructed during inference.
  • Existing approach: Siamese trackers enable end-to-end training by representing the target model as template features and locating the most similar image region.They learn an embedding offline and compare regions through cross-correlation.
  • Limitations: Siamese model prediction uses only target appearance, ignoring background information needed to distinguish the target from similar objects.The framework also has limitations in generalization and model updating, with template averaging used for updates.
  • Proposed approach: The proposed architecture predicts a discriminative target model through iterative optimization derived from a discriminative learning loss.Steepest-descent step lengths and an effective initializer reduce the required optimization steps, while the loss itself is learned during offline training.
  • Results: State-of-the-art results were reported on all 7 evaluated tracking datasets while running at over 40 FPS.The experiments covered VOT2018, LaSOT, TrackingNet, GOT10k, NFS, OTB-100, and UAV123.

2. Related Work

Related tracking methods span efficient Siamese similarity models, online discriminative classifiers, neural formulations of optimization procedures, and meta-learning approaches. The central distinction is how these methods incorporate background information and embed target-model learning into end-to-end training.

  • Siamese approaches: Siamese trackers learn an offline similarity metric and offer high efficiency, but cannot incorporate background or previous-frame information effectively into model prediction.Their inability to use these cues limits robustness relative to state-of-the-art online-learning trackers.
  • Discriminative online learning: Online discriminative trackers distinguish target from background and achieve strong robustness, but their complex learning procedures resist straightforward end-to-end formulation.This creates a trade-off between discriminative online learning and end-to-end trainability.
  • Neural optimization: Prior neural approaches embed correlation-filter solutions or unroll BACF optimization iterations to learn tracking components and hyper-parameters.The cited approaches use either a closed-form single-sample solution or a multi-stage ADMM-based training procedure.
  • Meta-learning: Predicting a target model from few images is closely related to meta-learning, which has also been applied to tracking-specific model and feature-space prediction.Examples include predicting tracking-model parameters and target-specific feature spaces.

3. Method

The architecture predicts a discriminative target model from annotated samples by embedding a learned loss and rapid optimization process into an end-to-end tracking network. It uses target and background information, learns key loss components, and predicts convolutional classifier weights in only a few iterations.

  • Model prediction architecture: The model prediction network combines a discriminative learning loss with an optimization strategy designed for rapid convergence.The predicted target model can be obtained in only a few iterations without compromising discriminative power.
  • Tracking architecture: The tracking architecture has target-classification and bounding-box-estimation branches that share a backbone feature extractor.The classification branch applies the predicted target-classifier weights to features extracted from test frames.
  • Model prediction architecture: The target model is a convolutional filter predicted from bounding-box annotated feature samples and used to classify target appearance against background.The predictor receives feature maps and target-center coordinates, then outputs convolutional filter weights.
  • Discriminative learning loss: The loss combines least-squares regression with a hinge-like background penalty to reduce negative-sample imbalance and avoid forcing calibrated scores for easy background samples.The residual uses a spatial weight, target mask, regression scores, and a clipped background response max(0, s).
  • Learning the discriminative loss: The target mask, spatial weight, regression label, and regularization factor are learned from data rather than fixed manually.These loss parameters are learned by the model prediction network and are parameterized using distance from the target center.
  • Optimization-based architecture: A Gauss-Newton-based steepest-descent procedure computes optimization steps using first-order derivatives to predict a strong discriminative filter efficiently.The method sets Q^(i)=(J^(i))^T J^(i), where J^(i) is the residual Jacobian at the current filter estimate.

4. Experiments

Experiments analyze the model-prediction architecture and compare DiMP with state-of-the-art trackers across seven benchmarks. The proposed approach combines discriminative model prediction with end-to-end training and operates above 40 FPS.

  • Runtime: 57 FPS with ResNet-18 and 43 FPS with ResNet-50 demonstrate that the tracker maintains high speed on a single Nvidia GTX 1080 GPU.Both versions use the same Python/PyTorch implementation.
  • Architecture analysis: 58.2% AUC for target-only initialization rises to 61.6% with gradient descent, while steepest descent improves on gradient descent by 2.2%.The comparison uses the combined OTB-100, NFS, and UAV123 datasets.
  • Architecture analysis: 1.3%, 2.6%, 0.7%, and 0.5% AUC gains result respectively from adding initialization, backbone fine-tuning, a classification block, and offline loss learning.These components are added incrementally to the steepest-descent baseline.
  • State-of-the-art comparison: The experiments report state-of-the-art results on all seven evaluated benchmarks.The benchmark set comprises VOT2018, LaSOT, TrackingNet, GOT10k, NFS, OTB-100, and UAV123.
  • State-of-the-art comparison: DiMP-50 outperforms SiamRPN++ by 6.3% relative EAO and has a 34% lower failure rate on VOT2018 with similar accuracy.Both methods use ResNet-50 features.

5. Conclusions

The paper presents an end-to-end tracking architecture that predicts target models through iterative optimization derived from a discriminative learning loss. It achieves state-of-the-art performance on six tracking benchmarks while operating at over 40 FPS.

  • The architecture uses a steepest-descent-based optimizer and an effective initializer to predict a powerful target model in only a few optimization steps.It also learns the discriminative loss during offline training by minimizing prediction error on unseen test frames.
  • The proposed tracker sets a new state-of-the-art on 6 tracking benchmarks while operating at over 40 FPS.

Supplementary Material

The supplementary material derives optimizer quantities and reports additional evaluations, ablations, and visual analyses. Across the reported benchmarks, the tracker achieves strong results, while retaining performance with substantially less training data.

  • S1. Closed-Form Expression for ∇L: The supplement derives a closed-form filter-gradient expression and explains its implementation using standard deep-learning operations.
  • S2. Calculation of h in Algorithm 1: The optimizer computes h = J(i)∇L(f(i)) to determine the optimal step length α in Algorithm 1.The supplement derives the squared L2 norm needed for step-length calculation.
  • S3. Detailed Results on VOT2018: 0.44 EAO is achieved by DiMP50 on VOT2018, where trackers are ranked by expected average overlap over typical sequence lengths.
  • S4. Detailed Results on LaSOT: 5.9% and 12.8% relative gains over ATOM are achieved by the ResNet-18 and ResNet-50 versions, respectively, on LaSOT normalized precision.
  • S5. Detailed Results on NFS, OTB-100, and UAV123: 3.5% absolute AUC gain over ATOM is achieved on NFS, while DiMP-50 obtains the best results on NFS and UAV123 and remains competitive on OTB-100.
  • S6. Impact of Training Data: 1.5% performance degradation occurs when training uses only 10% of the total videos, while 1% still yields an AUC score of around 58%.

S7. Visualizations of learned yc, mc, and vc

The visualization shows learned regression labels, target masks, and spatial weights as continuous quantities sampled on a discrete feature grid. Samples near the target-background transition receive less weight because they are ambiguous.

  • The network learns continuous regression labels yc, target masks mc, and spatial weights vc, visualized at discrete feature-grid points.
  • Samples in the target-background transition region receive less weight because of their ambiguous nature.
Loading 1904.07220v2…