PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic Regression for Visual Tracking

Martin Danelljan, Luc Van Gool, Radu Timofte

arXiv:2003.12565v1cs.CVcs.LG

TL;DR

Visual tracking needs uncertainty estimates, but conventional confidence scores lack a clear probabilistic interpretation. The paper predicts a conditional target-state density, models annotation noise and task ambiguity through label distributions, and trains with KL divergence. The resulting tracker improves over DiMP and reaches state-of-the-art performance on six datasets.

  • Problem

    Visual trackers need interpretable uncertainty estimates because confidence scores lack a clear probabilistic meaning, while tracking itself must estimate targets from an initial test-time example.

  • Method

    The network predicts p(y|x, θ), a conditional density of the target state, and minimizes KL divergence to a label distribution modeling annotation noise and task ambiguities.

  • Results

    PrDiMP outperforms previous state-of-the-art trackers, with +2.9% AUC on LaSOT and +1.8% Success on TrackingNet.

  • Takeaways & Limitations

    The probabilistic formulation provides a probabilistic representation of tracking outputs while substantially improving tracker performance across benchmark datasets.

  • Takeaways & Limitations

    The Gaussian label variance σ2 is treated as a hyper-parameter rather than estimated from multiple annotations in this work.

Abstract

from arXiv · show

Visual tracking is fundamentally the problem of regressing the state of the target in each video frame. While significant progress has been achieved, trackers are still prone to failures and inaccuracies. It is therefore crucial to represent the uncertainty in the target estimation. Although current prominent paradigms rely on estimating a state-dependent confidence score, this value lacks a clear probabilistic interpretation, complicating its use. In this work, we therefore propose a probabilistic regression formulation and apply it to tracking. Our network predicts the conditional probability density of the target state given an input image. Crucially, our formulation is capable of modeling label noise stemming from inaccurate annotations and ambiguities in the task. The regression network is trained by minimizing the Kullback-Leibler divergence. When applied for tracking, our formulation not only allows a probabilistic representation of the output, but also substantially improves the performance. Our tracker sets a new state-of-the-art on six datasets, achieving 59.8% AUC on LaSOT and 75.8% Success on TrackingNet. The code and models are available at https://github.com/visionml/pytracking.

1. Introduction

Visual tracking regresses a target’s state frame by frame, but confidence-based outputs lack a clear probabilistic interpretation for uncertainty. The paper proposes predicting a conditional state density and reports substantial tracking improvements.

  • Problem: Visual tracking estimates a target object’s state in every video frame, typically using a bounding box.In the general setting, the target’s initial state is given during inference, so the tracker must learn the target model during tracking.
  • Existing approach: Confidence-based regression estimates the target by maximizing a state-dependent score s(y, x).This paradigm underlies both discriminative correlation filters and Siamese trackers.
  • Existing approach: The confidence score can encode uncertainty but has no clear interpretation because its scale and characteristics depend on the loss and pseudo-label strategy.This complicates reasoning about uncertainty and decisions such as whether to update the tracker or declare the target lost.
  • Proposed approach: The proposed formulation predicts the conditional probability density p(y|x) of the target state given the input image.Unlike a confidence score, the density has a direct probabilistic interpretation.
  • Results: PrDiMP improves tracking performance over previous state-of-the-art trackers, including gains of +2.9% AUC on LaSOT and +1.8% Success on TrackingNet.The approach is integrated into DiMP and evaluated on seven benchmark datasets.

2. Regression by Confidence Prediction

Confidence-based regression learns a scalar score over candidate target states and maximizes it to obtain the estimate. Visual tracking uses this formulation because direct regression is less suitable for its challenging, test-time adaptation setting.

  • General formulation: Direct regression learns a mapping from images to continuous target states by minimizing a prediction discrepancy loss.Popular losses include the L_p family.
  • Visual tracking: Confidence prediction has prevailed over direct regression in tracking, detection, and pose estimation because it supports candidate-state scoring rather than a single committed output.Tracking approaches include DCF, Siamese trackers, and bounding-box regression heads based on IoU prediction.
  • Confidence prediction: Confidence-based regression learns a scalar function s_θ(y, x) over output-input pairs and obtains the estimate by maximizing the score over y.The formulation transforms regression into confidence prediction using pseudo labels and a task-dependent loss.
  • Visual tracking: Visual tracking is challenging because only the target’s initial location is provided at test time, requiring the tracker to learn the target model during inference.The target state is commonly an axis-aligned bounding box y ∈ R4.
  • DCF and Siamese methods: DCF trackers compute confidence maps by convolving image features with a target model and evaluate the result at each spatial location.Their squared confidence loss enables efficient optimization in the Fourier domain.
  • DCF and Siamese methods: Siamese trackers learn an embedding space offline so similarities between a target template and a frame can be computed as a correlation.They commonly treat target localization as dense binary classification with binary cross-entropy.

3. Method

The method replaces confidence-based regression with a predictive probability density over outputs, trained against a conditional ground-truth distribution that represents annotation noise and task ambiguity. It uses KL-divergence training with grid or Monte Carlo integration, depending on the regression space.

  • Probabilistic representation: The model predicts p(y|x, θ) and obtains point estimates by maximizing the predicted density, while representing uncertainty probabilistically.This replaces confidence values whose interpretation depends on the loss and pseudo-label strategy.
  • Probabilistic representation: The density is formed by exponentiating a scalar network score and dividing by the input-dependent normalizing constant Zθ(x).This formulation generalizes SoftMax from discrete outputs to an arbitrary output space.
  • Label uncertainty: Target-center regression is ambiguous because small appearance changes, such as a raised tail, can move the bounding-box center into the background.The tracker’s visually natural prediction may remain near the earlier location despite the changed center of mass.
  • Label uncertainty: The conditional ground-truth distribution p(y|yi) models label noise and ambiguity associated with an annotated output yi.Multiple annotations can empirically estimate its variance; in the Gaussian case, variance may instead be treated as a hyper-parameter.
  • Learning objective: Training minimizes KL divergence rather than relying only on negative log-likelihood, thereby incorporating uncertainty information from p(y|yi).Negative log-likelihood uses only each training pair and makes no additional assumptions, whereas the proposed loss incorporates the label distribution.
  • Numerical integration: Grid sampling approximates the loss for two-dimensional image-coordinate regression, while Monte Carlo importance sampling handles more general, higher-dimensional regression problems.Grid sampling can scale poorly in higher dimensions and may introduce bias; proposal samples should cover both the label distribution and regions of high predicted density.

4. Tracking Approach

The tracking approach integrates probabilistic regression into both output branches of the DiMP tracker while preserving its tracking procedure with minimal changes. The center and bounding-box branches use branch-specific approximations to train predictive densities and model annotation uncertainty.

  • Target Center Regression: During tracking, target-center localization evaluates confidence densely over a wide search region centered on the previous target location.This retains the baseline branch’s coarse localization role while replacing its scalar confidence interpretation with a predictive density.
  • Bounding Box Regression: The bounding-box branch applies the probabilistic formulation to the target conditional IoU-Net-based architecture used for accurate box fitting.Inference fits the box through gradient-based maximization with respect to the bounding-box state.
  • Probabilistic DiMP: The proposed tracker integrates fully probabilistic output representations into both the target-center and bounding-box branches of DiMP.The predicted densities are visualized for both branches and preserve uncertainty information in the output state.
  • Target Center Regression: The target-center branch predicts p(ytc|x, θ) by applying the probabilistic formulation to its fully convolutional confidence output.Its KL-divergence loss is approximated using grid sampling, with a Gaussian conditional ground-truth density based on the baseline pseudo-label scale.
  • Bounding Box Regression: Bounding-box annotation noise and uncertainty are modeled with an isotropic Gaussian label distribution using σbb = 0.05, and KL divergence is estimated by Monte Carlo sampling.The method uses the same proposal distribution for a fair comparison with the baseline setting.
  • Training and inference: The entire network is trained jointly end-to-end with the original DiMP strategy and settings, while the tracking procedure is otherwise preserved with minimal changes.Training uses LaSOT, GOT10k, TrackingNet, and COCO training splits for 50 epochs with 1000 iterations each.

5. Experiments

Experiments evaluate regression formulations, uncertainty modeling, and state-of-the-art performance across tracking benchmarks. Probabilistic regression improves DiMP and performs strongly across diverse datasets.

  • Comparison of Regression Models: For BBR, the KL-based model with annotation uncertainty adds 0.8% AUC beyond the NLL formulation and 2.0% beyond the L2 baseline.The NLL formulation itself provides a 1.2% gain over the 63.8 AUC L2 baseline.
  • Comparison of Regression Models: For TCR, explicitly modeling label uncertainty achieves a 2.3% AUC gain, while probabilistic NLL is unsuitable because it does not account for inherent ambiguity.
  • Label Uncertainty: AUC varies with the label-noise standard deviation: excessive uncertainty produces overly uncertain predictions, whereas small σ causes overfitting and over-confidence.This analysis separately varies σbb and σtc for bounding-box and target-center regression.
  • State-of-the-Art: On LaSOT, PrDiMP improves over DiMP by 3.2% and 2.9% AUC with ResNet-18 and ResNet-50, respectively.The improvement is most prominent for overlap thresholds T > 0.3.
  • State-of-the-Art: On UAV123, PrDiMP50 reaches 68.0% AUC, compared with 65.3% for DiMP50 and 64.2% for ATOM.

6. Conclusions

The paper proposes probabilistic regression for tracking and reports improved performance with probabilistic output representations. It concludes that the approach achieves state-of-the-art results on six datasets.

  • The method predicts the conditional output density p(y|x, θ) and trains by minimizing KL divergence to a label distribution modeling annotation noise and task ambiguities.
  • Applied to tracking, the probabilistic formulation outperforms the baseline DiMP.
  • The tracker achieves new state-of-the-art performance on six datasets.

A. Derivation of KL Divergence Loss

The appendix derives the training loss from the KL divergence between the predicted distribution and ground-truth density. The derivation then removes a constant entropy term and substitutes the partition function.

  • The derivation starts from the KL divergence between p(y|xi, θ) and the ground-truth density p(y|yi).
  • The resulting loss is obtained after inserting the predicted-distribution formulation into the divergence.
  • The derivation discards the negative entropy of p(y|yi) and substitutes the definition of the partition function Zθ(xi).

B. Target Center Regression Module

The PrDiMP target-center optimizer minimizes a KL-divergence-based loss, using Newton-approximated steepest descent implemented through closed-form operations. Training uses labeled support images and regularization, while spatial outputs are represented through grid-based SoftMax quantities.

  • Target Center Regression Module: The optimizer predicts target-center regression weights from support images and corresponding bounding-box annotations.During tracking, support images come from first-frame augmentations and gradual memory updates.
  • Target Center Regression Module: The module replaces DiMP’s least-squares formulation with a KL-divergence-based learning loss and adds L2 regularization.The regularization term controls generalization to unseen frames, while sample weights control individual support-image contributions.
  • Target Center Regression Module: The per-sample loss is a grid approximation of the original KL-divergence objective and corresponds to SoftMax-Cross Entropy.Spatial locations are represented as vectors or 2D maps, with K = HW grid positions.
  • Target Center Regression Module: The optimizer uses steepest descent with a quadratic Newton approximation to compute step lengths.Closed-form gradient, Hessian, and denominator expressions enable implementation as deep neural network layers.
  • Target Center Regression Module: Algorithm 1 iteratively extracts support-image features, initializes weights, and executes the optimizer loop.The update uses efficiently computed gradient and Hessian quantities for the current weight estimate.

C. Detailed Results

This section presents more detailed results from the state-of-the-art comparison described in Section 5.3.

  • C. Detailed Results: The section provides additional results from the state-of-the-art comparison performed in Section 5.3.The passage introduces the section as a more detailed examination of that comparison.
  • C. Detailed Results: The detailed results supplement the comparison reported in Section 5.3.No dataset-specific outcome is stated in the supplied passage.
  • C. Detailed Results: The section focuses on elaborating the tracker’s state-of-the-art comparison.The supplied passage identifies the comparison as the section’s subject.

C.1. LaSOT

On LaSOT, the approach is evaluated with normalized precision and success plots, and it outperforms prior trackers by a large margin, improving both accuracy and robustness.

  • C.1. LaSOT: The normalized precision score measures the percentage of frames whose normalized center distance is below threshold D, ranked by curve area.The LaSOT test set contains 280 videos, and thresholds range from D ∈[0, 0.5].
  • C.1. LaSOT: The LaSOT normalized precision plot reports average normalized precision in its legend, where the approach outperforms previous trackers by a large margin.This figure provides the corresponding visual comparison across trackers.
  • C.1. LaSOT: Success plots report the percentage of frames exceeding an IoU threshold, with each tracker’s AUC shown in the legend.The supplied caption covers GOT10k, OTB-100, and NFS comparisons rather than a LaSOT-specific success result.
  • C.1. LaSOT: PrDiMP50 achieves absolute gains of 11.9% over SiamRPN++ and 3.8% over DiMP-50.The gains reflect improvements at both small thresholds, associated with accuracy, and large thresholds, associated with robustness.

C.4. NFS

On the challenging 30 FPS NFS dataset, both ResNet-18 and ResNet-50 PrDiMP versions significantly outperform the previous state-of-the-art DiMP-50.

  • C.4. NFS: The NFS success plot covers the 30 FPS version of the challenging dataset and compares available results from leading trackers.Compared trackers include DiMP, ATOM, UPDT, CCOT, ECO, and MDNet.
  • C.4. NFS: Both PrDiMP ResNet-18 and ResNet-50 versions significantly outperform the previous state-of-the-art DiMP-50.The two PrDiMP versions achieve similar results to each other in this comparison.

D. Visualization of Predicted Distributions

The tracker visualizes predicted probability densities for target center and bounding box regression on challenging sequences. These distributions expose uncertainty from distractors, ambiguity, and difficult box estimation.

  • The target center density is computed by the fully convolutional center regression branch on the search region.
  • Because the bounding box state is 4-dimensional, its full density cannot be shown as a 2-dimensional heatmap.The visualization therefore uses separate density slices for box location and size.
  • Bounding box center uncertainty is visualized by evaluating density over center coordinates while holding the box size fixed at the current estimate.
  • Bounding box size uncertainty is visualized by evaluating density over log-size coordinates while holding the box center fixed at the current estimate.
  • For the cat and mirror-image sequence, similarity and proximity make exact bounding-box prediction difficult, while the size probability is shown relative to the estimated bounding box size.
  • In a sequence with multiple distractors, a strong secondary mode indicates the true target even when the tracker briefly jumps to a distractor.The distributions also reflect the difficulty of bounding box regression when small targets have nearby similar-looking objects.
Loading 2003.12565v1…