PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Learning for Deep Visual Tracking

Di Yuan, Xiaojun Chang, Yi Yang, Qiao Liu, Dehua Wang, Zhenyu He

arXiv:2110.13259v2cs.CV

TL;DR

Deep CNN trackers need many labeled samples, but annotating them is time-consuming and expensive. The paper uses active learning with multi-frame collaboration, nearest-neighbor screening, and Tversky loss, and reports competitive accuracy and speed against state-of-the-art trackers on seven benchmarks.

  • Problem

    Deep CNN tracking requires numerous diverse labeled samples, but obtaining and annotating large training sets is time-consuming and prohibitively expensive.

  • Method

    ALT selects diverse video training sequences through multi-frame collaboration, screens isolated or low-quality samples using nearest-neighbor discrimination, and applies Tversky loss to bounding-box estimation.

  • Results

    ALT achieves competitive tracking accuracy and speed compared with state-of-the-art trackers on seven challenging benchmarks.

  • Takeaways & Limitations

    A limited labeling budget can support competitive deep tracking when selected samples preserve dataset diversity and representativeness.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have been successfully applied to the single target tracking task in recent years. Generally, training a deep CNN model requires numerous labeled training samples, and the number and quality of these samples directly affect the representational capability of the trained model. However, this approach is restrictive in practice, because manually labeling such a large number of training samples is time-consuming and prohibitively expensive. In this paper, we propose an active learning method for deep visual tracking, which selects and annotates the unlabeled samples to train the deep CNNs model. Under the guidance of active learning, the tracker based on the trained deep CNNs model can achieve competitive tracking performance while reducing the labeling cost. More specifically, to ensure the diversity of selected samples, we propose an active learning method based on multi-frame collaboration to select those training samples that should be and need to be annotated. Meanwhile, considering the representativeness of these selected samples, we adopt a nearest neighbor discrimination method based on the average nearest neighbor distance to screen isolated samples and low-quality samples. Therefore, the training samples subset selected based on our method requires only a given budget to maintain the diversity and representativeness of the entire sample set. Furthermore, we adopt a Tversky loss to improve the bounding box estimation of our tracker, which can ensure that the tracker achieves more accurate target states. Extensive experimental results confirm that our active learning-based tracker (ALT) achieves competitive tracking accuracy and speed compared with state-of-the-art trackers on the seven most challenging evaluation benchmarks.

I. INTRODUCTION

The paper addresses costly labeling for deep CNN tracking by selecting diverse, representative video training samples under a limited budget and improving bounding-box estimation with Tversky loss. ALT achieves competitive results on challenging tracking benchmarks.

  • I. INTRODUCTION: ALT selects diverse and representative training samples under a fixed budget, reducing labeling costs while maintaining competitive tracking performance.The method applies active learning to unlabeled video sequences rather than individual images.
  • I. INTRODUCTION: Multi-frame collaboration uses temporal relations to reduce background interference and preserve diversity in selected video sequences.The selection strategy differs from image-based active learning by operating on video sequences.
  • I. INTRODUCTION: Tversky loss improves bounding-box estimation by emphasizing the target over the background, producing more accurate target states.The paper identifies bounding-box estimation accuracy as a factor affecting tracking performance.
  • I. INTRODUCTION: Nearest-neighbor discrimination screens isolated or low-quality samples to improve the representativeness of the selected training set.The method uses average nearest-neighbor distance for this screening step.
  • I. INTRODUCTION: ALT achieves more competitive results than state-of-the-art trackers on OTB100, UAV123, TrackingNet, LaSOT, GOT10k, VOT2019, and VOT2020.The paper reports extensive experiments across seven challenging tracking benchmarks.

A. Deep learning-based tracking methods

Deep tracking methods use CNN representations and correlation-based architectures, while active-learning work queries samples using informativeness, representativeness, hybrid, or performance-based strategies. The paper highlights isolated selected sequences and bounding-box estimation as relevant design concerns.

  • A. Deep learning-based tracking methods: Siamese trackers formulate tracking as cross-correlation and include fully convolutional and attention-based architectures.SiamFC and HASN are cited as representative examples.
  • A. Deep learning-based tracking methods: Deep correlation trackers integrate correlation filters into networks and use off-the-shelf feature extractors to balance tracking speed and representation.Examples include VGGNet or AlexNet feature extraction and network-layer correlation filters.
  • A. Deep learning-based tracking methods: Active-learning query strategies include informativeness, representativeness, hybrid, and performance-based selection across several vision tasks.The cited applications include text classification, image classification, and object detection.
  • A. Deep learning-based tracking methods: Nearest-neighbor verification reorders selected sequences to exclude isolated and abnormal training samples.This distinguishes the paper’s sequence-selection procedure from methods that indiscriminately use selected samples.
  • A. Deep learning-based tracking methods: Existing bounding-box strategies use fixed scale factors or proposal IoU scores, but the paper identifies limitations in boundary accuracy and target-area emphasis.The proposal-based strategy does not distinguish the importance of intersection and union between estimated and real boxes.

III. ACTIVE LEARNING FOR OBJECT TRACKING

The tracker uses an end-to-end DiMP architecture combining target classification with bounding-box estimation. Its losses train confidence prediction and IoU-based box estimation jointly.

  • III. ACTIVE LEARNING FOR OBJECT TRACKING: The framework combines target classification and bounding-box estimation, with total loss Ltot = ηLcl + Libb.Classification predicts target confidence, while box estimation maximizes overlap with the ground-truth box.
  • III. ACTIVE LEARNING FOR OBJECT TRACKING: Target classification learns filter weights from labeled image patches and applies them to new patches to produce target confidence scores.The classification loss measures residuals between predicted confidence scores and the ground-truth target center.
  • III. ACTIVE LEARNING FOR OBJECT TRACKING: Discriminative learning rewrites the classification objective using regression labels modeled as a Gaussian centered on the target.The feature-extraction network contributes weight parameters f′, while zc denotes the Gaussian regression label.
  • III. ACTIVE LEARNING FOR OBJECT TRACKING: Bounding-box estimation selects candidate boxes by maximizing their IoU with the ground-truth bounding box.The component uses an overlap-maximization strategy based on target candidates generated from classification confidence.

B. Active learning for training sequences selection

The proposed active-learning selector chooses video sequences under a fixed budget by combining multi-frame diversity selection with neighborhood-based representativeness screening. FPS promotes diversity, while nearest-neighbor validation removes isolated or poor-quality samples.

  • B. Active learning for training sequences selection: FPS iteratively selects the sequence most distant from the already selected subset, producing a more diverse sample set than random selection.The first frame represents each sequence when applying FPS to unannotated video samples.
  • B. Active learning for training sequences selection: Multi-frame collaboration reduces background interference by exploiting temporal consistency, improving selection of target-relevant sequences.Single-frame selection can overemphasize background information relative to the moving target.
  • B. Active learning for training sequences selection: In the example, random selection is uncertain, SAL covers animals and aircraft, MAL adds humans, and KMAL covers human, animal, vehicle, and aircraft categories.The figure uses bold green boxes to mark samples selected by each method.
  • B. Active learning for training sequences selection: Neighborhood validation compares each candidate’s nearest-neighbor distance with the average nearest-neighbor distance to screen isolated or low-quality sequences.Such sequences can negatively affect the trained model if included for labeling and training.
  • B. Active learning for training sequences selection: The final selector maintains sample diversity and representativeness under a fixed budget by combining FPS with neighborhood validation.The algorithm selects a diverse and representative subset subA from an unlabeled sequence set A with budget B.

C. Tversky loss for bounding-box estimation

The tracker uses Tversky loss for bounding-box estimation, allowing different penalties for background-as-target and target-as-background errors. Its adjustable weighting is intended to improve boundary accuracy by emphasizing the target over the background.

  • C. Tversky loss for bounding-box estimation: Tversky loss replaces the bounding-box estimation loss with a coefficient based on overlap, background-as-target error, and target-as-background error.The coefficient uses α and β to weight the two mismatch terms.
  • C. Tversky loss for bounding-box estimation: The figure distinguishes intersection and union regions from the two asymmetric error regions used by Tversky loss.|B − Bgt| treats background as target, whereas |Bgt − B| treats target as background.
  • C. Tversky loss for bounding-box estimation: Adjusting α and β controls whether the model pays more attention to target or background errors during bounding-box estimation.The paper describes this control as a way to avoid treating background as the target.
  • C. Tversky loss for bounding-box estimation: The loss is combined with target classification loss, with η controlling the relative contribution of the two loss components.The total tracking objective combines the Tversky bounding-box loss and classification loss.

IV. EXPERIMENTS

The experiments evaluate the active learning-based tracker through ablations and comparisons with state-of-the-art trackers. The evaluation covers seven standard tracking benchmarks.

  • IV. EXPERIMENTS: The experiments compare ALT with state-of-the-art trackers after ablation studies assess active learning rules and Tversky loss.The stated purpose is to evaluate both component effectiveness and overall tracking competitiveness.
  • IV. EXPERIMENTS: Evaluation spans OTB100, UAV123, GOT10k, LaSOT, TrackingNet, VOT2019, and VOT2020.These are the seven benchmark datasets listed for comparison.

A. Experimental Details

The tracker uses a ResNet-50 backbone and is trained with fixed optimization settings on active-learning-selected GOT10k samples under varying budgets.

  • A. Experimental Details: ALT uses ResNet-50 and trains for 50 epochs with SGD momentum 0.9, learning rate 1e-5, weight decay 5e-4, and batch size 32.The target classification weight is η = 102.
  • A. Experimental Details: Unlike DiMP, the model is trained only on GOT10k samples selected by active learning under different budgets.The selected subsets are used for training and subsequent ablation analysis.

B. Ablation Study

The ablation study evaluates active learning rules and training budgets on LaSOT, while Fig. 4 reports success rate across budgets. Introducing Tversky loss improves tracking performance at each tested budget.

  • B. Ablation Study: The ablation study uses LaSOT to analyze different budgets and individual active learning rules under a fixed training setting.The selection process begins with single-frame image sampling and incrementally changes the selection rules.
  • B. Ablation Study: Fig. 4 presents success rate measured by AUC across different training budgets.The figure is specifically scoped to budget-dependent success-rate evaluation.
  • B. Ablation Study: Tversky loss improves tracking performance for each given training budget in the ablation study.This is the reported effect of adding Tversky loss to the active learning tracker.

C. State-of-the-art comparison

Across seven benchmark datasets, ALT delivers competitive or best-reported tracking performance despite using limited training samples. Results also show advantages over comparable trackers in accuracy, speed, or both.

  • OTB100: On OTB100, ALT matches ROAM for the best precision scores while requiring fewer training samples and outperforms DiMP50 with the same ResNet-50 backbone.ROAM is described as requiring large-scale, time-consuming, and expensive training data.
  • UAV123: On UAV123, ALT achieves the best reported precision and success scores, exceeding DiMP50 despite limited training samples.DiMP50 is reported at 64.8% AUC and 85.8% precision, while ALT improves on both indices.
  • GOT10k: On GOT10k, ALT surpasses DiMP50 in tracking performance and speed, achieves the best SR0.75 score, and exceeds DCFST50 in speed.ALT is slightly below DCFST50 on AO and SR0.50 but is faster.
  • LaSOT: On LaSOT, ALT obtains the best normalized precision and success scores among the compared state-of-the-art trackers.It outperforms DiMP50, Ocean, and SiamCAR on each reported performance metric.
  • VOT2019 and TrackingNet: On VOT2019, ALT achieves the best EAO score and exceeds trackers using the same backbone, while TrackingNet results resemble large-scale-data trackers with fewer samples.The supplied passages describe these comparisons without providing the underlying table values.
  • ALT achieves competitive tracking performance across seven standard benchmarks while using limited training samples.The paper reports this overall conclusion across the benchmark comparisons.

D. Qualitative comparison

Qualitative comparisons on challenging sequences show ALT producing more satisfactory tracking results than several state-of-the-art trackers. The paper attributes this to active-learning-selected samples and Tversky-loss bounding-box estimation.

  • ALT shows more accurate tracking results than seven compared trackers in complex tracking scenarios.The qualitative comparison includes SiamRPN++, PrDiMP50, DiMP50, ROAM, Ocean, SiamBAN, and ATOM.
  • Tversky loss improves bounding-box estimation by directing more attention to the target than the background.The paper links this strategy to more accurate target states in challenging sequences.

V. CONCLUSIONS

The paper concludes that multi-frame active learning can select diverse and representative training sequences under a limited labeling budget. Combined with Tversky-loss bounding-box estimation, ALT achieves results comparable to trackers trained on extensive samples.

  • ALT combines multi-frame cooperation, nearest-neighbor screening, and Tversky loss for limited-budget deep visual tracking.Multi-frame selection targets diversity, nearest-neighbor discrimination screens isolated or low-quality sequences, and Tversky loss improves target-state estimation.
  • With limited-budget samples, ALT achieves comparable tracking results to state-of-the-art trackers requiring extensive training samples.This is the paper's concluding experimental claim.
Loading 2110.13259v2…