PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Target Candidate Association to Keep Track of What Not to Track

Christoph Mayer, Martin Danelljan, Danda Pani Paudel, Luc Van Gool

arXiv:2103.16556v2cs.CV

TL;DR

Appearance-based trackers struggle when distractors resemble the target, especially as the target changes appearance over time. The paper tracks target candidates and distractors jointly using learned association, trained with partial annotations and self-supervision, achieving state-of-the-art performance on six of seven benchmarks.

  • Problem

    Appearance-based tracking struggles to distinguish the target from similarly appearing distractors, particularly when the target undergoes substantial appearance changes.

  • Method

    The tracker propagates target-candidate identities across frames with a learned association network trained using single-target annotations, self-supervision, and mined informative samples.

  • Results

    The proposed tracker achieves a new state-of-the-art on six of seven generic object-tracking benchmarks.

  • Takeaways & Limitations

    Tracking distractors alongside the target provides a basis for more robust target identification in challenging visual-tracking sequences.

Abstract

from arXiv · show

The presence of objects that are confusingly similar to the tracked target, poses a fundamental challenge in appearance-based visual tracking. Such distractor objects are easily misclassified as the target itself, leading to eventual tracking failure. While most methods strive to suppress distractors through more powerful appearance models, we take an alternative approach. We propose to keep track of distractor objects in order to continue tracking the target. To this end, we introduce a learned association network, allowing us to propagate the identities of all target candidates from frame-to-frame. To tackle the problem of lacking ground-truth correspondences between distractor objects in visual tracking, we propose a training strategy that combines partial annotations with self-supervision. We conduct comprehensive experimental validation and analysis of our approach on several challenging datasets. Our tracker sets a new state-of-the-art on six benchmarks, achieving an AUC score of 67.1% on LaSOT and a +5.8% absolute gain on the OxUvA long-term dataset.

1. Introduction

Appearance-based tracking struggles when distractors resemble the target or when the target changes appearance, motivating the use of alternative cues. The paper proposes associating target candidates across frames while learning from partial annotations and self-supervision.

  • Motivation: Similar-looking distractors commonly cause appearance-based trackers to misidentify the target and eventually fail.Drastic target appearance changes further complicate target–distractor discrimination.
  • Approach: The proposed approach actively tracks distractors alongside the target to support more robust target identification.A base appearance tracker supplies target candidates for subsequent association.
  • Approach: Each candidate is represented using its classifier score, location, and appearance before candidates are associated across frames.The candidate encodings are jointly processed by a graph-based embedding network, which produces association scores.
  • Training challenges: Existing tracking datasets lack ground-truth correspondences for distractors because they annotate only one object per frame.The definition of a distractor also depends on the employed appearance model.
  • Training challenges: The training strategy combines single-target annotations with self-supervision and mines rare, challenging tracker-output cases.These cases include situations where the tracker follows a distractor or loses confidence in the target.

2. Related Work

Prior tracking work mainly handles distractors through appearance modeling, recurrent scene information, handcrafted rules, or detection association. The paper distinguishes its approach by using a learnable network for frame-to-frame association of target candidates.

  • Distractor handling: Discriminative trackers suppress distractors by incorporating background information into online target classifiers, but online appearance-model capacity remains limited.This limitation has motivated more specialized distractor-handling strategies.
  • Distractor handling: Related methods propagate scene information with an RNN, use handcrafted rules based on distractor locations, or form short tracklets from detections.These approaches address distractors or detections using different mechanisms.
  • Candidate association: In contrast, the paper introduces a learnable network that explicitly associates target candidates from frame to frame.This differs from hand-crafted association scores used to connect subsequent detections.
  • Relation to multi-object tracking: Object association is also central to multi-object tracking, where tracking-by-detection commonly associates detections over time as a graph partitioning problem.Such methods are typically non-causal and use detections from all frames, while multi-object tracking often targets limited object classes.

3. Method

The method associates target candidates, including distractors, across consecutive frames and propagates object identities to support target selection. It combines candidate features, graph-based embeddings, assignment prediction, object management, redetection, and confidence-aware memory updates.

  • Candidate extraction: The pipeline uses a discriminative base tracker to extract candidate score, location, and appearance features from each frame.Each candidate is encoded into a feature vector before association processing.
  • Object association: The tracker maintains visible target and distractor objects, adding unmatched candidates and deleting objects without current-frame correspondences.This object database supports reasoning about which candidate currently represents the target.
  • Candidate association: Candidate identities are propagated by predicting assignments between previous- and current-frame candidate sets, with dustbins representing unmatched candidates.The assignment matrix marks correspondences, while dustbins handle newly appearing or disappearing candidates.
  • Candidate association: A graph-based embedding network uses message passing and attention across candidate graphs to produce representations for association scoring.The architecture incorporates both neighboring candidates and cross-frame relationships.
  • Object association: When the target is absent, redetection selects a high-scoring candidate above η = 0.25, while object history determines whether it becomes the target or remains a distractor.A candidate is declared target when its score exceeds the relevant historical score; otherwise, it can be selected later if its score rises sufficiently.
  • Memory sample confidence: Memory updates remove the sample with the smallest combined age-confidence score α_kβ_k rather than always removing the oldest sample.The confidence score combines target-classifier confidence with association confidence and relies fully on the classifier once the target is lost.

4. Experiments

The tracker is evaluated through component ablations and comparisons across seven benchmarks, showing gains from candidate association and strong state-of-the-art performance. It also reports real-time-capable speeds for two tracker variants.

  • Evaluation setup: 18.3 and 29.6 FPS are achieved by KeepTrack and KeepTrackFast, respectively, on a single Nvidia GTX 2080Ti GPU.Both variants are implemented in Python using PyTorch.
  • Ablation study: +1.3% AUC on LaSOT is obtained after adding the target candidate association network to the ablation pipeline.The association component also provides substantial improvements on NFS and UAV123.
  • Ablation study: 1.5% improvement on LaSOT follows the addition of memory sample confidence, while enlarging the search area yields further gains across all three ablation datasets.The ablation averages five runs on NFS, UAV123, and LaSOT.
  • State-of-the-art comparison: 6.8% and 3.5% are the reported improvements over LTMU and SuperDiMP, respectively, on LaSOTExtSub.The extension subset contains long sequences averaging 2500 frames.
  • State-of-the-art comparison: 5.8% improvement in MaxGM over the previous best method establishes a new state-of-the-art on the OxUvA long-term test set.OxUvA requires predicting target presence or absence alongside the bounding box.
  • State-of-the-art comparison: 69.7% AUC establishes a new state-of-the-art on the reported OPT evaluation, exceeding TransT by 0.6% and TrDiMP by 2.2%.The AUC is computed over IoU thresholds T.
  • State-of-the-art comparison: 0.8% and 1.1% AUC gains over the SuperDiMP baseline are reported for KeepTrack and KeepTrackFast on OTB-100.The tracker also sets a new state-of-the-art on NFS.

5. Conclusion

The paper presents candidate association as a tracking pipeline for jointly following targets and distractors, supported by partial annotations and self-supervision. It validates the approach on seven benchmarks and reports new state-of-the-art results on six.

  • Conclusion: The learned target candidate association network tracks both target and distractor objects by propagating all candidate identities through the sequence.The approach is framed as a tracking pipeline rather than appearance-only distractor suppression.
  • Conclusion: Partial annotations combined with self-supervision address missing ground-truth correspondences between distractor objects in visual tracking.The training strategy compensates for the fact that datasets annotate only one object per frame.
  • Conclusion: Seven generic object tracking benchmarks are used for comprehensive experimental validation and analysis.The conclusion reports new state-of-the-art performance on six benchmarks.

Appendices

The appendices provide supplementary training and inference details, expanded analysis and results, and a project video offering visual comparisons with the baseline tracker.

  • Supplementary material: The supplementary material covers training details, inference details, a more detailed analysis, and expanded experimental results.These topics are organized in Sections A through D of the supplement.
  • Project video: The project video visualizes distractor tracking and marks identical object identities across consecutive frames with matching colors.It also compares the proposed tracker visually with SuperDiMP.

A. Training

The training appendix describes data generation, sample selection, training procedures, and the employed network architecture.

  • Training: Training details include data generation and sample selection intended to train the network more effectively.The appendix also discusses batching, augmentations, and synthetic sample generation.
  • Training: The appendix summarizes batching, augmentations, synthetic sample generation, and the employed network architecture.These details complement the description of the training procedure.

A.1. Data-Mining

The data-mining procedure emphasizes challenging frames and tracking failures, which are more informative for learning candidate associations than routine frames with one obvious candidate.

  • Frames are categorized using candidate count, classifier score, target selection, and selection correctness, with target correspondence determined by spatial distance.
  • 70% of frames belong to category D, containing a single target candidate matching the annotated target object.
  • 18.4% of frames belong to category H, which contains multiple candidates and supports self-supervised distractor association learning.
  • Categories J and K represent base-tracker failures where the target is respectively undetected or detected but not selected.
  • Self-supervised training samples categories H, K, and J at a 2 : 1 : 1 ratio while excluding category D because it lacks multiple candidates.

A.2. Training Data Preparation

The preparation pipeline augments candidate features and trains association components around a frozen base tracker, while adapting search areas and runtime processing for robustness and efficiency.

  • The candidate association network uses augmented candidate features to enable self-supervised training and improve generalization.
  • SuperDiMP is used as the frozen base tracker, with ResNet-50 features providing visual representations for target candidates.
  • The embedding network alternates self- and cross-attention layers and uses Sinkhorn optimal matching for 10 iterations.
  • The association module matches candidates to previous-frame objects using assignment probabilities above ω = 0.75 and includes target redetection logic.
  • Search-area rescaling increases the area after target loss using averages of up to 30 previous search areas until redetection.
  • KeepTrack uses a larger 480 × 480 search area and 30 × 30 score map, while KeepTrackFast reduces candidate extraction and association frequency.

C. More Detailed Analysis

Ablations show that search-area adaptation and candidate-association design both improve tracking, with a smaller network offering nearly the same accuracy at substantially higher speed.

  • Using both larger search areas and search-area rescaling produces the best results across NFS, UAV123, and LaSOT.
  • The candidate association network outperforms the baseline even without GNN layers, while two or nine layers improve performance further on all three datasets.
  • Two GNN layers and 10 Sinkhorn iterations reduce performance by only 0.1 on UAV123 and LaSOT while accelerating runtime by 44%.
  • The best configuration uses nine GNN layers and 50 Sinkhorn iterations but runs at 12.7 FPS.

D. Experiments

Experiments evaluate the tracker across LaSOT, LaSOTExtSub, UAV123, OTB-100, NFS, and VOT2018LT, showing strong benchmark performance and gains over the SuperDiMP baseline.

  • On LaSOT and LaSOTExtSub, the approach outperforms all other methods by a large margin in the reported plots.
  • Normalized precision on LaSOT and LaSOTExtSub measures the percentage of frames whose normalized center error is below threshold D across D ∈[0, 0.5].
  • On UAV123 and NFS, the method provides a significant gain over SuperDiMP and ranks among the top methods on OTB-100.
  • VOT2018LT contains 35 videos averaging 4200 frames, and trackers are ranked by F-score across confidence thresholds.
  • The tracker achieves the best results on all three VOT2018LT metrics and outperforms SuperDiMP by almost 10% in F-score.

E. Speed Analysis

KeepTrack achieves real-time operation with a conditional association module, but complex interactions between similar objects still produce target-identity failures.

  • Speed Analysis: 19.3 ms of overhead is added compared with the baseline tracker.Candidate extraction takes 2.7 ms, while candidate association takes 16.6 ms.
  • Speed Analysis: 18.3 FPS for KeepTrack and 29.6 FPS for KeepTrackFast are achieved with SuperDiMP as the base tracker.These timings were measured on a single Nvidia GTX 2080Ti GPU.
  • Speed Analysis: The association network runs only when more than one candidate is detected, while candidate extraction runs every frame.Frames with fewer than two candidates therefore avoid the association-module cost.
  • Failure Cases: When two hands approach and merge into one detected candidate, the tracker can lose certainty about which hand is the target.Positional information becomes limited when objects approach or cross, motivating deeper scene and target understanding.
  • Failure Cases: A distractor can take over tracking when the target candidate disappears and the distractor receives a high target score.The tracker may continue following the distractor even after the target candidate reappears.

G. Attributes

KeepTrack and KeepTrackFast are especially strong on sequence attributes associated with occlusion, clutter, motion, and similar objects, while KeepTrack remains less accurate for bounding-box regression.

  • UAV123: +2.5/2.6% AUC is achieved over the runner-up for UAV123 sequences with Aspect Ratio Change.The trackers also outperform the runner-up on Full Occlusion, Partial Occlusion, Background Clutter, Illumination Variation, and Similar Object.
  • UAV123: +1.8/1.1% AUC is achieved over the runner-up on UAV123 sequences with the Similar Object attribute.The largest listed gains include +2.5/2.6% for Aspect Ratio Change and +2.5/2.3% for Partial Occlusion.
  • UAV123: KeepTrack achieves the highest or second-highest AUC across UAV123 attributes except Out-of-View, while maintaining lower bounding-box regression accuracy than some trackers.The attribute analysis links strongest performance to distractor-heavy conditions such as clutter, occlusion, and similar objects.
  • LaSOT: +4.1/3.5% AUC is achieved on LaSOT sequences with Fast Motion.KeepTrack and KeepTrackFast also outperform SuperDiMP by +7.0/6.4% for Fast Motion.
  • LaSOT: +2.7/1.8% AUC is achieved on LaSOT sequences with Full Occlusion.The comparison with SuperDiMP shows improvements of +6.0/5.2% for Full Occlusion.
Loading 2103.16556v2…