PaperlayerView exact revision on arXiv

Source-linked AI summary

Tracking by Instance Detection: A Meta-Learning Approach

Guangting Wang, Chong Luo, Xiaoyan Sun, Zhiwei Xiong, Wenjun Zeng

arXiv:2004.00830v1cs.CV

TL;DR

Visual tracking can be treated as instance detection, but a detector must adapt to a new target from very limited initial-frame data. The paper uses MAML to initialize modern detectors for rapid domain adaptation, producing Retina-MAML and FCOS-MAML trackers. Both are competitive with state-of-the-art trackers and run at 40 FPS, with strong results on OTB-100 and TrackingNet.

  • Problem

    Tracking requires adapting a detector to a particular instance from the initial frame without overfitting.

  • Method

    The paper uses MAML to meta-train any gradient-based modern detector, then fine-tunes it on the initial frame for tracking.

  • Results

    Both trackers achieve competitive state-of-the-art performance across four benchmarks; Retina-MAML reaches 0.712 AUC on OTB-100, while FCOS-MAML reaches 0.757 AUC on TrackingNet.

  • Takeaways & Limitations

    The procedure converts modern object detectors into template-free trackers that support rapid adaptation and real-time operation at 40 FPS.

  • Takeaways & Limitations

    Performance on LaSOT still has room for improvement, particularly for long sequences where online updating remains an open issue.

Abstract

from arXiv · show

We consider the tracking problem as a special type of object detection problem, which we call instance detection. With proper initialization, a detector can be quickly converted into a tracker by learning the new instance from a single image. We find that model-agnostic meta-learning (MAML) offers a strategy to initialize the detector that satisfies our needs. We propose a principled three-step approach to build a high-performance tracker. First, pick any modern object detector trained with gradient descent. Second, conduct offline training (or initialization) with MAML. Third, perform domain adaptation using the initial frame. We follow this procedure to build two trackers, named Retina-MAML and FCOS-MAML, based on two modern detectors RetinaNet and FCOS. Evaluations on four benchmarks show that both trackers are competitive against state-of-the-art trackers. On OTB-100, Retina-MAML achieves the highest ever AUC of 0.712. On TrackingNet, FCOS-MAML ranks the first on the leader board with an AUC of 0.757 and the normalized precision of 0.822. Both trackers run in real-time at 40 FPS.

1. Introduction

The paper reframes tracking as instance detection and proposes adapting modern object detectors to new target instances using MAML initialization and a few gradient updates. The resulting Retina-MAML and FCOS-MAML trackers achieve competitive benchmark performance while retaining real-time speed.

  • Tracking locates a particular object instance across frames, unlike detection, which identifies objects from predefined classes without distinguishing intra-class instances.
  • The central challenge is initializing a detector so it can absorb a new instance from one image without overfitting.Before adaptation, the detector behaves like a general object detector; after adaptation, it can memorize and locate the target in subsequent frames.
  • MAML enables a three-step conversion: select a gradient-trained detector, meta-train it on tracking sequences, then fine-tune it on the initial frame.Further online samples can be used to continue adapting the detector during tracking.
  • Retina-MAML and FCOS-MAML are built from RetinaNet and FCOS, respectively, and evaluated on OTB, VOT, TrackingNet, and LaSOT.
  • 40 FPS is achieved by both trackers, while OTB-100 AUCs reach 0.712 for Retina-MAML and 0.704 for FCOS-MAML.Retina-MAML also achieves an EAO of 0.452 on VOT-2018, and FCOS-MAML reaches an AUC of 0.757 on TrackingNet.

2. Related Work

Prior tracking methods either use explicit templates or fine-tune networks, but template-free adaptation remains difficult to make both effective and fast. The paper applies meta-learning to prepare detectors for rapid instance learning from few examples.

  • CNN-based visual object tracking: Template-based trackers store target appearance in an explicit template and commonly avoid online CNN updates for speed.
  • CNN-based visual object tracking: Template-based methods face a performance ceiling because most lack an effective mechanism for integrating new target appearances into the template online.
  • CNN-based visual object tracking: Template-free trackers store target appearance in fine-tuned network parameters, but quickly adapting without overfitting remains challenging.
  • Meta learning and its application to tracking: Meta-learning trains across varied tasks so a model can learn a new task from only a few training samples.For tracking, the tasks are instance-detection problems derived from different tracking sequences.
  • Meta learning and its application to tracking: MAML learns initialization parameters that support good generalization after a small number of gradient steps and can be used with gradient-trained detectors.The paper positions this capability as a way to convert modern detectors into trackers rather than merely accelerate existing trackers.

3. Learning an Instance Detector with MAML

The method learns an initialization that lets a differentiable detector adapt to a new tracking instance from limited labeled frames. It combines support-set adaptation, target-set evaluation, and MAML refinements to produce a detector-compatible tracker.

  • Instance-detector training: A good initialization lets a detector quickly adapt to a new instance when only the initial frame is available.The training data are videos with ground-truth labels for the target object on each frame.
  • Instance-detector training: The inner optimization updates detector parameters with k-step gradient descent on a support set sampled from a training video.The support set is denoted Ds_i and consists of labeled image examples.
  • Instance-detector training: The target set evaluates whether the adapted detector generalizes within the same video.The target-set loss is computed after applying the trained detector to additional samples from that video.
  • Instance-detector training: The outer objective learns an initialization θ0 across tracking videos by back-propagating through inner-level updates.Only differentiability of the detector is assumed, making the approach applicable to most deep learning-based detectors.
  • MAML refinements: Multi-step loss optimization includes losses after every inner update, including the initial parameters, and the authors find this stabilizes gradients.Each step receives a weight γk in the outer-level loss.
  • MAML refinements: Kernel-wise learnable learning rates assign one rate per convolutional kernel instead of per parameter, adding only Cout learnable parameters per convolution layer.This design improves gradient-based updating while avoiding the model-size increase of per-parameter rates.

4. Retina-MAML and FCOS-MAML

Retina-MAML and FCOS-MAML convert modern single-stage detectors into trackers through MAML initialization, detector simplifications, and online domain adaptation. The resulting pipeline uses ResNet-18-based branches, gradient updates from the initial frame, and candidate-box post-processing for subsequent tracking.

  • Detector choices: MAML permits modern gradient-trained detectors, including RetinaNet and FCOS, to serve as bases for tracker construction.The approach selects single-stage detectors because they are faster and easier to manipulate than two-stage detectors.
  • Detector choices: RetinaNet predicts anchor classifications and box offsets, whereas FCOS classifies pixels and directly estimates four boundary offsets.These represent the anchor-based and anchor-free detector designs used for the two trackers.
  • Offline training: The networks use ResNet-18 with frozen first three blocks, removed block-5, and independently trained block-4 copies for classification and regression.The architecture produces stride-8 feature maps, while online training updates only a subset of trainable layers.
  • Offline training: Offline MAML training uses four-step gradient adaptation, kernel-wise learnable learning rates, multi-step loss weighting, and Adam outer optimization.Training uses MS-COCO, GOT10k, TrackingNet, and LaSOT-train with cropped and augmented images.
  • Online training and tracking: Online adaptation extracts a 263 × 263 patch from the initial bounding box, augments it into a support set, and applies 5-step gradient descent.During tracking, candidate boxes are scored and post-processed with shape penalties and a cosine window; high-confidence results can enlarge the support set.
  • Online training and tracking: Table 1 presents MAML training as the mechanism enabling rapid detector adaptation to a new domain and conversion into a tracker.Figure 5 compares training and testing losses and score maps for MAML and baseline detectors during domain adaptation.

5. Experiments

The experiments show that MAML enables rapid detector adaptation for tracking, while online updating improves performance with speed and sequence-length trade-offs. Retina-MAML and FCOS-MAML achieve competitive results across major benchmarks.

  • Offline MAML training: 0.671 AUC on OTB-100 after 5-step domain adaptation shows the MAML detector substantially outperforms standard-GD initialization.The baseline detector benefits only slightly from domain adaptation.
  • Offline MAML training: After 5-step GD, the MAML detector quickly locates targets in training and testing images, whereas the baseline makes little progress after 20 steps.The MAML loss drops rapidly, while the baseline loss remains large and can rise on testing images.
  • Learnable learning rates: KLLR improves FCOS-MAML performance in both the classification and regression branches.The method uses kernel-wise learnable learning rates to guide gradient updates.
  • Online updating strategy: 85 FPS is achieved without online updating, while enabling online updating increases performance at slightly reduced speed.Online updating both branches is reported as the best choice for short sequences, except on very long LaSOT sequences.
  • Benchmark evaluation: 0.712 AUC is achieved by Retina-MAML on OTB-100, while FCOS-MAML achieves 0.704.The paper reports Retina-MAML as the best-performing tracker ever on OTB at the time of evaluation.
  • Benchmark evaluation: FCOS-MAML performs favorably against state-of-the-art trackers on TrackingNet and LaSOT, with a gain over DiMP-18 on TrackingNet but a slight loss on LaSOT.The authors suspect straightforward online updating is unsuitable for very long sequences.

6. Conclusion

The paper proposes converting general object detectors into template-free trackers through a three-step procedure centered on offline MAML training. The resulting trackers are competitive on short-video datasets, while LaSOT performance remains an area for improvement.

  • Conclusion: The proposed three-step procedure converts a general object detector into a tracker using offline MAML training, initial-frame domain adaptation, and online updating.Offline MAML prepares the detector for quick domain adaptation and efficient online updating.
  • Conclusion: The resulting instance detector is a template-free tracker that benefits from advances in object detection.The conclusion describes the design as elegant and template-free.
  • Conclusion: Performance remains limited on LaSOT, despite competitive results on datasets with short videos.The authors plan to investigate online updating for long sequences.

A. Shared blocks in backbone network

The appendix evaluates whether the final ResNet-18 block should be shared between FCOS-MAML’s classification and regression branches. Both shared and separate-block designs are compared quantitatively.

  • Shared blocks in backbone network: FCOS-MAML uses independent copies of ResNet-18 block-4 for classification and regression in the main design.The authors state that this separation is intended for more comprehensive analysis of branch roles, not as a necessary design choice.
  • Shared blocks in backbone network: A comparison is made between trackers with shared and separate block-4 parameters, with block-4 remaining online trainable.The quantitative results are reported in Table 7.

B. Results on VOT-2019

On VOT-2019, Retina-MAML and FCOS-MAML are competitive with state-of-the-art trackers. FCOS-MAML also obtains the highest accuracy among the compared trackers.

  • Results on VOT-2019: Both Retina-MAML and FCOS-MAML achieve competitive performance against state-of-the-art trackers on VOT-2019.The evaluation compares the two trackers with five state-of-the-art trackers.
  • Results on VOT-2019: FCOS-MAML achieves the highest accuracy among the trackers evaluated on VOT-2019.The result is summarized in Table 8.

C. More powerful backbone

FCOS-MAML benefits from using a more powerful backbone, with the improvement especially clear on VOT-18.

  • C. More powerful backbone: More powerful backbones improve FCOS-MAML tracking performance.The section compares FCOS-MAML trackers using ResNet-18 and ResNet-50 backbones.
  • C. More powerful backbone: ResNet-50 is particularly beneficial on the VOT-18 benchmark.
Loading 2004.00830v1…