PaperlayerView exact revision on arXiv

Source-linked AI summary

Quadruplet Network with One-Shot Learning for Fast Visual Object Tracking

Xingping Dong, Jianbing Shen, Dongming Wu, Kan Guo, Xiaogang Jin, Fatih Porikli

arXiv:1705.07222v3cs.CV

TL;DR

One-shot visual tracking must learn from a single exemplar, while pair-based Siamese training does not fully exploit relationships among multiple samples. The paper introduces a shared-branch quadruplet network with adaptive pair-and-triplet losses, and reports superior tracking performance at 78 fps.

  • Problem

    One-shot learning from a single exemplar is difficult, and Siamese or triplet training does not fully exploit relationships among multiple training instances.

  • Method

    The quadruplet network uses four shared branches, selects most similar and dissimilar instances for triplet learning, and combines pair and triplet losses with automatically adjusted weights.

  • Results

    The tracker achieves superior visual-tracking performance with real-time processing at 78 frames-per-second.

  • Takeaways & Limitations

    Multi-tuple training and adaptive loss weighting produce more robust one-shot representations for visual object tracking.

Abstract

from arXiv · show

In the same vein of discriminative one-shot learning, Siamese networks allow recognizing an object from a single exemplar with the same class label. However, they do not take advantage of the underlying structure of the data and the relationship among the multitude of samples as they only rely on pairs of instances for training. In this paper, we propose a new quadruplet deep network to examine the potential connections among the training instances, aiming to achieve a more powerful representation. We design four shared networks that receive multi-tuple of instances as inputs and are connected by a novel loss function consisting of pair-loss and triplet-loss. According to the similarity metric, we select the most similar and the most dissimilar instances as the positive and negative inputs of triplet loss from each multi-tuple. We show that this scheme improves the training performance. Furthermore, we introduce a new weight layer to automatically select suitable combination weights, which will avoid the conflict between triplet and pair loss leading to worse performance. We evaluate our quadruplet framework by model-free tracking-by-detection of objects from a single initial exemplar in several Visual Object Tracking benchmarks. Our extensive experimental analysis demonstrates that our tracker achieves superior performance with a real-time processing speed of 78 frames-per-second (fps).

I. INTRODUCTION

The paper targets one-shot learning settings where labeled data are limited and conventional pair- or triplet-based training underuses relationships among instances. It proposes a quadruplet network with adaptive loss weighting and applies it to fast visual tracking.

  • Motivation: One-shot learning is challenging because deep networks typically require large labeled corpora, whereas the task learns from a single exemplar.
  • Motivation: Siamese and triplet networks use pairs or triplets, but the authors argue that larger tuples can better exploit relationships among training instances.
  • Method: The quadruplet network uses four shared branches for exemplar, instance, positive, and negative inputs, combining pair loss with a triplet-style contractive loss.
  • Contribution: The method is introduced as a quadruplet network for one-shot learning and applied to single-object tracking.
  • Method: A weight layer automatically adjusts the combination between triplet and pair loss during training instead of relying on manually selected weights.
  • Results: 78 frames-per-second is reported for the tracker, which achieves state-of-the-art results without online updating during tracking.

II. RELATED WORK

The related work spans generative and embedding-based one-shot learning and visual tracking methods based on correlation filters or deep networks. The paper positions its approach as an offline-training improvement to Siamese tracking while retaining real-time operation.

  • One-shot learning: One-shot learning research includes generative models and discriminative embedding methods such as Siamese networks.
  • Visual tracking: Visual object tracking can be formulated as one-shot learning, with correlation-filter and deep-neural-network methods forming major lines of work.
  • Deep tracking: The quadruplet tracker focuses on improving offline training to obtain a more robust feature representation while maintaining speed comparable to SiamFc-3s.
  • Speed comparison: The tracker operates at 78 fps, while the cited related methods include real-time systems above 30 fps and nearly real-time systems above 20 fps.
  • Deep tracking: SiamFc-3s uses fully convolutional Siamese layers for tracking without frame-by-frame online updating and reports nearly 86 fps, or 78 fps on the authors' machine.

III. SIAMESE NETWORKS FOR TRACKING

Siamese tracking embeds an exemplar and candidate patches with shared branches, then selects the most similar candidate in the learned embedding space. Its pairwise similarity function underlies the tracking formulation.

  • Tracking formulation: Tracking uses the first-frame object patch as an exemplar and patches from later search regions as candidate instances in an embedding space.
  • Siamese architecture: A fully convolutional Siamese network processes exemplar and instance inputs through two parameter-sharing branches to support real-time tracking.
  • Similarity function: The similarity function is f(z, x) = g(φ(z), φ(x)), where shared transformation φ maps the exemplar and instance before metric g compares them.
  • Pairwise training: Pairwise training assigns labels y[u] ∈ {+1, −1} to exemplar-instance pairs and uses their similarity scores v[u] = f(z, u).
  • Quadruplet extension: The proposed quadruplet architecture extends Siamese and triplet designs with four shared branches connected by a common loss function.

A. Network branches

The quadruplet network uses four shared branches and combines pairwise and triplet losses to exploit relationships among multiple instances. A learned weight layer balances these losses during training.

  • Network branches: Four shared branches process exemplar, instances, positive, and negative inputs within the quadruplet network.The branches apply an identical transformation to different inputs before similarity scoring.
  • Pair loss: The pair loss compares the exemplar with instances using weighted logistic loss over positive and negative labels.Instance weights are positive, normalized, and may be changed during precomputation.
  • Triplet loss: The triplet loss uses exemplar, positive, and negative branches to compare soft-max similarity scores against the desired ordering.Positive and negative inputs are selected according to similarity scores from the instance set.
  • Loss combination: The final loss is a weighted sum of pair and triplet losses, with combination weights learned during training rather than fixed beforehand.The loss layer receives L1 and L2 and optimizes the branch parameters and loss weights with SGD.
  • Training iteration: Precomputation selects feature maps and adapts pair-loss weights before forward and backward training computations.The positive and negative feature maps are selected from instance feature maps, while the exemplar feature is updated during backpropagation.

C. Framework of our tracker

The tracker trains a powerful embedding offline with the quadruplet network, then performs online detection by comparing an exemplar with search-image sub-windows. The maximum of an upsampled similarity map determines the object location.

  • Offline and online stages: Quadruplet training learns an embedding representation offline, while online testing uses the exemplar and instances branches.The shared convolutional architecture follows SiamFc-3s during offline training.
  • Training inputs: Training uses an exemplar and larger search image, labeling central score-map locations positive and other locations negative.The positive area uses radius R = 2 in the experiments.
  • Instance selection: Before each iteration, the tracker selects the center instance and highest-scoring negative-area instance for triplet loss.The precomputed score map also constructs the weighted pair loss.
  • Online localization: During tracking, the score map is upsampled from 17 × 17 to 272 × 272 and the maximum score gives the object location.The exemplar and search images are resized to 127 × 127 and 255 × 255, respectively.

A. Implementation details

The implementation trains the shared network and loss weights with SGD on sampled video pairs, then performs online tracking using a fixed exemplar embedding and multi-scale search.

  • Training: Training minimizes the combined loss with SGD, initializing the shared network from SiamFc-3s and loss weights at (0.9, 0.1).The training and validation sets match those used by SiamFc-3s.
  • Training data: The training data contains almost 4500 videos across 30 animal and vehicle classes, with 10 epochs of 53,200 sampled pairs each.Ten percent of pairs are randomly selected for validation at each epoch.
  • Optimization: Mini-batches contain 8 pairs, the learning rate decays from 10^-2 to 10^-5, and 25% of pairs are converted to grayscale.The final testing network is selected using minimum validation mean distance error.
  • Tracking: Online tracking computes the exemplar embedding once and searches three scales, updating scale by linear interpolation with factor 0.59.The reported hardware is an Intel Core i7-6700 with a single NVIDIA GeForce 1080.

B. Benchmarks and evaluation metric

The tracker is evaluated against recent methods on OTB-2013, OTB-50, OTB-100, and VOT2015 using overlap, precision, accuracy, robustness, and EAO measures.

  • Benchmarks: Evaluation covers OTB-2013, OTB-50, OTB-100, and VOT2015 against recent state-of-the-art trackers.OTB-2013 has 50 sequences, OTB-100 has 100, OTB-50 contains 50 challenging sequences, and VOT2015 has 60 sequences.
  • OTB metrics: OTB evaluation uses overlap success rate at IoU threshold 0.5 and distance precision at threshold 20.Success measures the fraction of predicted boxes exceeding an IoU threshold, while precision measures locations within a distance threshold.
  • VOT metrics: VOT2015 evaluates short-term tracking with accuracy, robustness, and Expected Average Overlap.EAO combines accuracy and robustness, while tracker restarts are used in the benchmark protocol.
  • Evaluation protocols: Figure 5 reports success and precision plots under OPE, SRE, and TRE evaluation protocols on OTB-2013.These protocols vary initialization through one-pass, spatial, and temporal robustness evaluations.

C. The OTB-2013 benchmark

Quad outperforms recent real-time trackers across OTB evaluation protocols and challenging attributes, while improving over the SiamFc-3s baseline. Its gains extend across OTB-50 and OTB-100 success and precision measures.

  • Quad outperforms recent real-time trackers in overlap success rate and precision under OPE, SRE, and TRE on OTB-2013.
  • Quad ranks fourth in TRE precision at 0.828, nearly matching SANF at 0.828 and MEEM at 0.829.
  • Quad outperforms other trackers on 8 of 9 OTB-100 challenge attributes, especially low resolution and in-plane rotation.
  • Quad outperforms SiamFc-3s in all evaluated OTB-100 attribute subsets.
  • On OTB-50, Quad reaches 0.7117 precision and raises success from 0.542 to 0.660; on OTB-100, it reaches 0.782 precision and raises success from 0.710 to 0.739.

E. Qualitative Evaluation

Qualitative and VOT-2015 evaluations show Quad handling deformation, fast motion, rotation, and occlusion while maintaining high speed. It ranks competitively against both real-time and higher-accuracy trackers.

  • Qualitative Evaluation: Quad tracks challenging OTB-100 sequences more accurately than Staple, CFnet-conv2, and SiamFc-3s in precision or overlap.
  • Qualitative Evaluation: Quad alone succeeds throughout Lemming, where rotation and occlusion challenge the compared trackers.
  • VOT-2015 Evaluation: Quad achieves the best EAO and highest accuracy among VOT-2015 trackers exceeding 15 fps.
  • VOT-2015 Evaluation: Quad ranks 7th of 64 in VOT-2015 EAO while the six higher-ranked trackers operate below 5 fps.
  • VOT-2015 Evaluation: Quad runs at 78 fps, faster than the higher-ranked VOT-2015 trackers and beyond real-time speed.

G. Ablation Study

The ablation study shows that learned loss weighting is necessary for combining pair and triplet losses effectively. Quad improves both precision and overlap success over the baseline and fixed-weight variants.

  • SiamInit improves precision but reduces overlap success rate relative to SiamFc-3s, showing that additional training alone is insufficient.
  • Adaptive weighted pair loss slightly improves overall performance by increasing overlap while slightly reducing precision versus SiamInit.
  • Fixed weights for pair and triplet losses can reduce performance because their prior combination may be suboptimal.
  • The learned weight layer enables Quad to improve both precision and overlap success over the fixed-weight QuadC variant.
  • The quadruplet network uses multi-tuples to mine instance relationships and produce more robust one-shot-learning representations.
Loading 1705.07222v3…