Source-linked AI summary
Learning Multi-Domain Convolutional Neural Networks for Visual Tracking
Hyeonseob Nam, Bohyung Han
TL;DR
Visual tracking needs representations trained on varied video data, but sequence-specific targets, backgrounds, and conditions complicate unified CNN learning. MDNet learns shared representations with domain-specific branches, then adapts a new classification layer online; it reports outstanding performance on OTB and VOT2014, with failures under appearance changes.
Problem
Visual tracking has limited CNN-specific training data, and differing targets, backgrounds, and sequence conditions make unified representation learning difficult.
Method
MDNet pretrains a CNN with shared layers and sequence-specific binary-classification branches, then fine-tunes shared fully connected layers and a new classification layer online.
Results
MDNet achieved outstanding performance compared with state-of-the-art tracking algorithms on the OTB and VOT2014 benchmarks.
Takeaways & Limitations
The framework learns domain-independent representations during pretraining and captures domain-specific information adaptively during tracking.
Takeaways & Limitations
The tracker can drift after slight appearance changes and completely miss targets after dramatic appearance changes.
Abstract
from arXiv · showhide
We propose a novel visual tracking algorithm based on the representations from a discriminatively trained Convolutional Neural Network (CNN). Our algorithm pretrains a CNN using a large set of videos with tracking ground-truths to obtain a generic target representation. Our network is composed of shared layers and multiple branches of domain-specific layers, where domains correspond to individual training sequences and each branch is responsible for binary classification to identify the target in each domain. We train the network with respect to each domain iteratively to obtain generic target representations in the shared layers. When tracking a target in a new sequence, we construct a new network by combining the shared layers in the pretrained CNN with a new binary classification layer, which is updated online. Online tracking is performed by evaluating the candidate windows randomly sampled around the previous target state. The proposed algorithm illustrates outstanding performance compared with state-of-the-art methods in existing tracking benchmarks.
1. Introduction
Visual tracking has not fully benefited from CNNs because tracking data and specialized training methods are limited, while sequence variation makes unified representation learning difficult. MDNet addresses this by separating shared and domain-specific information, then adapting online to new sequences.
- CNN-based visual tracking remains constrained by limited large-scale video data and the mismatch between object classification and target localization.
- Different sequences vary in targets, backgrounds, appearances, motion patterns, and challenges such as occlusion, deformation, lighting changes, and motion blur.
- MDNet uses shared layers for common target representations and separate domain-specific branches for binary target-background classification.
- The network trains each sequence domain separately and iteratively while updating shared layers to learn domain-independent information.
- During tracking, pretrained shared layers are combined with a new classification branch that is fine-tuned online for the test sequence.
- MDNet achieved outstanding performance against state-of-the-art trackers on the Object Tracking Benchmark and VOT2014.
2. Related Work
Prior visual tracking work includes generative, discriminative, correlation-filter, and CNN-based approaches. MDNet places CNN pretraining within a multi-domain learning framework that incorporates domain information from multiple training sources.
- Generative trackers model target appearance, whereas discriminative trackers distinguish targets from background.
- Correlation filters became prominent because they offer computational efficiency with competitive tracking performance.
- CNNs have achieved strong visual representation performance in classification and detection, often using large-scale pretraining followed by task-specific fine-tuning.
- Earlier CNN trackers were limited by predefined object classes, scarce training data, or reduced accuracy after online learning.
- Multi-domain learning incorporates domain information when training data originate from multiple domains.
- MDNet applies this framework to tracking by using multiple sequence-specific branches alongside shared CNN layers.
3. Multi-Domain Network (MDNet)
MDNet uses a compact CNN with shared layers and sequence-specific classification branches to learn target-background representations across domains. Its training procedure updates one domain branch at a time while extracting generic information in shared layers.
- 3.1. Network Architecture: The network accepts 107×107 RGB inputs and contains three convolutional, two shared fully connected, and K domain-specific classification branches.Each branch performs binary target-background classification with softmax crossentropy loss.
- 3.1. Network Architecture: The architecture is smaller than AlexNet and VGG-Nets because tracking requires binary classification, precise localization, small inputs, and online efficiency.
- 3.2. Learning Algorithm: The learning objective is to retain common target properties across domains despite differing notions of target and background.Examples of common properties include robustness to illumination changes, motion blur, and scale variations.
- 3.2. Learning Algorithm: SGD handles one domain exclusively per iteration, enabling shared layers to model domain-independent information and produce generic target representations.
- 3.2. Learning Algorithm: Hard negative mining identifies difficult negative samples as the network becomes more discriminative during training.
4. Online Tracking using MDNet
MDNet adapts a pretrained multi-domain CNN to a new sequence by replacing domain-specific branches with a new online-updated classifier. Tracking samples candidates around the previous state, updates the model using short- and long-term samples with hard-negative mining, and refines localization with bounding-box regression.
- MDNet replaces pretrained domain-specific branches with a single new classification branch and fine-tunes it with shared fully connected layers for the test sequence.
- Long-term updates use positives collected over extended periods, while short-term updates respond to potential failures using recent positives.Negative samples are used in both update modes.
- Each frame evaluates N sampled target candidates and selects the candidate with the maximum positive network score.Candidates are sampled around the previous target state.
- Hard-negative mining selects the highest-scoring negatives for minibatches, emphasizing difficult distractors during online learning.The method tests many negative samples and retains a predefined number of hard examples without explicitly running a detector.
- Bounding-box regression refines target locations when the estimated target is reliable, using a linear model trained from conv3 features in the first frame.The regression model is trained only once because online updating is time-consuming and may be risky.
- The implementation fixes convolutional layers during tracking and updates fully connected layers, balancing computational efficiency with reduced overfitting.The algorithm initializes the final layer for a new sequence and updates the fully connected layers online.
5. Experiment
The proposed tracker is evaluated on OTB and VOT2014, including benchmark comparisons, internal component studies, challenge attributes, qualitative sequences, and failure cases. MDNet outperforms competing methods overall but still encounters drift and complete misses under substantial appearance changes.
- Evaluation on OTB: The OTB evaluation uses 100 annotated videos and compares MDNet with established trackers using center location error and bounding box overlap ratio.The comparison includes MUSTer, CNN-SVM, MEEM, TGPR, DSST, KCF, SCM, and Struck.
- Evaluation on OTB: MDNet significantly outperforms state-of-the-art trackers on OTB in both center location error and bounding box overlap ratio.It handles challenging situations including low resolution, where trackers based on low-level features are unsuccessful.
- Internal comparisons: Each component improves performance: MDNet outperforms versions without bounding box regression or without both bounding box regression and hard negative mining.The internal comparisons evaluate multi-domain pretraining against single-domain learning and remove individual tracking components.
- Evaluation on VOT2014: MDNet ranks first overall on VOT2014, placing first in accuracy and first or second in robustness, with better accuracy than all other methods despite fewer re-initializations.The VOT2014 evaluation reports average accuracy and robustness scores and ranks under baseline and region-noise initialization settings.
- Evaluation on VOT2014: MDNet works with imprecise re-initializations, while qualitative and attribute analyses examine challenging sequences, visual attributes, and failure cases.Failure cases include drift after slight appearance change in Coupon and a complete miss after dramatic appearance change in Jump.
6. Conclusion
MDNet learns domain-independent target representations through multi-domain CNN pretraining and captures domain-specific information online. It achieves outstanding performance on the OTB and VOT2014 tracking benchmarks compared with state-of-the-art algorithms.
- MDNet is a CNN-based tracking algorithm trained in a multi-domain learning framework.
- Offline pretraining learns domain-independent representations, while online learning captures domain-specific target information.The fully connected layers, including a single domain-specific layer, are fine-tuned online.
- MDNet achieved outstanding performance on the OTB and VOT2014 public tracking benchmarks compared with state-of-the-art algorithms.