Source-linked AI summary
Learning to Track at 100 FPS with Deep Regression Networks
David Held, Sebastian Thrun, Silvio Savarese
TL;DR
Generic trackers commonly train online and cannot exploit large collections of offline videos. GOTURN learns appearance–motion relationships offline with a feed-forward regression network, tracking novel objects without online fine-tuning at 100 fps. It outperforms previous trackers on a standard benchmark, and performance improves with additional training videos.
Problem
Generic object trackers are commonly trained from scratch online instead of using readily available offline videos.
Method
GOTURN trains a neural network offline to learn a generic relationship between object appearance and motion, then uses frozen weights to track novel objects.
Results
GOTURN runs at 100 fps and outperforms previous trackers on a standard benchmark.
Takeaways & Limitations
Adding offline training videos improves tracking performance, including for novel objects absent or sparsely represented in training.
Takeaways & Limitations
The tracker is constrained when targets become occluded or move too quickly; larger search regions increase network complexity.
Abstract
from arXiv · showhide
Machine learning techniques are often used in computer vision due to their ability to leverage large amounts of training data to improve performance. Unfortunately, most generic object trackers are still trained from scratch online and do not benefit from the large number of videos that are readily available for offline training. We propose a method for offline training of neural networks that can track novel objects at test-time at 100 fps. Our tracker is significantly faster than previous methods that use neural networks for tracking, which are typically very slow to run and not practical for real-time applications. Our tracker uses a simple feed-forward network with no online training required. The tracker learns a generic relationship between object motion and appearance and can be used to track novel objects that do not appear in the training set. We test our network on a standard tracking benchmark to demonstrate our tracker's state-of-the-art performance. Further, our performance improves as we add more videos to our offline training set. To the best of our knowledge, our tracker is the first neural-network tracker that learns to track generic objects at 100 fps.
1 Introduction
GOTURN addresses the limitations of online-trained generic trackers by learning object appearance–motion relationships offline. It tracks novel objects without fine-tuning and reaches 100 fps while outperforming prior approaches on a standard benchmark.
- Motivation: Generic trackers traditionally train entirely online, preventing them from exploiting the many available offline training videos.Offline videos can teach trackers to handle rotations, viewpoint changes, lighting changes, and other challenges.
- Contribution: GOTURN trains a neural network offline to track generic objects, with frozen weights and no online fine-tuning for novel objects.The method learns a generic relationship between appearance and motion from videos and images with bounding-box labels but no class information.
- Efficiency: 100 fps is achieved through offline training and a regression-based single feed-forward pass rather than online training or classification of many patches.These design choices produce a significant speed-up over previous neural-network tracking approaches.
- Evaluation: GOTURN outperforms previous trackers on a standard tracking benchmark while training on labeled videos and images without class-level object information.The benchmark result supports the proposed generic offline-training framework.
2 Related Work
Prior tracking methods either train online, limiting speed, or specialize to object classes. GOTURN instead compares two image crops and directly regresses the target location, enabling 100 fps tracking.
- Online training: Traditional generic trackers train foreground–background classifiers online from patches sampled around the target and background.This online procedure is used to estimate the target’s location in the next frame.
- Online training: Previous neural-network trackers that train online run from 0.8 fps to 15 fps, with top-performing examples running at 1 fps on a GPU.The cited methods are therefore much slower than real-time tracking requirements described by the paper.
- Model-based tracking: Model-based trackers train offline but are limited to specific object classes, whereas GOTURN is trained offline for generic object tracking.The distinction is between class-specific detectors and a tracker intended for generic objects.
- Neural tracking frameworks: Patch-matching trackers evaluate many candidate patches, while GOTURN passes only two images and directly regresses the target bounding-box location.Avoiding candidate-patch scoring enables the reported 100 fps speed.
3 Method
GOTURN tracks a target by comparing a crop from the previous frame with a search region from the current frame. A convolutional and fully connected network regresses the target box without online fine-tuning.
- Method overview: The tracker is trained entirely offline on video sequences and images, then successively outputs the tracked object’s location in each frame.At test time, it uses no online training and can track novel objects.
- Input / output format: The previous-frame target crop identifies which object to track and provides contextual information around it.The network tracks whatever object is supplied in this crop, including novel objects.
- Input / output format: The current-frame search region is centered on the previous predicted location, using object motion smoothness to restrict where the target is sought.The network then regresses the target location within this region.
- Input / output format: The method uses k1 = k2 = 2 for the target crop and current-frame search region in practice.These factors determine the contextual crop and search radius relative to the previous bounding-box size.
- Limitations: The tracker is reliable when the target is not occluded and is not moving too quickly; larger search regions increase network complexity.Long occlusions or large movements may require combination with another approach, which the authors leave for future work.
- Network architecture: The network compares convolutional features from the target and current-frame crops through fully connected layers to find the target’s displacement.Its output is the top-left and bottom-right coordinates of the current bounding box.
4 Training
GOTURN is trained offline with successive video frames and augmented still-image crops, using an L1 bounding-box loss. Laplace-based cropping teaches preference for small motion and limited size changes.
- Training from Videos and Images: The network is trained with an L1 loss on predicted versus ground-truth bounding boxes using video pairs and still-image crops.Video pairs encode object motion between successive frames, while still images provide apparent translation and scale changes.
- Training from Videos and Images: Still-image training broadens object diversity and helps prevent overfitting to objects in the training videos.Random crops are treated as if they came from different video frames, although their motions are less varied than real video motions.
- Learning Motion Smoothness: Bounding-box size changes are modeled with mean-one Laplace variables, favoring sizes near those in the previous frame.The current width and height are obtained by scaling the previous dimensions with γw and γh.
- Learning Motion Smoothness: Laplace-distributed crop sampling favors small object motions because smaller changes are sampled more often than larger changes.The motion variables ∆x and ∆y have mean-zero Laplace distributions, whose scale parameters are selected by cross-validation.
- Training Procedure: Training alternates between randomly selected video examples and image examples, with additional random crops used for augmentation.The convolutional layers are ImageNet-pretrained and kept fixed to reduce overfitting given the limited training set.
5 Experimental Setup
The experiments train on labeled videos and still images, then evaluate on 25 VOT 2014 videos using accuracy and robustness measures. The benchmark also reports attribute-specific and overall rankings.
- Training Data: The video training set contains 307 ALOV300++ sequences after removing seven videos that overlap with the test set.Approximately every fifth frame in each training video is labeled with the tracked object’s location.
- Training Data: The still-image training data comes from ImageNet Detection, where 478,807 objects have labeled bounding boxes.Random crops create apparent translation and scale changes for training.
- Test Set: The test set contains 25 videos from the VOT 2014 Tracking Challenge.The authors did not evaluate on VOT 2015 because of excessive overlap between its test set and the training data.
- Evaluation Metrics: VOT evaluates trackers with accuracy and robustness metrics ranging from 0 to 1, along with corresponding errors and an overall error.The overall error is defined as 1 − (A + R)/2.
- Evaluation Metrics: Trackers are ranked separately across occlusion, illumination, motion, size, and camera-motion attributes, then averaged into final accuracy, robustness, and overall rankings.Attribute-level rankings are averaged across attributes before the final rankings are computed.
6 Results
GOTURN achieves strong benchmark performance while operating at real-time speed, and its behavior reflects both temporal comparison and generic object detection. More offline training improves generalization and specialization, although failures remain under occlusion and training-set overfitting.
- 6.1 Overall performance: GOTURN outperformed all previous trackers on VOT 2014 overall rank, the average of accuracy and robustness ranks.It showed good robustness and near-top accuracy after training on 307 short videos.
- 6.1 Overall performance: 100 fps was achieved on a GTX 680 through offline training and a single feed-forward pass at test time.The tracker ran at 165 fps on a GTX Titan X and 100 fps on a GTX 680.
- 6.1 Overall performance: The tracker was among the fastest of 38 VOT 2014 baselines while achieving the best overall rank.Runtime was compared using EFO units to normalize hardware differences.
- 6.2 How does it work?: Previous-frame input helps most during occlusion and large camera motion, while current-frame detection dominates under large size changes or no variation.Without the previous frame, performance worsens substantially in occlusion and large-camera-motion cases but is slightly better or similar for size change and no variation.
- 6.3 Generality vs Specificity: Performance improves with more training videos for both novel and familiar object classes, enabling generic tracking and application-specific specialization.Class labels are used only for analysis; at least 25 same-class training videos yield a large improvement for familiar classes.
- 6.4 Design analysis: Laplace-distributed crops reduced errors by 20%, while replacing L2 with L1 loss reduced overall tracking errors from 0.43 to 0.24.The crop distribution encourages preference for small motions, and L1 better penalizes slightly incorrect bounding boxes.
- 6.4 Design analysis: Training on videos improved performance more than training on images alone, while combining videos and images produced the maximum performance.Small numbers of labeled videos taught invariance to background motion, out-of-plane rotations, deformations, lighting changes, and minor occlusions.
7 Conclusions
The paper demonstrates that generic object tracking can be trained offline and run purely feed-forward at 100 fps. The learned appearance–motion relationship improves with more training videos and supports tracking novel objects in real time.
- 7 Conclusions: Offline training lets GOTURN improve by watching more training videos without online fine-tuning during test time.The network runs in a purely feed-forward manner.
- 7 Conclusions: GOTURN learns a generic relationship between object appearance and motion that supports real-time tracking of novel objects.The tracker runs at 100 fps.
A Offline training
Offline training improves tracking as more videos are added, with large gains possible from expanding the labeled training set.
- Large tracking gains could be achieved by labeling more training videos.
- Tracking errors drop dramatically as the number of offline training videos increases.The reported state-of-the-art results used only 307 short videos, averaging 52 annotations per video.
B Online training
Offline training makes online training unnecessary for comparable performance, while online adaptation can trade accuracy for robustness through overfitting.
- 100 fps tracking is enabled by offline training, which removes the need for slow online neural-network training.Previous neural-network trackers trained online at 0.8–15 fps, with top-performing methods running at 1 fps.
- Online training does not significantly improve performance beyond offline training.The online-trained version achieves approximately the same performance as the offline-only tracker.
- Online training can slightly increase robustness while reducing accuracy because it overfits to the first few video frames.Such overfitting limits generalization to new deformations or viewpoint changes.
- Offline training has taught the network to track varied objects without requiring online fine-tuning.
C Generality vs Specificity
GOTURN generalizes accuracy to novel object classes, while robustness benefits more from training on classes represented in the training data.
- The tracker can generalize to novel objects, including objects with few or no similar training examples.
- More examples of a particular object class can improve tracking for applications where that class is prevalent.
- Accuracy errors decrease similarly for represented and unrepresented object classes as training videos increase.
- Robustness errors decrease more for object classes represented in the training set than for unrepresented classes.Robustness generalization is more limited in difficult tracking situations involving unknown-object motion.
- The tracker accurately estimates object borders with bounding boxes but has difficulty tracking unknown-object motion in challenging situations.The paper identifies labeling more videos or learning from unlabeled videos as future directions for improving robustness.
D Speed analysis
The tracker combines high benchmark ranking with real-time speed, while its motion model uses observed training-set movement statistics to guide random cropping.
- 100 fps: the tracker has the top overall rank, the second-highest accuracy rank, and one of the top robustness ranks.The overall rank averages accuracy and robustness ranks.
- The tracker is one of the fastest methods among 38 VOT 2014 baseline trackers while outperforming them in overall rank.
- The network’s tracking behavior is analyzed using motion comparison and local generic-object-detection hypotheses.The comparison removes the previous frame to distinguish whether the network relies on frame-to-frame comparison or nearest-object localization.
- Random cropping uses Laplace distributions fitted to training-set object motion to encode that small motions are more likely than large motions.
- Three fully connected layers perform best, although performance is similar with two, three, or four layers.
H Data augmentation
The study varies how many on-the-fly augmented images are included in each training batch. With a batch size of 50, performance is best with 49 augmented images and remains similar when more than 20 are used.
- Augmentation setup: Each batch contains 50 images, with 0–49 created as augmented images so at least one real image remains.New augmented images are generated on-the-fly for every batch.
- Performance: 49 augmented images per batch produces the best tracking performance, leaving only 1 real image.The augmented images show the same examples as real images, with the target object translated or w…
- Performance: Performance is similar for all settings with more than 20 augmented images per batch.For a batch size of 50, this corresponds to using at least 40% augmented images.