Source-linked AI summary
POI: Multiple Object Tracking with High Performance Detection and Appearance Feature
Fengwei Yu, Wenbo Li, Quanquan Li, Yu Liu, Xiaohua Shi, Junjie Yan
TL;DR
MOT performance is strongly affected by detection quality and learned appearance features, which many recent methods underemphasize. The paper develops high-performance detection, deep appearance features, and online and offline POI trackers, showing state-of-the-art results even with a simple online tracker. It publicly releases the detections and re-identification features for MOT2016.
Problem
Recent MOT works often focus on hand-crafted features and association algorithms while underemphasizing detection and learning-based appearance features.
Method
The paper combines high-performance detection, deep learning-based appearance features, and online and offline Person of Interest trackers.
Results
The authors report state-of-the-art MOT results, including with a very simple online tracker, and find no expected offline-tracker advantage when detection and appearance features are strong.
Takeaways & Limitations
The publicly released MOT2016 detections and deep learning-based re-identification features may support more sophisticated trackers.
Takeaways & Limitations
The trackers are not learning-based; detection-score thresholds are tuned on training sequences and transferred to similar test scenes.
Abstract
from arXiv · showhide
Detection and learning based appearance feature play the central role in data association based multiple object tracking (MOT), but most recent MOT works usually ignore them and only focus on the hand-crafted feature and association algorithms. In this paper, we explore the high-performance detection and deep learning based appearance feature, and show that they lead to significantly better MOT results in both online and offline setting. We make our detection and appearance feature publicly available. In the following part, we first summarize the detection and appearance feature, and then introduce our tracker named Person of Interest (POI), which has both online and offline version.
1 Detection
The tracker uses a high-performance Faster R-CNN detector trained with additional pedestrian data and multi-scale feature strategies. These detection optimizations significantly reduce false positives and false negatives, improving MOTA-related detection quality.
- Detection: Faster R-CNN is fine-tuned from VGG-16 on ImageNet with ETHZ, Caltech, and self-collected surveillance pedestrian data.The self-collected dataset contains 365653 boxes in 47556 frames.
- Detection: Multi-scale training, skip pooling, and multi-region strategies combine features across scales and levels, while testing uses one scale and one model.
- Detection: Detection optimization significantly decreases the sum of false positives and false negatives, which strongly affects MOTA.The evaluation is reported on the MOT16 train set in Table 1.
2 Appearance Feature
The appearance module learns a 128-dimensional representation for data association, where feature distances should distinguish identities. It trains jointly with softmax and triplet losses on a large person re-identification dataset.
- Appearance Feature: A GoogLeNet-like network extracts 128-dimensional appearance features from detection patches resized to 96 × 96.Its pool5 kernel is 3 × 3, and cosine distance measures appearance affinity.
- Appearance Feature: Appearance-feature distance is used as the affinity value for data association, with same-identity pairs intended to be closer than different-identity pairs.
- Appearance Feature: Training uses nearly 119 K patches from 19835 identities across PRW, Market-1501, VIPeR, and CUHK03, combining softmax and triplet losses.The softmax loss supports discriminative features, while the triplet loss makes same-identity cosine distances small.
3 Online Tracker
The online tracker builds motion, shape, and appearance affinities, then applies quality-aware Kuhn-Munkres matching to update tracklets. It predicts, removes, initializes, merges, and filters tracklets across frames.
- Affinity Matrix Construction: It constructs affinity by combining appearance, motion, and shape terms before Kuhn-Munkres data association.The appearance term uses the learned feature described in Section 2.
- Online Tracker: For each new frame, the tracker takes detections and previous tracklets and outputs an updated tracklet set.
- Data Association: Tracklets are divided into high- and low-quality sets using τt, matched to detections, and classified as successful or failed using τa.
- Tracklet Update: Association-success tracklets are updated with Kalman filtering and feature aggregation, while failed tracklets are predicted or removed after τm missing frames.
- Tracklet Update: Unmatched detections initialize new tracklets, candidate subsets are merged, and candidates outside the image border are removed.
4 Offline Tracker
The offline tracker improves H2T using dense-neighbor association, CNN appearance features, and scale-aware handling of big targets.
- Offline tracker: The offline tracker is an improved H2T version based on K-Dense Neighbors for complex tracking scenarios.It is described as more robust and efficient than H2T.
- Offline tracker: It divides videos into temporal segments, builds short tracklets with dense-neighbor search, and repeatedly merges nearby segments into longer tracklets.The same search associates existing tracklets after each merge.
- Improvements over H2T: The affinity matrix combines appearance, motion, and smoothness affinities, with CNN appearance features replacing H2T’s hand-crafted feature.Appearance affinity is identified as the most important of the three.
- Improvements over H2T: Two object-scale thresholds prevent mismatched associations and reduce motion and smoothness weights when large-target detections are unreliable.The thresholds are τs for scale compatibility and τr for deciding when to reduce motion and smoothness weights.
- Offline tracker: Dense-neighbor search operates on an affinity matrix encoding similarity between two tracklets.The passage points to prior work for details and comparisons with GMCP.
5 Evaluation
Evaluation compares the proposed detector and appearance feature with official and default CNN alternatives on the MOT16 train set.
- Evaluation: Detection-score thresholds are set to 0.1 for MOT16-03 and MOT16-04 and 0.3 for other sequences.The lower threshold is used for the two surveillance sequences because of their high detection precision.
- Detector comparison: On the MOT16 train set, the proposed detector improves MT, ML, FP, and FN compared with the official detector.The comparison covers both online and offline trackers.
- Appearance-feature comparison: On the MOT16 train set, the proposed appearance feature reduces IDS and FM compared with the default CNN feature.This comparison is reported for both online and offline trackers.
6 ECCV 2016 Challenge Results.
In the ECCV 2016 Challenge, both proposed trackers outperformed state-of-the-art approaches, while the offline tracker achieved the best FN but only moderate FP.
- Challenge results: The offline tracker achieves the best FN performance, but its FP performance is moderate because of the interpolation module.The passage attributes the FP limitation specifically to interpolation.
7 Conclusion
The paper concludes that high-performance detection and deep learning appearance features produce state-of-the-art MOT results even with a simple online tracker.
- Conclusion: High-performance detection and deep learning appearance features lead to state-of-the-art multiple object tracking results in the submission.The conclusion covers the paper’s central emphasis on detection and appearance representation.
- Conclusion: With strong detection and appearance features, the state-of-the-art offline tracker does not show expected advantages over the simpler online tracker.The authors note that this observation is not reported in many current MOT papers using weaker detections.
- Conclusion: The authors publicly release their MOT2016 detections and deep learning re-identification features.They hope these resources will help more sophisticated trackers achieve better performance.