Source-linked AI summary
MOT20: A benchmark for multi object tracking in crowded scenes
Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, Laura Leal-Taixé
TL;DR
MOT20 addresses the need to evaluate multi-object tracking in extremely crowded scenes with a standardized, fair benchmark. It introduces eight carefully annotated sequences and shared evaluation resources, with the release emphasizing substantially denser and more varied data for challenging trackers and detectors. The benchmark is intended to support fairer comparisons and encourage methods that generalize to unconstrained, crowded environments.
Problem
Existing tracking methods can be overfitted to particular datasets, motivating evaluation on diverse, unseen, and extremely crowded scenes.
Method
MOT20 constructs eight carefully selected and annotated sequences, provides standardized data, metrics, scripts, and public detections, and uses known and unknown scenes for evaluation.
Results
The benchmark contains circa 3 times more bounding boxes than MOT17, with mean crowd density reaching 246 pedestrians per frame.
Takeaways & Limitations
MOT20 provides a benchmark for fairer comparison of tracking methods in unconstrained and very crowded scenes.
Takeaways & Limitations
MOTP primarily quantifies detector localization accuracy and therefore provides little information about the tracker's actual performance.
Abstract
from arXiv · showhide
Standardized benchmarks are crucial for the majority of computer vision applications. Although leaderboards and ranking tables should not be over-claimed, benchmarks often provide the most objective measure of performance and are therefore important guides for research. The benchmark for Multiple Object Tracking, MOTChallenge, was launched with the goal to establish a standardized evaluation of multiple object tracking methods. The challenge focuses on multiple people tracking, since pedestrians are well studied in the tracking community, and precise tracking and detection has high practical relevance. Since the first release, MOT15, MOT16, and MOT17 have tremendously contributed to the community by introducing a clean dataset and precise framework to benchmark multi-object trackers. In this paper, we present our MOT20benchmark, consisting of 8 new sequences depicting very crowded challenging scenes. The benchmark was presented first at the 4thBMTT MOT Challenge Workshop at the Computer Vision and Pattern Recognition Conference (CVPR) 2019, and gives to chance to evaluate state-of-the-art methods for multiple object tracking when handling extremely crowded scenarios.
1 INTRODUCTION
MOT20 extends MOTChallenge with eight carefully annotated sequences from three very crowded scenes, including varied settings and data splits designed to assess generalization. It provides images, training annotations, and public detections under the established evaluation framework.
- MOTChallenge supports standardized comparison through carefully annotated datasets and clear metrics for tracking algorithms and pedestrian detectors.The framework has attracted more than 1,000 active users across five challenges, 44 sequences, 2.7M bounding boxes, and 36k seconds.
- MOT20 introduces eight novel sequences from three very crowded scenes, with pedestrian density reaching 246 per frame.
- The sequences include indoor and outdoor locations and day and night shots, with training and testing splits across known and unknown scenes.The unknown-scene test data is intended to measure detector and tracker generalization.
- MOT20 provides images for all sequences, ground-truth annotations for training, and public Faster R-CNN detections for the tracking challenge.
- The MOT20 challenges, data, current ranking, and submission guidelines are made available through the benchmark platform.
2 ANNOTATION RULES
The annotation protocol follows MOT16 and organizes relevant objects into three categories, separating trackable moving pedestrians from target-like people and vehicles or occluders. Dataset figures show the sequence views, labels, and detections supplied for the challenges.
- MOT20 follows the MOT16 protocol and accurately annotates every moving person or vehicle with a bounding box.A consistent annotation protocol is applied throughout the dataset.
- The annotations divide pertinent classes into moving pedestrians, target-like people, and vehicles or occluders.
- Figure 1 presents eight sequences from three scenes, separating training sequences from testing sequences that include known and unknown scenes.
- Figure 2 shows sequence frames, training-set ground-truth labels for all classes, and public detections from Faster R-CNN.
- Moving pedestrians visible in the field of view are benchmark targets, including people who briefly bend over or squat.Submitting algorithms are expected to track these targets.
- Static or ambiguously classified people are annotated so tracking them neither penalizes nor rewards an algorithm.
- Vehicles and other occluders are supplied for training and pedestrian-occlusion analysis but do not affect evaluation.
3 DATASETS
MOT20 introduces eight sequences from three extremely crowded scenes, with dense pedestrian coverage, standardized annotations, public detections, and prescribed file formats for benchmark participation.
- 246 pedestrians per frame is the reported density in some sequences, making the dataset substantially more crowded than previous challenges.The new data contains circa 3 times more bounding boxes than MOT17, and mean crowd density reaches 246 pedestrians per frame.
- 8 sequences from 3 scenes form the MOT20 benchmark, with half allocated to training and half to testing.
- Testing includes sequences from known and unknown scenes to measure detector and tracker generalization.One scene is reserved for test time, while testing annotations are withheld to avoid overfitting.
- The benchmark provides images, training ground truth, and public Faster R-CNN detections for tracking experiments.The public detections are obtained from a Faster R-CNN trained on the training data and serve as a baseline for trackers.
- MOT20 annotations distinguish moving pedestrians from people-like or other classes, while evaluation considers only annotated pedestrians and filters distractors and non-target classes.Static persons and other classes can be marked so they neither count as false negatives nor true positives.
- Annotation and result files use CSV records containing frame and identity information, bounding-box coordinates, confidence or activity flags, class labels, and visibility ratios.Each sequence requires a separate CSV file following the specified format, compressed into a single ZIP file for evaluation.
4 EVALUATION
The evaluation framework supports fair comparison of tracking methods by standardizing ground truth, metrics, scripts, and detections. This isolates tracker performance from other components.
- Standardized ground truth data, evaluation metrics, scripts, and precomputed detections enable comparison under identical conditions.
- The framework is designed to isolate the performance of the tracker from everything else.
4.1 Evaluation metrics
The benchmark combines established CLEAR and track-quality measures because no single metric fully captures multi-object tracking performance. Its evaluation matches tracker outputs to targets, computes error and localization measures, and reports robustness across diverse sequences.
- Metric selection: Established CLEAR metrics and track-quality measures are used because choosing one universally appropriate tracking metric remains unresolved.The benchmark makes evaluation scripts publicly available.
- Tracker-to-target assignment: Tracker outputs are matched to annotated targets so each true object is recovered at most once and each hypothesis accounts for at most one target.The matching uses the Hungarian algorithm and considers temporal correspondence rather than treating frames independently.
- Multiple Object Tracking Accuracy: MOTA combines false negatives, false positives, and identity switches into one performance measure, but can become negative when errors exceed ground-truth objects.The benchmark reports percentage MOTA over the range (−∞, 100].
- Robustness: The benchmark reports MOTA standard deviation across sequences to indicate tracker robustness under diverse and potentially unseen conditions.The dataset is designed to reduce bias and expose overfitting to particular settings.
- Multiple Object Tracking Precision: MOTP measures localization precision as the average overlap between correctly matched hypotheses and their corresponding ground-truth objects.For bounding-box overlap, MOTP ranges from the matching threshold of 50% to 100%.
- Multiple Object Tracking Precision: MOTP primarily quantifies detector localization accuracy and therefore provides little information about the tracker’s actual performance.It should not be confused with positive predictive value in precision–recall analysis.
5 CONCLUSION AND FUTURE WORK
MOT20 adds substantially more crowded sequences to MOTChallenge, broadening evaluation across varied environments and encouraging trackers that generalize to unconstrained, crowded scenes.
- MOT20 introduces sequences with substantially more targets and crowding than previous MOTChallenge releases.
- The benchmark includes indoor and outdoor scenes recorded during both daytime and nighttime.
- MOT20 is intended to support fairer comparisons and motivate more generic tracking methods for unconstrained, very crowded environments.