Source-linked AI summary
MOT16: A Benchmark for Multi-Object Tracking
Anton Milan, Laura Leal-Taixe, Ian Reid, Stefan Roth, Konrad Schindler
TL;DR
MOT15’s inconsistent annotations, overfitted sequences, metric differences, and missing standardized data hinder fair comparison of multi-target tracking methods. The paper introduces MOT16, with strictly consistent annotations, broader object classes, and substantially more challenging sequences, including over three times as many training and testing bounding boxes as MOT15.
Problem
Inconsistent ground truth, varying evaluation metrics, and a lack of predefined training and test data make fair comparison of multi-target tracking methods difficult.
Method
The paper presents the MOT16 benchmark, created with a strict annotation protocol applied consistently across sequences and extended to multiple object classes and visibility-related information.
Results
Over 3 times more bounding boxes and 3 times higher mean crowd density than MOT15 make MOT16 a larger and more challenging benchmark, while MOTChallenge methods had improved accuracy by over 10%.
Takeaways & Limitations
MOT16 provides a fairer comparison of state-of-the-art tracking methods and challenges researchers to develop methods that perform well in unconstrained environments and on unseen data.
Takeaways & Limitations
MOT15 used inconsistent annotation protocols and included easy, well-known sequences to which methods were overfitted, limiting its suitability for training.
Abstract
from arXiv · showhide
Standardized benchmarks are crucial for the majority of computer vision applications. Although leaderboards and ranking tables should not be over-claimed, benchmarks often provide the most objective measure of performance and are therefore important guides for reseach. Recently, a new benchmark for Multiple Object Tracking, MOTChallenge, was launched with the goal of collecting existing and new data and creating a framework for the standardized evaluation of multiple object tracking methods. The first release of the benchmark focuses on multiple people tracking, since pedestrians are by far the most studied object in the tracking community. This paper accompanies a new release of the MOTChallenge benchmark. Unlike the initial release, all videos of MOT16 have been carefully annotated following a consistent protocol. Moreover, it not only offers a significant increase in the number of labeled boxes, but also provides multiple object classes beside pedestrians and the level of visibility for every single object of interest.
1 INTRODUCTION
MOT16 addresses inconsistent data, annotation, and evaluation practices that hinder fair comparison in multi-target tracking. It introduces a more challenging benchmark with standardized annotations and evaluation-oriented goals.
- Motivation: MOT15 comparisons were hindered by inconsistent annotations, unbalanced crowd densities, overfitted sequences, and weak provided detections.These shortcomings could make results difficult to compare fairly and reduce the benchmark’s usefulness for training.
- MOT16 benchmark: MOT16 introduces 14 sequences with more crowded scenarios, varied viewpoints, camera motions, and weather conditions.The release aims to increase difficulty relative to the initial benchmark.
- Annotation protocol: All MOT16 sequences were annotated from scratch by qualified researchers under a strict protocol and double-checked for accuracy.The annotations include pedestrians, vehicles, sitting people, occluding objects, and other significant classes, enabling visibility and cropping analysis.
- Paper goals: The paper’s goals are to present MOT16 for fair evaluation, document its annotation protocol, and expose strengths and weaknesses of state-of-the-art trackers.The benchmark uses shared ground truth and evaluation methodology to support principled comparisons.
2 ANNOTATION RULES
MOT16 defines a consistent protocol for annotating moving people and vehicles, separating evaluation targets from ambiguous objects and training or occlusion-support classes. The rules specify tight object boxes, track boundaries, IDs, and comprehensive treatment of cropping and occlusion.
- Object classes: Annotations cover moving or standing upright pedestrians, including people on bikes or skateboards, as the benchmark’s central tracking targets.Briefly bending or squatting does not change a person’s standard pedestrian class.
- Object classes: Ambiguous people-like objects, including sitting people, mannequins, pictures, reflections, and people behind glass, are annotated as distractors rather than evaluation targets.The evaluation is designed neither to penalize nor reward algorithms for recovering or missing these cases.
- Object classes: Moving vehicles and other potential occluders are annotated for training and pedestrian-occlusion computation, but they do not contribute to evaluation.Static vehicles are omitted unless they occlude pedestrians.
- Bounding box alignment: Bounding boxes tightly contain the object’s full extent, extend beyond image borders for cropped people, and may be split across multiple boxes for irregular occluders.The annotation excludes unrelated pixels and uses multiple boxes when one box cannot accurately represent an occluder.
- Track boundaries and identity: Tracks begin when roughly 10% of a person is visible and end when precise localization is no longer possible; reappearances after ambiguous occlusions receive new IDs.A person leaving and later reappearing in the field of view is also assigned a new ID.
- Completeness and occlusion: The dataset annotates all distinguishable target sizes and performs a detector-based sanity check to add missed humans or distractors.Occlusion levels are computed automatically from annotations and a ground-plane assumption rather than explicitly labeled.
3 DATASETS
MOT16 compiles 14 varied sequences with richer annotations and substantially denser data than MOT15. It standardizes data formats, supplies detections, and withholds test annotations to reduce overfitting.
- Dataset composition: 14 sequences are split evenly between training and testing, with test annotations withheld to avoid overfitting.
- Sequence diversity: The sequences vary by camera motion, viewpoint, and weather conditions, including static or moving cameras and high, medium, or low viewpoints.
- Dataset scale: MOT16 contains almost 3 times more bounding boxes and has 3 times higher mean crowd density than MOT15.
- Annotation scope: Annotations cover pedestrians plus vehicles, bicycles, sitting people, occluding objects, and other object classes.
- Provided detections: DPM detections are released because DPM provides the highest recall at the best precision among the evaluated pedestrian detectors.
- Data format: Detection files use one CSV row per object instance, encoding frame, identity, bounding-box geometry, confidence, and reserved fields.
4 EVALUATION
MOT16 evaluates trackers through standardized benchmark components intended to make comparisons fair and transparent. It provides common ground truth, metrics, scripts, detections, and documented file formats.
- Standardized evaluation: The framework compares tracking methods under the same conditions using standardized ground truth, evaluation metrics, scripts, and precomputed detections.
- Detector inputs: Figure 3 compares three pedestrian detectors on training and test sets, marking the operating point used as tracker input.
4.1 Evaluation metrics
MOT16 evaluates trackers through standardized matching, error measures, localization measures, and trajectory-recovery measures. Its metric suite captures multiple error types but does not reduce tracking quality to one unqualified number.
- The benchmark uses CLEAR metrics and track quality measures, with publicly available evaluation scripts.
- Tracker-to-target assignment: Tracker evaluation first identifies true positives, false positives, and false negatives, then establishes one-to-one correspondences between targets and hypotheses.Matching uses a distance threshold, and the Hungarian algorithm finds the optimal assignment.
- Tracker-to-target assignment: ID switches occur when a target’s assigned track changes, while fragmentation counts interruptions followed by later reacquisition.These cases are illustrated through tracker-to-target assignments in Fig. 4.
- Target-like annotations: Target-like classes are excluded from penalties and rewards, and result boxes overlapping them by more than 50% are removed before pedestrian-only evaluation.These classes include distractors, static people, reflections, and people on vehicles.
- Multiple Object Tracking Accuracy: MOTA combines false negatives, false positives, and ID switches relative to ground-truth objects, but the score can be negative and is insufficient alone as a universal measure.MOTA is reported as a percentage, while the authors note that one score may not capture individual algorithmic errors.
- Multiple Object Tracking Precision: MOTP measures localization precision as the average overlap of correctly matched hypotheses with ground-truth objects, ranging from 50% to 100%.The authors caution that MOTP mostly quantifies detector localization accuracy and provides little information about tracking performance.
5 BASELINE METHODS
The baseline suite covers graph-based, continuous-energy, motion-based, tracking-by-detection, and probabilistic data-association approaches. Baselines use publicly available implementations with benchmark-specific parameter selection, while reported values may not be optimal for every method.
- Baseline results use publicly available code and common training procedures, but may not represent each method’s best possible performance.
- Parameters are tuned through 20 independent training-set runs, and the setting with the highest MOTA is evaluated once on the test set.The selected parameter set is reported with each baseline on the benchmark website.
- Network-flow methods: Network-flow methods represent detections as graph nodes and transitions as edges, then solve a minimum-cost flow problem with constraints preventing multiple assignments and track splitting.The baselines include successive shortest paths with non-maxima suppression and a linear-programming solver.
- CEM: CEM models tracking as a continuous energy combining data fidelity, trajectory dynamics, exclusion, and persistence terms.The baseline omits explicit occlusion reasoning and an appearance model.
- SMOT: SMOT emphasizes motion over appearance by linking similarly moving tracklets into longer trajectories with generalized linear assignment.Tracklet motion and dynamics are modeled using linear regressors.
- Tracking-by-detection: The tracking-by-detection baseline links detections into tracklets, bridges occlusions up to 20 frames, and uses the Hungarian algorithm in both stages.
- JPDA: JPDA forms joint assignment hypotheses and estimates marginal target-to-measurement probabilities, but its full formulation is impractical because hypotheses grow exponentially.An efficient approximation retains the m strongest hypotheses.
6 CONCLUSION AND FUTURE WORK
MOT16 expands MOTChallenge with more targets, stricter annotations, and additional object classes. The authors position it as a fairer test of generalization to unconstrained and unseen tracking environments.
- MOT16 contains three times more targets than the initial MOT15 release.
- All sequences received more accurate annotations under a strict protocol, with extra classes including vehicles, sitting people, reflections, and distractors.
- The release is intended to provide fairer comparisons of state-of-the-art trackers and encourage methods that work across unconstrained environments and unseen data.Future plans include continued workshops and targeted sub-benchmarks such as sports analysis and biomedical cell tracking.