PaperlayerView exact revision on arXiv

Source-linked AI summary

MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking

Patrick Dendorfer, Aljoša Ošep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, Laura Leal-Taixé

arXiv:2010.07548v2cs.CV

TL;DR

Single-camera multi-target tracking lacked fair, standardized evaluation because ground truth, metrics, and datasets were difficult to define consistently. The paper presents MOTChallenge as a benchmark framework with expanded annotated releases, public detections, and centralized evaluation, then analyzes published trackers. It reports broad tracker analyses and identifies end-to-end learning as a promising direction while acknowledging benchmark and tracking limitations.

  • Problem

    Single-camera multi-target tracking lacked large-scale, fair comparisons because ground truth, evaluation metrics, and predefined training and test data were difficult to standardize.

  • Method

    MOTChallenge standardizes tracking evaluation through shared datasets, annotations, centralized scoring, public detections, and analyses of published trackers across MOT15, MOT16, and MOT17.

  • Results

    The benchmark collected over 1,000 public method tests and supported analysis of tracker trends, weaknesses, and research directions across its releases.

  • Takeaways & Limitations

    End-to-end learning that combines optimization with learning is identified as a promising direction for improving multi-target tracking.

  • Takeaways & Limitations

    Evaluation remains sensitive to differing intersection-over-union thresholds, implementation details, and annotation policies; current trackers also fall short across large occlusion gaps.

Abstract

from arXiv · show

Standardized benchmarks have been crucial in pushing the performance of computer vision algorithms, especially since the advent of deep learning. Although leaderboards should not be over-claimed, they often provide the most objective measure of performance and are therefore important guides for research. We present MOTChallenge, a benchmark for single-camera Multiple Object Tracking (MOT) launched in late 2014, to collect existing and new data, and create a framework for the standardized evaluation of multiple object tracking methods. The benchmark is focused on multiple people tracking, since pedestrians are by far the most studied object in the tracking community, with applications ranging from robot navigation to self-driving cars. This paper collects the first three releases of the benchmark: (i) MOT15, along with numerous state-of-the-art results that were submitted in the last years, (ii) MOT16, which contains new challenging videos, and (iii) MOT17, that extends MOT16 sequences with more precise labels and evaluates tracking performance on three different object detectors. The second and third release not only offers a significant increase in the number of labeled boxes but also provide labels for multiple object classes beside pedestrians, as well as the level of visibility for every single object of interest. We finally provide a categorization of state-of-the-art trackers and a broad error analysis. This will help newcomers understand the related work and research trends in the MOT community, and hopefully shed some light on potential future research directions.

1 Introduction

MOTChallenge addresses difficult and inconsistent evaluation of single-camera multi-target tracking by standardizing data, annotations, detections, and scoring. Its releases expand challenge diversity and annotation detail while enabling broad analysis of tracking methods and trends.

  • Motivation: Tracking comparisons are hindered by ambiguous ground truth, conflicting metrics, and the lack of predefined training and test data.Occlusions, cropping, reflections, similar-looking objects, and metric choices make evaluation difficult and fairness hard to ensure.
  • Benchmark design: MOTChallenge combines collected datasets, centralized evaluation, and infrastructure for crowdsourcing data, evaluation methods, and annotations.MOT15 included 11 training and 11 testing sequences with shared detections, training annotations, and a common evaluation method.
  • Impact: Over 1,000 methods were publicly tested and over 1833 users registered, establishing MOTChallenge as a standardized large-scale benchmark for single-camera multiple people tracking.The benchmark included more than 1,000 public tests across its releases.
  • Benchmark releases: MOT16 introduced 14 diverse crowded sequences with researcher-produced, double-checked annotations covering pedestrians, vehicles, sitting people, and occluding objects.The annotations also enabled object-level occlusion and cropping measurements.
  • Benchmark releases: MOT17 improved annotation consistency and evaluated trackers with three public detector outputs whose results were averaged to assess robustness to detection quality.A separate MOT17Det subset evaluated object detectors.
  • Analysis: The paper categorizes and analyzes 73 MOT15, 74 MOT16, and 57 MOT17 trackers to identify trends, weaknesses, and promising research directions.The benchmark goals include fair evaluation and analysis of state-of-the-art tracking methods over time.

2 Related work

Prior MOT evaluation lacked consistent quantitative standardization, with results affected by data selection, training, detections, metrics, and evaluation scripts. Standardized challenges address this by centralizing data, evaluation, and public comparison, while MOTChallenge uses diverse sequences and established tracking metrics.

  • Single-camera MOT had limited standardized quantitative evaluation, with PETS primarily focused on surveillance applications.
  • PETS results were often inconsistent because researchers used different data subsets, training procedures, evaluation scripts, and detection inputs.
  • Standardized challenges upload results to a server, evaluate them centrally, and present comparisons publicly.
  • MOT evaluation commonly uses CLEAR-MOT measures based on bounding-box intersection over union, while IDF1 assesses long-term track quality.
  • Evaluation outcomes can change with the intersection over union threshold and annotation bounding-box size, complicating fair comparisons.

3 History of MOTChallenge

MOTChallenge evolved from MOT15 into progressively more challenging, accurately annotated releases with broader detection and tracking evaluation. Its growing participation and later extensions support analysis of tracking trends and expansion beyond pedestrian tracking.

  • MOT15: MOT15 launched with 11 training and 11 testing sequences, provided detections and evaluation scripts, and quickly attracted substantial leaderboard activity.The testing sequences were withheld publicly, and 692 tracking results were submitted by the cited point in the history.
  • MOT16: MOT16 introduced 14 longer, more crowded sequences and a stricter, more accurate annotation policy.
  • MOT17: MOT17 improved annotation consistency and required trackers to run on three public detection sets, with results averaged across detectors.The protocol was designed to assess tracker robustness to detections of different quality, and MOT17Det separately evaluated object detectors.
  • Beyond pedestrians: MOTChallenge later expanded toward general multi-object tracking through zebrafish tracking and the TAO dataset spanning 833 classes and 17,287 tracks.
  • Weaknesses of MOT15: MOT15 weaknesses included inaccurate collected annotations, especially for moving-camera scenes, and sequences that were too simple for state-of-the-art trackers.

5 MOT16 and MOT17 Releases

MOT16 introduced more diverse, challenging sequences with carefully verified annotations, while MOT17 refined those labels and tested tracker robustness across detector qualities.

  • MOT17: MOT17 reused MOT16 sequences but improved bounding boxes, added missed pedestrians and occluders, and incorporated a second round of sanity checks.These changes followed comments from benchmark users and aimed to increase annotation accuracy.
  • MOT17: MOT17 evaluated trackers with three public detection sets from different detectors and averaged their results to assess robustness to noisy detections.A separate MOT17Det subset was released for evaluating object detectors.
  • Evaluation: The benchmark standardizes tracker comparisons by providing common ground truth, evaluation metrics, scripts, and precomputed detections.This setup isolates tracker performance from other experimental factors, although selecting a single universally appropriate metric remains unresolved.
  • Evaluation: MOTA combines false negatives, false positives, and identity switches into one measure based on frame-to-frame matching.The benchmark reports percentage MOTA, which can be negative when tracker errors exceed the number of ground-truth objects.

Justification.

The evaluation framework combines complementary measures because tracking quality includes localization, event errors, identity preservation, trajectory coverage, and fragmentation.

  • CLEAR-MOT: MOTA summarizes false negatives, false positives, and identity switches, but emphasizes detection performance more than temporal continuity.The paper notes that MOTA has been criticized for not balancing error sources properly, despite its continued widespread use.
  • Localization: MOTP measures average bounding-box overlap for correctly matched hypotheses, making it a localization-precision measure rather than a tracker-quality measure.It ranges from 50% to 100% under the stated overlap formulation.
  • Identity: IDF1 evaluates identity preservation over an entire sequence by matching predicted and ground-truth trajectories using temporal overlap.It balances identification precision and recall through the harmonic mean of correctly identified detections.
  • Trajectory coverage: Trajectory coverage classifies targets as mostly tracked, partially tracked, or mostly lost according to the recovered percentage of each ground-truth trajectory.A target is mostly tracked when at least 80% of its lifespan is successfully covered and lost below 20%.
  • Trajectory continuity: Fragmentation counts interruptions in which a ground-truth trajectory changes from tracked to untracked and later resumes.The benchmark also reports relative fragmentation as FM / Recall.

7 Analysis of State-of-the-Art Trackers

The paper analyzes published MOT trackers across three benchmarks, organizing methods and examining performance patterns, errors, and runtime across varied scenes.

  • Scope: The authors categorize tracking methods and conduct error and runtime analyses to identify approaches performing well across diverse scenes.The stated goal is to help newcomers navigate the literature and suggest future research directions.
  • Scope: The analysis covers 101 published trackers: 73 evaluated on MOT15, 74 on MOT16, and 57 on MOT17.It includes peer-reviewed submissions using the benchmark’s public detections and considers results available before April 17th, 2020.
  • Tracking-by-detection: Tracking-by-detection separates MOT into object detection and temporal data association, which can be represented as disjoint paths through a detection graph.Graph nodes represent detections, while links encode feasible associations.
  • Global optimization: Earlier methods emphasized globally optimized association formulations, including k-shortest paths, linear programming, and conditional random fields.These approaches focused primarily on linking detections into consistent trajectories.
  • Global optimization: The top-15 overview orders entries from easier sequences and stronger methods to harder sequences and poorer performance, with a thick black line showing mean performance.The analysis spans all three benchmark releases.

Motion models.

MOT tracking research moved from hand-crafted motion and appearance cues toward learned association models, with sparse trajectories and learned representations associated with stronger reported performance.

  • Motion models: Motion-model methods using simple distances or appearance cues achieved around 38% MOTA on MOT16 and 25% on MOT15.These approaches used motion models as additional association affinity cues.
  • Hand-crafted affinity measures: The shift toward robust geometric and appearance similarities improved tracker performance and handling of complex scenarios.Examples include sparse appearance models, integral channel features, long-term interest-point trajectories, and learned sparse feature matching.
  • Hand-crafted affinity measures: NOMT achieved 33.67% MOTA on MOT15 and 46.42% on MOT16, with sparse flow or trajectories yielding a reported performance boost.The first methods surpassing NOMT on MOT16 were published in 2017.
  • Towards learning: Learning-based methods improved association by learning cost parameters, track-management policies, or appearance representations instead of relying only on hand-crafted cues.The paper describes a progression from structured learning and transfer learning toward end-to-end appearance models and online adaptation.
  • Towards learning: SiameseCNN learned spatio-temporal embeddings from appearance and optical flow, reaching 29% MOTA on MOT15.Its embeddings were combined with contextual cues for data association.
  • Towards learning: Later methods adapted target appearance models online, extending beyond general embedding vectors for detections.The paper identifies online adaptation as a source of further performance gains.

Online appearance model adaptation.

The analysis traces MOT progress from association and appearance modeling toward target regression and end-to-end learning, while identifying persistent challenges with occlusion gaps and runtime.

  • Graph-based learning: MPNTrack combines Tracktor-style regression with learned graph-based data association, reaching 58.56% MOTA on MOT16 and 58.85% on MOT17.It still requires projection from graph-network output to feasible network-flow solutions, so it is not fully end-to-end.
  • Performance trends: Deep-learning trackers increasingly learn appearance models, association affinities, online adaptation, and target regression.Figure 6 visualizes submitted-model performance over time and by model type.
  • Top-performing components: Top-performing methods commonly combine learned single-target regressors with re-identification modules.These components improve tracking but still fall short on large occlusion gaps.
  • Runtime analysis: Most submitted methods remain below the 25 Hz real-time threshold, with an average processing rate of approximately 5 Hz.Reported runtimes cannot be independently verified, so the efficiency comparison requires caution.
  • Error analysis: Since 2018, tracking-by-regression methods have sharply reduced false negatives by regressing targets into visible regions missed by detectors.The resulting reduction in false negatives coincides with a major leap in MOTA performance.

8 Conclusion and Future Work

MOTChallenge provides a standardized benchmark with carefully annotated data and a systematic analysis of tracker performance, revealing trends and future directions for multi-target tracking.

  • Benchmark contribution: MOTChallenge standardizes fair evaluation of single-camera multi-person tracking through its benchmark and data releases.The first two releases contain about 35,000 frames and almost 700,000 annotated pedestrians.
  • Dataset construction: The second release adds strictly validated annotations and labels for vehicles, sitting people, reflections, and distractors.These annotations provide additional information beyond pedestrian identities.
  • Systematic analysis: The paper analyzes 101 trackers across MOT15, MOT16, and MOT17, covering 73, 74, and 57 trackers respectively.The analysis follows changes from global association optimization to learned affinities, target regression, and online appearance adaptation.
  • Future work: End-to-end learning that combines optimization with learning is identified as a promising direction for future research.The authors present this trend as having substantial potential for further progress.

A Benchmark Submission

The benchmark submission system records method details, standardizes result uploads and evaluation, and enforces policies intended to support objective comparisons without test-set tuning.

  • Tracker registration: Users create trackers with a unique name and provide method descriptions, publication references, operating mode, code availability, and training-data information.These metadata requirements distinguish online from batch methods and provided from external data.
  • Challenge participation: Participants submit separate result files for selected challenges, identifying the challenge and publication reference.The system supports submissions to different benchmarks after tracker creation.
  • Evaluation and publication: Submitted results are automatically evaluated and shown on the user profile, while public ranking and anonymous publication remain user-controlled.Public submissions also expose visual results, annotations, and detections on tracker pages.
  • Submission policy: The benchmark requires additional training data and nonstandard detections to be disclosed, while forbidding test-set ground-truth use.Misuse can lead to account and result deletion.
  • Submission frequency: A 72-hour waiting period applies before resubmitting to the same challenge, discouraging server-based test-data training and parameter tuning.The policy allows recomputation after implementation bugs while limiting repeated submissions.

A.2 Challenges and Workshops

MOTChallenge organizes open and workshop-linked challenges around balanced training and testing sequences, while documenting dataset conditions and standardized result-file formats.

  • Challenges and workshops: The main challenges include 2D MOT 2015, 3D MOT 2015, MOT16, and MOT17, which remain open for submissions.Special challenges are linked to workshops and evaluate methods on selected sequence collections.
  • Dataset split: The benchmark uses 22 sequences split evenly between training and testing, withholding test annotations to reduce overfitting.The test data contains over 10 minutes of footage and 61,440 annotated bounding boxes.
  • Sequence characterization: Sequences are categorized by camera motion, viewpoint, and weather or illumination conditions, including static or moving cameras and low, medium, or high viewpoints.The conditions include normal scenes, shadows, night, and indoor settings.
  • Sequence characterization: Training and testing sequences are divided to maintain a balanced distribution across the characterized conditions.Figure 9 presents the comparison histogram for these categories.
  • Submission format: Each sequence requires a separately named result file containing standardized object-level records, compressed into one ZIP upload for evaluation.The file format records frame and identity information, bounding boxes, confidence or activity flags, and optional 3D positions.

C.1 Annotation rules

MOT16/MOT17 annotations follow a consistent protocol covering target classes, bounding-box extent, and track start and end conditions.

  • C.1 Annotation rules: The protocol defines annotation consistency across the MOT16 and MOT17 datasets through explicit rules applied throughout the sequences.Every moving person or vehicle is annotated with a bounding box as accurately as possible.
  • C.1 Annotation rules: Annotations cover upright pedestrians, ambiguous people-like objects, vehicles, and occluders under separate class categories.Vehicles and occluders support training and occlusion computation rather than evaluation.
  • C.1 Annotation rules: Bounding boxes tightly contain all pixels belonging to each distinguishable object and estimate full extent through partial occlusion.The box follows the object’s extent as accurately as possible, including portions inferred behind occlusions.
  • C.1 Annotation rules: People on vehicles are annotated separately only when clearly visible, excluding children inside strollers or people inside cars.Motorcyclists and bikers are annotated separately from their vehicles.
  • C.1 Annotation rules: Tracks begin when approximately 10% of a person is visible and end when its location can no longer be pinpointed accurately.Coordinates may extend beyond the image, and reappearing people receive new IDs.

C.1.4 Minimal size

The benchmark annotates distinguishable objects regardless of image size, computes occlusion from annotations, and records standardized image and file-format information.

  • C.1.4 Minimal size: All distinguishable objects are annotated independently of image size, although evaluation considers pedestrians only above a minimum pixel height.The annotation policy is broader than the evaluation-size threshold.
  • C.1.4 Minimal size: Occlusion levels are computed automatically from overlapping bounding boxes using their vertical positions relative to the camera.Targets remain fully annotated through occlusions when extent and location can be determined accurately.
  • C.1.4 Minimal size: A detector-based sanity check adds high-confidence human or distractor detections that were missed from the annotation list.The check runs on all videos after sequence annotation.
  • C.1.4 Minimal size: Annotation and detection files use comma-separated values, with each row representing one object instance and containing nine values.Images are converted to JPEG and named with sequential six-digit filenames.
  • C.1.4 Minimal size: Each record stores frame and trajectory identifiers, bounding-box coordinates, an evaluation flag, object type, and visibility ratio.The bounding box is represented by its top-left position, width, and height.

D Implementation details of the evaluation

Evaluation matches tracker outputs to annotated targets, computes tracking errors across concatenated test sequences, and excludes target-like special classes from scoring.

  • D Implementation details of the evaluation: Tracker outputs are classified as true positives or false positives using a distance threshold, while unmatched targets become false negatives.The evaluation seeks to minimize false positives and false negatives and also reports their ratio.
  • D Implementation details of the evaluation: At most one hypothesis may recover each target, and each hypothesis may correspond to only one target.This correspondence constraint is applied before computing evaluation counts.
  • D Implementation details of the evaluation: The Hungarian algorithm finds optimal frame-level matching, while target re-identification after leaving the field of view is not explicitly handled.A reappearing target is treated as unseen and assigned a new ID.
  • D Implementation details of the evaluation: Results are computed after concatenating all test sequences rather than averaging per-sequence figures.This is intended to account for large differences in the number of targets across sequences.
  • D Implementation details of the evaluation: Matching uses bounding-box intersection over union with a threshold of 0.5, or 50%.The criterion is also called the Jaccard index.
  • D.3 Target-like annotations: Outputs overlapping distractors, static people, reflections, or people on vehicles by more than 50% are excluded before final pedestrian-only evaluation.The procedure first performs Hungarian matching at each frame, then applies the exclusion rule.
Loading 2010.07548v2…