PaperlayerView exact revision on arXiv

Source-linked AI summary

MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking

Laura Leal-Taixé, Anton Milan, Ian Reid, Stefan Roth, Konrad Schindler

arXiv:1504.01942v1cs.CV

TL;DR

Multi-target tracking lacks standardized benchmarks, and inconsistent data, training, and evaluation practices make comparisons difficult. MOTChallenge responds with diverse datasets, centralized evaluation, and submission infrastructure; one new-sequence baseline reached 12.7% MOTA, while the 3D category was restricted because moving-camera errors were too large for tracking.

  • Problem

    Multi-target tracking lacks large-scale standardized benchmarks, while inconsistent data, training procedures, metrics, and evaluation scripts hinder fair quantitative comparison.

  • Method

    MOTChallenge provides diverse existing and new sequences, centralized evaluation, and infrastructure for submitting tracking methods, data, annotations, and evaluation methodologies.

  • Results

    12.7% MOTA was achieved by a baseline on newly collected sequences, demonstrating their difficulty.

  • Takeaways & Limitations

    The benchmark provides a platform for fairer comparison and aims to guide development of more generic trackers for unconstrained environments and unseen data.

  • Takeaways & Limitations

    The 3D category is limited to a few static-camera sequences because moving-camera calibration and annotation errors are too large for tracking purposes.

Abstract

from arXiv · show

In the recent past, the computer vision community has developed centralized benchmarks for the performance evaluation of a variety of tasks, including generic object and pedestrian detection, 3D reconstruction, optical flow, single-object short-term tracking, and stereo estimation. Despite potential pitfalls of such benchmarks, they have proved to be extremely helpful to advance the state of the art in the respective area. Interestingly, there has been rather limited work on the standardization of quantitative benchmarks for multiple target tracking. One of the few exceptions is the well-known PETS dataset, targeted primarily at surveillance applications. Despite being widely used, it is often applied inconsistently, for example involving using different subsets of the available data, different ways of training the models, or differing evaluation scripts. This paper describes our work toward a novel multiple object tracking benchmark aimed to address such issues. We discuss the challenges of creating such a framework, collecting existing and new data, gathering state-of-the-art methods to be tested on the datasets, and finally creating a unified evaluation system. With MOTChallenge we aim to pave the way toward a unified evaluation framework for a more meaningful quantification of multi-target tracking.

1 INTRODUCTION

MOTChallenge addresses the lack of standardized, large-scale multi-target tracking benchmarks by combining diverse data with centralized evaluation and submission infrastructure. It targets inconsistent datasets, training procedures, metrics, and evaluation scripts that make published results difficult to compare fairly.

  • Motivation: Multi-target tracking lacks large-scale benchmarks, while ambiguous ground truth, variable metrics, and undefined train/test data hinder fair comparison.Occlusions, reflections, cropped targets, and similar-looking objects make ideal annotations difficult to define; IoU thresholds and implementation details also vary.
  • Benchmark design: MOTChallenge combines a public dataset, centralized evaluation method, and infrastructure for crowdsourced data, annotations, and evaluation methodologies.The first release contains 22 sequences split evenly between training and testing, totaling 11,286 frames or 996 seconds of video.
  • Paper goals: The paper analyzes benchmark-creation challenges, existing datasets and evaluation methods, and the strengths and weaknesses of state-of-the-art tracking methods.The benchmark uses standardized ground truth and evaluation methodology so that comparisons focus on the tracking method.
  • Related benchmarks: Existing PETS results are often incomparable because studies use different subsets, detections, training procedures, and evaluation scripts.Although PETS sometimes enabled fair workshop comparisons, its approximately annual submission schedule limited broader use.
  • Benchmark design: The benchmark increases difficulty through sequences with varied viewpoints, lighting conditions, and crowd densities, emphasizing performance across diverse data rather than individual sequences.It includes sequences from PETS and KITTI alongside existing and newly collected data.
  • Evaluation: The evaluation framework relies on matching annotations and results using IoU-based measures, but differing thresholds from 25% to 50% can significantly affect results.CLEAR metrics measure localization precision and tracking accuracy through distinct errors such as misses, ghost trajectories, and identity switches.

2 BENCHMARK SUBMISSION

The benchmark submission system combines registration, standardized metadata, automated evaluation, and public-result controls with policies intended to limit test-set overfitting. It also supports recurring challenges with fixed deadlines and workshop-based result disclosure.

  • Submission policy: Participants register trackers with a method description, operating mode, code availability, runtime, hardware, and training-data details.The system records whether methods operate online or in batches and whether they use external training or detection data.
  • Submission policy: Submitted tracking results are automatically evaluated and can remain private, public, or anonymous for blind-review purposes.Public submissions, annotations, detections, and visual results are available through dedicated profile and visualization pages.
  • Training: The benchmark requires training only with training-sequence ground truth and forbids use of test labels, with misuse leading to account and result deletion.Additional training data must be disclosed and shown in the public ranking table.
  • Detections: Tracking-by-detection methods are expected to use the provided detections, while alternative detections or detection-free methods must be clearly identified.This disclosure keeps differing detection inputs visible in the results table.
  • Submission policy: The submission policy generally permits one submission per approach and imposes a 72-hour waiting period before recomputation and resubmission.The policy is intended to discourage training and parameter tuning on test data.
  • Challenges: Recurring challenges use smaller, potentially harder datasets, fixed deadlines, and workshop announcements of results and winners.The first WACV 2015 challenge used six new outdoor sequences captured with moving and static cameras.

3 DATASETS

MOTChallenge combines diverse existing and new sequences into a common benchmark, balancing training and testing data across camera, viewpoint, and weather conditions. Its six new sequences include especially difficult moving-camera, low-light, and dense-pedestrian settings.

  • 3 DATASETS: MOTChallenge compiles 22 sequences, split evenly between training and testing, to provide a common framework for evaluating tracking methods.The test set contains over 10 minutes of footage and 61,440 annotated bounding boxes, while test annotations remain unreleased to reduce overfitting.
  • 3 DATASETS: Sequences vary by camera motion, viewpoint, weather, frame rate, pedestrian density, and illumination.Weather categories distinguish sunny scenes with shadows and saturation, cloudy scenes with fewer such artifacts, and night scenes with substantial motion blur.
  • 3 DATASETS: Training and testing sequences are divided to maintain a balanced distribution across the reported sequence characteristics.Figure 1 compares the splits by camera type, viewpoint, and weather.
  • 3.1.1 New sequences: Six new challenging sequences comprise four static-camera sequences and two moving-camera sequences filmed at pedestrian height.Three are particularly difficult: one moving-camera night sequence and two outdoor sequences with high pedestrian density.
  • 3.1.1 New sequences: 12.7% MOTA was achieved by the best-performing algorithm on the six new sequences in the first workshop challenge.The result indicates the difficulty of these newly introduced sequences.

3.2 3D MOT 2015 sequences

The 3D benchmark derives pedestrian world positions from image annotations and camera calibration, but calibration and bounding-box limitations make moving-camera 3D tracking unreliable. Static-camera calibration is considered accurate enough for tracking, whereas problematic sequences are excluded or deferred for future annotation and estimation improvements.

  • 3.2 3D MOT 2015 sequences: 3D pedestrian positions are obtained by projecting bounding-box foot positions into the world using camera calibration and ground-plane geometry.Static-camera sequences use image-to-ground-plane homographies, while moving-camera sequences backproject feet and intersect rays with an estimated ground plane.
  • 3.2 3D MOT 2015 sequences: Bounding-box-based 3D localization is highly sensitive to small 2D placement errors, especially for distant or cropped pedestrians.Projective geometry can amplify slight image-space shifts into large 3D position errors.
  • 3.2 3D MOT 2015 sequences: The 3D category is limited to a few static-camera sequences because moving-camera calibration and annotation errors are too large for reliable tracking.Future work includes using additional views and directly annotating pedestrian feet to strengthen 3D estimation.
  • 3.2.1 Static camera sequences: Most static-camera pedestrians have mean speeds between 1 and 1.5 m/s, supporting the conclusion that static-camera calibration is accurate enough for tracking.Some apparent high speeds arise from bounding-box variation, cropping, or difficult annotations rather than actual running.
  • 3.2.2 Moving camera sequences: 200–700 m/s velocities in moving-camera sequences reveal clear problems in 3D position estimation, particularly for distant pedestrians visible briefly.Automatic ground-plane angle estimation adds further inaccuracy, making the 3D information unreliable for these sequences.

3.3 Detections

The benchmark supplies a standardized pedestrian-detector output while allowing trackers to use alternative detections or features if they disclose those choices.

  • 3.3 Detections: The default detections use Dollár et al.’s aggregated channel features detector with its default parameters and an INRIA-trained pedestrian model rescaled by 0.6.The rescaling enables detection of smaller pedestrians, with a minimum benchmark bounding-box height of 59 pixels.
  • 3.3 Detections: Trackers may use different detections or features, but their choices must be documented and displayed in the ratings table.This preserves transparency when comparing methods using non-default inputs.

3.4 Data format

The benchmark uses sequential JPEG images and CSV files to represent detections, annotations, and tracking results. Files encode frame and identity information, bounding boxes, confidence or activity flags, and must be submitted per sequence in a compressed archive.

  • File organization: Images are converted to JPEG and named sequentially with six-digit filenames, while detection and annotation files use comma-separated values.Each CSV line represents one object instance.
  • Record fields: Each object record begins with its frame number and identity, followed by a 2D bounding box and detection confidence information.The bounding box uses the top-left corner, width, and height; detection identities are initially set to −1.
  • Evaluation flags: A confidence-field flag determines whether ground-truth or result entries participate in evaluation: 0 means ignored and 1 means active.This flag is distinct from a detector confidence score in detection files.
  • Annotations: Annotation examples show identity continuity across frames, exclusion of pedestrians that are too small, and 2D files setting 3D positions to −1.All values, including bounding-box coordinates, are 1-based.
  • Submission format: A valid benchmark submission requires one correctly formatted CSV file per sequence, with all files compressed into a single zip archive for upload.The required filename is ‘‘Sequence-Name.txt’’.

3.5 Expansion through crowdsourcing

MOTChallenge is designed to expand through community contributions of challenging data, detections, annotations, and evaluation components. Future categories include varied tracking scenarios beyond pedestrian sequences, each handled through separate submissions.

  • Dataset evolution: The benchmark plans yearly dataset updates that add more challenging sequences and remove outdated or repetitive ones.The stated goal is to increase data difficulty as methods improve.
  • Community contributions: Community members are invited to contribute sequences, detections, and annotations to broaden the variety of benchmark data.The benchmark commits to extending its coverage with relevant categories.
  • Target categories: Planned expansion covers outdoor cars and bicycles, biological tracking, sports tracking, and large-scale multi-view sequences.Examples include cell, bird, fish, basketball, hockey, and soccer data.
  • Candidate data: Existing literature already contains candidate data sources such as thermal-camera bat footage, cell tracking, sports games, indoor multi-view sequences, and large-scale pedestrian trajectories.The authors encourage researchers to submit interesting data for inclusion.
  • Submission structure: Each expanded category will have its own separate submissions within the benchmark structure.This preserves category-specific evaluation organization as coverage grows.

4 BASELINE METHODS

The benchmark includes diverse baseline trackers spanning graph optimization, continuous-energy modeling, motion-based association, occlusion bridging, social-force modeling, and online appearance learning. Baselines are standardized through shared code and training procedures, with parameter search used when methods lack dataset-specific learning.

  • Baseline protocol: The baseline collection uses publicly available code and identical training procedures, although reported values may not equal each method’s best possible performance.The authors explicitly qualify the baseline results as benchmark configurations.
  • Training and testing: For methods without parameter-learning algorithms, 100 training-set runs sample parameters around author-provided defaults to find a strong benchmark setting.The default parameter set is Θ := {θ1, . . . , θP}.
  • Training and testing: The selected parameter set is the one achieving the highest MOTA across the 100 runs, after which it is run once on the test set.TBD is excepted because it does not disclose obvious free parameters.
  • Network flow: Network-flow baselines model detections as graph nodes and transitions as edges, then obtain trajectories by solving a minimum-cost flow problem.The benchmark uses DP NMS plus LP2D and LP3D solvers.
  • Continuous-energy tracking: CEM represents target trajectories with continuous image coordinates and combines data, dynamics, exclusion, and persistence terms in a continuous energy.The listed basic approach omits explicit occlusion reasoning and an appearance model.
  • Motion-based association: SMOT emphasizes motion over appearance by linking similarly moving tracklets into longer trajectories through generalized linear assignment.Tracklet dynamics and motion similarity are modeled with linear regressors.
  • Occlusion and online tracking: TBD bridges occlusions of up to 20 frames, while TC ODAL combines confidence-based tracklet association with global matching and online discriminative appearance learning.Both use Hungarian matching; TC ODAL is the only online baseline.

5 EVALUATION

Evaluating multi-object tracking remains difficult because prior comparisons are affected by ambiguous ground truth, imprecise metric definitions, and implementation variations. The paper therefore presents a complete evaluation procedure to address these sources of inconsistency.

  • Evaluation challenges: Comparing a new multi-object tracking method with prior work is not straightforward because ground truth can be ambiguous and evaluation metrics can vary in definition or implementation.The authors identify these as diverse, persistent evaluation problems.
  • Evaluation procedure: The section introduces the paper’s entire evaluation procedure in response to these benchmark-comparison difficulties.The supplied passage frames the procedure as the next step after identifying the problems.

5.1 Annotations

MOTChallenge combines annotated videos from public and newly collected sources, while acknowledging substantial variation in annotation policies. The benchmark provides training annotations, withholds test annotations to reduce overfitting, and plans standardized annotations for a later release.

  • Human supervision makes reliable video ground truth costly, contributing to the scarcity of datasets with publicly available annotations.
  • The benchmark uses publicly available ground truth for most sequences and annotates six new sequences with VATIC.
  • Training-set ground truth is released, while test annotations are withheld to reduce overfitting on unseen data.
  • Variation in the annotations: Public annotations vary in bounding-box extent, centering, scale with distance, and treatment of occlusions across sequences.
  • Variation in the annotations: Annotation-policy variation can change the performance of state-of-the-art methods, motivating reannotation with common strict rules.

5.2 Evaluation metrics

The benchmark evaluates multi-target trackers through standardized matching and complementary metrics, while addressing robustness, localization, and runtime. Its results show that localization varies little across methods, speed and accuracy trade off, and MOTA alone is insufficient as a complete measure.

  • Evaluation metrics: The benchmark combines CLEAR metrics with track-quality measures because no single evaluation metric fully captures the application-dependent errors of multi-target tracking.
  • Matching: Tracker outputs are classified as true positives or false positives using a distance threshold, while missed targets are false negatives.
  • Matching: Matching enforces one-to-one correspondence between annotated and hypothesized objects and uses temporally consistent Hungarian-algorithm assignments.
  • Distance measures: 2D matching uses bounding-box intersection over union with a 50% threshold, whereas 3D matching uses Euclidean distance with a 1-meter pedestrian threshold.
  • MOTA: MOTA combines false negatives, false positives, and identity switches, but can be negative and is debatable as a standalone performance measure.
  • Robustness: The benchmark reports MOTA standard deviation across sequences to indicate tracker robustness under diverse and unseen conditions.
  • MOTP: MOTP ranged from 69.6% to 71.6% across methods and was primarily determined by detections and annotations rather than tracker output.
  • Runtime: The fastest approach, DP NMS, performed worse on average, while LP2D provided a balance between speed and performance.

6 CONCLUSION AND FUTURE WORK

The paper presents MOTChallenge as a centralized, open platform combining public and new challenging videos for multi-target tracking evaluation. Future work targets standardized annotations, recurring challenges, and additional benchmarks for other disciplines.

  • MOTChallenge provides a centralized platform with existing public videos and new challenging sequences, open to new submissions.
  • The authors aim for fairer comparisons and research toward generic methods that perform well in unconstrained environments and on unseen data.
  • Future work includes standardizing annotations, continuing workshops and challenges, and introducing additional sub-benchmarks for other disciplines.
Loading 1504.01942v1…