PaperlayerView exact revision on arXiv

Source-linked AI summary

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixe, Bastian Leibe

arXiv:2009.07736v2cs.CV

TL;DR

Evaluating multi-object tracking is difficult because accurate detection, localization, and association must be assessed together. The paper introduces HOTA, analyzes its design and alternatives, and finds that human visual assessments align better with HOTA than with MOTA or IDF1.

  • Problem

    MOT evaluation is difficult because tracking requires accurate detection, localization, and association over time.

  • Method

    The paper proposes HOTA as a unified MOT metric, analyzes its design and existing metrics, and integrates scores across localization thresholds.

  • Results

    HOTA aligns better with human visual assessment than MOTA or IDF1, with human agreement of 61.6% versus 38.4% against MOTA and 72.0% versus 28.0% against IDF1.

  • Takeaways & Limitations

    HOTA supports clearer analysis of tracking performance and better reflects human judgments of tracking accuracy than established metrics.

  • Takeaways & Limitations

    The default HOTA weighting is recommended unless a different weighting is justified for a particular desired outcome.

Abstract

from arXiv · show

Multi-Object Tracking (MOT) has been notoriously difficult to evaluate. Previous metrics overemphasize the importance of either detection or association. To address this, we present a novel MOT evaluation metric, HOTA (Higher Order Tracking Accuracy), which explicitly balances the effect of performing accurate detection, association and localization into a single unified metric for comparing trackers. HOTA decomposes into a family of sub-metrics which are able to evaluate each of five basic error types separately, which enables clear analysis of tracking performance. We evaluate the effectiveness of HOTA on the MOTChallenge benchmark, and show that it is able to capture important aspects of MOT performance not previously taken into account by established metrics. Furthermore, we show HOTA scores better align with human visual evaluation of tracking performance.

1 Introduction

MOT evaluation is difficult because tracking requires accurate detection, localisation, and association over time. HOTA addresses this by balancing these aspects, including localisation, while supporting unified ranking and detailed error analysis.

  • MOT requires accurate detection, localisation, and association over time, making its evaluation difficult.
  • HOTA evaluates detection, association, and localisation together, unlike MOTA and IDF1, which overemphasize detection and association respectively.
  • HOTA serves as a unified metric for ranking trackers and decomposes into sub-metrics for separately analysing tracking errors.
  • DetA measures aligning detections, AssA averages alignment between matched trajectories, and HOTA combines them through a geometric mean across localisation thresholds.
  • The paper evaluates HOTA theoretically, on MOTChallenge, and through a user study comparing metric agreement with human tracking judgments.

2 Related Work

Earlier MOT evaluation metrics often focused on individual error types or lacked desirable evaluation properties. HOTA combines error-type analysis into a unified metric and is assessed against established benchmark practices and human judgments.

  • MOT benchmarks commonly ranked trackers with MOTA, while IDF1 was adopted in some multi-camera and single-camera evaluation settings.
  • IDF1 became attractive because it focuses more on association accuracy than detection accuracy.
  • Confidence-ranked tracking results motivated extensions such as PR-MOTA and AMOTA, while HOTA also supports confidence-ranked evaluation.
  • Other proposed metrics were not widely adopted by MOT benchmarks, limiting their role in standard tracker evaluation.
  • Secondary trajectory-based metrics often target single error types and can be simple or easy to game when used for comprehensive ranking.
  • A prior framework identified five MOT error types and two desired properties, while HOTA combines corresponding components into one differentiable metric.
  • Earlier human-evaluation work found MOTA most representative among the metrics it tested, despite noting MOTA’s limitations.

3 Preliminaries

MOT outputs encode detection, localisation, and identity association, and evaluation compares these outputs with ground-truth tracks. The preliminaries define similarity, matching, metric terminology, and relevant error categories.

  • MOT outputs specify which objects appear, where they are located, and which detections belong to the same object across frames.
  • Benchmark metrics compare tracker predictions with ground-truth tracking results, so metric design can influence which aspects of MOT research are prioritised.
  • Predictions contain detections with persistent trajectory identities, while Track-mAP additionally assigns confidence scores to predicted trajectories.
  • MOT can track multiple classes, with previous metrics typically evaluated per class and averaged, while HOTA also provides a multi-class extension.
  • The paper focuses on detection, association, and localisation errors because they are common and broadly applicable across tracking scenarios.
  • Tracking detections may use boxes, masks, points, or pose skeletons, provided the representation has an appropriate similarity measure.
  • Similarity scores range from 0 for no overlap to 1 for perfect alignment, and bijective matching penalises unmatched predictions and ground-truth detections.
  • In this paper, an evaluation metric is a ranking score and need not satisfy every condition of a strict mathematical metric.

4 Overview of Previous Metrics

MOTA, IDF1, and Track-mAP use different matching levels and optimisation objectives to score tracking. Their designs variously emphasise detection, identity association, trajectory similarity, or confidence-ranked precision and recall.

  • MOTA: MOTA matches predicted and ground-truth detections one-to-one within each frame, classifying matched detections as true positives and unmatched detections as errors.
  • MOTA: MOTA measures association through identity switches and does not include localisation error in its main score.
  • MOTA: MOTA matching first maximises true positives and then maximises the mean similarity across true positives.
  • IDF1: IDF1 matches ground-truth and predicted trajectories one-to-one, then counts overlapping matches as identity true positives and remaining detections as identity errors.
  • IDF1: IDF1 selects trajectory matches to minimise identity false positives and identity false negatives, without minimising localisation accuracy.
  • Track-mAP: Track-mAP matches trajectories using trajectory similarity and a threshold, selecting eligible predictions according to confidence.
  • Track-mAP: Track-mAP orders predicted trajectories by confidence and integrates an interpolated precision-recall curve over recall values.
  • Track-mAP: For multiple classes, Track-mAP is usually calculated separately per class and averaged across classes.

5 The HOTA Evaluation Metric

HOTA is a unified MOT metric that balances detection, association, and localisation while decomposing tracking performance into interpretable components. Its formulation matches detections, measures trajectory alignment, and averages performance across localisation thresholds.

  • HOTA evaluates detection, association, and localisation within one metric for MOT performance.It addresses shortcomings of metrics that emphasize detection or association alone.
  • Matching Predictions and Ground-Truth: At detection level, HOTA uses bijective matching to identify true positives, false positives, and false negatives.The actual matching is selected to maximise the final HOTA score.
  • Measuring Association: For each true positive, TPAs share its prID and gtID, FPAs share its prID, and FNAs share its gtID.These association concepts quantify alignment between the matched ground-truth and predicted trajectories.
  • Scoring Function: HOTA uses a double Jaccard formulation in which detection performance is weighted by an association score for each true positive.The association score is itself a Jaccard measure over TPAs, FPAs, and FNAs.
  • Integrating over Localisation Thresholds: HOTA averages scores computed at localisation thresholds from 0.05 to 0.95 in 0.05 intervals.Matching is performed separately at each threshold so the final score incorporates localisation accuracy.
  • HOTA in One Sentence: HOTA measures trajectory alignment over matching detections while penalising detections that do not match.

6 Decomposing HOTA into Different Error Types.

HOTA decomposes tracking performance into detection, association, and localisation components that distinguish the basic error types made by trackers. This decomposition supports both metric-based ranking and analysis or tuning for application-specific requirements.

  • HOTA supports a single ranking score while decomposing performance into sub-metrics for analysing tracker errors.This combines comparison between methods with diagnosis of how trackers can be improved.
  • Taxonomy of Error Types: The taxonomy separates detection, association, and localisation errors, with detection and association each divided into recall and precision.
  • Taxonomy of Error Types: The five basic error types are false negatives, false positives, fragmentations, mergers, and deviations.These correspond respectively to detection recall, detection precision, association recall, association precision, and localisation.
  • Taxonomy of Error Types: HOTA is designed to be error-type differentiable and monotonic because it decomposes into sub-metrics for each basic error type.
  • Metrics for Different Tracking Scenarios with Different Requirements: The decomposed sub-metrics allow algorithms or hyper-parameters to be selected according to application-specific tracking requirements.For example, some applications prioritise identity-preserving trajectories, while driving assistance prioritises detecting pedestrians and avoiding extra detections.
  • Measuring Localisation: HOTA averages HOTAα across localisation thresholds so the final score accounts for actual spatial alignment.Localisation accuracy can also be measured separately as LocA.
  • Separating Detection and Association: DetA and AssA are combined as the geometric mean of detection and association scores, balancing both components in the final HOTA score.Both components use Jaccard-index structures, with each detection contributing equally.
  • Separating Recall and Precision: Association recall decreases when a tracker splits one object across multiple predicted tracks, whereas association precision decreases when one track spans multiple objects.

7 Analysing the Design Space of HOTA

HOTA’s design choices target higher-order association and matching while balancing detection and association through Jaccard-based components and a geometric mean. These choices preserve monotonicity, symmetry, metric properties, and interpretable error separation.

  • Higher-Order vs First-Order Association: HOTA measures association per detection over time, unlike MOTA’s one-time-step ID-switch association measure.This higher-order association enables HOTA to measure long-term association.
  • Higher-Order vs First-Order Matching: HOTA performs higher-order trajectory matching through detection-level matches, allowing each trajectory to match all possible trajectories rather than one bijective partner.IDF1 uses unique bijective trajectory matching, whereas HOTA measures matches at detection level.
  • Jaccard Formulation: The Jaccard formulation is the only compared formulation that is monotone, symmetric, and satisfies the triangle inequality.F1 can score a less-correct track higher, while MODA is non-symmetric when predictions and ground truth are swapped.
  • Detection Errors in Association: Including false negatives and false positives in association keeps detection and association balanced, preventing added correct detections from decreasing the overall score.Excluding these errors can make the score non-monotonic because AssA may decrease faster than DetA increases.
  • Geometric Mean: The geometric mean balances DetA and AssA, approaches zero when either component approaches zero, and prevents double-counting shared detection errors.In single-object tracking, the association and detection scores coincide, so the square root yields a Jaccard score.

Final Tracks vs Potential Tracks with Confidence

HOTA is designed for final tracking results, while a confidence-ranked extension evaluates ordered potential tracks. The paper also identifies online evaluation and fragmentation as important scope boundaries, addressed by separate extensions.

  • Final Tracks vs Potential Tracks with Confidence: HOTA evaluates final tracking results rather than confidence-ranked potential tracks, while CR-HOTA provides a confidence-ranked alternative for scenarios such as monocular 3D tracking.CR-HOTA reduces to default HOTA when evaluating a fixed set of detections above a confidence threshold.
  • Drawbacks of HOTA: HOTA may be unsuitable for online tracking because each detection’s association score depends on associations throughout the future video.This future dependence is undesirable when tracker outputs are used for decisions at each time-step.
  • Drawbacks of HOTA: HOTA does not measure fragmentation because it is designed to measure global, long-term association alignment.This can be a drawback when short-range fragmentation matters, although fragmentation-aware HOTA is presented as an extension.

8 HOTA Extensions

HOTA is extended for online, fragmentation-aware, weighted, classification-aware, federated, and confidence-ranked tracking scenarios. These variants adapt the metric’s association, weighting, matching, or evaluation scope to different application requirements.

  • Online HOTA: OHOTA computes association accuracy using only time-steps up to the current time-step, adapting HOTA for online tracking.This avoids relying on future associations when evaluating each current time-step.
  • Fragmentation-Aware HOTA: FA-HOTA jointly measures short-term fragmentation alignment and long-term association by taking their geometric mean for each true-positive detection.FA-HOTA equals HOTA when no fragmentation occurs because fragmentation accuracy is bounded by association accuracy.
  • Importance Weighted HOTA: W-HOTA lets users assign different weights to detection, association, recall, and precision according to application requirements.The default weighting is recommended unless a different weighting serves a specific desired outcome.
  • Classification-Aware HOTA: CA-HOTA weights each true-positive contribution by the prediction’s probability of belonging to the ground-truth class and includes classification in matching.The method also defines a separate classification accuracy, ClaA, and a class-averaged CA2-HOTA for many-class datasets.
  • Federated HOTA: Fed HOTA adapts HOTA to federated evaluation by counting false positives only for classes whose images confirm that unannotated objects are absent.This addresses datasets where not all objects are annotated in every image.
  • Confidence-Ranked HOTA: CR-HOTA orders detections by decreasing confidence and evaluates HOTA across fixed recall values, integrating HOTA over detection recall.The formulation reduces to original HOTA when evaluating detections above a fixed threshold.

9 Analysing Previous Evaluation Metrics.

The analysis identifies distinct weaknesses in MOTA, IDF1, and Track-mAP, and explains how HOTA addresses them through balanced detection-association scoring, global association, localisation integration, and error decomposition.

  • Problems with MOTA and IDF1: HOTA balances detection and association, unlike MOTA’s detection bias and IDF1’s association bias, by matching detections while scoring association globally.HOTA’s detection and association components contribute equally to the final score.
  • Problems with MOTA: 100 times: on average, detection affects MOTA’s final score as much as association on the MOT17 benchmark.For the top ten MOT17 trackers, the detection-to-association effect ratio averaged 98.6.
  • Problems with MOTA: R2=99.4: MODA explains more than 99% of MOTA variation across 175 MOT17 trackers, whereas IDSWs have R2=23.7.This comparison indicates that MOTA is strongly tied to its detection-only counterpart.
  • Problems with MOTA: HOTA evaluates global association, correctly ranking trackers that recover from identity mistakes and those with greater predicted-to-ground-truth trajectory alignment.In the alignment example, one predicted trajectory is 83% similar in the better tracker versus 50% at best in the worse tracker.
  • Localisation: HOTA incorporates localisation by evaluating scores across α thresholds, so improved localisation increases the metric rather than being ignored beyond a fixed matching threshold.This combines localisation with detection and association in one score.
  • Problems with IDF1 and Track-mAP: HOTA evaluates all trajectory pairs and decomposes error types, avoiding IDF1’s best-match-only limitation and Track-mAP’s hard positive threshold that hides intermediate improvements.Track-mAP counts trajectories as positive only when their detection-association Jaccard reaches at least 50%.

10 Evaluating Trackers with HOTA on MOTChallenge

HOTA was evaluated against MOTA and IDF1 on published MOT17 trackers, revealing both shared broad trends and meaningful ranking differences. Its decomposed scores expose detection, association, localization-threshold behavior, and multidimensional tracker trade-offs.

  • Ranking Methods by HOTA: HOTA, MOTA, and IDF1 generally identify strong and weak MOT17 trackers similarly, but their rankings can differ substantially.The top method under each metric remains within the top four across the other metrics, while HOTA rankings generally align more closely with IDF1 than MOTA.
  • HOTA Components: HOTA computes a geometric mean of DetA and AssA, so its score lies between detection and association accuracy.Both components improve for stronger trackers; top performers tend to be better at association than detection, while association varies more across trackers.
  • Multidimensional Analysis: HOTA-based multidimensional analysis identifies tracker benefits, pitfalls, and Pareto-optimal methods for applications with different tracking requirements.Trackers on the Pareto front are state-of-the-art in at least one aspect of tracking performance.
  • Localization Analysis: Performance across localization thresholds reveals whether trackers remain effective under loose matching or stricter localization requirements.This threshold sweep exposes tracker properties that a single evaluation score can obscure.
  • HOTA Components: DetRe, DePr, AssRe, and AssPr alone do not fully explain tracker differences; higher-level DetA, AssA, and HOTA provide more informative unified summaries.The higher-level metrics combine multiple aspects of tracking performance while retaining separate detection and association interpretations.
  • Analysing Metrics across Detectors: MOTA is most sensitive to detector choice, whereas HOTA and IDF1 show similar detector trends and association scores depend less on detector input.MOTA follows DetA closely because it is largely a proxy for detection accuracy; better detectors still aid association.

11 Human Visual Assessment Study

The study compared metric rankings with human judgments using controlled head-to-head videos from MOT17. HOTA agreed with users more often than MOTA and IDF1, especially among MOT researchers.

  • Study Design: The study used 6-second MOT17 clips and randomized tracker and video ordering to reduce presentation and attention-related bias.The test set was divided into 36 clips, with 6 seconds selected after trials of 3-, 6-, and 9-second videos.
  • Study Design: Metric-disagreement pairs were selected when the metrics ranked trackers oppositely and each difference exceeded 0.05.The sign constraint ensured disagreement, while the magnitude constraints ensured meaningful score differences for both metrics.
  • Results: HOTA agreed with human evaluators on 61.6% of comparisons against MOTA, versus 38.4% for MOTA when equal ratings were excluded.Against IDF1, HOTA agreed with users 72.0% of the time compared with 28.0% for IDF1.
  • Interpretation: HOTA better aligned with human judgments of tracking accuracy than the previous metrics, while the study notes that human evaluation itself is difficult.Participants took more than two minutes per pair on average, and the authors frame researcher agreement as evidence of practical relevance.

12 Conclusion

The paper introduces HOTA as a balanced MOT metric addressing limitations in prior measures. Experiments and user studies support its use for evaluating detection, association, and tracking accuracy.

  • Contribution: HOTA is a novel MOT metric that equally weights detection and association accuracy through a simple formulation.The paper presents it as addressing shortcomings of previously used metrics, including MOTA's association limitations and non-intuitive detection behavior in IDF1 and Track-mAP.
  • Evidence: Experiments on state-of-the-art MOTChallenge trackers and a large-scale user study support HOTA over MOTA and IDF1.The user study found human visual assessment aligned better with HOTA than with either alternative.
  • Implication: The authors argue that HOTA can support future tracking algorithms designed and benchmarked around both detection and association quality.This proposed role is presented as groundwork for changes in tracking research.
Loading 2009.07736v2…