PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual object tracking performance measures revisited

Luka Čehovin, Aleš Leonardis, Matej Kristan

arXiv:1502.05803v3cs.CV

TL;DR

Visual tracking evaluation lacks consensus on measures, making cross-paper comparisons difficult and potentially biasing results toward particular performance aspects. The paper theoretically and experimentally revisits popular measures and visualizations, identifies equivalent or brittle measures, and narrows evaluation to complementary accuracy and robustness measures. These measures support intuitive visualization and have been adopted as the foundation of recent VOT challenge methodology.

  • Problem

    Visual tracking lacks consensus on evaluation protocols and measures, hindering rigorous comparison across publications.

  • Method

    The paper analyzes popular performance measures and visualizations through theoretical relations, systematic experiments, and theoretical trackers.

  • Results

    The analysis narrows the candidate measures to the A-R pair: accuracy using average overlap and robustness using failure rate.

  • Takeaways & Limitations

    The A-R measures and plots provide an intuitive evaluation foundation adopted by VOT2013 and VOT2014.

Abstract

from arXiv · show

The problem of visual tracking evaluation is sporting a large variety of performance measures, and largely suffers from lack of consensus about which measures should be used in experiments. This makes the cross-paper tracker comparison difficult. Furthermore, as some measures may be less effective than others, the tracking results may be skewed or biased towards particular tracking aspects. In this paper we revisit the popular performance measures and tracker performance visualizations and analyze them theoretically and experimentally. We show that several measures are equivalent from the point of information they provide for tracker comparison and, crucially, that some are more brittle than the others. Based on our analysis we narrow down the set of potential measures to only two complementary ones, describing accuracy and robustness, thus pushing towards homogenization of the tracker evaluation methodology. These two measures can be intuitively interpreted and visualized and have been employed by the recent Visual Object Tracking (VOT) challenges as the foundation for the evaluation methodology.

I. INTRODUCTION

Visual tracking lacks standardized evaluation protocols and a consensus set of performance measures, hindering rigorous comparison across publications. The paper surveys and analyzes existing measures to identify complementary choices for short-term single-target tracking evaluation.

  • Visual tracking lacks definite answers about standard sequences, evaluation protocols, and performance measures.
  • Nonhomogeneous protocols and arbitrary measure choices make cross-publication tracker comparisons difficult and can limit performance insight.
  • The paper combines theoretical relations with systematic experiments to analyze popular performance measures and their comparison value.
  • The study focuses on monocular short-term single-target tracking without complete target disappearance requiring later redetection.
  • Its contributions include surveying measures, identifying equivalent measure clusters, selecting complementary accuracy and robustness measures, and introducing theoretical trackers.
  • The large-scale experiment evaluates 16 state-of-the-art trackers on 25 video sequences, with selected-measure results provided as a byproduct.

II. PERFORMANCE MEASURES

Single-target tracking measures compare predicted annotations with ground truth, but popular measures have different annotation requirements and failure sensitivities. Center error is simple but can misrepresent failures, while overlap measures incorporate position and size.

  • Popular single-target tracking measures lack a de facto standard and assume manual sequence annotations.
  • An object state records center x_t and region R_t, which may require separate annotation for articulated objects.
  • Performance measures summarize agreement between tracker-predicted annotation ΛT and ground-truth annotation ΛG.
  • A. Center error: Center prediction error measures the distance between predicted and ground-truth target centers with minimal point-only annotation.
  • A. Center error: Normalized center error can be misleading because estimated target size proportionally reduces error and may overpower failure information during drift.

B. Region overlap

Region overlap measures compare predicted and ground-truth regions while jointly accounting for position and size. They support sequence-level and thresholded frame-based summaries, but threshold choices and frame-wise detection interpretations impose limitations.

  • Region overlap: Region overlap measures compute overlap between tracker-predicted and ground-truth target regions.Unlike center error, they require region annotations.
  • Region overlap: Overlap accounts for predicted and ground-truth bounding-box position and size simultaneously, avoiding arbitrarily large errors after tracking failure.When a tracker drifts to the background, overlap becomes zero regardless of the distance from the target.
  • Region overlap: Average overlap summarizes all region overlaps across an entire sequence.
  • Region overlap: Pτ counts frames whose overlap exceeds threshold τ and can be normalized by dividing correctly tracked frames by the total frame count.This measure is also called the percentage of correctly tracked frames.
  • Region overlap: The commonly used overlap threshold τ = 0.5 is too high for general-purpose tracking evaluation, especially for non-rigid articulated objects.The threshold can be reached by visually well-overlapping rectangles.
  • Region overlap: Detection-based measures disregard the sequential nature of tracking, while MOTA simplifies to the percentage of correctly tracked frames in short-term single-target tracking.In this setting, there is one object, no false positives or identity switches, and misses are binary per frame.

C. Tracking length

Tracking length measures the number of frames successfully tracked from initialization until the first failure. They explicitly address failure cases but discard the sequence after that failure, making early difficult conditions consequential.

  • Tracking length: Tracking length reports successfully tracked frames from tracker initialization to its first failure.The failure criterion may be manual or automated using a center or overlap threshold; the paper denotes the overlap-based version Lτ.
  • Tracking length: Automated failure criteria improve repeatability over manual visual inspection, but the chosen threshold or measure can affect tracker comparisons.The overlap-based criterion is described as more robust to size changes.
  • Tracking length: Tracking length uses only the sequence portion before the first failure, discarding the remainder of the video.This differs from simple average center-error and overlap measures, which do not explicitly address failure cases.
  • Tracking length: A difficult situation or poor initialization at the sequence beginning can cause failure and make the remainder unusable for the measure.Reliable statistics therefore require many sequences exhibiting varied properties at their beginnings.

D. Failure rate

The failure rate measures how often a tracker requires manual intervention after failure, using the full sequence for evaluation. Fragmentation complements it by describing how failures are distributed over time.

  • Failure rate: The failure rate records manual interventions per frame when a human operator reinitializes the tracker after failure.It models supervised tracking and reflects a real-world setting in which an operator corrects tracker errors.
  • Failure rate: Unlike tracking length, failure rate uses the entire sequence and reduces the influence of its beginning.
  • Failure rate: The failure-rate score depends on the chosen failure threshold, and changing that threshold requires repeating the experiment.
  • Fragmentation: Trajectory fragmentation measures the temporal distribution of failures, reaching 1 for uniformly spaced failures and decreasing as intervals become uneven.It is meaningful only when more than one failure occurs.
  • Fragmentation: Fragmentation supplements failure rate because it cannot indicate how many failures occurred, but distinguishes concentrated failures from uniform failures.

E. Hybrid measures

Hybrid measures combine tracking accuracy and failure information, but their aggregate scores can obscure the separate properties being compared. The paper therefore favors complementary measures and cautions against distorted or threshold-dependent visual comparisons.

  • Hybrid measures: CoTPS combines an accuracy score and a failure score into one threshold-independent overlap-based score, with higher values indicating poorer tracking.
  • Hybrid measures: CoTPS rankings do not reveal whether a tracker scores higher because of average overlap or fewer failed frames.
  • Hybrid measures: Different combinations of non-failure average overlap and failure proportion can produce the same CoTPS score, making interpretation difficult.
  • Hybrid measures: The paper recommends a few complementary measures with well-defined meanings rather than fusing performance properties into one early score.
  • Performance plots: Center-error plots can mislead multi-tracker comparisons when an inferior tracker distorts the scale; bounded overlap plots are an alternative.
  • Performance plots: Measure-threshold plots avoid selecting one threshold, while Pτ summarizes correctly tracked frames across sequences and its AUC equals average overlap.This equality permits direct averaging instead of ROC-like integration and gives AUC a straightforward interpretation.
  • Performance plots: Survival curves summarize overall success but cannot support sequence-wise comparison because sequence order varies across trackers.

III. EXPERIMENTAL COMPARISON OF PERFORMANCE

The experiment compares existing tracking measures across diverse trackers and sequences, showing that several measures carry equivalent information while others are threshold-sensitive or brittle. The analysis identifies average overlap as the most appropriate general comparison measure.

  • Experimental setup: The experiment evaluates selected performance measures by running 16 diverse trackers on 25 visual tracking sequences.The tracker set includes detection-based, holistic generative, and part-based approaches selected to fail in different scenarios.
  • Correlation analysis: A correlation analysis over approximately 10,000 tracker-sequence samples identifies five statistically significant clusters of measures.The clusters group measures with similar information content for tracker comparison.
  • Correlation analysis: The three center-error-based measures form one cluster and are effectively equivalent for tracker comparison.Their similarity is expected because all are based on center error with different averaging methods.
  • Correlation analysis: Overlap, correct-frame, and tracking-length measures form another cluster, but threshold choices and failure timing can change tracker ordering.For example, a tracker that recovers after an early loss may rank higher under average overlap or correct frames than under tracking length.
  • Measure selection: Average overlap is selected as the most appropriate comparison measure because it is simple, scale- and threshold-invariant, sequence-wide, and easy to interpret.It is also highly correlated with the more complex percentage-of-correctly-tracked-frames measure.
  • Measure selection: Failure rate and average overlap on re-initialized trajectories are the least correlated measures, supporting their use for complementary performance aspects.The finding remains unchanged when the analysis is repeated on subsets of approximately half the sequences.

B. Accuracy vs. robustness

The paper represents short-term tracking performance through complementary accuracy and robustness measures. It pairs average overlap with failure rate and visualizes the pair in a two-dimensional plot, interpreting robustness as reliability for display.

  • Accuracy and robustness: The proposed A-R pair measures accuracy with average overlap and robustness with failure rate on re-initialized trajectories.Average overlap accounts for object size without requiring a threshold, while failure rate counts tracker failures.
  • Accuracy and robustness: Average overlap and failure rate are selected because they measure different aspects of tracker performance with low correlation.Average overlap alone does not adequately capture robustness when a tracker fails early.
  • Failure threshold: The failure threshold is set to τ = 0 so that only complete failures with zero overlap trigger reinitialization.Higher thresholds can restart trackers after small errors and make the final score difficult to interpret.
  • Visualization: The A-R pair is visualized as a two-dimensional scatter plot, with higher vertical position indicating greater accuracy and farther-right position indicating greater robustness.Because robustness has no upper bound, the visualization interprets it as reliability.
  • Visualization: Reliability is scaled as an exponential failure-distribution quantity that can be interpreted as the probability of tracking successfully for S frames after a failure.The formulation and choice of S change visualization scaling but not tracker ordering.

C. Theoretical trackers

The paper introduces theoretical trackers as parameter-free reference points for interpreting accuracy–robustness results. Their contrasting behaviors clarify how tracker scores relate to sequence properties and practical tracking choices.

  • Tracker behaviors: TTA reports the full image region, avoiding failures but producing loose predictions that are accurate mainly when the target occupies much of the frame.Its behavior measures the target’s average relative size and provides a reference for high robustness with potentially poor accuracy.
  • Tracker behaviors: TTF tracks one frame and then deliberately fails, preserving high accuracy while incurring an extremely high failure rate.This places TTF at the high-accuracy, low-robustness extreme of the A-R plot.
  • Tracker behaviors: TTO is an oracle-like fixed-size tracker that predicts the target center correctly but cannot adapt its bounding-box size.It represents a practical performance limit for trackers using the initialization bounding-box size.
  • Interpretation guide: Theoretical trackers provide simple, parameter-free reference points that place evaluated trackers into context in A-R visualizations.Their scores can be computed directly from ground truth and interpreted without additional parameters.
  • Interpretation guide: In averaged results, LGT is most robust, while KCF, IVT, and TLD surpass it in accuracy; TLD’s low robustness resembles TTF because of frequent reinitializations.FOF shows the opposite trade-off, combining high robustness with very low accuracy like TTA.
  • Sequence dependence: Per-sequence plots reveal that tracker rankings vary with sequence properties, including object-size changes, deformation, motion, and background variation.TTO often performs best, but size-adaptive trackers can surpass it when the object changes scale.

E. Fragmentation

Fragmentation complements failure rate by showing how failures are distributed over a sequence. Theoretical trackers further support sequence interpretation and dataset construction from basic tracker behaviors.

  • Fragmentation indicator: Low fragmentation indicates that failures are likely clustered around a specific event in the sequence.This can reveal a problematic event that aggregate failure counts alone do not identify.
  • Fragmentation indicator: Fragmentation provides additional tracker information beyond the number of failures by describing their temporal distribution.It can reduce the need to inspect raw tracking results to infer whether failures are concentrated or dispersed.
  • Theoretical trackers: Theoretical trackers can be applied to any annotated sequence without parameters to expose sequence properties from basic tracker behaviors.These properties can assist evaluation-dataset construction and result interpretation.
  • Theoretical trackers: TTA’s robustness remains good because it never fails, while its accuracy is high mainly when the target covers a large part of the image.This makes TTA a reference for the sequence property of relative target size.
  • Evaluation framework: The proposed A-R pair combines average overlap for accuracy with failure rate for robustness, while the A-R plot visualizes both dimensions.The paper also introduces fragmentation as an additional indicator and uses theoretical trackers to annotate sequence properties automatically.
  • Evaluation framework: The measures became the foundation of the VOT2013 and VOT2014 evaluation methodology, although consistent evaluation requires more than narrowing the measure set.The authors identify automatic sequence labeling and reducing annotated frames as future directions.

APPENDIX A REFORMULATION OF COTPS [6] MEASURE

The appendix reformulates the CoTPS measure in terms of average overlap and failure percentage. It also proves that the AUC measure equals average overlap for a sequence of frame overlaps.

  • CoTPS reformulation: The tracking failure factor λ0 is the percentage of frames with zero overlap between the predicted and ground-truth regions.The corresponding weight β equals one minus λ0 because it counts frames with positive overlap.
  • CoTPS reformulation: The accuracy component Ω equals one minus the average positive overlap, 1−φ̂, after reformulating its threshold integral.The derivation uses the average overlap over frames whose overlap is greater than zero.
  • CoTPS reformulation: CoTPS is therefore a function of average overlap and the percentage of frames with zero overlap.The reformulation follows from writing whole-sequence average overlap as (1−λ0)φ̂.
  • AUC reformulation: The AUC measure integrates P(τ), the number of frame overlaps at least τ, over thresholds from 0 to 1.The overlap values are treated through the step-function behavior of P.
  • AUC reformulation: The appendix proves that AUC equals average overlap for the sequence of frame overlaps.This establishes equivalence between the integral-based AUC and the average-overlap measure.
  • AUC reformulation: Because P is a step function, its integral reorganizes into the sum of the frame-overlap values.The proof handles repeated overlap values by accounting for their multiplicity.
Loading 1502.05803v3…