PaperlayerView exact revision on arXiv

Source-linked AI summary

A Novel Performance Evaluation Methodology for Single-Target Trackers

Matej Kristan, Jiri Matas, Ales Leonardis, Tomas Vojir, Roman Pflugfelder, Gustavo Fernandez, Georg Nebehay, Fatih Porikli, Luka Cehovin

arXiv:1503.01313v3cs.CV

TL;DR

Single-target tracker evaluation lacks standardized, objective comparison across performance measures, datasets, and evaluation systems. The paper proposes an interpretable ranking methodology with statistical and practical equivalence tests, a diverse per-frame annotated dataset, and an integrable multi-platform system. The methodology is demonstrated in a large benchmark involving 38 trackers, while its theoretical analysis shows advantages for reinitialization-based and per-frame evaluation.

  • Problem

    Tracker proposals have expanded without a standardized methodology for objective comparison, motivating requirements for interpretable measures, datasets, and evaluation systems.

  • Method

    The paper combines interpretable performance measures, reinitialization-based evaluation, statistical and practical equivalence tests, rank visualization, clustered diverse data, and per-frame annotation.

  • Results

    38 trackers are comparatively analyzed in the paper’s largest benchmark to date, while WIR is unbiased and has variance no greater than NOR.

  • Takeaways & Limitations

    Per-frame annotation and reinitialization-based evaluation provide more accurate and less biased estimates than the compared non-reinitialized or per-sequence alternatives.

  • Takeaways & Limitations

    The ten-frame burn-in choice is reasonable for this dataset’s approximately consistent frame rate and non-high-speed sequences, but burn-in depends on frame rate and object speed.

Abstract

from arXiv · show

This paper addresses the problem of single-target tracker performance evaluation. We consider the performance measures, the dataset and the evaluation system to be the most important components of tracker evaluation and propose requirements for each of them. The requirements are the basis of a new evaluation methodology that aims at a simple and easily interpretable tracker comparison. The ranking-based methodology addresses tracker equivalence in terms of statistical significance and practical differences. A fully-annotated dataset with per-frame annotations with several visual attributes is introduced. The diversity of its visual properties is maximized in a novel way by clustering a large number of videos according to their visual attributes. This makes it the most sophistically constructed and annotated dataset to date. A multi-platform evaluation system allowing easy integration of third-party trackers is presented as well. The proposed evaluation methodology was tested on the VOT2014 challenge on the new dataset and 38 trackers, making it the largest benchmark to date. Most of the tested trackers are indeed state-of-the-art since they outperform the standard baselines, resulting in a highly-challenging benchmark. An exhaustive analysis of the dataset from the perspective of tracking difficulty is carried out. To facilitate tracker comparison a new performance visualization technique is proposed.

1 INTRODUCTION

The paper identifies a lack of standardized objective comparison for single-target trackers and proposes coordinated requirements for measures, datasets, and evaluation systems. Its methodology combines interpretable ranking, tracker-equivalence tests, diverse per-frame annotation, and accessible evaluation infrastructure.

  • 1 INTRODUCTION: Tracker proposals have grown rapidly, but objective comparison lacks a standardized evaluation methodology.
  • 1 INTRODUCTION: The study focuses on single-camera, single-target, model-free, causal trackers for short-term tracking with rotated bounding-box outputs.
  • 1 INTRODUCTION: Evaluation requires interpretable measures supporting tracker comparison and well-defined equivalence, diverse annotated datasets, and systems that fully use data consistently.
  • 1 INTRODUCTION: The proposed methodology uses two interpretable measures, analyzes reinitialization at failures, tests statistical and practical accuracy differences, and visualizes tracker ranks.
  • 1 INTRODUCTION: The dataset is built through visual-property clustering and per-frame attribute annotation, while the multi-platform system supports tracker integration and distributed VOT2013 and VOT2014 experiments.
  • 1 INTRODUCTION: A comparative analysis evaluates 38 trackers, making the benchmark the largest to date, alongside a dataset analysis focused on tracking success.

2 RELATED WORK

Prior work offers many tracking measures, datasets, visualizations, and evaluation systems, but these approaches can obscure diverse performance, bias comparisons, or limit tracker integration. The paper’s related-work discussion motivates more interpretable, diverse, and interactive evaluation.

  • 2 RELATED WORK: Existing measures range from center error and overlap to failure rate, F-score, precision, and combined scores, but combined measures provide limited insight.
  • 2 RELATED WORK: Using many measures can bias results because correlated measures may overrepresent particular aspects of tracking performance.
  • 2 RELATED WORK: At 0.5 overlap, bounding boxes can still fit objects well, illustrating limitations of a single overlap threshold for judging localization quality.
  • 2 RELATED WORK: Center-distance threshold measures depend strongly on object size, while normalized center error can be used to alleviate that problem.
  • 2 RELATED WORK: Rank averaging combines rankings across several performance measures when a single tracker list is required.
  • 2 RELATED WORK: Precision and success plots summarize tracking performance using center error or region overlap, while point-based visualizations support easier comparison than curves.
  • 2 RELATED WORK: Common datasets often lack standard ground truth, balanced visual phenomena, or per-frame attributes, allowing sequence composition to bias results.
  • 2 RELATED WORK: Existing evaluation systems focus on surveillance, utilities, or predetermined trackers, and none support tracker interaction; publication-mined ranks also lack sufficient comparisons for recent trackers.

3 VISUAL OBJECT TRACKER EVALUATION

The evaluation methodology combines interpretable accuracy and robustness measures with repeated runs, failure re-initialization, statistical and practical equivalence testing, and ranking-based visualization. Theoretical and experimental analyses show that re-initialization and per-frame annotation reduce estimator bias or variance under the stated conditions.

  • Performance measures: Accuracy measures bounding-box overlap with ground truth, while robustness counts tracker failures that require re-initialization.A failure triggers re-initialization when the overlap drops to zero.
  • Re-initialization: The methodology delays re-initialization by Nskip = 5 frames and excludes Nburnin = 10 post-reinitialization frames from accuracy computation.These choices address immediate repeated failures and transient stabilization bias after re-initialization.
  • Performance estimation: Trackers are run repeatedly to average per-frame accuracy and compute average failures across experiment repetitions.Per-sequence results are combined using weights proportional to sequence lengths, equivalently forming a dataset super-sequence.
  • Ranking and visualization: Equivalent trackers are grouped when their performance is indistinguishable, and corrected ranks support comparison while accuracy-robustness plots visualize performance and rankings.The methodology treats tracker performance as a hidden state and studies evaluation estimators through bias and variance.
  • Tracker equivalence: The Wilcoxon Signed-Rank test evaluates statistical accuracy differences because frame accuracies are not always normally distributed.Statistical difference is complemented by a practical-difference threshold because statistically significant changes may be negligible in practice.
  • Estimator analysis: The WIR estimator is unbiased with variance no greater than NOR, whereas NOR becomes increasingly biased and can estimate half the true overlap when p = 1.With increasing sequence count, WIR remains asymptotically unbiased and has lower variance; unequal sequence lengths can increase NOR variance further.
  • Estimator analysis: Per-frame annotation yields a less biased visual-attribute estimator than per-sequence labeling: under the example, MGLA bias is 0.67µB versus 0.09µB for MPFA.The analysis reports that per-frame annotation remains preferable even with a reasonable number of mislabeled frames.

4.1 VOT2014 challenge

The VOT2014 challenge applied the proposed tracker-comparison methodology through an annotated dataset and a portable evaluation kit. It evaluated 38 trackers using standardized experiments, repeated runs, and common parameter-selection requirements.

  • Challenge setup: VOT2014 implemented the proposed comparison methodology with an annotated dataset, a Matlab/Octave evaluation system, and a minimal tracker-integration API.The evaluation kit was designed for multi-platform and multi-programming-language compatibility.
  • Challenge setup: Participants downloaded the evaluation kit, integrated their trackers, and ran predefined experiments locally with one parameter set shared across experiments.This placed parameter selection and correction of implementation errors with the tracker authors.
  • Experiments: Experiment 1 initializes trackers on ground-truth bounding boxes and runs them on all VOT2014 sequences as the baseline.The experiment establishes the standard initialization condition for comparison.
  • Experiments: Experiment 2 repeats the baseline with perturbations of ±10% in position and size and ±0.1 radians in rotation, using 15 runs per sequence.The evaluation kit performed the experiments automatically to obtain better performance statistics.
  • Trackers: The challenge included 38 trackers, comprising 33 original submissions and 5 highly cited baseline trackers contributed by the VOT committee.Most participating trackers had been published recently and were described as state-of-the-art.

4.2 The VOT2014 Dataset

The VOT2014 dataset was designed to maximize visual diversity while remaining compact, using feature-based clustering, manual selection, bounding-box annotation, and per-frame attribute labels. It also supports sequence-specific analysis of annotation variability and practical differences.

  • Dataset construction: The dataset construction sought broad visual diversity with relatively few sequences because a large collection alone does not guarantee diverse visual properties.The approach was intended to avoid redundant sequences that would slow evaluation without adding visual variety.
  • Dataset construction: An initial pool of 394 sequences was reduced to 193 for automatic feature computation, then clustered into 12 groups and manually reduced to 25 sequences.The features represented visual properties including illumination, size change, motion, clutter, camera motion, blur, aspect-ratio change, and color change.
  • Dataset construction: Each sequence was represented by a 10-dimensional visual-feature vector before unsupervised affinity-propagation clustering.Manual selection preserved phenomena such as occlusion within the compact final selection.
  • Annotation: Targets were manually annotated with axis-aligned or rotated bounding boxes, with rotated boxes used for elongated, rotating, or deforming targets.Most axis-aligned boxes contained at least 60% target pixels for most frames.
  • Annotation: Every frame received one of five challenge attributes—camera motion, illumination change, motion change, size change, or occlusion—or the neutral label when none applied.Figure 3 summarizes frame sizes, ground-truth box sizes, sequence lengths, and attribute proportions.
  • Practical differences: Seven experts annotated four frames per sequence three times, producing 15,960 difference samples per sequence for practical-difference thresholds.The resulting boxplots showed that the threshold varied across sequences, especially with object properties.

4.3 Study of the methodology parameters

The methodology parameters were tested empirically to reduce re-initialization bias and stabilize performance estimates. Results support a 10-frame burn-in period, five skipped frames, and difference tests for more stable tracker ranking.

  • Burn-in period: 3249 overlap sequences showed that overlap-change rate stabilizes 10 frames after re-initialization, setting Nburnin = 10.The study used seven trackers on VOT2013 sequences recorded at approximately 20 frames per second.
  • Burn-in period: Including burn-in frames slightly reduces average accuracy, with larger drops for trackers that fail more often.The effect was quantified on four state-of-the-art trackers and two baseline trackers using VOT2014.
  • Re-initialization skipping: Failure counts change most between one and three skipped frames, then remain stable as Nskip increases.Relative changes are consistent across trackers, and the ranking remains stable across skipping values.
  • Difference tests: Difference tests reduce rank variance compared with rankings that assign distinct ranks to statistically or practically indistinguishable trackers.The experiment sampled 50 random subsets of 15 VOT2014 sequences and ranked six trackers under multiple settings.

4.4 Comparison with related methodologies

The paper experimentally compares re-initialization-based and non-re-initialization estimators on VOT2014. Repeated sequence sampling estimates their average overlap and variability across different numbers of sequences.

  • Estimator comparison: Re-initialization-based estimator Y(WIR) and non-re-initialization estimator N(NOR) were compared by repeatedly sampling K VOT2014 sequences.Each tracker was run once with re-initializations and once without, and average overlap was computed for each sampled set.
  • Estimator comparison: Average overlap values are reported for each estimator and tracker, with standard deviations shown in brackets.The table varies K and distinguishes the re-initialization condition using the R column.

K R DSST KCF SAMF CT FRT Struck

The VOT2014 analysis compares trackers across robustness, accuracy, visual attributes, and sequence difficulty. It identifies leading trackers, attribute-specific weaknesses, and challenging sequences or frames.

  • Attribute analysis: MGLA cannot distinguish performance for motion change, scale change, and occlusion, whereas MPFA shows clear performance differences.MPFA has lower variance than MGLA at p < 0.01, while an F-test finds dispersion differences at p < 0.05.
  • Robustness: PLT13, PLT14, MatFlow, and DGT are the top-performing trackers in robustness across baseline and noise experiments.PLT13 and PLT14 use holistic models, whereas MatFlow and DGT are part-based trackers.
  • Accuracy: DSST, SAMF, KCF, and DGT are the top-performing trackers in accuracy.DSST, SAMF, and KCF are correlation-filter-based trackers derived from MOSSE and use holistic HOG models.
  • Attribute analysis: DGT and eASMS lose performance under illumination changes, while DSST, KCF, SAMF, and PLT13 remain top performers in the attribute analysis.The DGT degradation is attributed in the text to its heavy reliance on color information in segmentation.
  • Ranking visualization: Per-attribute normalization changes local tracker ranks because sequence-pooled plots depend on the dataset’s visual-attribute distribution.The overall tracker layout remains similar between normalized and raw plots.
  • Tracker comparison: Most tested trackers are highly competitive, while baseline trackers NCC, MIL, CT, FRT, and IVT occupy the bottom-left of AR-rank plots.NCC is the least robust tracker, and Struck lies in the middle of the AR plots.
  • Attribute analysis: Occlusion, motion change, and size change are the most difficult visual attributes, followed by camera motion and illumination change.Difficulty is measured by the median expected failures per hundred frames across trackers.
  • Sequence analysis: The most challenging sequences include motocross, hand2, diving, fish2, bolt, and hand1, driven mainly by camera motion, object motion, size changes, or pose variation.Motocross combines camera and object motion with rapid size changes; hand2 emphasizes challenging hand-pose variation.

5 CONCLUSION

The paper presents an interpretable tracker-evaluation methodology, a visually diverse annotated dataset, and a cross-platform evaluation system. Experiments with 38 trackers reveal specialization between robustness and accuracy, attribute-dependent difficulty, and application-dependent notions of the best tracker.

  • The methodology evaluates trackers using interpretable measures and accounts for tracker equivalence through statistical significance and practical differences.
  • 25 color sequences are annotated per frame with visual attributes and rotated bounding boxes, supporting per-attribute and sequence-level analysis.
  • 38 trackers were compared, making the evaluation the largest benchmark to date; most tested trackers outperformed standard baselines and performed favorably to Struck.
  • Trackers tend to specialize either for robustness or accuracy, and none consistently outperformed others across all measures and sequence attributes.
  • Occlusion, motion change, and size change were the most challenging visual attributes, while trackers rarely failed on the traditionally difficult David sequence.
  • The notion of the best tracker varies by application because some settings prioritize accuracy while others prioritize autonomous robustness.

APPENDIX A DERIVATION OF NOR AND WIR STATISTICS

The appendix derives the NOR and WIR accuracy statistics using a mixture model for sequence-level tracking accuracy. It computes means and variances separately for successful and failed sequences before substituting them into the general mixture formulas.

  • The appendix models sequence accuracy as a random variable drawn from distributions for sequences with and without tracking failure.
  • NOR scenario: For NOR, the no-failure distribution parameters are computed directly, while failure cases account for overlap dropping to zero after a failure.
  • NOR scenario: The resulting NOR means and variances are substituted into the general formulas to obtain equations (5) and (6).
  • WIR scenario: For WIR, the tracker is reset after failure and frames immediately following the reset are excluded from accuracy computation.
Loading 1503.01313v3…