PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Multi-Object Tracking Baselines: Hyperparameter Optimization with Multi-Fidelity Greedy Coordinate Search

Momir Adžemović

arXiv:2609.12261v1cs.CV

TL;DR

Manual tuning and limited reporting of tuning procedures leave the scope of MOT baseline performance uncertain, while HPO remains rarely used in MOT. The paper compares HPO methods across two datasets and four trackers, introducing MFGCS; tuned methods outperform manual and published configurations, and MFGCS reaches the target faster than TPE in most combinations.

  • Problem

    MOT hyperparameters are commonly chosen by hand, reported tuning procedures are often poorly documented, and prior HPO studies are limited or predate modern deep-detector trackers and HOTA evaluation.

    Source: §1 · p. 2

  • Method

    The study compares eight optimizers and evaluates MFGCS and TPE across two datasets and four trackers, with MFGCS screening coordinate-wise candidates on scene subsets before full-dataset evaluation.

    Source: §1 · p. 2, Object detection · p. 8

  • Results

    Across all tracker–dataset cells, both optimizers outperform published scores by 1.45 to 16.05 HOTA points and hand-tuned configurations by 0.24 to 4.38 points; MFGCS reaches the per-tracker target faster in seven of eight.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Commonly used configurations under-tune these trackers on both benchmarks, and validation-selected configurations also exceed published numbers on held-out test splits.

    Source: §6 · p. 13

    Coordinate-wise search assumes useful local structure, although tracker hyperparameters interact and the assumption is only approximate.

    Source: §4 · p. 5

Abstract

from arXiv · show

Multi-object tracking (MOT) is dominated by the tracking-by-detection paradigm, whose methods typically rely on a small set of hyperparameters that are conventionally chosen by hand. Tuning them requires repeated expert-guided experimentation, while the procedures used to select reported values are often not systematically evaluated or fully documented. Hyperparameter optimization (HPO) automates this process, yet it remains rarely used in MOT, and existing studies applying HPO to MOT predate modern deep-detector-based trackers and HOTA evaluation. We systematically apply HPO across two datasets and four tracking-by-detection methods. We also propose Multi-Fidelity Greedy Coordinate Search (MFGCS), which optimizes one hyperparameter at a time by first evaluating candidate values on a small subset of scenes and re-evaluating only promising candidates on the full dataset. Across all eight tracker-dataset combinations, the Tree-structured Parzen Estimator (TPE) and MFGCS outperform both our hand-tuned configurations and the corresponding published results, with improvements of up to 4.38 and 16.05 HOTA points, respectively. MFGCS also reaches a predefined HOTA target faster than TPE in seven of the eight combinations. Within each tracker-dataset pair, all optimizers share the same search space and evaluation pipeline, isolating the effect of the search strategy. We release the code and tuned configurations to enable future work to compare against systematically optimized rather than default or manually tuned baselines.

1. Introduction

Manual tuning can obscure the performance of tracking-by-detection baselines, while prior MOT HPO studies have limited scope. The paper compares optimization strategies and introduces MFGCS, reporting gains over manual and published configurations.

1. Introduction

  • Tracking-by-detection methods add hyperparameters as association mechanisms grow more complex, including thresholds, gating margins, and track-lifecycle settings.The detector and association pipeline can be developed or replaced independently, but association remains largely based on hand-designed methods.Source: §1 · pp. 1–2
  • Manual tuning takes hours to about a day per tracker, and incompletely documented tuning can confound comparisons between trackers or limit transfer across benchmarks.The study focuses on DanceTrack and SportsMOT, whose motion and appearance characteristics differ substantially from those of many original tracking-by-detection benchmarks.Source: §1 · p. 2
  • Prior MOT HPO remains rare; earlier work predates deep-detector trackers and HOTA evaluation, and typically examines one strategy or benchmark.The study presents its work as a broad comparison of HPO strategies for modern tracking-by-detection methods.Source: §1 · pp. 2–3
  • The study compares eight optimizers, then evaluates MFGCS and TPE across two datasets and four tracking-by-detection methods using shared search spaces per tracker.MFGCS compares one hyperparameter's candidate values on a small scene subset and evaluates the full dataset only for proposed updates, which are accepted only if they improve the full-dataset score.Source: §1 · p. 2
  • MFGCS and TPE improve hand-tuned configurations across all eight tracker–dataset combinations, with gains of up to 4.38 HOTA points.The better validation-selected configuration also exceeds the corresponding published test result for every tracker–dataset pair, by up to 8.69 HOTA points.Source: §1 · p. 3

2. Related work

The related work covers black-box and multi-fidelity HPO, then places MOT applications within a sparse and limited literature. The paper distinguishes its multi-strategy, multi-dataset evaluation from earlier MOT tuning studies.

Black-box hyperparameter optimization

  • Black-box HPO includes random search, model-based samplers such as TPE and GP-BO, and derivative-free direct search represented here by Greedy Coordinate Search.TPE models densities for well-performing and remaining configurations; GP-BO uses a posterior and acquisition function.Source: §2 · p. 3

Multi-fidelity optimization

  • Multi-fidelity methods allocate more evaluation resources to promising configurations, with this study defining fidelity by evaluated scenes and using full-dataset scores to accept MFGCS updates.Hyperband and BOHB are evaluated on that scene-subset fidelity axis; MFGCS applies it within coordinate-wise search.Source: §2 · p. 4

Hyperparameter optimization applied to MOT

  • Prior MOT HPO applications are sparse; the only earlier multi-optimizer comparison cited predates deep-detector trackers and HOTA, while more recent work generally uses one strategy in one domain.The closest prior study also tunes four trackers with HOTA, but uses one optimizer on one domain-specific dataset.Source: §2 · p. 4

3. Tracker hyperparameter optimization

Tracker tuning is formulated as single-objective black-box optimization over a bounded hyperparameter space. Because evaluations vary in cost, the paper compares computational effort using cumulative scenes evaluated rather than trial counts.

3. Tracker hyperparameter optimization

  • The optimization seeks the configuration θ* in bounded search space Θ that maximizes tracking performance f(θ; D) on dataset split D.The discrete tracking and evaluation pipeline provides no usable gradient with respect to tracker hyperparameters, so the objective is treated as a black box.Source: §3 · p. 4
  • The paper uses cumulative scenes evaluated as its common cost measure because subset-based optimizers incur different costs per trial.The measure counts every objective evaluation, including initial, low-fidelity, and full-dataset evaluations, whether or not an update is accepted.Source: §3 · p. 5
  • The scene-count cost measure assumes scenes have approximately similar lengths; when lengths differ substantially, evaluated frames would better represent computational effort.A shared budget is needed because evaluation costs vary enough to make trial counts alone unsuitable for comparison.Source: §3 · pp. 4–5

4. Multi-Fidelity Greedy Coordinate Search

MFGCS uses low-cost scene subsets to screen coordinate-wise candidate values, reserving full-dataset evaluation for proposed updates. Its subset comparisons are paired across candidates, while full-data performance determines acceptance.

4. Multi-Fidelity Greedy Coordinate Search

  • MFGCS first compares candidate values on a small scene subset, then evaluates only promising candidates on the complete dataset to reduce full-dataset evaluation cost.This design targets comparable tracking performance with fewer evaluated scenes than methods that assess every candidate on the full split.Source: §4 · p. 5
  • All candidate values for a hyperparameter are evaluated on the same scene subset, reducing scene-selection effects in relative comparisons; full-dataset improvement is required to accept an update.Subset scores identify promising candidates but may be noisy estimates of complete-split performance.Source: §4 · p. 5

4.1. Algorithm

  • The algorithm sweeps hyperparameter coordinates, searches one coordinate at a time while holding others fixed, and skips full-dataset evaluation when no candidate beats the current value on the subset.A proposed update is accepted only if its full-dataset score improves by more than ε; this coordinate-wise approach assumes useful local structure despite hyperparameter interactions.Source: §4.1 · p. 5, §4 · p. 5

19 end

MFGCS tunes one hyperparameter at a time, using scene subsets to compare candidates before evaluating the selected value on the full dataset. Numeric, integer, and categorical parameters use different search procedures, and coordinate-level stopping concentrates later sweeps on parameters that continue to improve.

4.2. Components

  • A shared subset of m scenes is used to compare all candidates for a coordinate, and only the selected candidate is evaluated on the full dataset.The common subset keeps low-fidelity comparisons consistent.Source: §4.2 · p. 6
  • Continuous parameters use coarse-to-fine grid search, integer candidates are rounded and deduplicated, and categorical parameters are exhaustively enumerated.The continuous search retains the best of g evenly spaced values and contracts the interval over r rounds.Source: §4.2 · pp. 6–7, Figure 1 · p. 7
  • A coordinate is removed after two consecutive sweeps without an accepted update, and the outer loop stops at a no-update sweep, the sweep limit, or the full-dataset budget.The bootstrap evaluation counts toward that budget.Source: §4.2 · p. 7

5. Experimental evaluation

The study evaluates four tracking-by-detection methods on DanceTrack and SportsMOT, searching on validation splits and reserving test splits for final evaluation. HOTA is the optimization objective, with detectors held fixed within each dataset so measured differences come from tracker configurations.

5.1. Experimental setup

  • DanceTrack validation has 25 scenes, while SportsMOT validation has 45 scenes and its test split is more than three times larger.DanceTrack features similar-appearing targets, crossings, and re-entries; SportsMOT has fast irregular motion and moving-camera displacement.Source: §5.1 · pp. 7–8
  • Searches run on validation splits, while test splits are reserved for final evaluation.Source: §5.1 · p. 8
  • HOTA is the optimization objective because it jointly captures detection and association through a geometric mean without an explicit weighting between them.The study also reports DetA, AssA, MOTA, IDF1, and IDSW.Source: §5.1 · p. 8
  • Cumulative scene count measures search effort, while wall-clock time is reported as time-to-target only for directly comparable runs.Time-to-target records when running-best HOTA first reaches a predefined target.Source: §5.1 · p. 8
  • The study tunes SORT, ByteTrack, SparseTrack, and MoveSORT-KF, while excluding appearance-based and observation-centric variants.Within each dataset, the public YOLOX-X detector is frozen and identical across configurations, optimizers, and trackers.Source: §5.1 · p. 8, Object detection · p. 8

Object detection.

Across the test comparisons, tuned configurations outperform every published baseline for which a published value is available, and validation results also favor optimization over published and hand-tuned configurations. The study selects MFGCS and plain TPE for the broader comparison, and its ablations identify grid search and scene-subset size as consequential design choices.

Object detection.

  • Eight optimizers span six families and share a tracker-specific search space, but the 100 full-fidelity-trial cap does not equalize computational budgets.Source: Object detection · p. 8, Table 1 · p. 9

5.2. Choosing the optimization methods

  • 53.69% HOTA makes MFGCS the top method in the initial comparison; plain TPE is selected over warm-started TPE despite the latter’s 53.35% versus 53.27%.The text rejects warm-starting because it would make the sampler depend on the hand-tuned configuration.Source: §5.2 · p. 9, Table 2 · p. 10

5.3. Main results

  • 6.15 HOTA points on DanceTrack and 8.69 on SportsMOT are the maximum reported test-split gains over published baselines, and every published baseline is beaten.Source: §5.3 · p. 9, Table 3 · p. 11
  • Both optimizers outperform published and hand-tuned scores in all eight validation settings, while MFGCS reaches the per-tracker target faster than TPE in seven.On DanceTrack, validation-selected scores move by at most 0.58 HOTA points on test; all four SportsMOT test scores are 6.06 to 8.83 points below validation.Source: §5.3 · p. 10, Table 4 · p. 11

5.4. Ablation studies

  • 53.69% HOTA for grid search exceeds the best ternary variant’s 53.19% and random variant’s 52.90%, so grid search is carried forward.Three contraction rounds beat two by 1.1 HOTA points, while resolution beyond g = 5 did not help.Source: Table 5 · p. 12, §5.4 · p. 12
  • At matched cost of 1297 scenes, the full-fidelity variant scores 53.08% HOTA versus MFGCS’s 53.69%, while MFGCS uses one-third the 3950-scene full-fidelity budget.Source: Table 5 · p. 12, §5.4 · p. 12
  • For grid search, m = 6 reaches 53.69% HOTA, compared with 52.95% at m = 3 and 53.33% at m = 9; ternary performs best at m = 3 with 53.19%.The text attributes the lower m = 3 grid result to noisier comparisons discarding good candidates before full evaluation.Source: §5.4 · p. 12

6. Conclusion

The controlled comparison shows that commonly used configurations are under-tuned and that systematic optimization improves results across the evaluated tracker–dataset pairs. MFGCS offers faster tuning in most combinations, while retuning can change the apparent gap between trackers.

6. Conclusion

  • Across all tracker–dataset cells, MFGCS and TPE beat published scores by 1.45–16.05 HOTA points and hand-tuned configurations by 0.24–4.38 points; better validation-selected configurations also beat published test scores by 1.9–8.7 points.The comparison uses a shared search space and evaluation pipeline within each tracker–dataset pair.Source: §6 · p. 13
  • MFGCS reaches a higher-scoring configuration in under two hours unattended, compared with hand-tuning that takes hours to a full day per tracker.MFGCS reaches the per-tracker target faster than TPE in seven of eight combinations; its scene-subset stage alone reduces the budget threefold but needs to be paired with the search to pay off.Source: §6 · p. 13
  • MFGCS and TPE differ by at most 1.41 HOTA points in any cell and by under 0.8 in seven of eight cells.At equal cost, MFGCS with scene-subset evaluation scores 0.61 HOTA points above full-fidelity coordinate search.Source: §6 · p. 13
  • The results indicate that tuning can alter comparisons between trackers, so a published score and a tracker’s achievable score are not equivalent.The authors release optimized configurations as tuned baselines for future comparisons.Source: §6 · p. 13

Appendix A. Experimental details

The appendix documents the data, inference, search-space, and optimizer settings used in the experiments, along with the additional-results material. These details specify how trials were made reproducible and how each tracker’s tuning problem was bounded.

Appendix A. Experimental details

  • The experimental-details section organizes the configurations by detector, inference and postprocessing pipeline, search spaces, and optimizer settings.These are the four topics the appendix says it covers.Source: Appendix A · p. 13

Appendix A.1. Object detection

  • A frozen YOLOX-X detector supplies cached detections, so every trial uses identical detections and the objective is deterministic.A warm-cache DanceTrack-val pass takes two to three minutes, compared with roughly 1.5 hours with the detector in the loop.Source: Appendix A.1 · p. 13

Appendix A.2. Tracker inference postprocessing

  • Every reported score is computed after dataset-specific trajectory interpolation and removal of tracks shorter than the specified minimum length.Interpolation fills gaps up to 10 frames on DanceTrack or 20 on SportsMOT when the surrounding tracklet is at least 30 frames; tracks under 20 or 10 frames, respectively, are removed.Source: Appendix A.2 · p. 14

Appendix A.3. Search spaces

  • The search spaces contain 7 hyperparameters for SORT, 9 for MoveSORT-KF, 12 for ByteTrack, and 14 for SparseTrack, with a shared block and tracker-specific additions.Table A.6 gives inclusive ranges; † marks candidates generated at or above the sampled detection threshold, and ‡ marks a logarithmic scale.Source: Appendix A.3 · p. 14, Table A.6 · p. 14

Appendix A.4. Optimizer configurations

  • All optimizers use seed 42 and a cap of 100 full-fidelity evaluations; the MFGCS default setting is abbreviated g5r3-n6-s6.TPE uses a univariate kernel, 24 expected-improvement candidates per step, and γ = 0.20; the quantile is selected here rather than being Optuna’s default.Source: Appendix A.4 · p. 15, Table A.7 · p. 15

Appendix B. Additional results

  • The appendix’s additional-results section provides metric-panel results, optimized hyperparameter values, and information for reproducing them.The supplied passage identifies these materials but does not report further result values.Source: Appendix B · p. 15

Appendix B.1. Full metrics and optimized hyperparameters

The appendix supplements the headline results with metric-level comparisons, optimized configurations, and ablations of MFGCS. It also explains how grid and ternary coordinate searches differ in their evaluation points and contraction behavior.

Appendix B.1. Full metrics and optimized hyperparameters

  • On DanceTrack MoveSORT-KF, TPE leads MFGCS on HOTA, DetA, AssA, MOTA, and IDF1, while MFGCS makes 98 fewer identity switches.The comparison concerns each optimizer’s best-HOTA trial; the cited table caption identifies the validation metrics and says the better value is bolded.Source: Appendix B.1 · p. 15, Table B.8 · p. 16
  • The searches selected substantially different hyperparameter values between datasets for the same tracker.The stored configurations are the values associated with the HOTA scores reported in Table 4.Source: Appendix B.1 · pp. 15–16

Appendix C.1. MFGCS configuration

  • The MFGCS sweep’s best grid configuration scores 53.69%, versus 53.19% for the best ternary variant and 52.90% for the best random variant.The best random variant uses 20 candidates per coordinate and 1553 scenes; a coarse grid can leave the search at its starting point, while three contraction rounds outperform two at g = 5.Source: Appendix C.1 · p. 16

Appendix C.2. Ternary section and grid search comparison

  • Ternary section and grid search are distinct one-dimensional contractive methods; no grid resolution reproduces ternary section’s evaluation set.Ternary evaluates two interior points, whereas grid search includes evenly spaced points and both endpoints.Source: Appendix C.2 · p. 17, Appendix C.2.1 · p. 17

Appendix C.2.1. Evaluation points

  • Grid search with g points evaluates g evenly spaced locations, including both endpoints; ternary section evaluates two interior points and never the endpoints.At g = 4, the grid includes both ternary interior points plus the endpoints.Source: Appendix C.2.1 · p. 17

Appendix C.2.2. Contraction rule

  • Ternary section always contracts the window to 2/3 its width, while grid search recenters on the best point and can contract faster when the optimum is at an endpoint.For an interior optimum, grid search at g = 4 contracts at the same rate as ternary section.Source: Appendix C.2.2 · p. 17

Appendix C.2.3. Evaluation cost

Ternary section and grid search trade evaluation cost differently depending on where the optimum lies and on the objective’s shape.

Appendix C.2.3. Evaluation cost

  • For an interior optimum under a strictly unimodal objective, ternary section is roughly twice as cheap as g = 4 grid search for the same contraction rate; grid reaches parity or wins otherwise.Halving the window takes about 3.4 ternary evaluations and 6.8 grid evaluations for an interior optimum, but grid needs only 2.5 at an endpoint; on integer coordinates, ternary points can round to the same lattice point, while unordered categoricals reduce both methods to enumeration.Source: Appendix C.2.3 · p. 18

Appendix C.3. Baseline optimizer configurations

This appendix compares optimizer configurations and Hyperband pruning, reporting the selected TPE setup, Gaussian-process results, and pruning’s budget–accuracy trade-off.

TPE

  • Plain TPE was carried forward despite the warm-started variant’s highest family score, because warm-starting depends on a manually tuned configuration.Other tested variants landed within a point of plain TPE; a split quantile of γ = 0.30 lost 1.5 points.Source: Appendix C.3 · pp. 18–19, Table C.11 · p. 20

Gaussian process

  • No Gaussian-process setting clearly beat uniform random search: two variants fell below its 52.28%, while the third reached 52.50%, an unresolved margin.The comparison does not establish that the 52.50% result is better than random search.Source: Appendix C.3 · p. 19, Table C.12 · p. 20

Hyperband pruning

  • 78% and 76% fewer trials reduced evaluation costs from 2500 scenes to 1534 and 1530, but equal-budget accuracy fell by 0.82 and 1.56 HOTA points.The accuracy comparison matches the unpruned samplers at the pruned runs’ scene budgets.Source: Appendix C.3 · pp. 19–20, Table C.13 · p. 21
  • MFGCS compares candidate values for one coordinate on a shared scene sample with other coordinates fixed, unlike Hyperband’s independently sampled whole-configuration trials.The authors present this as an explanation, not a measurement of the mechanism, and note that shared subsets could also pair Hyperband comparisons.Source: Appendix C.3 · p. 21
Loading 2609.12261v1…