Source-linked AI summary
Cross-Domain Tracker Adaptation Without Target-Domain Labels via Vision-Language Agents
Daniel Davila, Ravikumar Balakrishnan, Mike Cochran
TL;DR
The paper addresses brittle tracker hyperparameter transfer across visual domains when target-domain labels are unavailable. It uses a VLM to inspect rendered tracking diagnostics and iteratively adjust exposed parameters. On MOT17→MOT20, the method recovers 67.8% of oracle headroom, while its effectiveness depends on visible, parameter-addressable failure modes and sufficient source mistuning.
Problem
Cross-domain tracker adaptation lacks practical methods for adjusting hyperparameters without target-domain ground truth or human feedback.
Method
A VLM diagnoses rendered tracking outputs and recommends sequential detection-then-tracking parameter updates through window-level analysis and sequence-level aggregation.
Results
67.8% of per-sequence oracle headroom is recovered on MOT17→MOT20, reaching within 0.029 HOTA of the target-domain oracle and up to 86.7% recovery on the highest-density sequence.
Takeaways & Limitations
VLM tuning improves poorly transferring configurations while avoiding significant perturbation when source configurations already transfer well.
Takeaways & Limitations
The approach is less effective when the source configuration is already near-optimal or when domain shifts do not produce visible failure modes mapped to exposed parameters.
Abstract
from arXiv · showhide
We present a system that uses a Vision-Language Model (VLM) as a diagnostic agent for adapting a detect-to-track pipeline to a new target domain without access to target-domain labels. Rather than optimizing against annotated metrics, the VLM directly inspects rendered tracking outputs, identifies visual failure modes, and recommends parameter updates through an iterative tuning loop. We first demonstrate that ground-truth-supervised hyperparameter transfer can be brittle. On MOT17->MOT20, applying a source-derived oracle configuration reduces mean HOTA by 0.090, from a target-domain ceiling of 0.357, to 0.267. Without using any target-domain labels, our VLM-based tuner recovers 67.8% of this lost headroom, finishing within 0.029 HOTA of the target ceiling; on the highest-density target sequence, it recovers up to 86.7%. We further show that label-free Bayesian optimization with handcrafted proxy objectives struggles under large domain shifts and can degrade configurations that are already strong. In contrast, the VLM tuner acts selectively: when its visual diagnosis reveals no clear failure mode, it declines to modify the configuration, preserving performance on easy transfers while improving hard ones. Finally, we characterize the conditions under which this approach succeeds, namely, when domain shift manifests through exposed detection-level parameters, versus where it is less effective, such as MOT17->DanceTrack, where the source oracle is already near-optimal.
1 Introduction
Cross-domain shifts make tracker configurations brittle, motivating label-free adaptation through VLM inspection of rendered outputs. The proposed tuner targets exposed detection and tracking parameters, recovering substantial oracle headroom without target labels.
- Domain shifts in scene density, viewpoint, and target dynamics degrade detect-to-track performance and make manual tuning time-consuming and error-prone.
- Oracle configurations transfer poorly across domains, with degradation especially pronounced on high-density or higher-motion target scenes.
- The VLM tuner inspects detection collages, dot panels, and identity strips to diagnose missed detections, false positives, identity switches, and fragmentation.
- 67.8% mean oracle-headroom recovery on MOT17→MOT20 leaves the VLM-tuned configuration within 0.029 HOTA of the target-domain oracle.
- 86.7% of per-sequence oracle headroom is recovered on the highest-density target sequence.
- The paper releases the complete toolkit and a 40+ iteration prompt-engineering experimentation log.
2 Related Work
Prior tracking adaptation methods commonly modify model weights or rely on supervised scalar feedback. This work instead adapts runtime operating parameters using structured visual diagnostics and VLM-driven interventions.
- 2.1 Multi-Object Tracking Pipelines: Tracking-by-detection pipelines separate per-frame detection from association and expose thresholds, NMS settings, IoU gates, and lost-track survival parameters.
- 2.2 Domain Adaptation for Tracking: Existing domain-adaptation methods generally modify detector, appearance, architecture, or tracker weights to bridge the domain gap.
- 2.2 Domain Adaptation for Tracking: The proposed regime keeps detector and tracker weights fixed while adjusting runtime thresholds and association gates for settings where retraining is impractical.
- 2.3 Hyperparameter Optimization and LLM Agents: Classical Bayesian optimization requires a labeled validation reward, while language-model agents improve search without removing that labeled-data requirement.
- 2.3 Hyperparameter Optimization and LLM Agents: The VLM replaces supervised feedback with detection collages, dot panels, and NMS-orphan overlays that expose target-domain visual failure modes.
- 2.4 VLMs as Diagnostic Agents: Unlike open-loop VLM evaluators, this system prescribes parameter interventions that are applied to the tracker in the next iteration.
3 Method
The method adapts a fixed detect-to-track pipeline to an unlabeled target domain through sequential detection and tracking phases driven by VLM diagnoses of rendered outputs. Window-level agents recommend parameter changes, while a meta-agent aggregates them into a final configuration.
- System Overview: The system initializes from a source-domain oracle, then performs unlabeled target adaptation in sequential detection and tracking phases.The source oracle is obtained by grid search over exposed parameters; after initialization, no source or target labels are used.
- Adaptation Loop: Each phase analyzes temporal windows independently and uses a meta-agent to aggregate window-level failure diagnoses and parameter recommendations.The phases terminate when windows report clean or no_lever, or when the phase iteration budget is exhausted.
- Pipeline: The tracker uses YOLOv11-Large detection followed by ByteTrack association, with exposed runtime parameters divided between the two adaptation phases.The pipeline is not architecturally modified; adaptation operates through its tunable operating parameters.
- Detection Phase: Detection diagnostics combine dot panels, thresholded detection collages, and NMS-orphan strips to distinguish recall, precision, threshold, and suppression failures.The agent uses scalar calibration metrics only to scale parameter-change magnitude; visual panels determine direction.
- Tracking Phase: The tracking phase targets continuity and identity failures using scene/dot panels, death-to-birth track-break views, and closest-approach strips for crossover analysis.Its exposed parameters include match_thresh, spawn_thresh, max_lost_age, velocity_decay, min_hits, and appearance_thresh.
4 Experiments
Experiments evaluate oracle transfer, proxy-based Bayesian optimization, and VLM adaptation across the 3 × 3 MOT17, MOT20, and DanceTrack source/target matrix. The comparison uses target-oracle HOTA as the ceiling and tests whether label-free methods recover transfer losses.
- Experimental Setup: The evaluation covers 9 source/target pairs across MOT17, MOT20, and DanceTrack, including same-domain self-transfer.Each pair reports oracle transfer, VLM adaptation from the source oracle, and the target per-sequence oracle ceiling.
- Bayesian Optimization: The label-free baseline optimizes a tracker-output proxy for 50 trials per sequence and reports HOTA for the proxy-maximizing configuration.The proxy combines confirmed-track count, mean track lifetime, and fragmentation ratio.
- Experimental Setup: Table 2 compares target-domain oracle HOTA, transferred-source-oracle HOTA, proxy Bayesian optimization, and VLM adaptation from the transferred oracle.The target oracle requires target-domain ground truth, whereas oracle transfer and VLM adaptation use source-domain supervision only for initialization.
5 Results
Cross-domain oracle transfer is brittle, especially under large density or motion shifts, while VLM-guided adaptation recovers much of the lost performance without target labels. The gains are concentrated in visible detection failures, whereas adaptation can regress when the source configuration is already near-optimal.
- 5.1 Brittleness of Oracle Transfer: 0.090 HOTA separates MOT17→MOT20 oracle transfer from the 0.357 target ceiling, with transferred performance at 0.267.The shortfall grows with domain-shift magnitude and is especially severe for denser crowds and faster motion.
- 5.1 Brittleness of Oracle Transfer: More than 0.15 HOTA is lost on dense MOT20-03 and MOT20-05, while moderate-density MOT20-01 and MOT20-02 lose under 0.032.Detection-stage under-coverage explains the concentration of degradation: MOT17 oracle conf_thresh values of 0.20–0.30 reject valid detections in dense MOT20 crowds.
- 5.2 VLM-Guided Recovery: 67.8% of the MOT17→MOT20 shortfall is recovered, lifting mean HOTA from 0.267 to 0.328 and ending within 0.029 of the 0.357 ceiling.Recovery reaches 86.7% on MOT20-03 and 65.4% on MOT20-05, while near-ceiling moderate-density sequences change little.
- 5.2 VLM-Guided Recovery: High-shift transfers close most of the gap, but low-shift transfers change little and MOT17→DanceTrack regresses by −0.015.The VLM is selective when diagnostics show no clear failure mode, unlike the BO-proxy baseline, which can move strong configurations away from good operating points.
- 5.3 Agent Ablation: Where Do the Gains Come From?: +0.042 HOTA comes from the detection agent alone versus +0.010 from the tracking agent on MOT17→MOT20, making detection-phase intervention the dominant contribution.The tracking phase preserves the detection gains, although detection slightly over-tunes moderate-density MOT20-01 and MOT20-02.
- 5.3 Agent Ablation: Where Do the Gains Come From?: Visual-only reaches 0.330 versus 0.328 for the full system, while text-only reaches only 0.281 because scalar metrics miss dense-crowd detection failures.On MOT17→DanceTrack, both isolated agents regress by −0.046 HOTA, but the full loop limits regression to −0.015 through inter-phase correction.
- 5.4 Failure Case: MOT17→DanceTrack: VLM tuning succeeds when domain shifts expose visible failures tied to adjustable parameters and the source configuration is mistuned beyond the intervention noise floor.MOT17→DanceTrack satisfies neither condition: the source oracle is already near-optimal, so local residuals perturb it rather than improve it.
6 Conclusion
The system uses a Vision-Language Model as a diagnostic agent to adapt a fixed detect-to-track pipeline across domains without target-domain labels. It improves difficult transfers selectively, but remains limited to exposed tracker parameters and can perturb already-good configurations.
- The system uses a Vision-Language Model as a diagnostic agent over rendered tracker output, calibration metrics, and a structured decision procedure.It tunes the exposed parameter space of a fixed detect-to-track pipeline without target-domain labels.
- 67.8% mean oracle-headroom recovery on MOT17→MOT20 and up to 86.7% on the highest-density target sequence demonstrate substantial label-free adaptation gains.The method recovers performance from source-oracle initializations without using target-domain labels.
- The method improves transfers that start poorly while avoiding significant perturbations when source configurations already transfer well.By contrast, label-free Bayesian optimization can degrade near-optimal configurations, including 0.440 →0.285 on MOT17→MOT17.
- Adaptation cannot address domain shifts requiring detector weights, association features, or appearance-model changes, because it operates only within the exposed parameter space.The system also requires source-domain labels to compute its oracle initialization.
- The study leaves open broader exploration of tracker tools, tuning variables, performance visualizations, and other perception tasks with similar open-loop failures.These directions concern both what the VLM can tune and how tracker performance is presented to it.