PaperlayerView exact revision on arXiv

Source-linked AI summary

YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Verification

Quansheng Hu, Qin Sun, Qiansen Dai, Jin Ding, Wan Zhang, Xue Zhou, Jianxiao Zou

arXiv:2609.02318v1cs.CV

TL;DR

RMOT must align language-described objects across video frames, while prior MLLM-based approaches commonly generate captions and then rely on external matching. YesTrack replaces that process with two-stage Yes/No verification, adds temporal consistency constraints, and reports superior performance with high efficiency across Refer-KITTI benchmarks and generic MOT.

  • Problem

    Existing MLLM-based RMOT methods primarily generate captions and require external matching, introducing latency and underusing direct vision-language alignment.

  • Method

    YesTrack reformulates referring as binary image-text matching in a two-stage MLLM verifier, augmented with Temporal Confidence Prior and Temporal Reference Propagation, and extends the paradigm to MOT.

  • Results

    Experiments on Refer-KITTI and Refer-KITTI-V2 report superior performance with high efficiency, including the smallest Qwen3-VL variant, while YesTrack-MOT generalizes the paradigm to generic MOT.

  • Takeaways & Limitations

    Direct MLLM-driven binary decisions provide a versatile tracking foundation spanning referring and generic multi-object tracking.

  • Takeaways & Limitations

    YesTrack remains constrained by its two-stage design and the quality of the off-the-shelf tracker; key-frame errors in TRP can propagate until re-evaluation.

Abstract

from arXiv · show

Referring multi-object tracking (RMOT) aims to track every instance in a video that matches a given language expression. Despite the recent integration of multimodal large language models (MLLMs) to enhance generalization, existing methods predominantly relegate them to the role of caption generators, necessitating external modules for final decision-making. This paradigm not only introduces extra latency but also severely underutilizes the inherent vision-language alignment capabilities of MLLMs. To address these limitations, we propose YesTrack, a novel two-stage RMOT method that reformulates referring as a discriminative task, directly leveraging MLLMs for Yes/No verification without explicit text generation. To further enhance the reliability and efficiency of this MLLM-based verification, we introduce two lightweight temporal consistency constraints: Temporal Confidence Prior (TCP) and Temporal Reference Propagation (TRP). We further validate the generality of this discriminative paradigm by proposing YesTrack-MOT, a straightforward yet highly effective instantiation for generic multi-object tracking (MOT). Experiments on Refer-KITTI and Refer-KITTI-V2 show that YesTrack significantly outperforms existing state-of-the-art methods while maintaining high efficiency, even when implemented with the smallest variant of Qwen3-VL. Code is released at https://github.com/ggbondrighthere24/YesTrack.

1 Introduction

RMOT requires aligning language-described objects across video frames, but existing MLLM-based approaches rely on caption generation and external matching. YesTrack instead uses direct Yes/No verification with temporal constraints, achieving strong results and efficiency across benchmarks.

  • Task and motivation: RMOT continuously localizes and associates objects matching a natural-language referring expression across video frames.The task differs from generic MOT by selecting only instances specified by language.
  • Task and motivation: Existing approaches often use MLLMs for caption generation followed by additional text matching, adding sequential decoding and alignment latency.This setup also underuses the MLLMs’ direct vision-language alignment capabilities.
  • YesTrack: YesTrack reformulates referring as binary image-text matching and directly determines whether each candidate trajectory matches the expression.The design avoids autoregressive decoding and requires a single forward pass per candidate.
  • Temporal refinement: TCP stabilizes referring decisions using identity-level temporal consistency, while TRP reduces verification by propagating decisions across sparsely evaluated frames.Together, the constraints target robustness and computational efficiency in two-stage RMOT.
  • Results and generalization: YesTrack generalizes the discriminative paradigm to generic MOT through YesTrack-MOT and reports state-of-the-art results on Refer-KITTI and Refer-KITTI-V2, including Qwen3-VL-2B-Instruct.The smallest Qwen3-VL family variant is explicitly reported as an efficient instantiation.

2 Related Work

RMOT research includes end-to-end and two-stage paradigms, while MLLM-based systems commonly generate descriptions before matching them. This generative strategy enriches semantic alignment but incurs latency that motivates discriminative Yes/No formulations.

  • RMOT paradigms: RMOT methods broadly follow end-to-end or two-stage pipelines that combine tracking with language-based trajectory selection.The two-stage pattern first performs generic tracking and then refers to target trajectories.
  • RMOT paradigms: End-to-end methods use task-specific expression decomposition, linguistic parsing, multimodal attention, or temporal modeling.Examples include temporal enhancement and decomposition into static and motion cues.
  • MLLM-based approaches: MLLM-based referring systems commonly generate textual descriptions and then align them with textual queries for matching.ReferGPT is cited as an RMOT example using sequential caption generation and query matching.
  • MLLM-based approaches: Autoregressive generation of many tokens creates a latency bottleneck for real-time tracking, motivating binary Yes/No decisions as a faster alternative.Discriminative formulations reduce multimodal reasoning to explicit binary judgments.

3 Method

YesTrack combines an off-the-shelf tracker with a two-stage MLLM verifier for RMOT. Temporal propagation and confidence regularization reduce redundant verification while routing uncertain candidates to video-level refinement.

  • Framework overview: YesTrack is a two-stage RMOT framework combining an off-the-shelf tracker with an MLLM-based discriminative referring module.The same paradigm is also applied to MOT data association in YesTrack-MOT.
  • Framework overview: The tracker produces candidate crops, track identities, and normalized bounding boxes for each frame.These candidate regions are passed to the verification pipeline.
  • Temporal refinement: TRP divides frames into key and non-key frames, performing full MLLM verification at key frames and propagating prior scores to active identities otherwise.This avoids unnecessary MLLM invocations across consecutive frames.
  • Binary verification: The verifier receives a cropped image or video clip with the referring expression and outputs a binary match decision.Frame and video modes retain the same binary decision format while differing in image count and prompt formulation.
  • Verification stages: Stage 1 performs single-frame candidate scoring with TCP regularization, while Stage 2 uses video-level evidence and a memory bank to refine uncertain cases.The two stages support early routing followed by more detailed temporal verification.

Binary probability extraction.

YesTrack extracts the MLLM logits for Yes and No and converts them into a continuous matching probability. This avoids free-form decoding while preserving confidence information for routing and temporal refinement.

  • Logit extraction: After jointly encoding visual and textual inputs, the MLLM produces vocabulary-level token logits used for decision scoring.The method operates on an input pair consisting of a candidate image crop and referring expression.
  • Probability computation: YesTrack computes the Yes matching probability by applying softmax only to the Yes and No decision-token logits.This directly converts the two logits into a binary probability rather than decoding text autoregressively.
  • Benefits: The logit-based score avoids instability from unexpected explanations or wording variations in free-form outputs.Its continuous confidence value retains uncertainty information needed for confidence-based routing and temporal refinement.

Training.

YesTrack trains binary image–text matching with BCE and adds inference-time temporal constraints to improve robustness and efficiency. TRP reduces redundant MLLM verification by propagating referring scores between key frames.

  • Training.: Each image–expression pair receives a binary match label, and the two decision logits produce a matching probability for BCE training.The positive label denotes a match; the negative label denotes no match.
  • Training.: TRP and TCP are inference-time constraints designed to improve robustness and efficiency without changing the core model architecture.They are intended for two-stage RMOT and can extend to other two-stage RMOT frameworks.
  • Training.: TRP exploits temporal stability in referring decisions to reduce redundant MLLM invocations without sacrificing referring accuracy.Tracking states update frame by frame, while an identity’s semantic relevance usually remains stable over a short window.
  • Training.: At key frames, tracking and referring run together; on non-key frames, active identities inherit the latest referring score until the next trigger.Referring is triggered periodically, when a new identity appears, or when the expression changes.

Temporal Confidence Prior (TCP).

TCP uses recent identity-level matching confidence to bias current referring predictions toward temporally consistent decisions. The prior is applied only after confident matching across the preceding temporal window and is clipped to preserve a valid probability.

  • Temporal Confidence Prior (TCP).: TCP is motivated by the assumption that an identity confidently matched in recent frames is likely to remain relevant in the current frame.This identity-level temporal consistency is intended to stabilize referring decisions under occlusion.
  • Temporal Confidence Prior (TCP).: TCP increases an identity’s current matching probability when its predictions remain above a confidence threshold throughout the previous K frames.The indicator activates the prior only under strict temporal consistency.
  • Temporal Confidence Prior (TCP).: The TCP adjustment uses K as the temporal window, α as the confidence threshold, λ > 0 as prior strength, and an indicator for whether the condition holds.The outer min operator clips the adjusted score to 1.

3.4 YesTrack-MOT

YesTrack-MOT extends YesTrack’s discriminative MLLM paradigm from referring to generic MOT data association. It uses pairwise identity verification with lightweight gating and Hungarian assignment in a deliberately minimal pipeline.

  • 3.4 YesTrack-MOT: The resulting minimal MOT pipeline combines distance gating, MLLM-based verification, and Hungarian assignment while retaining unmatched tracks for possible re-association.This design intentionally avoids sophisticated motion models.
  • 3.4 YesTrack-MOT: YesTrack-MOT replaces conventional embedding-based ReID with an MLLM-based pairwise identity verifier for data association.The verifier is a straightforward instantiation of the same discriminative paradigm used for referring.
  • 3.4 YesTrack-MOT: The verifier compares a tracked crop with a detection crop using image–image inputs and outputs a continuous confidence score.Referring instead uses image–text verification.
  • 3.4 YesTrack-MOT: Distance-based gating keeps only plausible track–detection pairs whose center displacement is within a threshold.The bounding-box center and gating threshold determine the retained pairs.

MLLM-based identity verification.

YesTrack-MOT computes identity-verification probabilities from logits for positive and negative outcomes. These probabilities support discriminative pairwise association between tracked and detected image regions.

  • MLLM-based identity verification.: The verification probability p_jk is computed by applying softmax to the positive and negative verification logits for each input pair.The input pair consists of the tracked crop and detection crop.

Hungarian matching with simple track management.

YesTrack-MOT forms verification-based costs, performs one-to-one Hungarian matching, and manages unmatched tracks through temporary lost-track retention and re-association. The pipeline deliberately uses only distance gating, MLLM verification, and Hungarian assignment while achieving strong practical performance.

  • Matching: Verification probabilities form a cost matrix C = [cjk] for one-to-one assignment with the Hungarian algorithm.The matching cost is defined from verification probabilities, as indicated by c_jk = 1 - p_jk.
  • Matching: The optimal feasible matching minimizes the summed costs over track-detection pairs in the gated candidate set.Matched pairs update their tracks, while unmatched tracks are marked as lost.
  • Track management: Lost tracks remain available for up to L subsequent frames, allowing unmatched detections one additional re-association attempt.Tracks are removed after exceeding the maximum lost age.
  • Design choice: YesTrack-MOT uses a minimal pipeline combining distance gating, MLLM-based verification, and Hungarian assignment without sophisticated motion models.The passage reports strong performance in practice for this deliberately simple design.

4 Experiment

Experiments evaluate YesTrack on Refer-KITTI and Refer-KITTI-V2 using benchmark comparisons, ablations, tracker-agnostic referential metrics, and qualitative analysis. YesTrack achieves strong tracking and referring results, while TCP improves stability and TRP improves efficiency, subject to two-stage and tracker-quality limitations.

  • Datasets: Refer-KITTI-V2 expands evaluation from 818 expressions and 49 vocabulary words to 9,758 expressions, 7,193 distinct expressions, and 617 vocabulary words.The expanded benchmark also covers all 21 KITTI sequences, whereas Refer-KITTI uses 18 sequences.
  • Benchmark Results: YesTrack achieves HOTA 54.00 and AssA 66.57 on Refer-KITTI with TempRMOT∗, while YesTrack-MOT obtains DetA 46.84.These are reported as the best overall HOTA, best association accuracy, and best detection accuracy in the corresponding comparisons.
  • Benchmark Results: On Refer-KITTI-V2, YesTrack-MOT attains HOTA 43.75, DetA 37.04, DetRe 48.78, and AssPr 83.95.The results are reported as leading values on the more challenging benchmark, while the TempRMOT∗ variant also yields substantial improvements.
  • Ablation Experiments: Combining frame-mode and video-mode inference balances detection and association quality, reflecting complementary roles in the ablation study.Video mode improves HOTA and association metrics over frame mode but increases inference time.
  • Ablation Experiments: TCP stabilizes trajectory selection and achieves the best overall HOTA and DetA, whereas TRP provides the fastest runtime with competitive performance.Combining TCP and TRP yields the best AssA with only a small runtime increase over TRP alone, exposing an accuracy–efficiency trade-off.
  • Tracker Generality: YesTrack consistently outperforms iKUN on Accuracy, Precision, Recall, and RMOT metrics across four underlying trackers, while YesTrack-MOT reaches HOTA 45.10 under standard MOT evaluation.The standard MOT comparison does not use referring expressions and uses standardized detection inputs for tracking-by-detection methods.

5 Conclusion

YesTrack reformulates MLLM inference as binary matching, eliminating text-generation latency. Temporal Confidence Prior and Temporal Reference Propagation improve robustness and efficiency in two-stage RMOT, while the discriminative paradigm generalizes to MOT.

  • YesTrack reformulates MLLM inference as binary matching, eliminating text-generation latency.
  • Temporal Confidence Prior and Temporal Reference Propagation improve robustness and efficiency in two-stage RMOT.
  • YesTrack’s discriminative MLLM paradigm generalizes to generic multi-object tracking through YesTrack-MOT.
  • Experiments show superior performance, demonstrating the effectiveness of discriminative MLLMs for multimodal video understanding.
Loading 2609.02318v1…