PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Unified Token Learning for Vision-Language Tracking

Yaozong Zheng, Bineng Zhong, Qihua Liang, Guorong Li, Rongrong Ji, Xianxian Li

arXiv:2308.14103v1cs.CV

TL;DR

VL tracking has relied on specialized modules and multiple task-specific losses, which can be difficult to design and tune. MMTrack represents language and bounding boxes as tokens and predicts target coordinates autoregressively. The authors report that it outperforms previous methods on four VL tracking benchmarks.

  • Problem

    Prior VL trackers use specialized modules and multiple task-specific losses, whose design and weight tuning are described as difficult and potentially limiting generalization.

    Source: §I · p. 1

  • Method

    MMTrack turns VL tracking into autoregressive generation of bounding-box tokens conditioned on language and visual representations.

    Source: §I · p. 2, §III · p. 3

  • Results

    MMTrack outperforms previous methods on four VL tracking benchmarks.

    Source: §I · p. 2, §V · p. 10

  • Takeaways & Limitations

    The authors present MMTrack as a simpler unified modeling approach and a potential new baseline for VL tracking.

    Source: §I · p. 2, §V · p. 10

    The reported qualitative visualization covers video sequences from TNL2K, rather than visual examples across all four benchmarks.

    Source: D. Analysis and Limitations · p. 10

Abstract

from arXiv · show

In this paper, we present a simple, flexible and effective vision-language (VL) tracking pipeline, termed \textbf{MMTrack}, which casts VL tracking as a token generation task. Traditional paradigms address VL tracking task indirectly with sophisticated prior designs, making them over-specialize on the features of specific architectures or mechanisms. In contrast, our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target in an auto-regressive manner. The design without other prior modules avoids multiple sub-tasks learning and hand-designed loss functions, significantly reducing the complexity of VL tracking modeling and allowing our tracker to use a simple cross-entropy loss as unified optimization objective for VL tracking task. Extensive experiments on TNL2K, LaSOT, LaSOT$_{\rm{ext}}$ and OTB99-Lang benchmarks show that our approach achieves promising results, compared to other state-of-the-arts.

I. INTRODUCTION

MMTrack recasts vision-language tracking as autoregressive generation of bounding-box tokens conditioned on language and visual cues. It avoids hand-designed prior modules and multiple task-specific objectives, and reports state-of-the-art results on four VL benchmarks.

I. INTRODUCTION

  • Earlier VL tracking approaches commonly cast tracking as grounding, retrieval, or matching and build task-specific modules around those formulations.The described designs include proposal and ROI-based mechanisms for cross-modal fusion and alignment.Source: §I · p. 1
  • Prior designs can rely on architecture-specific components, including bounding-box proposal mechanisms, whereas MMTrack learns unified VL features without those priors.The paper identifies region proposal networks and ROI pooling as examples of such components.Source: §I · pp. 1–2
  • The authors describe multi-task VL tracking as difficult to optimize because several task-specific losses require careful weight tuning and may limit generalization.They contrast this with MMTrack’s single cross-entropy objective.Source: §I · pp. 1–2
  • MMTrack serializes language and bounding-box information as discrete tokens and predicts the target box autoregressively.Its conditional queries use language and bounding-box cues; the framework uses cross-entropy as a unified objective.Source: Figure 1 · p. 1, §I · p. 2
  • MMTrack achieves state-of-the-art tracking results on four VL benchmarks.The introduction presents it as a potential new baseline for VL tracking.Source: §I · p. 2

II. RELATED WORK

The related work contrasts vision-only trackers with multimodal tracking and positions MMTrack as a unified token-generation alternative to modular VL tracking pipelines.

A. Visual Tracking

  • Vision-only tracking methods may perform well on visual benchmarks but are described as inflexible for video and language inputs.The paper motivates a unified pipeline that uses a transformer to model both modalities as token sequences.Source: A. Visual Tracking · p. 2

B. Vision-Language Tracking

  • VL tracking locates a target using a language description and an initial video frame, while prior methods use grounding, retrieval, or matching pipelines.The cited examples include separate sub-tasks, modules, and loss functions.Source: B. Vision-Language Tracking · p. 2
  • MMTrack treats VL tracking as token generation to learn unified multimodal representations instead of relying on independent modules and multiple sub-tasks.The paper presents it as a new baseline approach.Source: B. Vision-Language Tracking · p. 2

C. Unified Modeling of Language and Vision

The paper situates MMTrack within unified vision-language modeling, distinguishes its multimodal tracking design from sequence-based work, and describes an encoder-decoder pipeline for generating target sequences.

C. Unified Modeling of Language and Vision

D. Discussion on the difference with sequence-based work

  • Unlike Pix2seq’s detection-focused sequence construction, MMTrack targets multimodal tracking and uses bounding-box and text information without categories or other noise strategies.The paper says this shorter construction reduces training load and is better adapted to VL tracking.Source: D. Discussion on the difference with… · pp. 2–3

III. APPROACH

  • MMTrack encodes image and language features, fuses them into a unified VL representation, and feeds that representation with language-and-box conditional queries to an autoregressive decoder.A task-agnostic sequence head directly predicts the final tracking results.Source: Figure 2 · p. 3, §III · p. 3

D. Discussion on the difference with sequence-based work

  • Unlike vision-only SeqTrack, MMTrack takes language and a template-search pair as input, and incorporates language tokens with fewer quantization bins.The paper links language tokens to more robust localization in complex scenarios.Source: D. Discussion on the difference with… · p. 3

A. Encoders Across Modalities

MMTrack encodes visual and linguistic inputs separately, then uses a shared multimodal encoder to fuse them into a unified VL representation.

A. Encoders Across Modalities

  • A vanilla ViT encodes the template and search frames as visual token sequences for feature extraction and relational modeling.The frames are projected, flattened into token embeddings, concatenated, and passed through transformer layers.Source: A. Encoders Across Modalities · pp. 3–4
  • RoBERTa encodes the language description as text-token embeddings used for multimodal fusion and language queries.The language encoder is pretrained on large corpora.Source: A. Encoders Across Modalities · p. 4
  • A shared multimodal encoder receives concatenated visual and linguistic features after bottleneck layers reduce their channel dimensions.The paper describes residual multiplication in the fusion formulation as beneficial for capturing semantic information and says the unified modeling supports deeper cross-modal understanding.Source: A. Encoders Across Modalities · p. 4

B. Multi-modal Decoder

MMTrack uses an autoregressive transformer decoder to turn multimodal query inputs into target-sequence representations. Causal self-attention and multimodal cross-attention support this sequence generation.

B. Multi-modal Decoder

  • An autoregressive transformer decoder converts multimodal inputs into target-sequence representations for VL tracking.The encoder updates VL features while the decoder predicts discrete coordinate tokens.Source: B. Multi-modal Decoder · p. 4
  • Masked self-attention makes each output depend only on preceding query tokens, while cross-attention combines queries with VL representations.The decoder receives linguistic and visual token sequences as query inputs and iteratively generates target sequences.Source: B. Multi-modal Decoder · p. 4, Figure 3 · p. 4
  • At decoding step j, D uses queries Q0:j−1 to produce Tseq across target length L.D denotes the multimodal decoding function; Q0:j−1 are the preceding input queries, and Tseq is the decoded target-sequence representation.Source: B. Multi-modal Decoder · p. 4

C. Conditional Queries

MMTrack constructs conditional queries from language and bounding-box cues, which perceive the target and predict its spatial coordinates autoregressively. Concatenating the queries combines appearance and semantic information.

C. Conditional Queries

  • MMTrack forms conditional queries from a language query Qtext and a box-coordinate vision query Qvis, which directly predict target coordinates autoregressively.Both query types are required to perceive the desired target from their respective modalities.Source: C. Conditional Queries · p. 4
  • Pooling word embeddings produces a sentence-level target feature, then an embedding layer converts it into a 1-D language-token sequence.The sentence-level feature comes from the linguistic encoder.Source: C. Conditional Queries · p. 5
  • Bounding-box coordinates are quantized using search-area coordinates and K bins, then serialized through an embedding layer as visual tokens.The coordinates are expressed at the search-area level, with s denoting search-area size and K the number of quantization bins.Source: C. Conditional Queries · p. 5
  • Concatenating Qtext and Qvis supplies appearance and high-level semantic cues for the same target.The paper describes these multimodal conditional queries as more flexible for multimodal scenarios and more robust for tracking.Source: C. Conditional Queries · p. 5

D. Task-agnostic Sequence Predictor

MMTrack replaces task-specific prediction heads and multitask learning with autoregressive coordinate-token prediction. It trains with cross-entropy and generates coordinates at inference without additional postprocessing.

D. Task-agnostic Sequence Predictor

  • Prior task-specific classification, regression, and grounding heads require separate objectives, while multitask loss weights add tuning complexity.The paper frames token generation as avoiding multitask misalignment, such as a highly classified box having lower regression accuracy, and reducing modeling complexity.Source: D. Task-agnostic Sequence Predictor · p. 5
  • The task-agnostic predictor outputs coordinate-token probabilities, whose top four score indices represent target localization in the current frame.Three linear layers on the multimodal decoder further learn coordinate tokens.Source: D. Task-agnostic Sequence Predictor · p. 5, Figure 4 · p. 5

E. Training and Inference

  • MMTrack uses only cross-entropy loss to train token generation, avoiding the need to tune weights for specific loss functions.Source: E. Training and Inference · p. 5
  • At inference, the language query starts autoregressive prediction and an EOS token ends it; the resulting coordinates need no additional postprocessing.The coordinates are transformed to obtain the final result, and inference uses only a forward network.Source: E. Training and Inference · p. 5

IV. EXPERIMENTS

Experiments compare MMTrack with existing trackers on four vision-language tracking benchmarks; the reported results are strongest on TNL2K and mixed across the other benchmarks.

B. Comparison with State-of-the-art

A. Implementation Details

B. Comparison with State-of-the-art

  • Across LaSOT attributes, MMTrack achieves good precision results against vision-only trackers and VLTTT across conditions including occlusion, motion, and scale change.Fig. 5 presents precision plots for the listed attributes.Source: B. Comparison with State-of-the-art · pp. 6–7, Figure 5 · p. 7
  • On LaSOText, MMTrack scores 49.4% success, 59.9% normalized precision, and 55.3% precision; versus VLTTT, it gains 1% in success and precision, with equal normalized accuracy.The paper describes its LaSOText results as comparable.Source: B. Comparison with State-of-the-art · p. 7
  • On LaSOT and LaSOText, performance remains slightly below OSTrack-384; the authors suggest initial-frame language annotations may not adapt to prolonged video changes.The authors frame this explanation as a possibility.Source: B. Comparison with State-of-the-art · p. 7
  • Fig. 6 compares MMTrack with two VL trackers on three challenging TNL2K sequences, while Fig. 7 depicts poor tracking amid similar distractors and occlusion exceeding 280 frames.The Fig. 7 caption notes these conditions also challenge VLTTT and TNL2K-II.Source: Figure 6 · p. 8, Figure 7 · p. 8
  • On OTB99-Lang, MMTrack scores 70.5% success and 91.8% precision, ranking second to VLTTT.The authors suggest relatively rough OTB annotations may cause uneven performance.Source: B. Comparison with State-of-the-art · p. 8

C. Ablation Study

The ablations examine quantization-bin count, bounding-box format, query cues, and language-annotated training data. They report gains from finer quantization, corner-point boxes, multi-cue queries, and broader language-annotated training.

Study on the number of quantization bins

  • From 50 to 1,000 quantization bins, AUC, PNorm, and P improve by 6.9%, 3.9%, and 16.2%, respectively.Performance stabilizes when the bin count becomes sufficiently large.Source: C. Ablation Study · p. 8, Table II · p. 9

Study on the format of bounding box

  • Corner-point boxes (#4) outperform center-point-plus-size boxes (#6) by 2.5% AUC, 0.7% PNorm, and 1.4% P.The authors suggest center-point-plus-size prediction may be harder because it requires learning both center representation and target scale.Source: C. Ablation Study · p. 9, Table II · p. 9

Study on query construction based multi-cues or single-cue

  • Multi-cue queries using language and bounding-box information (#4) outperform box-only queries (#7) by 0.7% AUC, 1.6% PNorm, and 1.9% P.The authors attribute a benefit to enhanced target semantic perception.Source: C. Ablation Study · p. 9, Table II · p. 9

Study on the importance of language-annotated data

  • Training on LaSOT, RefCOCOg-google, TNL2K, and OTB99-Lang performs better than training only on TNL2K, which degrades on other datasets.The authors conclude that language-annotated data is crucial for VL tracking.Source: C. Ablation Study · p. 9, Table III · p. 9

D. Analysis and Limitations

MMTrack outperforms VLTTT and JointNLT in inference speed and model performance, despite having about 176.9M parameters. Visualizations report better results than VLT and TNL2K-II in selected challenging sequences, while similar distractors and long occlusions remain challenging.

Speed Analysis

  • MMTrack outperforms VLTTT and JointNLT in both inference speed and model performance.The tracker has approximately 176.9M parameters; the linguistic encoder runs once per video and has little effect on inference speed.Source: D. Analysis and Limitations · pp. 9–10

Visualization

  • On sequences with target changes, background clutter, and scale variation, MMTrack is reported to outperform VLT and TNL2K-II.The authors attribute this to high-level semantics enhancing the unified VL representation.Source: D. Analysis and Limitations · p. 10

Limitations

  • Similar distractors and long-term occlusion remain challenging; one example has more than 280 frames of occlusion and interference from similar actors.The authors suggest multi-frame target association as a potential solution.Source: D. Analysis and Limitations · p. 10

V. CONCLUSION

The conclusion presents MMTrack as a unified token-generation approach to VL tracking that simplifies modeling. Experiments report that it outperforms previous methods on four VL tracking benchmarks.

V. CONCLUSION

  • MMTrack reformulates VL tracking as token generation, avoids sensitive loss functions and sub-task learning, and outperforms previous methods on four VL benchmarks.The authors hope the work encourages further research on VL tracking.Source: §V · p. 10
Loading 2308.14103v1…