PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Visual Grounding and Tracking with Natural Language Specification

Li Zhou, Zikun Zhou, Kaige Mao, Zhenyu He

arXiv:2303.12027v1cs.CV

TL;DR

Natural-language tracking systems often separate grounding from tracking, despite language providing target semantics for both. This paper unifies the tasks in a joint model with multi-source relation modeling and semantic-guided temporal clues. It reports favorable performance against state-of-the-art algorithms on three natural-language tracking datasets and one visual-grounding dataset.

  • Problem

    Existing approaches separate visual grounding and tracking, overlooking their connection; tracking may also ignore language cues that describe target variation and semantics.

    Source: §1 · pp. 1–2

  • Method

    A joint framework models relations between language, template, and test image, while semantic-guided temporal modeling uses language to focus historical target information.

    Source: §1 · p. 2, §3.5 · p. 5

  • Results

    The method achieves favorable performance against state-of-the-art algorithms on three natural-language tracking datasets and one visual-grounding dataset.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The unified framework connects grounding and tracking, and its semantics-guided temporal module effectively improves tracking performance.

    Source: §5 · p. 8

    The method is sensitive to ambiguous natural-language descriptions; a clear description or an additional bounding box can resolve the ambiguity in the described example.

    Source: §4.6 · pp. 7–8

Abstract

from arXiv · show

Tracking by natural language specification aims to locate the referred target in a sequence based on the natural language description. Existing algorithms solve this issue in two steps, visual grounding and tracking, and accordingly deploy the separated grounding model and tracking model to implement these two steps, respectively. Such a separated framework overlooks the link between visual grounding and tracking, which is that the natural language descriptions provide global semantic cues for localizing the target for both two steps. Besides, the separated framework can hardly be trained end-to-end. To handle these issues, we propose a joint visual grounding and tracking framework, which reformulates grounding and tracking as a unified task: localizing the referred target based on the given visual-language references. Specifically, we propose a multi-source relation modeling module to effectively build the relation between the visual-language references and the test image. In addition, we design a temporal modeling module to provide a temporal clue with the guidance of the global semantic information for our model, which effectively improves the adaptability to the appearance variations of the target. Extensive experimental results on TNL2K, LaSOT, OTB99, and RefCOCOg demonstrate that our method performs favorably against state-of-the-art algorithms for both tracking and grounding. Code is available at https://github.com/lizhou-cs/JointNLT.

1. Introduction

Tracking by natural language specification uses a description to locate a target across video frames, but existing systems separate initial grounding from tracking. The paper proposes a joint model that uses language and visual references for both stages, alongside semantic-guided temporal modeling.

1. Introduction

  • Language can describe long-term target variation and provide semantics that bounding boxes lack, while boxes may leave the target ambiguous.The paper frames these properties as advantages of language-based specification for tracking.Source: §1 · p. 1
  • Existing systems separate first-frame visual grounding from later tracking, overlooking their connection and often preventing end-to-end training.The tracking stage may also rely only on the template, overlooking the natural-language description.Source: §1 · p. 2
  • The paper unifies grounding and tracking as localization from visual-language references, using a multi-source relation module that adapts to the references available for each task.Grounding uses language; tracking uses language and the historical target patch.Source: §1 · p. 2
  • Semantic-guided temporal modeling uses historical predictions and language guidance to provide temporal clues and improve adaptability to target appearance variations.Language guides the module toward the target region rather than noise in prior predictions.Source: §1 · p. 2

2. Related work

Prior visual trackers commonly initialize from a bounding box, while natural-language tracking methods use several approaches to ground and track described targets. The paper's related-work discussion contrasts these lines and their limitations.

2.1. Tracking with bounding box specification

  • Classical visual trackers initialize with a first-frame bounding box, and transformer-based trackers model template–search relations or spatial and temporal dependencies.MixFormer and OSTrack use one-stream pipelines, but manually specified initialization remains a barrier to direct real-world use.Source: §2.1 · p. 2

2.2. Tracking with natural language description

  • Natural-language tracking approaches include separable grounding, tracking-by-detection, and target-specific retrieval, while prior methods often leave grounding and tracking disconnected.The authors position their unified framework as performing grounding or tracking according to different inputs.Source: §2.2 · p. 3

3. Method

The method uses one end-to-end model for grounding and tracking, adapting its visual-language references to each stage. It combines multi-source relation modeling with a semantic-guided temporal module that supplies recent target information during tracking.

3. Method

  • The framework combines language and vision encoders, multi-source relation modeling, a target decoder, semantic-guided temporal modeling, and a localization head in one end-to-end model.The model predicts a target box from test-image embeddings enhanced by the references.Source: §3.1 · p. 3, §3 · p. 3

3.1. Joint grounding and tracking framework

  • Grounding uses language and the first frame to obtain a template, while each later tracking step uses language, that template, and the current search image.Zero padding stands in for the unavailable template during grounding.Source: §3.1 · pp. 3–4

3.2. Vision and language feature extraction

  • BERT encodes the language query and Swin-Transformer encodes image patches, producing token embeddings for the language, template, and test image.The described vision encoder retains the first three Swin-Transformer stages.Source: §3.2 · p. 4

3.3. Multi-source relation modeling

  • A transformer encoder jointly models language, template, and test-image embeddings to capture cross-modality and cross-time relations.For grounding, zero-padding placeholders fill the missing template embeddings and are masked during self-attention.Source: §3.3 · p. 4

3.4. Target decoder and localization head

  • The target decoder combines enhanced test-image embeddings with a target query, and a shared localization head predicts the referred target’s box.For tracking, the query adds the temporal clue to an offline learned embedding; for grounding, it uses the offline embedding.Source: §3.4 · p. 4

3.5. Semantics-guided temporal modeling

  • SGTM learns recent target-appearance clues from historical target states, using language semantics to focus on target regions rather than prediction noise.It encodes historical target-patch embeddings with the NL CLS token, then uses a temporal-query decoder to form a historical target representation.Source: §3.5 · pp. 4–5

3.6. End-to-end learning and inference

  • Training combines grounding and tracking losses in one forward pass, and inference grounds on the first frame before tracking subsequent frames with language, template, and temporal clues.The model can also be initialized with language and bounding-box specifications together.Source: §3.6 · p. 5

4. Experiments

Experiments evaluate tracking on OTB99, LaSOT, and TNL2K, and visual grounding on RefCOCOg. They compare model variants and report results for language-only tracking, language-plus-box tracking, and grounding, alongside an ambiguity limitation.

4.1. Experimental setup

  • The evaluation uses tracking benchmarks with success and precision metrics, and evaluates grounding with Top-1 accuracy on RefCOCOg’s Google-split validation set.The method was tested on an NVIDIA 3090 GPU at about 39 FPS; template and search images are 128 × 128 and 320 × 320, respectively.Source: §4.1 · p. 5

4.2. Ablation study

  • Ablations show gains from multi-source relation modeling, the target decoder, temporal modeling, and adding language CLS tokens to SGTM.The temporal module supports adaptation to appearance variation; adding language CLS tokens brings gains especially in precision.Source: Table 1 · p. 6, §4.2 · p. 6

4.3. Tracking by only language specification

  • On TNL2K, gains over TNL2K-1 and CTRNLT were 40.6% AUC and 46.0% precision, respectively; on LaSOT, gains over CTRNLT were 4.9% AUC and 8.3% precision.On OTB99, the method achieved an AUC score of 59.2%. SGTM’s temporal clue attends to the real target and helps predict a precise box.Source: Table 3 · p. 6, §4.3 · p. 7

4.4. Tracking by language and box specification

  • With language and box initialization, the method improves over VLTTT on TNL2K by 3.8% AUC and 4.8% precision, while VLTTT performs better on LaSOT and OTB99.On OTB99, VLTTT’s AUC was 0.653 and precision was 0.856; the proposed method performed on par with SNLT and TNL2K-2.Source: Table 3 · p. 6, §4.4 · p. 7

4.5. Performance on visual grounding

  • On RefCOCOg’s Google-split validation set, the method performs comparably with the state-of-the-art visual grounding algorithm VLTVG.The comparison reports visual grounding performance on the validation set.Source: Table 4 · p. 7, §4.5 · p. 7

4.6. Limitations

  • Ambiguous natural-language descriptions can cause an incorrect initial target localization and failure throughout the sequence; a clear description or bounding box resolves the example.The method is designed to track using the natural-language description, and the authors note sensitivity to ambiguity.Source: §4.6 · pp. 7–8, Figure 6 · p. 8

5. Conclusion

The paper unifies visual grounding and tracking by modeling cross-modal and cross-temporal relations among references and the test image. It also reports favorable performance on three natural-language tracking datasets and one visual-grounding dataset.

5. Conclusion

  • The framework unifies grounding and tracking by modeling relations among multiple references and the test image, including visual-language and historical-patch/current-frame relations.Source: §5 · p. 8
  • The method achieves favorable performance against state-of-the-art algorithms on three natural-language tracking datasets and one visual-grounding dataset.Source: §5 · p. 8
  • A semantics-guided temporal module models historical target states using global semantic information as guidance, improving tracking performance.Source: §5 · p. 8
Loading 2303.12027v1…