PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, Feng Wu

arXiv:2103.16746v1cs.CVcs.AI

TL;DR

Tracking by natural language lacks a benchmark that reflects its semantic and flexible initialization advantages. The paper introduces TNL2K, diverse evaluation settings, and AdaSwitcher, with reported gains over prior joint language-and-box trackers across three datasets.

  • Problem

    Existing natural-language-initialized trackers are evaluated on bounding-box-oriented benchmarks that omit relative location and significant appearance variation.

  • Method

    The paper constructs TNL2K with densely annotated videos and proposes AdaSwitcher, which adaptively switches between local tracking and global grounding.

  • Results

    0.42|0.50|0.42 on TNL2K is reported for joint language-and-BBox tracking, alongside 0.88|0.68 on OTB-Lang and 0.55|0.51 on LaSOT, outperforming cited prior trackers.

  • Takeaways & Limitations

    TNL2K provides a dedicated platform with adversarial samples, RGB–thermal modality switches, and baseline results for natural-language tracking research.

Abstract

from arXiv · show

Tracking by natural language specification is a new rising research topic that aims at locating the target object in the video sequence based on its language description. Compared with traditional bounding box (BBox) based tracking, this setting guides object tracking with high-level semantic information, addresses the ambiguity of BBox, and links local and global search organically together. Those benefits may bring more flexible, robust and accurate tracking performance in practical scenarios. However, existing natural language initialized trackers are developed and compared on benchmark datasets proposed for tracking-by-BBox, which can't reflect the true power of tracking-by-language. In this work, we propose a new benchmark specifically dedicated to the tracking-by-language, including a large scale dataset, strong and diverse baseline methods. Specifically, we collect 2k video sequences (contains a total of 1,244,340 frames, 663 words) and split 1300/700 for the train/testing respectively. We densely annotate one sentence in English and corresponding bounding boxes of the target object for each video. We also introduce two new challenges into TNL2K for the object tracking task, i.e., adversarial samples and modality switch. A strong baseline method based on an adaptive local-global-search scheme is proposed for future works to compare. We believe this benchmark will greatly boost related researches on natural language guided tracking.

1. Introduction

The introduction motivates tracking by natural language as a more flexible alternative to bounding-box initialization and presents TNL2K with an adaptive baseline for evaluating it.

  • Motivation: Bounding-box initialization is inconvenient, can ambiguously represent the target, and may fail under abrupt appearance or species variation.Examples include confusion between a pedestrian’s bike and lower body, and changes in face or clothing.
  • Motivation: Natural language can specify spatial and semantic target information, helping address bounding-box ambiguity and appearance variation.Descriptions may include attributes, category, shape, properties, and relations to other objects.
  • Benchmark: 2,000 TNL2K videos are densely annotated with per-frame target locations and one English sentence, with 1,300 for training and 700 for evaluation.The dataset includes adversarial samples and RGB–thermal modality switches.
  • Benchmark: TNL2K is intended as a dedicated platform for developing and assessing natural-language-based tracking algorithms.The benchmark is specifically proposed for natural-language-based tracking rather than only conventional bounding-box tracking.
  • Baseline: AdaSwitcher adaptively switches between a local tracking algorithm and a global grounding module as a baseline for comparison.The paper also evaluates more than 40 representative bounding-box-based trackers on TNL2K.

2. Related Work

Prior work covers bounding-box tracking and emerging natural-language tracking, but existing benchmarks remain largely designed for bounding-box initialization and omit key language-tracking conditions.

  • Tracking by Bounding Box: Bounding-box tracking includes classification-, Siamese-network-, correlation-filter-, and regression-based approaches, with deep Siamese trackers achieving state-of-the-art results on multiple benchmarks.These methods begin with an initialized bounding box in the first frame.
  • Tracking by Natural Language: Natural-language tracking is a rising topic with relatively few algorithms, including modular lingual-visual trackers, language-based global proposal methods, and GTI.GTI decomposes tracking into grounding, tracking, and integration subtasks operating simultaneously.
  • Benchmarks for Tracking: Existing benchmarks are mainly designed for bounding-box tracking, and their language descriptions often omit relative target location.This makes them more suitable for language-assisted tracking than language-initialized tracking.
  • Benchmarks for Tracking: Existing benchmarks also lack videos with significant appearance variation, limiting evaluation under conditions such as clothing changes.The discussed benchmark includes diverse natural, animation, infrared, and virtual-game videos for broader evaluation.
  • Benchmarks for Tracking: The benchmark provides baseline results for three settings: natural language only, bounding box only, and joint bounding box and language.These settings are intended to support comparison among future trackers.

3. Tracking by Natural Language

TNL2K combines language, frame-level bounding boxes, and challenging attributes to benchmark natural-language tracking. AdaSwitcher combines global grounding with local tracking and switches search regions when tracking failure is detected.

  • TNL2K Dataset: TNL2K contains 2,000 video sequences with one English sentence and one bounding box per frame.The dataset includes descriptions of target category, shape, attributes, properties, and spatial location.
  • TNL2K Dataset: TNL2K defines 17 attributes for evaluating tracking under challenging factors, including camera motion, occlusion, scale variation, and adversarial samples.The attribute list also includes modality-related and appearance-related challenges.
  • TNL2K Dataset: The benchmark includes 144 evaluation videos longer than 1,000 frames, supporting assessment of long-term trackers.The evaluation split contains 700 videos overall.
  • Proposed Approach: The visual grounding module takes a video frame and language description, fuses visual, language, and spatial features, and predicts the target location globally.Grounding is used especially at the beginning of a video and when local tracking needs recovery.
  • Proposed Approach: The framework initializes SiamRPN++ from the grounding prediction, then combines local tracking with adaptive grounding-based recovery.Two fully connected layers determine whether to switch search regions.
  • Proposed Approach: AdaSwitcher detects tracking failure as an anomaly and switches candidate search regions from local tracking to visual grounding when its prediction exceeds a threshold.Its inputs include confidence, bounding-box, image, response-map, language, and historical frame information.

4. Experiments

Experiments evaluate tracking-by-language, tracking-by-BBox, joint language-and-BBox tracking, and AdaSwitcher components on OTB-Lang, LaSOT, and TNL2K. Results include benchmark comparisons, component analyses, history-information tests, and attribute-specific evaluation.

  • Evaluation Protocol: Precision Plot and Success Plot evaluate tracking using center-location accuracy and bounding-box overlap, respectively.The Success Plot measures the percentage of frames whose predicted-ground-truth IoU exceeds a given ratio.
  • Benchmark Results: 0.49|0.51 on LaSOT and 0.06|0.11|0.11 on TNL2K are reported for tracking by natural language only.The reported values correspond to the metrics listed for Table 3.
  • Benchmark Results: 0.88|0.68 on OTB-Lang and 0.55|0.51 on LaSOT are achieved for joint language-and-BBox tracking.The joint setting combines language with bounding-box initialization or tracking.
  • Ablation Study: 0.344/0.353 improves to 0.355/0.370 when AdaSwitcher is integrated with SiamRPN++, outperforming the naive fused method's 0.347/0.362.The passage attributes the improvement to the adaptive switch mechanism.
  • Ablation Study: 0.353/0.369 improves to 0.355/0.370 with frame attention, while spatial coordinates raise grounding results from 0.103/0.124 to 0.143/0.159.The component analyses evaluate frame attention and spatial coordinates separately.
  • Ablation Study: Discarding predicted BBox information reduces performance from 0.355/0.370 to 0.350/0.365, indicating geometric history contributes to tracking.The history-information analysis compares results with and without predicted BBox input.
  • Attribute Analysis: SiamRCNN performs best across the selected attributes, while RTAA performs worse than DaSiamRPN on Adversarial Samples.The selected attributes are Adversarial Samples, Scale Variation, Background Clutter, and Full Occlusion.

5. Conclusion and Future Works

The paper proposes TNL2K as a large-scale benchmark for natural-language-guided tracking, with dense annotations and an adaptive-switch baseline. It evaluates language-only, BBox-only, and joint tracking settings and identifies benchmark expansion and visual-grounding improvements as future directions.

  • Conclusion: TNL2K contains 2,000 video sequences densely annotated with target bounding boxes and natural-language descriptions.The benchmark is intended as a dedicated platform for natural-language-based tracking.
  • Conclusion: AdaSwitcher is proposed as the adaptive-switch baseline, alongside evaluations of natural-language-only, BBox-only, and joint language-and-BBox tracking.The evaluated settings cover language, bounding-box, and combined initialization or tracking information.
  • Future Works: Future work will extend the benchmark with more videos and baseline trackers and improve the visual grounding module.These directions are stated as planned extensions and improvements.

A. The TNL2K Benchmark

TNL2K is designed specifically for tracking by natural language because existing datasets inadequately represent its distinctive challenges. Its protocol combines dense annotations with difficult visibility, appearance, reasoning, and cross-modality conditions.

  • Motivation and Protocols: TNL2K targets tracking-by-language rather than maximizing dataset size, balancing video length and quantity.The authors describe this as a compromise between the short-term GOT-10k and the larger but less suitable LaSOT setting.
  • Motivation and Protocols: GOT-10k lacks enough videos with special properties and uses short-term sequences that cannot reflect language-based re-detection gains.The authors also report that many LaSOT language annotations do not clearly identify the target.
  • Motivation and Protocols: The collection protocol seeks severe first-frame occlusion, major appearance variation, and targets identifiable only through reasoning.Thermal-tracking videos are added to test robustness to domain adaptation and modality switching.
  • Why add Attribute Modality Switch (MS)?: Modality Switch connects RGB and thermal tracking for long-term scenarios in which daytime and nighttime cameras have complementary usefulness.The benchmark introduces MS because existing tracking studies generally treat these modalities separately or in integrated RGB-T approaches.

A.3. Highlights of TNL2K Dataset

TNL2K is designed specifically for tracking by natural language, combining language and dense bounding-box annotations with challenging and heterogeneous video content. It also supplies diverse baselines for comparing tracking settings.

  • TNL2K is the first benchmark specifically designed for tracking by natural language.
  • Each sequence provides language describing attributes, category, shape, properties, and spatial relationships, alongside dense bounding-box annotations.
  • TNL2K includes actively introduced adversarial samples and significant appearance variation, including cloth or face changes.
  • The heterogeneous dataset contains RGB, thermal, cartoon, and synthetic game videos, supporting domain-adaptation studies.
  • TNL2K provides baselines for tracking by bounding box, language, and joint bounding-box-language input.

B. The Proposed Method

The proposed method combines global visual grounding with adaptive local-global search to provide a strong baseline for language-guided tracking. Its experiments report dataset-level tracking results and implementation details for evaluated trackers.

  • The visual grounding module is optimized with a modified YOLO loss using recalibrated anchor boxes and a softmax layer.
  • The evaluated BBox trackers are compared using publication, feature representation, updating, pretraining, search scheme, efficiency, Precision Plot, and Success Plot.
  • TANet replaces the grounding module for global search in joint language-and-BBox tracking and outputs attention maps highlighting possible search regions.
  • TANet can integrate with a baseline tracker and AdaSwitcher module for robust and accurate tracking.

C.1. Attribute Analysis

Attribute analysis shows that SiamRCNN performs best on most listed attributes, while other trackers lead on selected conditions such as low resolution and thermal crossover.

  • SiamRCNN achieves the best performance on most attributes, including adversarial samples, deformation, fast motion, occlusion, and viewpoint change.
  • KYS achieves the top result on Low Resolution, while SuperDiMP, LTMU, and PrDiMP also perform well across several attributes.
  • MDNet performs best on Thermal Crossover, whose videos are all thermal.
  • The reported attribute results demonstrate strong performance from Siamese trackers using pretraining and joint local-global search.

C.2. Efficiency Analysis

The proposed natural-language tracking baselines provide different speed-accuracy trade-offs on OTB-Lang, with the language-initialized method running faster than the language-guided method.

  • 24.39 FPS is reported for Our-I on OTB-Lang using a laptop with an Intel Core i7 and RTX2070.
  • 12.44 FPS is reported for Our-II on OTB-Lang.

C.3. More Visualization

The visual grounding module generally locates targets accurately in clean backgrounds and some challenging videos, but its localization of position and scale remains insufficient in certain cases.

  • Visual grounding accurately locates targets when the background is relatively clean.This finding comes from visualizations of 20 OTB-Lang video sequences.
  • The module also performs well on some challenging videos, including car and human-head sequences.
  • In some cases, grounding is inaccurate in the target’s central location and scale, limiting its suitability for tracking.The passage identifies further improvement of visual grounding as necessary for more accurate tracking.
  • Figure 12 provides additional tracking results for the proposed baseline and other trackers.
Loading 2103.16746v1…