PaperlayerView exact revision on arXiv

Source-linked AI summary

Anti-UAV: A Large Multi-Modal Benchmark for UAV Tracking

Nan Jiang, Kuiran Wang, Xiaoke Peng, Xuehui Yu, Qiang Wang, Junliang Xing, Guorong Li, Jian Zhao, Guodong Guo, Zhenjun Han

arXiv:2101.08466v3cs.CV

TL;DR

UAV tracking lacks a high-quality multimodal benchmark despite the need to monitor UAV operation status. The paper introduces Anti-UAV and DFSC, and reports that DFSC improves tracker performance while leveraging cross-sequence semantics without additional inference time.

  • Problem

    Existing RGB trackers can lose useful cues in low light, infrared images can have low resolution, and no multimodal UAV tracking benchmark was available.

  • Method

    The paper constructs Anti-UAV with RGB and infrared tracking data and proposes DFSC, which uses class-level and instance-level semantic modulation across sequences.

  • Results

    DFSC significantly outperforms baselines, and its semantic modulation improves tracker robustness and discrimination without additional inference time.

  • Takeaways & Limitations

    Anti-UAV provides a large-scale benchmark, protocols, metrics, and baselines for advancing UAV tracking research.

Abstract

from arXiv · show

Unmanned Aerial Vehicle (UAV) offers lots of applications in both commerce and recreation. With this, monitoring the operation status of UAVs is crucially important. In this work, we consider the task of tracking UAVs, providing rich information such as location and trajectory. To facilitate research on this topic, we propose a dataset, Anti-UAV, with more than 300 video pairs containing over 580k manually annotated bounding boxes. The releasing of such a large-scale dataset could be a useful initial step in research of tracking UAVs. Furthermore, the advancement of addressing research challenges in Anti-UAV can help the design of anti-UAV systems, leading to better surveillance of UAVs. Besides, a novel approach named dual-flow semantic consistency (DFSC) is proposed for UAV tracking. Modulated by the semantic flow across video sequences, the tracker learns more robust class-level semantic information and obtains more discriminative instance-level features. Experimental results demonstrate that Anti-UAV is very challenging, and the proposed method can effectively improve the tracker's performance. The Anti-UAV benchmark and the code of the proposed approach will be publicly available at https://github.com/ucas-vg/Anti-UAV.

I. INTRODUCTION

The paper motivates UAV tracking with the need to monitor UAV locations and trajectories, then introduces Anti-UAV and DFSC to support multimodal tracking research.

  • I. INTRODUCTION: UAV tracking is important because UAVs are increasingly used commercially and recreationally, making operation status, location, and trajectory monitoring necessary.
  • I. INTRODUCTION: RGB trackers can fail in low light, while infrared imagery often has low resolution, motivating fusion of visible and infrared information.
  • I. INTRODUCTION: Anti-UAV addresses the lack of a multimodal UAV tracking benchmark by pairing RGB and infrared sequences with bounding boxes, attributes, and target-existence flags.The dataset supports both single-modal and multimodal UAV tracking.
  • I. INTRODUCTION: DFSC uses class-level and instance-level semantic modulation to exploit shared information across UAV sequences and improve feature robustness and discrimination.The method is designed specifically for the single-category UAV setting.
  • I. INTRODUCTION: The benchmark contains 318 RGB-T video pairs and is intended to facilitate research on UAV tracking across visible and thermal modalities.

II. RELATED WORK

Prior tracking datasets and methods span RGB, thermal, and RGB-T settings, while Anti-UAV targets UAVs with larger, unaligned multimodal video pairs.

  • II. RELATED WORK: Existing tracking benchmarks cover general RGB, thermal, and RGB-T objects, but Anti-UAV is specifically constructed for UAV tracking.
  • II. RELATED WORK: Thermal tracking research includes benchmarks and methods using deep features, motion, similarity modeling, particle filtering, and sparse representations.
  • II. RELATED WORK: Anti-UAV differs from prior RGB-T datasets because its paired video sequences are unaligned, creating a more challenging multimodal tracking setting.
  • II. RELATED WORK: RGB tracking methods include correlation-filter, deep, and Siamese-based approaches, with performance depending on search areas and discriminative background samples.
  • II. RELATED WORK: Anti-UAV is reported as much larger than most RGB-T tracking datasets and provides a corresponding training set for UAV tracking.

C. Image-based Training Strategy

The image-based training discussion places DFSC among strategies that exploit cross-image or cross-sequence instances to improve tracker robustness and discrimination.

  • C. Image-based Training Strategy: Intra-image strategies reuse instances from one image, while inter-image strategies use instances across images to address limitations of single-image training.
  • C. Image-based Training Strategy: DFSC uses cross-sequence pairs in its class-level modulation stage because UAV is the only tracking category in Anti-UAV.The strategy operates within the current batch and shares feature-extraction computation.
  • C. Image-based Training Strategy: The dataset includes 318 RGB-T video pairs, with each pair containing an RGB video and a thermal video.
  • C. Image-based Training Strategy: The videos cover multiple UAV types, day and night conditions, visible and infrared modes, diverse backgrounds, and a frame rate of 25 FPS.

B. Annotation

Anti-UAV combines progressive bounding-box annotation with attributes and target-presence flags, enabling analysis of varied UAV tracking conditions and difficulties.

  • B. Annotation: The annotation process progressively refines UAV bounding boxes and records attributes, target presence, and difficult frames such as occlusion or blur.An empty bounding-box list denotes that the target does not exist.
  • B. Annotation: The displayed attribute annotation table defines the dataset’s attribute labels used for per-condition tracking analysis.
  • B. Annotation: Anti-UAV provides seven binary attributes and sequence statistics to analyze tracker strengths and weaknesses across conditions.
  • B. Annotation: Large inter-frame motion exceeding 60 pixels is common and represents a major UAV tracking difficulty.
  • B. Annotation: Most sequences are captured during the day, while target-size distributions across splits remain similar and average target sizes stay below 40 pixels.
  • B. Annotation: The dataset includes many target-confusion sequences, and TChard is identified as the most challenging tracking attribute.

E. Evaluation Metrics

Anti-UAV evaluates tracking beyond localization by incorporating whether the UAV is present, while supporting separate visible and infrared protocols and unaligned multimodal data.

  • E. Evaluation Metrics: mSA averages per-frame state accuracy across video sequences, combining IoU-based localization with UAV visibility predictions.The metric uses ground-truth visibility flags and predicted presence states; precision and success remain standard tracking measures.
  • E. Evaluation Metrics: Protocol I evaluates visible or infrared trackers without UAV-containing training data to test generalization.Protocol II instead permits Anti-UAV training or fine-tuning for a dedicated UAV-tracking evaluation.
  • E. Evaluation Metrics: Anti-UAV evaluates visible and infrared annotations separately, with paired modalities that are not spatially aligned.The unaligned RGB-T design creates a multimodal tracking setting distinct from standard aligned datasets.

A. Class-level Semantic Modulation

Class-level semantic modulation uses cross-sequence query features to generate UAV-relevant proposals and trains the tracker with same-sequence and cross-sequence supervision.

  • A. Class-level Semantic Modulation: CSM modulates each search feature with query features from another sequence to find candidate regions containing UAVs.When the sequence indices match, the operation becomes intra-sequence modulation; otherwise it uses cross-sequence modulation.
  • A. Class-level Semantic Modulation: The modulated feature retains the search-feature size, while projected query and search features are combined and channel-aligned before proposal generation.The convolution operator combines the projected features, and the output is aligned with the search feature channels.
  • A. Class-level Semantic Modulation: CSM trains classification and regression with a loss combining intra-sequence Lsame and cross-sequence Lcross through weight α.The cross term represents RPN prediction after cross-sequence modulation, while the same term represents intra-sequence prediction.

B. Instance-level Semantic Modulation

Instance-level semantic modulation refines CSM-selected proposals using the query from their own sequence, emphasizing instance-specific discrimination before final classification and regression.

  • B. Instance-level Semantic Modulation: ISM uses same-sequence query features to distinguish selected UAV proposals from similar-looking instances and complex backgrounds.The stage follows class-level proposal selection and focuses on instance-related information.
  • B. Instance-level Semantic Modulation: The modulator combines projected query and proposal features through Hadamard production while preserving their feature size.The selected proposals are indexed by k, and the query comes from the sequence containing the current proposal.
  • B. Instance-level Semantic Modulation: The modulated ROI proposals undergo classification and bounding-box regression, with the ISM loss averaged over proposals selected by CSM.Npnum denotes the number of proposals passed from CSM into this stage.

V. EXPERIMENTS

The experiments compare deep-learning and correlation-filter trackers under Anti-UAV protocols, using DFSC implementation settings and attribute-based evaluation on the validation set.

  • V. EXPERIMENTS: More than 40 trackers are evaluated on Anti-UAV, including deep-learning and correlation-filter trackers under Protocol I.The supplied passages identify the tracker families and the Protocol I comparison setting, but do not provide numerical rankings.
  • V. EXPERIMENTS: The experiments include success and precision plots for the top 20 trackers under Protocol I.The supplied figure passage specifies the plotted metrics and truncates the display to the leading 20 trackers.
  • V. EXPERIMENTS: Protocol II uses GlobalTrack as the baseline with separate visible and infrared hyper-parameter settings.Visible training uses 12 epochs, while infrared training uses 18 epochs; both begin with a 0.02 learning rate and decay it twice.
  • V. EXPERIMENTS: CSM and ISM use cross-entropy for classification, smooth l1 for regression, and batch size 2 per GPU.These settings define the reported DFSC training configuration.
  • V. EXPERIMENTS: Table III reports attribute-based mSA for baseline trackers on the Anti-UAV validation set under Protocol I.Higher mSA is better, and the table marks the first-, second-, and third-place trackers.

C. Evaluations under Protocol I

Protocol I evaluates trackers across overall performance, state accuracy, speed, and challenging attributes on Anti-UAV. SiamRCNN and GlobalTrack lead many results, while long-term tracking improves performance but challenges real-time use.

  • Overall Performance: 95.70% precision and 71.52% success make SiamRCNN best on the infrared validation set, while its 63.60% infrared test success is state of the art.
  • Overall Performance: 87.13% precision makes GlobalTrack best on the infrared test set, whereas SiamRCNN leads the visible test set by 4.68 success points and 3.06 precision points.
  • State Accuracy: 74.33% and 65.41% are SiamRCNN’s best infrared state accuracy scores on the validation and test sets, respectively.
  • Tracking Attributes: Long-term trackers generally perform better because larger search areas can recover targets that disappear from view, but they are challenging to deploy in real time.
  • Attribute-based Performance: SiamRCNN and GlobalTrack outperform most trackers on many attributes, with SiamRCNN particularly strong on OV, TC, and LR while GlobalTrack leads slightly on OV and SV in some settings.

D. Evaluation under Protocol II

Protocol II compares normal, large-scale, and DFSC training strategies using mSA, precision, success, visualizations, and attribute-based analyses. DFSC achieves the best overall performance and improves most attributes, although gains are limited on others.

  • Qualitative Analysis: Visualized successful sequences indicate that DFSC can handle challenging UAV-tracking cases, while precision and success comparisons show consistent improvement.
  • Overall Performance: DFSC obtains the best overall performance on infrared and visible sequences, gaining 0.49 mSA on validation and 0.68 mSA on test over normal training.
  • Overall Performance: DFSC improves visible-sequence mSA by 0.48 on validation and 0.57 on test compared with normal training, while test-set gains decrease because of training-set overfitting.
  • Attribute-based Performance: DFSC improves most validation attributes, gaining 1.28%, 0.97%, and 0.78% mSA on OC, SV, and LR, respectively, while other attributes improve little.
  • Attribute-based Performance: On the test set, DFSC reaches 62.75% mSA on TCall, 1.07% above normal training, and gains 0.89%, 1.21%, and 0.98% on TCeasy, TCmed, and TChard.

E. Ablation Study

The ablations show that DFSC benefits from instance-level semantic modulation, balanced supervision, and a moderate cross-sequence loss ratio. DFSC performs favorably against DFSC-all, while α=0.25 balances intra-sequence and cross-sequence semantics.

  • Supervised Task: DFSC-cls achieves the best mSA but the lowest infrared success and precision, whereas DFSC-reg obtains the highest precision and success scores.Classification supervision improves UAV state discrimination, while regression supervision favors localization.
  • Semantic modulation: DFSC performs favorably compared with DFSC-all, indicating that later class-level modulation can confuse cross-sequence information and instance-level modulation strengthens discrimination.The ablation identifies instance-level semantic modulation as necessary for enhancing the tracker's discrimination power.
  • Influence of Ratio: α=0.25 achieves the best trade-off between intra-sequence semantics and cross-sequence semantics, whereas smaller or larger ratios reduce performance.Performance first increases and then decreases as α changes; excessive cross-sequence weighting hinders class-level semantic learning, while a smaller value degenerates toward normal training.
  • Overall conclusion: DFSC improves tracker robustness and discrimination ability by leveraging semantic information across video sequences without adding inference time.The conclusion frames this semantic transfer as the central benefit of the proposed training strategy.
Loading 2101.08466v3…