PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Domain Adaptation for Nighttime Aerial Tracking

Junjie Ye, Changhong Fu, Guangze Zheng, Danda Pani Paudel, Guang Chen

arXiv:2203.10541v2cs.CV

TL;DR

Nighttime aerial tracking poses a domain gap because low-light images differ from daytime data, while existing trackers generalize poorly. UDAT adapts daytime trackers using object discovery and Transformer-based feature alignment with adversarial learning, and experiments report favorable nighttime tracking performance across benchmarks.

  • Problem

    Nighttime images have low contrast, brightness, and signal-to-noise ratio, creating a feature-distribution gap that causes current trackers to generalize poorly.

    Source: §1 · p. 1

  • Method

    UDAT discovers objects in unlabelled nighttime videos to generate training patches, then uses Transformer bridging and day/night discrimination for adversarial feature adaptation.

    Source: §1 · p. 2

  • Results

    UDAT-BAN reaches 0.469 success rate versus SiamBAN’s 0.437 on NAT2021-test, while UDAT trackers raise their baselines by ~4% on UAVDark70.

    Source: §5.2.1 · p. 6

  • Takeaways & Limitations

    Experiments on NAT2021-test and UAVDark70 verify UDAT’s effectiveness and domain adaptability for nighttime aerial tracking.

    Source: §1 · p. 2

Abstract

from arXiv · show

Previous advances in object tracking mostly reported on favorable illumination circumstances while neglecting performance at nighttime, which significantly impeded the development of related aerial robot applications. This work instead develops a novel unsupervised domain adaptation framework for nighttime aerial tracking (named UDAT). Specifically, a unique object discovery approach is provided to generate training patches from raw nighttime tracking videos. To tackle the domain discrepancy, we employ a Transformer-based bridging layer post to the feature extractor to align image features from both domains. With a Transformer day/night feature discriminator, the daytime tracking model is adversarially trained to track at night. Moreover, we construct a pioneering benchmark namely NAT2021 for unsupervised domain adaptive nighttime tracking, which comprises a test set of 180 manually annotated tracking sequences and a train set of over 276k unlabelled nighttime tracking frames. Exhaustive experiments demonstrate the robustness and domain adaptability of the proposed framework in nighttime aerial tracking. The code and benchmark are available at https://github.com/vision4robotics/UDAT.

1. Introduction

UDAT adapts daytime trackers to nighttime aerial tracking without manually annotated target-domain labels, using object discovery and Transformer-based feature adaptation. The paper also introduces NAT2021 and reports effectiveness on nighttime tracking benchmarks.

  • Night images have low contrast, brightness, and signal-to-noise ratio, producing a day/night feature discrepancy that hurts tracker generalization and impedes aerial applications.Source: §1 · p. 1
  • Object discovery generates training patches from unlabelled nighttime data, while Transformer bridging and discriminator components support feature alignment and adversarial learning.Source: §1 · p. 2
  • NAT2021 provides 180 fully annotated test sequences and over 276k unlabelled nighttime tracking frames for unsupervised domain adaptive tracking.Source: §1 · p. 2
  • UDAT is presented as the first unsupervised domain adaptation framework for object tracking, targeting adaptation from daytime general conditions to nighttime aerial perspectives.Source: §1 · pp. 1–2
  • Experiments on NAT2021-test and UAVDark70 verify UDAT’s effectiveness and domain adaptability in nighttime aerial tracking.Source: §1 · p. 2

2. Related work

Related work covers Siamese and correlation-filter trackers, then reviews domain adaptation across vision tasks. The authors identify unsupervised domain adaptation for object tracking as an unaddressed task.

2.1. Object tracking

  • Siamese trackers benefit from large-scale training and end-to-end learning; anchor-free variants use per-pixel regression, while Transformer variants model global information.Source: §2.1 · p. 2
  • Nighttime aerial tracking remains understudied; prior low-light enhancement approaches are described as limited by transferability, handcrafted features, or weak collaboration with the tracker.Source: §2.1 · p. 2

2.2. Domain adaptation

  • The proposed framework uses saliency-based object discovery for target-domain training patches and adversarial feature alignment between source and target domains.Source: Figure 2 · p. 3
  • Domain adaptation has been applied to detection and segmentation, but the authors state it had not yet been investigated for object tracking.Source: §2.2 · pp. 2–3

3. Proposed method

UDAT preprocesses unlabelled nighttime video into paired training patches, then adapts daytime trackers through Transformer-based feature alignment and adversarial domain discrimination. Its training objective combines the baseline tracking loss with an adversarial loss, while the discriminator is optimized in alternating steps.

3. Proposed method

  • UDAT preprocesses unlabelled target-domain video into paired training patches, then uses adversarial feature alignment to adapt daytime trackers for nighttime scenes.The framework applies saliency-based object discovery before the training pipeline modulates extracted features and reduces source–target feature discrepancy.Source: §3 · p. 3

3.1. Data preprocessing

  • Object discovery enhances nighttime frames for saliency detection, filters candidate boxes across time with dynamic programming, and interpolates boxes for frames without a selection.Training patches are cropped from the original images; enhanced crops are only for visualization.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2. Network architecture

  • A Transformer bridging layer adds positional encodings to backbone features and applies self-attention to modulate nighttime features before tracking localization.The paper reports that the bridging layer narrows day/night feature discrepancy.Source: §3.2 · p. 4, Figure 4 · p. 4
  • A day/night discriminator uses a gradient reversal layer and two Transformer layers to distinguish source- from target-domain features during adversarial learning.The tracker head cross-correlates modulated template and search features before classification and regression.Source: §3.2 · p. 4

3.3. Objective functions

  • The training network combines the baseline tracking loss with a least-squares adversarial loss, while the discriminator is optimized in alternating steps.The implementation sets the loss-balancing weight λ to 0.01.Source: §3.3 · p. 5

4. NAT2021 benchmark

NAT2021 provides annotated nighttime aerial tracking sequences for evaluation and unlabelled nighttime videos for unsupervised training. Its sequence collection, annotation process, and illumination-related attributes support analysis of nighttime tracking performance.

4.1. Sequence collection

  • NAT2021-test contains 180 sequences with more than 140k frames, while its training set has 1,400 unlabelled sequences and over 276k frames.NAT2021-L-test is a long-term subset of 23 sequences longer than 1,400 frames.Source: §4.1 · p. 5

4.2. Annotation

  • Test sequences were manually annotated, with low-light enhancement, expert visual inspection, and iterative bounding-box refinement used to ensure annotation quality.The benchmark includes diverse nighttime scenes and targets.Source: §4.2 · p. 5, §4.1 · p. 5

4.3. Attributes

  • The benchmark labels 12 tracking attributes and defines low ambient intensity (LAI) using sequences with average ambient intensity below 20.Ambient intensity is the sequence-level average of frame-level mean pixel intensity in the local region centered on the object; illumination variation is judged by the maximum difference across a sequence.Source: §4.3 · pp. 5–6, Figure 6 · p. 6

4. NAT2021 benchmark

  • SOTA trackers can hardly achieve satisfactory performance on NAT2021-test’s nighttime aerial views.The benchmark is intended to evaluate nighttime aerial tracking and provide unlabelled nighttime videos for unsupervised training.Source: §4 · p. 5, §4.3 · p. 6

5.1. Implementation details

The implementation trains the discriminator and bridging layer alongside pretrained baseline trackers, using separate learning-rate settings and a 20-epoch schedule. Evaluation uses one-pass evaluation and three tracking metrics.

5.1. Implementation details

  • Training runs for 20 epochs, with top-ranked trackers used as baselines and pretrained models from general datasets used to accelerate convergence.The setup reuses the source tracking datasets on which the pretrained models were trained and introduces no new source-domain datasets.Source: §5.1 · p. 6
  • The discriminator uses Adam with a 0.005 base learning rate and a polynomial decay policy with power 0.8; the bridging layer also uses a 0.005 base learning rate.The bridging layer is optimized with the baseline tracker.Source: §5.1 · p. 6
  • Performances are ranked using one-pass evaluation, success rate, precision, and normalized precision.The experiments are implemented in PyTorch on an NVIDIA RTX A6000 GPU.Source: §5.1 · p. 6

5.2. Evaluation results

UDAT improves baseline trackers across nighttime benchmarks, including long-term and illumination-related evaluations, while producing only slight performance fluctuation on a daytime benchmark.

5.2.2 Long-term tracking evaluation

  • UDAT achieves competitive long-term tracking performance and considerably improves upon baseline trackers on NAT2021-L-test.The reported top-10 results are in Table 3.Source: §5.2.2 · p. 6, Table 3 · p. 7

5.2.1 Overall performance

  • UDAT-BAN and UDAT-CAR rank first and second on NAT2021-test; success rates rise from 0.437 to 0.469 for SiamBAN and from 0.453 to 0.483 for SiamCAR.On UAVDark70, UDAT trackers raise baseline performance by approximately 4%, despite its different data distribution from NAT2021.Source: §5.2.1 · p. 6, Figure 7 · p. 7

5.2.3 Illumination-oriented evaluation

  • For illumination variation, UDAT-CAR reaches success rates of 0.442 on NAT2021-test and 0.485 on UAVDark70; for LAI, UDAT-BAN raises SiamBAN’s normalized precision by over 9% on both benchmarks.Existing trackers’ best success rates for illumination variation are 0.408 on NAT2021-test and 0.468 on UAVDark70.Source: §5.2.3 · pp. 7–8, Figure 8 · p. 7

5.2.4 Visualization

  • Confidence-map visualizations show the baseline fails to focus on objects in adverse illumination, while UDAT substantially improves nighttime perception ability.The maps are generated with Grad-CAM.Source: §5.2.4 · p. 8

5.2.5 Source domain evaluation

  • On daytime UAV123, UDAT causes only slight performance fluctuation: within 2% in success rate and 0.5% in precision.The evaluation tests whether nighttime adaptation degrades source-domain performance.Source: §5.2.5 · p. 8

5.3. Empirical study

The empirical study examines domain-adaptive training, object-discovery preprocessing, and the bridging layer. Its results distinguish the contributions of preprocessing and feature bridging to nighttime tracking performance.

5.3. Empirical study

  • The ablation evaluates domain-adaptive training, object-discovery preprocessing, and the bridging layer on NAT2021-test.Table 5 identifies the three components as DA, OD, and BL.Source: §5.3 · p. 8, Table 5 · p. 8
  • Object discovery instead of random cropping boosts target-domain success rate by 3.24%, while the bridging layer doubles the promotion brought by object discovery.The study describes random cropping as producing abundant meaningless samples that hinder learning the target-domain distribution.Source: §5.3 · p. 8
  • Complete UDAT reaches 0.687 precision and 0.483 success rate on nighttime tracking.The authors report that the bridging layer enables the tracker to generate discriminative features from nighttime images.Source: §5.3 · p. 8

6. Conclusion

UDAT uses object discovery and Transformer-based feature alignment and adversarial learning to adapt daytime trackers to nighttime aerial tracking. The authors report improved nighttime performance, while identifying absent target-domain pseudo supervision as a limitation.

6. Conclusion

  • UDAT substantially improves nighttime tracking performance upon state-of-the-art approaches and shows effectiveness and domain adaptability on nighttime tracking benchmarks.The conclusion attributes the improvement to a framework combining object discovery, a Transformer bridging layer, and adversarial learning with a Transformer discriminator.Source: §6 · p. 8
  • Object discovery preprocesses unlabelled target-domain data, while a Transformer bridging layer narrows the gap between daytime and nighttime image features.Source: §6 · p. 8
  • Adversarial learning with a Transformer discriminator optimizes the adapted model for nighttime tracking.Source: §6 · p. 8
  • The work identifies the absence of target-domain pseudo supervision as its limitation and proposes reliable pseudo supervision as future work.Source: §6 · p. 8
  • NAT2021 is presented as a benchmark intended to facilitate research on nighttime and other adverse-condition visual tracking.Source: §6 · p. 8

Related papers

Loading 2203.10541v2…