PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation

Changhong Fu, Liangliang Yao, Haobo Zuo, Guangze Zheng, Jia Pan

arXiv:2307.01024v2cs.CV

TL;DR

Nighttime UAV tracking is challenged by the day–night domain gap and by target-domain samples that may lack precise object boundaries and diversity. SAM-DA uses SAM to generate many high-quality adaptation samples from each nighttime image. The paper reports robust, domain-adaptable tracking with fewer raw nighttime images.

  • Problem

    Nighttime tracking adaptation is limited by target-domain samples that may lack precise pixel-level object locations and boundaries, while nighttime image differences from daytime images create a domain gap.

    Source: §I · p. 1

  • Method

    SAM-DA uses SAM-powered one-to-many sample generation to obtain many high-quality target-domain training samples from each challenging nighttime image.

    Source: §I · p. 2

  • Results

    Experiments on extensive nighttime UAV videos report robustness and domain adaptability, with better performance than SOTA domain adaptation using fewer raw nighttime images.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    The fewer-image training approach is reported to promote quick validation and deployment of UAV algorithms.

    Source: §I · p. 2

Abstract

from arXiv · show

Domain adaptation (DA) has demonstrated significant promise for real-time nighttime unmanned aerial vehicle (UAV) tracking. However, the state-of-the-art (SOTA) DA still lacks the potential object with accurate pixel-level location and boundary to generate the high-quality target domain training sample. This key issue constrains the transfer learning of the real-time daytime SOTA trackers for challenging nighttime UAV tracking. Recently, the notable Segment Anything Model (SAM) has achieved a remarkable zero-shot generalization ability to discover abundant potential objects due to its huge data-driven training approach. To solve the aforementioned issue, this work proposes a novel SAM-powered DA framework for real-time nighttime UAV tracking, i.e., SAM-DA. Specifically, an innovative SAM-powered target domain training sample swelling is designed to determine enormous high-quality target domain training samples from every single raw nighttime image. This novel one-to-many generation significantly expands the high-quality target domain training sample for DA. Comprehensive experiments on extensive nighttime UAV videos prove the robustness and domain adaptability of SAM-DA for nighttime UAV tracking. Especially, compared to the SOTA DA, SAM-DA can achieve better performance with fewer raw nighttime images, i.e., the fewer-better training. This economized training approach facilitates the quick validation and deployment of algorithms for UAVs. The code is available at https://github.com/vision4robotics/SAM-DA.

I. INTRODUCTION

Nighttime UAV tracking faces a day–night feature gap, while high-quality nighttime annotations are costly and existing adaptation samples can miss precise object boundaries and diversity. SAM-DA uses SAM-powered one-to-many sample generation for adaptation, and reports better tracking with fewer raw nighttime images.

  • Nighttime images’ limited illumination, low contrast, and noise create a day–night feature-distribution gap, while collecting high-quality nighttime tracking annotations is costly and time-consuming.Source: §I · p. 1
  • Existing adaptation samples may lack precise pixel-level object locations and boundaries and often generate only one target-domain sample per nighttime image.Source: §I · p. 1
  • SAM’s zero-shot generalization, attributed to training with over one billion masks, offers potential to discover numerous and diverse objects.Source: §I · p. 1
  • Directly applying SAM to real-time nighttime UAV tracking is difficult because UAVs have limited payload and computation resources.Source: §I · p. 2
  • SAM-DA introduces SAM into day–night domain adaptation and swells each challenging nighttime image into many high-quality target-domain training samples.The paper describes this as one-to-many generation.Source: §I · p. 2
  • SAM-DA reports better nighttime UAV tracking with fewer raw images, and Fig. 1 reports superior NUT-L performance using about 10% of UDAT’s training images.The figure compares SAM-DA-Track training-image shares of about 10.0%, 33.2%, 50.1%, and 100% of NAT2021-train.Source: Figure 1 · p. 1, §I · p. 2

A. Nighttime UAV tracking

Prior nighttime UAV tracking work includes low-light enhancement, while day–night domain adaptation has been used in other vision tasks and SAM has demonstrated zero-shot use across vision applications. The section describes gaps in applying these approaches to nighttime UAV tracking.

A. Nighttime UAV tracking

  • Low-light enhancers address inadequate illumination and noise, but their limited connection to tracking and added model parameters leave feature-distribution gaps and strain UAV resources.Source: A. Nighttime UAV tracking · p. 2

B. Day-night domain adaptation

  • Day–night domain adaptation transfers information from daytime source domains to nighttime targets, but tracking adaptation remains constrained by low-quality samples with imprecise boundaries and only one sample per image.Source: B. Day-night domain adaptation · p. 2

C. Segment anything model

  • SAM’s zero-shot generalization has been explored for camouflaged-object detection and medical image segmentation, but not UAV tracking, especially at night.Source: C. Segment anything model · p. 2

III. PROPOSED METHOD

SAM-DA uses SAM to identify potential objects in nighttime images and expand each image into many target-domain training samples for day–night adaptation. The samples are paired template and search crops, while daytime source samples and automatically generated nighttime samples train the tracker.

III. PROPOSED METHOD

  • SAM-DA applies SAM to nighttime images to locate potential objects and generate high-quality target-domain samples for day–night adaptation.The framework uses manually annotated daytime source samples alongside automatically generated nighttime target samples.Source: §III · pp. 2–3, Figure 2 · p. 3

A. SAM-powered target domain training sample swelling

  • An encoder and decoder produce potential-object masks with accurate locations and boundaries, whose surrounding boxes define the target objects.The image is patchified before encoding; a mask-deciding head generates masks, and boxes around them are used to form target samples.Source: A. SAM-powered target domain training… · p. 3
  • Each nighttime image serves as both template and search frame, producing paired crops around each boxed object and swelling one image into many samples.Template and search crops are resized to fixed sizes and paired when they contain the same target; the one-to-many process increases sample quantity and quality.Source: A. SAM-powered target domain training… · p. 4

B. Tracking-oriented day-night domain adaptation

SAM-DA-Track shows greater efficacy than the Baseline in the low-light confidence-map comparison. Its tracking-oriented day–night adaptation aligns daytime and nighttime features through a framework with a backbone, bridging module, tracker head, and discriminator.

B. Tracking-oriented day-night domain adaptation

  • Day–night adaptation aligns source daytime and target nighttime features to improve nighttime tracking performance.The framework follows the Baseline paradigm and includes a backbone, bridging module, tracker head, and discriminator.Source: B. Tracking-oriented day-night domain… · p. 4
  • SAM-DA-Track demonstrates notable efficacy over the Baseline in low-light confidence-map comparisons on NUT-L.The target objects are marked by red boxes.Source: Figure 4 · p. 4
  • An adversarially trained discriminator distinguishes whether features come from the source or target domain.The caption attributes improved nighttime tracking to the substantial high-quality target samples used by the adaptation framework.Source: B. Tracking-oriented day-night domain… · p. 4

A. Implementation details

SAM-DA-Track significantly surpasses the Baseline and other SOTA methods in the reported overall comparison. The experiments vary the share of NAT2021-train images used for sample swelling and evaluate trackers on nighttime benchmarks.

A. Implementation details

  • The four SAM-NAT training-set versions use approximately 10.0%, 33.2%, 50.1%, or 100% of NAT2021-train raw images.The versions are nano (N), tiny (T), small (S), and base (B), respectively.Source: A. Implementation details · p. 4
  • Training uses a single NVIDIA A100 GPU, adds no additional data, and follows the Baseline setup with the base tracker.Evaluation covers DarkTrack2021 and NUT-L, a long-term benchmark formed from NAT2021-test and UAVDark135 with 42 sequences and 95,274 images.Source: A. Implementation details · pp. 4–5
  • SAM-DA-Track significantly surpasses the Baseline and other SOTA methods in overall performance on DarkTrack2021 and NUT-L.The evaluation ranks performance by success rate, precision, and normalized precision.Source: Figure 5 · p. 5, A. Implementation details · p. 5

B. Overall evaluation

SAM-DA-Track outperforms Baseline on DarkTrack2021 and ranks first on NUT-L, with reported improvements across both benchmarks’ metrics.

B. Overall evaluation

  • On NUT-L, SAM-DA-Track ranks first and raises Baseline by 14.1%, 13.4%, and 13.3% across three metrics.On DarkTrack2021, it improves over Baseline by 7.8%, 5.8%, and 5.0% on success rate, normalized precision, and precision, respectively.Source: B. Overall evaluation · p. 5
  • On DarkTrack2021, SAM-DA-Track improves over Baseline by 7.8% on success rate, 5.8% on normalized precision, and 5.0% on precision.These are the three reported DarkTrack2021 metric improvements.Source: B. Overall evaluation · p. 5
  • The authors attribute the gains to SAM-powered high-quality training samples and report better adaptability across variant nighttime conditions.The evaluation covers DarkTrack2021 and NUT-L.Source: B. Overall evaluation · p. 5

C. Attribute-based performance

SAM-DA-Track improves on illumination variation and low ambient intensity, while SAM-powered sample swelling expands and diversifies target-domain training data, particularly in low light.

C. Attribute-based performance

  • For illumination variation, SAM-DA-Track improves over Baseline by 7.1%, 5.0%, and 4.0% on DarkTrack2021 and 15.3%, 15.2%, and 15.1% on NUT-L across three metrics.The passage reports the improvements in metric order without naming each metric separately.Source: C. Attribute-based performance · p. 5, Table I · p. 6
  • For low ambient intensity, SAM-DA-Track improves over Baseline by 7.8%, 6.2%, and 5.4% on DarkTrack2021 and 12.6%, 10.8%, and 11.1% on NUT-L across three metrics.The passage reports the improvements in metric order without naming each metric separately.Source: C. Attribute-based performance · p. 5, Table I · p. 6

D. Analysis on training sample swelling

  • SAM produces pixel-level masks with clear boundaries in low light and adds candidates beyond common cars and people, including monitors and traffic signs.The authors note that no low-light image enhancement is required.Source: D. Analysis on training sample swelling · pp. 5–6
  • SAM-powered swelling enriches diverse lighting conditions, especially samples with AI values below 20, which indicate low ambient intensity.The figure compares four SAM-NAT versions with NAT2021-train.Source: Figure 6 · p. 6, Figure 7 · p. 7
  • With the same NAT2021-train image count, SAM-NAT-B contains 16,073,740 samples, 58.2 times the original 276,081 samples.SAM-NAT-N contains 1,608,843 samples using 10.0% of the images; SAM-NAT-T has 5,314,760 using 33.2%, and SAM-NAT-S has 8,042,926 using 50.1%.Source: D. Analysis on training sample swelling · p. 6, Table II · p. 6

E. Fewer-better training

SAM-DA-Track performs better with fewer raw nighttime images and can improve further with more training data and time.

E. Fewer-better training

  • With 10.0% of training images and about 2.4 hours of training, SAM-DA-Track scores 0.411 on NUT-L versus Baseline’s 0.378.The passage presents this as evidence of better performance with constrained data and training time.Source: E. Fewer-better training · p. 7
  • With SAM-NAT-B swelling the whole dataset and the same training time as Baseline, SAM-DA reaches 0.430 on NUT-L.The passage reports further improvement with more training data and longer training time.Source: E. Fewer-better training · p. 7
  • The fewer-better evaluation supports the practicality of training with fewer raw images and the authors describe the method as not data-hungry.The passage identifies limited training data and the need for quick training as relevant to nighttime UAV applications.Source: E. Fewer-better training · p. 7

F. Day-night feature distribution

The t-SNE visualization indicates that SAM-DA reduces the day-night feature discrepancy relative to the base tracker and Baseline by leveraging high-quality target-domain samples.

F. Day-night feature distribution

  • The t-SNE visualization indicates that SAM-DA further significantly reduces day-night domain discrepancy compared with the base tracker and Baseline, leveraging enormous high-quality target-domain training samples.Source: F. Day-night feature distribution · p. 7
  • The authors link the reduced discrepancy to SAM-DA’s use of enormous high-quality target-domain training samples.The passage presents this visualization as evidence of domain adaptability.Source: F. Day-night feature distribution · p. 7
  • Figure 8 presents real-world test results, with CLE below the green dashed line indicating successful tracking.The passage does not state a broader result for those tests.Source: Figure 8 · p. 7

G. Real-world tests

SAM-DA expands high-quality target-domain training samples from nighttime images and is reported to show robustness and domain adaptability. Real-world tests report 32 fps amid challenging visual conditions.

G. Real-world tests

  • In real-world tests, SAM-DA-Track ran at 32 fps and performed well amid fast motion, low resolution, occlusion, and illumination variation.A laptop with an NVIDIA RTX 3060 GPU served as the ground control station.Source: G. Real-world tests · p. 7

V. CONCLUSION

  • SAM-DA’s one-to-many generation expands high-quality target-domain samples, reducing raw-image reliance while the paper reports robustness and domain adaptability across extensive nighttime videos.The conclusion also links reduced raw-image reliance with enhanced generalization and prevention of overfitting.Source: §V · p. 7
Loading 2307.01024v2…