PaperlayerView exact revision on arXiv

Source-linked AI summary

Catastrophic Forgetting in Sequential Thermal Anti-UAV Detection: The Role of Scale-Conditioned Gradient Imbalance

Khac Duc Giang Nguyen, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansoor Alsahag

arXiv:2610.08315v1cs.CV

TL;DR

Continual fine-tuning across evolving thermal anti-UAV datasets risks catastrophic forgetting, a problem not systematically evaluated in this domain. This study measures retention in a sequential three-benchmark YOLOMG curriculum and tests distillation and stratified replay. Naive CST fine-tuning yields FM = −0.605, while replay reduces forgetting; the study’s controls limit causal attribution.

  • Problem

    Continual learning methods have not been systematically evaluated for thermal anti-UAV detection across operational datasets, despite the need to retain earlier-task performance as distributions shift.

    Source: §1 · pp. 1–2

  • Method

    The study sequentially trains a single-stream YOLOMG detector across three thermal anti-UAV datasets and evaluates knowledge distillation and scale-stratified replay for retention.

    Source: §2.2 · p. 3, §3.5 · p. 5

  • Results

    FM = −0.605 for naive CST fine-tuning against the Stage 1 ceiling, while SSH reduces FM to −0.311 and random-stratified replay reaches −0.221.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The results suggest scale-stratified replay can offset forgetting in this curriculum, while the random-stratified control indicates the gain is not specific to herding selection.

    Source: §6 · p. 14

    No Stage 2 no-KD control isolates distillation’s contribution, and Stage 3 is single-seed, limiting causal attribution and estimate precision.

    Source: §5.3 · pp. 12–13

Abstract

from arXiv · show

Counter-UAV systems based on thermal infrared detection must stay accurate as operational datasets evolve, yet sequential fine-tuning causes catastrophic forgetting of prior tasks, a problem that remains insufficiently characterized in this domain. This continual-learning study measures the stability-plasticity trade-off in YOLOMG, a YOLOv5-based detector run as a single thermal-infrared stream with the motion channel disabled, trained sequentially across three anti-UAV benchmarks of rising scale difficulty: Anti-UAV-RGBT, Anti-UAV410, and CST Anti-UAV. Naive fine-tuning on CST yields a Forgetting Measure of -0.605 against the Stage 1 ceiling, corresponding to a 90% capability loss, with -0.572 occurring in Stage 3 alone. In contrast, knowledge distillation from a frozen teacher is associated with FM = -0.033 +/- 0.004 across three seeds, corresponding to 95% retention. Because no Stage 2 no-KD control is included, this result establishes retention under KD training rather than a causal KD effect. Per-stratum analysis shows large-target detection collapsing to near zero within the first epoch, despite an inter-stage cosine similarity of 0.987 over the gradient-updated weights, pointing to scale-conditioned gradient imbalance, rather than weight drift, as a candidate mechanism. Scale-Stratified Herding (SSH), a 300-exemplar buffer balanced across four UAV size strata, roughly halves the forgetting (FM = -0.605 to -0.311) and keeps large-target detection non-zero. An ablation attributes the gain primarily to scale stratification rather than herding: random-stratified replay performs at least as well (FM = -0.221 versus -0.311 for SSH). These replay results are single-seed and should therefore be treated as preliminary.

1 Introduction

Thermal anti-UAV systems must retain detection accuracy as datasets and target scales shift, but continual-learning methods have not been systematically evaluated in this domain. This study chains three thermal benchmarks to measure forgetting and investigate scale-related effects.

Continual-learning challenge

  • Sequential fine-tuning can improve a new task while interfering with prior representations, causing abrupt performance loss; continual learning addresses this stability-plasticity trade-off.Existing approaches include rehearsal, regularisation, gradient projection, and knowledge distillation.Source: §1 · p. 1

Research gap

  • Thermal anti-UAV benchmarks have not been chained into one model’s training history to measure stability-plasticity, nor has gradient starvation been examined as a driver of forgetting after distribution shift.The benchmarks capture complementary regimes: standard-scale drones, wider size ranges, and extreme tiny targets.Source: §1 · pp. 1–2

Study design

  • This study trains YOLOMG sequentially on Anti-UAV-RGBT, Anti-UAV410, and CST Anti-UAV, holding thermal input constant while moving from moderate domain shift to extreme scale shift.The motion channel is disabled; the Stage 3 comparison is descriptive because Stage 2 uses KD and Stage 3 uses naive fine-tuning.Source: §1 · p. 2

Research questions and findings

  • Stage 2 KD training retained most Task 1 performance, but the missing no-KD control prevents attributing that retention causally to distillation.Stage 3 results tentatively fit scale-conditioned gradient imbalance better than outright weight overwriting; SSH roughly halves forgetting.Source: §1 · p. 2

2 Related Work

Prior work spans thermal anti-UAV detection, continual learning, and incremental object detection, but the supplied review identifies no systematic study of sequential forgetting across real thermal anti-UAV datasets. It frames scale-conditioned gradient imbalance as a candidate mechanism requiring investigation.

2 Related Work

  • No prior work had combined continual learning with thermal anti-UAV detection across multiple operational datasets or examined scale-conditioned gradient imbalance as a candidate driver of forgetting.The work positions its gap across thermal anti-UAV detection, sequential adaptation, and scale-related gradient starvation.Source: §2 · p. 2

2.1 Thermal Anti-UAV Detection

  • Thermal anti-UAV benchmarks progress from standard-scale drones in Anti-UAV-RGBT and Anti-UAV410 to extreme tiny targets in CST Anti-UAV, creating a near-complete scale inversion.YOLO-family detectors are common for their speed–accuracy balance, while prior reliability and adaptation work does not study catastrophic forgetting after sequential cross-dataset training.Source: §2.1 · p. 2

2.2 Continual Learning

  • Continual learning balances retention against acquiring new capabilities; this study uses domain-incremental learning because the UAV class stays fixed while the input distribution changes.The review contrasts continual learning with domain generalisation, one-shot adaptation, and parameter-efficient adaptation, and discusses rehearsal, regularisation, architecture, gradient projection, and distillation methods.Source: §2.2 · p. 3

2.3 Incremental Object Detection

  • Incremental object-detection research includes class-incremental distillation and domain-incremental adaptation, but prior methods do not quantify per-stratum forgetting or examine gradient starvation in this setting.The review identifies sequential training across real thermal anti-UAV datasets and scale-distribution shift as unaddressed in prior incremental detection work.Source: §2.3 · p. 3

2.4 Gradient Starvation and Scale Distribution Shift

  • Gradient starvation describes dominant features suppressing updates to weaker relevant features; here, large-target signals may vanish when CST training lacks large targets, despite limited aggregate weight displacement.The review distinguishes this candidate mechanism from gradient interference: projection methods cannot project gradients that do not exist.Source: §2.4 · pp. 3–4

3 Methodology

The methodology uses one detector in a sequential, domain-incremental curriculum across three thermal anti-UAV datasets, measuring Task 1 retention after training stages. It includes a diagnostic naive CST stage and a scale-stratified replay design.

3 Methodology

  • A single detector is trained sequentially on Anti-UAV-RGBT, Anti-UAV410, and CST Anti-UAV, with prior-task detection measured after each stage to quantify forgetting.The protocol is domain-incremental: the UAV class remains fixed while dataset distributions change.Source: §3 · p. 4, Figure 1 · p. 5

3.1 Backbone: YOLOMG

  • YOLOMG is run as a single-stream appearance detector: the motion channel is zeroed, and its three detection heads predict at small, medium, and large scales.The detector is YOLOv5-based, with dataset-specific anchors and 318 layers.Source: §3.1 · p. 4, Figure 1 · p. 5

3.2 Stage 1: Supervised Training on Anti-UAV-RGBT

  • Stage 1 trains YOLOMG on Anti-UAV-RGBT infrared frames from ImageNet-pretrained weights; the best validation-mAP checkpoint initializes Stage 2 and sets the Task 1 ceiling.Training uses 100 epochs of SGD with momentum 0.937 and weight decay 5 × 10^-4.Source: §3.2 · p. 4

3.3 Stage 2: Knowledge Distillation on Anti-UAV410

  • Stage 2 fine-tunes the Stage 1 checkpoint on Anti-UAV410, using the frozen Stage 1 model as a teacher to constrain the student’s output distribution.Detection and distillation losses are equally weighted with λ_kd = 1.0, and three seeds are run.Source: §3.3 · p. 4

3.4 Stage 3: Naive Fine-tuning on CST Anti-UAV

  • Stage 3 fine-tunes the best Stage 2 checkpoint on CST Anti-UAV without memory protection, measuring forgetting after training reaches peak Task 3 mAP.This naive baseline is designed to diagnose forgetting and parameter-space changes under the shift from normal/large targets toward tiny/small targets.Source: §3.4 · p. 4

3.5 Replay Design: Scale-Stratified Herding

  • Scale-Stratified Herding allocates 75 of its 300 Anti-UAV-RGBT training exemplars to each of four size strata, then replays one exemplar per four CST samples.The random-stratified control keeps the same buffer size and quotas but selects randomly within strata, separating stratification from herding.Source: §3.5 · p. 5

3.6 Datasets

  • The curriculum datasets span distinct scale distributions: Anti-UAV-RGBT is dominated by normal targets, Anti-UAV410 covers a wider range, and CST is 87.9% tiny with 0% large targets.ARD100 is used only to pre-compute motion masks and is not part of the continual-learning curriculum.Source: §3.6 · p. 5

3.7 Evaluation Metrics; 3.8 Compute Infrastructure

  • Forgetting Measure uses the Anti-UAV-RGBT validation split as the Task 1 reference, while per-stratum mAP and cosine similarity characterize scale-specific performance and parameter drift.Training and evaluation used Snellius A100/H100 GPU partitions; on-board or edge deployment was not investigated.Source: §3.7 · p. 6, §3.8 · p. 6

4 Results

Across three thermal anti-UAV datasets, the scale distribution shifts sharply toward tiny targets in CST, where naive sequential training is associated with severe, stratum-specific forgetting. Scale-stratified replay mitigates that forgetting in single-seed results, while gradient probes support scale-conditioned imbalance as a candidate mechanism.

4.1 Scale-Distribution Shift Across Domains

  • Anti-UAV-RGBT is dominated by normal targets (67.8%), whereas CST is 87.9% tiny and 99.8% tiny or small, with modality held constant.Anti-UAV410 is intermediate, with 46.6% normal and 24.8% tiny targets.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 Catastrophic Forgetting Under Naive Sequential Training

  • FM reached −0.605 against the Stage 1 ceiling after naive CST fine-tuning, with −0.572 incurred in Stage 3 alone.T1 mAP@0.5 fell from 0.640 after Stage 2 to 0.068 at the best T3 checkpoint; the Stage 3 versus Stage 2 comparison is descriptive because Stage 2 used KD and Stage 3 did not.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Knowledge Distillation During Stage 2 Training

  • FM was −0.033 ± 0.004 across three Stage 2 seeds, corresponding to 95% retention of Stage 1 T1 mAP under KD training.Every T1 stratum was preserved through Stage 2, but the absence of a no-KD control prevents attributing the retention causally to KD.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Scale Shift and the Forgetting Pattern in Stage 3

  • On CST without replay, large-stratum T1 mAP@0.5 collapsed to 0.000 by epoch 1; separately, learnable-tensor inter-stage cosine similarity was higher at Stage 3 than Stage 2 (0.987 vs 0.935).Source: Figure 3 · p. 8, §4.4 · pp. 8–9

4.5 Direct Gradient Probe at the Stage 2 →3 Boundary

  • On the Stage 2 checkpoint, CST drove a box-regression loss of approximately 0.096 versus 0.020 on T1, while large targets received no positive reinforcement.The probe supports scale-conditioned gradient imbalance as a candidate mechanism; separating its causal contribution from batch-size and learning-rate differences requires a controlled rerun.Source: §4.5 · p. 10, Table 5 · p. 10

4.6 Scale-Stratified Replay and Stage 3 Forgetting

  • FM was −0.311 for SSH and −0.221 for random-stratified replay versus −0.605 for naive fine-tuning; both replay variants kept large-target T1 mAP non-zero.Random-stratified replay slightly exceeded SSH, suggesting the gain came from scale stratification rather than herding; both replay runs were single-seed.Source: §4.6 · pp. 10–11, Table 7 · p. 11

4.7 Tracking Baseline on CST Anti-UAV

  • The naive CST detector’s tracking baseline was SR@0.5 = 0.012, AUC-SR = 0.039, and PR@20 = 0.219 across 40 validation sequences.The gap between PR@20 and SR@0.5 suggests that target centres were often near the target while box sizing failed.Source: §4.7 · p. 11

4.8 Qualitative Detection Across Stages

  • On a single Anti-UAV-RGBT validation frame, the Stage 1 and Stage 2 models localised the target confidently, while the Stage 3 naive checkpoint produced no large-target detection.This example qualitatively matches the measured large-stratum erasure.Source: §4.8 · p. 11

5 Discussion

The discussion interprets Stage 3 forgetting as scale-conditioned gradient imbalance, while distinguishing it from weight drift and comparing it with earlier continual-learning results. It also qualifies causal attribution and generalisability, and presents scale-stratified replay as a targeted response.

5.1 Comparison with Prior Continual Learning Results

  • Stage 2 FM of −0.033 ± 0.004 and 95% T1 retention place the KD result in a range consistent with prior distillation-based continual-learning results.The study does not include a Stage 2 no-KD baseline, so it does not isolate how much retention is caused by KD.Source: §5.1 · p. 11, §5.3 · p. 13
  • The Stage 3 FM is 18 times the Stage 2 FM, a comparison consistent with stronger forgetting under the extreme scale inversion but not isolating its cause.Stage 2 uses KD, Stage 3 uses naive fine-tuning, and the stages also differ in optimisation settings.Source: §5.1 · p. 11, §5.3 · p. 13

5.2 Gradient Starvation as a Candidate Forgetting Mechanism

  • Learnable-weight cosine similarity was 0.987 in Stage 3 versus 0.935 in Stage 2, despite 18-fold greater forgetting in Stage 3.The analysis excludes BatchNorm running-statistic buffers from the learnable-weight comparison.Source: §5.2 · p. 11, §5.1 · p. 11
  • The proposed starvation account links small learnable-weight changes to weak gradient flow, while BatchNorm running statistics shift more during CST training.The reported relative L2 shift is approximately 0.24 for running statistics versus 0.12 for learnable weights.Source: §5.2 · pp. 11–12
  • Large-stratum mAP fell to 0.000 by epoch 1 while the tiny stratum was essentially unaffected, consistent with missing large-target reinforcement and dominant small-target gradients.The gradient probe describes the collapse as scale-conditioned gradient imbalance, refining the gradient-starvation interpretation.Source: §5.2 · p. 12

5.3 Limitations

  • The Stage 3 result may partly reflect optimisation differences: batch size 16 and learning rate 5 × 10−4 were used, versus 64 and 10−3 in Stage 2.The authors note that a Stage 3 rerun at Stage 2 settings would help isolate the effect.Source: §5.3 · p. 12
  • The conclusions are bounded by DJI-heavy semi-controlled data and the specific Anti-UAV410-to-CST scale inversion; transfer to other UAV types, detectors, or smaller scale gaps is unknown.Stage 3 and replay results are single-seed, and the replay comparison is preliminary.Source: §5.3 · pp. 12–13

5.4 Value of the Research and Alternative Interpretations

  • Scale-stratified replay roughly halves forgetting, and the ablation attributes the gain to balanced size-stratum coverage rather than the herding selection rule.The relative ordering of replay variants remains indicative rather than definitive because the comparison is single-seed and short.Source: §5.4 · p. 13, §5.3 · p. 13

6 Conclusion

Sequential evaluation across three thermal anti-UAV benchmarks finds substantial forgetting under naive fine-tuning, while scale-stratified replay reduces it. The findings point to scale-conditioned gradient imbalance as a candidate mechanism, with conclusions bounded by optimization and replication limitations.

6 Conclusion

  • FM = −0.605 against the Stage 1 ceiling, with −0.572 incurred in Stage 3 alone, leaves Task 1 detection capability at roughly 10% of its post-Stage-1 level.The study evaluates YOLOMG as a single thermal-infrared stream across Anti-UAV-RGBT, Anti-UAV410, and CST Anti-UAV.Source: §6 · p. 14
  • FM = −0.033± 0.004 across three seeds under Stage 2 KD training corresponds to 95% T1 retention, but no no-KD control isolates distillation’s causal benefit.This establishes retention under KD training, not a causal KD effect.Source: §6 · p. 14
  • Large-target mAP reaches 0.000 by Stage 3 epoch 1 despite 0.987 cosine similarity over gradient-updated weights, pointing to scale-conditioned gradient imbalance as a candidate mechanism.The interpretation is consistent with an absent positive gradient signal for the large stratum, rather than weight drift as the primary mechanism.Source: §6 · p. 14
  • SSH changes FM from −0.605 to −0.311 and keeps large-target T1 mAP non-zero at 0.079 versus 0.000; random-stratified replay performs at least as well at −0.221.The comparison indicates that scale stratification, rather than herding selection, accounts for the replay gain; Stage 3 and replay findings are single-seed and short-run.Source: §6 · p. 14
  • Stage 3 used a smaller batch size and lower learning rate than Stage 2, and the scale-inversion finding may not generalize beyond dataset pairs with large scale gaps.The Stage 3 and replay results use a single seed and short runs, making the narrow margin between replay variants indicative rather than definitive.Source: §6 · p. 14
  • The study provides a quantified forgetting protocol and calls for multi-seed, longer replay runs, comparison with regularization, and extension to detection-to-tracking.The authors suggest naive sequential training across operational datasets is unsuitable for deployed C-UAV systems (FM = −0.605, 90% capability loss).Source: §6 · p. 14

A Dataset Statistics

Figures 7 and 8 report UAV-present frame counts and target-visibility rates by dataset split.

A Dataset Statistics

  • Figure 7 reports UAV-present frame counts by split, while Figure 8 reports corresponding target-visibility rates.UAV-present means the exist=1 flag; visibility marks actively flying UAVs, with occluded or out-of-frame instances marked absent.Source: A Dataset Statistics · p. 15

B Experimental Configuration

The three training stages use the same YOLOMG architecture and Nesterov SGD with cosine learning-rate decay.

B Experimental Configuration

  • All stages use Nesterov SGD with cosine learning-rate decay and the same 318-layer YOLOMG architecture with 335 named parameter tensors and one UAV class.Table 8 summarizes the stage hyperparameters.Source: B Experimental Configuration · p. 15

C Additional Training Dynamics

The appendix points readers to tables of epoch-level Stage 2 and Stage 3 training dynamics and describes the motion-mask procedure and its use in this study. The configuration caption specifies stage seed counts and the epoch-indexing convention.

C Additional Training Dynamics

  • Table 9 reports seed-42 Stage 2 epoch dynamics, and Table 10 reports per-epoch Stage 3 naive dynamics.Both tables are discussed in the main text, in Sections 4.3 and 4.2, respectively.Source: C Additional Training Dynamics · p. 15

Motion Mask Generation (mask32)

  • Table 8 states that Stage 2 uses three seeds while Stages 1 and 3 use seed 42, and that epochs are 0-indexed.Epoch 0 is the first epoch.Source: Table 8 · p. 16
  • Stage 3 variants comprise naive training, SSH, and random-stratified replay; replay uses a 300-example buffer with 75 examples per stratum, a 25% replay ratio, and replay weight 4.0.The strata are tiny, small, normal, and large UAVs.Source: Motion Mask Generation (mask32) · p. 16
  • Figure 7 gives UAV-present frame counts by split, and Figure 8 gives target-visibility rates.Anti-UAV410 has 428,703 UAV-present frames, Anti-UAV-RGBT 293,209, CST Anti-UAV 208,221, and ARD100 199,291; ARD100 is used only for motion-mask pre-computation.Source: Figure 7 · p. 16, Figure 8 · p. 16
  • Stage 1 and the purely thermal datasets use a zero tensor for the motion channel, so YOLOMG functions as a single-stream thermal appearance detector.The motion masks were unavailable within the study’s scope for Anti-UAV-RGBT, Anti-UAV410, and CST Anti-UAV.Source: Motion Mask Generation (mask32) · p. 16
  • The described FD5 scheme uses frames t−2, t, and t+2 after ECC-based alignment to compute motion differences that retain object-relative motion and suppress camera shake.The five-frame sliding window is described alongside global motion compensation and frame-difference residuals.Source: Motion Mask Generation (mask32) · p. 16

Dataset Loading Strategy

The section describes native-format dataset loading, consistent target-scale strata, and the procedures used for stratified replay, distillation, and forgetting measurement.

Frame Extraction

  • Anti-UAV-RGBT infrared video is decoded on demand or read as pre-extracted JPEGs, while Anti-UAV410 and CST Anti-UAV frames are read directly as JPEGs.The paired visible stream is ignored, and no intermediate converted files are written.Source: Frame Extraction · p. 16, Dataset Loading Strategy · p. 16

Algorithm 2: Stage 2 Knowledge Distillation

  • Three-seed mean Stage 2 FM is −0.033 ± 0.004; the clean post-Stage-2 evaluation reports T1 mAP 0.640 and FM −0.033.The table notes a ∼0.004 gap between the training-time evaluation and the clean re-evaluation.Source: Table 9 · p. 17

Frame Extraction

  • Large-stratum T1 mAP reaches 0.000 at Stage 3 epoch 1 and never recovers; small and normal strata fall to 0.060 and 0.079 at the best checkpoint.The Stage 3 naive condition starts with pre-training T1 mAP 0.640; the T1 ceiling is 0.6725.Source: Table 10 · p. 17

Scale Stratification

Algorithm 1: Scale-Stratified Herding (SSH)

  • SSH partitions T1 reference data by target-size stratum before greedy herding, ensuring representation of all four scale categories regardless of their source frequency.This is the stated departure from standard iCaRL herding.Source: Algorithm 1: Scale-Stratified Herding… · p. 17

Algorithm 2: Stage 2 Knowledge Distillation

  • The Stage 2 teacher is initialized from the Stage 1 checkpoint and remains frozen; teacher and student receive the same infrared input, with ground-truth supervision added for the student.Source: Algorithm 2: Stage 2 Knowledge… · p. 17

Algorithm 3: Forgetting Measure Computation

F Stage 3 Tracking Evaluation

The Stage 3 best checkpoint is evaluated on CST Anti-UAV using one-pass tracking plots and sequence-level success rates, which expose variation across sequences.

Per-sequence success rate

  • Figure 11 shows substantial per-sequence variance in SR@0.5 that is not visible in aggregate mAP; sequences dominated by tiny targets have the lowest rates.The caption associates this pattern with near-zero tiny-stratum mAP.Source: F Stage 3 Tracking Evaluation · p. 18, Figure 11 · p. 18

Precision plot

  • Figure 9 plots the proportion of frames whose predicted centre falls within a pixel-distance threshold of the ground-truth centre.The representative PR@20 score is reported in the main tracking evaluation table.Source: F Stage 3 Tracking Evaluation · p. 18, Figure 9 · p. 18

Success plot

G Compute Environment

Training and analysis ran on the Snellius HPC cluster in a dedicated Python 3.9 Conda environment.

G Compute Environment

Library versions.

H Experimental Run Summary

The section points to a table summarizing the experimental runs underlying the reported results.

H Experimental Run Summary

Loading 2610.08315v1…