PaperlayerView exact revision on arXiv

Source-linked AI summary

ARMOR: Manifold-Oriented Training for Adversarially Robust Aerial Object Detection under Data Scarcity

Haoran Wang, Matthew Lau, Alec Helbling, Matthew Hull, ShengYun Peng, Mansi Phute, Martin Andreoni, Willian T. Lunardi, Duen Horng Chau, Wenke Lee

arXiv:2608.29510v1cs.CVcs.CRcs.LG

TL;DR

Aerial object detectors must resist physical universal patches despite scarce labeled deployment data. ARMOR adapts on-manifold adversarial training with label-based background masking and randomized object patches, maintaining over 0.90 clean model confidence and improving adversarial robustness by up to 0.32 in model confidence over state-of-the-art defenses.

  • Problem

    Aerial object detectors remain vulnerable to physically realizable universal patches, while deployment sites often lack enough labeled data for data-intensive robustness methods.

  • Method

    ARMOR uses bounding-box labels to mask backgrounds and insert randomized patches on objects, adapting OMAT’s feature-learning insights without extra annotation cost.

  • Results

    Over 0.90 clean model confidence is maintained, while adversarial robustness improves by up to 0.32 in model confidence over state-of-the-art defenses.

  • Takeaways & Limitations

    ARMOR translates manifold-based training insights into a data-efficient defense whose gains persist against physically printed patches in deployment.

Abstract

from arXiv · show

Aerial object detection is increasingly deployed in real-world applications, but models remain vulnerable to physical, universal adversarial patches that cause them to miss objects. Furthermore, defenders face the practical constraint of training data scarcity: aerial imagery is costly to collect and label, so a deployment site typically yields hundreds of images rather than the tens of thousands that adversarial robustness benchmarks assume. To tackle model vulnerability and training data scarcity, we propose Adversarial Robustness with Manifold-Oriented Training (ARMOR), a novel defense that realizes the core insights of on-manifold adversarial training (OMAT) in low-data regimes. ARMOR builds on the insight of OMAT to model the data manifold - the compact structure capturing the data's relevant features - to learn and robustify these features during training. While OMAT relies on the data-intensive operations of training large generative models and adversarial training to achieve this, ARMOR adopts a data-efficient approach that reuses labels the detection task already supplies: ARMOR (i) masks image backgrounds to retain object-relevant features, and (ii) injects randomized patches on objects to improve feature robustness. Our low-data experiments with physically-realizable adversarial patches evaluate both query-free transfer attacks and defense-aware attacks. ARMOR maintains strong clean performance of over 0.90 model confidence, while improving adversarial robustness by up to 0.32 in model confidence over state-of-the-art defenses. Physical experiments with printed patches confirm that these gains survive deployment. Overall, ARMOR translates insights from manifold-based training to defend object detectors amidst training data scarcity.

I. INTRODUCTION

ARMOR addresses physically realizable adversarial patches and scarce labeled aerial data by adapting on-manifold adversarial training with label-based image masking and randomized object patches. It maintains clean performance while improving adversarial robustness in low-data experiments.

  • Motivation: Aerial detectors face physically realizable universal patches and scarce site-specific labeled data, motivating robustness methods that preserve clean performance.Deployment sites may have only hundreds of labeled images, while aerial datasets can span hundreds to tens of thousands of higher-resolution images.
  • Related approach: OMAT targets relevant features through manifold projection and adversarial training, but its generative modeling and optimization requirements are impractical with little data.The manifold hypothesis treats realistic high-dimensional images as lying on a lower-dimensional structure capturing relevant features.
  • ARMOR: ARMOR replaces OMAT’s data-intensive operations with bounding-box background masking and randomized patches placed on objects during training.These operations reuse labels already supplied for detection, adding no labeling cost over standard fine-tuning.
  • ARMOR: ARMOR deliberately trains on masked images and patched objects even though test images contain backgrounds and lack object patches, while retaining clean accuracy and increasing attack robustness.The authors interpret this mitigation of robust overfitting as evidence of a data-efficient adaptation of OMAT’s insights.
  • Results: 0.90 model confidence is maintained on clean data, while adversarial robustness improves by up to 0.32 in model confidence over state-of-the-art defenses.Physical experiments with printed patches confirm that the gains hold in deployment.

II. RELATED WORKS

Prior defenses address adversarial patches, robustness, or certification, but often require extensive data, robust pretraining, costly inference, or assumptions mismatched to physically realizable aerial attacks.

  • Object Detection: Object detection localizes objects with bounding boxes; this work evaluates YOLOv3 and YOLOv11 in aerial imagery applications.The aerial setting supports monitoring of large areas for traffic and environmental applications.
  • Adversarial Defenses: Pre-processing defenses can detect or mask patches, but adaptive patches and dependence on segmentation models limit their reliability.Information-theoretic defenses also involve threshold estimation or segmentation assumptions.
  • Model Robustness: Standard augmentations improve robustness but leave adversarial performance 20% lower than clean performance, while robust detectors may require extensive data or sacrifice clean mAP.Low-data fine-tuning methods may additionally require a robust pretrained model unavailable for object detection.
  • On- and Off-Manifold Attacks: On-manifold attacks target failures to use relevant features, whereas off-manifold attacks target irrelevant features such as backgrounds.OMAT projects data onto the manifold before adversarial training, but both steps are difficult in low-data regimes.
  • Certified Robustness and Model Modification: Certified defenses remain uncommon for object detection because multiple or large patches can break them, noisy inference is impractical, or their threat model is limited to ℓp perturbations.Other defenses modify detector features, such as by clipping intermediate-layer feature norms.

III. PROBLEM SET-UP

The case study examines car detection from stationary aerial imagery, a setting motivated by common applications, limited training data, and realistic physical attacks.

  • Case Study: The study focuses on aerial car detection because traffic monitoring and self-driving applications commonly use this viewpoint and often face limited training data.The main experiments use a stationary aerial camera looking onto a street.

A. Threat Model: Realistic Evasion Attacks

The threat model uses a single printed patch placed on a car roof to evade aerial detection, evaluated under transfer and defense-aware attacker knowledge.

  • Attacker’s Goal: The attacker places one universal ON patch on the car roof to evade detection and does not consider partially hidden patches or cars.The patch is required to remain robust across environmental conditions.
  • Attacker’s Knowledge: Transfer attacks use a pretrained model without defense knowledge, while defense-aware attacks optimize directly against the defended model.Transfer attacks represent practical feasibility; defense-aware attacks stress-test robustness.
  • Attacker’s Capabilities: The attacker digitally optimizes patches, prints them, and affixes them to the car roof, without perturbing the rest of the scene or using test-time data.The perturbation is restricted to a small localized patch on the object.
  • ARMOR Training: ARMOR’s training diagram combines standard fine-tuning, background-masked images, randomized object patches, and hybrid training to mitigate trivial localization solutions.The final hybrid phase addresses behavior such as treating non-black regions as objects.
  • Attacker’s Strategy: Physical attack patches use non-printability, total-variation, and expectation-over-transformation losses to support printability, smoothness, and environmental robustness.Transfer and defense-aware patches are optimized for physical deployment conditions.

B. Defender Model

ARMOR adapts OMAT for low-data aerial detection by using bounding-box labels to isolate object-relevant features and improve their robustness. Its training phases also suppress false positives and address residual vulnerability that background removal alone cannot eliminate.

  • Method: ARMOR adapts OMAT by replacing generative modeling and adversarial training with label-driven background masking and randomized object patches.Both operations reuse bounding-box labels supplied by the detection task, adding no labeling cost over standard fine-tuning.
  • Manifold Projection: Background masking uses bounding-box labels to remove pixels outside objects, treating retained object pixels as the task-relevant manifold features.The transformation is described as projection-like and is idempotent when applied again to an already masked dataset.
  • Mitigating Potential Issues: ARMOR uses three fine-tuning phases: conventional adaptation, background-masked training, and hybrid training that combines masked and raw images.The hybrid phase regularizes against localizing objects in unmasked background regions, aiming to suppress false positives while preserving existing solutions.
  • Robust Feature Training: Background removal alone does not necessarily eliminate adversarial vulnerability, so ARMOR adds randomized object-level noise to address remaining on-manifold vulnerability without unstable adversarial training.The paper distinguishes background-related off-manifold vulnerability from residual on-manifold vulnerability addressed by OMAT’s second step.

B. On-Manifold Fine-Tuning

ARMOR addresses low-data training by modestly augmenting images and replacing unstable optimized perturbations with random patches applied to objects during fine-tuning. Its evaluation follows an established attack, model, and dataset setup.

  • Data Augmentation: ARMOR makes 10 copies of each image and randomly applies weather augmentations to enrich the low-data training set.The augmentation strategy is intentionally modest because background removal is expected to remove many irrelevant features.
  • Randomized Patches: Instead of optimizing adversarial noise through an unstable min-max objective, ARMOR adds uniform random noise during training.This resembles randomized smoothing, but the noise is used during training rather than at test time.
  • Evaluation Setup: The experiments benchmark ARMOR using attacks, models, and a dataset primarily drawn from prior work, including random, transfer, and defense-aware patches.The evaluation is organized around four questions covering effectiveness, mechanism, generalizability, and practical usefulness.

A. Evaluation and Metrics

The evaluation measures detector confidence and average precision across clean, random-patch, transfer, and defense-aware attack settings. ARMOR preserves strong clean performance and substantially outperforms other defenses against physically realizable attacks.

  • Metrics: Average model confidence measures patch-induced evasion across ground-truth objects, while average precision captures the trade-off among true positives, false positives, and false negatives.Successful attacks lower confidence; successful defenses mitigate that decrease.
  • Evaluation Setup: The test set contains 63 held-out images from an 843-image aerial video feed, with 780 images used for fine-tuning.The undefended YOLOv3-based model is initialized from pretraining on an aerial car dataset.
  • Main Results: Across defenses, clean confidence is generally 0.91–0.96, but ON-T reduces confidence to 0.31–0.48 and ON-DA to 0.14–0.30, except for ARMOR.These results indicate that the transferable ON attack preserves vulnerability across most defenses.
  • Baseline Comparisons: Randomized smoothing obtains 0.46 Conf and 0.72 AP against random ON patches, while SHIELD and PAD also have limited efficacy against physically realizable ON attacks.The paper attributes SHIELD’s weaker performance to lower-frequency patches and notes that PAD-S localizes only one of two patches.
  • Ablation: The ablation reports that random patches improve confidence under ON-T and ON-DA, while hybrid fine-tuning after background masking mitigates false positives.Background-masked training alone trades slight clean performance for stronger ON-DA confidence but has poor AP.
  • Main Results: ARMOR achieves 0.92 clean Conf, a 0.05 Conf drop under random ON patches, 0.73 Conf against ON-T, and 0.62 Conf against ON-DA.Against ON-DA, ARMOR reaches 0.62 Conf versus 0.14–0.30 Conf for other defenses, an improvement of 0.32.

1) Ablations:

The ablations show that random object patches and background removal improve adversarial robustness under scarce-data training, while manifold metrics indicate more compact and consistent ARMOR representations.

  • Standard Fine-Tuning and Augmentations: Fine-tuning raises clean Conf from 0.83 to 0.89 and ON-T from 0.14 to 0.18, while weather augmentation raises them to 0.96 and 0.36.ON-DA remains poor at 0.11–0.14 Conf, similar to the undefended model.
  • Random Patch Augmentation: Random ON patches raise ON-DA Conf from 0.14 to 0.25, whereas weather augmentation alone reaches only 0.14.The patches also improve clean and ON-T performance despite clean test images containing no patches.
  • Background Removal: Background-masked fine-tuning raises ON-DA Conf from 0.25 to 0.59, while hybrid fine-tuning alone lowers it to 0.49.Adding hybrid fine-tuning after background masking raises ON-DA Conf slightly to 0.62 and mitigates false positives.
  • Manifold Analysis: ARMOR reduces intrinsic dimensionality by 5.7%, from 1.74±0.57 to 1.64±0.54, suggesting representations focus more on car-relevant features.The analysis compares ARMOR with the undefended model using penultimate-layer features.
  • Manifold Analysis: ARMOR lowers local-PCA reconstruction error from 0.41±0.14 to 0.37±0.06 and variance from 0.14 to 0.06.These results indicate a more structured, locally linear, and consistent manifold.
  • Manifold Analysis: Both models exceed 0.90 mean tangent-space alignment, but ARMOR has 6× higher variance at 0.90±0.24 versus 0.96±0.04.The undefended model is more tightly aligned, allowing attacks to exploit the same directions more reliably.

D. Generalizability of ARMOR

ARMOR generalizes across weather corruption, a different drone-imagery setting, and a smaller detector, obtaining the best confidence against both attack types in every setting.

  • Weather Corruptions in Test Data: Under weather corruption, ARMOR retains 0.61 Conf against ON-T and 0.54 against ON-DA.The attacks reduce undefended and fine-tuned confidence more substantially, to 0.23 and 0.34–0.35 respectively.
  • Different Setting: Drone Imagery: The drone-imagery evaluation uses a moving carpark scene with constantly changing backgrounds and only 541 images.The passage reports that ON-T lowers undefended and fine-tuned confidence to 0.04 and 0.55, while ON-DA decreases it further.
  • Different Model: The cross-model test evaluates YOLOv11n, a 5 MB detector with 3M parameters, using a patch trained on YOLOv3.YOLOv3 has 241 MB and over 60M parameters in this comparison.

1) Stress-Test with Other Attacks:

ARMOR is stress-tested against modified defense-aware attacks and OFF patches, with the reported physical ON-T figure showing detection where undefended and fine-tuned models fail.

  • Physical ON-T Attack: ARMOR detects cars with average confidence 0.75 under the realistic physical ON-T attack, while undefended and fine-tuned models do not detect the car.This result is shown in Fig. 5.
  • OFF Patches: Under OFF-T patches, ARMOR reaches 0.78 Conf versus 0.73 for ON-T, while other defenses obtain 0.21–0.66 Conf.Defense-aware OFF patches are over 600 times more expensive to generate and are limited to stationary cars.

E. Evaluation on Realistic Physical Attacks

Realistic physical evaluations constrain attacks to printed patches affixed to car roofs and test both deployment conditions and motion blur. ARMOR remains robust, but motion blur exposes a clean-performance trade-off.

  • Physical Attack Evaluation: ARMOR obtains the best Conf under realistic physical attacks in both the original sidestreet setting and motion blur.Table IV evaluates printed patches affixed to car roofs in these two settings.
  • Motion Blur: With motion blur, undefended and fine-tuned models fall to 0.11 and 0.23 Conf against ON-T, while ARMOR retains 0.59 Conf.The experiment uses palm-sized physical replicas and reuses models fine-tuned on the original sidestreet dataset.
  • Limitations and Extensions: ARMOR’s motion-blur clean performance trails the baselines, which the authors attribute to object-semantic distribution shift.ARMOR trains on object-relevant features from cars without motion blur, while test cars exhibit motion blur.
  • Limitations and Extensions: ARMOR requires bounding box labels at fine-tuning time, excluding fully unlabeled adaptation.Bounding box labels remain standard for detection, but this is a stated scope boundary.
  • Conclusion: ARMOR uses background masking and random object patches, drawing both operations from existing bounding-box labels without extra annotation cost.These steps target relevant, robust features while reducing data demands.
Loading 2608.29510v1…