PaperlayerView exact revision on arXiv

Source-linked AI summary

DroneGround: Open-Vocabulary Drone Payload Characterization Using Synthetic Data and Grounded Vision-Language Models

Ami Pandat, Rajasekhar Punna, Gopika Vinod, Rohit Shukla

arXiv:2609.07780v2cs.CV

TL;DR

Payload characterization is difficult because annotated real-world data is scarce and closed-set systems cannot recognize unseen payloads. DroneGround combines synthetic data with drone localization and grounded vision-language description. Experiments report better performance than closed-set detectors and full-image VLM inference, including improved generalization to unseen payload categories.

  • Problem

    Payload characterization is constrained by scarce diverse annotated real-world data and closed-set approaches that cannot recognize previously unseen payloads.

    Source: §1 · p. 2

  • Method

    DroneGround pairs a photorealistic synthetic dataset with drone localization and a LoRA-fine-tuned PaliGemma model that generates grounded, open-vocabulary payload descriptions.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Experiments on synthetic and real-world imagery show DroneGround outperforming conventional closed-set payload detectors and full-image VLM inference, with improved generalization to unseen categories.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    DroneGround provides visual evidence by localizing payload-relevant regions, while temporal reasoning and edge deployment remain future-work directions.

    Source: §7 · pp. 9–10

    Performance is challenged by tiny, motion-blurred, or heavily occluded payloads, and errors can propagate from unsuccessful drone localization.

    Source: §6.4 · p. 9

Abstract

from arXiv · show

Automated drone surveillance has become increasingly important for public safety, critical infrastructure protection,and restricted airspace monitoring. While existing vision-based systems achieve strong performance for drone detection and tracking, reliable payload characterization remains highly challenging under long-range imaging conditions due to limited availability of annotated real-world datasets, and substantial distribution shifts encountered during deployment. Existing approaches formulate payload characterization as a closed-set object detection problem, limiting their ability to recognize previously unseen payloads and generalize beyond the training distribution. To address these challenges, we generate a photorealistic synthetic drone-payload dataset using Unreal Engine 5 and Cosys-AirSim and propose DroneGround: Grounded Vision-Language Payload Characterization, a two-stage framework for robust open-vocabulary payload analysis. DroneGround first employs a YOLO26s detector to localize drones and extract drone-centric image crops, followed by a LoRA-fine-tuned PaliGemma vision-language model that generates seman- tic descriptions of the detected drones and their attached payloads, enabling open-vocabulary payload characterization beyond predefined categories. An occlusion-based grounding module further provides interpretable payload localization by identifying image regions responsible for the generated descriptions. Extensive experiments on both synthetic and real-world drone imagery demonstrate that DroneGround substantially improves robustness under synthetic-to-real distribution shifts, outperforming a conventional closed-set payload detector by improving the F1-score from 82.5% to 96.3%, while achieving significantly better generalization to previously unseen payload categories (80.4%versus 42.7% F1). Dataset and code will be released upon request.

1. Introduction

DroneGround addresses the scarcity of diverse real-world payload data and the limits of closed-set recognition with synthetic data and grounded vision-language generation. Experiments report stronger performance than closed-set detectors and full-image VLM inference, including generalization to unseen payloads.

  • Closed-set systems struggle when payload appearance, distance, illumination, and attachment configurations differ from training data, and cannot recognize unseen payloads.Source: §1 · pp. 1–2
  • Vision-language models can describe visual observations beyond fixed labels, but adapting them to payload characterization is constrained by scarce domain-specific training data.Collecting annotated payload imagery across platforms, viewpoints, environments, and payload types is expensive and often impractical.Source: §1 · p. 2
  • DroneGround localizes drones, crops them, and uses a LoRA-fine-tuned PaliGemma model to generate semantic descriptions beyond predefined payload categories.The framework decouples drone localization from payload characterization, allowing integration with existing detection systems.Source: §1 · p. 2
  • Experiments on synthetic and real-world imagery show DroneGround outperforming closed-set payload detectors and full-image VLM inference, with improved generalization to unseen categories.Source: §1 · p. 2
  • The authors develop a photorealistic synthetic drone-payload dataset to address the scarcity of diverse annotated real-world data.Source: §1 · p. 2
  • An occlusion-based grounding mechanism produces payload-localization heatmaps as visual evidence for semantic predictions.Source: §1 · p. 2

2. Related Work

Related work spans drone sensing and detection, payload analysis, vision-language reasoning, and visual explanation. The paper identifies limits in closed-set payload tasks and a gap in applying open-vocabulary reasoning and grounding to drone payload analysis.

2. Related Work

  • RF, radar, LiDAR, and acoustic systems support drone detection and tracking but offer less detailed payload characterization than vision.Source: §2 · p. 2

2.1. Drone Detection, Payload Characterization, and Datasets

  • Prior detection methods use closed-set labels, while payload studies often classify predefined categories or loaded-versus-unloaded status; existing synthetic benchmarks omit attached payloads, unlike this dataset.The described dataset includes payload-equipped drones, bird confounders, varied environments, and annotations for drone and payload localization.Source: §2.1 · p. 3

2.2. Vision-Language Models and Open-Vocabulary Reasoning

  • Vision-language models such as PaliGemma align visual representations with language, enabling reasoning beyond fixed categories, but drone-payload applications remain largely unexplored.Source: §2.2 · p. 3

2.3. Explainability and Visual Grounding

  • Existing explanations highlight image regions contributing to predictions, and vision-language grounding links text tokens to visual evidence; these approaches have seen limited investigation in drone payload analysis.Source: §2.3 · p. 3

3. Data Generation

The dataset is generated from assembled drone–payload models rendered in varied simulated scenes. Randomized payload configurations and bird confounders add variation relevant to drone detection and payload-aware evaluation.

  • Drone and payload models are assembled using 3D hooks, then rendered in Unreal Engine 5 and Cosys-AirSim across diverse operational conditions.Payload type, orientation, and attachment configuration are randomized across scenes.Source: §3 · pp. 3–4
  • The dataset includes drones carrying bags, boxes, and guns alongside bird models used as aerial confounders.Individual birds are annotated, while Niagara-generated flocks are left unlabeled to add background motion and clutter.Source: Figure 1 · p. 4, §3 · p. 4
  • The dataset combines labeled birds and payload-equipped drones, a pairing the authors say existing synthetic drone-detection datasets do not provide.Scenes span urban, suburban, park, rural, and forested environments.Source: §3 · p. 4

Multi-Camera Configuration

Six synchronized cameras provide panoramic coverage across varied drone viewpoints, and the cleaned dataset spans multiple platforms, payloads, environments, and weather conditions.

Multi-Camera Configuration

  • Six synchronized cameras, each with a 60° horizontal field of view, are uniformly distributed around a rigid rig for full 360° panoramic coverage.Drone motion and camera placement vary to capture diverse viewpoints.Source: Multi-Camera Configuration · p. 4
  • Drone trajectories follow randomized circular and spiral paths to increase viewpoint diversity.Source: Multi-Camera Configuration · p. 4
  • Rendered frames are retained when the drone bounding-box width ranges from 5 pixels to 20% of the image width, preserving varied target scales.This filtering removes frames with extremely small or excessively large targets while keeping drones sufficiently visible for annotation and evaluation.Source: Multi-Camera Configuration · p. 4
  • Approximately 100,000 annotated RGB images span 15 drone platforms, 6 payload categories, 5 bird species, 7 virtual environments, and 5 weather conditions.The dataset is intended as a benchmark for drone detection and open-vocabulary payload characterization under synthetic-to-real distribution shifts.Source: Multi-Camera Configuration · p. 4

4. Proposed Method

DroneGround localizes drones, analyzes expanded drone-centric crops with a LoRA-fine-tuned vision-language model, and parses semantic descriptions for payload presence and type. An occlusion-based module produces heatmaps of image evidence contributing to payload-token generation.

4. Proposed Method

  • DroneGround localizes drones, sends their crops to fine-tuned PaliGemma for semantic descriptions, then parses them for payload presence and type.The modular design invokes the vision-language stage only for detected drone crops requiring payload characterization.Source: §4 · p. 5, Figure 4 · p. 5

4.1. Drone-Centric Payload Characterization

  • YOLO26s-guided crops address reduced payload visibility after full-frame resizing, while crop expansion preserves suspended payload structures.The expanded crop is resized and passed to PaliGemma for payload-aware caption generation.Source: §4.1 · pp. 5–6

4.2. Vision-Language Fine-Tuning

  • LoRA freezes the pretrained PaliGemma backbone and adds trainable low-rank matrices to attention layers; fine-tuning uses image-caption pairs rather than payload-level boxes.Captions specify payload presence and, when applicable, payload type; keyword matching parses the generated descriptions at inference.Source: §4.2 · p. 6

4.3. Occlusion-Based Payload Grounding

  • Payload-token confidence guides occlusion analysis: local patches are masked, confidence drops are accumulated and normalized into a heatmap of supporting visual evidence.Payload-specific tokens are extracted from the generated caption and their prediction confidence is used as the reference score.Source: §4.3 · p. 6

5. Experimental Setup

The experiments train a drone-localization stage and a fine-tuned vision-language stage using synthetic and real drone imagery. They compare DroneGround with closed-set and vision-language baselines and describe the setup’s latency and grounding procedure.

5. Experimental Setup

  • Stage 1 trains YOLO26s on about 45,000 synthetic images and 10,000 real images to detect drones and birds, then uses it to extract drone crops.The detector is used for localization and cropping, not payload characterization.Source: §5 · p. 6, §5.1 · p. 7
  • Stage 2 fine-tunes PaliGemma with LoRA on about 5,500 image-description pairs covering payload presence and type, without payload-level bounding-box supervision.The pairs comprise about 5,200 synthetic crops and 250 real drone images.Source: §5 · p. 6

5.1. Baselines and Implementation Details

  • Occlusion-based grounding generates a payload-aware caption, selects a payload-related token, and occludes image patches to build a heatmap from changes in token confidence.The heatmap is normalized and overlaid on the drone crop for visualization.Source: §5.1 · p. 7
  • The closed-set YOLO26s baseline predicts six payload categories plus no_payload; gun-related samples are held out to evaluate unseen-category generalization for both approaches.The same unseen-payload protocol is applied to the closed-set baseline.Source: §5.1 · p. 7
  • DroneGround’s end-to-end inference latency is about 22 ms per frame (45 FPS), with 10 ms for localization and 12 ms for vision-language inference.The vision-language stage runs only on detected drone crops.Source: §5.1 · p. 7

6. Experimental Results

DroneGround improves payload characterization over the closed-set detector, with gains from drone-centric cropping and fine-tuning. The experiments also assess localization, grounding, unseen-payload generalization, and failure conditions.

6.1. Drone Detection Performance

  • DroneGround’s detector achieves 98.3% drone mAP50, supporting localization before semantic payload reasoning.The detector recognizes drone and bird classes.Source: §6.1 · p. 7

6.2. Payload Characterization Performance

  • Drone-centric crops raise PaliGemma’s F1 from 88.2% to 91.6%; with fine-tuning, DroneGround reaches 92.8% on full images and 96.3% on crops.The reported comparison indicates that cropping and task-specific adaptation each contribute to the stronger results.Source: §6.2 · p. 8

6.2.1. E

  • With β = 0.6, α = 0.3 yields the highest reported accuracy, 93.89%, balancing payload visibility against background suppression.Smaller expansions can truncate suspended payloads, while larger ones add background content that can interfere with reasoning.Source: §6.2.1 · p. 8

6.3. Grounding Evaluation

  • 77.15% Pointing Game accuracy, or 2,700 successful localizations out of 3,500 images, is achieved without explicit localization supervision.Evaluation uses manually annotated payload boxes in held-out synthetic and real-world images.Source: §6.3 · p. 8
  • Grounding heatmaps focus on payload regions in qualitative examples, and the model identifies payload-free drones without spurious payload predictions.Figure 5 pairs cropped drone inputs with token-specific heatmaps highlighting payload-relevant visual evidence.Source: §6.3 · p. 8, Figure 5 · p. 9

6.4. Generalization to Unseen Payloads

  • DroneGround substantially outperforms the closed-set detector on previously unseen payload categories, generalizing through semantic reasoning and describing them with generalized expressions even when the exact category was not observed during training.Source: §6.4 · pp. 8–9

6.2. Payload Characterization Performance

  • 96.3% F1 for DroneGround with detector-guided cropping exceeds the closed-set YOLO26s detector’s 82.5% F1 by 13.8 percentage points on 3,500 synthetic and real-world test images.The closed-set detector’s precision is 97.1% and recall is 71.7%.Source: §6.2 · p. 8, Table 3 · p. 9

6.4. Generalization to Unseen Payloads

  • Failures primarily involve extremely small, motion-blurred, or drone-occluded payloads; false Stage-1 detections can also propagate to vision-language reasoning.Under severe long-range conditions, payload regions may occupy only a few pixels, making semantic reasoning difficult even after cropping.Source: §6.4 · p. 9

7. Conclusion

The conclusion presents DroneGround as a two-stage framework combining synthetic data, drone localization, vision-language characterization, and occlusion-based grounding. It reports advantages over closed-set detectors and identifies directions for future work.

7. Conclusion

  • DroneGround combines a synthetic drone-payload dataset, YOLO26s localization, fine-tuned PaliGemma image captioning, and occlusion-based grounding without explicit localization supervision.The dataset includes diverse drone models, payload configurations, environments, and bird confounders.Source: §7 · p. 9
  • The experiments report that DroneGround outperforms conventional closed-set payload detectors and generalizes better to previously unseen payload categories.Source: §7 · pp. 9–10
  • Future work will investigate larger vision-language models, temporal reasoning over video sequences, and edge-platform deployment for real-time payload-aware surveillance.Source: §7 · p. 10
Loading 2609.07780v2…