Comparative Study of Out-of-the-Box Technology for Automatic Target Detection and Recognition

Alma M. Liezenga; Lotte Nijskens; Henrik R. Baumann; Stefan Becker; Simon Bensberg; Niccolò Camarlinghi; Håvard R. Eiring; Alexander W. Johnsgaard; Tanel Liiv; Giuseppe Martino; Matteo Marturini; Matthias Rapp; Jan Erik van Woerden; Alexander Wolpert; Hugo J. Kuijf

Source-linked AI summary

Comparative Study of Out-of-the-Box Technology for Automatic Target Detection and Recognition

Alma M. Liezenga, Lotte Nijskens, Henrik R. Baumann, Stefan Becker, Simon Bensberg, Niccolò Camarlinghi, Håvard R. Eiring, Alexander W. Johnsgaard, Tanel Liiv, Giuseppe Martino, Matteo Marturini, Matthias Rapp, Jan Erik van Woerden, Alexander Wolpert, Hugo J. Kuijf

arXiv:2608.17917v1Aug 18, 2026cs.CVcs.AI

TL;DR

Military ATD/R remains difficult because models trained on civilian data often generalize poorly to military imagery. This study benchmarks YOLO and DETR models on military data, finding larger and DETR-based models promising, while small A2G targets remain challenging and in-domain training remains crucial.

Problem
Models trained on publicly available civilian datasets often generalize poorly to military imagery, making military vehicle ATD/R a persistent challenge.
Method
The study evaluates six YOLO iterations and two DETR variants on the STORE military dataset, with additional fine-tuning on the civilian VisDrone A2G dataset.
Results
Larger models generally outperform smaller ones, DETR-based models perform promisingly, and fine-tuning improves A2G performance while all models struggle with small A2G targets.
Takeaways & Limitations
Despite advances in architectures and pretraining, in-domain training remains crucial for developing capable military ATD/R systems.
Takeaways & Limitations
Aggregating fine-grained military vehicle classes into a coarse vehicle class limits applicability because human operators must assess detected threats.

Abstract

from arXiv · show

Automatic Target Detection and Recognition (ATD/R) is critical for military decision support and (semi-)autonomous operations. Recent advances in object detection and artificial intelligence (AI) significantly boosted the potential performance of ATD/R. However, the scarcity of publicly available military datasets limits the application of these systems. As a solution, this paper explores the use of publicly available models and civilian datasets to achieve reasonable performance in military contexts. We benchmark several state-of-the-art models, including six iterations of the YOLO series and two variations on the DETR framework, on a newly acquired military relevant dataset. This dataset features military vehicles and challenging circumstances, including various degrees of occlusions and small targets. The out-of-the-box version of each model is validated alongside a version finetuned on the VisDrone dataset. This dataset features small objects, an Air-to-Ground (A2G) perspective and relevant classes, potentially generalizing to our military ATD/R task. We compare the performance of the models using mAP@0.5 and mAP@0.5:0.95, across A2G and Ground-to-Ground (G2G) perspective, target size and model size, giving insight into the real-time capabilities of models. Our main findings are: (1) bigger models outperform smaller models, (2) DETR-based models show promising results compared to the YOLO series,(3) fine-tuning models on an out-of-domain A2G dataset, improves their A2G performance and slightly improves their performance on small objects, but (4) all models still struggle with detecting small objects in an A2G scenario. We conclude that, despite recent advances in object detection, in-domain training is still crucial for creating capable ATD/R systems.

I. INTRODUCTION · II. METHOD · A. Automatic Target Detection and Recognition

The study examines whether publicly available object-detection models and civilian data can support military Automatic Target Detection and Recognition (ATD/R). It evaluates YOLO and DETR models on a new military-relevant dataset across operational perspectives, object sizes, and model sizes, including fine-tuning on VisDrone.

I. INTRODUCTION: AI and deep learning have advanced image and video object detection, but models trained on civilian datasets often generalize poorly to military use-cases.This out-of-domain limitation motivates evaluating publicly available models for military ATD/R.
I. INTRODUCTION: Potential responses to limited military data include acquiring domain datasets, applying zero-shot or few-shot strategies, and generating synthetic imagery.Publicly available techniques, including DETR, have demonstrated few- and zero-shot capabilities.
I. INTRODUCTION: The study uses the newly acquired STORE military dataset for an operationally representative and objective quantitative evaluation.The dataset provides the basis for assessing publicly available detection methods in a military-relevant setting.
I. INTRODUCTION: Because the dataset was unavailable outside consortium partners, it was not used to pretrain publicly available models.The evaluation covers six YOLO iterations and two DETR variants in A2G and G2G scenarios, object sizes, and model sizes.
II. METHOD: The method investigates how publicly available datasets and models perform on a military ATD/R task.The experiments cover the task definition, datasets, models, and evaluation procedures.
A. Automatic Target Detection and Recognition: The ATD/R task detects and recognizes persons and vehicles in military A2G and G2G settings.The vehicle category combines civilian cars and buses with military tanks and trucks to support evaluation of models trained on civilian data.
A. Automatic Target Detection and Recognition: Evaluation uses a newly acquired military-relevant dataset containing occlusions and small objects.These conditions are included in both the dataset design and the assessment of detection performance.

B. Datasets

The study uses six datasets spanning common pretraining resources, VisDrone for fine-tuning, and the military STORE dataset for evaluation. STORE contains annotated military and civilian vehicles and personnel across airborne and ground perspectives under varied operational conditions.

Dataset overview: Six datasets supported the study: MS COCO, Object365, GoldG, CC3M-Lite, VisDrone, and STORE.Most models were pretrained on the first four datasets, fine-tuned on VisDrone, and evaluated on STORE.
VisDrone: VisDrone contains 10,209 images, 263 videos, and 2.5 million annotations across ten classes, with varied weather, viewpoints, and altitude from an A2G perspective.The official training split was used for fine-tuning, while the validation split selected the best checkpoint.
STORE: STORE contains 3,316 frames from 80 videos and 10,683 annotations for person and vehicle classes, including military and civilian vehicles.The annotations comprise 5,414 persons and 5,269 vehicles.
STORE: STORE combines drone, stationary, and moving-ground-vehicle recordings from A2G and G2G perspectives in snowy and rainy conditions with occasional tree or rainfall occlusion.Fontevraud contributed 654 A2G frames from 13 videos, while Kongsberg contributed 2,662 G2G frames from 67 videos.

C. Models

The study evaluates state-of-the-art object detectors using both out-of-the-box class mapping and VisDrone fine-tuning. It includes multiple YOLO variants and DETR-based models spanning real-time, open-vocabulary, and localization-focused designs.

Model preparation: Models were evaluated either out of the box, mapping original classes to “person” and “vehicle,” or after VisDrone fine-tuning with default hyperparameters.The model overview also records each version’s original pretraining datasets.
YOLO models: YOLO-World adapts the YOLO series for open-vocabulary detection by combining large-scale pretraining, CLIP text embeddings, YOLOv8 image features, and region-text fusion.Its architecture adds RepVL-PAN and a region-text contrastive loss.
DETR models: DETR formulates detection as direct set prediction and uses attention to exploit object context, although its original version generally performed poorly on small objects.DETR provides an efficiency improvement and an NMS-free alternative to YOLO detectors.
DETR models: RT-DETR reduces DETR’s computational cost with a hybrid multi-scale encoder, while D-FINE refines localization through probability distributions and self-distillation.D-FINE’s components are Fine-grained Distribution Refinement and Global Optimal Localization Self-Distillation.

III. EXPERIMENTS AND RESULTS

The experiments evaluated out-of-the-box and fine-tuned models on the full STORE dataset using COCO average precision across IoU thresholds, perspectives, object sizes, and model parameter counts.

Inference was run on both out-of-the-box and fine-tuned models across the full STORE dataset.
Performance was evaluated using COCO %AP at IoU thresholds of 0.5 and 0.5:0.95.
Results were separated by A2G and G2G perspectives and by standard COCO object-size categories: small, medium, and large.
Model parameter counts were reported to compare models and indicate their real-time capability.

A. Parameters vs. performance

Across models, larger parameter counts generally correspond to better out-of-the-box and fine-tuned performance, while DETR-based models and YOLO-World outperform other YOLO models. Fine-tuning slightly reduces YOLO-based performance but leaves DETR-based performance relatively stable.

Parameters vs. performance: Higher parameter counts correspond to better performance across out-of-the-box and fine-tuned models.This trend changes when considering only A2G images, discussed in Section B.
Parameters vs. performance: DETR-based models and YOLO-World outperform the other YOLO-series models out of the box.
Parameters vs. performance: After fine-tuning, YOLO-based performance decreases while DETR-based performance remains relatively constant.The difference between the model families becomes stronger for fine-tuned versions.

B. G2G vs. A2G performance

Fine-tuning on VisDrone improved A2G performance but harmed G2G performance, making out-of-the-box models better overall. RT-DETR achieved the highest performance across categories, approaching 30% AP in both A2G and G2G.

A2G performance: Fine-tuned models consistently outperformed out-of-the-box models on A2G data across model types.The improvement was observed when evaluating A2G scenarios after fine-tuning on VisDrone.
G2G performance: Out-of-the-box models performed better than fine-tuned models in the G2G scenario.Fine-tuning on VisDrone harmed G2G performance.
Overall comparison: Fine-tuned models performed worse overall because their G2G performance losses outweighed their A2G gains.The passage states that out-of-the-box versions perform better overall as a consequence of their stronger G2G performance.
Best-performing model: RT-DETR reached the highest performance across all categories, approaching 30% AP for both A2G and G2G.This result was reported across the evaluated models and perspectives.

C. Performance across object sizes

Performance varied by object size: A2G imagery was more challenging than G2G, and fine-tuning generally helped small-object detection while different models led for small, medium, and large targets.

Perspective and object size: A2G imagery was more challenging than G2G imagery, at least partly because it contained smaller objects.The paper evaluates performance across object sizes using small, medium, and large target categories.
Fine-tuning effects: VisDrone fine-tuning slightly improved small-object performance over out-of-the-box models, except for YOLO11, YOLO-World, and D-FINE DETR.The fine-tuning advantage was less pronounced than the A2G-versus-G2G pattern and did not consistently extend to medium or large objects.
Best-performing models: RT-DETR performed best overall and for small objects, YOLO-World performed best for medium-sized objects, and YOLOv5 achieved the highest score for large objects.The strongest small-object result came from a VisDrone fine-tuned model, whereas the strongest larger-object results came from an out-of-the-box model.

D. Worst and best performing tasks

Models performed best on detecting large persons and worst on detecting small objects from an A2G perspective. Large-person detection exceeded 60% AP@0.5:0.95, whereas no model exceeded 10% for small A2G objects.

Best-performing task: Over 60% AP@0.5:0.95 was achieved only for detecting large persons, with all out-of-the-box models reaching at least 58%.Performance was better from a G2G perspective than an A2G perspective.
Performance analysis: Model-parameter plots compared AP@0.5:0.95 and AP@0.5 across out-of-the-box and VisDrone-finetuned models, excluding YOLOv10 l because of inconsistent results.The plots relate model parameter count to performance metrics.

IV. CONCLUSION AND DISCUSSION · A. Discussion

The study finds that larger models and DETR-based architectures, particularly after VisDrone fine-tuning, can improve military ATD/R performance, but small A2G targets remain difficult. The authors nevertheless conclude that context-dependent evaluation and in-domain training remain essential.

IV. CONCLUSION AND DISCUSSION: The study evaluated six YOLO iterations and two DETR variants, both out of the box and fine-tuned on the civilian VisDrone dataset, using a military operational dataset.This tested whether publicly available models and civilian data could achieve reasonable military-context performance.
IV. CONCLUSION AND DISCUSSION: Larger models outperformed smaller models, while DETR-based models showed promising results compared with the YOLO series, especially after fine-tuning.RT-DETR and D-FINE DETR outperformed the YOLO series after VisDrone fine-tuning, and YOLO-World also performed relatively well.
IV. CONCLUSION AND DISCUSSION: VisDrone fine-tuning improved performance in A2G settings and, to a lesser extent, on small objects, even though the dataset came from another domain.The authors associate DETR’s results with generalization ability and, for some models, broader or more varied pretraining.
IV. CONCLUSION AND DISCUSSION: All evaluated models still struggled to detect small objects in A2G scenarios despite the benefits of fine-tuning.This limitation persists across the model families and is central to the paper’s military ATD/R findings.
A. Discussion: VisDrone fine-tuning produced modest A2G improvements but harmed G2G performance, potentially because VisDrone is civilian and pretrained models already contain substantial G2G imagery.The proposed explanation contrasts out-of-domain adaptation with the G2G capabilities learned from existing pretraining datasets.
A. Discussion: What constitutes reasonable performance in a military operational context remains an open, highly context-dependent question.Therefore, model evaluation results cannot be interpreted independently of the operational setting and task requirements.
A. Discussion: Some poor evaluation results reflected annotation misalignment around vehicle barrels, which reduced IoU even when the vehicle itself was detected.The discussion questions whether this localization discrepancy is operationally important when detection of the vehicle is the primary concern.
A. Discussion: Despite promising results from larger pretraining datasets and architectures such as DETR, in-domain training remains crucial for military ATD/R systems.The conclusion treats domain-specific training as necessary even when model architecture and pretraining scale improve performance.

B. Limitations

The evaluation is limited by excluded large D-FINE DETR models, an imbalanced dataset design, coarse vehicle classes, and a restricted metric set. These constraints reduce robustness, applicability, and operational relevance, although the current metrics still provide a comprehensive overview.

Model coverage: Larger D-FINE DETR versions were excluded because their training costs were computationally prohibitive, despite the model’s promising results.The study nevertheless demonstrated the added value of D-FINE DETR.
Dataset balance: The STORE dataset selection is unbalanced because all A2G images came from Fontevraud and all G2G imagery from Kongsberg.Mixing both perspectives across locations would have enabled more robust results and an ablation study of perspective and location.
Class granularity: Combining VisDrone and STORE required aggregating fine-grained military vehicle classes into the coarse ‘vehicle’ class, limiting model applicability.A human operator would therefore need to assess the threat level associated with detected vehicles or people; this limitation was accepted because the models were not intended for operational use.
Evaluation metrics: Metrics using a lower IoU threshold might have shown higher and more operationally relevant performance, but the current metrics still provide a comprehensive overview.The metrics were retained for simplicity and to indicate the feasibility of open-source models and datasets in a military use case.

C. Future work

Future work should prioritize in-domain training for military ATD/R, especially A2G scenarios and small-object detection, while expanding evaluation metrics and dataset availability. Proposed directions include fine-tuning promising detector families and generating synthetic training data.

C. Future work: Future studies will incorporate in-domain training within EDF STORE, particularly for A2G scenarios and small objects.The paper emphasizes fine-tuning DETR-based models and YOLO-World, especially D-FINE DETR and its larger versions.
C. Future work: Future evaluations should measure gains from specific training strategies using a wider, more representative set of military ATD/R metrics.The proposed metrics should align particularly with small-object detection goals.
C. Future work: Synthetic data generation and training, especially with Transformer backbones, is proposed to address limited availability of diverse in-domain datasets.The paper recommends investing in the creation of datasets such as STORE.