PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLO Evolution: A Comprehensive Benchmark and Architectural Review of YOLOv12, YOLO11, and Their Previous Versions

Nidhal Jegham, Chan Young Koh, Marwan Abdelatti, Abdeltawab Hendawi

arXiv:2411.00201v4cs.CV

TL;DR

The paper addresses the need for comprehensive evidence comparing YOLO versions across varied object-detection challenges. It benchmarks YOLOv3 through YOLOv12 across three diverse datasets and multiple accuracy, efficiency, and complexity metrics, finding that YOLO11 most consistently balances these objectives while YOLOv12 underperforms.

  • Problem

    The study addresses limited comprehensive evidence comparing YOLOv3 through YOLOv12 across varied object sizes, aspect ratios, small objects, and multiple performance dimensions.

  • Method

    The paper fine-tunes pre-trained YOLO models and compares them across three diverse datasets using consistent hyperparameters and metrics including Precision, Recall, mAP, processing times, GFLOPs, and model size.

  • Results

    YOLO11 most consistently balances accuracy, efficiency, and model size; YOLOv10 is faster and more efficient but slightly less accurate, while YOLOv12 performs underwhelmingly.

  • Takeaways & Limitations

    The findings guide selection of YOLO algorithms for different applications and inform future architectural enhancements.

  • Takeaways & Limitations

    YOLOv12’s Area Attention and R-ELAN introduce architectural complexity without clear advantages and make balancing accuracy and speed difficult.

Abstract

from arXiv · show

This study presents a comprehensive benchmark analysis of various YOLO (You Only Look Once) algorithms. It represents the first comprehensive experimental evaluation of YOLOv3 to the latest version, YOLOv12, on various object detection challenges. The challenges considered include varying object sizes, diverse aspect ratios, and small-sized objects of a single class, ensuring a comprehensive assessment across datasets with distinct challenges. To ensure a robust evaluation, we employ a comprehensive set of metrics, including Precision, Recall, Mean Average Precision (mAP), Processing Time, GFLOPs count, and Model Size. Our analysis highlights the distinctive strengths and limitations of each YOLO version. For example: YOLOv9 demonstrates substantial accuracy but struggles with detecting small objects and efficiency whereas YOLOv10 exhibits relatively lower accuracy due to architectural choices that affect its performance in overlapping object detection but excels in speed and efficiency. Additionally, the YOLO11 family consistently shows superior performance maintaining a remarkable balance of accuracy and efficiency. However, YOLOv12 delivered underwhelming results, with its complex architecture introducing computational overhead without significant performance gains. These results provide critical insights for both industry and academia, facilitating the selection of the most suitable YOLO algorithm for diverse applications and guiding future enhancements.

1. Introduction

YOLO evolved from a real-time one-stage detector into a broad family of architectures balancing accuracy, speed, and accessibility. This paper evaluates that evolution comprehensively, emphasizing architectural changes, applications, and comparative performance.

  • YOLO predicts bounding boxes and class probabilities in a single evaluation, enabling real-time object detection.
  • YOLOv1 through YOLOv6 introduced progressively different backbones, feature-fusion designs, frameworks, and inference simplifications.Examples include Darknet-53 and multi-scale detection in YOLOv3, SPP and PAN in YOLOv4, PyTorch and SPPF in YOLOv5, and RepVGG in YOLOv6.
  • Later versions extended YOLO toward scalable deployment and additional tasks while introducing architectural changes for gradient flow, feature fusion, and NMS-free inference.
  • YOLO has been applied across autonomous driving, surveillance, agriculture, healthcare, and other real-world domains.
  • The study provides a comprehensive evaluation of YOLOv3 through YOLOv12 across diverse datasets and metrics, using pre-trained models, fine-tuning, and consistent hyperparameters.Metrics include speed, efficiency, accuracy, GFLOPs, and model size.

2. Related Work

Prior YOLO reviews and benchmarks have often been limited by model coverage, evaluation metrics, or lack of direct comparative experiments. This paper addresses these gaps through a broader benchmark and architectural review spanning scaled YOLO variants and diverse datasets.

  • Several related studies emphasized selected applications or custom detector improvements rather than comprehensive comparisons across YOLO families.
  • Earlier reviews commonly focused on YOLO versions through YOLOv8 or YOLOv4, limiting coverage of newer architectural developments.
  • Existing reviews often omitted important measures such as Precision, Recall, speed, or efficiency by concentrating primarily on mAP or FPS.
  • One prior evolution paper covered YOLOv1 to YOLOv10 but did not conduct benchmarking or comparative performance analysis.
  • This study compares YOLOv12 with predecessors across nano-to-extra-large variants, three diverse datasets, and metrics extending beyond mAP and FPS.

3.1. Datasets

The benchmark uses three publicly accessible datasets selected to represent distinct object-detection conditions and application domains. Their differences include dataset scale, class structure, object size, aspect ratio, density, and overlap.

  • The study evaluates Ultralytics YOLO models on datasets selected for autonomous driving, satellite imagery, and wildlife conservation applications.
  • Traffic Signs Dataset: The Traffic Signs dataset contains varied sign sizes and environments, with preprocessing leaving 24 classes and 3,233 images split 70%/20%/10%.
  • Africa Wildlife Dataset: The Africa Wildlife dataset contains four animal classes with at least 376 images per class and uses a 70%/20%/10% train-validation-test split.Varying aspect ratios, overlapping objects, and multiple occurrences create detection challenges.
  • The experiments use manually annotated YOLO-format data without data augmentation for the Africa Wildlife and Ships/Vessels datasets.
  • Ships/Vessels Dataset: The Ships/Vessels dataset contains approximately 13.5k images for single-class ship detection.Small targets, varying aspect ratios, and rotations pose detection challenges, particularly for small objects.

3.2. Models

The study compares Ultralytics-supported YOLO versions and scaled variants, focusing on fair benchmarking and architectural differences across the selected models.

  • Ultralytics vs. Original YOLO: The benchmark focuses exclusively on Ultralytics-supported versions because substantial modifications make direct comparisons with original models inequitable.The authors exclude unsupported versions and original-model comparisons to maintain consistent benchmarks.
  • Model variants: 33 models from 7 YOLO versions were trained, with suffixes indicating model size and complexity and exposing accuracy–inference-speed trade-offs.Variants range from nano and tiny models to extra-large, balanced, compact, and extended models.
  • YOLOv3: YOLOv3 uses Darknet-53, residual connections, skip connections, upsampling, and three feature-map scales for multi-scale detection.The three scales correspond to down-sampling factors of 32, 16, and 8; the model struggles with medium and large objects.
  • YOLOv5: YOLOv5 uses a CSPDarknet backbone, CSP connections, strided convolution, SPPF, and augmentation methods to improve feature representation and robustness.Its architecture includes the CSPDarknet backbone, PANet neck, and YOLO head.
  • YOLOv8: YOLOv8 uses the C2f module and supports detection, classification, pose estimation, instance segmentation, and oriented bounding-box tasks.The C2f module combines high-level features with contextual information for enhanced detection accuracy.
  • YOLOv9–YOLOv12: YOLOv9 emphasizes gradient propagation and feature fusion, while YOLOv10 uses PAN, dual assignment, and one-to-one inference to improve efficiency and localization.YOLOv12 adds Area Attention and R-ELAN in an attention-centric architecture intended to improve speed, accuracy, training stability, and convergence.

3.3. Hardware and Software Setup

Experiments used a specified software stack and two NVIDIA RTX 4090 GPUs, with similar hyperparameters across models for fair comparison.

  • Software: The experiments used Python 3.12, Ubuntu 22.04, CUDA 12.4, cuDNN 8.9.7, Ultralytics 8.2.55, and WandB 0.17.4.Ultralytics handled model training and WandB tracked the experiments.
  • Hardware: Training ran on 2 NVIDIA RTX 4090 GPUs, each with 16,384 CUDA cores.The hardware setup provided GPU acceleration for the experiments.
  • Experimental control: Similar hyperparameters were applied across models to ensure a fair comparison.The hyperparameters are listed in Table 3.

3.4. Metrics

The study evaluates YOLO models with accuracy, computational-efficiency, and model-complexity metrics to support comprehensive performance comparisons.

  • Accuracy metrics: Accuracy is measured with Precision, Recall, mAP50, and mAP50-95.Precision reflects false positives, Recall reflects false negatives, mAP50 uses IoU 0.50, and mAP50-95 spans IoU thresholds from 0.50 to 0.95.
  • Efficiency metrics: Computational efficiency is assessed using Preprocessing Time, Inference Time, and Postprocessing Time.These measure data preparation, prediction generation, and conversion of raw outputs into final results.
  • Complexity metrics: GFLOPs and model size measure computational power, storage requirements, and parameter count.Together, these metrics support comprehensive comparisons for real-world applications.

4. Benchmark Results

Across traffic signs, Africa Wildlife, and Ships and Vessels, YOLO performance varied with object scale and dataset characteristics, revealing accuracy–efficiency trade-offs. YOLO11 was consistently strong overall, while YOLOv9 excelled on the small wildlife dataset and YOLOv12x was slow on several benchmarks.

  • Traffic Signs Dataset: YOLOv5ul achieved the highest Traffic Signs accuracy, with mAP50 0.866 and mAP50-95 0.799.
  • Traffic Signs Dataset: YOLOv10n was most efficient on Traffic Signs at 2ms per image and 8.3 GFLOPs, while YOLOv9 variants were least efficient.The results indicate a clear trade-off between accuracy and computational efficiency.
  • Traffic Signs Dataset: YOLO11m performed best overall on Traffic Signs, balancing accuracy, model size, and efficiency across the reported figures.YOLO11 and YOLOv12 families were reported to outperform other families in accuracy and computational efficiency on this dataset.
  • Africa Wildlife Dataset: YOLOv9s achieved Africa Wildlife mAP50 0.956 and mAP50-95 0.832, demonstrating strong accuracy on the small dataset.YOLOv9c and YOLOv9t also followed closely in accuracy.
  • Africa Wildlife Dataset: YOLOv9t and YOLOv9s excelled across Africa Wildlife metrics, combining high accuracy with small model sizes, low GFLOPs, and short inference times.Larger models such as YOLOv12x and YOLOv5ux were reported to have suboptimal accuracy despite larger sizes and longer inference times.
  • Ships and Vessels Dataset: YOLOv12x was slowest on Ships and Vessels, requiring 10.9ms and 198.5 GFLOPs.YOLOv3u tiny was fastest at 2ms, while YOLO11n achieved 2.5ms with 6.3 GFLOPs.
  • Ships and Vessels Dataset: YOLO11s and YOLOv10s performed strongly on Ships and Vessels while maintaining compact sizes, low GFLOPs, and quick processing times.This dataset contains tiny objects with varying orientations, and the mAP50-to-mAP50-95 disparity indicates difficulty at higher IoU thresholds.

5. Discussion

The discussion compares YOLO families and model scales using accuracy, speed, GFLOPs, and size rankings across three datasets. It identifies YOLO11 as the most balanced family, YOLOv10 as especially efficient, YOLOv9 as accurate but slower, and YOLOv12 as accurate but computationally costly.

  • Ranking Method: Rankings used mAP50-95 for accuracy and total processing time for speed, with models categorized by scale for fair comparison.Family-wide performance was analyzed by averaging rankings across scales.
  • Model-Scale Analysis: YOLOv9t ranked highest among nano models in accuracy and speed, while YOLOv10n traded lower accuracy for computational efficiency.The discussion attributes YOLOv10n’s lower accuracy to its One-to-One head and NMS-Free training on densely packed objects.
  • Model-Scale Analysis: YOLOv12s led small models in accuracy but suffered slower speed from Area Attention, whereas YOLOv10s and YOLO11s offered more balanced performance.YOLO11s used C3k2 for feature extraction, while YOLOv10s benefited from an efficient head design.
  • Model-Scale Analysis: YOLO11m provided the best medium-model balance, while YOLOv10m led computational efficiency and YOLOv9m incurred efficiency costs despite strong accuracy.The discussion links YOLOv9m’s overhead to gradient-based optimizations.
  • Model-Scale Analysis: YOLO11l and YOLO11x achieved the strongest large and extra-large balances of accuracy and efficiency, respectively.The discussion attributes these results to C2PSA, refined convolutions, and C3k2-based optimization.
  • Family Analysis: YOLOv12 combined strong accuracy with increased latency and computational demand from Area Attention, R-ELAN, and FlashAttention-related overhead.These costs limited its overall practicality despite its accuracy ranking.
  • Family Analysis: YOLO11 ranked first in accuracy and GFLOPs and among the first in speed and size, establishing it as a balanced alternative to YOLOv8.The discussion presents YOLO11 as suitable for applications requiring both speed and precision.
  • Family Analysis: YOLOv10 emphasized speed and size but had lower accuracy, whereas YOLOv9 maintained strong accuracy at the cost of slow inference.YOLOv8, YOLOv5u, and YOLOv3u lagged newer models in overall efficiency.

6. Real-Life Applications

The paper maps YOLO model families and scales to deployment scenarios involving resource limits, real-time detection, object size, overlap, orientation, and dataset scale. The recommendations emphasize matching model efficiency and detection strengths to application constraints.

  • Resource-Constrained Applications: Nano, small, and medium YOLO11 and YOLOv10 models suit constrained devices because they combine accuracy, compact size, low inference time, and low GFLOPs.
  • Real-Time Detection: Nano, small, and medium YOLO11 and YOLOv10 models support real-time detection by maintaining fast inference while detecting objects across scales.
  • Object-Size Challenges: Medium, large, and extra-large YOLO11 and YOLOv12 models are recommended for small-object detection, while YOLOv9 and YOLOv8 models perform robustly on large objects.
  • Overlapping Objects: YOLOv9 models are effective for overlapping objects, including densely packed animals or traffic signs.
  • Rotated Objects: YOLO11 and YOLOv12 support rotated-object detection through oriented bounding boxes and outperform YOLOv8 models in this setting.
  • Dataset Scale: Medium, large, and extra-large YOLO11 and YOLOv12 models suit large diverse datasets, whereas YOLOv9 models are recommended for tiny datasets where overfitting is a concern.

7. Conclusion

The benchmark compares YOLO versions across diverse datasets and metrics, identifying distinct trade-offs in accuracy, efficiency, and model complexity. YOLO11m offers the strongest balance, while YOLOv12 underperforms and remains a target for future improvement.

  • Benchmark scope: The study evaluates YOLO algorithms across Traffic Signs, African Wildlife, and Ships and Vessels datasets using accuracy, speed, efficiency, and model-size metrics.The datasets represent varying object sizes, aspect ratios, and object densities.
  • Comparative findings: YOLO11m achieves the most consistent performance by balancing accuracy, efficiency, and model size.The YOLO11 family is identified as the most consistent overall.
  • Comparative findings: YOLOv10 offers lower accuracy than YOLO11 but excels in speed and efficiency for fast-processing applications.
  • Comparative findings: YOLOv9 performs well overall and stands out particularly on smaller datasets.
  • Limitations and future work: YOLOv12 underperforms because its Area Attention Module and R-ELAN add complexity without clear advantages, making accuracy-speed balance difficult.The results highlight challenges in integrating advanced attention mechanisms into the YOLO framework.
  • Limitations and future work: Future work should improve YOLOv10 accuracy, simplify YOLOv12 attention mechanisms, and pursue more efficient and accurate architectures.

Declarations

The declarations report no applicable funding, ethics approval, consent, or materials statement, while affirming no conflicts of interest and author approval. Data and source code are available on GitHub and are intended for public release with the final version.

  • Declarations: The authors report that funding, ethics approval and consent, and materials availability are not applicable.
  • Declarations: The authors declare no conflict of interest and confirm that all publication permissions have been obtained.
  • Declarations: Data and source code are available on GitHub and will be publicly available upon publication of the final version.
  • Author contribution: All authors contributed to the study, approved the final manuscript, and identified Dr. Abdeltawab Hendawi as supervisor.
Loading 2411.00201v4…