PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS

Juan Terven, Diana Cordova-Esparza

arXiv:2304.00501v7cs.CV

TL;DR

YOLO’s evolution addresses the need for real-time object detection that balances speed and accuracy across diverse applications. This paper reviews 16 YOLO versions and related variants, tracing architectural, training, and postprocessing developments. It identifies recurring shifts such as anchor-free detection, changing backbones, and continued speed–accuracy tradeoff tuning, while noting early localization and nearby-object limitations.

  • Problem

    Real-time object detection requires balancing speed and accuracy, while existing reviews omit recent YOLO developments and variants.

  • Method

    The paper comprehensively reviews 16 YOLO versions from the original model through YOLO-NAS, covering architectures, training developments, and related variants.

  • Results

    YOLO development progressed from anchor-based improvements to YOLOX’s anchor-less state-of-the-art approach, alongside major backbone changes including CSP, reparameterization, and neural architecture search.

  • Takeaways & Limitations

    YOLO models prioritize speed–accuracy balance and offer different scales for application contexts and hardware requirements.

  • Takeaways & Limitations

    YOLOv1 had larger localization error and could detect at most two objects of the same class within a grid cell.

Abstract

from arXiv · show

YOLO has become a central real-time object detection system for robotics, driverless cars, and video monitoring applications. We present a comprehensive analysis of YOLO's evolution, examining the innovations and contributions in each iteration from the original YOLO up to YOLOv8, YOLO-NAS, and YOLO with Transformers. We start by describing the standard metrics and postprocessing; then, we discuss the major changes in network architecture and training tricks for each model. Finally, we summarize the essential lessons from YOLO's development and provide a perspective on its future, highlighting potential research directions to enhance real-time object detection systems.

1 Introduction

This introduction positions YOLO as a real-time object-detection framework whose successive versions address limitations through architectural and training improvements. The review surveys YOLO’s evolution, related detectors, applications, evaluation metrics, and speed–accuracy tradeoffs.

  • Review scope: The review covers YOLOv1 through YOLOv8 and extends coverage to YOLOX, PP-YOLOs, transformer-based YOLO, and YOLO-NAS.It emphasizes network design, loss functions, anchor boxes, and input-resolution scaling.
  • Related detection methods: YOLO and other detectors occupy different speed, accuracy, and complexity tradeoffs suited to application-specific requirements.The introduction contrasts two-stage R-CNN methods, single-shot SSD, segmentation-capable Mask R-CNN, RetinaNet, and EfficientDet.
  • Applications: YOLO models have been applied to autonomous vehicles, surveillance, agriculture, medicine, remote sensing, traffic systems, wildlife monitoring, robotics, and drones.These applications include object tracking, crop and disease detection, medical detection, satellite-image analysis, and defect inspection.

4 YOLO: You Only Look Once

YOLO introduced an end-to-end, single-pass approach that unified object detection and enabled real-time performance. Its grid-based predictions and training design delivered strong speed, but localization and several object-detection cases remained challenging.

  • YOLO: You Only Look Once: YOLO presented a real-time end-to-end detector that performs object detection with a single network pass.This replaced repeated sliding-window classification and two-stage region-proposal pipelines.
  • How YOLOv1 works?: YOLO divides an image into an S × S grid and predicts B boxes with confidence and class information for each grid element.Each box contains confidence, center coordinates, height, and width, producing an S × S × (B × 5 + C) tensor followed optionally by NMS.
  • How YOLOv1 works?: On PASCAL VOC, YOLOv1 used S = 7, B = 2, and C = 20, producing a 7 × 7 × 30 output prediction.A simplified three-by-three, three-class, single-box example produces 3 × 3 × 8.
  • YOLOv1 results: YOLOv1 achieved an average precision (AP) of 63.4 on PASCAL VOC2007.The model used 24 convolutional layers followed by two fully connected layers, with ImageNet pretraining and higher-resolution detection fine-tuning.
  • YOLOv1 Training: The YOLOv1 loss combines localization, confidence, and classification errors, weighting coordinate loss by λcoord = 5 and reducing no-object confidence loss with λnoobj = 0.5.Localization errors are computed for boxes containing objects, while classification loss applies when an object appears in the cell.
  • YOLOv1 Strengths and Limitations: YOLO was faster than existing detectors but had larger localization error, limited same-class detections per cell, difficulty with unseen aspect ratios, and coarse-feature learning.The grid allowed at most two same-class objects per cell, while downsampling contributed to coarse object features.

5 YOLOv2: Better, Faster, and Stronger

YOLOv2 added architectural, training, and prediction refinements while preserving the framework’s speed. These changes improved detection accuracy and expanded the model’s category coverage through joint classification and detection training.

  • YOLOv2 improvements: YOLOv2 introduced batch normalization, high-resolution classifier fine-tuning, a fully convolutional design, and anchor boxes.Its backbone, Darknet-19, used 19 convolutional layers and five max-pooling layers with 1 × 1 convolutions for parameter reduction.
  • YOLOv2 results: YOLOv2 achieved an average precision (AP) of 78.6% on PASCAL VOC2007 compared with 63.4% for YOLOv1.The comparison reports the AP values for the same dataset.
  • YOLOv2 prediction: YOLOv2 predicts five bounding boxes, each with five values and 20 classes when using the PASCAL VOC dataset.The model defines multiple anchor boxes for each grid cell and parameterizes box coordinates relative to prior boxes.
  • YOLO9000: YOLOv2 jointly trained classification and detection using COCO detection data and ImageNet classification data.This produced YOLO9000, capable of detecting more than 9000 categories.

6 YOLOv3

YOLOv3 expanded the detector with a larger residual backbone, multi-label classification, and predictions at three scales. These changes improved small-object detection while maintaining real-time performance, with YOLOv3-spp reaching 36.2% AP at 20 FPS on COCO.

  • YOLOv3 replaced softmax classification with independent logistic classifiers trained using binary cross-entropy, allowing multiple labels per bounding box.
  • Its Darknet-53 backbone contains 53 convolutional layers with residual connections, replacing max-pooling layers with strided convolutions.
  • YOLOv3 predicts three boxes at different scales, producing finer boxes and significantly improving small-object prediction.The three outputs use 13 × 13, 26 × 26, and 52 × 52 feature-map resolutions.
  • Each COCO scale outputs an N × N × [3 × (4 + 1 + 80)] tensor, representing three boxes per cell, four coordinates, objectness, and 80 categories.
  • YOLOv3-spp achieved 36.2% AP and 60.6% AP50 at 20 FPS on MS COCO, reported as state-of-the-art at the time.

7 Backbone, Neck, and Head

Modern object detectors are organized into a backbone, neck, and head. The backbone extracts hierarchical features, the neck refines multi-scale representations, and the head produces task-specific predictions.

  • The backbone extracts hierarchical features from the input image, from low-level edges and textures to higher-level object and semantic information.
  • The neck connects backbone and head features by aggregating and refining spatial and semantic information across scales.
  • The head generates predictions through task-specific subnetworks for classification, localization, and potentially instance segmentation or pose estimation.
  • The paper uses the backbone, neck, and head framework to describe the remaining YOLO architectures.

8 YOLOv4

YOLOv4 combined architectural modules with training strategies to balance accuracy and inference speed, while YOLOv5 adapted many of these ideas in PyTorch. Their reported COCO results reached 43.5% AP above 50 FPS for YOLOv4 and 50.7% AP for YOLOv5x at 640-pixel input.

  • YOLOv4 categorized improvements into bag-of-freebies, which raise training cost without inference cost, and bag-of-specials, which slightly increase inference cost while improving accuracy.
  • YOLOv4 used CSPDarknet53 with Mish, SPP, modified PANet, SAM, and multi-scale predictions; CSP connections reduced computation while SPP expanded the receptive field without affecting inference speed.
  • YOLOv4 achieved 43.5% AP and 65.7% AP50 at more than 50 FPS on an NVIDIA V100 using MS COCO test-dev 2017.
  • YOLOv5 moved the implementation to PyTorch and added AutoAnchor, which evolves dataset-specific anchor boxes after k-means initialization.
  • YOLOv5 uses a modified CSPDarknet53 backbone, SPPF, CSP-PAN, SiLU activations, and a YOLOv3-like head.
  • YOLOv5x achieved 50.7% AP at 200 FPS with 640-pixel input on an NVIDIA V100, while 1536-pixel input with test-time augmentation reached 55.8% AP.

10 Scaled-YOLOv4

Scaled-YOLOv4 introduced explicit scaling up and down to target different hardware and accuracy-speed requirements. Subsequent models broadened the design space through unified representations, anchor-free detection, decoupled heads, efficient backbones, and quantization.

  • Scaled-YOLOv4: Scaling up increases accuracy at lower speed, whereas scaling down increases speed with reduced accuracy and lower computing requirements.
  • Scaled-YOLOv4: YOLOv4-tiny targeted low-end GPUs, reaching 22% AP on COCO at 46 FPS on Jetson TX2 and 440 FPS on RTX2080Ti.
  • Scaled-YOLOv4: YOLOv4-large targeted cloud GPUs with P5, P6, and P7 variants and achieved 56% AP on MS COCO.
  • YOLOR: YOLOR used multi-task learning to learn a general representation and derive task-specific representations for classification, detection, and pose estimation.
  • YOLOX: YOLOX introduced anchor-free detection, multiple positives, a decoupled head, advanced label assignment, and strong augmentations, achieving 50.1% AP at 68.9% FPS on Tesla V100.
  • YOLOv6: YOLOv6 combined efficient RepVGG- or CSPStackRep-based backbones, a PAN neck, an efficient decoupled head, and quantization techniques.
  • YOLOv6: The largest YOLOv6 model achieved 57.2% AP at around 29 FPS on an NVIDIA Tesla T4.

14 YOLOv7

YOLOv7 combined architectural changes and training techniques to improve efficiency and accuracy across a broad speed range. Its E-ELAN design, modified re-parameterized convolution, and auxiliary-head training supported strong benchmark results with reduced computational cost.

  • Overview: YOLOv7 surpassed known object detectors in speed and accuracy across 5 FPS to 160 FPS, while its bag-of-freebies increased accuracy without affecting inference speed.Training time increased, but inference speed was preserved.
  • Architecture: E-ELAN shuffles and merges features from different groups to improve learning while preserving the original gradient path.The design supports models with unlimited stacked computational blocks.
  • Architecture: Removing RepConv’s identity connection produced RepConvN because the connection disrupted residual and concatenation structures.This change was motivated by interactions with ResNet and DenseNet designs.
  • Training: YOLOv7 trains an auxiliary head with coarse labels while the lead head uses fine labels for the final output.The auxiliary head assists training, whereas the lead head produces the final predictions.
  • Results: Compared to YOLOv4, YOLOv7 reduced parameters by 75% and computation by 36% while improving AP by 1.5%.These reductions were reported alongside the AP improvement.
  • Results: YOLOv7-E6 achieved 55.9% AP and 73.5% AP50 at 50 FPS on an NVIDIA V100 using 1280-pixel inputs.The evaluation used MS COCO test-dev 2017.

15 DAMO-YOLO

DAMO-YOLO combines neural architecture search with an efficient large neck and small head for real-time detection. The later YOLOv8 broadens the YOLO framework across several vision tasks while retaining high-speed detection.

  • DAMO-YOLO: DAMO-YOLO uses MAE-NAS to automatically search for an efficient architecture, alongside an Efficient-RepGFPN neck and a compact prediction head.The head retains one linear layer for classification and one for regression.
  • DAMO-YOLO: DAMO-YOLO’s best scaled model achieved 50.0% AP at 233 FPS on an NVIDIA V100.The model family includes Tiny, Small, and Medium variants.
  • YOLOv8: YOLOv8 provides nano, small, medium, large, and extra-large versions and supports detection, segmentation, pose estimation, tracking, and classification.These capabilities extend YOLOv8 beyond object detection alone.
  • YOLOv8: YOLOv8 replaces YOLOv5’s CSPLayer with the C2f module, combining high-level features with contextual information to improve detection accuracy.The backbone remains similar to YOLOv5 with this modification.
  • YOLOv8: YOLOv8 uses an anchor-free decoupled head whose branches independently process objectness, classification, and regression.The design allows each branch to focus on its respective task.
  • Results: YOLOv8x achieved 53.9% AP at 280 FPS on an NVIDIA A100 and TensorRT with 640-pixel images.YOLOv5 achieved 50.7% AP at the same input size in the cited comparison.

17 PP-YOLO, PP-YOLOv2, and PP-YOLOE

The PP-YOLO line progressively improves YOLOv3-derived detection through backbone, training, loss, preprocessing, and head refinements. PP-YOLOE adopts anchor-free detection and task alignment while maintaining a speed–accuracy balance.

  • PP-YOLO: PP-YOLO combines a ResNet50-vd-dcn backbone, larger-batch training, parameter moving averages, FPN DropBlock, IoU-aware losses, and Matrix NMS.These changes target feature extraction, training stability, localization, regularization, and postprocessing.
  • PP-YOLO: PP-YOLO achieved 45.9% AP and 65.2% AP50 at 73 FPS on an NVIDIA V100.The evaluation used MS COCO test-dev 2017.
  • PP-YOLOE: PP-YOLOE achieved 51.4% AP at 78.1 FPS on an NVIDIA V100 after adopting an anchor-free architecture, RepResBlocks, task alignment learning, and an efficient task-aligned head.Its loss design includes Varifocal Loss and Distribution Focal Loss.
  • PP-YOLOE: PP-YOLOE uses dynamic label assignment and task-alignment loss to reduce mismatch between classification confidence and localization accuracy.The method jointly considers classification and localization quality through IoU-aware classification scores.

18 YOLO-NAS

YOLO-NAS uses neural architecture search and quantization-aware design to target efficient real-time deployment. Its model family balances latency, throughput, and accuracy across hardware and precision settings.

  • Deployment: YOLO-NAS is designed for small-object detection, improved localization accuracy, and performance-per-compute efficiency in real-time edge-device applications.Its architecture is open source for research use.
  • Quantization: QSP and QCI modules combine re-parameterization with 8-bit quantization to minimize accuracy loss during post-training quantization.These modules are part of YOLO-NAS’s quantization-aware design.
  • Quantization: Hybrid quantization selectively quantizes model parts to balance latency and accuracy instead of quantizing every layer.This differs from standard quantization applied uniformly across the model.
  • Architecture search: YOLO-NAS uses AutoNAC to automatically design architectures that balance latency and throughput, producing small, medium, and large variants.The variants differ in depth and QSP/QCI block placement.
  • Results: YOLO-NAS achieved 52.2% AP on MS COCO with 16-bit precision.The released models support FP32, FP16, and INT8 precisions.

19 YOLO with Transformers

Transformer-based YOLO variants combine attention mechanisms with established YOLO components to improve feature representation across scales and applications. These approaches include hybrid backbones, transformer-enhanced detection heads, additional prediction stages, and specialized modules for small or difficult objects.

  • Transformer-based detection: YOLOS adapts a pretrained Vision Transformer for detection by replacing one [CLS] token with one hundred [DET] tokens and using bipartite matching loss.It achieved 42.0% AP on MS COCO.
  • ViT-YOLO: ViT-YOLO combines CSP-Darknet and MHSA-Darknet in the backbone, BiFPN in the neck, and five multi-scale detection heads.The architecture integrates convolutional, self-attention, feature-pyramid, and multi-scale detection components.
  • Application-specific variants: NRT-YOLO adds a prediction head, feature-fusion layers, and a residual transformer module to address tiny objects in remote-sensing images.These changes improved YOLOv5l by 5.4% on the DOTA dataset.
  • Application-specific variants: YOLO-SD combines YOLOX with multi-scale convolution and feature-transformer modules to improve small-ship detection in SAR imagery.The reported comparison showed higher accuracy than YOLOX on HRSID.
  • YOLO-DETR combinations: DEYO uses a YOLOv5-based first stage to generate queries and anchors for a DETR-like second stage, achieving 52.1% AP on COCO.The authors also report faster convergence and better performance than DETR.

20 Discussion

The review identifies recurring architectural and design shifts across 16 YOLO versions, including changes in anchors, frameworks, backbones, and speed–accuracy balancing. It concludes by outlining future directions involving newer techniques, benchmarks, applications, task domains, and hardware platforms.

  • Anchors: YOLOv2 introduced anchors, while YOLOX later adopted an anchor-less approach; subsequent YOLO versions abandoned anchors.This marks a shift in bounding-box prediction design across the family.
  • Framework: YOLO development moved from Darknet to PyTorch after Ultralytics ported YOLOv3, contributing to a surge in enhancements.PaddlePaddle is another framework used in the YOLO ecosystem.
  • Backbone: Backbones evolved from simple Darknet convolution and pooling layers to CSP connections, reparameterization, and neural architecture search.CSP appeared in YOLOv4, reparameterization in YOLOv6 and YOLOv7, and neural architecture search in DAMO-YOLO and YOLO-NAS.
  • Performance: YOLO models generally improve detection while balancing speed and accuracy rather than optimizing accuracy alone.This balance supports real-time detection across varied applications.
  • Performance: The original YOLO used a single CNN for direct prediction and enabled real-time processing, but accuracy suffered especially for small or overlapping objects.Later versions introduced refinements intended to address these limitations while preserving real-time capabilities.
  • Evolution of capabilities: YOLOv2 added anchor boxes and passthrough layers, while YOLOv3 introduced multi-scale feature extraction to improve localization and detection across object sizes.These changes addressed limitations associated with the original model’s speed-focused design.
  • Performance: YOLOv4 and YOLOv5 improved accuracy through new backbones, data augmentation, and training strategies without drastically reducing real-time performance.From YOLOv5 onward, model scales target different application and hardware requirements.
Loading 2304.00501v7…