Source-linked AI summary
You Only Look Once: Unified, Real-Time Object Detection
Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi
TL;DR
Object detection pipelines were slow and difficult to optimize because they separately generated, classified, and refined candidate boxes. YOLO instead predicts boxes and class probabilities with one end-to-end neural network, achieving real-time detection while generalizing well to new domains.
Problem
Existing detection methods used complex, separately trained pipelines that were slow and difficult to optimize.
Method
YOLO reframes detection as a single regression problem, with one neural network jointly predicting bounding boxes and class probabilities from full images.
Results
Fast YOLO is reported as the fastest general-purpose object detector, while YOLO achieves real-time speeds and high average precision.
Takeaways & Limitations
YOLO generalizes well to new domains, supporting applications that rely on fast, robust object detection.
Takeaways & Limitations
YOLO’s loss treats localization errors equally across box sizes, and its main source of error is incorrect localization.
Abstract
from arXiv · showhide
We present YOLO, a new approach to object detection. Prior work on object detection repurposes classifiers to perform detection. Instead, we frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. Our unified architecture is extremely fast. Our base YOLO model processes images in real-time at 45 frames per second. A smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors. Compared to state-of-the-art detection systems, YOLO makes more localization errors but is far less likely to predict false detections where nothing exists. Finally, YOLO learns very general representations of objects. It outperforms all other detection methods, including DPM and R-CNN, by a wide margin when generalizing from natural images to artwork on both the Picasso Dataset and the People-Art Dataset.
1. Introduction
YOLO reframes object detection as a single, end-to-end regression network that predicts boxes and classes directly from the full image. This unified design delivers real-time speed, fewer background errors, and strong generalization, while sacrificing some localization accuracy.
- Unified formulation: YOLO replaces classifier-based detection pipelines with regression from image pixels directly to bounding-box coordinates and class probabilities.A single convolutional network predicts multiple boxes and classes simultaneously while training on full images.
- Real-time performance: 45 frames per second is the base network’s speed, while the fast version runs at more than 150 fps with less than 25 milliseconds of latency.The system requires only one neural-network evaluation at test time, avoiding a complex detection pipeline.
- Global reasoning: Less than half the background errors compared to Fast R-CNN shows how global image reasoning reduces false detections in background patches.YOLO sees the entire image during training and testing, allowing it to encode contextual information about objects and their surroundings.
- Generalization: YOLO outperforms DPM and R-CNN by a wide margin when trained on natural images and tested on artwork.This cross-domain result indicates that YOLO learns representations that transfer to new or unexpected inputs.
- Limitations: YOLO still lags behind state-of-the-art detectors in accuracy, particularly when precisely localizing small objects.The paper explicitly frames speed and robustness against background false positives as tradeoffs with localization precision.
2. Unified Detection
YOLO unifies object detection in one neural network that predicts boxes and classes directly from the full image. Its grid-based representation enables end-to-end training and real-time inference, but introduces localization and crowding limitations.
- Unified model: YOLO uses the entire image to predict all bounding boxes and class probabilities simultaneously, enabling end-to-end optimization and global reasoning.The network replaces separate detection components with a single model that predicts across all classes and boxes at once.
- Grid representation: Each grid cell predicts B boxes, confidence scores, and C conditional class probabilities, producing an S × S × (B × 5 + C) tensor.A cell is responsible for an object when that object's center falls inside it.
- Box predictions: A box contains x, y, w, h, and confidence, with center coordinates relative to the cell and width and height relative to the whole image.Confidence reflects both whether the box contains an object and how accurately it overlaps the ground truth.
- Training: YOLO’s training objective uses sum-squared error, but localization and classification errors are weighted equally and many empty cells can destabilize training.The method increases coordinate-loss weight and decreases no-object confidence-loss weight using λcoord = 5 and λnoobj = .5.
- Limitations: YOLO struggles with grouped small objects, unusual aspect ratios, coarse features, and incorrect localizations.Its grid permits only two boxes and one class per cell, while the loss treats errors equally across small and large boxes.
3. Comparison to Other Detection Systems
YOLO replaces the fragmented detection pipelines used by systems such as DPM and R-CNN with one jointly optimized network. This design reduces proposal overhead while supporting general-purpose detection across multiple object classes.
- R-CNN: R-CNN uses region proposals, feature extraction, SVM scoring, box adjustment, and non-max suppression as separately tuned stages.Selective Search produces potential boxes, making the resulting system slow at test time.
- R-CNN: 98 boxes per image compares with about 2000 from Selective Search in R-CNN.YOLO combines proposal scoring and detection into one jointly optimized model.
- Other fast detectors: Fast and Faster R-CNN improve speed and accuracy over R-CNN but still fall short of real-time performance.The comparison highlights that architectural simplification, rather than incremental pipeline optimization, is central to YOLO’s speed.
- DPM: 30Hz DPM is identified as the only DPM variant in this comparison that actually runs in real time.Other efforts accelerate HOG computation, use cascades, or shift computation to GPUs.
- Pipeline design: YOLO eliminates the large detection pipeline instead of optimizing its individual components.The unified model performs feature extraction, bounding box prediction, non-maximal suppression, and contextual reasoning concurrently.
- General-purpose detection: YOLO detects multiple object classes simultaneously, unlike detectors optimized for a single class.This positions YOLO as a general-purpose detector rather than a narrowly specialized system.
4. Experiments
YOLO’s experiments show that a unified detector reaches real-time speed, complements Fast R-CNN’s error profile, and transfers better across visual domains. Its main weaknesses remain localization accuracy and small-object detection.
- Comparison to Other Real-Time Systems: 52.7% mAP makes Fast YOLO more than twice as accurate as prior real-time detectors, while YOLO reaches 63.4% mAP at real-time speed.Fast YOLO is reported as the fastest detector on PASCAL VOC, while the larger YOLO model improves accuracy further.
- Comparison to Other Real-Time Systems: 0.5 fps keeps Fast R-CNN far from real-time because selective search takes around 2 seconds per image to generate proposals.Faster R-CNN improves proposal generation but its most accurate model reaches 7 fps and a smaller model reaches 18 fps.
- VOC 2007 Error Analysis: 13.6% of Fast R-CNN’s top detections are background false positives, making it almost 3x more likely than YOLO to predict background.YOLO instead makes more localization errors, revealing a complementary error profile.
- Combining Fast R-CNN and YOLO: 75.0% mAP results when YOLO is combined with Fast R-CNN, a 3.2% increase over Fast R-CNN’s 71.8% mAP.Other Fast R-CNN ensembles produce only .3–.6% increases, indicating that the gain comes from differing error types rather than generic ensembling.
- VOC 2012 Results: 57.9% mAP on VOC 2012 remains below state of the art, with YOLO especially struggling on small objects such as bottles, sheep, and tv/monitors.YOLO performs better than competing methods on some categories, including cat and train.
- Generalizability: Person Detection in Artwork: YOLO’s average precision degrades less than other methods on artwork because object size, shape, and relationships transfer beyond pixel appearance.The evaluation uses the Picasso Dataset and People-Art Dataset for person detection in artwork.
5. Real-Time Detection In The Wild
YOLO extends real-time detection beyond benchmark images to interactive webcam use and evaluation on artwork datasets. Its visual outputs remain largely accurate, while the experiments clarify both the evaluation setup and qualitative behavior.
- Webcam deployment: YOLO maintains real-time performance when connected to a webcam, including image capture and displaying detections.The resulting system behaves interactively, tracking objects as they move and change appearance.
- Artwork generalization: Figure 5 compares YOLO with other detection methods on Picasso and People-Art generalization results.The Picasso evaluation includes precision-recall curves and quantitative results using AP and best F1 score.
- Qualitative behavior: Qualitative results on artwork and natural images are mostly accurate, though YOLO incorrectly identifies one person as an airplane.
6. Conclusion
YOLO concludes with a unified, jointly trained model for object detection that combines simplicity with strong real-time performance. Fast YOLO leads general-purpose detection speed, while YOLO also transfers effectively to new domains.
- Unified model: YOLO is a unified object-detection model that is simple to construct and trained directly on full images.Unlike classifier-based approaches, its entire model is trained jointly with a loss function aligned to detection performance.
- Real-time performance: Fast YOLO is described as the fastest general-purpose object detector in the literature, while YOLO advances real-time detection.
- Domain transfer: YOLO generalizes well to new domains, supporting applications that require fast and robust object detection.