PaperlayerView exact revision on arXiv

Source-linked AI summary

A Robust Real-Time Automatic License Plate Recognition Based on the YOLO Detector

Rayson Laroca, Evair Severo, Luiz A. Zanlorensi, Luiz S. Oliveira, Gabriel Resende Gonçalves, William Robson Schwartz, David Menotti

arXiv:1802.09567v6cs.CV

TL;DR

The paper targets ALPR systems that lack robustness under real-world variation. It develops a YOLO-based, stage-specific CNN pipeline with augmented character processing and evaluates it on SSIG and the more varied UFPR-ALPR dataset, achieving higher reported recognition rates than the cited baselines and commercial systems.

  • Problem

    Many ALPR solutions remain insufficiently robust in real-world scenarios because they depend on constrained cameras, viewpoints, backgrounds, lighting, search regions, and vehicle types.

  • Method

    The system uses YOLO-based CNNs trained for ALPR stages, temporal redundancy for video predictions, and separate character segmentation and recognition with targeted augmentation.

  • Results

    93.53% recognition on SSIG and 78.33% on UFPR-ALPR, exceeding the cited previous and commercial-system results on both datasets.

  • Takeaways & Limitations

    A larger, varied public dataset exposes the difficulty of real-world ALPR while the proposed system achieves stronger reported recognition than the evaluated baselines.

Abstract

from arXiv · show

Automatic License Plate Recognition (ALPR) has been a frequent topic of research due to many practical applications. However, many of the current solutions are still not robust in real-world situations, commonly depending on many constraints. This paper presents a robust and efficient ALPR system based on the state-of-the-art YOLO object detector. The Convolutional Neural Networks (CNNs) are trained and fine-tuned for each ALPR stage so that they are robust under different conditions (e.g., variations in camera, lighting, and background). Specially for character segmentation and recognition, we design a two-stage approach employing simple data augmentation tricks such as inverted License Plates (LPs) and flipped characters. The resulting ALPR approach achieved impressive results in two datasets. First, in the SSIG dataset, composed of 2,000 frames from 101 vehicle videos, our system achieved a recognition rate of 93.53% and 47 Frames Per Second (FPS), performing better than both Sighthound and OpenALPR commercial systems (89.80% and 93.03%, respectively) and considerably outperforming previous results (81.80%). Second, targeting a more realistic scenario, we introduce a larger public dataset, called UFPR-ALPR dataset, designed to ALPR. This dataset contains 150 videos and 4,500 frames captured when both camera and vehicles are moving and also contains different types of vehicles (cars, motorcycles, buses and trucks). In our proposed dataset, the trial versions of commercial systems achieved recognition rates below 70%. On the other hand, our system performed better, with recognition rate of 78.33% and 35 FPS.

I. INTRODUCTION

The paper addresses limited robustness in real-world ALPR by combining YOLO-based detection, stage-specific CNNs, character-focused augmentation, and a broader benchmark dataset.

  • Many ALPR solutions remain constrained by camera, viewpoint, background, lighting, search-region, and vehicle-type assumptions.These constraints can exclude motorcycles, trucks, and buses.
  • UFPR-ALPR expands evaluation beyond SSIG with 4,500 images from varied vehicles, cameras, positions, distances, backgrounds, and lighting conditions.The dataset includes cars, motorcycles, buses, and trucks, with some vehicles only partially visible.
  • The proposed system uses YOLO object-detection CNNs in a real-time end-to-end ALPR pipeline.The paper positions YOLOv2 and Fast-YOLO as speed/accuracy alternatives for real-time detection.
  • Character segmentation and recognition use a robust two-stage approach with inverted LPs and flipped characters for training augmentation.These augmentations target character-processing robustness.
  • The evaluation compares the proposed approach with previous work and two commercial systems across SSIG and UFPR-ALPR.The introduction reports that the proposed system outperforms those comparisons in both datasets.

II. RELATED WORK

Prior ALPR research has used deep learning across pipeline stages, but evaluation is often limited by datasets and settings that poorly represent real-world conditions. The UFPR-ALPR dataset addresses these limitations with diverse, annotated imagery from multiple cameras and vehicle types.

  • Deep-learning ALPR research: Recent ALPR studies apply deep learning to license-plate detection, character segmentation, and character recognition, including CNN-, HMM-, and end-to-end approaches.The reviewed literature includes methods that address individual stages as well as unified systems.
  • Research limitations: Many prior datasets and methods do not represent real-world scenarios or support real-time recognition, limiting evaluation and deployment.The review specifically notes constraints in experimental scenarios and computational speed.
  • UFPR-ALPR dataset: 4,500 images were captured from 150 one-second videos recorded in urban traffic using three cameras.The dataset uses 30 FPS videos and images captured from inside a moving vehicle.
  • UFPR-ALPR dataset: The dataset includes cars, motorcycles, buses, trucks, complex backgrounds, varied lighting, and changing vehicle positions and distances.Some images also contain vehicles that are not fully visible.
  • Dataset distribution: The UFPR-ALPR vehicle and plate locations are better distributed than those in SSIG, while letters A and B occur more frequently because of Paraná plate ranges.The letter imbalance reflects the regional registration range AAA-0001 to BEZ-9999.
  • UFPR-ALPR dataset: UFPR-ALPR provides annotations for camera, vehicle attributes, vehicle and plate locations, and character positions.The annotations are stored in text files and include vehicle type, manufacturer, model, and year.

IV. PROPOSED ALPR APPROACH

The proposed ALPR approach uses separate CNNs for the pipeline stages, allowing each task to be tuned independently. It combines Fast-YOLO, YOLOv2, and the CR-NET architecture for vehicle and plate processing.

  • Pipeline design: The approach divides ALPR into vehicle and plate detection, character segmentation, character recognition, and temporal redundancy stages.The complete pipeline is illustrated in Fig. 5.
  • Pipeline design: Specific CNNs are assigned to each stage so their parameters can be tuned separately for the corresponding task.The models are Fast-YOLO, YOLOv2, and CR-NET.
  • Model selection: Fast-YOLO, YOLOv2, and CR-NET provide the detection, segmentation, and recognition components of the proposed system.CR-NET is inspired by Fast-YOLO and is used for character segmentation and recognition.

A. Vehicle and LP Detection

Vehicle and plate detection is performed hierarchically with separate CNNs, using Fast-YOLO or YOLOv2 according to scenario complexity. The design adjusts output filters and training inputs for dataset classes and robust plate coverage.

  • Detection pipeline: Two CNNs detect vehicles in the input image and plates within the detected vehicle region.The vehicle-first strategy is also used in related work.
  • Model selection: Fast-YOLO is evaluated for simpler data, while YOLOv2 is considered for more realistic scenarios requiring greater depth.The choice reflects a speed-versus-accuracy trade-off.
  • YOLO configuration: YOLO output filters are configured as filters = (C + 5) × A, with five anchor boxes and class-dependent predictions.Each anchor predicts four coordinates, confidence, and C class probabilities.
  • YOLO configuration: 30 filters support one-class detection in SSIG, whereas 35 filters support two vehicle classes in UFPR-ALPR.Using separate car and motorcycle classes produced better results than a single vehicle class in the tests.
  • Training inputs: The plate detector is trained on a vehicle patch with a margin selected to keep plates inside imperfect vehicle bounding boxes.The margin is validated to reduce plate loss when vehicle detection or segmentation is inaccurate.
  • Detection thresholds: Detection thresholds are selected on validation data, with a zero threshold used for plate detection to retain low-confidence plate candidates.When multiple plates are detected, only the highest-confidence detection is kept.

B. Character Segmentation

Character segmentation and recognition use separate CNN stages tailored to Brazil’s three-letter, four-digit plate format, with augmentation and padding choices designed to preserve character information.

  • Character Segmentation: The system first segments characters, then uses separate recognition networks for letters and digits.The Brazilian format identifies the first three characters as letters and the last four as digits.
  • Character Segmentation: 30 filters in the final segmentation layer represent the single character-segmentation class.
  • Character Segmentation: Negative LP images double segmentation training samples and simulate character colors from other vehicle categories.
  • Character Recognition: Padding of 1–3 pixels compensates for imperfect segmentation, while larger padding also adds noise from neighboring characters or the LP frame.
  • Character Recognition: Flipped-character augmentation creates new recognition instances, but validation found it improved letters while hampering digits.

D. Temporal Redundancy

The system aggregates frame-level plate predictions across each vehicle using temporal redundancy, producing a final character sequence by majority vote.

  • Temporal Redundancy: Final recognition uses the most frequently predicted character at each plate position across all frames of the same vehicle.
  • Temporal Redundancy: Prior ALPR studies also reported greatly increased recognition rates when using majority voting.
  • Temporal Redundancy: Experiments compare the proposed system with previous work and the Sighthound and OpenALPR commercial systems on SSIG and UFPR-ALPR.
  • Temporal Redundancy: The commercial systems used larger private datasets, although they were not tuned for these evaluation datasets.

A. Evaluation on the SSIG Dataset

The SSIG evaluation uses a constrained, static-camera dataset with annotated vehicle plates and a predefined train-validation-test protocol; vehicle detection was manually supported and achieved near-perfect test performance.

  • Dataset: SSIG contains 2,000 images from 101 vehicles with annotations for plate position, plate identity, and character positions.
  • Dataset: The dataset uses a static camera, similar simple backgrounds, no motorcycles, and few misaligned plates.
  • Evaluation Protocol: SSIG is split into 40% training, 20% validation, and 40% testing to support a larger test set.
  • Vehicle Detection: Vehicle bounding boxes were manually labeled because SSIG lacks vehicle annotations.
  • Vehicle Detection: 100% recall and above 99% precision were achieved for vehicle detection using a 0.125 confidence threshold.

2) LP Detection:

The reported ALPR stages achieve high detection and segmentation performance, while recognition improves through task-specific networks, temporal aggregation, and data augmentation.

  • LP Detection: 100% recall and precision were obtained for LP detection on both validation and test sets.
  • LP Detection: A 10% margin was used so detected plates retained all characters for test-time processing and segmentation training.
  • Character Segmentation: 99.75% character-segmentation recall was achieved on 5,628 characters using a 0.1 confidence threshold.
  • Character Recognition: Recognition without temporal redundancy reached 85.45%, with all letters and digits recognized in 86.32% and 98.63% of cases, respectively.
  • Character Recognition: 93.53% final recognition was achieved with temporal redundancy, with all digits and letters correct in 93.53% of vehicles.
  • Comparison: Recognition improved from 81.8% to 93.53%, and the proposed system recognized at least 6 of 7 characters in every plate.

B. Evaluation on the UFPR-ALPR Dataset

On the UFPR-ALPR dataset, the system used YOLOv2 for vehicle detection and Fast-YOLO for LP detection, with margins accommodating challenging LP placements. The LP detector achieved high recall, while low-confidence false positives remained a consideration for downstream processing.

  • Vehicle Detection: 97.33% was the best recall rate achieved by Fast-YOLO for vehicle detection, but recognition rates were considered unsatisfactory.The dataset’s variability in vehicle types and positions contributed to the challenge.
  • Vehicle Detection: 100% recall and 99% precision were achieved with YOLOv2 for vehicle detection at a confidence threshold of 0.125.YOLOv2 was selected despite its higher computational cost.
  • LP Detection: A 10% margin was used around predicted vehicle bounding boxes so the entire LP would be included, especially in challenging motorcycle images.A 5% margin was sufficient in validation, whereas the test and LP-detection training used 10%.
  • LP Detection: 98.33% recall was attained for LP detection, with only one vehicle missed because a false positive received higher confidence than the actual LP.The missed vehicle contained 30 frames, and the failure is illustrated in Fig. 9.
  • LP Detection: Post-processing multiple LP detections using character segmentation was considered, but low-confidence true LPs could require analyzing false negatives and increase computational cost.Actual LPs could be detected at confidence levels of 0.1 or lower.

2) LP Detection:

LP detection and subsequent recognition were evaluated on the challenging UFPR-ALPR dataset. The system achieved strong segmentation and recognition results, while temporal redundancy substantially improved full-vehicle recognition.

  • LP Detection: 97.59% recall was obtained when undetected LPs from the previous stage were excluded, compared with 95.97% on the whole test set.LPs were segmented even when tilted or affected by shadows.
  • Character Recognition: 78.33% recognition was achieved with temporal redundancy, compared with 64.89% when frames were processed individually.The best results used 1 pixel of padding and data augmentation for letters and digits.
  • Comparison: 70% recognition was achieved by OpenALPR with temporal redundancy, while commercial systems produced substantial errors on motorcycle images.The results suggest that the commercial systems were not sufficiently trained for motorcycles.
  • Runtime: 35 FPS was achieved despite using YOLOv2 for vehicle detection, compared with 47 FPS using Fast-YOLO.The reported 35 FPS was considered sufficient for cameras recording at 30 FPS.
  • Qualitative Results: Qualitative results included correctly detected and incorrectly recognized LPs, along with successful recognition across different LP categories and lighting conditions.Examples are shown in Figs. 10 and 11.

VI. CONCLUSIONS

The paper presents a real-time YOLO-based ALPR system and a publicly available, richly annotated dataset for realistic Brazilian driving scenarios. Its strongest results occur on SSIG, while UFPR-ALPR remains challenging and broader deployment requires further development.

  • Contributions: The system trains CNNs for each ALPR stage, using separate CNNs for letter and digit recognition.It is presented as a robust, real-time, end-to-end ALPR system based on YOLO object-detection CNNs.
  • Dataset: 4,500 fully annotated images from 150 vehicles form the public UFPR-ALPR dataset, including over 30,000 LP characters in moving-camera, moving-vehicle scenarios.Compared with SSIG, the dataset contains more than twice the images and greater variety.
  • Character Processing: Character segmentation and recognition are identified as the current ALPR bottleneck, addressed with augmentation that simulates vehicle categories and increases underrepresented characters.The authors state these simple strategies were essential for their results.
  • Results: 93.53% full recognition was achieved on SSIG, exceeding previous results and slightly surpassing the commercial-system result of 93.03%.Recognition without temporal redundancy was 85.45%.
  • Results: 78.33% recognition was achieved on UFPR-ALPR, while both commercial systems remained below 70%; the result was still considered unsatisfactory for some real-world applications.This dataset was designed to represent more challenging real-world conditions.
  • Future Work: Future work targets faster detection architectures, alignment correction, vehicle metadata, and a character-recognition module independent of LP layout.The system was conceived and evaluated on two country-specific Brazilian datasets.
Loading 1802.09567v6…