Source-linked AI summary
Reading Car License Plates Using Deep Convolutional Neural Networks and LSTMs
Hui Li, Chunhua Shen
TL;DR
The paper tackles license plate detection and recognition in natural-scene images, where varied appearance and complex backgrounds make conventional methods difficult. It uses a CNN cascade for character-based detection and plate verification, then applies CNN features with an LSTM sequence-labeling model for segmentation-free recognition. The system reports high detection precision and recall, while recognition achieves appealing performance with sufficient training data, but the authors note that detection is not yet real time.
Problem
License plate detection and recognition remains challenging in open environments because of appearance variation, image degradation, occlusion, blur, and complicated backgrounds.
Method
A CNN cascade detects characters and rejects non-plate text, while CNN-extracted sequential features are recognized by an RNN/LSTM with CTC without character segmentation.
Results
The detector achieves 95.24% recall and 97.56% precision on the Caltech cars dataset, while the sequence-labeling method produces impressive recognition performance given sufficient training data.
Takeaways & Limitations
Combining CNN feature learning with sequence labeling supports license plate recognition without pre-segmenting characters and uses context information during recognition.
Takeaways & Limitations
The detection system still cannot be used in real time, despite GPU assistance.
Abstract
from arXiv · showhide
In this work, we tackle the problem of car license plate detection and recognition in natural scene images. Inspired by the success of deep neural networks (DNNs) in various vision applications, here we leverage DNNs to learn high-level features in a cascade framework, which lead to improved performance on both detection and recognition. Firstly, we train a $37$-class convolutional neural network (CNN) to detect all characters in an image, which results in a high recall, compared with conventional approaches such as training a binary text/non-text classifier. False positives are then eliminated by the second plate/non-plate CNN classifier. Bounding box refinement is then carried out based on the edge information of the license plates, in order to improve the intersection-over-union (IoU) ratio. The proposed cascade framework extracts license plates effectively with both high recall and precision. Last, we propose to recognize the license characters as a {sequence labelling} problem. A recurrent neural network (RNN) with long short-term memory (LSTM) is trained to recognize the sequential features extracted from the whole license plate via CNNs. The main advantage of this approach is that it is segmentation free. By exploring context information and avoiding errors caused by segmentation, the RNN method performs better than a baseline method of combining segmentation and deep CNN classification; and achieves state-of-the-art recognition accuracy.
1 Introduction
The paper addresses license plate detection and recognition in difficult natural scenes using CNN-based detection and segmentation-free sequence recognition. Its cascade combines character detection with plate verification, while an LSTM-based model recognizes the whole plate as a sequence.
- Motivation: Open-environment LPDR remains difficult because character variation, viewpoint distortion, poor image quality, occlusion, blur, and cluttered backgrounds complicate detection and recognition.Existing systems often work only under controlled capture conditions.
- Problem definition: A complete LPDR system comprises plate detection, which produces bounding boxes, and recognition, which identifies characters inside them.
- Framework overview: The paper presents segmentation-based baseline and LSTM-based recognition methods as independent alternatives within the overall framework.
- Detection framework: The proposed cascade first detects characters with a 37-class CNN and then uses a plate/non-plate CNN to reject general text and other false positives.Character regions are clustered to generate text-like candidate regions before plate verification.
- Recognition framework: The recognition contribution treats license characters as a sequence-labelling problem, avoiding character segmentation while using CNN features and an RNN with CTC decoding.The model concatenates local and global CNN features and employs a bidirectional LSTM network.
2 Related work
Prior license plate methods use handcrafted visual cues, segmentation pipelines, and character classifiers, each with documented weaknesses in complex scenes. The paper positions CNN character-based detection and RNN sequence recognition as alternatives that avoid some of these limitations.
- License plate detection: License plate detection research includes edge-, color-, texture-, and character-based approaches for locating candidate plate regions.These categories exploit different visual properties such as edges, color contrast, texture distributions, or character presence.
- License plate detection: Edge-based methods are fast but sensitive to unwanted edges and ineffective when plates are blurry.
- License plate detection: Character-based detection can provide high recall, but general text in image backgrounds substantially affects performance.
- License plate recognition: Conventional license plate recognition typically segments characters before recognizing each segment with OCR or a classifier.Segmentation methods include projection-based and connected-component approaches.
- License plate recognition: Recognition classifiers include template matching and learning-based methods using features such as density, gradients, direction, and LBP.Learning-based approaches can handle differences in font, illumination, and rotation more robustly.
- CNNs and RNNs: RNNs have been used for sequential recognition without segmentation, motivating the paper’s CNN-plus-RNN approach with local and global features and no language model or lexicon.
3 Car License Plate Detection
The detection pipeline uses character-focused CNNs to generate high-recall candidates, filters false positives, and refines bounding boxes using plate-edge information.
- Candidate generation and verification: A character-based detector uses CNNs to identify plate characters, while a second plate/non-plate CNN rejects text-like false positives.The cascade is designed to preserve recall during candidate generation and improve precision during verification.
- Candidate generation and verification: The 4-layer detector classifies 26 uppercase letters, 10 digits, and a non-character class across image patches.Its outputs form a character saliency map used for subsequent candidate extraction.
- Candidate generation and verification: Input images are evaluated at 12 scales, with padding and RLSA-based grouping used to generate character-string bounding boxes.Connected-component analysis then produces initial candidate boxes from grouped probability peaks.
- Candidate generation and verification: Geometric constraints, saliency scores, and bounding-box-level NMS filter the generated candidates before plate verification.Boxes scoring below the average box score are eliminated.
- Bounding-box refinement: Bounding-box refinement uses vertical-edge detection and horizontal and vertical projections to locate the plate boundaries more accurately.The procedure addresses boxes that include textual background or fail to contain the whole plate.
- Candidate generation and verification: The plate/non-plate CNN verifies remaining candidates using positive license-plate samples and negative non-text or general-text patches.Only candidates classified as plates proceed to recognition.
4 Car License Plate Recognition
License plate recognition is treated as a second LPDR stage with two alternatives: segmentation followed by character classification, or segmentation-free sequence labelling of the whole plate.
- Recognition approaches: The recognition stage compares a segmentation-based baseline with a sequence-labelling method that recognizes all plate characters at once.The sequence approach avoids the challenging character-segmentation step but requires many labelled license plates for training.
- Recognition approaches: The segmentation-free alternative treats the plate as an unsegmented character sequence and uses CNN features with an RNN and CTC output layer.This design is intended to exploit sequence context while avoiding segmentation errors.
- Recognition approaches: The segmentation-based baseline first segments characters and then recognizes each character individually.It provides the comparison method for evaluating whole-plate sequence recognition.
4.1 Character Segmentation Based Plate Recognition
The segmentation-based recognizer enhances and binarizes cropped plates, extracts connected character blocks, and classifies each segment with a deeper CNN.
- Image preprocessing: Cropped plate images undergo intensity adjustment before binarization to increase contrast for later processing.The adjustment saturates 1% of pixels at both low and high original intensities.
- Image preprocessing: The method prefers local thresholding, selecting Non-linear Niblack’s algorithm after evaluating several binarization methods.Global thresholding is considered vulnerable to unbalanced illumination.
- Image preprocessing: Binary images use white pixels as foreground because connected-component analysis forms blocks from white regions.The image may be inverted under the assumption that plate background occupies more area than text.
- Character segmentation: Connected-component analysis groups pixels into character blocks but can create false blocks or split or merge characters under plate artifacts and uneven lighting.The procedure removes such false blocks before recognition.
- Character recognition: A 9-layer CNN performs 36-way character recognition on segmented candidates, excluding the non-character class.The model contains six convolutional and three fully connected layers followed by softmax probabilities.
- Character recognition: Local binary pattern features are concatenated with grayscale input to provide a 24 × 24 × 2 representation for recognition.LBP is used as a texture feature intended to be robust to illumination variation.
- Character recognition: Recognition predictions are averaged over five randomly cropped neighborhood patches around each segmented character.The final result uses the average softmax probabilities.
- Character recognition: When more than the country-specific maximum number of character blocks is detected, lower-probability blocks are rejected.The system returns a result constrained to N characters.
4.2 Sequence Labelling based Plate Recognition
The paper treats license-plate recognition as sequence labelling, avoiding character-level segmentation by combining CNN feature extraction with bidirectional LSTM processing and CTC decoding.
- Sequence Labelling based Plate Recognition: Character segmentation is avoided by viewing the license plate as an unsegmented sequence and recognizing the entire plate directly.The approach addresses segmentation errors caused by uneven lighting, shadows, and noise.
- Sequence Feature Generation: A pre-trained 9-layer CNN slides across the plate to extract sequential feature vectors while preserving left-to-right order and contextual information.The resulting sequence is x = {x1, x2, ..., xL}, with each xt in R256.
- Sequence Labelling: Bidirectional LSTMs process each feature using both forward and backward contextual information to model dependencies across the character sequence.LSTM memory cells and gates support long-range dependencies, while the two hidden layers share the output layer.
- Sequence Labelling: A soft-max layer converts LSTM states into probability distributions over 37 classes for every position in the input sequence.The output probability sequence has the same length as the input feature sequence.
- Sequence Decoding: CTC decodes the recurrent network’s probability sequence directly into character strings without requiring pre-segmented training data.It selects likely paths and removes repeated and non-character labels to produce the final sequence.
5 Experiments
Experiments evaluate the CNN-based detection and recognition system on two datasets using precision, recall, and recognition accuracy. The results show efficient character classifiers, strong plate-detection performance, complementary CNN features, and an advantage for sequence labelling under difficult plate conditions.
- 5.1 Dataset: The experiments use Caltech Cars (Real) and AOLP, whose subsets cover cluttered backgrounds, varied illumination and weather, roadside enforcement, and arbitrary viewpoints.AOLP contains 2049 Taiwan-plate images divided into AC, LE, and RP subsets.
- 5.2 Evaluation Criteria: The system is evaluated with precision and recall for detection, plus plate-level and character-level recognition accuracy.Precision measures false alarms, recall measures detected groundtruth objects, and plate-level recognition requires every character to be correct.
- 5.3 Character Classification Performance: 3−5 times faster in training and testing, our smaller CNN has comparable classification accuracy to Jaderberg et al.’s model while using fewer parameters.The comparison concerns the 37-way character classification model.
- 5.3 Character Classification Performance: LBP features reduce classification error, and combining CNN I and CNN II improves accuracy at both character and plate levels because their learned features are complementary.CNN I uses grayscale images, whereas CNN II includes LBP features.
- 5.4 License Plate Detection: 95.24% recall and 97.56% precision are achieved on Caltech Cars, outperforming all five comparison methods on both datasets and both detection metrics.The recall is 4.77 percentage points above the second-best method, while precision is 2.06 points higher; processing takes about 5 seconds per Caltech image and 2−3 seconds per AOLP image with GPU.
- 5.5 License Plate Recognition: Sequence labelling improves recognition by avoiding character separation and exploiting context, with experiments demonstrating its superiority over the segmentation-based method.BRNN output maps produce clearer and more correct character probabilities than the CNN soft-max outputs.
6 Conclusion
The system combines CNN-based detection with deeper recognition models for license plates in natural scenes. It reports strong performance with sufficient training data, while detection speed remains a limitation for real-time use.
- Conclusion: The 4-layer CNN detects characters with 37-class outputs, while the deeper 9-layer CNN supports more accurate recognition.The conclusion contrasts the smaller model for fast detection with the deeper model for accurate recognition.
- Conclusion: BRNN confidence maps provide better recognition results, allowing characters on license plates to be read directly.
- Conclusion: The recognition approach produces impressive performance when sufficient training data is available.
- Limitations: Detection speed remains insufficient for real-time use, motivating future work on reducing the detection area with proposal-based approaches.
- Conclusion: The method demonstrates robustness across varied illuminations and orientations on the AOLP dataset.