Towards End-to-End Car License Plates Detection and Recognition with Deep Neural Networks

I. INTRODUCTION

• Natural-scene plate reading remains difficult because existing algorithms often work well only under controlled conditions or sophisticated image capture, while scenes bring cluttered backgrounds and variable illumination, distortion, occlusion, or blur.Source: §I · p. 1
• The unified network detects plates and recognizes labels together, shares low-level features, and is trained end-to-end without heuristic rules or intermediate character grouping or separation.Training requires only images, plate positions, and labels.Source: §I · p. 1
• The paper identifies shared convolutional features and joint optimization as reducing parameters and boosting both detection and recognition performance.The integrated system also avoids cropping detected plates for a separate recognition network and takes 0.3–0.4 seconds per image on a Titan X GPU.Source: §I · p. 1

1) License Plate Detection:

• License-plate detection methods are grouped as edge-, color-, texture-, and character-based approaches, with trade-offs including sensitivity to clutter or illumination and higher computational complexity.Character-based methods can achieve high recall but are affected by other text in the background.Source: 1) License Plate Detection · pp. 1–2

2) License plate recognition:

• Earlier recognition pipelines commonly segmented plate characters first and then recognized each character with OCR techniques.Source: 2) License plate recognition · p. 2
• Character segmentation is vulnerable to uneven lighting, shadows, and noise, and improper segmentation can prevent correct recognition even with a strong recognizer.Source: 2) License plate recognition · p. 2
• Segmentation-free approaches recognize the whole plate directly, including sequence-labeling methods using CNN features and RNNs with CTC to label data without character separation.Source: 2) License plate recognition · pp. 2–3

III. MODEL

• The model uses a single forward evaluation for simultaneous plate detection and recognition, jointly optimizing localization and recognition losses.Its components include convolutional features, a plate-specific region proposal network, RoI pooling, detection and box-regression MLPs, and RNNs with CTC.Source: §III · p. 3

1) Low-level Feature Extraction:

• VGG-16 convolutional features retain higher spatial resolution by using two pooling layers instead of five, and serve both detection and recognition.The resulting feature maps are one fourth the original input size.Source: 1) Low-level Feature Extraction · p. 3

2) Plate Proposal Generation:

• The plate-specific RPN uses six anchors per feature-map position, with heights 5, 8, 11, 14, 17, and 20 and width/height ratio 5; two rectangular filters capture local and contextual information.At test time, NMS selects 100 higher-confidence proposals for subsequent processing.Source: 2) Plate Proposal Generation · p. 3
• RPN training samples 256 anchors per image, labels IoU above 0.7 positive and below 0.3 negative, and applies classification and box-regression losses.Regression is performed only for positive anchors; the training loss uses binary logistic classification and smooth L1 regression.Source: 2) Plate Proposal Generation · p. 3

3) Proposal Processing and Pooling:

• Positive proposals associated with one ground-truth plate are combined into an encompassing box, then paired with three times as many sampled negatives to form a balanced training batch.The encompassing boxes become positive samples for later detection and recognition.Source: 3) Proposal Processing and Pooling · p. 4
• RoI pooling converts variable-size proposal features to fixed-size representations using a 4 × 20 grid, preserving horizontal detail for character recognition while constraining parameters and computation.The authors report these features are sufficient for classification and recognition.Source: 3) Proposal Processing and Pooling · p. 4

• The network judges whether each proposed region is a plate and refines its bounding-box coordinates.Source: 4) Plate Detection Network · p. 4
• Two fully connected layers with 2048 neurons and 0.5 dropout encode each region’s features before separate classification and regression heads.Source: 4) Plate Detection Network · p. 4
• The classification head outputs two plate/non-plate probabilities, while the regression head predicts bounding-box coordinate offsets.Source: 4) Plate Detection Network · p. 4

• Plate recognition is framed as sequence labeling with BRNNs and CTC, avoiding character segmentation.Source: 5) Plate Recognition Network · p. 4
• RoI features are transformed into a 512-dimensional sequence of length 19, using a rectangular pooling window suited to narrow characters such as 1 and I [20].Source: 5) Plate Recognition Network · p. 4
• Two separate 512-unit RNN layers process the feature sequence forward and backward to capture context from both directions.Source: 5) Plate Recognition Network · pp. 4–5
• Concatenated hidden states produce probabilities over 37 outputs: 26 capital letters, 10 digits, and a non-character class.Source: 5) Plate Recognition Network · p. 4
• CTC decodes the output sequence by selecting a probable path and removing repeated labels and non-character labels to produce the plate string.Source: 5) Plate Recognition Network · p. 5

• The recognition objective is the negative log probability of the correct plate label, summing probabilities over output paths that map to that label through CTC.Source: B. Loss Functions and Training · p. 5
• Approximate joint training ignores derivatives with respect to proposed-box coordinates; the authors report this has little influence on performance.Source: B. Loss Functions and Training · p. 5
• Training runs for 200K iterations with one randomly sampled image per iteration; VGG-16 initializes low-level CNN features, and ADAM uses learning rates of 10^-5 for pretrained parameters and 10^-4 for others.The first four convolutional layers are not fine-tuned; other CNN layers are fine-tuned only in the first 50K iterations.Source: B. Loss Functions and Training · p. 5

IV. EXPERIMENTS

• Experiments were implemented in Torch 7 and run on an NVIDIA Titan X GPU with 12GB memory.Source: §IV · p. 5

A. Datasets

• CarFlag-Large contains 450K training images and 7,378 test images from fixed frontal surveillance cameras under varied weather and illumination; only the nearest plate is labeled.Source: A. Datasets · p. 5
• AOLP contains 2,049 Taiwan-plate images in AC, LE, and RP subsets; cross-subset training and testing plus rotation and affine augmentation address its small training set.Source: A. Datasets · p. 5
• PKUData contains 3,977 Chinese-plate images from varied scenes, but supports detection evaluation only because its ground truth provides bounding boxes alone.Source: A. Datasets · p. 5
• Figure 3 shows joint detection and recognition examples from CarFlag-Large, AOLP, and PKUData under varied photographing conditions, including day and night and sunny and rainy weather.Source: Figure 3 · p. 6

B. Evaluation Criterion

• End-to-end correctness requires IoU > 0.5 and matching plate labels; detection-only correctness additionally requires the plate to be fully enclosed.The evaluation treats all Chinese plate characters as '*', so the trained network cannot distinguish other Chinese characters.Source: B. Evaluation Criterion · p. 6

C. Performance Evaluation on CarFlag-Large

• The Faster R-CNN detection baseline used 450K training images, while CRNN was fine-tuned on cropped, resized ground-truth plates.Source: C. Performance Evaluation on… · pp. 6–7
• 96.13% end-to-end F-measure on 7,378 CarFlag-Large test images was around 2% higher than the two-stage approach.Source: Table I · p. 7, C. Performance Evaluation on… · p. 7
• 300 ms per image on a single small input scale compared with around 450 ms for the two-stage approach.The unified method avoids cropping and shares convolutional feature extraction rather than recalculating features.Source: Figure 4 · p. 6, C. Performance Evaluation on… · p. 7
• 98.15% detection accuracy from the jointly trained network was 1% higher than the detection-only Faster R-CNN result.Source: C. Performance Evaluation on… · p. 7
• The jointly trained network handled images under different illumination conditions in the reported CarFlag-Large examples.Source: C. Performance Evaluation on… · p. 7

D. Performance Evaluation on AOLP

• On AOLP, the method performed better on AC and LE end-to-end evaluation and had the best plate detection performance on all three subsets.Its detection performance averaged 2% higher than Li et al. [13] and 4% higher than Hsu et al..Source: D. Performance Evaluation on AOLP · p. 7
• 400 ms for both detection and recognition compared with Li et al.'s [13] 2–3 s and Hsu et al.'s average 260 ms.Source: D. Performance Evaluation on AOLP · p. 7
• RP end-to-end performance was worse than [13], possibly because RP plates have substantial rotation and projective orientation that the method does not explicitly handle.The authors identify integrating a spatial transformer network as future work.Source: D. Performance Evaluation on AOLP · p. 7

E. Performance Evaluation on PKUData

• Because PKUData ground truth contains only plate bounding boxes, evaluation there covers detection only, using the model trained on CarFlag-Large.Source: E. Performance Evaluation on PKUData · p. 7
• 99.80% average detection ratio on PKUData, 2% above the previous best, with 100% detection on G4 and an advantage across all five subsets.Source: E. Performance Evaluation on PKUData · p. 7, Table III · p. 8
• The jointly trained network performs better than one trained only with detection information, consistent with detection being boosted by label information.Source: Table III · p. 8, E. Performance Evaluation on PKUData · p. 8
• Yuan et al.’s method is relatively faster than ours because it uses linear SVMs rather than deep CNNs and RNNs.Source: E. Performance Evaluation on PKUData · p. 8

V. CONCLUSION

• The conclusion describes a jointly trained network that detects and recognizes plates in one forward pass, sharing convolutional features to reduce model size without cropping or character separation.Source: §V · p. 8
• Evaluation across three datasets supports the method’s advantage; the authors identify multi-oriented plates and NMS, which takes about half the processing time, as future work.Source: §V · p. 8