PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Large-Scale Traffic-Sign Detection and Recognition

Domen Tabernik, Danijel Skočaj

arXiv:1904.00649v1cs.CV

TL;DR

Traffic-sign inventory management needs detection beyond the small category sets emphasized by prior systems. The paper applies an improved Mask R-CNN to 200 challenging categories in a novel dataset, reporting below 3% error rates and improved overall performance for practical inventory-management deployment.

  • Problem

    Existing traffic-sign systems perform well mainly on a small subset of categories, leaving large-scale detection and recognition for inventory management insufficiently established.

  • Method

    The paper uses Mask R-CNN for end-to-end traffic-sign detection, recognition, and localization, with domain-specific improvements and a 200-category dataset.

  • Results

    Below 3% average error rates are reported, alongside improved miss rate, recall, and mean average precision for the proposed system.

  • Takeaways & Limitations

    The authors conclude that deep learning achieves extremely good performance across many complex traffic-sign categories, including those with large intra-class variability.

Abstract

from arXiv · show

Automatic detection and recognition of traffic signs plays a crucial role in management of the traffic-sign inventory. It provides accurate and timely way to manage traffic-sign inventory with a minimal human effort. In the computer vision community the recognition and detection of traffic signs is a well-researched problem. A vast majority of existing approaches perform well on traffic signs needed for advanced drivers-assistance and autonomous systems. However, this represents a relatively small number of all traffic signs (around 50 categories out of several hundred) and performance on the remaining set of traffic signs, which are required to eliminate the manual labor in traffic-sign inventory management, remains an open question. In this paper, we address the issue of detecting and recognizing a large number of traffic-sign categories suitable for automating traffic-sign inventory management. We adopt a convolutional neural network (CNN) approach, the Mask R-CNN, to address the full pipeline of detection and recognition with automatic end-to-end learning. We propose several improvements that are evaluated on the detection of traffic signs and result in an improved overall performance. This approach is applied to detection of 200 traffic-sign categories represented in our novel dataset. Results are reported on highly challenging traffic-sign categories that have not yet been considered in previous works. We provide comprehensive analysis of the deep learning method for the detection of traffic signs with large intra-category appearance variation and show below 3% error rates with the proposed approach, which is sufficient for deployment in practical applications of traffic-sign inventory management.

I. INTRODUCTION

Traffic-sign inventory management is largely manual, while existing detection systems cover only a small subset of categories. The paper addresses large-scale detection and recognition using deep learning and introduces a 200-category dataset for challenging signs.

  • Manual localization and recognition across thousands of road kilometers is time-consuming; automation could reduce labor and speed detection of damaged or missing signs.
  • Existing traffic-sign solutions mainly target the small category set needed for advanced driver-assistance and autonomous vehicles.
  • Large-scale traffic-sign detection remains open because many categories combine high intra-category appearance variation with low inter-category variation.
  • The paper proposes a CNN-based Mask R-CNN system with end-to-end detection, recognition, and localization, plus domain-specific improvements for traffic signs.
  • The novel dataset contains 200 categories, 13,000 traffic-sign instances, and 7000 high-resolution images, with at least 20 instances per category.

II. RELATED WORK

Prior traffic-sign benchmarks vary widely in task and scale, making large-category detector comparisons difficult. The paper positions its dataset and end-to-end Mask R-CNN approach as broader than existing benchmarks and hand-crafted methods.

  • Existing benchmarks often address recognition without detection or cover only subsets of categories important for ADAS and autonomous vehicles.
  • The proposed dataset contains 200 traffic-sign categories with significant intra-category variability, enabling comprehensive detector analysis for inventory management.
  • The Tsinghua-Tencent 100K dataset contains 45 categories, but its evaluation focuses on simple traffic signs rather than broad category coverage.
  • Earlier work uses hand-crafted features such as HOG, SIFT, LBP, and integral channel features alongside classifiers including SVMs and random forests.
  • The proposed approach uses full feature learning, a region proposal network, deeper VGG16 and ResNet-50 backbones, ImageNet pretraining, and several extensions.

III. TRAFFIC-SIGN DETECTION WITH MASK R-CNN

The paper uses Mask R-CNN as a unified detector and recognizer for traffic signs. Its region proposal and classification modules share convolutional features and are trained end to end.

  • Mask R-CNN combines a Region Proposal Network that generates object proposals with Fast R-CNN, which classifies and refines them.
  • The unified network shares convolutional features between proposal generation and region classification, with the RPN directing Fast R-CNN where to look.
  • Feature Pyramid Networks extend the architecture to improve detection across object scales.
  • Mask R-CNN learns the proposal and classification networks simultaneously through end-to-end stochastic-gradient-descent training.
  • At test time, each input image produces bounding boxes paired with category labels and softmax scores in [0, 1].

B. Adaptation to traffic-sign detection

The traffic-sign adaptation modifies Mask R-CNN training, proposal handling, and data preparation for small signs and large appearance variation. Synthetic distortions augment real examples using estimated geometry and appearance variability.

  • B. Adaptation to traffic-sign detection: Online hard-example mining selects high-loss regions instead of random ROIs for the classification module.
  • B. Adaptation to traffic-sign detection: The RPN training procedure is adjusted to reduce foreground-background and small-large object imbalance in selected regions.
  • B. Adaptation to traffic-sign detection: Sample weighting addresses missed region proposals because regions lost by the RPN cannot be recovered by later classification.
  • B. Adaptation to traffic-sign detection: The detector increases pre-NMS region pass-through from 1000 to 10,000 regions per FPN level, retaining 2000 regions after merging and NMS.
  • C. Data augmentation: Data augmentation modifies segmented real signs with geometric and appearance distortions, including perspective, scale, brightness, and contrast changes.
  • C. Data augmentation: Synthetic distortions sample geometry, appearance, and scale from estimated training-set distributions, using class-specific appearance distributions.
  • C. Data augmentation: Augmented signs are inserted into street-environment backgrounds at random non-overlapping locations, with two to five signs per image.

IV. THE DFG TRAFFIC-SIGN DATASET

The DFG traffic-sign dataset was collected from Slovenian roads and contains 200 categories with substantial real-world variation, controlled splits, and augmentation procedures. It excludes categories or instances that fail quality and size criteria.

  • Data acquisition: Images were collected with a vehicle-mounted RGB camera across rural and urban municipalities in Slovenia.Only images containing at least one traffic sign were selected from the collected corpus.
  • Data augmentation: Synthetic instances were generated by applying geometric distortions and brightness or contrast changes to segmented real-world training samples.The distortions included perspective and scale changes, plus appearance variations.
  • Dataset composition: The dataset contains 6957 images and 13,239 tightly annotated traffic-sign instances across 200 categories.Each image annotates traffic signs larger than 25 pixels using tight polygons.
  • Dataset composition: Approximately 30% of categories exhibit significantly larger appearance variability than the remaining categories.The dataset includes signs with differing real-world appearance while maintaining some visual consistency within each category.
  • Dataset construction: Each category included at least 20 instances with a minimal 30-pixel bounding-box size, and 25% of instances were assigned to testing.For categories with 20 instances, this yielded at least 15 training and 5 test samples.

V. EXPERIMENTAL EVALUATION

The evaluation compares region-proposal-based Faster R-CNN and Mask R-CNN methods, first establishing a baseline on STSD and then evaluating the proposed DFG traffic-sign dataset.

  • Experimental evaluation: The study evaluates Faster R-CNN and Mask R-CNN as state-of-the-art region-proposal-based methods for traffic-sign detection and recognition.The evaluation begins with STSD for baseline comparison before extending to the newly proposed DFG dataset.

A. Implementation details

The implementation uses Detectron-based Faster R-CNN and Mask R-CNN systems, with Mask R-CNN employing ResNet-50 and feature pyramids to retain information about small objects.

  • Model implementation: Both detection systems use publicly available Caffe2-based Detectron implementations and are initialized with ImageNet-pretrained models.Faster R-CNN uses VGG16, while Mask R-CNN uses ResNet-50.
  • Implementation details: The proposed implementation and improvements were made publicly available in a GitHub repository.The repository contains the Detectron-based traffic-sign adaptations.
  • Model implementation: Mask R-CNN uses a Feature Pyramid Network to collect features across layers and preserve information about small objects lost through down-sampling.The architecture contains 50 convolutional layers through ResNet-50.
  • Training configuration: Training uses end-to-end learning with shared momentum of 0.9 and fixed method-specific learning rates and weight decay values.Faster R-CNN uses a learning rate of 0.001 and weight decay of 0.0005; Mask R-CNN uses 0.0025 and 0.0001.

B. Performance metrics

Performance is evaluated primarily with mAP under PASCAL-style and COCO-style overlap criteria, supplemented by precision, recall, and corresponding error rates.

  • Mean average precision: The primary metrics are mAP50 and mAP50:95, averaged over all traffic-sign categories.mAP50 uses a fixed IoU threshold of 0.50, whereas mAP50:95 averages values from 0.50 to 0.95 in 0.05 increments.
  • Mean average precision: mAP50:95 places more emphasis on region-overlap quality than mAP50.The COCO-based metric evaluates a range of IoU thresholds, while the PASCAL-based metric uses only 0.50.
  • Additional metrics: The study also reports precision and recall at the best F-measure, together with false-positive and miss rates.False-positive rate is 1−precision, while miss rate is 1−recall.

C. Comparison to the state-of-the-art

On the Swedish traffic-sign dataset, the proposed Mask R-CNN adaptations outperform the compared detection methods overall, while FCN retains the lowest false-positive rate. The evaluation follows a ten-category protocol using annotated signs of at least 50 pixels.

  • The comparison uses ten categories from STSD, training on Set1Part0 and testing on Set2Part0 with annotations at least 50 pixels in size.
  • 2.3 % false-positive rate is achieved by FCN, while its miss rate is 7.1 %; Faster and Mask R-CNN have miss rates lower by 1 percentage point.
  • 94.9 % mAP50 is obtained by Mask R-CNN with ResNet-50, compared with 94.3 % for Faster R-CNN.
  • 95.2 % mAP50, 2.5 % average false-positive rate, and 3.3 % average miss rate are achieved by Mask R-CNN with the proposed adaptations.
  • 2 pp higher F-measure is obtained by the proposed approach than by the state-of-the-art comparison.

D. Evaluation on DFG traffic-sign dataset

On the DFG dataset, the proposed Mask R-CNN adaptations improve region proposals and full-pipeline detection across 200 traffic-sign categories, including small objects. The remaining full-pipeline misses indicate classification still contributes errors after proposal quality improves.

  • More than 99 % of traffic signs are found by all methods among top-n proposals, but adapted proposals retain higher recall at larger IoU overlaps.
  • Near-zero proposal miss rate is achieved by the adapted method at IoU 0.50, compared with 3 % for the standard approach.
  • Over 95 % mAP50 is achieved with the proposed Mask R-CNN adaptations, while the original Mask R-CNN reaches 93 %.
  • 3.5 % miss rate is achieved with the adaptations without augmentation, down from 5.4 % for the original Mask R-CNN.
  • 84.4 % mAP50:95 is achieved with the adaptations and data augmentation, improving on the original Mask R-CNN’s 82−83 %.
  • The adaptation eliminates the performance gap between small and large signs, while ResNet-101 differs from ResNet-50 by below 0.4 pp.

VI. QUALITATIVE ANALYSIS

Qualitative analysis shows strong detection of traffic signs with substantial variation in content, size, aspect ratio, occlusion, and viewing angle. The weakest cases involve very similar categories, small signs, and severe occlusion.

  • 108 traffic-sign classes achieve 100 % AP50, while 60 additional classes exceed 90 % AP and 23 exceed 80 % AP.
  • The detector successfully handles large-direction signs with varied colors, sizes, aspect ratios, and small instances.
  • Most right-gray-direction instances are detected despite variable content and side-by-side layouts that complicate region proposals.
  • 57 % AP50 is obtained for left-into-right-lane-merger signs, whose near-identical distance-label categories cause apparent false detections.
  • Three of eleven work-in-progress signs are missed, with most missed objects being small and one substantially occluded.
  • Complex full-resolution images containing occluded and small signs still show extremely strong overall detection.

VII. DISCUSSION AND CONCLUSION

The proposed Mask R-CNN system targets large-scale traffic-sign inventory management through adaptations, data augmentation, and a 200-category dataset. It achieves low error rates and is already deployed, while missed detections from classification remain the main area for improvement.

  • VII. DISCUSSION AND CONCLUSION: The system combines Mask R-CNN adaptations and geometric- and appearance-based data augmentation to detect and recognize many traffic-sign categories.The paper also introduces the DFG traffic-sign dataset, containing categories with low inter-class and high intra-class variability.
  • VII. DISCUSSION AND CONCLUSION: 2−3% average error rates were reflected in strong detections across many traffic-sign categories, including complex categories with large intra-class variability.Problematic categories were mainly associated with similarity to other categories, wide viewing angles, and large occlusions.
  • VII. DISCUSSION AND CONCLUSION: The approach was already being deployed for traffic-sign inventory management on Slovenian roads.The paper states that errors can be mitigated by detection across several video frames or by matching 3D locations from stereo cameras.
  • VII. DISCUSSION AND CONCLUSION: Missed detections from the classification network prevent ideal performance and remain the main target for future improvement.The limitation concerns losses occurring after region proposals reach the classification network.
Loading 1904.00649v1…