PaperlayerView exact revision on arXiv

Source-linked AI summary

Simultaneous Traffic Sign Detection and Boundary Estimation using Convolutional Neural Network

Hee Seok Lee, Kang Kim

arXiv:1802.10019v1cs.CV

TL;DR

Traffic-sign detection systems often provide bounding boxes, although pixel-accurate boundaries are needed when signs serve as 3D landmarks. This paper predicts sign pose and shape with a single CNN and projects template boundaries into the image, reporting accurate boundary estimates at mobile-platform frame rates above 7 FPS.

  • Problem

    Previous traffic-sign detectors provide bounding boxes, but 3D mapping requires pixel-accurate boundary corners for triangulation.

    Source: §I · p. 1

  • Method

    A single CNN predicts each sign’s 2D pose and shape class, then projects the corresponding template’s boundary corners into the input image.

    Source: §I · p. 1, C. Boundary Corner Estimation · p. 3

  • Results

    Higher than 0.88 mean average precision (mAP), less than 3 pixels boundary error at 1280 × 720, and higher than 7 frames per second (FPS) on mobile platforms are reported.

    Source: §I · p. 2

  • Takeaways & Limitations

    In simulation, boundary-based mapping gives accurate, stable sign-position estimates across 2D orientation changes, unlike bounding-box mapping.

    Source: F. Mapping Accuracy · p. 11

Abstract

from arXiv · show

We propose a novel traffic sign detection system that simultaneously estimates the location and precise boundary of traffic signs using convolutional neural network (CNN). Estimating the precise boundary of traffic signs is important in navigation systems for intelligent vehicles where traffic signs can be used as 3D landmarks for road environment. Previous traffic sign detection systems, including recent methods based on CNN, only provide bounding boxes of traffic signs as output, and thus requires additional processes such as contour estimation or image segmentation to obtain the precise sign boundary. In this work, the boundary estimation of traffic signs is formulated as a 2D pose and shape class prediction problem, and this is effectively solved by a single CNN. With the predicted 2D pose and the shape class of a target traffic sign in an input image, we estimate the actual boundary of the target sign by projecting the boundary of a corresponding template sign image into the input image plane. By formulating the boundary estimation problem as a CNN-based pose and shape prediction task, our method is end-to-end trainable, and more robust to occlusion and small targets than other boundary estimation methods that rely on contour estimation or image segmentation. The proposed method with architectural optimization provides an accurate traffic sign boundary estimation which is also efficient in compute, showing a detection frame rate higher than 7 frames per second on low-power mobile platforms.

I. INTRODUCTION

Traffic signs can serve as landmarks for mapping and localization, but 3D reconstruction requires pixel-accurate boundary corners. The paper addresses the gap between box-only detection and slower pixel-wise segmentation with a CNN that predicts sign pose and shape for template-based boundary estimation.

I. INTRODUCTION

  • 3D sign reconstruction needs pixel-accurate boundary correspondences for triangulation, whereas earlier detectors provide only bounding boxes and segmentation can be time-consuming.Traffic signs’ standardized appearance also makes them suitable landmarks for mapping and localization.Source: §I · p. 1
  • A single CNN predicts traffic-sign position, 2D pose, and shape labels; template corners are projected into the image to estimate boundaries.The predicted pose and shape label determine which template is used.Source: §I · pp. 1–2
  • Higher than 0.88 mean average precision (mAP), less than 3 pixels boundary error at 1280 × 720, and higher than 7 frames per second (FPS) on mobile platforms are reported.The method uses template shapes and an architecture selected for an accuracy/speed trade-off.Source: §I · p. 2

A. Traffic Sign Detection

  • Earlier traffic-sign detectors used handcrafted features in staged pipelines, while CNN-based detectors applied classifiers to candidate regions and remained dependent on handcrafted candidate extraction,.Handcrafted-feature detector accuracy could also be affected by environmental conditions and hyper-parameters.Source: A. Traffic Sign Detection · p. 2

B. CNN for Object Detection

CNN object detectors localize objects with bounding boxes, but obtaining precise boundaries can require further processing. The proposed method replaces box-coordinate prediction with pose estimation and template-based boundary estimation, avoiding pixel-wise segmentation.

B. CNN for Object Detection

  • Direct CNN detectors predict object position and class from feature layers, while region-proposal methods use candidate regions and are relatively slower.The text notes latency as important for traffic-sign detection under limited computational resources.Source: B. CNN for Object Detection · p. 2
  • CNN detectors usually predict boxes and class labels, so precise boundaries require further processing; pixel-wise masks can be slow.Prior work used an OverFeat -style CNN for simultaneous box, mask, and sign-category prediction, while this method avoids pixel-wise segmentation.Source: B. CNN for Object Detection · pp. 2–3

III. PROPOSED METHOD

  • The proposed CNN predicts 2D poses and shape probabilities from feature maps, then uses them to compute traffic-sign boundary corners.The network is based on SSD and uses separate pose-regression and shape-classification layers.Source: §III · p. 3

A. Convolutional Feature Extraction

The detector uses pretrained CNNs as feature-extraction base networks, prioritizing the accuracy/speed trade-off. For pose prediction, it generalizes box regression to four-point vertex regression, using template vertices to estimate sign boundaries.

A. Convolutional Feature Extraction

  • Pretrained CNN base networks provide feature maps for the detection layers, and network choice prioritizes the accuracy/speed trade-off.The experiments use VGG16 and InceptionV2; SSD-style pose and shape layers operate across multiple feature-map resolutions to detect signs at different scales.Source: A. Convolutional Feature Extraction · p. 3

B. 2D Pose Prediction

  • The method generalizes box regression to an 8-dimensional regression vector for four points that determines a planar target’s 2D pose.Regression values are offsets relative to default-box coordinates.Source: B. 2D Pose Prediction · p. 3
  • Virtual template vertices define the sign’s minimum bounding quadrilateral and serve as regression points for precise boundary estimation.At inference, the network outputs are used to recover the template vertices.Source: B. 2D Pose Prediction · p. 3

C. Boundary Corner Estimation

The method combines a predicted shape label and 2D pose to recover sign boundaries from shape-specific templates. It transforms the template boundary corners into the input image coordinates.

C. Boundary Corner Estimation

  • The predicted shape label selects a corresponding template, whose known boundary corners are transformed into the image coordinates using the estimated 2D pose.The shape classifier outputs N-way softmax values for rectangle, diamond, octagon, and background in the experiments.Source: C. Boundary Corner Estimation · pp. 3–4
  • The transform is computed from predicted template vertices and normalized template-image corners, then applied to each template boundary corner.Template width and height are normalized to 1; an octagon template has eight boundary corners.Source: C. Boundary Corner Estimation · p. 4
  • Triangles and shields can use an affine transform, while shapes with more than three corners can use a perspective transform.For circular signs, the paper describes using four endpoints of the observed ellipse’s major and minor axes to compute the perspective transform.Source: C. Boundary Corner Estimation · p. 4
  • Experiments use rectangle, diamond, and octagon signs; the paper reports that rectangle and diamond types account for more than 90% of U.S. traffic sign types.The shaded shapes in Fig. 5 are the shapes used in the experiment.Source: C. Boundary Corner Estimation · p. 4, Figure 5 · p. 4

D. Training

Training combines shape classification and vertex regression, with pretrained base networks and hard negative mining. The SDTS data includes annotated signs and excludes occluded or partially observable signs from training.

D. Training

  • Default boxes are positive when their intersection over union with a ground-truth sign exceeds 0.5; unmatched boxes are negative samples.Base networks are pretrained on ImageNet and fine-tuned, while pose-regression and shape-classification layers use Xavier initialization.Source: D. Training · pp. 4–5
  • The overall loss combines shape-classification loss for positive and negative samples with vertex-regression loss for positive samples.Shape classification uses softmax loss, and vertex regression uses smooth L1 loss.Source: D. Training · p. 5
  • Hard negative mining selects the highest-scoring negatives, using K_n = 3K_p; optimization runs for 60,000 iterations with SGD.The learning rate drops from 10^-3 to 10^-4 after 48,000 iterations; batch size is 32.Source: D. Training · p. 5

E. Dataset Manipulation

  • SDTS contains 37,079 road scenes, split into 33,360 training and 3,719 test images captured on different routes, with sign corners and shape labels annotated.The images were collected around San Diego, and slight radial distortion was not corrected for annotation, training, or testing.Source: E. Dataset Manipulation · p. 5
  • The authors do not use GTSDB or LISA because those datasets provide only bounding-box annotations.Their SDTS annotations include traffic-sign boundary corners and shape labels.Source: E. Dataset Manipulation · p. 5
  • The authors exclude images containing occluded or partially observable signs from training because their boundary corners are difficult to annotate accurately.They report that the trained detector can still estimate boundaries for such signs; the dataset manipulation also includes pruning, augmentation, and hard negative mining.Source: 1) Pruning training data · p. 5, E. Dataset Manipulation · p. 5

1) Pruning training data:

Training-data preparation uses cropping and perspective transforms to address variation in sign position, size, and viewpoint, while adding mined negative-only images to improve robustness to false positives.

2) Data augmentation:

  • Perspective transforms augment close, large-sign examples to address their scarcity and stronger perspective distortion as signs approach the camera.The authors generated 10 transformed images per selected original, yielding 6,200 augmented images from about 620 of 33,360 training images.Source: 2) Data augmentation · pp. 5–6

1) Pruning training data:

  • Image cropping is used to make detection less sensitive to traffic-sign position and size.The passage also identifies perspective transforms as a means to improve robustness to viewpoint changes.Source: 2) Data augmentation · p. 5

2) Data augmentation:

  • The authors add the top 5,000 hard negatives from sign-free images to the initial 33,360 training images.They mine these examples by running an initial model on images without traffic signs; the confusing objects include advertisements and some trees.Source: 3) Hard negative mining · p. 6

3) Hard negative mining:

The detector builds on SSD, strengthens its lower-resolution prediction layers with additional convolutions, and is evaluated using Caffe for training and Snapdragon hardware for testing.

3) Hard negative mining:

  • The detector uses SSD feature layers at six resolutions, with default boxes and non-maximum suppression to filter redundant predictions.The six feature-layer scales range from 1/8 to 1/256 of the network input.Source: 3) Hard negative mining · p. 6
  • Additional convolutional layers before pose-regression and shape-classification layers strengthen the 1/8- and 1/16-scale prediction paths.The paper says these lower layers are shallower and that the extension is intended to help detect frequently occurring small signs.Source: 3) Hard negative mining · p. 6

B. Experimental setup

  • Training used Caffe, while testing used Qualcomm’s Snapdragon Neural Processing Engine on a Snapdragon 820A processor.RGB images and different base-network input resolutions were used to evaluate the accuracy–speed trade-off.Source: B. Experimental setup · pp. 6–7

C. Accuracy Evaluation

  • Table II is identified as a comparison of bounding-box accuracy on GTSDB, but the supplied passage gives no table values.The accuracy-evaluation section introduces VGG16 as the base network for its initial evaluation.Source: Table II · p. 7, C. Accuracy Evaluation · p. 7

1) Evaluation protocol:

Evaluation distinguishes bounding-box detection from boundary estimation and reports results on GTSDB and SDTS, including comparisons across sign shapes and qualitative examples.

1) Evaluation protocol:

  • The VGG16 detector achieves the best average box overlap across GTSDB sign categories, with competitive area under the precision-recall curve.The benchmark comparison is limited to box annotations because the method requires polygon annotations for training and average-vertex-error evaluation.Source: 2) ‘GTSDB’ test dataset · p. 7

2) ‘GTSDB’ test dataset:

  • Both SDTS models exceed 0.8 mAP at 0.5 IoU and approach 0.8 mAP at 0.7 IoU; their mAP-versus-IoU curves are reported in Figure 9.The SDTS test set contains 3,719 images and 6,324 traffic-sign annotations; the figure caption describes a comparable detection rate for the optimized InceptionV2 detector and VGG16.Source: 3) ‘SDTS’ test dataset · p. 7, Figure 9 · p. 7

3) ‘SDTS’ test dataset:

  • Rectangle signs have worse detection accuracy than other shapes despite having substantially more training samples.The authors suggest varied appearances and lookalike objects such as advertisements as possible explanations.Source: 3) ‘SDTS’ test dataset · p. 7, Table III · p. 7
  • At 800 × 450 input resolution, signs smaller than 13 pixels per side are ignored; examples show detections and boundary estimates across sizes and shapes, alongside false positives on lookalikes.The segmentation-based comparison uses detected boxes to crop patches, then applies segmentation, polygon fitting, and simplification.Source: 3) ‘SDTS’ test dataset · p. 7, 4) Comparison with segmentation-based… · p. 7

4) Comparison with segmentation-based methods:

The paper compares its boundary estimates with a segmentation-based approach, reports timing for both, and tests an additional local-refinement step.

4) Comparison with segmentation-based methods:

  • 2.7014 pixels AVE for the VGG-16 model, versus 3.2917 pixels for the segmentation-based method.The segmentation approach requires an additional boundary-fitting process because its masks cannot be restricted to traffic-sign shapes.Source: 4) Comparison with segmentation-based… · p. 8
  • 126ms per image for the proposed method, versus 189ms for segmentation before boundary fitting is counted.The segmentation timing comprises 106ms for SSD and 83ms for the segmentation network; total time can increase depending on the fitting algorithm.Source: 4) Comparison with segmentation-based… · p. 8

5) Sign boundary refinement:

  • Local refinement reduces average vertex error from 2.679 to 2.256 and takes 1.2ms per image on PC.The refinement estimates a local 2D transform using patch gradients and discards results far from the network prediction.Source: 5) Sign boundary refinement · p. 8
  • The authors note that CNN-based detection requires many training images and that it is unclear whether their prepared dataset is sufficient.The passage describes more than 44,000 images, including annotated, negative-only, and augmented examples.Source: 6) Volume of training data · p. 8

6) Volume of training data:

Training-data experiments found only marginal detection-accuracy gains above 15,000 images. The speed studies reduce computation through lighter networks and input-resolution trade-offs, with higher frame rates at lower resolutions.

6) Volume of training data:

D. Speed Optimized Network

  • Replacing VGG16 with InceptionV2 reduces base-network computation from more than 100 billion MAC to about 14 billion MAC per forward pass at 800 × 450 resolution.InceptionV2 initially yields less than 3 FPS, motivating further network simplification.Source: D. Speed Optimized Network · p. 8, 1) Lighter base network · p. 8

1) Lighter base network:

  • Removing all upper layers from inception4b reduces MAC by almost half while marginally degrading detection accuracy.The experiments report that removing most upper InceptionV2 layers does not significantly affect detection accuracy.Source: 1) Lighter base network · p. 9
  • 533 × 300 input runs near 5 FPS with competitive accuracy; 400 × 225 exceeds 7 FPS with a little more accuracy sacrificed.The paper identifies around 5 FPS as a possible near-real-time option when paired with robust object tracking.Source: 2) Input image resolution · p. 9

2) Input image resolution:

The speed-up strategy exploits the traffic-scene relationship between sign size and image position, combining a center crop with a resized image. The paper also tests SSD design choices that affect accuracy and computation.

2) Input image resolution:

  • Traffic-scene sign size correlates with position: distant, smaller signs tend to appear near the image center, while closer, larger signs appear near the border.This pattern motivates using differently sized inputs for the two detectors.Source: 3) Detection speed up by input cropping · p. 9

3) Detection speed up by input cropping:

  • A center-cropped image targets distant signs near the center, while a half-resized image targets larger signs near the borders; together, the detectors take two-thirds the single-detector running time.The combined method uses half as many total input pixels, and its results are described as comparable to detection on original images with much less time.Source: 3) Detection speed up by input cropping · pp. 9–10, Figure 13 · p. 10

E. Model Analysis

  • Perspective-transform augmentation improves mAP by 1.1%, and adding lower-path convolutional layers improves mAP by 0.9%.Five default-box aspect ratios—1, 2, 3, 1/2, and 1/3—produce the best model; lighter added layers reduce MAC with marginal accuracy degradation.Source: E. Model Analysis · p. 10

F. Mapping Accuracy

In simulation, mapping based on precise boundary estimates remains accurate as sign orientation changes, while bounding-box-based mapping errors grow. The paper concludes with a 7 FPS mobile detector and identifies generalization to other standard-shaped planar objects as future work.

F. Mapping Accuracy

  • The comparison uses synthetic 1280 × 720 image pairs, forward camera motion, z-axis rotation, and Gaussian observation noise with 1-pixel variance.The reported reconstruction-error curves show averages and standard deviations across 100 simulations.Source: F. Mapping Accuracy · p. 11, Figure 14 · p. 11
  • As 2D sign orientation changes, bounding-box mapping errors grow, while boundary-based mapping remains accurate and stable.The comparison varies orientation through in-plane camera rotation and supports the paper's conclusion that boundary estimation is crucial for robust 3D mapping from 2D observations.Source: F. Mapping Accuracy · pp. 10–11, Figure 14 · p. 11

V. CONCLUSIONS

  • The proposed detector estimates sign locations and boundaries together and runs at 7 FPS on low-power mobile platforms; generalization to other planar objects with standard shapes remains future work.The conclusion also identifies newer detection architectures and traffic-sign-specialized base networks as directions for improving accuracy and speed-accuracy trade-offs.Source: §V · p. 11
Loading 1802.10019v1…