The Mapillary Traffic Sign Dataset for Detection and Classification on a Global Scale

1. Introduction

• Traffic signs are difficult to detect and classify because of visual confusion, degraded or occluded signs, fine-grained differences, and small size.Source: §1 · p. 2
• Prior traffic-sign studies used benchmarks constrained by country, class count, and diversity in weather, sensors, and seasons.Source: §1 · p. 2
• MTSD contains 100K global images, including more than 52K fully annotated images covering 313 known classes and about 48K partially annotated images.Source: §1 · p. 2
• Pretraining on MTSD boosts binary-detection average precision by 4–6%, which the authors attribute to the dataset’s completeness and diversity.Source: §1 · p. 2
• The baselines use Faster-RCNN with FPN and a separate fine-grained classifier, and the paper studies transfer learning across traffic-sign datasets.Source: §1 · pp. 2–3

2. Mapillary Traffic Sign Dataset

• MTSD combines 52,453 fully annotated images with 47,547 nearby images carrying 87,358 automatically generated labels, totaling 100,000 images and 313 classes.Source: §2 · p. 3

2.1. Image Selection

• Greedy per-country sampling balances continental targets, rural and urban locations, sign-bearing images, capture variation, and distance between selected images.The resulting heat map covers almost all habitable areas, with higher density in populous regions.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2. Annotation Process

• Expert annotators perform image approval, sign localization, and classification, with continuous quality control and a second annotator checking each sign.Quality assurance found corrections needed for 0.5% of boxes, a 0.89% false-negative rate, and a 2.45% false-positive rate; a neural network helps sort templates for classification.Source: §2.2 · p. 4

2.3. Traffic Sign Class Taxonomy

• The 313-class taxonomy groups templates with similar semantics and appearance, separates shared semantics with different appearances, and can be extended incrementally.Source: §2.3 · p. 5

2.4. Partial Annotations

• Partial labels are propagated from manual annotations to API detections in neighboring images using camera-pose estimates and verified sign correspondences.The authors identify the set as a resource for semi-supervised learning and semantic matching.Source: §2.4 · p. 5

2.5. Dataset Splits

• The fully annotated data are split into 36,589 training, 5,320 validation, and 10,544 test images; test annotations are withheld, and class distributions are balanced across splits.The 47,547 partially annotated images are also available for training.Source: §2.5 · p. 5

3.1. Image Properties

• MTSD spans over 200 camera manufacturers, compared with TT100K’s single sensor setup.The camera-device distribution reflects varied image properties.Source: §3.1 · p. 6, Figure 5 · p. 6
• Image sizes range from 1 to more than 16 MPixels, and MTSD includes 1,138 360-degree panoramas.TT100K contains images at only 2048^2 pixels; MTSD also has a larger fraction of low-resolution images.Source: §3.1 · p. 6, Figure 6 · p. 6

3.2. Traffic Sign Properties

• MTSD’s fully annotated set contains 257,543 sign boxes, including 82,724 labels across 313 classes; it has about twice TT100K’s classes and about three times its trainable classes.The remaining boxes include ambiguous, directional, information, highway, exterior, and barrier signs, as well as signs without a selected template or outside the current taxonomy.Source: §3.2 · p. 6
• Sign sizes have an almost uniform distribution up to 2562 px, with MTSD containing a higher fraction of extreme sizes than TT100K.MVD has a similar size distribution but lower overall volume.Source: §3.2 · pp. 6–7
• MTSD has a larger fraction of images with more than 12 signs than TT100K, whose annotations cover only specific sign types.The right panel of Figure 7 groups images by their number of traffic-sign instances.Source: §3.2 · p. 7, Figure 7 · p. 7

3.3. Annotator Interactions

• 52% of MTSD boxes originated from existing automatic detections, but final boxes overlap the originals by only 76% IoU, and annotators added about three new boxes per image.Table 2 reports statistics of manual annotator interactions.Source: §3.3 · p. 7, Table 2 · p. 7

4. Traffic Sign Detection

• A detection matches ground truth at IoU > 0.5, with duplicate matches counted only for the highest-scoring detection; AP is the area under the precision-recall step curve.AP is also reported by sign scale as APs, APm, and APl.Source: §4 · p. 8
• The baseline uses Faster R-CNN with FPN and residual-network backbones, training on full-resolution 1000 × 1000 crops to preserve small signs.Inference uses images scaled to a maximum long side of 2048 or 4000 pixels, or full resolution when the original image is smaller.Source: §4 · p. 8
• MTSD pre-training boosts detection performance by a large margin on both TT100K and MVD, regardless of input resolution.The experiments compare ImageNet and MTSD initialization, with MTSD-initialized models fine-tuned for half as many epochs.Source: §4 · p. 8

5. Simultaneous Detection and Classification

• Evaluation reports mAP over 313 classes, matching boxes without regard to class and excluding other-sign instances.A straightforward 314-way classification head on the binary detector did not yield acceptable performance in preliminary experiments.Source: §5 · p. 8
• The method uses a binary detector followed by a decoupled shallow classifier, addressing scale issues and under-represented class variation in training batches.The classifier has a 314-way output: 313 taxonomy classes plus other-sign, alongside a foreground/background output.Source: §5 · p. 9
• The ResNet101-detector combination reaches 83.4 mAP over all 313 classes; the ResNet50 variant is about 2 points lower.Figure 10 shows typical classification-network failure cases.Source: §5 · p. 9, Table 4 · p. 9
• On TT100K, the two-stage approach outperforms baseline by 8.3 points despite similar binary-detector performance.The reported comparison is in the simultaneous detection and classification results.Source: §5 · p. 9, Table 4 · p. 9
• MTSD pre-training of the classifier and detector further improves the TT100K result.The authors report this as evidence of MTSD’s generalization effectiveness.Source: §5 · p. 9

6. Conclusion

• MTSD includes 100K images with full and partial bounding-box annotations, covering 313 traffic-sign classes from around the world.Source: §6 · p. 9
• For evaluation, prior results were converted to MTSD’s format and evaluated using its metrics.Source: §6 · p. 9
• Baseline experiments found that decoupling detection and fine-grained classification yields superior results on previous traffic-sign datasets.Source: §6 · p. 10
• Transfer-learning experiments show that MTSD facilitates fine-tuning and substantially improves accuracy for traffic-sign datasets in a narrow domain.Source: §6 · p. 10
• The authors frame MTSD as a first step toward global fine-grained traffic-sign research and further research using partial annotations for semi-supervised learning.Source: §6 · p. 10

Scale Supplementary Material

• The author list names Christian Ertler, Jerneja Mislej, Tobias Ollmann, Lorenzo Porzi, Gerhard Neuhold, and Yubin Kuang.Source: Scale Supplementary Material · p. 12
• The listed arXiv version is 1909.04422v2, dated 7 May 2020.Source: Scale Supplementary Material · p. 12

1. Scene Classification for Image Selection

• A ResNet50 scene classifier, pretrained on ImageNet, was trained on filtered BDD100K images labeled residential, highway, or city street to guide image selection.Training used cross-entropy loss and SGD, with the learning rate reduced by a factor of 0.1 when validation accuracy plateaued.Source: §1 · p. 12
• The selection scheme targeted a uniform scene-class distribution, combining city street and residential because even human annotators did not always distinguish them clearly.The authors say this weakly supervised scheme facilitated increasing scene-class diversity among the large pool of candidate images.Source: §1 · p. 12

2. Template Proposal Network

• The proposer ranks template images by predicted similarity to each sign crop, helping annotators navigate a large template set that would otherwise be time-consuming to handle.Source: §2 · p. 12
• The three-layer metric-learning network maps 40 × 40 × 3 image vectors into a 128-dimensional embedding and learns higher similarity for samples in the same template group.For samples from different groups, the similarity should be lower than a margin.Source: §2 · p. 12
• The annotation UI shows crop-based template groups and templates similar to the currently selected template, allowing browsing even when the crop-based proposal misses a match.Source: Figure 2 · p. 13, §2 · p. 13
• The proposer aims to place a matching template and similar templates within the top-k, not necessarily to rank the correct class first.New missing templates could be added to the UI without training data or retraining the proposer network.Source: §2 · p. 13
• Text-based template search was also provided for cases when the proposer failed to suggest good templates.Source: §2 · p. 13

• For each fully annotated training image, nearby Mapillary images within a predefined distance are grouped into a cluster, and OpenSfM estimates their relative camera poses.Source: §3 · p. 13
• Same-label signs form tentative matches; their bounding-box centers are triangulated as 3D points, and pose-based voting selects geometrically feasible correspondences.The triangulation uses traffic-sign centers from corresponding 2D bounding boxes.Source: §3 · p. 13
• Human-verified class labels from fully annotated training signs are propagated to corresponding automatic detections to create the partially annotated dataset.Source: §3 · p. 13

4. Qualitative Examples

• Qualitative detection comparisons use score thresholds chosen to make either recall or precision comparable.Source: §4 · p. 14

4.1. Examples in MTSD

• MTSD examples span cities, highways, residential and rural areas, with varying lighting, weather, and viewpoints.Source: §4.1 · p. 14

4.2. Impact of Transfer Learning

• At precision 0.95, TT100K recall was 0.91 versus 0.81; at precision 0.8, MVD recall was 0.67 versus 0.61 after MTSD pre-training.Source: §4.2 · p. 14