Source-linked AI summary
CityPersons: A Diverse Dataset for Pedestrian Detection
Shanshan Zhang, Rodrigo Benenson, Bernt Schiele
TL;DR
Pedestrian detection still lacks clear evidence about suitable architectures, training data, and cross-dataset performance. The paper adapts FasterRCNN and introduces CityPersons, a diverse Cityscapes-based annotation dataset. The resulting models achieve strong benchmark performance, generalize across datasets, and improve difficult-case detection and localization.
Problem
Existing progress leaves suitable architectures, the benefit of more data, and open-world across-dataset generalization unclear.
Method
The paper adapts FasterRCNN for pedestrian detection and introduces high-quality CityPersons bounding-box annotations on Cityscapes.
Results
CityPersons supports the best reported across-dataset generalization results, while FasterRCNN achieves state-of-the-art benchmark performance and improves small, occluded, and localization outcomes.
Takeaways & Limitations
A diverse CityPersons training set enables one convnet model to generalize across multiple benchmarks and further improve target-dataset detection after finetuning.
Takeaways & Limitations
For fair comparison, the dataset statistics consider only the training subset because annotations for other datasets are not publicly available.
Abstract
from arXiv · showhide
Convnets have enabled significant progress in pedestrian detection recently, but there are still open questions regarding suitable architectures and training data. We revisit CNN design and point out key adaptations, enabling plain FasterRCNN to obtain state-of-the-art results on the Caltech dataset. To achieve further improvement from more and better data, we introduce CityPersons, a new set of person annotations on top of the Cityscapes dataset. The diversity of CityPersons allows us for the first time to train one single CNN model that generalizes well over multiple benchmarks. Moreover, with additional training with CityPersons, we obtain top results using FasterRCNN on Caltech, improving especially for more difficult cases (heavy occlusion and small scale) and providing higher localization quality.
1. Introduction
Pedestrian detection has advanced through benchmarks and convnet variants, but cross-dataset generalization and the value of additional data remain open questions. The paper introduces CityPersons and reports improved detection and generalization results.
- Existing benchmarks have driven progress, but performance in open-world and across-dataset settings remains unclear.
- The authors question whether custom convnet architectures and high-capacity models benefit sufficiently from more data.
- CityPersons adds high-quality pedestrian bounding-box annotations to Cityscapes, covering train, validation, and test sets.
- The paper reports state-of-the-art FasterRCNN results on Caltech and KITTI, with gains especially for small or occluded pedestrians and localization precision.
- CityPersons enables the best reported across-dataset generalization results for pedestrian detection.
- Its diversity spans multiple cities, seasons, weather conditions, and crowd densities, supporting experiments on training and evaluation.
2. A convnet for pedestrian detection
The paper adapts FasterRCNN to pedestrian detection by addressing scale, feature resolution, ignore regions, and optimization. These modifications substantially improve Caltech miss rate while retaining a simple architecture.
- The reference detector is designed as a straightforward FasterRCNN architecture for evaluating subsequent data and training experiments on Caltech-USA.
- 20.98 to 10.27 MRO reduces Caltech miss rate through five adaptations targeting pedestrian-specific detection difficulties.
- RPN scales are quantized using ten equal-sample quantile bins to generate more proposals for the dataset’s many small pedestrians.
- The adapted FasterRCNN gains 10.71 MR points overall and is the simplest variant among architectures reaching approximately 10% MR.
- 3.74 MRO percentage points is gained by up-sampling input images 2x, while larger factors provide no further improvement.
- Removing VGG16’s fourth max-pooling layer reduces feature stride from 16 to 8 pixels, helping the detector handle small objects.
- 1.33 MR percentage points is gained by preventing RPN proposals from sampling ignore regions during training.
3. CityPersons dataset
CityPersons adds high-quality pedestrian annotations to Cityscapes, combining substantial volume, geographic and environmental diversity, fine-grained categories, and heavier occlusion than established datasets. It functions both as a training resource and as a challenging benchmark.
- Dataset construction: CityPersons adds high-quality pedestrian bounding-box annotations to the 5,000-image fine-annotation subset of Cityscapes.The dataset also includes annotations for train, validation, and test sets.
- Bounding-box annotations: Pedestrians and riders receive full-body boxes generated from head-to-feet annotations with a fixed aspect ratio of 0.41, while visible boxes cover segmentation masks.Sitting and other persons receive only segment bounding boxes because no uniform full-body alignment applies.
- Dataset scale: Approximately 35k person and 13k ignore-region annotations are provided across 5,000 images.Annotation density is consistent across the train, validation, and test subsets.
- Diversity: CityPersons spans 18 cities, three seasons, varied weather, and approximately seven persons per image, compared with one person per image in Caltech and KITTI.It also contains approximately 20,000 unique pedestrians, versus approximately 1,300 in Caltech and 6,000 in KITTI.
- Fine-grained categories: Fine-grained labels divide humans into pedestrians, riders, sitting persons, and other persons; pedestrians comprise 83% of the approximately 35k persons.Riders and sitting persons comprise 10% and 5%, respectively.
- Occlusion: CityPersons contains substantially more occlusion than Caltech, including more occluded cases within the commonly used reasonable subset.Caltech has more than 60% fully visible pedestrians, whereas CityPersons has fewer than 30%.
- Baseline experiments: FasterRCNN outperforms ICF detectors on CityPersons, while all tested detectors perform worse there than on Caltech, indicating a more challenging benchmark.Performance continues improving as the amount of training data increases.
- Role of the dataset: CityPersons is positioned as both a large, diverse training database and a challenging test base for future pedestrian-detection research.Its diversity supports generalization to other test sets.
4. Improve quality using CityPersons
The paper uses CityPersons to improve pedestrian detection through cross-dataset training, target-domain pre-training, and semantic-label supervision. These strategies improve generalization, difficult-case detection, localization quality, and small-person performance.
- Cross-dataset generalization: Training on CityPersons improves mean MR across test sets, and CityPersons-trained models generalize better than models trained on Caltech or KITTI.FasterRCNN generalizes better across datasets than ACF when trained with the same data, although it benefits from ImageNet pre-training.
- Cross-dataset generalization: CityPersons serves as effective external training data both without target-domain data and when combined with target-domain fine-tuning.The latter setting further boosts performance.
- Caltech transfer: CityPersons→Caltech improves reasonable-set performance by approximately 1 pp over Caltech-only training.The comparison evaluates a model pre-trained on CityPersons and then fine-tuned on Caltech against one trained only on Caltech.
- Caltech transfer: The gain reaches approximately 9 pp for heavily occluded pedestrians and is also larger for smaller-scale persons.The authors attribute the heavy-occlusion improvement to more occluded training samples in CityPersons.
- Localization quality: Raising the Caltech matching IoU threshold from 0.50 to 0.75 increases the CityPersons-training gain from 1 pp to 5 pp.This result indicates improved alignment of detections associated with the higher-quality CityPersons annotations.
- Caltech state of the art: 5.1% MRN at IoU 0.50 outperforms the previous best reported 7.3% MRN by 2.2 pp on the Caltech test set.At IoU 0.75, the reported gap becomes approximately 20 pp.
- KITTI transfer: On KITTI, CityPersons pre-training improves reasonable evaluation by 2.5 pp, smaller-scale detection by 10.7 pp, and IoU 0.75 performance by 4.1 pp.The gains again favor difficult cases and better-aligned detections.
- Semantic supervision: Adding semantic channels from Cityscapes improves reasonable-set performance by approximately 0.6 pp and helps more with small persons.An FCN-8s model predicts semantic labels from coarse annotations, which are concatenated with RGB channels as detector inputs.
5. Summary
The paper combines an adapted FasterRCNN with CityPersons, a diverse Cityscapes-based dataset, to improve pedestrian detection across datasets and challenging conditions.
- An adapted FasterRCNN achieves state-of-the-art performance on Caltech.
- CityPersons provides diverse pedestrian annotations from 27 cities, 3 seasons, varied weather, and more common crowds.
- Training on CityPersons yields reasonable performance across six benchmarks and supports further gains after target-data finetuning.
- Additional target-data finetuning especially improves detection of small-scale and heavily occluded persons while enhancing localization quality.
- CityPersons also serves as a test benchmark with greater occlusion and environmental diversity, using online evaluation with held-out test annotations.
- CityPersons includes semantic segmentations, stereo and GPS modalities, and neighboring frames; preliminary semantic-label results show promising complementarity.
A. Content
The supplementary material presents additional illustrations, discussions, and experiments, including annotation examples and analyses of annotation statistics and quality.
- The supplementary material contains further illustrations, discussions, and experiments for CityPersons.
- Section B shows examples of the dataset’s annotations.
- Section C analyzes annotation height statistics and annotation quality.
B. CityPersons annotation examples
Figure 10 presents bounding-box and segmentation annotation examples from different cities, illustrating diversity in people’s appearance, clothing, and background objects.
- Figure 10 shows bounding-box annotations and Cityscapes segmentation annotations from different cities.
- The examples illustrate diversity in people’s appearance, clothing, and background objects.
C. Analysis of CityPersons annotations
This section analyzes CityPersons annotation height statistics and annotation quality.
- The analysis covers annotation height statistics and the quality of CityPersons annotations.
C.1. Height statistics
CityPersons has a more diverse pedestrian-height distribution than Caltech, covering a broader range and distributing examples more evenly across scales.
- CityPersons covers a larger range of pedestrian heights because it consists of larger images.
- More than 70% of Caltech pedestrians fall within the [50,100] height bin, whereas CityPersons pedestrians are distributed more evenly across scale ranges.
C.2. Quality
High-quality, aligned bounding-box annotations and explicit ignore-region labels improve detection quality on CityPersons and support generalization to Caltech.
- High-quality annotations improve performance by ~7 pp, with ~6 pp from better alignment and ~1 pp from ignore-region handling.
- Training with aligned bounding-box and ignore-region annotations outperforms training with segment bounding boxes on CityPersons validation.
- Using aligned bounding boxes for training obtains better quality on both CityPersons and Caltech.
- Segment-derived training bounding boxes fail on both CityPersons and Caltech because other benchmarks also provide aligned bounding-box annotations.
- Annotations distinguish pedestrians, riders, sitting persons, other persons, groups of people, and ignore regions through different masks.