The EuroCity Persons Dataset: A Novel Benchmark for Object Detection

Markus Braun; Sebastian Krebs; Fabian Flohr; Dariu M. Gavrila

Source-linked AI summary

The EuroCity Persons Dataset: A Novel Benchmark for Object Detection

Markus Braun, Sebastian Krebs, Fabian Flohr, Dariu M. Gavrila

arXiv:1805.07193v2May 18, 2018cs.CVcs.AIcs.LGcs.RO

TL;DR

Person detection needs larger and more representative benchmarks because existing datasets are limited in size, diversity, detail, or annotation quality. The paper introduces EuroCity Persons and evaluates four deep learning baselines, finding that performance continues to benefit from more diverse data and transfers better after pre-training on the new dataset. Its remaining challenges include small objects, occlusion, duplicate detections, and confusions with depictions, reflections, and clothing.

Problem
Person detection remains an open problem because urban scenes contain substantial variation in pose, clothing, background, weather, time of day, and visibility.
Method
The paper constructs EuroCity Persons and optimizes Faster R-CNN, R-FCN, SSD, and YOLOv3 as baselines while studying data size, diversity, annotation detail, quality, transfer, and errors.
Results
Detectors pre-trained on EuroCity Persons and fine-tuned on target datasets outperform target-only training, improving KITTI by 5-11 points and CityPersons by 2-9 points.
Takeaways & Limitations
Additional diverse training data remains useful for person detection, particularly for non-standard poses, rainy or night-time conditions, and partial occlusions.
Takeaways & Limitations
Small objects, occlusion-related duplicate suppression, and confusions with depictions, reflections, and clothing remain prominent detector error sources.

Abstract

from arXiv · show

Big data has had a great share in the success of deep learning in computer vision. Recent works suggest that there is significant further potential to increase object detection performance by utilizing even bigger datasets. In this paper, we introduce the EuroCity Persons dataset, which provides a large number of highly diverse, accurate and detailed annotations of pedestrians, cyclists and other riders in urban traffic scenes. The images for this dataset were collected on-board a moving vehicle in 31 cities of 12 European countries. With over 238200 person instances manually labeled in over 47300 images, EuroCity Persons is nearly one order of magnitude larger than person datasets used previously for benchmarking. The dataset furthermore contains a large number of person orientation annotations (over 211200). We optimize four state-of-the-art deep learning approaches (Faster R-CNN, R-FCN, SSD and YOLOv3) to serve as baselines for the new object detection benchmark. In experiments with previous datasets we analyze the generalization capabilities of these detectors when trained with the new dataset. We furthermore study the effect of the training set size, the dataset diversity (day- vs. night-time, geographical region), the dataset detail (i.e. availability of object orientation information) and the annotation quality on the detector performance. Finally, we analyze error sources and discuss the road ahead.

1 INTRODUCTION

Person detection remains an open and challenging task, while existing benchmarks are limited by outdated recording conditions, small manually annotated training sets, and dataset bias. EuroCity Persons addresses these limitations with a substantially larger, more diverse, detailed, and quality-controlled benchmark for urban traffic scenes.

Person detection supports intelligent vehicles, surveillance, and robotics but remains difficult because appearance varies with pose, clothing, background, weather, time of day, and visibility.
Existing Caltech and KITTI benchmarks use recording conditions that no longer reflect current image resolution and quality, while their small training sets risk dataset bias and over-fitting.
31 cities across 12 European countries provide geographic coverage, while recordings span day and night and all four seasons, increasing dataset diversity.
Over 238200 manually labeled person instances in more than 47300 images make EuroCity Persons nearly one order of magnitude larger than previously used person benchmarks.
Annotations include bounding boxes, occlusion and truncation tags, body orientation, and object- and image-related tags, with overall accuracy supported by quality-control procedures.
The paper evaluates how training-data volume, temporal and geographic bias, annotation detail and quality, orientation multitasking, and dataset transfer affect detection performance.

2 RELATED WORK

Prior vehicle-context benchmarks and detection methods span established pedestrian datasets, newer diverse benchmarks, two-stage and one-stage detectors, and orientation-estimation approaches. The paper positions EuroCity Persons as a larger, richer benchmark and evaluates four deep learning detector baselines alongside dataset-characteristic analyses.

Person detection benchmarks: Cityscapes provides pixel-wise semantic segmentation, and CityPersons extends part of it with full pedestrian bounding boxes that enable occlusion analysis.
Person detection benchmarks: KITTI and Caltech were established vehicle-context pedestrian benchmarks, while CityPersons and Tsinghua-Daimler Cyclist offered comparatively greater annotation quantity and diversity.
Person detection benchmarks: Caltech includes many annotations, but only an unspecified subset was manually labeled and the remainder was interpolated, with about 2300 unique persons.
Deep learning detectors: Recent detectors divide into two-stage methods with proposal and classification stages and one-stage methods without a distinct proposal stage.
Deep learning detectors: Faster R-CNN combines a region proposal network with downstream processing in a jointly trainable end-to-end network, whereas YOLO uses grid cells and SSD uses default boxes across feature layers.
This work: EuroCity Persons contributes baselines based on Faster R-CNN, R-FCN, SSD, and YOLOv3, and analyzes training size, dataset bias, annotation detail and quality, and error sources.

3 BENCHMARK

EuroCity Persons combines broad urban-traffic coverage with detailed, manually validated annotations and evaluation procedures designed to separate data conditions and reduce overlap across splits.

3.1 Dataset Collection: 53 hours of imagery were collected across 31 cities in 12 European countries using an automotive-grade 1920 x 1024 camera.Frames were sampled every 80th image for the detection benchmark.
3.2 Dataset Annotation: Pedestrians and riders receive tight full-extent bounding boxes, occlusion and truncation labels, class distinctions, ignore regions, and additional event tags.Rider types include bicycle, buggy, motorbike, scooter, tricycle, and wheelchair.
3.2 Dataset Annotation: All annotations were manual and quality-checked, targeting missed and hallucinated objects within 1% and bounding-box accuracy within two pixels.Orientation annotations were required to be accurate within 20 degrees, with independent validation and feedback iterations.
3.3 Dataset Splits: Each city recording is chunked into at least 20-minute segments, then split 60%/10%/30% into training, evaluation, and test data.Splits occur when vehicle speed exceeds 7 km/h and validation is interleaved to reduce repeated people across sets.
3.5 Evaluation Metrics: The benchmark evaluates miss-rate against false positives per image, uses IoU > 0.5 matching, and additionally reports AP and orientation similarity.The log average miss-rate provides a single image-level detection indicator, while AOS is bounded above by AP.

4 EXPERIMENTS

The experiments benchmark optimized detectors on EuroCity Persons, test their transfer across datasets and conditions, and analyze sources of detection error. Faster R-CNN variants generally perform best, while dataset diversity, localization, annotation quality, and post-processing materially affect results.

Experiments use daytime EuroCity Persons pedestrian data for baseline and generalization comparisons, with the same scope applying to many dataset-aspect analyses.
About 100% proposal recall is achieved for reasonable pedestrians larger than 40 px at IoU 0.5, while 1.3× upscaling increases recall especially for smaller pedestrians.
Faster R-CNN variants perform best overall across the three EuroCity Persons test scenarios, while YOLOv3 and SSD outperform R-FCN on at least reasonable and occluded scenarios.Faster R-CNNall performs best overall; Faster R-CNNsmall is best for small pedestrians and slightly better on the reasonable setup.
Localization errors account for about 60% of errors at 6 fppi and about 40% at 4 × 10^-3 fppi, while missed detections commonly involve small or occluded pedestrians.Greedy NMS also suppresses some highly overlapping pedestrians, and adding 10% pedestrian-like false positives causes a 3.3-point performance drop.
ECP pre-training improves transfer performance on KITTI and CityPersons after fine-tuning, including KITTI test AP increasing from 65.9 to 72.6.The authors attribute the benefit to ECP’s diversity, while noting that cross-domain training without fine-tuning performs worse because of dataset bias.
Daytime-trained models perform significantly worse on nighttime data than models trained and tested at the same time of day.The time-of-day experiment uses reduced training sets, and overall results are lower because those sets are comparatively small.

5 DISCUSSION

EuroCity Persons experiments show that detector performance continues to benefit from more diverse training data, while dataset biases, annotation choices, small objects, occlusion, and false positives remain important constraints. The discussion also identifies transfer-learning gains, application-specific factors, and several directions for improving detection and orientation estimation.

Training data and diversity: Training data remains a driving factor: Faster R-CNN and SSD do not saturate even with roughly one order of magnitude more training data than existing datasets.The authors attribute this lack of saturation to EuroCity Persons’ geographic, temporal, and weather diversity.
Training data and diversity: Pre-training on EuroCity Persons improves transfer to KITTI and CityPersons, especially for hard or occluded cases, but target-dataset fine-tuning remains necessary.KITTI average precision increases by 5, 8, and 11 percentage points for easy, moderate, and hard validation sets; CityPersons LAMR decreases by 2, 9, and 7 points for reasonable, small, and occluded sets.
Dataset bias: Night-time performance is worse than day-time performance at equal training sizes, with a LAMR four points higher; near-range recall may also be reduced by vehicle headlights.The discussion links the near-range recall drop to bright spots on lower-body regions caused by the recording vehicle’s headlights.
Annotation quality: Annotation choices materially affect results: correct ignore-region handling boosts performance by 1.5 points, and joint rider-pedestrian training lowers pedestrian miss rates when precise classes are enforced.The 1.5-point annotation effect exceeds the difference observed between using 75% and 100% of the training data.
Error sources and future directions: Small objects and severe occlusion remain difficult, while approximately 75% of false positives at 0.3 fppi are smaller than 80 pixels.Other persistent false positives involve depictions, reflections, and clothing, motivating small-object methods and multi-task networks using broader scene context.

6 CONCLUSIONS

The paper establishes EuroCity Persons as a large, diverse benchmark and finds that additional data continues to improve urban person detection. Its experiments also quantify transfer benefits, environmental and regional variation, orientation-related costs, and promising directions for future progress.

EuroCity Persons provides a benchmark with annotations taken to a new level in quantity, diversity and detail.The dataset supports person detection in urban traffic scenes and serves as the basis for the benchmark.
A Faster R-CNN variant performs best overall, reaching log-averagemiss-rates of 8.1, 17.1 and 33.9 in the reasonable, small and occluded scenarios.
5-11 points on KITTI and 2-9 points on CityPersons are achieved when detectors pretrained on EuroCity Persons are fine-tuned on the target datasets.These improvements exceed performance from training on the target datasets alone.
Detection performance does not saturate at training sizes about one order of magnitude larger than previous datasets, while night performance is a few percentage points lower than daytime performance.Performance also varies significantly across large-scale European regions by a similar order.
Adding orientation estimation lowers detection performance by a single percentage point for the Biternion loss.
Future improvement is expected to come partly from additional data, especially for non-standard poses, rain, nighttime conditions and partial occlusions.The paper also points to annotation accuracy and multi-task networks using scene context and known biases as promising directions.