PaperlayerView exact revision on arXiv

Source-linked AI summary

The Cityscapes Dataset for Semantic Urban Scene Understanding

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele

arXiv:1604.01685v2cs.CV

TL;DR

Existing urban scene datasets do not adequately capture the complexity and diversity needed for semantic scene understanding. Cityscapes introduces a large benchmark with pixel- and instance-level annotations, and shows that an off-the-shelf network trained on it reaches or surpasses prior KITTI and CamVid performance while the task remains challenging.

  • Problem

    Existing urban scene datasets are often smaller and insufficiently representative of complex inner-city street scenes, despite their importance for scene understanding.

  • Method

    Cityscapes constructs a diverse 50-city street-scene benchmark with fine and coarse annotations, stereo depth information, and evaluation for pixel- and instance-level labeling.

  • Results

    An off-the-shelf fully convolutional network trained only on Cityscapes easily reaches and sometimes outperforms state-of-the-art semantic labeling performance on KITTI and CamVid.

  • Takeaways & Limitations

    Cityscapes establishes a demanding urban-scene benchmark whose performance ordering differs from generic datasets, suggesting generic datasets may not suffice for serious progress.

Abstract

from arXiv · show

Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations; 20000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.

1. Introduction

Visual scene understanding remains challenging despite advances, while large-scale datasets have helped drive deep-learning progress. Cityscapes addresses the demands of complex urban driving scenes with richer, more variable annotations spanning pixel- and instance-level labeling.

  • Motivation: Scene understanding spans global scene recognition, local pixel-level scene labeling, and object-centric detection, yet remains challenging relative to human performance.Scene labeling identifies constituent parts and their interrelations, while specialized methods detect subsets of mostly dynamic constituents.
  • Motivation: Large-scale public datasets such as ImageNet, PASCAL VOC, PASCAL-Context, and Microsoft COCO have contributed to deep neural networks’ strong performance across applications.The paper identifies dataset availability as a major factor behind deep learning’s success.
  • Problem: Self-driving cars require highly performant and reliable understanding of complex traffic scenes, motivating substantial research into vision technologies for driving scenarios.Urban autonomous driving involves a particularly popular and challenging scene-understanding application.
  • Contribution: Cityscapes covers complex inner-city street scenes recorded in 50 different cities and exceeds previous efforts in size, annotation richness, scene complexity, and variability.Its annotations and evaluation metrics extend beyond pixel-level semantic labeling to include instance-level semantic labeling.

2. Dataset

Cityscapes was designed to capture highly variable outdoor street scenes through diverse multi-modal recordings and extensive pixel- and instance-level annotations. Its annotation scheme, representative city-level splits, and comparisons with prior driving datasets establish its coverage and scene complexity.

  • Data recording: Several hundreds of thousands of frames were recorded over several months across spring, summer, and fall in 50 cities, primarily in Germany and neighboring countries.Adverse weather such as heavy rain and snow was deliberately excluded.
  • Data modalities: The dataset provides rectified 16 bit HDR and 8 bit LDR stereo pairs, vehicle odometry, outside temperature, and GPS tracks.The stereo camera system used an automotive-grade 22 cm baseline, while LDR images support compatibility with existing datasets.
  • Annotation: 5000 fine annotations and 20 000 coarse annotations provide pixel-level labels, including instance-level labels for humans and vehicles.Fine annotations required more than 1.5 h per image on average, whereas coarse annotations targeted less than 7 min per image.
  • Annotation: The annotation scheme defines 30 visual classes grouped into eight categories, with 19 classes retained for benchmark evaluation.Fine layered polygons also encode object depth ordering and can be extended to additional or more fine-grained classes.
  • Dataset split: The city-level split contains 2975 training, 500 validation, and 1525 test images, balancing geographic location, city population size, and recording time.Coarsely annotated images are used only for additional training, and split representativeness was evaluated with performance variation below 1.5 % points for most characteristics.
  • Dataset comparison: Compared with CamVid, DUS, and KITTI, Cityscapes targets dense inner-city traffic with wide roads and large intersections, yielding different class composition and scene complexity.CamVid has over 700 annotated frames, DUS has 500 annotated images from a 5000-image sequence, and KITTI has approximately 700 independently annotated frames.

3. Semantic Labeling

Cityscapes evaluates per-pixel semantic labeling with both class/category IoU and an instance-level IoU metric designed to address scale variation. Control and baseline experiments show that urban-scene variability, output resolution, and annotation quality strongly affect performance.

  • The task predicts a per-pixel semantic label without modeling higher-level object instances or boundaries.
  • The evaluation reports IoUcategory and IoUclass, while instance-level IoU measures how well individual traffic-participant instances are represented despite scale variation.Global IoU is biased toward large instances, motivating the additional instance-level metric.
  • Control experiments: 21 % IoUclass from nearest-annotation retrieval using 128-times subsampled training labels demonstrates substantial dataset variability and the need for methods that generalize well.The result outperforms static predictors but remains poor.
  • Baseline experiments: Downscaling has a strong, consistent negative influence on performance, with IoUclass ranking strictly following the degree of downscaling.The authors attribute this effect to the dataset’s large scale variation.
  • Baseline experiments: Models trained on 20 000 weakly annotated images achieve comparable IoU performance to models trained on 3475 densely annotated images, but both underperform models trained on all densely annotated images.Coarse-only training remains viable, while many fine labels are important for standard training and testing.
  • Cross-dataset evaluation: A Cityscapes-trained half-resolution FCN matches or outperforms methods specifically trained and specialized for Camvid and KITTI test data without additional training or fine-tuning.

4. Instance-Level Semantic Labeling

Instance-level semantic labeling jointly detects traffic participants and assigns each a confidence score and segmentation mask, extending both object detection and semantic labeling. Performance is assessed with region-level average precision across multiple overlap thresholds, while baseline and oracle experiments examine proposal and classification limitations.

  • Task Definition: The task simultaneously detects objects and segments each instance, treating every instance as a separate semantic label.It extends traditional object detection with per-instance segments and semantic labeling with instance-specific labels.
  • Related Work: Existing approaches fall into segmentation-then-detection, detection-then-segmentation, and simultaneous detection-and-segmentation categories.The categories differ in proposal generation, prediction coupling, cues, and inference methods.
  • Baseline Experiments: 4.6 % is the best main AP score for the evaluated baseline, obtained with convex hull proposals.The baseline trains an FRCN detector to score MCG proposals and tests bounding boxes, region proposals, or convex hulls as instance segmentations.
  • Oracle Experiments: Oracle experiments show that an oracle classifier is only slightly better than FRCN with MCG proposals, whereas perfect proposals yield decent FRCN results.These experiments decouple object localization from classification by replacing proposals or the classifier with ground truth or an oracle.

5. Conclusion and Outlook

Cityscapes is presented as a comprehensive benchmark suite designed to advance semantic urban scene understanding through a large, diverse dataset, sound evaluation, dataset analysis, and state-of-the-art evaluation. Its significance is underscored by performance rankings that differ from generic datasets and by strong results relative to KITTI and CamVid.

  • Contributions: Cityscapes combines the largest and most diverse street-scene dataset to date with high-quality and coarse annotations.The benchmark was designed to spark progress in semantic urban scene understanding.
  • Contributions: Cityscapes develops an evaluation methodology for pixel-level and instance-level semantic labeling.The work also provides an in-depth analysis of the dataset’s characteristics.
  • Significance: Performance rankings for state-of-the-art methods on Cityscapes differ notably from those on generic datasets such as PASCAL VOC.The authors conclude that serious progress in urban scene understanding may not be achievable through such generic datasets.
  • Significance: State-of-the-art semantic-labeling performance on KITTI and CamVid is easily reached and to some extent even outperformed on Cityscapes.The supplied passage presents this as a second observation demonstrating Cityscapes’ significance.

A. Related Datasets

The paper compares Cityscapes with related datasets across annotation types, provided metadata, camera perspectives, scene types, and dataset size. The comparison focuses on datasets that are either large-scale or centered on street scenes.

  • The comparison covers annotation types, metadata, camera perspectives, scene types, and dataset size.
  • The selected related datasets are either large-scale or focused on street scenes.
  • These dataset characteristics are presented in Tab. 7 as a comparison with other related datasets.

B. Class Definitions

Class definitions precisely guided annotation and quality control, with typical examples supplied for each class. The labeling procedure used scene depth and occlusion ordering, shared boundaries, and simple polygonal object representations.

  • B. Class Definitions: Table 8 precisely defines the annotated classes and provides a typical example for each, guiding labeling and quality control.These definitions were used throughout the labeling process and its quality checks.
  • B. Class Definitions: Annotators labeled distant objects first and occluded regions with conservative boundaries, then placed the occluder polygon in front to ensure shared, consistent boundaries.The procedure used scene depth ordering and occlusions to accelerate labeling.
  • B. Class Definitions: Holes revealing background were included as part of the object, allowing objects to be represented by simple polygons forming simply-connected sets.This convention kept the labeling effort within reasonable bounds.

C. Example Annotations

Figure 7 illustrates the Cityscapes dataset’s diversity and difficulty through annotated frames selected from the train and validation splits. The examples represent extremes in the number of traffic-participant instances per scene.

  • C. Example Annotations: Figure 7 shows annotated frames exemplifying the dataset’s diversity and difficulty.All examples come from the train and val splits.
  • C. Example Annotations: The frames were selected by searching for scenes with extreme numbers of traffic-participant instances.Figure 7 provides the corresponding details.

D. Detailed Results

This section expands the main-paper evaluation with class-level scores, baseline training details, and additional qualitative results. It focuses on control experiments and baseline analyses.

  • The section reports additional details on control experiments and baselines.
  • It provides individual class scores to complement the main paper’s aggregated scores.
  • It documents the training procedure for all baselines and presents additional qualitative results for every method.

D.1. Semantic labeling

The semantic-labeling experiments compare baselines trained primarily on monocular Cityscapes images, with custom FCN training and a best-performing Dilated10 architecture emphasizing dilated convolutions and contextual aggregation.

  • Evaluation: Class-level IoU and instance-normalized iIoU scores are reported for all control experiments and baselines, alongside qualitative examples.Tables 9 and 11 provide IoU scores, while Tables 10 and 12 provide iIoU scores; Figs. 8 and 9 show qualitative results.
  • Basic setup: All baselines used single-frame monocular LDR images pretrained on ImageNet before Cityscapes fine-tuning.Most baselines also subsampled the input image, likely because of time or memory constraints.
  • Own baselines: Our FCN experiments used four-stage training from FCN-32s through FCN-8s, then reduced the learning rate by a factor of 10.Each stage initialized from the preceding stage, following the training procedure in [41].
  • Dilated10: The best-performing baseline, Dilated10, combines a VGG-16-based dilated-convolution front end with a context module that expands receptive fields and aggregates contextual information.Its context module has 10 layers and uses C = 19 feature maps per layer.

D.2. Instance-level semantic labeling

The instance-level semantic labeling baselines use Fast R-CNN with proposal regions from MCG or ground-truth annotations. At inference, class scores, thresholding, and non-maximum suppression produce instance masks from bounding boxes, proposal regions, or convex hulls, with results reported using several region-level average precision metrics.

  • D.2. Instance-level semantic labeling: Fast R-CNN baselines use proposal regions from either MCG or the ground-truth annotations.MCG denotes Multiscale Combinatorial Grouping.
  • D.2. Instance-level semantic labeling: Training starts from an ImageNet-pre-trained model, uses a learning rate of 0.001, and stops after validation error plateaus beyond 120 000 iterations.The experiments otherwise use Fast R-CNN’s standard training and testing parameters.
  • D.2. Instance-level semantic labeling: At test time, each object proposal receives one score per class, followed by thresholding and non-maximum suppression.The resulting instance masks are generated from bounding boxes, original proposal regions, or their convex hulls.
  • D.2. Instance-level semantic labeling: The baselines are evaluated with region-level average precision AP, AP50% at 50% overlap, AP100m within 100 m, and AP50m within 50 m.All reported values in Tables 13–16 are percentages and are provided per class.
Loading 1604.01685v2…