Source-linked AI summary
DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images
Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, Ramesh Raskar
TL;DR
Satellite imagery is structured and information-rich but less investigated than everyday imagery, motivating a shared benchmark for satellite image understanding. DeepGlobe 2018 creates three datasets and public competitions for roads, buildings, and land cover, defines evaluation criteria, and provides baselines; by May 15, 2018, the competitions had over 950 registered participants and more than 90 valid submissions.
Problem
Satellite imagery is a structured information source that has received less computer vision attention than everyday-image datasets.
Method
DeepGlobe provides three task-specific datasets, public competitions, evaluation criteria, and preliminary baselines, supported by a collaborative workshop.
Results
By May 15, 2018, more than 950 participants had registered and more than 90 valid submissions appeared across the three competition tracks.
Takeaways & Limitations
The DeepGlobe datasets are intended to serve as fair, durable reference benchmarks for future satellite image analysis research.
Takeaways & Limitations
Land-cover IoU scores can decrease when predictions correctly identify small structures or finer-grained regions absent from the ground truth.
Abstract
from arXiv · showhide
We present the DeepGlobe 2018 Satellite Image Understanding Challenge, which includes three public competitions for segmentation, detection, and classification tasks on satellite images. Similar to other challenges in computer vision domain such as DAVIS and COCO, DeepGlobe proposes three datasets and corresponding evaluation methodologies, coherently bundled in three competitions with a dedicated workshop co-located with CVPR 2018. We observed that satellite imagery is a rich and structured source of information, yet it is less investigated than everyday images by computer vision researchers. However, bridging modern computer vision with remote sensing data analysis could have critical impact to the way we understand our environment and lead to major breakthroughs in global urban planning or climate change research. Keeping such bridging objective in mind, DeepGlobe aims to bring together researchers from different domains to raise awareness of remote sensing in the computer vision community and vice-versa. We aim to improve and evaluate state-of-the-art satellite image understanding approaches, which can hopefully serve as reference benchmarks for future research in the same topic. In this paper, we analyze characteristics of each dataset, define the evaluation criteria of the competitions, and provide baselines for each task.
1. Introduction
DeepGlobe 2018 introduces three public satellite-image challenges spanning roads, buildings, and land cover, designed to bring greater attention and shared benchmarking to satellite image understanding.
- Motivation: Satellite imagery provides structured information, but computer vision research has only recently focused on it compared with everyday-image datasets.Applications mentioned include map composition, population analysis, precision agriculture, and autonomous driving.
- Challenge design: DeepGlobe organizes three datasets, three public challenges, and a collaborative workshop for satellite image understanding.The workshop gathers researchers from diverse fields around related tasks.
- Challenge tracks: The three tracks target road extraction for crisis-response mapping, building detection for urban demographic information, and land-cover classification for development-related applications.The tracks address connectivity, spatial settlement distribution, agriculture, forestry, and urban planning.
- Benchmark structure: The combined datasets contain over 10K satellite images, with defined annotations, data splits, task metrics, and preliminary baselines.The paper describes dataset characteristics, annotation processes, training, validation, and test divisions.
- Participation: More than 950 participants registered and more than 90 valid submissions appeared across the three tracks by May 15, 2018.Competition results were scheduled for presentation at the DeepGlobe 2018 Workshop co-located with CVPR 2018.
2. Datasets
DeepGlobe assembles datasets for road extraction, building detection, and land-cover classification from diverse satellite imagery. The datasets specify annotation procedures, geographic and sensor characteristics, splits, and class-label conventions.
- Road Extraction: Road imagery covers Thailand, Indonesia, and India at 50 cm/pixel resolution using three-channel RGB images from DigitalGlobe +Vivid Images.Images were tiled, sampled across rural and urban areas, and selected to represent varied road surfaces and scene types.
- Road Extraction: 8′570 road images were split into 6′226 training, 1′243 validation, and 1′101 testing images across 2′220 km2.The intended split distribution was approximately 70%/15%/15%.
- Building Detection: The building dataset contains 24′586 nonoverlapping 200m × 200m scenes from Las Vegas, Paris, Shanghai, and Khartoum, totaling 302′701 footprints.Source imagery comes from WorldView-3, combining 31 cm panchromatic and 1.24 m eight-band multispectral data.
- Building Detection: Building areas are easier to model because each area is covered by a single satellite image rather than imagery with varying acquisition conditions.Different sun angles, satellite orientations, atmospheric conditions, and shadow lengths can complicate models used across acquisition times.
- Land Cover Classification: The land-cover dataset provides 1′146 RGB images of 2′448 × 2′448 pixels at 50 cm resolution, covering 1′716.9 km2 of rural areas.The train/validation/test split is 803/171/172 images, and masks contain seven Anderson Classification classes.
- Land Cover Classification: Land-cover masks are pixel-wise annotations sampled to represent all classes, with instances larger than roughly 20m × 20m targeted for annotation.The authors note that small human annotation errors are inevitable and intentionally exclude roads and bridges.
3. Tasks and Evaluations
DeepGlobe defines three satellite-image understanding tasks and their evaluation procedures: road extraction, building detection, and land-cover classification. Each task uses a task-specific output representation and metric, with IoU-based evaluation for road and land-cover segmentation and F1-based evaluation for buildings.
- Road Extraction: Road extraction predicts a same-sized binary road/non-road mask for each satellite image.
- Road Extraction: Pixel-wise IoU is computed per road-extraction image, then averaged across n images using true-positive, false-positive, and false-negative pixels.
- Building Detection: Building detection outputs building polygons and scores detections with an F1 measure using matching based on footprint overlap.
- Building Detection: A detected building is a true positive when its IoU with a ground-truth building exceeds 0.5; buildings of 20 pixels or fewer are discarded as tiling artifacts.
- Land Cover Classification: Land-cover classification is a multi-class segmentation task that predicts an RGB mask whose pixel colors indicate class labels.
- Land Cover Classification: Land-cover evaluation averages pixel-wise IoU across k classes, excluding the unknown class associated with clouds from scoring.
4. State-of-the-art and Baselines
The paper reviews existing approaches for road extraction, building detection, and land-cover classification, then provides preliminary baselines for each DeepGlobe track. These baselines establish reference performance while exposing dataset-specific challenges such as geographic variation, small objects, and annotation mismatch.
- State-of-the-art and Baselines: The authors compare prior datasets and methods for road extraction, building detection, and land-cover classification before presenting DeepGlobe baselines.The baselines are intended to set expected success figures and guide challenge participants.
- Road Extraction: A modified DeepLab with a ResNet18 backbone and Focal Loss achieved the best road-extraction baseline results among the tested approaches.Only simple rotation augmentation was added for this baseline.
- Building Detection: SpaceNet building-detection results averaged IoU above 0.8, but performance was strongest in Las Vegas and weakest in Khartoum.The text attributes the geographic difference probably to structural organization and illumination variance; the algorithm also struggles with small or closely spaced buildings.
- Building Detection: The SpaceNet baseline uses blue outlines for ground truth, green outlines for true positives, and red outlines for false positives across Las Vegas, Khartoum, and Paris examples.The displayed examples emphasize both geographic variation and detection errors.
- Land Cover Classification: The land-cover baseline uses a DeepLab-based CNN with ResNet18, ASPP, batch normalization, rotations, and class weighting, reaching IoU 0.433 at epoch 30.The reported score comes from 512×512 patches and direct segmentation without post-processing.
- Land Cover Classification: The land-cover model correctly classifies some small structures absent from ground truth, while CRF or clustering post-processing could improve IoU scores.The examples also show strong distinctions among farms, rangelands, and forests.
5. Conclusions
DeepGlobe combines datasets, public challenges, and a workshop around road, building, and land-cover understanding from satellite imagery. The authors present the resulting benchmarks and baselines as a basis for collaborative, fairly comparable research across computer vision, remote sensing, and related fields.
- Conclusions: DeepGlobe 2018 provides datasets, challenges, and a CVPR 2018 workshop covering roads, buildings, and land-cover types in satellite images.The paper also analyzes dataset characteristics and challenge evaluation metrics.
- Conclusions: The authors present baselines compared with state-of-the-art approaches and argue that the datasets can become valuable benchmarks for satellite image understanding.They connect these benchmarks to collaborative interdisciplinary research and fair comparison.