Source-linked AI summary
Functional Map of the World
Gordon Christie, Neil Fendley, James Wilson, Ryan Mukherjee
TL;DR
fMoW addresses the limited availability of large, geographically diverse datasets for learning functional building and land-use categories from satellite imagery. It constructs a dataset combining temporal views, multispectral imagery, metadata, and bounding-box-guided classification, then evaluates baseline models using temporal and metadata information. The paper reports that joint reasoning about metadata and temporal information is important, while category bias and unregistered imagery constrain some uses.
Problem
Large datasets have largely focused on ground-level imagery, leaving remote sensing without a large dataset that characterizes its temporal, metadata-rich problems.
Method
The paper constructs fMoW with temporal satellite sequences, multispectral imagery, metadata, and bounding boxes supplied as inputs for classification across 63 categories.
Results
Baseline analyses show the importance of jointly reasoning about metadata and temporal information, while LSTM models generally perform similarly to probability-summing approaches.
Takeaways & Limitations
fMoW provides a publicly released resource for developing models that combine imagery and metadata and can transfer toward real-world detection systems.
Takeaways & Limitations
Several categories retain geographic bias, and unregistered image sequences prevent easy use of methods exploiting pixel-level registration.
Abstract
from arXiv · showhide
We present a new dataset, Functional Map of the World (fMoW), which aims to inspire the development of machine learning models capable of predicting the functional purpose of buildings and land use from temporal sequences of satellite images and a rich set of metadata features. The metadata provided with each image enables reasoning about location, time, sun angles, physical sizes, and other features when making predictions about objects in the image. Our dataset consists of over 1 million images from over 200 countries. For each image, we provide at least one bounding box annotation containing one of 63 categories, including a "false detection" category. We present an analysis of the dataset along with baseline approaches that reason about metadata and temporal views. Our data, code, and pretrained models have been made publicly available.
1. Introduction
fMoW addresses the need for geographically diverse annotated satellite data by combining temporal imagery, multispectral information, metadata, and bounding-box-guided classification.
- Dataset motivation and design: fMoW combines temporal image sequences, multispectral imagery, metadata, and bounding boxes to support reasoning beyond visual information.Metadata includes information enabling reasoning about location, time, sun angles, and physical sizes.
- Dataset motivation and design: Observing multiple temporal views can distinguish a flooded road from a lake or an ordinary road and vegetation scene.
- Dataset motivation and design: The dataset provides at least one bounding box per temporal sequence, mapped to one of 63 categories including false detection.
- Annotation strategy: Bounding boxes are supplied as algorithm inputs rather than detection outputs, allowing collection to focus on high-confidence instances and global diversity.This avoids exhaustive annotation difficulties caused by incomplete map data, dense objects, and unfamiliar aerial viewpoints.
- Release and accessibility: fMoW was released with code, data, pretrained models, and a public challenge, making the dataset and baseline methods broadly accessible.
2. Related Work
Prior large vision datasets primarily use first-person or ground-level imagery, while remote-sensing work has adapted some of those techniques. fMoW relates to earlier temporal, land-use, and aerial-imagery datasets but targets a broader remote-sensing setting.
- Large vision datasets have typically focused on first-person or ground-level imagery, leaving a research gap for remote-sensing data.
- PlaNet formulates image localization as classification over geographic buckets, an approach reused in one fMoW baseline.
- The UC Merced Land Use Dataset provides 21 categories with 100 images per category at roughly 30cm resolution and 256x256 image sizes.
- Related resources include TorontoCity’s seasonal aerial imagery, SpaceNet’s building footprints and points of interest, AID’s 10,000 images across 30 categories, and NWPU-RESISC45’s 31,500 images across 45 categories.
3. Dataset Collection
fMoW was assembled through category selection, geographically diverse location selection, satellite-image filtering, and trusted crowd annotation. The resulting workflow combines multispectral and pan-sharpened imagery with metadata, while validation revealed category-specific geographic bias.
- Category selection: The category-selection process targeted 50–100 categories for a dataset of 1 million images and drew on OSM and NATO feature sources.
- Location selection: Flooded roads, debris or rubble, and construction sites were difficult to collect because open-source data generally lacks temporal information.
- Image selection: fMoW imagery pairs 4-band or 8-band multispectral data with pan-sharpened RGB imagery from the DigitalGlobe constellation.
- Location selection: Location selection used curated, deduplicated volunteered geographic information datasets and removed nearby locations to promote geographic diversity.
- Image selection: The image-selection pipeline searched the DigitalGlobe archive, created image chips, and removed approximately 30% of candidate images for excessive cloudiness.
- Annotation: Trusted crowd users drew category-specific bounding boxes, which a second trusted crowd graded for quality; 642 unique GeoHIVE users participated.
- Annotation: Despite crowd validation and geographic-diversity procedures, some categories retained bias, including limited United States examples for wind farms.
- Metadata: Dataset metadata includes normalized UTM-zone and latitude-band coordinates, timestamp components, viewing and illumination angles, and image-plus-box sizes.
4. Dataset Analysis
fMoW provides multiple image modalities, temporal views, and metadata across geographically diverse locations. Its sequences, imagery formats, and capture times support analysis of objects over space and extended periods.
- Category statistics: Figure 3 reports category instance totals, including false detections, across imagery with different numbers of bands and temporal views.The figure distinguishes the three-band pan-sharpened RGB imagery from the 4- and 8-band imagery.
- Temporal views: 21.2% of sequences contain one view, and 95% contain 10 or fewer images.The distribution of temporal views excludes duplicate counting of pan-sharpened and multispectral images with nearly identical metadata.
- Geographic diversity: fMoW covers much of the globe through data collected from over 400 unique UTM zones.Figure 5 uses green for UTM zones with more sequences and blue for zones with fewer sequences.
- Capture timing: The average time between the earliest and most recent images in a sequence is approximately 3.8 years.The dataset also records distributions of capture years and local capture times.
5. Baselines and Methods
The paper evaluates image-only, metadata-aware, and temporal models for fMoW, including CNN and LSTM approaches. Results indicate that metadata remains useful beyond dataset bias, while multiple views generally outperform single-view predictions.
- Five approaches vary in their use of metadata and temporal reasoning, including metadata-only LSTM, image-only CNN, metadata-fused CNN, and CNN-feature LSTM models.
- Because fMoW sequences are not registered, methods that directly exploit aligned pixel coordinates cannot be used easily.
- The base CNN uses DenseNet-161 initialized with pretrained ImageNet weights and trained with 224x224 crops and Adam at an initial learning rate of 1e-4.
- DenseNet-161 features are fused with metadata through concatenation, two 4096-d fully connected layers with 50% dropout, and a 63-output softmax.
- Metadata improves performance even after removing test instances correctly classified by the metadata-only baseline, indicating benefits beyond bias exploitation.
- Seeing an area multiple times generally outperforms single-view predictions, although this comparison is unfair for categories that may be absent from some views.The results are reported as average F1 scores excluding false-detection scores, with qualitative LSTM-I examples shown in Figure 8.
6. Conclusion and Discussion
The paper presents fMoW as a large, publicly released satellite-imagery dataset combining temporal views, multispectral imagery, and metadata. Its task design supports joint reasoning about these inputs while avoiding the full annotation burden of geographically diverse detection datasets.
- fMoW contains over 1 million satellite images with temporal views, multispectral imagery, and metadata such as timestamps and UTM zones.
- The task lies between detection and classification by supplying bounding boxes, avoiding challenges associated with collecting a large geographically diverse detection dataset.
- The presented methods demonstrate the importance of jointly reasoning about metadata and temporal information.
- The authors publicly release the code, data, and pretrained models and identify potential uses including humanitarian disaster-relief applications.
Appendix Overview
The appendices provide supporting material on metadata, collection details, additional results, and dataset examples.
- Appendix I describes metadata features and their distributions.
- Appendix II supplies additional collection details, Appendix III reports additional results, and Appendix IV provides dataset examples.
Appendix I. Metadata Features and Statistics
The metadata includes geographic, temporal, acquisition, illumination, cloud, and resolution information for satellite image strips.
- fMoW includes 207 of 247 possible ISO Alpha-3 country codes.
- UTM metadata identifies one of 60 longitudinal zones and one of 20 latitude bands.
- Each image records a UTC timestamp in Python datetime format.
- Cloud coverage records the percentage of the image strip obscured by clouds on a 0–100 scale.
- Acquisition metadata records whether the sensor points forward or reverse relative to the orbital path.
- The dataset provides panchromatic and multispectral ground sample distances, including start, end, minimum, and maximum scan-line values.
- Sensor and sun azimuth metadata describe viewing and illumination directions, with range values available across scan lines.
10. Sun Elevation
Sun elevation records the angle of the sun above the horizontal for each image, with minimum and maximum values also provided.
- Sun elevation is measured from the horizontal in degrees.
- The metadata includes minimum and maximum sun-elevation values across all scan lines.
11. Off-Nadir Angle
Off-nadir angle describes the satellite’s viewing angle relative to the image-strip center, while country codes and UTM zones summarize sequence locations.
- 11. Off-Nadir Angle: Off-nadir angle is measured in degrees relative to the center of the image strip.
- 11. Off-Nadir Angle: Metadata records off-nadir angle at the first and last scan lines, plus its minimum and maximum across scan lines.
- Country Codes: Country-code counts are incremented once per sequence rather than once per metadata file.
- UTM Zones: UTM-zone counts are incremented once per sequence rather than once per metadata file.
Appendix II. Dataset Collection
fMoW’s collection process combines geographically diverse candidate locations, crowd validation, seed-guided localization, and bounding-box input to manage difficult global annotations.
- Location Selection: Candidate locations were selected from conflated and curated VGI datasets to remove duplicates and ensure geographic diversity.
- Location Validation: GeoHIVE users validated category presence in satellite images using centerpoint information and keyboard responses for existence, absence, or cloud obscuration.
- Object Localization: Annotators drew object bounding boxes from initial seed points and received class-specific guidance about box size.
- Annotation Challenges: Full-image annotation is difficult because surrounding buildings may have unclear functional purposes and incomplete annotations can make detector scoring unreliable.
- Annotation Challenges: Educational institutions can be difficult to annotate because annotators must decide which buildings belong in one category box.
- Additional Results: Individual-view tests may disadvantage categories absent from some temporal views, including construction sites.
- Dataset Examples: Category examples vary in image quality, recognizability, weather, season, and surrounding context.
- Additional Results: Table 4 reports per-image F1 scores, with averages excluding false-detection scores and sequence-level predictions repeated for most methods.