Source-linked AI summary
Satellite Image Classification with Deep Learning
Mark Pritt, Gary Chern
TL;DR
Satellite-image analysis requires automation because large geographic areas and limited analysts make manual searches impractical, while traditional methods are too inaccurate and unreliable. The paper develops an ensemble-CNN system that combines image features with satellite metadata for 63-class fMoW classification. It reports 83% accuracy, a 0.797 F1 score, and 15 classes with at least 95% accuracy.
Problem
Large satellite-image search tasks require automation, but traditional object detection and classification methods are too inaccurate and unreliable for the problem.
Method
The system ensembles CNNs and combines their image features with satellite metadata using additional neural networks.
Results
83% total accuracy, a 0.797 F1 score, and at least 95% accuracy for 15 classes were achieved on fMoW validation data.
Takeaways & Limitations
The system classifies objects and facilities in high-resolution multispectral satellite imagery across 63 fMoW classes.
Abstract
from arXiv · showhide
Satellite imagery is important for many applications including disaster response, law enforcement, and environmental monitoring. These applications require the manual identification of objects and facilities in the imagery. Because the geographic expanses to be covered are great and the analysts available to conduct the searches are few, automation is required. Yet traditional object detection and classification algorithms are too inaccurate and unreliable to solve the problem. Deep learning is a family of machine learning algorithms that have shown promise for the automation of such tasks. It has achieved success in image understanding by means of convolutional neural networks. In this paper we apply them to the problem of object and facility recognition in high-resolution, multi-spectral satellite imagery. We describe a deep learning system for classifying objects and facilities from the IARPA Functional Map of the World (fMoW) dataset into 63 different classes. The system consists of an ensemble of convolutional neural networks and additional neural networks that integrate satellite metadata with image features. It is implemented in Python using the Keras and TensorFlow deep learning libraries and runs on a Linux server with an NVIDIA Titan X graphics card. At the time of writing the system is in 2nd place in the fMoW TopCoder competition. Its total accuracy is 83%, the F1 score is 0.797, and it classifies 15 of the classes with accuracies of 95% or better.
I. INTRODUCTION
Deep learning, especially CNNs, offers a way to automate satellite-image recognition, but satellite imagery presents unusual scale, viewpoint, occlusion, and labeled-data challenges.
- Deep learning models represent data at multiple abstraction levels through multiple processing layers.
- CNN-based methods achieved major success in photograph object detection and classification, motivating their use for satellite imagery.
- CNNs learn image features through successive convolutional layers and produce predicted probabilities for each class.
- Satellite-image recognition is needed for applications including law enforcement, disaster response, and monitoring agricultural or oil-development activity.
- Traditional detection and classification algorithms are described as too inaccurate and unreliable, while analysts cannot manually cover the required geographic expanses.
- Limited success in satellite imagery is also attributed to the lack of large labeled datasets needed for training.
- Satellite objects may be very large or small, long and narrow, inverted, or obscured by clouds, complicating fixed-size CNN preprocessing.
III. PRIOR WORK
Prior remote-sensing datasets supported land-cover classification, building detection, or geographically narrow tasks, but did not provide the global scale needed for versatile classification.
- The UC Merced Land Use Dataset contains 2,100 aerial images across 21 land-cover classes.
- SpaceNet provides high-resolution satellite imagery from five cities with building footprints for segmentation and footprint extraction.
- SpaceNet is limited in geographic coverage and usefulness for training a general classifier.
- Existing remote-sensing datasets lacked the hundreds of thousands of globally distributed images needed for a versatile classification system.
IV. DATASET
The fMoW dataset supplies one million high-resolution, multispectral satellite images with broad class coverage and metadata, enabling large-scale classification research.
- fMoW contains one million satellite images labeled across 62 classes plus a separate false-detection class.
- The dataset includes categories such as airstrips, oil and gas facilities, surface mines, tunnel openings, and shipyards.
- fMoW imagery comes from DigitalGlobe satellites at a nominal 0.5-meter-per-pixel ground sample distance with at least four spectral bands, including NIR.
- The images are accompanied by metadata including ground sample distance, timestamps, bounding-box measurements, acquisition angles, and sun information.
- The dataset’s size and global scope are described as unprecedented and suitable for developing satellite-image classification systems.
V. METHODS
The system classifies satellite imagery into 63 classes using an ensemble of CNNs, image preparation, and neural networks that integrate image features with metadata.
- The system inputs a satellite image and bounding-box metadata, then classifies it into 63 classes including false detection.
- The architecture combines an ensemble of CNNs with image-preparation operations and neural networks that integrate image features and metadata.
- Unweighted averaging merges neural-network outputs into class probabilities, and the maximum probability determines the classification.
- The system was implemented in Python with Keras and TensorFlow and trained and tested on CentOS Linux servers with NVIDIA Titan X GPUs.
A. Image Preparation
Image preparation enlarges or squares object bounding boxes before cropping and resizing, while an ensemble of deep CNNs processes the resulting images.
- A. Image Preparation: The preparation step enlarges the bounding box proportionally before cropping and resizing to provide context pixels around the object.
- A. Image Preparation: Squaring the bounding box expands its smaller dimension to preserve the aspect ratio of image features.
- A. Image Preparation: The system uses DenseNet-161, ResNet-152, Inception-v3, and Xception in an ensemble rather than relying on a single CNN.
- A. Image Preparation: The authors found that deeper CNNs produced better results than initially tested shallower VGG and ResNet-50 models.
C. Metadata
Metadata is normalized and combined with CNN-derived image features in neural networks that produce 63-class predictions, with unweighted averaging selecting the final class.
- C. Metadata: CNN prediction probabilities provide 63 image features that are combined with normalized metadata features before entering the neural networks.
- C. Metadata: The feature-extraction step converts satellite metadata into 27 normalized floating-point values intended to improve classification accuracy.
- C. Metadata: The neural networks combine 27 metadata features and 63 image features through a 1024-node layer and a 63-node softmax output layer.
- C. Metadata: Using the 63 CNN predictions as image features performed better than using 2048 values from the next-to-last CNN layer.
- C. Metadata: Unweighted averaging combines neural-network predictions, and the class with maximum probability is selected.
E. Image Sequences
For image sequences, the system classifies each image separately, averages class predictions across the sequence, and uses the maximum resulting prediction.
- E. Image Sequences: Each sequence represents one object or facility across different dates and times, with varying acquisition, sun, lighting, and weather conditions.
- E. Image Sequences: The task assigns one class to each image sequence, including activity classes for road flooding, construction, and debris deposition.
- E. Image Sequences: The system separately classifies each image, averages predictions for each class across the sequence, and takes the maximum prediction as the classification.
F. False Detection Class
The fMoW false detection images were added as a 63rd class after testing alternative threshold-based definitions, which performed worse.
- 11,000 false detection images were split 90–10% and added to the training and validation datasets as a 63rd class.
- Ad hoc thresholding based on maximum predicted probability produced inferior results.
G. Training
Training used reserved validation data, transfer learning, and separate metadata integration networks, with preprocessing exclusions and empirically selected procedures.
- A training–validation split reserves data to check whether neural networks overfit the training data.
- The dataset used 363 thousand training images and 53 thousand validation images, while omitting images with excessive cloud cover or tiny bounding boxes from training.The supplied split allocated 87% of the data to training; false detections were split separately 90–10%.
- Transfer learning fine-tuned CNNs pretrained on ImageNet rather than training them from scratch.Fine-tuned networks produced higher accuracies and trained faster than networks trained from scratch.
- Metadata-integration networks consumed each CNN’s 63 softmax probabilities and 27 metadata features after CNN training.They were trained for 20 epochs with early stopping.
- Separate training of the metadata networks and CNNs yielded shorter training times and higher accuracies than simultaneous training.
VI. RESULTS AND DISCUSSION
The system achieved strong overall validation performance, but accuracy varied substantially across classes, with distinctive visual patterns easier to classify than visually similar facilities.
- Overall performance: 83% total accuracy and a 0.797 F1 score were achieved on the 53 thousand-image validation dataset.Fifteen classes reached accuracies of 95% or better, and the system ranked second among 50 competitors with a TopCoder score of 765,663.
- Confusion patterns: The system’s strongest confusion was between shipyards and ports, which were confused 56% of the time.The multi-unit residential and single-unit residential classes formed the next most-confused pair.
- Weak classes: Office_building was the weakest class at 24% accuracy, followed by police_station at 37%.Office buildings were confused with several other building-related classes, while police stations were often confused with places of worship and fire stations.
- Strong classes: Wind_farm reached 99% accuracy and crop_field reached 97% accuracy.Distinctive tower, blade, shadow, crop-row, and color features supported these classifications.
- Related system: The ensemble system combined CNN outputs and metadata-processing networks, while the related Johns Hopkins APL solution used DenseNet-161, metadata features, and an LSTM.
- Data augmentation: Training-image flips and rotations increased the effective training-dataset size by a factor of eight.
- Confusion matrix: The 63×63 confusion matrix places predicted labels on the x-axis and true labels on the y-axis, with classes ordered by accuracy.
VII. CONCLUSION
The paper presents a deep learning system for classifying objects and facilities in high-resolution multispectral satellite imagery and reports practical applications when combined with detection.
- The system classifies objects and facilities in high-resolution multispectral imagery using an ensemble of CNNs and metadata-integrating post-processing networks.On the one-million-image, 63-class fMoW dataset, it achieved 0.83 accuracy, a 0.797 F1 score, and at least 95% accuracy on 15 classes.
- Combined with a detection component, the system could search large amounts of satellite imagery for objects or facilities of interest.The paper identifies possible uses in law enforcement, disaster response, and monitoring crop growth or oil well construction.