Source-linked AI summary
iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images
Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, Xiang Bai
TL;DR
Aerial instance segmentation lacks large-scale, densely annotated datasets suited to images with dense objects, tiny instances, and major scale variation. The paper introduces and benchmarks iSAID, a dataset with precise instance masks across diverse high-resolution aerial imagery. Its scale and annotations expose challenges for existing algorithms and support further development of specialized aerial-image solutions.
Problem
Aerial imagery lacks large-scale, densely annotated instance-segmentation datasets with accurate masks, despite challenging density, appearance, scale, and class-imbalance conditions.
Method
The paper constructs iSAID by independently annotating aerial imagery from scratch and benchmarks it with Mask R-CNN, PANet, and modified variants.
Results
655,451 instances across 15 categories and 2,806 high-resolution images establish a large benchmark whose existing instance-segmentation baselines require aerial-specific adaptations.
Takeaways & Limitations
iSAID provides a challenging, precisely annotated benchmark intended to drive new instance-segmentation developments for aerial imagery.
Abstract
from arXiv · showhide
Existing Earth Vision datasets are either suitable for semantic segmentation or object detection. In this work, we introduce the first benchmark dataset for instance segmentation in aerial imagery that combines instance-level object detection and pixel-level segmentation tasks. In comparison to instance segmentation in natural scenes, aerial images present unique challenges e.g., a huge number of instances per image, large object-scale variations and abundant tiny objects. Our large-scale and densely annotated Instance Segmentation in Aerial Images Dataset (iSAID) comes with 655,451 object instances for 15 categories across 2,806 high-resolution images. Such precise per-pixel annotations for each instance ensure accurate localization that is essential for detailed scene analysis. Compared to existing small-scale aerial image based instance segmentation datasets, iSAID contains 15$\times$ the number of object categories and 5$\times$ the number of instances. We benchmark our dataset using two popular instance segmentation approaches for natural images, namely Mask R-CNN and PANet. In our experiments we show that direct application of off-the-shelf Mask R-CNN and PANet on aerial images provide suboptimal instance segmentation results, thus requiring specialized solutions from the research community. The dataset is publicly available at: https://captain-whu.github.io/iSAID/index.html
1. Introduction
Aerial imagery exposes instance-segmentation challenges that natural-scene datasets and existing aerial datasets do not adequately address. iSAID is introduced as a large, densely annotated benchmark designed for these conditions.
- Instance segmentation predicts category labels and pixel-level masks for every object of interest.
- Aerial images contain densely packed objects with arbitrary shapes and orientations, large aspect ratios, and substantial scale variation.
- Existing aerial datasets for detection or semantic labeling lack accurate per-instance pixel annotations, while public instance-segmentation datasets usually cover only one category.
- iSAID provides 655,451 instances across 15 categories and 2,806 high-resolution images for instance segmentation.
- The dataset includes high-resolution imagery, many instances per image, varied object scales and orientations, ambiguous small objects, and professionally validated instance annotations.
2. Related Work
Earth Vision research lacks aerial datasets with the scale, category diversity, instance counts, and annotation quality of natural-scene benchmarks. iSAID addresses this gap with broad, precise instance-level annotations for challenging aerial imagery.
- Large-scale datasets have advanced scene understanding, but aerial datasets lag in category count, instance count, and annotation quality.
- Off-the-shelf methods trained on conventional datasets perform unsatisfactorily on aerial imagery because of domain shift and large density, orientation, and scale variations.
- SSD achieves an mAP of 17.84 on DOTA, illustrating the difficulty of transferring conventional object detectors to aerial images.
- Aerial instance segmentation requires precise per-pixel localization, but existing public datasets contain only single-category instances such as ships or buildings.
- iSAID offers 15× the object categories and 5× the instances of existing aerial instance-segmentation datasets.
3. Dataset Details
iSAID is constructed as a densely annotated aerial instance-segmentation dataset, using independent annotation and quality-control procedures to address limitations in existing aerial datasets. Its images and annotations capture the density, scale, aspect-ratio, and resolution challenges characteristic of aerial imagery.
- Dataset construction: 2,806 images from DOTA were independently annotated from scratch, producing 655,451 instances instead of DOTA’s 188,282.The source images were collected from multiple sensors and platforms, while the original DOTA annotations contained incorrect labels, missing instances, and inaccurate bounding boxes.
- Dataset construction: iSAID annotates 15 frequently occurring overhead-imagery categories, including planes, ships, vehicles, bridges, harbors, and sports facilities.The category list also includes storage tanks, helicopters, roundabouts, swimming pools, and soccer ball fields.
- Annotation procedure: The annotation pipeline combines guidelines, annotator training, image annotation, quality checks, and iterative refinement for consistent, accurate, and complete masks.Guidelines require annotating visible objects, matching visual margins, refining boundaries through zooming, escalating difficult cases, and using one facility and software.
- Dataset properties: iSAID images range from 800 to 13,000 pixels in width, creating substantially higher-resolution inputs than typical natural-image datasets.The paper notes that applying conventional detection and instance-segmentation methods to such high-resolution aerial images is challenging.
- Dataset properties: 8,000 instances can occur in one image, while the dataset averages approximately 239 instances per image.MSCOCO, Cityscapes, PASCAL-VOC, ADE20K, and NYU Depth V2 average 7.1, 2.6, 10.3, 19.5, and 23.5 instances per image, respectively.
- Dataset properties: Object scale and shape vary sharply: areas span from 10-pixel vehicles to 1,297,121-pixel fields, area ratios reach 20,000, and aspect ratios reach 90.The average instance aspect ratio is 2.4, and small and large objects commonly appear together.
4. Experiments
The experiments benchmark Mask R-CNN, PANet, and aerial-adapted variants for instance segmentation and object detection on iSAID. PANet variants generally outperform Mask R-CNN variants, while PANet++ achieves the strongest reported performance and qualitative results.
- Evaluation setup: Mask R-CNN and PANet baselines, their variants, and class-wise results are evaluated on iSAID using instance segmentation and object detection metrics.The evaluation uses COCO-style mask AP metrics and horizontal bounding-box AP results.
- Model variants: Three modifications adapt the baselines to aerial imagery: 1000 detection boxes, six-scale augmentation, and a 0.6 NMS threshold.The baselines use 100 detections, a single 800-pixel scale, and a 0.5 NMS threshold.
- Quantitative results: 7.8 AP is the absolute improvement of Mask R-CNN+ over the original Mask R-CNN baseline.The paper describes this as a marginal improvement from minor aerial-image adaptations.
- Quantitative results: PANet and its variants outperform Mask R-CNN and its variants for instance segmentation, while the strongest result comes from PANet++ with a ResNet-152-FPN backbone.The same method-ranking trend is reported for bounding-box object detection.
- Class-wise results: PANet++ shows significant AP50 gains of approximately 5 points or more for categories including baseball diamond, basketball court, and harbor.These gains are reported in the class-wise instance segmentation analysis.
- Qualitative results: Qualitatively, simple baseline modifications improve performance on very small and very large objects, and PANet++ produces the most convincing masks.Figure 9 also reports that original Mask R-CNN has missing object instances and the least accurate visual results.
5. Conclusion
The paper introduces iSAID to address the lack of large-scale, densely annotated aerial imagery with accurate instance masks. Its benchmarks show that aerial imagery poses challenges for existing instance segmentation algorithms and motivate further development.
- Contribution: iSAID provides 15 object categories and 655,451 annotated instances for aerial-image instance segmentation.The dataset is presented as a response to limited progress caused by the lack of large-scale, densely annotated imagery with accurate instance masks.
- Challenges: Aerial imagery challenges existing instance segmentation algorithms through many objects per image, limited appearance details, small objects, scale variation, and class imbalance.These properties distinguish the aerial setting addressed by the paper.
- Implication: The authors hope iSAID will lead to new developments in instance segmentation for aerial imagery.