Source-linked AI summary
The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot
Lucas Prado Osco, Qiusheng Wu, Eduardo Lopes de Lemos, Wesley Nunes Gonçalves, Ana Paula Marques Ramos, Jonathan Li, José Marcato Junior
TL;DR
Remote sensing segmentation requires accurate object delineation while often relying on costly, labor-intensive labeled data. This study evaluates SAM across aerial and satellite imagery with multiple prompts and introduces automated one-shot refinement using a text-prompt-derived example. SAM shows promise for remote sensing segmentation and reducing annotation workload, although lower-resolution imagery and one-shot generalization remain important limitations.
Problem
Remote sensing segmentation requires large volumes of labeled ground-truth data, whose acquisition and annotation are time-consuming, labor-intensive, and difficult for variable imagery.
Method
The study benchmarks SAM’s zero-shot and one-shot capabilities across aerial and satellite datasets, using text, point, box, and general prompts, with text-prompt-derived segmentation as an automated one-shot training sample.
Results
SAM showed notable performance against ground-truth masks, while its prompt capabilities and minimal human supervision can reduce annotation workload.
Takeaways & Limitations
SAM has promising potential as a resource for remote sensing applications and for supporting faster, more cost-effective workflows.
Takeaways & Limitations
One-shot learning may reduce generalization across heterogeneous remote sensing data, while SAM’s accuracy decreases on lower spatial resolution imagery, particularly above 30 cm.
Abstract
from arXiv · showhide
Segmentation is an essential step for remote sensing image processing. This study aims to advance the application of the Segment Anything Model (SAM), an innovative image segmentation model by Meta AI, in the field of remote sensing image analysis. SAM is known for its exceptional generalization capabilities and zero-shot learning, making it a promising approach to processing aerial and orbital images from diverse geographical contexts. Our exploration involved testing SAM across multi-scale datasets using various input prompts, such as bounding boxes, individual points, and text descriptors. To enhance the model's performance, we implemented a novel automated technique that combines a text-prompt-derived general example with one-shot training. This adjustment resulted in an improvement in accuracy, underscoring SAM's potential for deployment in remote sensing imagery and reducing the need for manual annotation. Despite the limitations encountered with lower spatial resolution images, SAM exhibits promising adaptability to remote sensing data analysis. We recommend future research to enhance the model's proficiency through integration with supplementary fine-tuning techniques and other networks. Furthermore, we provide the open-source code of our modifications on online repositories, encouraging further and broader adaptations of SAM to the remote sensing domain.
1 Introduction
Remote sensing segmentation is important but traditionally labor-intensive, motivating evaluation of SAM’s zero-shot and one-shot capabilities across diverse imagery and prompts. The study introduces an automated one-shot refinement using a text-prompt-derived training example.
- Remote sensing segmentation isolates objects or regions in airborne and satellite images for environmental monitoring, disaster management, and urban planning.
- Traditional segmentation often requires extensive human intervention, despite advances in AI and deep learning.
- SAM generalizes across diverse image datasets and can process unfamiliar objects without additional training, but domain-specific conditions can limit performance.
- The study evaluates SAM’s zero-shot and one-shot segmentation across multiple remote sensing datasets and image scales using varied prompts.
- The proposed one-shot method uses a text-prompt-based segmentation as a training sample instead of a human-labeled sample, automating SAM refinement.
2 Remote Sensing Image Segmentation: A Brief Summary
Remote sensing segmentation must handle complex imagery while reducing dependence on large labeled datasets. The section reviews conventional and deep-learning approaches before presenting promptable segmentation and SAM as minimal-input alternatives.
- Remote sensing segmentation partitions imagery into regions ideally corresponding to specific objects or classes.
- Traditional pixel-based and object-based methods can struggle with the complexity, variability, and high spatial resolution of modern remote sensing imagery.
- Segmentation methods: Interactive segmentation incorporates user input to make results more precise and tailored to task requirements.
- Segmentation methods: Superpixelization groups similar pixels into larger units, while object proposal generation suggests candidate regions or bounding boxes for later identification.
- Segmentation methods: Semantic, instance, and panoptic segmentation differ in whether they assign class labels, distinguish same-class objects, or combine both objectives.
- Promptable segmentation: Promptable segmentation uses designed prompts to guide versatile models toward desired segmentation outputs.
- Deep-learning approaches: CNNs, GANs, Vision Transformers, and transfer learning support remote sensing segmentation, but deep-learning methods commonly require substantial labeled ground-truth data.
- Promptable segmentation: SAM offers a potential alternative by generalizing to new distributions and tasks while using minimal inputs such as boxes, points, or text prompts.
3 Materials and Methods
The study evaluates SAM for remote-sensing segmentation across aerial and satellite datasets using multiple prompting conditions and a one-shot extension. The workflow combines text-based object selection with one-shot training and uses PerSAM-F as the selected approach after early experiments showed better accuracy.
- 3 Materials and Methods: SAM is tested with general segmentation, bounding-box prompts, point prompts, and text-based prompting to assess its segmentation capacity under different guidance conditions.General segmentation provides a zero-shot baseline, while boxes restrict segmentation per object and points test precision potential.
- 3.3 One-Shot Text-Based Approach: The one-shot implementation supplies SAM with one target-class example selected through text-prompt segmentation using the highest-logit candidate object.This combines text-prompt-based object selection with one-shot learning to create an automated training example.
- 3.1 Description of the Datasets: The evaluation spans UAV, airborne, and satellite datasets covering different spatial resolutions, geographic contexts, coverage areas, and binary or multi-class problems.UAV imagery offers very-high spatial resolution but limited coverage; satellite imagery offers wider coverage but generally lower resolution and greater atmospheric challenges.
- 3.2 Protocol for Promptable Image Segmentation: The study processes remote-sensing imagery with SamGeo and the ViT-H SAM model, using image masks, shapefile-derived boxes, and coordinate-based point prompts.General prompting produces non-classified masks, while box and point workflows pass geospatial prompts to the predictor.
- 3.3 One-Shot Text-Based Approach: PerSAM-F was selected for incorporation into the text-based approach because early experiments showed better segmentation accuracy than PerSAM.PerSAM-F uses one-shot fine-tuning to address mask ambiguity and hierarchical structures.
4 Results and Discussion
SAM’s remote-sensing performance varied with spatial resolution, object characteristics, and prompt type. The automated text-derived one-shot approach improved performance in most cases, while several limitations remained for coarse imagery, ambiguous boundaries, and text-grounding failures.
- General Segmentation: Higher spatial resolution generally produced more accurate unprompted segmentation, although satellite imagery challenged SAM at contiguous-object boundaries.General segmentation was difficult to measure against available labels because it did not return specific classes.
- Zero-Shot Segmentation: Prompt effectiveness depended on the target: text prompts led for UAV trees, boxes suited larger geometric objects, and points worked best for fine-resolution individual plants.At 0.04 m plantation resolution, bounding boxes outperformed other prompts as canopies covered entire rows.
- Zero-Shot Segmentation: Point prompts were effective for airborne trees, vehicles, and lakes, while satellite multiclass segmentation favored points because boxes overshot boundaries and produced more false positives.The satellite multiclass task was generally the most difficult, with lower metrics than the other datasets.
- Zero-Shot Segmentation: Bounding boxes efficiently captured larger objects but often overestimated smaller objects by including shadows, whereas point prompts gave precise results for distinct objects.Single-point prompting was especially useful for individual plants, trees, and vehicles.
- Zero-Shot Segmentation: Text prompts were promising for very-high-resolution imagery and easy to implement, but their effectiveness depended on translating remote-sensing language into image information.They performed worse on lower-resolution imagery and the 0.04 m plantation dataset; the authors associate this limitation partly with GroundDINO.
- One-Shot Segmentation: Traditional one-shot learning outperformed zero-shot learning across all datasets, while the automated text-based one-shot method improved performance in most cases and often approached the human-sampled approach.Human labeling remained preferable when GroundDINO could not identify the object, and standard-deviation differences sometimes narrowed the gap.
5 Future Perspectives on SAM for Remote Sensing
Future work should improve SAM’s handling of varied remote sensing data, especially multispectral imagery, low spatial resolutions, and geographically limited datasets. The study also identifies integrations with other models and GIS, alongside expanded one-shot learning, as promising directions.
- SAM may require adaptation for multispectral and hyperspectral data because it was trained primarily on RGB imagery.Suggested approaches include rotated three-band combinations or domain-adaptation fine-tuning.
- SAM can support land cover mapping and monitoring urban growth and land-use changes across remote sensing applications.The study reports segmentation of forests, urban areas, and agricultural fields, with potential use in urban planning and change monitoring.
- Limited regional diversity may not capture worldwide variation in object characteristics, motivating evaluation on datasets from more diverse regions.The discussion specifically mentions buildings and water bodies as candidates for geographically diverse datasets.
- Expanding one-shot learning to few-shot settings could improve adaptability by using 2 to 10 examples instead of one.The proposed direction uses small sets of labeled objects for each land-cover type.
- Combining SAM with other methods, including ViT frameworks, GroundDINO alternatives, or GIS, could support annotation and segmentation workflows.The discussion also notes possible increases in model complexity and computational requirements when integrating other techniques.
- Low spatial resolution remains a key challenge, with accuracy decreasing at resolutions above 30 cm in the reported tests.Super-resolution before SAM is proposed cautiously because super-resolution errors could offset potential benefits.
6 Conclusions
SAM shows promise for remote-sensing segmentation across aerial and satellite datasets, but its performance remains constrained in complex scenarios and lower-resolution imagery. The study identifies reduced annotation demands as a practical benefit while calling for further refinement.
- SAM was benchmarked for zero-shot and one-shot segmentation using aerial and satellite remote-sensing datasets.
- SAM’s prompt capabilities and minimal human supervision can reduce the workload involved in image annotation.
- The study’s datasets covered diverse contexts and scales but were not designed to represent complex or challenging scenarios.
- Future work should address these constraints and explore integrating SAM with other methods for more challenging remote-sensing scenarios.
- In complex scenarios, SAM can produce less optimal segmentation by overestimating object boundaries.
- SAM’s performance varies with input spatial resolution, with more mistakes reported as imagery resolution decreases.
Supplementary
The supplementary repository provides open-access code, packages, and notebooks for applying SAM to remote-sensing imagery with point, bounding-box, text-prompt, and one-shot workflows.
- The repository provides code and packages for combining SAM with point- and bounding-box-based shapefiles.
- Supplementary notebooks demonstrate text-based prompting and one-shot modifications of SAM.
- These resources are intended to improve the usability of SAM for remote-sensing applications.
Conflicts of Interest
The authors report no known competing financial interests or personal relationships that could have influenced the work.
- The authors declare no known competing financial interests.
- The authors declare no personal relationships known to have influenced the reported work.
- The declaration covers potential financial and personal conflicts of interest.
Abbreviations
The manuscript defines abbreviations used for artificial intelligence, neural networks, geographic information systems, remote-sensing platforms, and SAM-related components.
- SAM stands for Segment Anything Model.
- AI, CNNs, and GANs denote Artificial Intelligence, Convolutional Neural Networks, and Generative Adversarial Networks.
- GIS, NLP, UAV, and ViT denote Geographic Information Systems, Natural Language Processing, Unmanned Aerial Vehicle, and Vision Transformer.