The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot

1 Introduction

• Remote-sensing segmentation supports environmental monitoring, disaster management, and urban planning, but traditional techniques often require extensive human input for accurate results.Deep-learning methods have automated segmentation, though effective segmentation with minimal human input remains challenging [3].Source: §1 · p. 1
• SAM generalizes across diverse image datasets, requiring no additional training for unfamiliar objects; specific domain conditions may limit it, while relearning with one example of a new class or object can improve results.Source: §1 · p. 1
• PerSAM localizes a reference-mask concept with a location prior, while PerSAM-F freezes SAM and fine-tunes two multi-scale-mask weights in as little as ten seconds.Both are training-free personalization approaches; PerSAM-F uses one-shot fine-tuning to reduce mask ambiguity.Source: §1 · p. 1
• SAM can segment from a bounding box, single point, or text prompt, potentially reducing annotation labor and delays in time-sensitive applications.Existing techniques may require intensive annotations for each new object, with computational overhead and delays.Source: §1 · pp. 1–2
• The study evaluates SAM’s zero- and one-shot segmentation of remote-sensing imagery across multiple datasets and scales, and reports improved performance from a single example of a new class.The study also adapts SAM to the data structure of remote-sensing imagery.Source: §1 · p. 2
• The one-shot innovation uses text-prompt-based segmentation as the training sample rather than a human-labeled example, automating SAM refinement for remote-sensing imagery.The study discusses implications, limitations, and future directions.Source: §1 · p. 2

2 Remote Sensing Image Segmentation: A Brief Summary

• Pixel- and object-based traditional methods can struggle with modern imagery; CNNs capture hierarchical features but are computationally intensive, while ViTs model global context and are reported to outperform CNNs in remote sensing [28] [29] [3].Source: §2 · pp. 2–3
• Interactive segmentation incorporates user input for precision, while superpixelization groups similar pixels, object proposals suggest candidate regions, and foreground segmentation separates subjects from backgrounds.Source: §2 · p. 2
• Semantic segmentation labels each pixel by class, instance segmentation distinguishes individual objects of the same class, and panoptic segmentation combines class labels with instance identifiers.Source: §2 · p. 2
• Promptable segmentation uses prompts to guide outputs and take on new tasks at inference, unlike fixed-task multi-task systems; it can be paired with an object detector for instance segmentation [39] [33].The section presents this approach as a development associated with visual foundation and large multimodal models.Source: §2 · p. 2
• Obtaining labeled ground-truth data is time-consuming, labor-intensive, and difficult for complex, variable imagery; transfer learning can help when task-specific labels are limited.Transfer learning adapts a model pretrained on a large dataset to a related task, such as remote-sensing segmentation.Source: §2 · p. 3
• SAM can segment remote-sensing images with minimal human input, generalize to new image distributions, and be retrained with one example of each new class at prediction time.Prompts may be bounding boxes, reference points, or text; this may reduce manual annotation and accelerate image-processing workflows.Source: §2 · p. 3

3.1 Description of the Datasets

• The datasets span UAV, airborne, and satellite imagery, covering binary single-class and multi-class tasks with differing spatial resolution and coverage.Airborne data exceed 90 GB and 10,000 images and patches; UAV imagery offers fine detail but limited coverage, while satellite imagery has wider coverage and generally lower resolution.Source: §3.1 · p. 4

3.2 Protocol for Promptable Image Segmentation

• Promptable segmentation used SamGeo with ViT-H for general, bounding-box, point, and text prompts, with GroundingDINO [30] linking text to SAM masks.For multiple instances, a loop detects the highest-logit object, masks and removes it, then repeats until manually set box and text thresholds are reached; general segmentation was assessed visually.Source: §3.2 · pp. 5–6

3.3 One-Shot Text-Based Approach

• PerSAM-F was selected for the text-based one-shot approach after early experiments found it more accurate and better suited to hierarchical objects.It freezes SAM and fine-tunes two parameters, producing masks at multiple scales with learnable relative weights; the comparison uses manually labeled one-shot examples from random dataset samples.Source: §3.3 · pp. 7–8

3.4 Model Evaluation

• SAM was evaluated against ground-truth masks using IoU, Pixel Accuracy, and Dice Coefficient, with TPR and FPR also used to assess detection.Larger crops or entire orthomosaics were processed together to preserve image context for SAM’s attention mechanism, at the cost of longer computation time.Source: §3.4 · p. 8

4.2 Zero-Shot Segmentation

• Zero-shot prompt effectiveness varied by platform and target: text worked well for UAV trees, points for airborne trees, vehicles, and lakes, and satellite multiclass tasks favored points.For UAV plantation crops, points worked better at 0.01 m, whereas boxes performed better at 0.04 m as resolution coarsened and plant canopies covered rows.Source: §4.2 · p. 9
• Boxes were effective for larger, well-defined objects but could include shadows, while text prompts were relatively comparable on aerial imagery and weaker at lower resolutions.Text performance depended on translating text into image information; the authors note that remote-sensing terminology and bird’s-eye imagery may challenge GroundDINO, and suggest domain-specific fine-tuning.Source: §4.2 · pp. 9–10

4.1 General Segmentation

• Without prompts, SAM segmented objects more accurately at higher spatial resolution, but could not assign specific classes for accuracy evaluation against available labels.At very high resolution it distinguished regions and could support object-based applications or speed labeling, but it struggled to separate contiguous objects in satellite imagery.Source: §4.1 · p. 9, Figure 5 · p. 10

4.3 One-Shot Segmentation

• SAM’s zero-shot errors included selecting shadows, confusing similar surroundings, struggling with grouped or small irregular objects, and producing spurious disconnected regions at lower satellite resolutions.Single-point prompting could also perform poorly on larger grouped instances.Source: §4.3 · p. 11
• Text-based one-shot automates example selection by choosing the highest-probability object and iteratively segmenting instances, reducing manual input.Its one-example policy favors objects similar to the selected sample and can exclude other objects in the image.Source: §4.3 · p. 11
• Traditional one-shot learning outperformed zero-shot learning on all datasets, while text-based one-shot was close to it in most cases.The comparison with human-sampled PerSAM was not consistently decisive given metric standard deviations; human labeling was more appropriate when GroundDINO failed to identify an object.Source: §4.3 · p. 11, Table 3 · p. 15

• SAM’s RGB training may limit multispectral and hyperspectral segmentation; rotated three-band combinations or domain-adaptation fine-tuning are proposed, but early tests warrant further exploration.Source: §5 · p. 15
• SAM can support land-cover mapping, urban-growth and land-use monitoring, and natural-resource management, with real-time applications identified for further study.Source: §5 · pp. 15, 18
• Limited regional diversity may constrain robustness; future work could use geographically diverse datasets to capture variation in buildings, water bodies, and other objects.The paper suggests publicly available datasets from diverse regions for building and water-body detection.Source: §5 · pp. 15–16
• One-shot training improved performance across most investigated datasets, especially at object borders, but a single example may impair generalization across heterogeneous data.For example, a healthy tree sample could bias the model against unhealthy trees or differently structured canopies; few-shot learning with 2 to 10 examples may improve adaptability but require more resources and training data.Source: §5 · p. 17
• Combining SAM with weakly supervised vision transformers or interchangeable detection models could improve segmentation, while increasing complexity; GIS integration could speed annotation, with AI security also a concern.The paper notes that SAM can provide labeled regions for transformer training and that GIS-assisted editing could make training datasets quicker and more robust; training-data contamination is a security risk.Source: §5 · p. 17
• Tests found a noticeable accuracy decrease at resolutions above 30 cm; super-resolution before SAM may help, but introduced errors require careful validation.The authors also propose adapting or training SAM on lower-resolution imagery.Source: §5 · p. 17

6 Conclusions

• Across aerial and satellite datasets, SAM showed promising zero- and one-shot segmentation performance against ground-truth masks, though task-specific improvement remains needed.The study benchmarked SAM on remote-sensing segmentation tasks and identified scope to refine its performance for specific implementations.Source: §6 · p. 18
• SAM’s text, point, box, and general prompts enable segmentation with minimal human supervision, potentially reducing annotation workload and streamlining other methods’ training workflows.The authors suggest reduced labeling input may expedite training schedules and support more cost-effective workflows.Source: §6 · p. 18
• The datasets covered diverse contexts and scales but emphasized straightforward segmentation; future work should improve SAM and integrate it with other methods for more challenging scenarios.The study deliberately avoided complex or challenging datasets to focus on fundamental segmentation aspects.Source: §6 · p. 18
• In complex scenarios, SAM can produce less optimal masks by overestimating object boundaries, and mistakes increase as input imagery resolution decreases.The authors identify these constraints as needing correction to improve SAM’s remote-sensing applicability.Source: §6 · p. 18

Supplementary

• The repositories provide code and packages for point- and bounding-box-based shapefiles with SAM, plus notebooks demonstrating text prompts and one-shot modifications.The resources are intended to support use across diverse remote-sensing contexts.Source: Supplementary · p. 18

• AI means Artificial Inteligence, CNNs means Convolutional Neural Networks, and GANs means Generative Adversarial Networks.Source: Abbreviations · p. 18
• GIS means Geographic Information Systems, NLP means Natural Language Processing, and SAM means Segment Anything Model.Source: Abbreviations · p. 18
• UAV means Unmanned Aerial Vehicle, and ViT means Vision Transformer.Source: Abbreviations · p. 18