Source-linked AI summary
Segment anything, from space?
Simiao Ren, Francesco Luzi, Saad Lahrichi, Kaleb Kassaw, Leslie M. Collins, Kyle Bradbury, Jordan M. Malof
TL;DR
SAM’s transfer from natural to overhead imagery is uncertain because overhead tasks pose distinct challenges and prior evaluation covered only one dataset. This paper evaluates SAM in model-composition and interactive-annotation experiments across overhead benchmarks. SAM sometimes generalizes well, but usually performs relatively poorly against supervised models and can fail on roads.
Problem
Whether SAM’s effectiveness transfers to overhead imagery is unclear because prior work examined only one dataset and overhead imagery presents distinct challenges.
Source: §1.1 · p. 2Method
The authors adapt SAM experiments for model composition and interactive annotation to evaluate its performance on overhead imagery.
Source: §1.1 · p. 2, §5 · p. 5Results
SAM sometimes generalizes well to overhead tasks, but usually performs relatively poorly against supervised models and can fail entirely on roads.
Source: §4.1 · p. 5Takeaways & Limitations
Overhead imagery can expose poorly defined instance classes such as roads, for which class-level segmentation or overhead-class fine-tuning may be relevant.
Source: §4.1 · p. 5, §6 · p. 7Road instance segmentation is ill-posed when connected road networks lack clear instance boundaries; SAM’s road-task IoU is below 10 across the reported experiments.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Recently, the first foundation model developed specifically for image segmentation tasks was developed, termed the "Segment Anything Model" (SAM). SAM can segment objects in input imagery based on cheap input prompts, such as one (or more) points, a bounding box, or a mask. The authors examined the \textit{zero-shot} image segmentation accuracy of SAM on a large number of vision benchmark tasks and found that SAM usually achieved recognition accuracy similar to, or sometimes exceeding, vision models that had been trained on the target tasks. The impressive generalization of SAM for segmentation has major implications for vision researchers working on natural imagery. In this work, we examine whether SAM's performance extends to overhead imagery problems and help guide the community's response to its development. We examine SAM's performance on a set of diverse and widely studied benchmark tasks. We find that SAM does often generalize well to overhead imagery, although it fails in some cases due to the unique characteristics of overhead imagery and its common target objects. We report on these unique systematic failure cases for remote sensing imagery that may comprise useful future research for the community.
1. Introduction
SAM is a segmentation foundation model designed to generate masks from lightweight prompts without task-specific adaptation. This work tests whether its natural-imagery capabilities transfer to overhead imagery through interactive annotation and model composition experiments.
1. Introduction
- SAM [18] is a segmentation foundation model that predicts masks from point, box, or mask prompts, aiming for zero-shot use without task-specific adaptation.Foundation models are trained on massive datasets to generalize to novel downstream tasks with little or no additional training.Source: §1 · p. 1
- On natural imagery, SAM often outperformed interactive annotation methods and achieved accuracy comparable to fully supervised models in model composition.Interactive annotation uses human prompts to refine masks; model composition uses prompts, such as another model’s box or mask, to guide SAM.Source: §1 · p. 2
1.1. Contributions of this Work
- Because prior overhead evidence covered only one dataset and overhead imagery presents distinct challenges, the work asks how well SAM transfers to this domain.The authors adapt experiments from [18] covering interactive annotation and model composition.Source: §1.1 · p. 2
- The study evaluates SAM on seven public overhead-imagery benchmarks spanning 5 million square kilometers and reports overhead-specific failure cases for future research.The evaluation includes both model composition and interactive annotation scenarios.Source: §1.1 · p. 2
2. Related Work
Prior SAM applications span natural and remote-sensing imagery, but this work presents the first comprehensive overhead-imagery evaluation across varied tasks and both interactive annotation and model composition.
2. Related Work
- SAM has been applied to dehazing [15], image tagging [34], shadow segmentation [29], sonar [28], foreground segmentation [30], electron microscopy [7], and medical segmentation [1] [10] [11] [12] [17] [20] [28] [32].Source: §2 · p. 2
- Remote-sensing applications include planetary landforms [16], glaciers, sea ice [31], roads [13], and buildings [13] [33].Source: §2 · p. 2
- This work is the first comprehensive SAM evaluation across overhead target classes, resolutions, and locations, and the first to assess both interactive annotation and model composition.Collectively, its results provide the first assessment of SAM’s transfer to overhead-imagery applications.Source: §2 · p. 2
3. Benchmark Datasets
The benchmarks were selected for public availability and segmentation labels, with softer preferences for community relevance, common target classes, and complementary dataset properties. Several datasets were adapted to define manageable experimental tasks.
3. Benchmark Datasets
- Eight publicly available datasets with class- or instance-level labels were selected, prioritizing community use, widely studied objects, and representative locations, resolutions, and classes.Source: §3 · p. 2
- Table 1 summarizes benchmark quantity, resolution in meters per pixel, and task type, distinguishing segmentation from edge detection.Inria and DeepGlobe Building were combined into DG+Inria Building to enable comparison with supervised models from [21].Source: §3 · p. 2, Table 1 · p. 3
- The DeepGlobe Land dataset was reduced to water, agriculture, and urban tasks, while isolated nonedge components in Parcel Delineation were treated as object instances.Source: §3 · p. 3
4. Model Composition Experiments
The experiments adapt model composition to overhead imagery using U-Net-derived boxes and ground-truth point prompts. SAM generally falls short of supervised models, with notable weaknesses on point prompts and several overhead classes.
4. Model Composition Experiments
- Dataset-specific U-Nets supplied component-derived boxes, while ground-truth components supplied random or center point prompts; instance masks were unioned into class-level predictions.The U-Net models were evaluated against disjoint training and testing partitions.Source: §4 · p. 3
4.1. Results and Discussion
- The SAM–supervised performance gap is larger on most overhead benchmarks, especially buildings and roads; the authors hypothesize a bias toward natural-image objects.The authors suggest complex buildings may not fit a generic objectness concept inferred from natural imagery.Source: §4.1 · pp. 4–5
- Task-specific supervised models almost always achieve the highest IoU; SAM performs significantly worse with point than box prompts, with little effect from point location.The comparison covers the Model Composition experiments.Source: §4.1 · pp. 3–4, Figure 4 · p. 5
- Buildings' complexity, solar-array underperformance despite apparent visual simplicity, and roads' poorly defined instance concept expose distinct overhead-imagery challenges.SAM's performance can depend strongly on object scale, and road performance is low overall.Source: §4.1 · p. 5
- Overall, SAM sometimes achieves competitive zero-shot results on overhead tasks, but often performs relatively poorly against supervised models and sometimes fails completely.The authors note that fine-tuning or class-level segmentation may address some challenges.Source: §4.1 · p. 5
5. Interactive Annotation Experiments
SAM generally outperforms RITM in the interactive annotation experiments, while its IoU varies with prompt informativeness and improves for both models as more points are added. The experiments compare box and point prompts, single-point mask selection, and iterative error-correcting prompts.
5. Interactive Annotation Experiments
- For single-box prompts, ground-truth class-level masks are converted into instance-level boxes that prompt SAM to produce masks.Predicted instance masks are aggregated into one class-level mask for scoring.Source: §5 · pp. 5–6
- Single-point experiments use random or centered points within ground-truth instances; a human can select the candidate mask with the highest true IoU.SAM returns three candidate masks with estimated IoUs, while RITM is evaluated using the same point prompts.Source: §5 · p. 6
- In iterative prompting, each new point targets the center of the largest false-positive or false-negative region in the current mask.The experiments evaluate SAM and RITM as the number of prompt iterations increases.Source: §5 · p. 6
5.1. Results and Discussion
- SAM almost always outperforms RITM across single-box and single-point prompts, with gains of 50 and 45 IoU points on Solar and Inria+DG Buildings.Both models perform very poorly on Inria+DG Roads.Source: §5.1 · p. 6, Figure 6 · p. 7
- As point prompts increase, SAM and RITM improve; SAM leads in nearly every setting, but their IoUs are usually similar at T=10.At T=10, SAM reaches IoU of 0.75 or larger on 6 of 10 benchmarks.Source: §5.1 · p. 6, Figure 7 · p. 7
6. Additional Analysis
The additional analysis finds that SAM’s IoU can change when overhead imagery is up-sampled, even without adding image information. It also identifies road instance segmentation as an ill-posed task where SAM performs very poorly.
Sensitivity of SAM to Image Resolution
- Up-sampling can significantly improve SAM’s IoU despite adding no image information, and the effect varies across the two benchmark problems.The authors hypothesize that SAM’s training on high-resolution natural imagery may bias it toward more pixels per object.Source: §6 · p. 7
Instance Segmentation is Sometimes Ill-Posed
- Road instance segmentation is sometimes ill-posed because connected road networks lack clear instance boundaries, and SAM’s Inria+DG Roads IoU is below 10.The paper suggests fine-tuning SAM’s decoder for class-level masks where that is more appropriate.Source: §6 · p. 7
7. Conclusions
Across overhead imagery benchmarks, SAM often performs well in interactive segmentation and can approach supervised models in model composition, but results vary substantially. Prompt type, image resolution, and object-class fit constrain performance.
7. Conclusions
- The study evaluates SAM on eight overhead-imagery benchmarks across model-composition and interactive-segmentation scenarios.Source: §7 · p. 8
- In interactive segmentation, SAM nearly always outperforms RITM, though its IoUs are somewhat lower than comparable natural-imagery results.With 10 point prompts, SAM achieves IoU above 0.75 on 6 of 10 benchmarks.Source: §7 · p. 8
- In model composition, SAM often approaches supervised state-of-the-art models, but its performance varies from superior to much worse across tasks.It generally performs somewhat worse; bounding-box prompts support the reported finding, while point prompts from other models generally perform poorly.Source: §7 · p. 8
- Artificially up-sampling overhead imagery can significantly improve or lower SAM’s performance, indicating sensitivity to image resolution.Source: §7 · p. 8
- SAM performs very poorly on some classes, such as roads, for which instance segmentation may be unsuitable.Source: §7 · p. 8