GeoSAM: Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentation

1 Introduction

• Pedestrian infrastructure such as sidewalks and crosswalks has received comparatively little attention, although accurate segmentation can help reveal accessible routes and destinations, especially for people with disabilities.Source: §1 · p. 1
• Limited high-quality labeled datasets constrain scalability and adaptability, and traditional models trained on limited, homogeneous data struggle with fine-grained classes and unseen regions.Roads and sidewalks can have thin boundaries, similar textures, and frequent occlusions; even regional distribution shifts can degrade performance.Source: §1 · p. 1
• SAM’s task-agnostic pretraining and prompt-based adaptation offer a route to segmentation across diverse regions without retraining the model for each new task.Source: §1 · pp. 1–2
• Zero-shot SAM struggles on geographical imagery, where extensive roads and sidewalks need multiple iterative prompts and natural-image pretraining does not cover their large, texture-similar structures.The paper proposes fine-tuning SAM on limited data to learn domain-specific patterns while remaining effective under regional shifts.Source: §1 · p. 2
• GeoSAM combines point prompts for spatial guidance with text prompts for semantic clarity, then fine-tunes SAM’s lightweight decoder for multi-class road and pedestrian segmentation.The prompts inject domain knowledge from traditional models, and the paper evaluates performance across datasets from two cities.Source: §1 · p. 2

2.1 Traditional Geographical Methods

• Traditional geographical segmentation approaches include UNet-based, encoder-decoder, CNN, machine-learning, and transfer-learning methods, but often rely on extensive supervision and retraining and fall short on generalizing to new locations.Source: §2.1 · p. 2

2.2 Geographical Foundation Models

• Geographical foundation-model studies have explored SAM, but existing approaches often target objects with clear boundaries and context, while automated prompting and direct natural-language prompts remain gaps.Some prompt-generation methods require substantial training data, and some prompt-free or auxiliary-input methods have narrower data or class requirements.Source: §2.2 · p. 2
• Prior geographical SAM fine-tuning has not, to the authors’ knowledge, focused on mobility tasks such as pedestrian infrastructure segmentation.Source: §2.2 · p. 2

2.3 Domain-Specific Geographical Foundation Models

• Domain-specific geographical foundation models often target roads while overlooking sidewalks and crosswalks, and limited public source code makes their pedestrian-segmentation effectiveness difficult to evaluate.Source: §2.3 · p. 2

3.1 Problem Definition

• GeoSAM maps each high-resolution RGB geographical image to a per-pixel class map, with classes including background, pedestrian infrastructure, and road infrastructure.Training labels are represented as one-hot multi-channel maps.Source: §3.1 · pp. 2–3

3.2 SAM: Background

• SAM encodes images and prompts separately, then its mask decoder uses attention-based interactions between their embeddings to predict a segmentation mask.Prompt embeddings provide spatial and semantic information to guide the decoder.Source: §3.2 · p. 3

3.3 Multi-Modal Prompt Generation

• GeoSAM uses multiple foreground and background points generated from a pretrained traditional model’s pseudo-labels as primary spatial guidance for sparse, spatially extensive roads and sidewalks.The method decomposes pseudo-labels into class-specific masks and samples points; approximate foreground or background placement is sufficient to guide segmentation.Source: §3.3 · p. 3
• Class-specific text prompts generated during training by GPT-4 add semantic context to point prompts, helping distinguish overlapping objects when a pixel can belong to multiple classes.Prompts are generated from class names without image content, then CLIP encodes them and a trainable projection maps embeddings to SAM’s dimension.Source: §3.3 · p. 4

3.4 Fine-Tuning the Decoder

• The decoder repeatedly applies self-attention among prompts and cross-attention with image features, refining prompt and visual embeddings into context-aware representations for segmentation.GeoSAM fine-tunes only the decoder and freezes the rest of the model.Source: §3.4 · p. 4

P + MLPP (CA(T ′

• GeoSAM freezes the rest of SAM and fine-tunes only its decoder, reducing adaptation updates to the task-specific component.The approach leverages the encoder’s general representations while reducing computational overhead.Source: P + MLPP (CA(T ′ · p. 4
• A shared decoder processes target classes jointly and outputs a multi-channel map, with one channel per class, such as roads or pedestrian infrastructure.The design is described as extensible to additional classes.Source: P + MLPP (CA(T ′ · p. 4
• Dice Focal Loss combines Dice and Focal losses to balance precise overlap accuracy against severe class imbalance in high-resolution imagery where infrastructure occupies little area.Focal Loss uses a balancing factor and focusing parameter to emphasize hard examples.Source: P + MLPP (CA(T ′ · p. 4

4.1 Datasets

• Training and test sets use separate Washington, D.C. regions, while Cambridge, MA is used for generalization; 1024 × 1024 tiles come from orthorectified aerial imagery and manually checked GIS masks.The masks represent roads and pedestrian infrastructure.Source: Table 1 · p. 5, §4.1 · p. 5

4.2 Implementation Details

• GeoSAM uses SAM ViT-H with nnU-Net point generation and CLIP ViT-B text encoding; all benchmarks share training settings and postprocessing, with no data augmentation.Erosion and dilation are applied to model outputs to remove isolated regions and improve path continuity.Source: §4.2 · p. 5

4.3 Results and Discussion

• On Washington, D.C.’s test set, GeoSAM surpasses second-best nnU-Net by 5% in mIoU and 6% in mAP, outperforming state-of-the-art models across both classes.Source: Table 2 · p. 6, §4.3 · p. 6
• On sampled test images, GeoSAM’s segmentation is closest to ground truth among the compared models, particularly around intricate boundaries.The comparison includes zero-shot SAM, UV-SAM, nnU-Net, and Swin UNETR.Source: Figure 5 · p. 6, §4.3 · p. 6
• A 2:1 ratio of 2,000 foreground to 1,000 background prompts performs best; using 150 total prompts reduces pedestrian-infrastructure performance by 8%.The authors attribute the selected ratio to extensive, sparse infrastructure and the ambiguity introduced by excess background prompts.Source: Table 3 · p. 7, §4.3 · p. 7
• Ablations report an 8% pedestrian-segmentation drop with point-only prompts, a 22% drop with text-only prompts, and a 26% pedestrian drop without decoder fine-tuning.The study presents text as secondary semantic guidance and decoder fine-tuning as adaptation to geographical imagery.Source: Table 4 · p. 7, §4.3 · p. 7
• Replacing nnU-Net with UNet or Swin UNETR for point generation causes only minor performance drops, although pedestrian infrastructure is more sensitive to the backbone.GeoSAM remains above the listed state-of-the-art models when these alternative backbones are used.Source: Table 5 · p. 7, §4.3 · p. 7
• GeoSAM is marginally slower than nnU-Net and DeepLabv3+, faster than Swin UNETR, and uses the fewest tunable training parameters.Precomputed pseudo labels enable point prompts without additional inference-time overhead.Source: Table 6 · p. 7, §4.3 · p. 7

• GeoSAM combines point and text prompts with decoder fine-tuning to segment roads and pedestrian infrastructure in geographical images.The authors describe the training and end-to-end inference pipeline as transferable across locations and classes using any pre-trained traditional model.Source: §5 · p. 7
• The authors characterize the approach as generic, reproducible, and adaptable to domain-specific segmentation tasks.Source: §5 · p. 7
• Future work will add cities to broaden generalizability analysis and extend support to more object types and imaging modalities.Proposed object types include stairs, islands/bridges, and potholes.Source: §5 · p. 7

• IoU is the predicted-to-ground-truth overlap area divided by their union area.Its range is 0 to 1, from no overlap to perfect overlap.Source: §A.1 · p. 9
• AP measures pixel-classification accuracy across thresholds as the area under the precision–recall curve.The integral is approximated by a finite sum over discrete recall levels, using recall values at successive thresholds and precision at threshold n.Source: §A.1 · p. 9
• The evaluation uses both IoU and AP to assess spatial accuracy and classification performance.Source: §A.1 · p. 9

• The erosion and dilation equations use image pixel coordinates and a structuring-element mask; T and S denote intersection and union.Source: §A.2 · p. 9
• The cited expressions reference image values at coordinates shifted by i and j, showing the indexed image value I(x + i, y + j) in both passages.Source: §A.2 · p. 9
• Erosion and dilation are applied to covered regions of a 1024×1024 segmentation map with a 10×10 filter for 10 iterations.Source: §A.2 · p. 9

• Target urban regions are selected, aerial-image tiles are downloaded and stitched into 1024×1024 composites at zoom level 20.Source: §A.3 · p. 9
• GIS shapefiles are aligned with the imagery and rasterized into pixel-level, class-specific masks for infrastructure such as roads and sidewalks.Source: §A.3 · p. 9
• Manual verification and correction address annotation quality and missing or noisy labels; the dataset is planned for public release upon paper acceptance.Source: §A.3 · p. 9

A.4 Benchmark Models

• The conventional benchmark architectures include UNet [27], nnU-Net, UNet++ [45], DeepLabv3+ [6], and HRNet [31].Their described design features include skip connections, multi-scale context, and high-resolution representations.Source: §A.4 · p. 9
• RSPrompter [4] adapts SAM for remote-sensing instance segmentation with semantic-aware learned prompts, while UV-SAM [40] adapts SAM for urban-village segmentation using four category-specific prompts fused by a prompt mixer.Source: §A.4 · p. 10

B The Algorithm

• GeoSAM freezes SAM’s image encoder and other components, concatenates point and fixed text embeddings, and updates the decoder using ground-truth masks and combined Dice Focal loss.Point prompts are generated automatically from pseudo-labels produced by a traditional segmentation model.Source: B The Algorithm · p. 10, §A.4 · p. 10