Source-linked AI summary
Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models
Jielu Zhang, Zhongliang Zhou, Gengchen Mai, Mengxuan Hu, Zihan Guan, Sheng Li, Lan Mu
TL;DR
Remote-sensing models often transfer poorly across sensors and class distributions, while SAM’s instance masks are impractical for semantic segmentation requiring same-label objects to share masks. Text2Seg uses visual foundation models to automatically generate text-guided points or bounding boxes that guide or filter SAM predictions, without additional training. The best pipeline improved zero-shot mIoU over original SAM on UAVid (124%), LoveDA (31%), Vaihingen (225%), and Potsdam (117%).
Problem
Remote-sensing models often transfer poorly across sensors and class distributions, while SAM’s instance masks are impractical for semantic segmentation requiring same-label objects to share masks.
Source: §1 · p. 2, §2.1 · p. 3Method
Text2Seg uses visual foundation models to automatically generate text-guided points or bounding boxes that guide or filter SAM predictions, without additional training.
Source: §1 · p. 3Results
The best pipeline improved zero-shot mIoU over original SAM on UAVid (124%), LoveDA (31%), Vaihingen (225%), and Potsdam (117%).
Source: §4.4 · p. 8Takeaways & Limitations
Results across four datasets support repurposing multiple visual foundation models together to improve zero-shot remote-sensing semantic segmentation.
Source: §6 · p. 8Improvements vary by dataset and class; sensor and resolution disparities may limit optimal performance, with smaller gains for low vegetation and water regions.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Remote sensing imagery has attracted significant attention in recent years due to its instrumental role in global environmental monitoring, land usage monitoring, and more. As image databases grow each year, performing automatic segmentation with deep learning models has gradually become the standard approach for processing the data. Despite the improved performance of current models, certain limitations remain unresolved. Firstly, training deep learning models for segmentation requires per-pixel annotations. Given the large size of datasets, only a small portion is fully annotated and ready for training. Additionally, the high intra-dataset variance in remote sensing data limits the transfer learning ability of such models. Although recently proposed generic segmentation models like SAM have shown promising results in zero-shot instance-level segmentation, adapting them to semantic segmentation is a non-trivial task. To tackle these challenges, we propose a novel method named Text2Seg for remote sensing semantic segmentation. Text2Seg overcomes the dependency on extensive annotations by employing an automatic prompt generation process using different visual foundation models (VFMs), which are trained to understand semantic information in various ways. This approach not only reduces the need for fully annotated datasets but also enhances the model's ability to generalize across diverse datasets. Evaluations on four widely adopted remote sensing datasets demonstrate that Text2Seg significantly improves zero-shot prediction performance compared to the vanilla SAM model, with relative improvements ranging from 31% to 225%. Our code is available at https://github.com/Douglas2Code/Text2Seg.
1 Introduction
Remote-sensing segmentation models face generalization challenges across regions, seasons, and sensors, while vanilla SAM produces instance masks rather than semantic masks. Text2Seg uses visual foundation models to generate text-guided prompts for SAM and improves zero-shot performance across four datasets.
1 Introduction
- Remote-sensing segmentation is important for environmental monitoring and land-use estimation, but models often generalize poorly across regions, seasons, and sensor types.Source: §1 · p. 1
- Text2Seg’s pipeline combines Grounding DINO–SAM, CLIP Surgery–SAM, and SAM–CLIP routes, with additional variants combining two or all three.Figure 2 also contrasts traditional segmentation with text-conditioned Text2Seg and depicts prompt augmentation.Source: Figure 1 · p. 2, Figure 2 · p. 3
- Vanilla SAM produces separate masks for objects, which does not directly meet semantic segmentation’s need to group objects sharing a label.Fine-tuning SAM may help specific applications but requires additional data and resources.Source: §1 · p. 2
- Text2Seg’s best pipeline improved zero-shot predictions over vanilla SAM on all four evaluated remote-sensing datasets.The introduction reports consistent improvement; dataset-specific mIoU values appear in the results section.Source: §1 · p. 3
- Text2Seg automatically generates text-guided points or bounding boxes with visual foundation models to guide or filter SAM predictions, without additional training.The approach is intended to reduce dependence on extensive per-pixel annotations and improve generalization across datasets.Source: §1 · pp. 2–3
2 Related work
Related work describes cross-dataset generalization challenges in remote-sensing segmentation and the use of prompt engineering to adapt foundation models without parameter tuning.
2.1 Remote Sensing Image Segmentation
- Models trained on one remote-sensing dataset often perform poorly on others because sensors and class distributions differ.Source: §2.1 · p. 3
2.2 Prompting for Foundation Models
- Prompt engineering adapts foundation models with task-specific instructions without further tuning their parameters.The section notes that prompt design has been explored for visual foundation models after successes in language models.Source: §2.2 · p. 3
3 Methods
Text2Seg is a training-free approach that combines visual foundation models to guide SAM for remote-sensing semantic segmentation. It conditions predictions on a chosen class and offers pre-SAM prompting, post-SAM filtering, and text-prompt augmentation.
3.1 Text2Seg
- Text2Seg reframes segmentation as producing a mask yk for a user-selected class Ck from image I and class text Tk.This targets settings where only a few classes are of interest.Source: §3.1 · p. 5
3.1.1 Pre-SAM Module
- Grounding DINO generates class-conditioned boxes for SAM, while CLIP Surgery samples class-similar pixel points as alternative pre-SAM prompts.CLIP Surgery forms pixelwise similarity maps and samples points above a threshold.Source: §3.1.1 · p. 5
3.1.2 Post-SAM Module
- Post-SAM filtering prompts SAM with grid points, then uses CLIP image-text similarity to retain object masks matching the target class.The selected object masks are combined into the final prediction for that class.Source: §3.1.2 · p. 5
- The experiments combine pre- and post-SAM methods, including Grounding DINO, CLIP Surgery, SAM, and CLIP.The tested combinations include a Grounding DINO and CLIP Surgery combination and a combination of all three methods.Source: §3.1.2 · p. 5
3.2 Text Prompt Augmentation
- Text prompt augmentation expands each class label with up to ten GPT-4-generated synonyms and supplies the augmented texts to the pipeline.For example, “Plant” is augmented with “Tree” and “Grass”; prompts or detections are merged according to the method used.Source: §3.2 · p. 6
4.1 Experiment Setup
The experiment compares Text2Seg pipeline combinations with vanilla SAM on four remote-sensing datasets. Because SAM does not produce semantic class labels, its evaluation uses a random point prompt and the best score across classes.
4.1 Experiment Setup
- Vanilla SAM receives a random point per image, and its best score across classes is used because it cannot provide class labels.The authors describe this comparison as an unfair but rough estimate of SAM’s performance.Source: §4.1 · p. 6
- Five tested pipelines combine Grounding DINO, CLIP Surgery, SAM, and CLIP in different configurations.The configurations include each of three two-model combinations and two combinations using additional foundation models.Source: §4.1 · p. 6
- Evaluation covers UAVid, Vaihingen, Potsdam, and LoveDA.Source: §4.1 · p. 6
4.2 Dataset
The evaluation uses four remote-sensing datasets with different imagery, class sets, and preprocessing. UAVid, Vaihingen, and Potsdam images are cropped into 1024 × 1024 patches; the supplied LoveDA description specifies its image size and coverage.
4.2.1 UAVid dataset
- UAVid comprises high-resolution urban street-scene UAV images with eight classes; 150 official images were padded and split into eight 1024 × 1024 patches each.Its imagery has varied spatial features, ambiguous categories, and complex scenes.Source: §4.2.1 · p. 7
4.2.2 LoveDA dataset
- LoveDA contains 5,987 optical images at 0.3 m GSD, each 1024 × 1024, covering seven land-cover categories across urban and rural scenes in three Chinese cities.The dataset includes multiscale objects, complex backgrounds, and inconsistent class distributions.Source: §4.2.2 · p. 7
4.2.3 Vaihingen dataset
- Vaihingen provides 33 high-resolution TOP tiles with five foreground classes and one background class; experiments used TOP imagery without DSM or NDSM, cropped to 1024 × 1024.Each tile averages 2494 × 2064 pixels and has near-infrared, red, and green bands.Source: §4.2.3 · p. 7
4.2.4 Potsdam dataset
- Potsdam provides 38 TOP tiles at 5 cm GSD with the same categories as Vaihingen; experiments used RGB bands and 1024 × 1024 crops.The original tiles are 6000 × 6000 pixels and also provide near-infrared, DSM, and NDSM data.Source: §4.2.4 · p. 7
4.3 Evaluation Metrics
The evaluation reports overall accuracy and mean intersection over union. For SAM, which cannot segment by class, the authors use its highest score across classes as a favorable evaluation.
4.3 Evaluation Metrics
- OA measures the proportion of correctly classified pixels among all pixels.The metric’s denominator is the total number of pixels n.Source: §4.3 · p. 7
- mIoU measures predicted-mask overlap with ground truth, averaged across classes.Source: §4.3 · p. 7
- For SAM, each metric uses its highest score across classes because the model cannot produce class-specific segmentation.The authors characterize this as the most favorable evaluation for class-agnostic SAM.Source: §4.3 · p. 7
4.4 Result
Text2Seg's best pipeline improves zero-shot mIoU over original SAM across four datasets, though gains vary by dataset, class, and pipeline.
4.4 Result
- 0.226→0.507 mIoU on UAVid, 0.237→0.311 on LoveDA, 0.099→0.322 on Vaihingen, and 0.215→0.468 on Potsdam versus original SAM.These are the reported best-pipeline results for each dataset.Source: §4.4 · p. 8
- Gains vary by dataset and class; sensor and resolution disparities may constrain performance, while object fragmentation and band-dependent colors are offered as explanations for weaker vegetation and water results.The authors speculate that SAM's tendency to split whole objects contributes to lower accuracy for vegetation and water.Source: §4.4 · p. 8
- Grounding DINO + SAM improved most across datasets, whereas CLIP Surgery + SAM improved least on some, possibly because rough class distributions yield weak SAM point prompts.The authors attribute CLIP Surgery's weaker results to its rough class-distribution estimates, which may not yield effective point prompts.Source: §4.4 · p. 8
5 Ablation Study
The ablation finds that text-prompt augmentation improves performance for both Pre-SAM and Post-SAM models across datasets.
5 Ablation Study
- Text augmentation improves mIoU for both Pre-SAM and Post-SAM models across the evaluated datasets.The ablation compares scores before and after augmentation using two methods from the proposed toolbox.Source: §5 · p. 8
- Text prompts serve as a surrogate for SAM's point or bounding-box prompt engineering, making prompt selection pivotal to performance.Source: §5 · p. 8
- The authors suggest GPT-4-augmented text concepts improve visual-language models' understanding of corresponding classes, contributing to better performance.Source: §5 · p. 8
6 Conclusion
The conclusion presents multiple visual foundation models as a way to repurpose their complementary capabilities for downstream tasks, including zero-shot remote sensing semantic segmentation.
6 Conclusion
- Combining visual foundation models with SAM leverages their complementary strengths for downstream tasks such as zero-shot remote sensing semantic segmentation.SAM segments instances, while Grounding DINO and CLIP provide semantic understanding and coarse-grained visual prompts.Source: §6 · p. 8
- The pipeline is reported effective on four remote sensing datasets, while still trailing fully supervised models.Source: §6 · p. 8
- The authors suggest limited representation of remote sensing images in VFM pretraining may contribute to weaker performance than supervised counterparts.Source: §6 · p. 8
- The authors advocate developing visual foundation models tailored to specific domains.Source: §6 · p. 8