PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Vision-Language Models Across Diverse Coastal Environments

Seth Knoop, Chad R. Samuelson, Gabriel R. Slade, Brady Moon, Joshua G. Mangelson

arXiv:2609.10855v1cs.CVcs.RO

TL;DR

Coastal VLM performance is understudied despite its relevance to robotic perception. The paper introduces a densely labeled Oahu dataset and evaluates seven VLMs through text-to-mask, mask-to-mask, and mask-to-text experiments. Landscape classes generally perform best, while coastal difficulties are linked to segmentation and linguistic representation rather than coastal context alone.

  • Problem

    VLM performance in coastal environments is largely unexplored, although coastal perception matters for autonomous marine robotic systems.

  • Method

    The study evaluates seven VLMs on a densely labeled Oahu coastal dataset using three complementary alignment experiments.

  • Results

    Landscape classes generally outperform conventional and coastal classes in text-to-mask evaluation, while cross-dataset comparisons show no consistent coastal-versus-terrestrial performance trend.

  • Takeaways & Limitations

    Alternative general or colloquial labels improve recognition of several coastal concepts, indicating that terminology influences vision-language alignment.

Abstract

from arXiv · show

Vision-language models (VLMs) enable robotic per- ception by associating visual observations with natural-language concepts. Yet their performance in coastal environments remains largely unexplored. We introduce a densely labeled coastal dataset containing more than 1,000 images collected across seven missions in three regions of Oahu, Hawaii, with 18 semantic classes and over 7,400 annotated instances. We evaluate seven modern VLMs through three complementary experiments mea- suring text-to-mask, mask-to-mask, and mask-to-text alignment. Broad landscape classes are generally recognized more accurately than conventional object and coastal classes, with coastal con- cepts presenting the greatest challenge. However, comparisons of shared conventional classes across coastal and terrestrial datasets reveal no consistent performance difference attributable solely to environmental context. Mask-to-mask matching also remains similar across conventional and coastal classes, while alternative textual labels substantially improve recognition of several coastal concepts. These results suggest that lower performance on coastal classes (at least on the objects/query categories evaluated) is heavily influenced by segmentation and linguistic representation.

I. INTRODUCTION

VLM performance in coastal environments is understudied despite the importance of visual-language perception for robotic systems. This work evaluates whether modern VLMs can generalize beyond terrestrial benchmarks and examines their intra- and inter-modal capabilities.

  • I. INTRODUCTION: Coastal VLM evaluation remains limited because existing benchmarks predominantly cover terrestrial and indoor environments.This restricts standardized evaluation and comparability for littoral domains.
  • I. INTRODUCTION: Ambiguous shorelines and under-represented marine classes weaken semantic labeling and can degrade risk assessment and navigation decisions.These challenges may contribute to unsafe real-world marine-robot deployments.
  • I. INTRODUCTION: The study evaluates state-of-the-art VLMs in coastal environments to expose gaps in their intra- and inter-modality capabilities.The evaluation targets generalization beyond conventional benchmark datasets.
  • I. INTRODUCTION: VLMs learn relationships between visual features and natural-language descriptions, while dense representations extend this alignment toward localized visual regions.Image-level alignment alone does not guarantee object- or region-level semantic understanding.

B. Applications of Vision-Language Alignment

Vision-language alignment methods support visual grounding, open-vocabulary segmentation, and cross-modal matching, but current benchmarks provide limited coverage of coastal concepts. Existing maritime datasets also omit important infrastructure categories, leaving coastal generalization insufficiently evaluated.

  • B. Applications of Vision-Language Alignment: Complementary methods combine or distill visual, language, and segmentation representations for grounding, open-vocabulary segmentation, and cross-modal alignment.Examples include Grounded-SAM, CLIP-DINOiser, and RADIO.
  • B. Applications of Vision-Language Alignment: Terrestrial datasets such as COCO and GOOSE have supported VLM development but offer comparatively little coverage of coastal environments and maritime categories.Their limited domain coverage motivates evaluation on coastal imagery.
  • B. Applications of Vision-Language Alignment: Existing maritime benchmarks provide limited opportunities to test whether VLMs recognize coastal concepts, especially infrastructure such as piers and wharves.LaRS focuses on boats and waterborne people, excluding these maritime structures.

III. COASTAL DATASET

The study constructs a densely labeled coastal dataset from diverse Oahu missions spanning harbors, rivers, and open coastal waters. It contains manually annotated semantic and instance masks across 18 classes, including specialized shoreline and maritime structures.

  • III. COASTAL DATASET: The dataset contains more than 1,000 images collected across seven missions in three geographically distinct Oahu regions.The regions include harbors, rivers, and open coastal waters.
  • III. COASTAL DATASET: Manual per-pixel labeling produces semantic and instance-level masks, with classes selected for prevalence, visual distinctiveness, and scientific relevance.The labeling process supports analysis of both scene-level and discrete-object categories.
  • III. COASTAL DATASET: The dataset includes 18 classes and more than 7,400 annotated instances, spanning scene-level classes and discrete objects.Scene-level categories are annotated once per image, whereas discrete objects are annotated by instance.
  • III. COASTAL DATASET: Maritime structures are operationally distinguished as piers, wharves, and floats according to their geometry and relationship to the water.Piers extend approximately perpendicular from shore, wharves run parallel to shore, and floats are surrounded by water.
  • III. COASTAL DATASET: Ego_vehicle and ego_person are excluded as void classes to prevent over-representation bias and are subtracted from overlapping annotations.These classes represent portions of the vessel and captured research-team members.

IV. EVALUATION METHODOLOGY

The evaluation probes VLM generalization through three complementary alignment experiments across seven modern architectures. It separates text-to-mask and mask-to-text inter-modal tasks from mask-to-mask intra-modal matching while controlling model and prompting choices where possible.

  • IV. EVALUATION METHODOLOGY: Three experiments measure text-to-mask, mask-to-mask, and mask-to-text alignment to test complementary intra- and inter-modal capabilities.The first and third experiments connect language with visual regions, while the second matches semantically similar regions across observations.
  • IV. EVALUATION METHODOLOGY: The benchmark compares seven VLM architectures, including CLIP, CLIP-DINOiser, Grounded-SAM, C-RADIOv3, SAM3, SigLIP, and YOLOE-seg.CLIP and SigLIP use FastSAM as a shared mask-generation module because they do not directly produce segmentation masks.
  • IV. EVALUATION METHODOLOGY: Models use comparable parameter counts where possible, while SAM3 remains an approximately 850M-parameter outlier.YOLOE-m-seg is selected because its largest model is about 70M parameters, and other listed variants are approximately 150–300M.
  • IV. EVALUATION METHODOLOGY: All models are evaluated without domain-specific tuning, using original-work prompt templates to assess out-of-the-box generalization fairly.The prompting strategy accounts for sensitivity in CLIP-style models.

B. Experiment 1: Input Text; Output Semantic Mask

Experiment 1 tests whether VLMs can use text prompts to localize corresponding semantic regions in coastal RGB images. Performance is measured with pixelwise segmentation metrics, with model-specific mask-generation strategies and exclusions.

  • B. Experiment 1: Input Text; Output Semantic Mask: Text prompts are matched to image regions, and predicted masks are evaluated against ground-truth annotations using pixelwise segmentation metrics.The reported metrics are mIoU, mF1, and mAP.
  • B. Experiment 1: Input Text; Output Semantic Mask: Grounded-SAM, SAM3, and YOLOE-seg directly output instance segmentations from natural-language prompts.These models automatically produce instance masks given the prompt list.
  • B. Experiment 1: Input Text; Output Semantic Mask: CLIP and SigLIP first generate class-agnostic FastSAM masks, then assign masks to prompts using cosine similarity thresholds.The thresholds are 0.25 for CLIP and 0.07 for SigLIP.
  • B. Experiment 1: Input Text; Output Semantic Mask: mAP counts predictions as true positives when their IoU reaches 0.50, while mAP75 uses a 0.75 IoU threshold.The general mAP metric is defined across IoU thresholds, whereas mAP75 fixes the threshold at 0.75.
  • B. Experiment 1: Input Text; Output Semantic Mask: CLIP-DINOiser and C-RADIOv3 produce dense patch-level features rather than instance masks, so mAP is not reported for them.Their outputs do not directly support the instance-level mAP evaluation used for the other models.

C. Experiment 2: Input Mask; Output Top-K Semantic Masks

Experiment 2 evaluates intra-modal semantic alignment by asking whether a ground-truth mask retrieves other masks from the same semantic class. Models rank mask embeddings by cosine similarity and use top-k retrieval accuracy.

  • C. Experiment 2: Input Mask; Output Top-K Semantic Masks: Given a ground-truth mask, the experiment ranks all other masks by embedding cosine similarity and checks whether a top-k result shares its semantic class.A retrieval is correct when at least one of the k most similar masks belongs to the query mask’s class.
  • C. Experiment 2: Input Mask; Output Top-K Semantic Masks: CLIP and SigLIP encode each mask directly, whereas the other included models use masked average pooling over dense visual features.The resulting embeddings are compared with cosine similarity for ranked retrieval.
  • C. Experiment 2: Input Mask; Output Top-K Semantic Masks: Top-k accuracy measures whether at least one of the k most similar retrieved masks matches the query mask’s class.The metric averages this indicator over the eligible mask instances.
  • C. Experiment 2: Input Mask; Output Top-K Semantic Masks: Classes with fewer than two instances are excluded because no other same-class mask can be retrieved for their queries.This restriction applies to the mask-to-mask evaluation setup.

D. Experiment 3: Input Mask; Output Top-K Text Classes

Experiment 3 evaluates whether a ground-truth mask can be associated with semantic class words. It compares visual and text embeddings in a shared space using cosine similarity and top-k selection, with several models excluded when direct comparison is unavailable.

  • D. Experiment 3: Input Mask; Output Top-K Text Classes: The experiment ranks class words by cosine similarity to each ground-truth mask embedding and evaluates whether the top-k words identify its semantic class.Visual and word embeddings are generated in a shared embedding space for direct comparison.
  • D. Experiment 3: Input Mask; Output Top-K Text Classes: CLIP, SigLIP, C-RADIOv3, and CLIP-DINOiser generate text embeddings through their respective text encoders.CLIP-DINOiser uses CLIP’s text encoder, while C-RADIOv3 uses its SigLIP2 adaptor’s text encoder.
  • D. Experiment 3: Input Mask; Output Top-K Text Classes: Grounded-SAM and SAM3 are excluded because their visual and text embeddings do not share a directly comparable embedding space.YOLOE-seg is also excluded because its visual encoder features are inaccessible.
  • D. Experiment 3: Input Mask; Output Top-K Text Classes: The experiment uses top-k accuracy to evaluate mask-to-text class association.The metric follows the top-k accuracy definition used for the preceding mask-to-mask experiment.

A. Experiment 1: Text-to-Semantic Mask

Experiment 1 evaluates text-to-mask alignment across coastal classes and datasets. Landscape classes perform best, while cross-dataset comparisons and correlation analyses provide no consistent evidence that coastal context alone explains weaker performance.

  • Experiment 1: Text-to-Semantic Mask: The evaluation measures text-to-mask performance on the Hawaii dataset using metrics defined for Experiment 1.Table I summarizes each model’s performance, while the supplied passages do not provide its numerical values.
  • Experiment 1: Text-to-Semantic Mask: Landscape classes generally achieve the highest F1-scores, followed by conventional classes, while coastal classes achieve the lowest.Fig. 5 groups Hawaii classes into conventional, landscape, and coastal categories.
  • Experiment 1: Text-to-Semantic Mask: Spatial extent has weak, inconsistent associations with segmentation performance, with Spearman’s ρ ranging from −0.2253 to 0.3846 and no significant correlations.None of the correlations reaches p < 0.05.
  • Experiment 1: Text-to-Semantic Mask: Conventional-class F1-scores show no consistent performance trend across Hawaii, COCO, and GOOSE datasets.Performance differences vary in direction and magnitude across models, leaving domain-shift effects unresolved.

B. Experiment 2: Mask-to-Top-K Semantic Masks

Experiment 2 tests whether models can match visual regions across images, while Experiment 3 tests associations between regions and classifying text. Visual matching is broadly similar across conventional and coastal classes, whereas alternative terminology improves recognition of several coastal concepts.

  • Experiment 2: Mask-to-Top-K Semantic Masks: Models generally achieve similar mask-to-mask top-1 accuracy for conventional and coastal classes.Some models outperform others overall, but individual models typically show comparable accuracy across the two semantic groups.
  • Experiment 3: Mask-to-Top-K Text Classes: Remapping pier and wharf to dock and piling to pole improves accuracy for all evaluated models.The result suggests that specific marine terminology can contribute to coastal classification difficulty.
  • Experiment 3: Mask-to-Top-K Text Classes: Remapping shore-artificial and shore-natural to shore does not consistently improve performance, but exceeds at least one original label for every model.The result further indicates that descriptive terminology influences vision-language alignment.

VI. CONCLUSION

The study evaluates seven VLMs on coastal visual-language alignment and finds that performance varies by class type, segmentation difficulty, and terminology. Its results motivate broader coastal evaluation and improvements in segmentation and domain-specific language.

  • Seven VLMs evaluated across three alignment experiments generally recognized landscape classes more accurately than conventional and coastal classes.The evaluation used text-to-mask, mask-to-mask, and mask-to-text alignment on a densely labeled coastal dataset from Oahu, Hawaii.
  • Lower coastal-class performance was not attributable solely to environmental context or visual representation, with segmentation and linguistic representation contributing to the observed gap.The conclusion specifically identifies these factors as important contributors.
  • Replacing specific marine terms with more general or colloquial alternatives improved recognition for several coastal classes, showing that terminology influences vision-language alignment.
  • Future work will extend evaluation to additional coastal environments and datasets, examine domain shift more systematically, and improve segmentation and domain-specific linguistic representations.
Loading 2609.10855v1…