PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoPrompt: Geometry-Conditioned Prompt Learning for Underwater Open-Vocabulary Instance Segmentation

Shanshan Han, Yuxi Wang, Can Qin, Yujie Chen, Cong Pan, Haitao Li, Guoqiang Zhong

arXiv:2610.34190v1cs.CV

TL;DR

Underwater appearance degradation weakens the visual-text alignment used by open-vocabulary instance segmentation, especially for unseen categories. GeoPrompt adapts visual features and textual classifiers using depth-derived geometry, and improves MARIS benchmark mAP over the baseline. Its results support geometry-conditioned adaptation in this setting, though difficult objects and noisy geometry remain challenging.

  • Problem

    Underwater appearance degradation can make visual region features unreliable matches for textual classifiers, particularly for unseen categories that rely on transferable semantic correspondence.

    Source: Introduction · p. 1

  • Method

    GeoPrompt injects depth-derived structural features into multi-scale visual representations and uses geometry-conditioned prompts to generate textual classifiers alongside fixed and image-conditioned classifiers.

    Source: Introduction · p. 2

  • Results

    6.73 points: GeoPrompt improves overall mAP from 44.37 to 51.10 over the MARIS baseline, and open-vocabulary mAP from 39.77 to 48.20.

    Source: Quantitative Results · p. 6

  • Takeaways & Limitations

    GeoPrompt improves geometry-conditioned vision-language alignment for seen and unseen underwater categories without requiring ground-truth depth annotations.

    Source: Introduction · p. 2, Conclusion · p. 7

    Extremely small objects, heavy occlusion, ambiguous boundaries, or noisy depth-derived geometry features may still lead to incomplete predictions.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Open-vocabulary instance segmentation (OVIS) provides a promising solution for marine perception by recognizing and segmenting categories beyond predefined label spaces. Existing OVIS methods usually rely on vision-language alignment, where visual region features are matched with textual category embeddings to recognize both seen and unseen classes. However, this alignment assumption becomes fragile in underwater environments. Color attenuation, scattering, turbidity, and low contrast distort visual appearance, causing region features to deviate from their corresponding semantic concepts. As a result, the similarity between degraded visual features and textual classifiers becomes unreliable, which particularly weakens the recognition of unseen underwater categories that cannot be corrected by category-specific supervision. To address these issues, we propose GeoPrompt, a geometry-conditioned prompt learning framework for underwater open-vocabulary instance segmentation. Our key insight is that geometric structure is more stable than visual appearance in degraded underwater scenes. Specifically, GeoPrompt introduces a geometry-guided visual adaptation module (GVAM) to extract depth-derived structural priors and inject them into multi-scale visual features. It further develops a geometry-conditioned prompt learning (GCPL) that generates geometry-aware textual classifiers from structural representations. These geometry-aware classifiers are integrated with fixed and image-conditioned textual classifiers to improve open-vocabulary semantic alignment. Experiments on the MARIS-OpenIS benchmark show that GeoPrompt achieves state-of-the-art performance and brings consistent improvements on underwater open-vocabulary instance segmentation.

Introduction

Underwater appearance degradation undermines vision-language alignment, particularly for unseen categories, while fixed or image-conditioned prompts may not adapt reliably. GeoPrompt uses depth-derived geometry to adapt both visual representations and semantic classifiers without requiring ground-truth depth annotations.

Introduction

  • Marine perception includes diverse, fine-grained, long-tail categories that can be difficult to predefine and annotate, motivating recognition beyond closed-set labels.Figure 1 contrasts incomplete or inaccurately localized appearance-only predictions with more complete GeoPrompt predictions using depth-derived geometry.Source: Introduction · p. 1, Figure 1 · p. 2
  • Underwater color attenuation, scattering, turbidity, and low contrast distort visual features, weakening alignment with textual concepts, especially for unseen categories.Fixed templates cannot adapt to scene-specific degradation, while image-conditioned prompts may inherit appearance drift from RGB features.Source: Introduction · p. 1
  • Depth-derived features provide complementary structural cues without requiring ground-truth depth annotations or explicit depth supervision.The paper uses them as structural priors rather than treating monocular depth as a target prediction.Source: Introduction · p. 2
  • GeoPrompt adapts both visual representations and semantic prompts, framing underwater OVIS as geometry-conditioned vision-language alignment.Its GVAM injects depth-derived structural priors into multi-scale visual features, while GCPL generates geometry-aware textual classifiers integrated with fixed and image-conditioned classifiers.Source: Introduction · p. 2

Related Work

OVIS matches visual region embeddings to textual classifiers to recognize seen and unseen categories, but underwater appearance degradation makes that alignment fragile. Prior approaches use prompts or geometric cues, while geometry-guided textual classifier adaptation remains underexplored.

Open-Vocabulary Instance Segmentation

  • OVIS commonly separates mask generation from category prediction, matching visual region embeddings to textual classifiers to recognize beyond predefined labels.Most approaches assume RGB features provide reliable vision-language alignment cues, an assumption that is fragile underwater.Source: Open-Vocabulary Instance Segmentation · p. 2
  • Geometry-guided self-distillation has also been explored for open-vocabulary 3D scene understanding, while domain-adaptation studies address distribution shifts in semantic segmentation.These related studies do not directly target underwater OVIS.Source: Open-Vocabulary Instance Segmentation · p. 3

Geometry-Conditioned Prompt Learning

  • Prior underwater OVIS work and RGB-D methods primarily use geometry to enhance visual representations, leaving geometry-conditioned textual classifier adaptation underexplored.GeoPrompt addresses this gap by generating textual classifiers from depth-derived structural features and integrating them with fixed and image-conditioned classifiers.Source: Open-Vocabulary Instance Segmentation · p. 3, Geometry-Conditioned Prompt Learning · p. 3
  • Prompt methods range from learned task-specific context to image-conditioned and multimodal prompts, but adaptive prompts depend on reliable feature inputs.Underwater RGB degradation can affect both region embeddings and image-level features used to generate prompts.Source: Geometry-Conditioned Prompt Learning · p. 3

Method

GeoPrompt adapts both visual representations and textual classifiers using depth-derived geometry. It combines these adaptations for open-vocabulary mask prediction under underwater degradation.

Method

  • GeoPrompt combines depth-guided visual adaptation, geometry-conditioned prompt learning, and vision-language alignment for seen and unseen underwater categories.GVAM injects structural priors into multi-scale visual features, while GCPL generates geometry-aware textual classifiers.Source: Method · p. 4
  • A pretrained monocular depth model supplies multi-stage structural features that a lightweight geometry adapter transforms into task-specific representations.These representations complement degraded RGB appearance.Source: Geometry-Guided Visual Adaptation · p. 4
  • At each scale, VGMI fuses adapted geometry with visual features through concatenation, channel transformation, and channel-wise gating.The channel gate emphasizes informative channels, while a residual connection preserves original visual semantics.Source: Geometry-Guided Visual Adaptation · p. 4
  • Geometry context is projected into prompt tokens, combined with category-name embeddings, and encoded by the frozen CLIP text encoder to form geometry-aware classifiers.The geometry context aggregates adapted multi-scale features; these structural cues complement RGB appearance.Source: Geometry-Conditioned Prompt Learning · pp. 4–5
  • The final textual classifier normalizes a weighted mixture of fixed, image-conditioned, and geometry-conditioned classifiers.The coefficients λ and αg control the image-conditioned and geometry-conditioned contributions, respectively.Source: Geometry-Conditioned Prompt Learning · p. 5
  • The decoder assigns open-vocabulary labels by matching visual mask embeddings with the final classifier, and training uses classification, mask, and DICE losses with Hungarian matching.Deep supervision is applied to auxiliary decoder outputs.Source: Geometry-Conditioned Vision-Language… · p. 5

Experiments

On MARIS-OpenIS, GeoPrompt outperforms the MARIS baseline and achieves the best performance across intersection, open-vocabulary, and overall categories. Ablations and visualizations examine its components and image-side feature responses.

Experimental Setup

  • MARIS-OpenIS contains more than 16K images, 9 super-classes, and 158 fine-grained subclasses; evaluation reports mask AP, AP50, and AP75, with category-split AP where available.The official split has 5,712 training images and 10,439 validation images; 41 categories overlap between the 84 training and 115 validation categories.Source: Experimental Setup · p. 5

Quantitative Results

  • 6.73 points overall mAP (44.37 to 51.10) and 8.43 points open-vocabulary mAP (39.77 to 48.20): GeoPrompt outperforms MARIS and is best across intersection, open-vocabulary, and overall categories.The comparison uses the same in-domain setting as the benchmark-reported methods.Source: Table 1 · p. 6, Quantitative Results · p. 6

Qualitative Results

  • GeoPrompt produces more complete masks, clearer boundaries, and more reliable localization and separation of instances than MARIS in challenging scenes.The examples include small or ambiguous instances, partially missed targets, and adjacent objects.Source: Qualitative Results · p. 6, Figure 3 · p. 6

Main Component Ablation

  • GVAM alone raises overall mAP from 44.37 to 50.83, GCPL alone reaches 46.55, and combining both yields 51.10 overall and 56.34 intersection-class mAP.The component results support the paper's description of visual-side geometry adaptation and prompt-side classifier generation as complementary.Source: Ablation Experiments · pp. 6–7, Table 2 · p. 7

Feature Response Visualization

  • After visual-geometric fusion, responses concentrate more on object regions, while geometry responses highlight contours and spatial layouts despite appearance degradation and background clutter.The visualization contrasts RGB visual responses, depth-derived geometry responses, and their fused response.Source: Ablation Experiments · p. 7, Figure 4 · p. 7

Conclusion

GeoPrompt combines depth-derived geometry with visual adaptation and adaptive textual classifiers for underwater OVIS, without requiring ground-truth depth. On MARIS, it raises OV mAP from 39.77 to 48.20, though difficult objects and noisy geometry remain challenges.

Conclusion

  • OV mAP improves from 39.77 to 48.20 on the MARIS benchmark.The paper reports stronger generalization to unseen underwater categories.Source: Conclusion · p. 7
  • GeoPrompt jointly enhances multi-scale visual features with depth-derived geometry and generates adaptive textual classifiers from image-level and geometry-level contexts.It does not require ground-truth depth annotations, making it practical for RGB-only marine datasets.Source: Conclusion · p. 7
  • Extremely small objects, heavy occlusion, ambiguous boundaries, and noisy depth-derived geometry can still lead to incomplete predictions.Future work will explore robust geometry priors and stronger semantic discrimination for complex marine scenarios.Source: Conclusion · p. 7

Related papers

Loading 2610.34190v1…