Source-linked AI summary
Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
Tilemachos Aravanis, Vladan Stojnić, Bill Psomas, Nikos Komodakis, Giorgos Tolias
TL;DR
OVS must bridge the gap between coarse VLM supervision and precise pixel-level prediction while handling ambiguous language. RNS adds pixel-annotated visual support and learns per-image fusion with textual features through a retrieval-augmented test-time adapter. Across benchmarks, it narrows the gap to supervised segmentation while retaining open-vocabulary generalization and supporting expanding support sets.
Problem
OVS remains behind fully supervised segmentation because VLMs use image-level supervision and natural language lacks precision for pixel-level tasks.
Method
RNS retrieves visual support features and fuses them with textual class features to train a lightweight per-image classifier on frozen VLM features.
Results
Across six benchmarks and two backbones, RNS consistently outperforms baselines and competitors while narrowing the gap between zero-shot and fully supervised segmentation.
Takeaways & Limitations
Dynamic support expansion makes RNS suited to open-world fine-grained applications such as personalized segmentation without sacrificing open-vocabulary generalization.
Takeaways & Limitations
SAM masks may over- or under-segment regions, merging distinct semantic areas or splitting single objects into multiple segments.
Abstract
from arXiv · showhide
Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.
1. Introduction
Open-vocabulary segmentation extends VLM-based zero-shot recognition to pixel-level prediction but remains limited by coarse supervision and language ambiguity. The paper introduces few-shot visual support and a retrieval-augmented adapter to improve segmentation while preserving open-vocabulary recognition.
- Fully supervised segmentation produces accurate masks but requires costly pixel annotations and cannot recognize categories unseen during training.
- OVS uses shared image-text embeddings to segment arbitrary categories specified at test time through text prompts or class names.
- The remaining supervision gap reflects both image-level training supervision versus fine-grained segmentation demands and ambiguity in natural-language supervision.
- The proposed few-shot setting supplements class-name prompts with pixel-annotated visual examples and trains a lightweight classifier for each test image.
- RNS retrieves relevant visual support, learns per-image fusion with textual prototypes, stores compact visual prototypes, and trains in under a second on an NVIDIA A100 GPU.The method avoids backbone retraining by steering predictions with a lightweight classifier over frozen VLM features.
- RNS supports partial and continually expanding support sets and extends to personalized segmentation while maintaining open-vocabulary predictions.
2. Related work
Prior work addresses OVS through segmentation-specific VLM training, few-shot adaptation, retrieval augmentation, and test-time adaptation. These approaches differ in supervision requirements, classifier construction, and assumptions about how test data arrive.
- OVS methods train VLMs with weak, image-caption, or fully supervised pixel-level data to improve dense localization beyond vanilla image-level supervision.
- Few-shot segmentation uses labeled support examples to form prototypes for novel classes, but typically assumes a closed world and does not leverage VLMs.
- Retrieval augmentation dynamically expands a model’s knowledge base, while segmentation methods apply it to in-context scene understanding.
- FREEDA retrieves generated visual examples to build a non-parametric visual classifier, whereas kNN-CLIP enhances OVS through retrieval-based methods using real support images for comparison.
- Many VLM test-time adaptation methods operate in batch or streaming settings with class-complete or i.i.d. assumptions, while single-image adaptation avoids those data-arrival assumptions.
3. Method
RNS formulates OVS with textual and pixel-annotated visual support, then adapts a lightweight classifier per test image using retrieved support features. It fuses visual and textual class information and produces patch- or region-level segmentation predictions.
- Task formulation: OVS assigns pixels to an arbitrary test-time class set specified by textual or visual support examples.Textual support consists of class names; visual support consists of annotated images, which may be partial or continually expanded.
- Zero-shot segmentation with VLMs: VLM image features form patch-level representations, while class names produce normalized textual class features for zero-shot prediction.Patch predictions are reshaped and upsampled to full resolution before assigning each pixel the highest-probability class.
- Test-time adaptation: For each query image, RNS retrieves nearest visual support features and trains a lightweight linear classifier on them and relevant fused features.Cross-entropy losses use class relevance weights to suppress irrelevant retrieved examples; the resulting classifier is applied to query patch features.
- Visual support features: Pixel annotations are down-sampled into patch-level labels used to pool features into per-image visual class features and a visual support feature set.The support feature set is the union of per-image class features extracted from annotated support images.
- Fused support features: RNS fuses each visual class feature with its textual counterpart using multiple mixing coefficients to create fused class features.The fused support feature set contains one feature for each class and coefficient, capturing complementary information from both modalities.
- Support maintenance and region predictions: RNS updates visual and fused support sets when new annotated images arrive, enabling dynamically expandable support during continual inference.With region proposals, region-level features replace patch features and predicted region labels are mapped back to full-resolution mask regions.
4. Experiments
Experiments evaluate RNS across open-vocabulary, partial-support, retrieval, closed-set, ablation, and personalized-segmentation settings. RNS improves over zero-shot and competing methods, narrows the supervised gap, and benefits from relevant retrieval, complementary support modalities, and test-time adaptation.
- Full textual and visual support: RNS consistently outperforms competitors across support sizes, backbones, and feature granularities, improving zero-shot segmentation by 7.3% with OpenCLIP and 18.4% with DINOv3.txt using one image per class.Textual priors help when support is sparse, while visual support dominates as the support set becomes denser.
- Partial support: RNS degrades smoothly when visual support is missing for classes and remains best as textual support becomes scarce, whereas kNN-CLIP and FREEDA deteriorate more sharply.The pseudo-label loss compensates for missing visual support, while text provides fusion and relevance weighting without making the model fragile when absent.
- Ablations: RNS components provide complementary benefits: class relevance weights suppress irrelevant retrieved classes, and a fixed λ = 0.8 particularly harms the low-shot regime.The ablation evaluates average mIoU for B = 1, B = 5, and B = 10.
- Retrieval analysis: Replacing retrieved features with random support causes substantial degradation, while restricting adaptation to semantically relevant retrieved classes improves performance over random sampling from all classes.Using the full retrieved-class feature set performs slightly below the default, indicating that RNS filters irrelevant examples.
- Closed-set comparisons: Offline pixel-level classifiers perform poorly in low-shot settings, while test-time retrieval dynamically selects relevant features and outperforms a frozen-backbone linear classifier across all shots.Jointly training the backbone is stronger with sufficient support but adds training complexity; combining those weights with RNS gives the best performance.
- Bridging the gap: With B = 20, RNS narrows the gap to fully supervised segmentation to 11.5 on average, improves the zero-shot baseline by 34, and surpasses CAT-Seg by 14.1.The improvement is more evident on fine-grained datasets such as CUB and Food.
5. Conclusion
RNS is a retrieval-augmented test-time adapter for open-vocabulary segmentation that learns lightweight per-image classifiers from frozen VLM features. It supports full and partial visual support and dynamically expanding support sets across diverse tasks.
- RNS fuses textual class features with retrieved visual support features to learn a lightweight per-image linear classifier on frozen VLM features.
- The method operates on patches or region proposals and handles full and partial support with a single objective.
- Across six benchmarks and two backbones, RNS consistently outperforms all baselines and competitors.
- Dynamic support makes RNS suited to open-world fine-grained tasks such as personalized segmentation.
Supplementary Overview
The supplementary material provides experimental details, additional results, and expanded comparisons supporting the main paper's evaluation. It covers protocols, implementations, competitors, baselines, and analyses across settings.
- The supplementary material details evaluation protocols, implementation specifics, competitor configurations, and offline baseline comparisons.
- Additional experiments cover out-of-domain support, partial visual support on fine-grained datasets, backbone comparisons, and seen/unseen class analysis.
- The supplement also includes a complete version of the main comparison table and expanded experimental results.
6. Experimental setup details
The experiments sample few-shot support sets from dataset training splits and evaluate RNS with multiple VLM backbones, feature configurations, and comparison protocols. Additional figures examine qualitative behavior, retrieval, tuning sensitivity, and backbone performance.
- Experimental protocol: Support images are sampled per class from training splits using four random seeds, with a procedure that preserves a realistic long-tail distribution.
- Support settings: RNS is evaluated with textual-only, visual-only, and combined textual-plus-visual support settings.
- Baselines and tuning: The linear offline classifier is tuned on an 85−15% train–validation split, while RNS remains comparatively stable across fixed hyperparameter configurations.
- Backbones: The study uses OpenCLIP ViT-B/16, DINOv3.txt ViT-L/16, and SigLIP2 ViT-L/16 backbones, with corresponding zero-shot baselines.
- Feature extraction: The evaluation includes patch features and optional SAM 2.1 region proposals, with dense support features extracted using sliding windows.
7. Additional experimental results
Additional experiments show that RNS remains effective across domains, retrieval conditions, backbones, support regimes, and inference budgets. Its relevance weighting and multi-neighbor retrieval improve robustness, while unsupported classes retain near-zero-shot behavior.
- Out-of-domain support: Out-of-domain visual support is weaker than in-domain support but still produces substantial gains over zero-shot segmentation and improves with more examples.
- Retrieval robustness: RNS benefits from support examples even with high retrieval error, while class relevance weights suppress irrelevant retrieved instances.
- Backbone comparison: DINOv3.txt ViT-L/16 achieves the highest mIoU and benefits most from additional support, while all tested backbones surpass their zero-shot baselines.
- Seen and unseen classes: Performance on seen classes remains nearly identical to full-support RNS when few classes are unseen, while unseen-class performance stays close to the zero-shot baseline.
- Efficiency and ablations: RNS remains robust with fewer training iterations, and K = 4–16 yields similar mIoU whereas K = 1 performs clearly worse.
8. Qualitative results
RNS produces sharper masks and fewer semantic confusions by combining learned textual and visual support, while SAM-based regions improve spatial alignment but can mismatch task-specific granularity.
- SAM mask vs patch-level predictions: SAM2.1 mask proposals yield noticeably sharper segmentation masks than fixed patches by following object boundaries and spatially denoising pooled features.They also group parts of the same object under challenging appearance changes such as shadows.
- SAM mask vs patch-level predictions: SAM can over- or under-segment regions, merging distinct semantic areas or splitting single objects despite improved image-structure alignment.
- Textual and visual support: Text-only predictions produce semantic ambiguity, while visual-only support confuses contextually similar objects; combining both modalities reduces these errors.Examples include house versus building, wall versus building, and train versus bus.
- Textual and visual support: RNS learns image-adaptive fusion instead of using fixed handcrafted fusion, yielding cleaner boundaries and fewer semantic confusions when modalities conflict or one is unreliable.
- Personalized segmentation: Figure 16 presents the visual support sets used for personalized segmentation, with support expanded from an initial set to a larger set.
- Comparative evaluations: The evaluated comparisons span zero-shot, RNS, kNN-CLIP, and FREEDA across full or partial textual and visual support, varying support images per class.The experiments use OpenCLIP or DINOv3.txt features and SAM2.1 region proposals in the reported figure settings.