Source-linked AI summary
ReCo: Retrieve and Co-segment for Zero-shot Transfer
Gyungin Shin, Weidi Xie, Samuel Albanie
TL;DR
Semantic segmentation is costly to annotate and existing unsupervised methods often need labelled target examples to name predictions, while language-image models lack comparable segmentation ability. ReCo retrieves concept-specific images with CLIP, co-segments them using modern visual representations, and constructs pixel-label-free segmenters; it performs favourably to unsupervised approaches, supports rare concepts, and enables zero-shot transfer. Its scope depends on retrieved concepts appearing in the unlabelled collection, and CLIP makes inference computationally heavy.
Problem
Semantic segmentation faces expensive pixel annotation, limited category flexibility, and a lack of methods combining nameable predictions with zero-shot transfer without labelled target examples.
Method
ReCo uses CLIP to retrieve unlabelled images for requested concepts, co-segments recurring entities with visual correspondences, and builds segmentation models without pixel labels.
Results
ReCo strongly outperforms existing zero-shot transfer approaches that forgo pixel supervision, while segmenting rare concepts and supporting unsupervised adaptation through ReCo+.
Takeaways & Limitations
ReCo provides nameable open-vocabulary segmentation and zero-shot transfer, including specialist segmenters for rare objects such as fire extinguishers.
Takeaways & Limitations
ReCo may struggle when a concept does not appear in the large unlabelled collection, and its use of computationally heavy CLIP during inference limits efficiency.
Abstract
from arXiv · showhide
Semantic segmentation has a broad range of applications, but its real-world impact has been significantly limited by the prohibitive annotation costs necessary to enable deployment. Segmentation methods that forgo supervision can side-step these costs, but exhibit the inconvenient requirement to provide labelled examples from the target distribution to assign concept names to predictions. An alternative line of work in language-image pre-training has recently demonstrated the potential to produce models that can both assign names across large vocabularies of concepts and enable zero-shot transfer for classification, but do not demonstrate commensurate segmentation abilities. In this work, we strive to achieve a synthesis of these two approaches that combines their strengths. We leverage the retrieval abilities of one such language-image pre-trained model, CLIP, to dynamically curate training sets from unlabelled images for arbitrary collections of concept names, and leverage the robust correspondences offered by modern image representations to co-segment entities among the resulting collections. The synthetic segment collections are then employed to construct a segmentation model (without requiring pixel labels) whose knowledge of concepts is inherited from the scalable pre-training process of CLIP. We demonstrate that our approach, termed Retrieve and Co-segment (ReCo) performs favourably to unsupervised segmentation approaches while inheriting the convenience of nameable predictions and zero-shot transfer. We also demonstrate ReCo's ability to generate specialist segmenters for extremely rare objects.
1 Introduction
ReCo combines language-image retrieval with visual co-segmentation to enable open-vocabulary semantic segmentation without pixel labels or labelled target examples. It preserves zero-shot transfer and nameable predictions while supporting unsupervised adaptation when target data is available.
- Motivation: Manual pixel-level annotation can require 90 minutes per image, while supervised models also struggle with rare or novel categories.These costs and flexibility limits hinder deployment across applications.
- Motivation: Modern visual representations provide spatial object extents and cross-image correspondences, while language-image pre-training supplies broad vocabularies and zero-shot transfer.ReCo is motivated by combining these complementary capabilities.
- Approach: ReCo retrieves concept-specific images with CLIP, co-segments their entities using visual correspondences, and constructs a segmentation model without pixel labels.The resulting predictions retain wide-vocabulary naming and zero-shot transfer.
- Contributions: ReCo enables open-vocabulary segmentation without pixel annotations or labelled target examples, and ReCo+ uses unsupervised adaptation when target-distribution data is available.The framework is evaluated on COCO-Stuff, Cityscapes, and KITTI-STEP, with additional rare-concept demonstrations.
2 Related work
Prior work addresses unsupervised segmentation, weak supervision, zero-shot transfer, large vocabularies, and co-segmentation through distinct mechanisms. ReCo differs by combining nameable open-vocabulary predictions with independence from labelled target examples and pixel annotations.
- Unsupervised semantic segmentation: Unsupervised segmentation methods use self-supervised objectives and often require labelled examples or Hungarian matching to assign names to segments.ReCo remains independent of labelled examples during training and inference.
- Weakly-supervised semantic segmentation: Weakly supervised methods reduce annotation demands but still attach weak annotations to training data, whereas ReCo can use an unlabelled image collection.Webly-supervised methods additionally depend on image-search infrastructure and may not leverage search knowledge during inference.
- Zero-shot semantic segmentation: Zero-shot segmentation methods use pretrained word or vision-language embeddings to generalize to unseen categories, while ReCo targets unseen datasets without target-distribution examples.ReCo adopts a variation of DenseCLIP within this formulation.
- Large-vocabulary/rare concept segmenters: Large-vocabulary and rare-concept segmenters have explored captions, grounded descriptions, annotation transfer, and incremental learning, but still require costly pixel-level annotations.ReCo instead inherits concept knowledge from language-image pre-training without pixel annotations.
- Co-segmentation: Co-segmentation methods identify common regions across image collections using classical, deep, iterative, or weakly supervised approaches.ReCo can use unsupervised co-segmentation and adopts a simple correlation strategy.
3 Method
ReCo constructs concept segmenters on the fly by retrieving relevant images, co-segmenting recurring regions, and applying the resulting reference embeddings to new images. Language-guided gating and context suppression reduce distraction from salient co-occurring backgrounds.
- Framework: ReCo takes unlabelled images and text descriptions, then combines retrieval and co-segmentation to build a segmenter applied without fine-tuning to a target distribution.This pipeline supports zero-shot transfer.
- Retrieve: The method assumes large unlabelled collections contain examples of desired concepts, enabling CLIP to retrieve a k-image archive for each text query.CLIP compares text and image embeddings and selects nearest-neighbour images.
- Co-segment: ReCo extracts L2-normalised dense features from each archive image with a pre-trained encoder, which may be distinct from CLIP.Each feature map has d channels over spatial dimensions h × w.
- Co-segment: It identifies archive seed pixels through cross-image feature similarities, averages the k seed embeddings, and L2-normalises the result into a concept reference embedding.The reference embedding serves as the classifier for the retrieved concept.
- Inference: For a new image, ReCo computes dot products between the reference embedding and dense features, then applies a sigmoid to produce an initial probability map.The map represents the estimated probability of the category at each spatial location.
- Inference: DenseCLIP supplies a concept saliency map, and ReCo multiplies it with the initial estimate to refine the final probability map.Multiple category maps are concatenated and resolved with argmax; a CRF is optionally evaluated as post-processing.
- Context control: Language-guided co-segmentation gates similarities toward pixels salient for the target concept, while context elimination suppresses common background categories such as trees, sky, and roads.These mechanisms address distractors that co-occur with target concepts.
4 Experiments
ReCo is evaluated through ablations, benchmark comparisons, and rare-concept segmentation experiments. Results show strong zero-shot transfer, improved performance from several components, and the ability to segment rare objects, while limitations remain around concept coverage, inference cost, and data bias.
- Experimental setup: Experiments evaluate ReCo through ablations, standard semantic-segmentation benchmarks, and rare-concept datasets.The evaluation covers ImageNet1K archive curation, PASCAL-Context ablations, COCO-Stuff, Cityscapes, KITTI-STEP, and FireNet.
- Ablations: ViT-L/14@336px performs particularly strongly for CLIP retrieval, while larger archives generally improve co-segmentation performance across visual encoders.The archive-size trend is improving but non-monotonic, and ViT-L/14@336px is selected for subsequent retrieval experiments.
- Ablations: DenseCLIP, language-guided co-segmentation, context elimination, and CRF post-processing each contribute to improved zero-shot transfer performance.The component study evaluates these additions on PASCAL-Context using DeiT-SIN for co-segmentation and ViT-L/14@336px for archive curation.
- Benchmark comparisons: ReCo strongly outperforms prior models across COCO-Stuff, Cityscapes, and KITTI-STEP under zero-shot transfer, while ReCo+ leads under unsupervised adaptation on Cityscapes and KITTI-STEP.On COCO-Stuff, ReCo+ has slightly lower pixel accuracy than [24] but considerably higher mIoU.
- Rare concepts: On FireNet, ReCo achieves pixel accuracy 93.3 and IoU 44.9 for fire extinguishers, and it co-segments fire extinguishers and Antikythera mechanisms without labelled examples.These experiments demonstrate segmentation of rare concepts beyond standard benchmarks.
- Limitations: ReCo may struggle when concepts are absent from billion-image datasets, uses a computationally heavy CLIP model during inference, and is primarily evaluated on object-centric ImageNet data.The authors also note that newly emerged concepts unavailable during CLIP training cannot be archived, and target-distribution data are still used for some procedures.
5 Conclusion
The conclusion presents ReCo as a framework for semantic-segmentation zero-shot transfer. It combines language-image pre-training with modern visual backbones to name and segment rare concepts without labelled target-domain examples, outperforming existing zero-shot approaches that forgo pixel supervision.
- Conclusion: ReCo combines large-scale language-image pre-training with modern visual backbones for semantic-segmentation zero-shot transfer.The framework supports direct concept naming and rare-concept segmentation without labelled examples from the target distribution.
- Conclusion: ReCo strongly outperforms existing zero-shot transfer approaches that forgo pixel supervision.
Appendices
The supplementary material expands the paper with broader-impact discussion, supervision analysis, dataset details, experiment and training details, and additional ablations.
- Appendix A: The appendices discuss broader impact, supervisory signals, alternative approaches, and the datasets used in the work.
- Appendices B–C: Additional material provides experiment details, training hyperparameters, and ablations on context-category selection and category-name ambiguity.
A Discussion of broader impact, supervision and data
ReCo has broad potential benefits but also lowers the barrier to both beneficial and harmful uses of semantic segmentation. Its evidence and deployment scope remain constrained by human involvement in data curation and by biases in benchmark imagery.
- Semantic segmentation supports applications in medical imaging, wildlife monitoring, and manufacturing, but can also enable unlawful surveillance or privacy invasion.
- ReCo relies on large-scale unlabelled image collections that may contain demographic biases and content inconsistent with users’ ethical values.
- The authors describe ReCo as a proof of concept requiring deployment-specific analysis and safeguards for archive curation before real-world use.
- ReCo and other methods may involve human participation in data curation, complicating claims of fully unsupervised learning.
- Human-curated photographs and ImageNet-based archives can make experimental performance optimistic relative to less curated deployment distributions.
A.3 Discussion of consent in used datasets
The study uses established computer-vision benchmarks and does not independently investigate consent for their imagery. For LAION-5B qualitative studies, the authors manually checked the curated archives for human presence.
- The authors do not conduct an independent consent investigation for widely used computer-vision benchmarks.
- For LAION-5B qualitative studies, the authors manually verified that curated archives contained no humans.
A.4 Discussion on whether data contains personally identifiable information or offensive content
The authors do not release data and primarily use public-domain benchmark imagery, which they assess as limiting privacy and harmful-content risks. For LAION-5B studies, they manually screened curated archives.
- The authors release no data and rely on widely used public-domain computer-vision benchmarks, which they believe keeps privacy and harmful-content risks low.
- For LAION-5B qualitative studies, the authors manually verified that ReCo-curated archives contained no personally identifiable information or harmful content as judged by the authors.
A.5 Dataset licenses
The paper states dataset licensing terms and supplies pseudocode for ReCo’s retrieval, co-segmentation, reference-embedding, and inference procedures. It also documents an ambiguity-related exclusion from one evaluation.
- Dataset licenses: PASCAL-Context and COCO images follow Flickr Terms of Use, while COCO-Stuff annotations use the Creative Commons Attribution 4.0 License.
- Dataset licenses: Cityscapes and ImageNet1K follow their official website terms, while KITTI-STEP uses the CC BY-NC-SA 3.0 licence.
- ReCo pseudocode: ReCo’s core pseudocode takes CLIP encoders, an unlabelled image collection, a concept, and archive size, then outputs a reference embedding and new-image prediction.
- ReCo pseudocode: Retrieval normalises image and text embeddings, scores their matrix products, and selects the top-k images for each concept.
- ReCo pseudocode: Co-segmentation computes dense feature affinities across the archive, aggregates per-image support, selects seed pixels, and averages their features into a normalised reference embedding.
- Evaluation scope: Two ambiguous ImageNet1K labels, including crane, are excluded, leaving 996 categories for the reported Fig. 3 left results.
B.4 Hyperparameters for ReCo+ training
ReCo+ training uses standard DeepLabv3+ with ResNet101 and common geometric and photometric augmentations. Additional experiments show that suppressing frequent context categories improves performance.
- Hyperparameters: ReCo+ uses DeepLabv3+ with a ResNet101 encoder and standard data augmentations.Geometric augmentation includes random scaling, 320×320 crops, and horizontal flips; photometric augmentation includes colour jittering and Gaussian blurring.
- Hyperparameters: Random scaling ranges from 0.5 to 2.0, with 320×320 random crops and horizontal flips applied with probability 0.5.
- Context categories: Suppressing five frequently appearing categories improves performance on PASCAL-Context, with all five categories performing best.Suppressing tree and sky already produces a notable gain.
C.2 Category name rephrasing to reduce ambiguity
ReCo benefits from concrete category names because its retrieval and co-segmentation depend on visually consistent images. The approach remains sensitive to ambiguous labels and struggles with small objects.
- Category name rephrasing: Rephrasing ambiguous labels such as parking and vegetation as parking lot and tree improves ReCo performance on Cityscapes.The more concrete names retrieve images with more similar visual appearance.
- Limitations: Dependence on concrete, specific concept names is a limitation because ReCo relies on text descriptions to disambiguate visual concepts without target-domain examples.This requirement is readily mitigated by rewriting ambiguous category names.
- Failure cases: ReCo and ReCo+ typically fail on small objects such as people inside buses, potentially because ReCo uses a 16×16 image-encoder stride.A smaller stride could improve this issue at increased computational cost.