PaperlayerView exact revision on arXiv

Source-linked AI summary

Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching

Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, Chunhua Shen

arXiv:2305.13310v2cs.CV

TL;DR

Vision foundation models have strong open-world potential but generally require task-specific structures and fine-tuning for perception tasks. Matcher combines off-the-shelf models with one in-context example and three components, achieving broad training-free segmentation generalization, including 52.7% mIoU on COCO-20i and 33.0% mIoU on LVIS-92i. Its instance-level matching remains limited for instance segmentation.

  • Problem

    Vision foundation models require task-specific structures and fine-tuning for varied perception tasks, unlike language models that directly support diverse language tasks.

  • Method

    Matcher integrates off-the-shelf vision foundation models and uses one in-context example with bidirectional matching, robust prompt sampling, and instance-level matching.

  • Results

    Matcher generalizes across diverse segmentation tasks without training, achieving 52.7% mIoU on COCO-20i and 33.0% mIoU on LVIS-92i.

  • Takeaways & Limitations

    Matcher demonstrates that combining foundation models can provide open-world generality and flexibility across few-shot segmentation tasks.

  • Takeaways & Limitations

    Matcher has relatively limited instance-level matching inherited from its image encoder, restraining performance for instance segmentation.

Abstract

from arXiv · show

Powered by large-scale pre-training, vision foundation models exhibit significant potential in open-world image understanding. However, unlike large language models that excel at directly tackling various language tasks, vision foundation models require a task-specific model structure followed by fine-tuning on specific tasks. In this work, we present Matcher, a novel perception paradigm that utilizes off-the-shelf vision foundation models to address various perception tasks. Matcher can segment anything by using an in-context example without training. Additionally, we design three effective components within the Matcher framework to collaborate with these foundation models and unleash their full potential in diverse perception tasks. Matcher demonstrates impressive generalization performance across various segmentation tasks, all without training. For example, it achieves 52.7% mIoU on COCO-20$^i$ with one example, surpassing the state-of-the-art specialist model by 1.6%. In addition, Matcher achieves 33.0% mIoU on the proposed LVIS-92$^i$ for one-shot semantic segmentation, outperforming the state-of-the-art generalist model by 14.4%. Our visualization results further showcase the open-world generality and flexibility of Matcher when applied to images in the wild. Our code can be found at https://github.com/aim-uofa/Matcher.

1 INTRODUCTION

Matcher introduces a training-free perception framework that combines vision foundation models and an in-context example to address diverse segmentation tasks. Its components support broad generalization across semantic, part, and video object segmentation.

  • Motivation: Matcher explores vision foundation models for diverse perception tasks without training, motivated by their gap with language models in directly supporting varied tasks.The framework targets semantic segmentation, part segmentation, and video object segmentation.
  • Approach: Matcher uses one in-context example to understand the task, match target semantics with DINOv2 features, and guide segmentation with foundation models.The framework combines correspondence extraction, prompt generation, and controllable mask generation.
  • Results: 52.7% mIoU on COCO-20i surpasses the state-of-the-art specialist model by 1.6%, while 33.0% mIoU on LVIS-92i exceeds SegGPT by 14.4%.These are reported one-shot semantic segmentation results.
  • Results: Matcher outperforms competing methods by about 10.0% mean mIoU on both proposed one-shot object part segmentation benchmarks and achieves competitive video object segmentation performance.The reported results span semantic, part, and video object segmentation.
  • Approach: Matcher combines bidirectional matching, robust prompt sampling, and instance-level matching to improve segmentation quality and open-set generality.These three components are designed to unleash the capabilities of vision foundation models.

2 RELATED WORK

Related work covers vision foundation models and efforts to unify multiple segmentation tasks under generalist models. Matcher is positioned within this broader movement toward reusable visual representations and task generalization.

  • Vision Foundation Models: Vision foundation models use large-scale pre-training strategies such as masked image modeling and image-text contrastive learning to learn transferable visual representations.The passage discusses MAE and CLIP as representative approaches.
  • Positioning: Matcher differs from these lines of work by targeting diverse segmentation tasks through a training-free framework using off-the-shelf foundation models.Its approach combines an all-purpose feature extractor with a class-agnostic segmentation model.
  • Vision Generalist for Segmentation: Generalist segmentation models unify multiple tasks with Transformer architectures and in-context training on supervised datasets.Painter represents task outputs as images, while SegGPT extends this approach with random coloring.

3 METHOD

Matcher is a training-free, one-shot framework that combines feature matching with class-agnostic segmentation to transfer object or part semantics from a reference image to a target image. Its pipeline extracts patch correspondences, generates diverse prompts, and selects high-quality masks through instance-level matching.

  • Framework overview: Matcher integrates an all-purpose feature extractor with a class-agnostic segmenter to segment target objects or parts matching a reference image and mask.The framework uses off-the-shelf models such as DINOv2 and SAM without training.
  • Prompts Generation: Bidirectional matching performs forward matching, reverse matching, and mask filtering to remove correspondence outliers before prompt generation.The final points retain forward matches whose reverse correspondences lie on the reference mask.
  • Correspondence Matrix Extraction: Patch-wise feature similarity forms a dense correspondence matrix that identifies regions in the target image matching the reference mask.The correspondence matrix uses cosine similarity between reference and target patch features, although appearance and category differences make matching challenging.
  • Prompts Generation: The robust prompt sampler clusters matched points and samples part-level, instance-level, and global prompts to provide varied semantic coverage.These prompts increase mask-proposal diversity and suppress fragmented false positives caused by matching outliers.
  • Controllable Masks Generation: Instance-level matching ranks mask proposals using feature-based structural distance together with purity and coverage, then merges selected masks into the final output.The score combines EMD, purity, and coverage, while the number of merged masks controls outputs for same-semantic instances.
  • Controllable Masks Generation: The proposal score combines EMD, purity, and coverage through tunable coefficients to select high-quality masks with controllable merging.The score is defined as score = α · (1 − emd) + β · purity · coverage^λ.

4 EXPERIMENTS

Matcher is evaluated across few-shot semantic, object-part, and video segmentation, with ablations and qualitative results assessing its training-free generalization. It performs competitively or better across these settings while combining foundation-model matching with SAM.

  • Few-shot semantic segmentation: 52.7% mean mIoU on COCO-20i is achieved in one-shot semantic segmentation, surpassing the state-of-the-art specialist model by 1.6%.
  • Few-shot semantic segmentation: Matcher surpasses training-free PerSAM and fine-tuned PerSAM-F by 29.2% mean mIoU on COCO-20i, 11.4% mIoU on FSS-1000, and 10.7% mean mIoU on LVIS-92i.
  • One-shot object-part segmentation: Matcher exceeds PerSAM by 12.8% mean mIoU on PASCAL-Part and 13.5% on PACO-Part for one-shot object-part segmentation.These benchmarks evaluate fine-grained part segmentation, where SAM alone lacks semantics for distinguishing ambiguous masks.
  • Video object segmentation: Matcher achieves competitive video object segmentation performance against models trained with video data and outperforms SegGPT and PerSAM-F on both DAVIS datasets.The results support generalization to VOS without training.
  • Ablation studies: Bidirectional matching improves performance by 2.1% mean mIoU on COCO-20i, 5.9% mIoU on FSS-1000, and 6.0% J&F on DAVIS 2017.The ablation attributes these gains to combining forward and reverse matching rather than reverse matching alone.
  • Ablation studies: Performance on DAVIS 2017 improves as more frames are used, reaching its optimum with four frames.The study also finds different mask-proposal metrics are preferable for COCO-20i and DAVIS 2017.

5 CONCLUSION

Matcher integrates off-the-shelf vision foundation models into a training-free framework for diverse few-shot segmentation tasks. Its experiments indicate strong performance and open-world flexibility, while instance segmentation remains constrained by limited instance-level matching inherited from the image encoder.

  • Matcher is a training-free framework that integrates off-the-shelf vision foundation models for diverse few-shot segmentation tasks.
  • Bidirectional matching, robust prompt sampling, and instance-level matching are universal components intended to unleash the capabilities of the foundation models.
  • Experiments show strong performance across few-shot segmentation tasks, while visualizations show open-world generality and flexibility on images in the wild.
  • Limited instance-level matching inherited from the image encoder restrains Matcher’s performance for instance segmentation.The paper identifies instance-level segmentation as future work despite comparable VOS performance and controllable mask outputs.

A MORE DETAILS OF INSTANCE-LEVEL MATCHING

Matcher uses optimal transport and Earth Mover’s Distance to compare reference and proposed-mask features, while purity and coverage guide whether selected masks represent parts or whole instances.

  • Instance-Level Matching: Earth Mover’s Distance formulates instance-level matching as an optimal-transport problem over dense semantic features inside reference and proposed masks.The objective is to minimize transportation cost between suppliers and demanders.
  • Instance-Level Matching: Reference-mask patches act as suppliers, while patches covered by SAM’s mask proposal act as demanders.Both sides use equal-valued goods in Matcher.
  • Instance-Level Matching: The transport cost c_ij is derived from a cost matrix computed using the mask proposal and reference mask, after which EMD is calculated.The transported quantity π_ij defines the plan between each supplier-demand pair.
  • Purity and Coverage: Higher purity favors part-level or single-instance masks, whereas higher coverage favors whole-object or multiple-instance masks.These criteria support filtering false-positive fragments and selecting high-quality masks.

B IMPLEMENTATION DETAILS

Matcher combines DINOv2 ViT-L/14 with SAM ViT-H and uses fixed, training-free settings tailored to semantic, part, and video segmentation.

  • Model and Training Setup: DINOv2 ViT-L/14 is the default image encoder and SAM ViT-H is the default segmenter, with no Matcher training in any experiment.Input sizes are 518 × 518 for semantic and part segmentation and 896 × 504 for video segmentation.
  • Configuration: Video object segmentation samples global prompts from centers and uses an EMD threshold of 0.75 with α = 0.4, β = 1.0, and λ = 1.0.The video configuration differs from the semantic and part-segmentation settings.

C DATASET DETAILS

PASCAL-Part and PACO-Part provide part-segmentation benchmarks organized by semantic categories and filtered examples, with their taxonomies documented in the accompanying tables.

  • PASCAL-Part: PASCAL-Part contains four superclasses: animals, indoor, person, and vehicles.Its categories are built from PASCAL VOC 2010 and body-part annotations.
  • PASCAL-Part: PASCAL-Part includes five animal subclasses, three indoor subclasses, one person subclass, and six vehicle subclasses.The reported taxonomy includes bird, cat, cow, dog, horse, sheep, bottle, potted plant, TV monitor, person, and vehicle categories.
  • PACO-Part: PACO-Part is a more difficult one-shot object-part benchmark derived from PACO after filtering categories with one sample and extremely small-area examples.The filtering leaves 303 object parts, divided into four folds of about 76 parts each.
  • Dataset Taxonomies: Table 5 documents the PASCAL-Part taxonomy, while Table 6 documents the PACO-Part taxonomy.The tables provide the dataset-level category organization for the two benchmarks.

D ADDITIONAL RESULTS AND ANALYSIS

Additional analyses show that Matcher benefits from stronger foundation-model components, task-appropriate prompts, larger models, and richer references, while visualizations support its broad segmentation scope.

  • Image Encoders: DINOv2 achieves the best performance among CLIP, MAE, and DINOv2 across all evaluated datasets.The paper attributes this advantage to DINOv2’s patch-level feature matching ability.
  • Prompt Types: Part-level prompts are needed for PACO-Part, global prompts suit FSS-1000, and instance-level points are most effective for COCO-20i.Prompt effectiveness depends on whether the task targets parts, full-image instances, or multiple instances.
  • Model Size: Matcher’s performance continuously improves as the sizes of its foundation-model components increase.Even SAM Base and DINOv2 Base provide strong generalization on the reported datasets.
  • Segmenters: Semantic-SAM performs comparably to SAM on four benchmarks and better on PACO-Part because it produces more fine-grained masks.These results indicate that Matcher can operate as a general segmentation framework.
  • Upper Bound Analysis: The upper bound consistently exceeds current performance by a large margin across four datasets, indicating additional framework potential.The authors propose Matcher as an evaluation criterion for vision foundation models from a general segmentation perspective.
  • Few-Shot Segmentation: Few-shot segmentation concatenates multiple reference features for matching, yielding richer visual details, fewer outliers, and improved performance.The remaining processing is unchanged from the one-shot setting.
  • Visualizations: Visualizations cover background concepts, matching components, semantic segmentation, part segmentation, controllable masks, and video object segmentation.The reported results demonstrate improved segmentation quality and open-set generality.
Loading 2305.13310v2…