PaperlayerView exact revision on arXiv

Source-linked AI summary

FoundYou: A Unified Model for Personalized Segmentation and Retrieval

Gabriele Trivigno, Marcos Alfaro, Claudia Cuttano, Gabriele Berton, Luis Payá, Carlo Masone

arXiv:2608.29917v1cs.CV

TL;DR

Personalized segmentation and retrieval both require identifying the same physical object across different images, yet they have largely been treated separately. FoundYou unifies them by adapting SAM 2 for cross-image instance matching while preserving its segmentation capability. It reports gains over unified and task-specific methods with a compact, efficient model.

  • Problem

    Personalized segmentation and retrieval share an instance-level matching objective but have largely evolved independently with different solutions.

  • Method

    FoundYou adapts SAM 2’s video-trained instance representations and memory attention for matching specific objects across independent images, using self-distillation to preserve segmentation structure.

  • Results

    FoundYou consistently outperforms unified and task-specific methods across personalized segmentation and retrieval while scaling with references and remaining robust to weaker prompts.

  • Takeaways & Limitations

    A frozen SAM 2 backbone can support unified instance-level segmentation and retrieval through lightweight adaptation for cross-image matching.

Abstract

from arXiv · show

Personalized segmentation and personalized retrieval both aim to identify the same physical object across different images. While the former localizes the object within a target image, the latter retrieves images where it appears. Despite this shared instance-level objective, the two tasks have largely evolved separately and are addressed with distinct solutions. In this work, we introduce FoundYou, a unified framework built on the observation that Segment Anything 2 (SAM 2), trained to preserve object identity across video frames, inherently captures instance-level cues. We leverage this property to match objects across independent images, enabling segmentation and retrieval to emerge as two outcomes of the same instance alignment process. This unified view unlocks new capabilities beyond traditional benchmarks, including few-shot personalized retrieval and promptable personalized segmentation with flexible prompts. Extensive experiments show consistent gains over unified and task-specific methods, including +18.4 mIoU on PerMIS and +17.8 mAP on ILIAS. Performance scales with additional references and remains robust to weaker prompts. Beyond personalization, FoundYou achieves state-of-the-art results on category-level retrieval benchmarks. Notably, our approach keeps the SAM 2-small model entirely frozen and adds only 5.9 M trainable parameters, yielding a 52 M-parameter model that is over 75x faster and 20x smaller than the only prior unified solution. Code is available at https://github.com/ga1i13o/FoundYou .

1 Introduction

Personalized segmentation and retrieval target the same physical instance across images, but have largely developed as separate tasks. FoundYou unifies them by adapting SAM 2’s instance-aware representations for cross-image matching, achieving strong performance and broader prompting capabilities.

  • Motivation: Personalized segmentation identifies a particular object in new images rather than merely recognizing its category.The task must distinguish the designated instance from visually similar objects.
  • Motivation: Personalized retrieval ranks images containing the exact referenced instance, making it another form of instance-level matching.Segmentation produces a mask, whereas retrieval produces image-level scores.
  • Motivation: Despite their shared objective, segmentation and retrieval have largely evolved independently with distinct task-specific solutions.PDM is described as the closest prior attempt to bridge the tasks.
  • FoundYou: FoundYou repurposes SAM 2’s video-tracking representations and memory attention to match a specific instance across independent images.Lightweight feature adaptation layers address SAM 2’s temporal continuity bias for unrelated scenes.
  • Results: +18.4 mIoU on PerMIS and +17.8 mAP on ILIAS are reported improvements over PDM.FoundYou also surpasses task-specific methods, supports multiple references and weaker prompts, and achieves category-level retrieval gains.
  • Efficiency: FoundYou keeps SAM 2 frozen while adding only 5.9 M trainable parameters.The resulting compact model supports the unified framework without updating the SAM 2 backbone.

2 Related Work

Prior work has applied SAM 2 and feature matching to downstream segmentation and retrieval, often emphasizing semantic or category-level generalization. FoundYou instead sharpens instance-specific cues to identify the same physical object across independent images.

  • SAM 2 for downstream tasks: SAM 2 downstream methods use external models or direct feature adaptation for few-shot, open-vocabulary, referring, and related segmentation tasks.These approaches derive prompts or adapt SAM 2 features for downstream objectives.
  • SAM 2 for downstream tasks: SANSA and FS-SAM2 use SAM 2 to guide matching in new images but target broader semantic categories.FoundYou differs by reinforcing instance-level discrimination rather than abstracting instance-specific cues.
  • Personalized segmentation: Personalized segmentation methods localize a referenced instance through feature matching before prompting SAM to segment it.Representation quality is central because SAM features may lack correspondence structure, while semantic features can collapse similar instances.
  • Instance retrieval: Instance retrieval has progressed from class-level discrimination toward domain-specific and diverse-object settings with clutter and similar distractors.These conditions expose limitations in global embeddings and motivate local or instance-aware refinement.

3 Method

FoundYou unifies personalized segmentation and retrieval by adapting SAM 2’s memory attention to match a reference instance across independent images. Shared matched features feed either a segmentation mask decoder or an image-level retrieval decoder.

  • Task Definition: FoundYou defines a visual prompt with a reference image and target annotation, then predicts the same physical instance’s mask or ranks gallery images by instance relevance.Prompts may use masks, boxes, or points; retrieval produces a similarity score for each gallery image.
  • Overview: FoundYou repurposes frozen SAM 2 memory attention, replacing temporal frame matching with instance matching across independent images.SAM 2’s memory stores the reference representation and matches target features to produce dense memory-conditioned features.
  • Feature Adaptation: AdaptFormer blocks in the final two Image Encoder layers reduce SAM 2’s continuity bias while preserving instance-aware representations for cross-image correspondence.The backbone remains frozen while only adapter projections are trained.
  • Task-Specific Decoders: The frozen SAM 2 Segmentation Decoder converts matched features into a binary mask, while a lightweight Retrieval Decoder aggregates them into an image-level similarity score.The retrieval decoder uses a learnable token and cross attention to summarize spatial correspondences.
  • Training Objectives: Training treats retrieval as binary instance matching over positive and negative candidates and uses mask-based self-distillation to preserve dense segmentation structure.The distillation objective regularizes adapted predictions toward the frozen SAM 2 mask outputs using pixel-wise KL divergence.

4 Experiments

Experiments evaluate FoundYou across personalized segmentation, retrieval, few-shot retrieval, scaling, category-level retrieval, prompt flexibility, and adaptation choices. Results show strong accuracy, scalability, generalization, prompt robustness, and a favorable efficiency–performance trade-off.

  • Unified Personalized Segmentation and Retrieval: FoundYou achieves the best reported personalized segmentation and retrieval results across the evaluated benchmarks.It reaches 92.1 mAP on PerMIR and 32.5 mAP@1k on ILIAS, while improving segmentation on PerMIS and PerSeg.
  • Unified Personalized Segmentation and Retrieval: +18.4 mIoU over PDM on PerMIS and +17.8 mAP@1k over PDM on ILIAS quantify FoundYou’s gains over the closest unified baseline.On ILIAS, it also surpasses AMES by +6.1 mAP@1k.
  • Unified Personalized Segmentation and Retrieval: FoundYou uses a single 52 M-parameter model running at 90 FPS while keeping SAM 2 frozen and training only 5.9 M additional parameters.The model is reported as over 75× faster than PDM and faster than segmentation methods.
  • Few-shot Retrieval: With four references, FoundYou improves from 30.1 to 34.2 mAP on ILIAS, a +4.1 gain that exceeds AMES and RoMa v2’s gains.The framework stores each reference in memory and jointly aggregates evidence across views.
  • Scaling the Re-Ranking Set: +9.1 mAP as K grows from 10 to 1000 shows FoundYou scaling from 23.4 to 32.5 mAP despite increasing distractors.Its margin over RoMa v2 grows from +0.3 mAP at K=10 to +6.5 at K=1000.
  • Category-Level Retrieval: FoundYou outperforms the retrieval baseline across category-level and landmark benchmarks, including 65.2 on iNat and 97.4 on RP2K.The reported gains are +1.1 over AMES on iNat, +0.9 on RP2K, +0.2 on SOP, +0.2 on RParis, and +0.7 on ROxford.
  • Personalized Segmentation with Flexible Prompts: FoundYou remains robust to weaker segmentation prompts, with PerMIS performance decreasing by 4.0 mIoU from masks to points.On PerSeg, performance is 96.4 mIoU with masks, 95.9 with boxes, and 95.2 with points.
  • Ablation Studies: Lightweight adaptation reaches 62.6 mIoU and 32.5 mAP, outperforming direct SAM 2 use and full or partial fine-tuning in the reported trade-off.Self-distillation preserves segmentation quality while maintaining strong retrieval performance.

5 Conclusion

FoundYou unifies personalized segmentation and retrieval by extending SAM 2 from video tracking to cross-image instance matching. The frozen-backbone 52 M-parameter model outperforms prior methods, scales with additional references, and remains robust to weaker prompts.

  • FoundYou extends SAM 2 from video tracking to cross-image instance matching for personalized segmentation and retrieval.
  • 52 M-parameter FoundYou outperforms prior unified and task-specific methods.The model keeps SAM 2 entirely frozen and uses lightweight adaptation layers with a compact retrieval head.
  • FoundYou scales effectively with additional references and remains robust to weaker prompts.
  • SAM 2 video pretraining provides identity-aware representations that extend beyond tracking to instance-level reasoning across tasks.

Appendix

The appendix organizes supplementary analyses, comparisons, qualitative examples, failure cases, and dataset details for FoundYou.

  • Additional experiments: Additional experiments report ablation studies of FoundYou’s method.
  • Comparison to few-shot segmentation: The appendix compares FoundYou with traditional few-shot segmentation methods, including similar SAM 2-based approaches.
  • Few-shot retrieval details: The appendix provides further details on the proposed few-shot personalized retrieval setting.
  • Qualitative results: Qualitative examples compare FoundYou with task-specific and unified approaches for segmentation and retrieval.
  • Failure cases: Failure cases are reported to provide insights for future work.
  • Dataset details: The appendix includes additional information on all datasets used.

A Additional Experiments

Additional experiments examine distillation, training-example counts, and feature backbones to analyze FoundYou’s design choices and SAM 2 representations.

  • Distillation weight: The distillation weight balances preserving segmentation-relevant structure against enforcing instance-discriminative cues during adaptation.Distillation supports mask prediction, while retrieval-oriented loss promotes instance-specific matching and reduces positional bias from video training.
  • Table 6 reports mIoU on PerMIS and mAP on ILIAS for the distillation, training-example, and backbone ablations.
  • Number of positive and negative training examples: Both positive and negative training examples are necessary for effective personalization learning.Positive examples provide target-instance appearance variability, while negatives supply the signal for separating matching from non-matching instances.
  • Pre-trained feature encoders: The backbone ablation replaces SAM 2 with alternative pretrained encoders while keeping the rest of the pipeline identical.The experiment assesses their instance-awareness using the same lightweight adapters.

B Few-Shot vs. Personalized Segmentation

SANSA and FS-SAM2 use SAM 2 to match objects from reference prompts, but they address category-level few-shot segmentation rather than personalized instance identification.

  • SANSA and FS-SAM2 segment all objects belonging to a reference category, unlike personalized segmentation, which identifies one physical instance.Personalized segmentation must reject other objects from the same category; results are reported on PerSeg and PerMIS.

C Few-Shot Retrieval Details

FoundYou extends personalized retrieval to multiple reference images by jointly matching target features against a shared memory of instance representations. This few-shot formulation aggregates complementary views and improves ranking of positives over hard negatives.

  • Memory-bank formulation: Each target location can attend to tokens from all references simultaneously, aggregating complementary evidence across viewpoints.
  • Memory-bank formulation: The multi-reference formulation reduces to the single-reference case when N = 1.
  • Memory-bank formulation: Few-shot retrieval stores multiple representations of the same physical instance in the Memory Bank for joint matching.Each reference may depict the instance from a different viewpoint or under different imaging conditions.
  • Baseline adaptation: Competing pairwise methods are adapted to few-shot retrieval by averaging their individual similarity scores across references.
  • Qualitative behavior: Multiple reference inputs enhance retrieval by moving positive gallery images to higher ranks and hard negatives to lower ranks.Figure 6 compares FoundYou under 1-shot and 4-shot settings for the same instances.

D Qualitative Results

Qualitative comparisons show FoundYou more consistently preserves instance identity across segmentation and retrieval, including cases with visually similar distractors. It also produces strong category-level retrieval examples across multiple datasets.

  • Personalized Segmentation: FoundYou more consistently localizes the correct instance and produces more accurate segmentation masks than PerSAM-F, PDM, and GF-SAM.Competing methods often miss the instance or produce incomplete masks when similar objects are present.
  • Personalized Retrieval: FoundYou correctly retrieves target instances despite visually similar negatives ranking first for competing retrieval methods.The comparison includes MASt3R, RoMa v2, and AMES.
  • Category-level Retrieval: Category-level retrieval examples span multiple datasets and show top-1 results for each query image.

E Failure Cases

FoundYou still fails in difficult instance-level cases. Segmentation errors involve imprecise localization or boundaries, while retrieval errors arise when visually similar distractors outrank the correct instance.

  • Personalized Segmentation: Segmentation failures include masks that overshoot object boundaries or complete failure to localize the target instance.
  • Personalized Retrieval: Retrieval can rank an incorrect instance highest when distractors closely resemble the reference in shape, color, or texture.
  • Personalized Retrieval: Correct retrievals may remain in the top results but appear second or third when subtle visual differences separate instances.

F Dataset details

The evaluation covers personalized segmentation, personalized retrieval, and category-level retrieval across datasets varying in domain, query modality, scale, and instance-matching requirements.

  • Evaluation scope: The evaluation datasets are organized by domain, query modality, and scale.
  • Personalized Segmentation: PerSeg contains 40 personal objects and 216 annotated images for reference-mask segmentation pairs.
  • Personalized Segmentation: PerMIS is built from 150 BURST video sequences and is harder than PerSeg because scenes often contain multiple same-category instances.
  • Personalized Retrieval: PerMIR contains 216 queries and 432 gallery images, while ILIAS spans 1,232 query instances and 4,715 positives with 100M distractors.
  • Category-level Retrieval: Category-level retrieval evaluation includes Cars196, iNaturalist, RP2K, Stanford Online Products, and Food2K.
Loading 2608.29917v1…