PaperlayerView exact revision on arXiv

Source-linked AI summary

INSID3: Training-Free In-Context Segmentation with DINOv3

Claudia Cuttano, Gabriele Trivigno, Christoph Reich, Daniel Cremers, Carlo Masone, Stefan Roth

arXiv:2603.28480v1cs.CV

TL;DR

ICS must segment arbitrary concepts from annotated examples while avoiding the limited generalization of fine-tuning and the complexity of multi-model pipelines. INSID3 uses only frozen DINOv3 features, combining debiased cross-image correspondence with region clustering and self-similarity aggregation. It achieves strong generalization across one-shot semantic, part, and personalized segmentation, outperforming prior fine-tuned and training-free methods.

  • Problem

    Existing ICS methods trade generalization for supervision or rely on multiple frozen VFMs with architectural complexity and fixed segmentation granularities.

  • Method

    INSID3 performs in-context segmentation using only frozen DINOv3 features, with positional debiasing, target-region clustering, seed selection, and self-similarity-guided aggregation.

  • Results

    +7.5 % mIoU over previous work is reported across one-shot semantic, part, and personalized segmentation, with strong generalization across these settings.

  • Takeaways & Limitations

    The results suggest that segmentation can emerge naturally from dense self-supervised representations without task-specific training or model composition.

  • Takeaways & Limitations

    INSID3 handles one target concept at a time and relies on masks to define the target concept.

Abstract

from arXiv · show

In-context segmentation (ICS) aims to segment arbitrary concepts, e.g., objects, parts, or personalized instances, given one annotated visual examples. Existing work relies on (i) fine-tuning vision foundation models (VFMs), which improves in-domain results but harms generalization, or (ii) combines multiple frozen VFMs, which preserves generalization but yields architectural complexity and fixed segmentation granularities. We revisit ICS from a minimalist perspective and ask: Can a single self-supervised backbone support both semantic matching and segmentation, without any supervision or auxiliary models? We show that scaled-up dense self-supervised features from DINOv3 exhibit strong spatial structure and semantic correspondence. We introduce INSID3, a training-free approach that segments concepts at varying granularities only from frozen DINOv3 features, given an in-context example. INSID3 achieves state-of-the-art results across one-shot semantic, part, and personalized segmentation, outperforming previous work by +7.5 % mIoU, while using 3x fewer parameters and without any mask or category-level supervision. Code is available at https://github.com/visinf/INSID3 .

1. Introduction

ICS seeks to segment arbitrary objects, parts, and personalized instances from annotated examples, but existing approaches trade generalization for supervision or complexity. INSID3 argues that DINOv3 alone can provide the spatial and semantic structure needed for training-free, flexible segmentation.

  • In-context segmentation adapts to arbitrary concepts and domains from one or more annotated examples at inference time.The task covers objects, parts, and personalized instances.
  • Fine-tuned VFM methods improve in-domain results but require supervision and narrow the model scope to the training distribution.
  • Training-free multi-VFM pipelines improve generalization but add computational overhead and inherit fixed segmentation priors.
  • DINOv3’s dense localized features preserve spatial structure and enable robust region-level grouping without decoders, fine-tuning, or model composition.
  • INSID3 selects discriminative target regions through cross-image similarity and expands them using self-similarity to recover coherent concept masks.
  • +7.5 % mIoU average improvement over fine-tuned and SAM-based training-free approaches is reported across diverse segmentation settings.The method also reports up to +6.6 % PCK on semantic correspondence after positional-bias correction.

2. Related Work

Related work develops ICS through generalist segmentation models, multi-model training-free pipelines, or supervised VFM extensions. INSID3 builds on DINOv3’s scaled dense representations to unify correspondence and segmentation without fine-tuning or model composition.

  • SegGPT and Painter train generalist models to handle multiple segmentation scenarios from contextual examples.
  • ICS has been formulated as one-shot semantic, part, and personalized segmentation across different semantic granularities.
  • Training-free pipelines combine DINOv2 correspondence with SAM masks, gaining generalization while inheriting multi-stage cost and fixed mask granularity.
  • Earlier self-supervised methods transferred suboptimally to pixel-level prediction, motivating localized and discriminative dense feature learning.
  • DINOv3 uses scaling, Gram anchoring, and high-resolution post-training to obtain expressive dense features usable for segmentation without fine-tuning or model composition.

3. In-context Segmentation with INSID3

INSID3 uses frozen DINOv3 features to decompose a target image into coherent regions, match those regions to an annotated reference, and expand a seed into a complete mask. Its correspondence analysis identifies positional bias and removes it before cross-image matching.

  • INSID3 removes positional components from frozen DINOv3 features for cross-image matching while retaining original features for clustering and intra-image similarity.
  • The target image is partitioned into semantically coherent regions using feature self-similarity.
  • A target cluster is selected by comparing each region with the annotated reference through cross-image similarity in the debiased feature space.
  • The selected seed region is expanded by aggregating clusters according to intra-image self-similarity, producing a complete coherent mask.
  • Task definition: Reference and target images are encoded as D-dimensional patch embeddings at patch resolution P = H′ × W′.
  • Unlocking the DINOv3 feature space: Cross-image similarity maps measure how strongly target patches align with the reference concept, while also revealing systematic same-position false activations.
  • Unlocking the DINOv3 feature space: A noise image estimates a low-dimensional positional-bias subspace that can be removed consistently during inference.

3.2. Fine-grained clustering

INSID3 partitions target-image DINOv3 features into spatially coherent regions, then selects and aggregates clusters that match the annotated reference at the appropriate granularity.

  • Fine-grained clustering: DINOv3 features group patches from the same object or part into semantically coherent regions without supervision.These regions provide the structural representation used for subsequent matching.
  • Fine-grained clustering: Agglomerative clustering avoids predefining the cluster count, supporting open-world concepts with variable segmentation granularities.The method is chosen because K-means requires a fixed number of clusters and density-based methods struggle in high-dimensional feature spaces.
  • Candidate localization: Backward correspondences retain target patches whose nearest reference neighbors lie inside the support mask, filtering candidate regions conservatively.Restricting precomputed clusters to these patches yields candidate clusters for seed selection.
  • Seed-cluster selection: Cross-image similarity selects the candidate cluster whose prototype is most semantically aligned with the annotated reference.Prototype-based comparison is performed in the debiased feature space for candidate clusters and the reference region.
  • Cluster aggregation: INSID3 combines semantic alignment with within-image structural coherence, merging candidate clusters whose multiplicative score exceeds threshold α.This expands the typically partial seed localization while accounting for self-similarity within the target image.

4. Experiments

INSID3 is evaluated across one-shot semantic, part, personalized segmentation, and semantic correspondence, with ablations examining debiasing, clustering, and aggregation. It consistently improves over competing approaches while using a unified DINOv3 feature space.

  • Evaluation setup: INSID3 is compared on one-shot semantic, part, and personalized segmentation using a single annotated reference mask.Semantic evaluation spans six datasets, while part segmentation uses PASCAL-Part and PACO-Part; personalized segmentation is evaluated on PerMIS.
  • Baselines: INSID3 is the only Table 1 method using no supervision and operating solely on the self-supervised DINOv3 backbone.Baselines include training-free SAM-based pipelines and fine-tuned methods using semantic or mask supervision.
  • One-shot semantic segmentation: +6.6 % pts. mIoU on LVIS-92i and +27.8 % pts. on Chest X-ray are among INSID3’s gains over GF-SAM in one-shot semantic segmentation.INSID3 performs correspondence estimation and segmentation in a unified space with 304 M versus 945 M parameters.
  • One-shot part segmentation: +6.0 % pts. mIoU on PASCAL-Part and +2.4 % pts. on PACO-Part are INSID3’s gains over GF-SAM for one-shot part segmentation.INSID3 also surpasses SegIC and DiffewS, while preserving object structure and segmentation granularity.
  • One-shot personalized segmentation: 67.0 % mIoU on PerMIS is INSID3’s personalized-segmentation result, exceeding GF-SAM by +12.9 % pts.Backward correspondences use negative evidence to suppress irrelevant regions and select among visually similar distractors.
  • Semantic correspondence: +0.9–6.6 PCK across DINOv3 model sizes shows that positional debiasing improves semantic correspondence on SPair-71k.Qualitative comparisons report that debiased features reduce systematic errors caused by mixed semantic and positional cues.
  • Ablation study: Rank s = 500 yields consistent gains of +3.1 % on COCO, +2.7 % on PASCAL, and up to +6.6 % PCK on SPair-71k.Removing too many positional-subspace dimensions eventually reverses the improvement.
  • Ablation study: Jointly leveraging cross- and self-similarity reaches 57.6 % and 50.5 % mIoU on COCO and PASCAL-Part, respectively.The aggregation strategy resolves the trade-off between object-level coherence and part-level granularity under a fixed intermediate clustering threshold.

5. Conclusion

The paper introduces INSID3, a training-free in-context segmentation framework built solely on DINOv3. Its results across semantic, part, and personalized segmentation suggest that segmentation can emerge from self-supervised dense representations without training or fine-tuning.

  • Conclusion: INSID3 uses DINOv3’s semantic alignment and spatial coherence to perform correspondence estimation and segmentation within one backbone.The framework remains fully unsupervised and relies on the in-context example for guidance.
  • Conclusion: INSID3 generalizes across one-shot semantic, part, and personalized segmentation while outperforming fine-tuned and training-free approaches.The conclusion presents this as evidence that segmentation can emerge naturally from self-supervised dense representations.
  • Conclusion: INSID3 suggests that reducing supervision may foster more robust and transferable representations for scalable visual understanding.This conclusion is framed as a concrete step toward more general-purpose visual understanding.

Supplementary Material

The appendix extends INSID3 with analyses of positional bias, fixed implementation settings, and additional experiments across settings, backbones, and evaluation cases.

  • Additional analyses: The appendix analyzes positional bias in DINOv3, including comparisons with DINOv2 and additional debiasing studies.
  • Implementation details: The paper reports hyperparameters used throughout, with the same set fixed across all datasets and tasks.
  • Additional experiments: Additional experiments cover the 5-shot setting, backbone comparisons, evaluation against SAM 3, and the empty-mask corner case.

A.1. DINOv2 vs. DINOv3: Positional bias

DINOv3 dense features exhibit stronger positional bias than DINOv2, producing coordinate-aligned false matches that debiasing suppresses while preserving semantic correspondence. The projection improves downstream segmentation with negligible inference overhead and retains a cleaner semantic subspace across dimensionalities.

  • Positional bias: DINOv3 exhibits stronger coordinate-aligned similarity artifacts than DINOv2, especially in background regions with little semantic content.Patches at the same absolute position as the reference keypoint produce spurious high responses.
  • Debiasing effects: Debiasing suppresses position-driven responses while keeping similarity patterns on the object stable.The correction targets positional activations without visibly disrupting object-related similarity.
  • Alternative strategies: 57.6 % mIoU is achieved by projection-based debiasing, compared with 55.7 % and 56.5 % mIoU for averaging 4 and 12 augmented views.The projection is computed offline and adds virtually no inference overhead beyond one matrix multiplication.
  • Semantic structure: PCA-debiased features consistently outperform PCA-original features on SPair-71k across retained dimensionalities.The comparison evaluates cross-image semantic correspondence using PCK@0.10.
  • Discussion: Positional information remains a trade-off: it can help tasks such as sky segmentation but hamper correspondence across spatial positions.The proposed decomposition removes positional directions without training and is intended for correspondence-sensitive matching.

B. Implementation details

INSID3 uses three scalar hyperparameters selected once on COCO-20i training data and fixed across datasets, domains, and segmentation tasks. No dataset-specific or task-specific tuning is performed.

  • Hyperparameters: INSID3 relies on only three scalar hyperparameters fixed across datasets, domains, and semantic, part, and personalized segmentation.The hyperparameters are selected using 3-fold cross-validation on the COCO-20i training split.
  • Additional results: The supplementary section provides additional experimental results complementing the main paper’s Section 4.

C.1. k-shot segmentation

INSID3 extends naturally from one-shot to multiple reference examples by modifying only correspondence, while retaining target clustering and aggregation. In the 5-shot setting, it benefits consistently from additional references and achieves gains across benchmarks, though performance varies by dataset.

  • C.1. k-shot segmentation: In k-shot segmentation, target clustering and aggregation remain unchanged; only the correspondence stage is adapted.The method computes nearest neighbors across references, filters candidates by majority support, and averages debiased reference prototypes.
  • Correspondence: Majority-vote filtering retains target correspondences supported by most reference images before prototype aggregation.One debiased prototype is computed per reference and then averaged for cross-image similarity.
  • Domain results: +8.7 % and +27.2 % pts. gains are reported on ISIC and X-Ray, while COCO shows a −1.7 % pts. gap.The reported boost is +3.6 % pts. on SUIM and +4.5 % pts. mIoU on iSAID; GF-SAM improves by +1.0 % pts. with DINOv3.
  • Backbone analysis: DINOv3 features provide the best segmentation accuracy among the evaluated backbones, supporting single-backbone ICS without decoders, fine-tuning, or external models.
  • Empty-mask corner case: INSID3 correctly predicts empty masks in 85% of 4 000 absent-concept COCO pairs.SAM-based methods always return non-empty masks by design in this setting.

D. Computational Cost of INSID3

INSID3 is substantially faster than existing training-free approaches because semantic alignment and mask generation occur in one DINOv3 feature space. On an RTX 4090, it runs in 302 ms, with clustering as the dominant cost.

  • Design: INSID3 performs semantic alignment and mask generation directly in the DINOv3 feature space, avoiding the two-model coordination used by SAM-based pipelines.Its single-backbone design accounts for the faster inference runtime.
  • Runtime comparison: 302 ms is the reported INSID3 runtime on one RTX 4090, versus 1 030 ms for GF-SAM.Runtimes are measured in 32-bit precision and averaged over 100 examples.
  • Runtime breakdown: 166 ms of INSID3 runtime is spent on agglomerative clustering, while the DINOv3 forward pass and debiasing take 78 ms.Similarity computation and cluster scoring together take 3 ms.
  • Qualitative behavior: INSID3 produces coherent masks across objects and fine-grained parts in aerial, marine, medical, and everyday scenes.

F. Limitations and Future Work

INSID3’s current scope is limited to one target concept at a time and mask-based prompts, while instance-level reasoning and lighter annotations remain future directions. Its segmentation quality also depends on the representational properties of the frozen DINOv3 backbone.

  • INSID3 handles one target concept at a time, requiring separate reference prompts when multiple concepts appear in the target image.
  • Mask-based prompting limits INSID3 relative to methods that accept cheaper point or bounding-box annotations.The authors identify lighter prompt modalities as a way to broaden practical and interactive use.
  • Instance-level reasoning remains a future direction because the current focus is semantic segmentation rather than recovering multiple instances jointly.The authors suggest exploiting affinities among clusters belonging to the same instance.
  • Final segmentation quality depends on the semantic structure encoded in DINOv3’s frozen self-supervised features, despite improved cross-image matching from debiasing.
Loading 2603.28480v1…