PaperlayerView exact revision on arXiv

Source-linked AI summary

REBASE: Reference-Background Subspace Elimination for Training-Free In-Context Segmentation

Mantha Sai Gopal, Jaison Saji Chacko, Harsh Nandwana, Sandesh Hegde, Debarshi Banerjee, Uma Mahesh

arXiv:2607.09082v1cs.CV

TL;DR

Training-free in-context segmentation avoids the retraining and storage costs of adding classes, but cross-image background similarities can mislocalize prompts. REBASE removes a reference-conditioned background subspace from reference and query features, and achieves the best training-free performance on four of five benchmarks while ranking second on PASCAL-Part.

  • Problem

    Class-incremental learning requires repeated optimization, stored replay data or parameter snapshots, and delays serving each newly introduced class until fine-tuning finishes.

  • Method

    REBASE projects reference and query features onto the orthogonal complement of a reference-derived background subspace, then uses similarity-weighted farthest-point prompts and a dense prior for SAM.

  • Results

    Among training-free methods, REBASE performs best on four of five benchmarks and second on PASCAL-Part; it reaches 63.8% mIoU on ISIC, outperforming INSID3 by +9.4 pp.

  • Takeaways & Limitations

    Reference-conditioned subspace debiasing consistently improves segmentation across natural-image, part-level, and medical benchmarks without training or test-time optimization.

  • Takeaways & Limitations

    The pipeline uses sparse point prompts and dense priors rather than natural-language prompting and exemplar-based concepts.

Abstract

from arXiv · show

Training-free in-context segmentation enables new object categories to be introduced at inference time from a single annotated reference image, eliminating the retraining and memory overhead of class-incremental learning. Recent approaches achieve this by combining vision foundation models for semantic correspondence with promptable segmentation networks like SAM. However, their performance is fundamentally limited by the quality of the cross-image similarity map; shared contextual backgrounds between the reference and query systematically elevate similarity in non-target regions, degrading prompt localization. We present REBASE, a training-free framework that explicitly suppresses these spurious contextual correspondences. Our method identifies the low-rank background feature subspace from the reference image and project the reference and query features onto its orthogonal complement in closed form, yielding cleaner semantic matching. We then generate positive point prompts using similarity-weighted farthest-point sampling, paired with a refined dense similarity prior. Without any training or parameter updates, our approach establishes a new state of the art among training-free methods on PACO-Part, FSS-1000, and cross-domain datasets such as ISIC2018, demonstrating that explicit background subspace removal is a highly effective principle for one-shot localization.

1. Introduction

Training-free one-shot segmentation addresses the continual arrival of new object categories without repeated fine-tuning, replay storage, or per-class hyperparameter tuning. REBASE targets spurious contextual correspondences by removing a shared reference-background subspace before generating prompts for SAM-based segmentation.

  • Motivation: New object categories continually emerge after deployment, making fixed closed-set vision systems inadequate for real-world use.
  • Motivation: Class-incremental learning repeatedly fine-tunes models for new classes, incurring optimization, storage, and hyperparameter costs while delaying service for the new class.
  • Training-free segmentation: Few-shot segmentation specifies a target class at inference using annotated references; in the one-shot regime, one masked reference image must suffice without training or parameter updates.
  • Existing paradigm: Vision foundation models decompose the task into semantic correspondence and geometric localization, combining dense self-supervised features with promptable segmentation models such as SAM and SAM 2.
  • REBASE: REBASE applies the symmetric projection eF = F(I −BB⊤) to remove shared scene-context directions from reference and query DINOv2 features, then uses similarity-weighted farthest-point sampling for prompting.

2. Related Work

Related work spans promptable segmentation foundations, trained few-shot semantic segmentation, and training-free pipelines based on frozen vision foundation models. Recent studies also mitigate representation biases before feature matching to improve in-context segmentation.

  • Promptable segmentation foundations: SAM introduced a general-purpose, promptable segmentation foundation model trained on over one billion masks with class-agnostic supervision.SAM 2 and SAM 3 extend this paradigm with improved architecture and stronger image and video segmentation capabilities.
  • Few-shot semantic segmentation: Trained few-shot methods use support–query architectures, prototype matching, attention, cycle-consistent transformers, and hypercorrelation networks.These approaches perform strongly within training domains but require dataset-specific optimization and retraining for unseen categories.
  • Training-free segmentation: PerSAM established a training-free pipeline that uses DINOv2 correspondences to generate positive and negative point prompts for SAM-based target segmentation.Matcher builds on this foundation with dense bidirectional correspondences and controllable prompting.
  • Bias mitigation in feature matching: INSID3 mitigates disruptive positional priors in frozen vision-transformer features by projecting them onto the orthogonal complement of a globally estimated frozen positional basis.The method reports marked improvements in downstream in-context segmentation.

3. Method

REBASE performs training-free one-shot in-context segmentation by removing a reference-derived background feature subspace from frozen DINOv2 features before similarity matching. It combines the cleaned similarity map with spatially diverse point prompts and a dense SAM mask prior.

  • Feature extraction and similarity: REBASE encodes reference and query images with a frozen DINOv2 encoder, then computes cross-image similarity from their patch features.The reference mask is area-average resized to the encoder’s patch grid to preserve sub-patch foreground coverage.
  • Prompt generation: SW-FPS selects K positive prompts by balancing normalized similarity and spatial distance among candidates above a confidence threshold.The weighting parameter α ∈ [0, 1] interpolates between top-K selection at α = 0 and pure farthest-point sampling at α = 1.
  • Dense similarity prior: REBASE injects the standardized similarity map into SAM’s auxiliary mask-input branch as a dense spatial prior alongside point prompts.Positive-support locations provide a coarse foreground estimate, while other locations receive a constant negative logit before bilinear resampling.
  • Background subspace elimination: Reference-background features define a low-rank basis, and a closed-form projector removes its dominant directions from both reference and query features.With B as the leading background basis, REBASE applies PB = I_C − BB^⊤ to every patch feature, retaining components orthogonal to the background subspace.
  • Background subspace elimination: The per-episode projection attenuates query patches sharing reference-background context while largely preserving target-related patches.The basis is estimated from the current reference background, making its directions scene-specific; only the rank s is exposed as a hyperparameter.

4. Experiments

REBASE is evaluated across natural-image, fine-grained part, and medical segmentation benchmarks against fine-tuning and training-free methods. It achieves leading training-free performance, with substantial medical gains and component ablations validating its prompt-sampling, dense-prior, and background-subspace projection design.

  • Benchmark Setup: The evaluation spans five benchmarks covering natural-image semantic segmentation, fine-grained part segmentation, and medical-domain segmentation.FSS-1000 uses a standard 240-class test split; PASCAL-Part covers 56 parts across 15 categories.
  • Main Results: REBASE achieves the best training-free mIoU on four benchmarks and ranks second on PASCAL-Part.The comparison includes fine-tuning methods such as Painter, SegGPT, SINE, DiffewS, and SegIC, alongside training-free methods including PerSAM, Matcher, GF-SAM, and INSID3.
  • Main Results: 63.8% mIoU on ISIC exceeds INSID3 by +9.4 pp, while 86.3% on Chest X-Ray improves over INSID3 by +7.5 pp.The method remains 1.2 pp below fully fine-tuned SegGPT on Chest X-Ray.
  • Main Results: 88.2% mIoU on FSS-1000 surpasses GF-SAM (88.0%) by +0.2 pp and INSID3 (83.7%) by +4.5 pp without additional training.It also outperforms the evaluated fine-tuning baselines SegGPT and SegIC on this benchmark.
  • Main Results: On PACO-Part, REBASE reaches 39.3% mIoU, while its 46.6% on PASCAL-Part ranks second behind INSID3 (50.5%).On PACO-Part, it improves over INSID3 (38.7%) by +0.6 pp and GF-SAM (36.3%) by +3.0 pp.
  • Ablation and Sensitivity: +4.41 pp on PACO-Part and +3.93 pp on ISIC result from adding background-subspace projection after similarity-weighted farthest-point sampling and the dense prior.The complete pipeline reaches 39.28% and 63.77% mIoU, respectively; increasing r from 0.005 to 0.9 drops performance by 13.7 pp on FSS-1000 and 8.4 pp on PASCAL-Part.

5. Conclusion

REBASE is a training-free in-context segmentation pipeline that improves cross-image matching by removing episode-specific background bias from feature representations. Its reference-conditioned subspace debiasing complements foundation-model pipelines and offers a direction for more robust training-free in-context visual understanding.

  • Conclusion: REBASE improves cross-image matching by removing episode-specific background bias from feature representations.It does so in a training-free pipeline for in-context segmentation.
  • Conclusion: Projecting reference and query features onto an empirical background subspace’s orthogonal complement enhances feature matching.The method uses reference-conditioned subspace debiasing as its central mechanism.
  • Conclusion: REBASE provides a promising direction for developing more robust training-free methods for in-context visual understanding.The perspective is presented as complementary to existing foundation-model pipelines.

Supplementary Material · Overview

The supplementary material provides additional analyses validating REBASE, including tests across visual encoders and more challenging semantic segmentation benchmarks.

  • Overview: The supplementary material adds analyses intended to further validate the proposed REBASE framework.
  • Overview: REBASE is evaluated with the DINOv2-L visual encoder.
  • Overview: REBASE is also evaluated with the successor encoder DINOv3-L.
  • Overview: The encoder comparison demonstrates that background-subspace elimination is not tied to a particular visual encoder.
  • Overview: The supplementary experiments extend evaluation to the COCO-20i benchmark.
  • Overview: They also extend evaluation to LVIS-92i, showing that REBASE’s gains persist on more challenging semantic segmentation benchmarks.

S1. Additional Experiments

Additional experiments show that REBASE transfers across DINOv2-L and DINOv3-L backbones, additional semantic-segmentation benchmarks, and newer SAM architectures. Ablations further indicate that REBASE is not a direct replacement for positional debiasing on natural images, while the two projections can be complementary on ISIC.

  • DINOv3-L backbone: The relative strengths of DINOv2-L and DINOv3-L are dataset-dependent, with neither backbone consistently outperforming the other.DINOv3-L performs better on COCO-20i, LVIS-92i, X-Ray, and FSS-1000, whereas DINOv2-L performs slightly better on ISIC, PASCAL-Part, and PACO-Part.
  • DINOv3-L backbone: REBASE improves the DINOv3-L baseline on six of seven benchmarks, with largest gains on X-Ray, LVIS-92i, and COCO-20i.The reported gains are +19.70 pp on X-Ray, +8.20 pp on LVIS-92i, and +7.10 pp on COCO-20i; FSS-1000 changes by −0.27 pp from an 89.52% mIoU baseline.
  • Additional semantic-segmentation benchmarks: REBASE improves COCO-20i and LVIS-92i baselines under both DINOv2-L and DINOv3-L backbones.Gains are +6.93 pp and +7.10 pp on COCO-20i, and increase from +4.82 pp to +8.20 pp on LVIS-92i when using DINOv3-L.
  • Newer SAM architectures: With SAM 2, REBASE improves the baseline across COCO-20i, PASCAL-Part, and ISIC2018.The gains are +6.25 pp on COCO-20i, +5.82 pp on PASCAL-Part, and +3.22 pp on ISIC2018; REBASE also continues to improve the baseline with SAM 3, albeit by smaller amounts.
  • Positional-debiasing ablation: Replacing positional debiasing with REBASE substantially degrades natural-image performance but has modest effects on the medical datasets.Relative to INSID3, substitution reduces mIoU by 13.19 pp on PASCAL-Part and 5.94 pp on COCO-20i, while improving Lung X-Ray by +0.59 pp and ISIC by +0.42 pp.
  • Positional-debiasing ablation: Stacking REBASE after positional debiasing adds little on PASCAL-Part, COCO-20i, and Lung X-Ray but benefits ISIC.The first three datasets differ by at most ±0.5 pp from substitution, whereas ISIC consistently outperforms both the baseline and substitution variant, suggesting complementary projections.

S2. Implementation Details

All experiments use fixed implementation settings across benchmarks. The dense prior is thresholded and morphologically refined before adaptive reference background-subspace estimation.

  • Implementation settings are fixed across all benchmarks, as summarized in Table S6.
  • The dense prior is thresholded by the image mean, eroded with a 3 × 3 elliptical kernel, and used to assign background logits.The erosion suppresses boundary leakage before background-logit assignment.
  • The reference background subspace is estimated from remaining background patches with adaptive basis rank s = ⌈r · n_BG⌉.The rank is determined by the number of available background patches.

S3. Qualitative Results

Qualitative examples show that REBASE suppresses background and distractor similarity responses, concentrating correspondence on the target. Across four benchmarks and varied visual conditions, its predicted masks remain well aligned with target regions.

  • Background-subspace elimination: REBASE suppresses similarity responses extending into background regions or visually related objects, concentrating correspondence on the target.Comparisons use original DINOv2 features versus features after background-subspace elimination (+REBASE).
  • Cross-benchmark qualitative results: Across PASCAL-Part, PACO-Part, ISIC dermoscopy, and chest X-ray, predictions remain well aligned with target parts and preserve clear boundaries.Examples include fine-grained parts, small or geometrically ambiguous regions, and low-contrast medical structures despite differences in appearance, pose, and imaging characteristics.
  • Background-subspace elimination: In COCO-20i examples, refined REBASE similarity maps produce corresponding segmentations with reduced distractor responses.Figure S3 compares original and REBASE cosine similarity maps alongside their segmentations.
Loading 2607.09082v1…