PaperlayerView exact revision on arXiv

Source-linked AI summary

FoRIS: Progressive Foreground Refinement for Training-Free In-Context Segmentation

Ming Hu, Jianfu Yin, Mingyu Dou, Miaomiao Zhang, Yao Wang, Cong Hu, Bingliang Hu, Quan Wang

arXiv:2609.03384v1cs.CV

TL;DR

Training-free in-context segmentation can confuse visually similar background regions with target objects, while target responses may be contaminated, ambiguous, or fragmented. FoRIS progressively purifies, localizes, and consolidates foreground evidence rather than treating correspondence as the final prediction. It reports competitive results across semantic, part, and cross-domain benchmarks, with average IoU improvements in both 1-shot and 5-shot settings.

  • Problem

    Reference-query correspondence can confuse visually similar background regions with targets, while foreground features and query responses may be contaminated, ambiguous, or fragmented.

    Source: §1 · p. 2

  • Method

    FoRIS uses a frozen visual foundation model to progressively purify foreground features, localize discriminative target regions, and consolidate scattered responses without additional segmentation supervision or auxiliary segmentation models.

    Source: §1 · pp. 2–3

  • Results

    FoRIS is competitive across semantic segmentation, part segmentation, and cross-domain benchmarks, with average IoU improvements of over 4.5 and 4.8 percentage points in the 1-shot and 5-shot settings, respectively.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The reported part-segmentation results indicate that progressive foreground refinement benefits fine-grained structural understanding beyond object-level segmentation.

    Source: §4.1 · p. 7

    Existing SAM- and DINO-based ICS methods achieve consistently low IoU on Fundus and DeepGlobe-18, indicating limited reliability on slender structures.

    Source: §1 · p. 3, §5 · p. 9

Abstract

from arXiv · show

In-Context Segmentation (ICS) aims to precisely segment arbitrary semantic concepts, such as objects or parts, given one or a few annotated visual exemplars. In this paper, we revisit ICS from a more classical segmentation perspective, viewing it as a coarse-to-fine progressive refinement process. Rather than directly predicting the final mask through reference-query matching, we progressively refine the segmentation from coarse and ambiguous foreground responses to precise and complete foreground structures. Building upon this perspective, we propose a training-free in-context segmentation framework, termed FoRIS. Specifically, FoRIS consists of three key stages: Foreground Purification, Foreground Localization, and Foreground Consolidation, which progressively suppress background distractions, localize discriminative target regions, and recover complete foreground structures through semantic aggregation. Experimental results demonstrate that FoRIS achieves SOTA performance across semantic and part segmentation tasks, with average improvements of 4.5 and 4.8 mIoU points over existing approaches in the 1-shot and 5-shot settings, respectively. Code: https://github.com/Xi-Mu-Yu/FoRIS.

1 INTRODUCTION

The introduction argues that reference-query correspondence alone can confuse visual similarity with target membership and proposes FoRIS as progressive refinement of foreground evidence. It reports average IoU gains over existing methods in both shot settings, while identifying slender and fragmented structures as a generalization challenge.

1 INTRODUCTION

  • Existing training-free ICS methods may match visually similar background regions to the target, exposing a mismatch between visual similarity and target membership.The introduction illustrates the issue with a query-image wall region resembling a reference refrigerator.Source: §1 · p. 2
  • Foreground contamination, ambiguous target localization, and fragmented responses jointly make reliable foreground representations difficult to derive from frozen features.An annotated mask marks target location but does not guarantee semantically pure corresponding features.Source: §1 · p. 2
  • FoRIS treats correspondence as intermediate semantic evidence in a three-stage refinement process: purification, localization, and consolidation.The stages suppress background interference, identify discriminative query regions, and aggregate scattered responses into complete structures.Source: §1 · pp. 2–3
  • FoRIS implements the refinement process with a frozen visual foundation model, without additional segmentation supervision or auxiliary segmentation models.The framework shifts from one-step semantic transfer toward extracting, localizing, and consolidating foreground evidence.Source: §1 · p. 3
  • Current VFM-based training-free ICS methods remain limited in generalizing to slender and fragmented structures, an open problem for future research.Such structures can have weak semantic responses, limited spatial support, and substantial appearance variation.Source: §1 · p. 3
  • Over 4.5 and 4.8 percentage points of average IoU improvement over existing methods are reported for the 1-shot and 5-shot settings, respectively.The paper presents FoRIS as a training-free ICS framework.Source: §1 · p. 3

2 RELATED WORK

Related work describes ICS as segmentation of arbitrary concepts specified by visual exemplars, while existing approaches commonly rely on reference-target correspondence. FoRIS instead treats correspondence as evidence to refine progressively into coherent object masks.

2 RELATED WORK

  • ICS targets arbitrary concepts specified at inference time across semantic granularities, unlike few-shot segmentation focused on predefined novel classes.Training-free and supervised approaches differ in whether they use task-specific training, but both commonly rely on correspondence to transfer foreground semantics.Source: §2 · p. 3
  • FoRIS reinterprets correspondence as intermediate semantic evidence that is progressively purified, localized, and consolidated into coherent object masks.The approach does not discard correspondence but makes foreground refinement the central inference process.Source: §2 · pp. 3–4
  • Classical segmentation methods commonly refine uncertain local evidence progressively into coherent foreground structures, using approaches such as region growing, graph cuts, active contours, and multiscale processing.These methods offer a coarse-to-fine perspective for foreground-background separation.Source: §2 · p. 4
  • Correspondence-centric training-free ICS can remain vulnerable to background interference, ambiguous semantic matches, and fragmented target responses, leading to incomplete predictions or semantic drift.Existing clustering, propagation, or refinement operations are still primarily built on correspondence-derived evidence.Source: §2 · p. 4

3 IN-CONTEXT SEGMENTATION WITH FORIS

FoRIS refines a foreground response through purification, localization, and consolidation, with successive stages producing Score 1, Score 2, and the final response (p21, p00).

3 IN-CONTEXT SEGMENTATION WITH FORIS

  • The framework extracts and channel-normalizes patch features from the encoder’s last intermediate layer, using reference image–mask pairs and a target image without fine-tuning or auxiliary heads.The overview specifies a frozen DINOv3 encoder and three successive response scores.Source: §3 · p. 4, Figure 3 · p. 5

3.1 FOREGROUND PURIFICATION

  • FP conditionally debiases features, then uses clustered foreground prototypes and confusable background patches for contrastive gating before semantic matching forms Score 1.Debiasing is applied when semantic alignment falls below θ; the hard-negative background prototype retains the top 20% of background tokens by similarity to the foreground prototype.Source: §3.1 · pp. 4–5

3.2 FOREGROUND LOCALIZATION

  • FL uses dense cross-image matching to identify majority-supported foreground candidates, lightly densifies them, and combines them with a seed prior from feature, color, and spatial clustering.When clusters overlap the candidate set, the seed cluster maximizes reference affinity weighted by candidate area.Source: §3.2 · p. 6

3.3 FOREGROUND CONSOLIDATION

  • FC transforms Score 2 into Score 3 by penalizing semantic disagreement and foreground–background coupling, then reweighting re-clustered target patches by foreground–background separation.The disagreement penalty is modulated by uncertainty and the cluster reweighting is mapped back to patches.Source: §3.3 · p. 6

4 EXPERIMENTS

Using the same datasets, baselines, and protocols as INSID3, the experiments report FoRIS’s semantic and part benchmark gains, then ablate its progressive modules and components.

4 EXPERIMENTS

  • The evaluation follows INSID3’s datasets, baselines, and protocols for one-shot semantic and part segmentation, reporting mIoU.Each setting uses one annotated reference mask to segment the corresponding concept in a target image.Source: §4 · p. 7

4.1 MAIN RESULTS

  • FoRIS achieves the best performance across semantic benchmarks and sets a new state of the art on part-level benchmarks, surpassing training-free and fine-tuned comparisons.The semantic results compare FoRIS with GF-SAM and INSID3; the part results also compare it with SegIC and DiffewS.Source: §4.1 · p. 7

4.2.1 FOREGROUND PURIFICATION

  • COCO mIoU rises from 59.1% to 60.9% and saturates at θ ≥0.75, while Chest X-ray falls from 87.6%–87.7% at θ ≤0.80 to 84.9% at θ=1.0.The contrasting threshold responses show that excessive debiasing can remove useful anatomical priors.Source: Figure 5 · p. 8, §4.2.1 · p. 8

4.2 ABLATION STUDY

  • Adding foreground purification, localization, and consolidation progressively improves performance across all benchmarks, supporting the complementarity of the three-stage design.The ablation begins from a baseline and adds the modules successively.Source: §4.2 · p. 8

4.2.2 FOREGROUND LOCALIZATION

  • Densification improves mIoU on COCO (60.1%→60.9%), Chest X-ray (87.4%→87.6%), and PACO-Part (41.8%→42.3%).Positional cues benefit Chest X-ray (+1.1 points), while RGB cues help SUIM (+0.2 points); combining both maintains strong performance across domains.Source: §4.2.2 · p. 8, Table 3 · p. 9

4.2.3 FOREGROUND CONSOLIDATION

  • SDP and SR together achieve the best performance on all benchmarks, with SDP suppressing unreliable activations and SR correcting region-level semantic bias.SDP gives its largest reported gain on PACO-Part (+1.9 points); SR improves ISIC and COCO more strongly.Source: Table 3 · p. 9, §4.2.3 · p. 9

5 CHALLENGES AND THE FUTURE OF VFM-BASED TRAINING-FREE ICS

Existing SAM- and DINO-based in-context segmentation methods have low IoU on the Fundus and DeepGlobe-18 slender-structure scenarios.

5 CHALLENGES AND THE FUTURE OF VFM-BASED TRAINING-FREE ICS

  • SAM- and DINO-based in-context segmentation methods achieve consistently low IoU on Fundus and DeepGlobe-18 slender-structure scenarios.The paper presents these results as revealing limitations of existing methods in slender-structure segmentation.Source: §5 · p. 9
  • The reported challenge concerns slender structures in Fundus and DeepGlobe-18, rather than a general segmentation setting.The passage frames this as an extension to slender-structure scenarios.Source: §5 · p. 9
  • The findings motivate questions about why slender structures challenge these methods and how performance might be improved.Further analysis and experiments are provided in Appendix F.Source: §5 · p. 9

6 CONCLUSION

FoRIS recasts in-context segmentation as progressive foreground refinement and reports competitive results across segmentation benchmarks. The paper also identifies slender and fragmented structures as an open challenge for training-free methods.

6 CONCLUSION

  • FoRIS progressively suppresses background interference, localizes target regions, and recovers coherent object structures through three refinement stages.The stages are foreground purification, foreground localization, and foreground consolidation.Source: §6 · p. 9
  • FoRIS achieves competitive performance across semantic segmentation, part segmentation, and cross-domain benchmarks, while exhibiting strong out-of-distribution generalization.Source: §6 · p. 9
  • Weak semantic responses, limited spatial support, and appearance variation make slender and fragmented structures an open problem for training-free in-context segmentation.The paper identifies preserving structural continuity for weakly salient, fine-grained, topology-sensitive targets as an unresolved challenge.Source: §6 · p. 9

A EXPERIMENTS SETTING

The appendix specifies one-shot semantic and part evaluation settings and summarizes FoRIS inference, from reference and target inputs to a refined binary mask.

A EXPERIMENTS SETTING

  • One-shot evaluation uses one annotated reference mask to segment the corresponding concept in a target image, for semantic instances or object parts.The semantic benchmarks span six datasets; part evaluation uses PASCAL-Part and PACO-Part.Source: A EXPERIMENTS SETTING · p. 14
  • The implementation uses a Large DINOv3 encoder, 1024 × 1024 inputs, patch-resolution masks, bilinear upsampling, and CRF refinement.Source: A EXPERIMENTS SETTING · p. 14

B ALGORITHM SUMMARY

  • The single-episode inference algorithm takes reference images and masks, a target image, and a frozen encoder, then produces a binary mask.Its listed stages include adaptive positional debiasing, foreground gating and matching, candidate voting and densification, semantic consolidation, and mask refinement.Source: B ALGORITHM SUMMARY · p. 14
  • Table 5 compares semantic correspondence on SPair-71k using PCK@T across DINOv3 backbones and original, debiased, and adaptive-debias variants.Source: Table 5 · p. 15

C SEMANTIC CORRESPONDENCE ANALYSIS OF POSITIONAL DEBIASING

Positional debiasing improves semantic correspondence, but suppressing positional information indiscriminately can harm downstream segmentation. Adaptive debiasing preserves useful spatial cues while selectively reducing harmful positional bias.

C SEMANTIC CORRESPONDENCE ANALYSIS OF POSITIONAL DEBIASING

  • PCK@0.10 rises from 46.80 to 52.60 and PCK@0.20 from 61.22 to 68.73 with DINOv3-Base positional debiasing.Improvements also appear across Small, Base, and Large backbones and evaluation thresholds.Source: C SEMANTIC CORRESPONDENCE ANALYSIS OF… · p. 15
  • Adaptive debiasing performs best across evaluated SPair-71k settings and avoids some degradation from indiscriminate suppression, supporting selective removal of harmful bias while retaining useful geometric priors.Spatial regularities can aid correspondence when reference and target images are already well aligned.Source: C SEMANTIC CORRESPONDENCE ANALYSIS OF… · p. 15
  • Unconditional debiasing can reduce segmentation performance, including ISIC mIoU falling from 46.8 to 41.2, whereas adaptive debiasing adjusts suppression to correspondence reliability.Adaptive debiasing suppresses position-dominated activations only when necessary.Source: C SEMANTIC CORRESPONDENCE ANALYSIS OF… · p. 15

D COMPUTATIONAL COST OF FORIS

On COCO-20i, FoRIS offers a more favorable accuracy–efficiency trade-off at lower resolution than INSID3 at 1 024 px. Increasing FoRIS resolution yields diminishing accuracy gains while sharply increasing runtime.

D COMPUTATIONAL COST OF FORIS

  • At 1 024 px on one RTX 3090, FoRIS takes 1 620.77 ms per example and is slower than INSID3 at the same resolution.The encoder pass and foreground localization and consolidation account for most of the cost; purification adds relatively little overhead.Source: D COMPUTATIONAL COST OF FORIS · p. 15, Table 6 · p. 16
  • At 512 px, FoRIS reaches 59.5% mIoU in 258 ms, exceeding INSID3’s 57.6% mIoU in 812 ms at 1 024 px, with 68.2% less inference time.This comparison supports a favorable lower-resolution accuracy–efficiency trade-off on COCO-20i.Source: D COMPUTATIONAL COST OF FORIS · pp. 15–16, Table 7 · p. 16
  • Raising FoRIS resolution from 512 to 1 024 px increases mIoU from 59.5% to 60.9% but runtime from 258 to 1 621 ms; at 1 760 px, mIoU remains around 61%.Runtime reaches 11 235 ms at 1 760 px, indicating diminishing returns in this evaluation.Source: D COMPUTATIONAL COST OF FORIS · p. 16, Table 7 · p. 16

E N-SHOT SEGMENTATION

FoRIS remains strong with five reference examples, outperforming the strongest training-free baseline across benchmarks and domains. It reuses its one-shot hyperparameters without additional tuning.

E N-SHOT SEGMENTATION

  • FoRIS achieves 64.7% average mIoU in 5-shot segmentation, beating INSID3 by 4.8 percentage points and ranking first on 7 of 8 benchmarks.It outperforms INSID3 across all datasets, with gains of 7.9 points on X-Ray, 6.2 on PASCAL, and 5.4 on PACO.Source: E N-SHOT SEGMENTATION · p. 16, Table 8 · p. 17
  • FoRIS uses the same DINOv3 encoder and parameter count as INSID3, so its reported gains do not rely on increased model capacity.The authors attribute gains to foreground purification, localization, and consolidation.Source: E N-SHOT SEGMENTATION · p. 16
  • All 5-shot hyperparameters are reused from the 1-shot setting without tuning, with results spanning natural, medical, underwater, and aerial imagery.The table describes five contextual examples per target image.Source: E N-SHOT SEGMENTATION · p. 16, Table 8 · p. 17

F LIMITATIONS OF VFM-BASED TRAINING-FREE ICS FOR SLENDER STRUCTURE SEGMENTATION

FoRIS improves over other training-free methods on slender-structure benchmarks, but all methods remain far below compact-object performance. Results show strong sensitivity to encoder depth and input resolution, underscoring unresolved limits in slender-structure recovery.

F LIMITATIONS OF VFM-BASED TRAINING-FREE ICS FOR SLENDER STRUCTURE SEGMENTATION

  • Slender structures challenge standard VFM-based ICS because weak local support and varied appearance make precise localization, boundary preservation, and continuity difficult.The evaluation treats this setting as a stress test, using Fundus and DeepGlobe-18 with frozen DINOv3-L and episodic evaluation.Source: F LIMITATIONS OF VFM-BASED… · p. 17, §F.1 · p. 18
  • When a slender structure occupies little of a patch, its token can be dominated by surrounding background, weakening foreground-context separability and training-free matching.This patch-level mixing is contrasted with compact objects, whose tokens usually contain enough foreground signal.Source: F LIMITATIONS OF VFM-BASED… · p. 17
  • Variation in scale, orientation, curvature, illumination, and local appearance disperses same-class features, so one prototype may not cover an elongated target’s appearance and topology.This is a limitation of the compact-cluster assumption used by prototype-based approaches.Source: F LIMITATIONS OF VFM-BASED… · p. 17

F.2 QUANTITATIVE COMPARISON ON SLENDER STRUCTURES

  • Even FoRIS’s best Table 9 result, 21.5% mIoU on Fundus in 5-shot, is an order of magnitude below compact-object benchmark scores.The authors conclude current VFM representations and prototype-based ICS pipelines remain insufficient for reliable slender-structure recovery.Source: §F.2 · p. 18, Table 9 · p. 18
  • FoRIS exceeds INSID3 and GF-SAM on both datasets in 1-shot, by 7.4 and 9.8 points on Fundus and 6.1 and 6.2 on DeepGlobe-18, respectively.The authors characterize these gains as modest relative to the low performance ceiling.Source: §F.2 · p. 18
  • Adding references offers limited benefit: FoRIS gains only 1.8 Fundus mIoU points from 1-shot to 5-shot, while all methods gain under 1.5 points on DeepGlobe-18.INSID3 Fundus mIoU falls from 12.3 to 11.7 with five references.Source: §F.2 · p. 18

F.3 LAYER-WISE AND RESOLUTION SENSITIVITY

  • Early layers 0–7 stay below 13% mIoU on Fundus and 10% on DeepGlobe-18, while mid-to-deep layers capture partial structure cues.At 2048 inputs in 1-shot, Fundus reaches 28.9–31.9% at layer 12 and DeepGlobe-18 reaches 23.9% at layer 19.Source: §F.3 · pp. 18–19
  • At 1024 in 1-shot, default layer 23 reaches 19.7% mIoU on Fundus and 13.3% on DeepGlobe-18, below the best layers and resolutions.Best overall results reach 41.7% on Fundus at layers 20–21 and 2048, and 23.9% on DeepGlobe-18 at layer 19 and 2048; higher resolution improves best-layer results but both remain below 45% and 27%, respectively.Source: §F.3 · p. 19, Table 10 · p. 19, Table 11 · p. 20

F.4 FEATURE REPRESENTATION ANALYSIS

The analysis visualizes layer-wise DINOv3 patch features on Fundus and DeepGlobe-18 episodes using feature norms, PCA colorization, and positional debiasing. It finds foreground-background mixing and dispersed foreground semantics; debiasing sharpens alignment but does not reconnect fragmented structures.

Feature visualization setup

  • The layer-wise DINOv3 analysis covers representative Fundus and DeepGlobe-18 episodes, displaying feature norms, raw-token PCA colorization, and PCA after positional debiasing.Figure 9 specifies the reference image, mask, target image, and ground truth in the top row; subsequent rows show reference and target feature views.Source: §F.4 · p. 21, Figure 9 · p. 22, Figure 10 · p. 23

Foreground–background mixing in patch tokens

  • Early and mid-layer patch features mix foreground with surrounding tissue or terrain, while high-magnitude responses often track illumination or texture rather than thin foreground branches.The analysis presents this as a patch-mixing limitation: a single token rarely represents a vessel or road in isolation.Source: §F.4 · p. 21

Dispersed foreground semantics across branches

  • In deeper layers, branches, crossings, and terminators remain separate PCA color modes, helping explain why prototype-based ICS misses thin side branches and peripheral capillaries.The passage attributes these misses to compressing the reference mask into a small set of cluster centers.Source: §F.4 · p. 21

Positional debiasing improves alignment but not topology

  • Positional debiasing reduces layout-driven gradients and sharpens cross-image correspondence, but does not reconnect fragmented segments in similarity maps.The passage links this to modest quantitative gains over INSID3: purification improves semantic alignment, while structural continuity remains largely unaddressed.Source: §F.4 · p. 21

Related papers

Loading 2609.03384v1…