PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation

Yijun Hu, Heng Fan, Libo Zhang

arXiv:2609.28949v1cs.CV

TL;DR

Few-shot segmentation relies on a small set of support examples, leaving target-specific information limited in challenging query images. MK-FSS uses MLLM-derived spatial and semantic knowledge through DMDF and PCPG, and reports state-of-the-art performance on PASCAL-5i and COCO-20i.

  • Problem

    Support-guided FSS remains limited in scenarios with appearance variation, occlusion, or visually similar distractors because few support images cannot describe targets across diverse conditions.

    Source: §1 · p. 1

  • Method

    MK-FSS uses an MLLM to derive spatial and semantic knowledge from the query image, incorporating them through DMDF memory fusion and PCPG multimodal prompting.

    Source: §1 · p. 2

  • Results

    MK-FSS establishes new state-of-the-art performance on PASCAL-5i and COCO-20i.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework combines complementary target cues beyond those available from limited support samples for a more comprehensive representation and robust segmentation.

    Source: §1 · p. 2, §5 · p. 9

    Target-knowledge elicitation uses the class label ξ, which is available in standard FSS benchmarks; the process requires no additional manual annotations.

    Source: §3 · p. 3

Abstract

from arXiv · show

Few-shot segmentation (FSS) aims to segment unseen object categories with a few (e.g., one or five) labeled examples, enabling efficient adaptation to novel classes. Conventional models typically rely on appearance-based visual matching between support and query images for segmentation. While straightforward, these methods often struggle to handle significant appearance discrepancies and occlusions in the query image due to insufficient target knowledge. To mitigate this, we introduce a novel framework that mines target knowledge using the strong reasoning capacity of Multimodal Large Language Models (MLLMs) and employs it to enhance FSS. Specifically, building on SAM 2, our method, named MK-FSS, exploits two forms of complementary knowledge derived from a query image by an MLLM for FSS, including spatial knowledge, which provides a spatial prior indicating the potential target location, and semantic knowledge, which describes the target using text. The spatial knowledge is first encoded into a memory representation, and then resulting memory is integrated with the support-guided memory feature from query image through a carefully designed dual-memory debate-fusion (DMDF) module, yielding a more robust target memory feature. In parallel, the semantic knowledge is encoded into the textual feature, which is fused with multi-scale query features via a progressive cross-modal prompt generator (PCPG), producing a target-aware multimodal prompt for segmentation. Working together, the dual-memory feature and the multimodal prompt provide a comprehensive representation of the target, enabling more robust segmentation. In our extensive experiments, MK-FSS shows promising results and largely surpasses existing methods. Code will be released.

1 INTRODUCTION

FSS adapts to novel classes from a few labeled examples, but support-only cues remain limited under appearance variation, occlusion, and visually similar distractors. MK-FSS addresses this with complementary query-image spatial and semantic knowledge, incorporated through DMDF and PCPG.

1 INTRODUCTION

  • FSS segments novel object classes from a few labeled examples, addressing settings where pixel-level annotation is time-consuming and labor-intensive.Examples named include medical imaging, remote sensing, and autonomous driving.Source: §1 · p. 1
  • Support-guided FSS remains limited by appearance variation, occlusion, and visually similar distractors, while few support images cannot describe targets across diverse conditions.Earlier foundation-model and category-name textual cues still exploit target-specific information only in a limited way in complex scenarios.Source: §1 · p. 1
  • MK-FSS uses MLLM-derived spatial knowledge in DMDF and semantic knowledge in PCPG to build a comprehensive target representation for few-shot segmentation.DMDF integrates spatial knowledge with support-guided memory; PCPG fuses text with multi-scale query features.Source: §1 · p. 2
  • Unlike prior LLM-based FSS methods using polygon predictions or category-level descriptions, MK-FSS extracts target-specific spatial and semantic knowledge directly from the query image.It incorporates that knowledge into target representation learning rather than using it only as direct segmentation guidance or visual prompts.Source: §1 · p. 2

2 RELATED WORK

Prior FSS methods use support prototypes or support-query feature matching, while later work adds foundation-model, category-text, or LLM-derived cues. MK-FSS instead derives query-specific spatial and semantic knowledge with an MLLM and incorporates it through dedicated modules.

Few-Shot Segmentation

  • Prototype-based methods summarize support information as prototypes, whereas feature-matching methods use pixel-level support-query correspondences to segment query images.Both paradigms rely heavily on limited support samples and may struggle with occlusions and visually similar distractors.Source: §2 · p. 2
  • Foundation models and category-name text provide auxiliary target knowledge, but prior auxiliary cues remain limited in resolving target ambiguity.Earlier LLM-based FSS methods use spatial localization cues or category-level textual descriptions, yet do not fully exploit query-specific target information.Source: §2 · p. 2

Multimodal Large Language Models

  • MK-FSS derives spatial and semantic target-relevant knowledge from the query image with an MLLM and incorporates it into FSS through two dedicated modules.This differs from other MLLM-guided visual tasks such as SEAL's visual cues for detection and DeepEeyes' potential target-region identification.Source: §2 · p. 3

3 OUR APPROACH

The approach describes how MK-FSS combines MLLM-derived spatial and semantic cues with support-based guidance for mask prediction, along with its CMF architecture, K-shot procedure, and Dice-loss training.

3.1 OUR MK-FSS FRAMEWORK

  • Spatial cues about target location and semantic descriptions provide complementary information, and the available class label elicits this knowledge without additional manual annotations.The episodic protocol trains on base classes and evaluates on disjoint novel classes.Source: §3 · p. 3, §3.1.1 · p. 4

3.1.1 TARGET KNOWLEDGE GENERATION

  • The MLLM predicts query-image target boxes to form a binary spatial prior and generates a target description encoded as sentence-level and word-level text features.The spatial and semantic knowledge are elicited using the query image and class label.Source: §3 · p. 3, §3.1.1 · p. 4

3.1.2 FEW-SHOT SEGMENTATION WITH TARGET KNOWLEDGE

  • Support-derived foreground and background prototypes produce a spatial prior for SAM 2 memory encoding, while MLLM-derived spatial memory and semantic text features yield a dual memory and multimodal prompt for mask prediction.The semantic pathway can help disambiguate visually similar query regions.Source: §3.1.2 · pp. 4–5

3.2 DUAL-MEMORY DEBATE-FUSION

  • DMDF exchanges complementary information between support-guided and MLLM-derived memories via bidirectional cross-attention, estimates channel-wise reliability weights, and fuses the updated memories.Its debate, verification, and integration stages produce the final dual-memory target feature.Source: §3.2 · pp. 5–6

3.3 PROGRESSIVE CROSS-MODAL PROMPT GENERATOR

  • PCPG fuses text with each of three query-feature scales through cross-modal attention, then progressively aggregates the resulting features from shallow to deep levels into a multimodal prompt.The cross-modal fusion uses two consecutive cross-attention layers followed by residual connections and an FFN.Source: §3.3 · pp. 6–7
  • Fig. 3 shows the architecture of CMF.Source: §3.3 · p. 7

3.4 K-SHOT EXTENSION AND OPTIMIZATION

  • In K-shot FSS, MK-FSS averages the support-guided prior masks from all support samples and then follows the one-shot procedure.Source: §3.4 · p. 7
  • During training, MK-FSS is optimized with standard Dice loss between its predicted foreground mask and the target ground-truth mask.Source: §3.4 · p. 7

4 EXPERIMENTS

MK-FSS leads the reported PASCAL-5i and COCO-20i comparisons, and ablations support combining MLLM-derived spatial and semantic knowledge. The experiments evaluate this approach across two benchmarks and examine its memory and prompt designs.

4.1 DATASETS AND EVALUATION METRICS.

  • Experiments fine-tune SAM 2-small memory attention and mask decoder, train new modules from scratch, and evaluate PASCAL-5i and COCO-20i using mIoU and FB-IoU.Both datasets use four disjoint folds, with three for training and one for testing; evaluation samples 1,000 episodes by default.Source: §4 · p. 7, §4.1 · p. 7

4.2 STATE-OF-THE-ART COMPARISON

  • MK-FSS achieves 87.1% and 87.7% mIoU on PASCAL-5i and 74.3% and 75.9% mIoU on COCO-20i in the 1-shot and 5-shot settings, surpassing prior methods.The reported comparisons are summarized in Tables 1 and 2.Source: §4.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.3 ABLATION STUDY

  • In PASCAL-5i 1-shot ablations, the support-only, DMDF-only, PCPG-only, and combined models achieve mean scores of 77.3%, 86.0%, 80.0%, and 87.1%, respectively.The combined DMDF and PCPG model achieves the best mean score.Source: §4.3 · p. 8, Table 3 · p. 8
  • Using either support-guided memory or MLLM-derived memory alone is suboptimal compared with combining both through DMDF.The ablation attributes complementary target guidance to the two memories.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • The full DMDF, with debate, verification, and integration stages, is reported to improve segmentation accuracy over settings using only partial stages.The compared partial settings use debate alone with simple addition, or verification and integration alone.Source: §4.3 · p. 9, Table 5 · p. 9
  • DMDF achieves the best performance across evaluation splits against learnable addition and concatenation followed by MLP fusion.These alternatives combine support-guided and MLLM-derived memory features using different fusion strategies.Source: §4.3 · p. 9, Table 6 · p. 9
  • PCPG’s progressive fusion outperforms using only the deepest textual feature or directly adding multi-scale features with learnable weights.The authors state that progressive integration captures both local details and global semantic cues.Source: §4.3 · p. 9, Table 7 · p. 9
  • Combining sentence-level and word-level text embeddings achieves superior performance to the alternative text-embedding strategies tested.The comparison is reported in Table 8.Source: §4.3 · p. 9, Table 8 · p. 9

5 CONCLUSION

MK-FSS combines MLLM-derived spatial and semantic knowledge through DMDF and PCPG to enrich target representations beyond limited support samples. Experiments on multiple FSS benchmarks report strong performance and generalization.

5 CONCLUSION

  • MK-FSS uses MLLM-derived spatial and semantic knowledge to enhance few-shot segmentation through DMDF and PCPG.DMDF fuses spatial-knowledge memory with support-guided memory, while PCPG uses semantic knowledge to enrich visual representations and generate a multimodal prompt.Source: §5 · p. 9
  • The framework captures complementary target cues beyond those available from limited support samples alone.The conclusion describes the resulting dual-memory feature and multimodal prompt as additional guidance for target segmentation.Source: §5 · p. 9
  • Experiments on multiple FSS benchmarks report strong performance and generalization for MK-FSS.Source: §5 · p. 9

A APPENDIX

The appendix lists supplementary prompt, architecture, visualization, robustness, and comparison analyses. It also includes checks of alternative MLLMs and fine-grained target-specific language descriptions.

A APPENDIX

  • Supplementary materials include MLLM prompt templates and generated-knowledge examples, PGM architecture details, and visualizations of PMLLM and Psupp.Source: A APPENDIX · p. 13
  • Additional evaluations cover random-seed error bars, testing-episode counts on COCO-20i, qualitative segmentation comparisons, linguistic cues, and PCPG cross-attention depth.Source: A APPENDIX · p. 13
  • The appendix also reports results with alternative MLLMs and compares target-specific with class-specific language descriptions.Source: A APPENDIX · p. 13

A.1 PROMPT DESIGN FOR MLLMS AND EXAMPLES OF GENERATED KNOWLEDGE

The section describes prompts for eliciting spatial and semantic knowledge from MLLMs, and illustrates how the resulting cues relate to segmentation examples. It reports that spatial localization and semantic attributes provide complementary target cues.

A.1 PROMPT DESIGN FOR MLLMS AND EXAMPLES OF GENERATED KNOWLEDGE

  • In Figure 4, blue marks ground-truth target regions, red marks MLLM-predicted spatial regions, and green marks MK-FSS segmentation results.Source: Figure 4 · p. 13
  • The spatial prompt requests accurate, complete bounding boxes in valid JSON, while the semantic prompt asks for short factual descriptions of visible target attributes and excludes background details and unsupported guesses.Source: §A.1 · p. 13
  • The examples show spatial knowledge roughly localizing potential targets while semantic knowledge describes discriminative visual attributes; the authors report that integrating them yields more accurate segmentation in challenging scenes.Source: §A.1 · p. 14

A.2 DETAILED ARCHITECTURE OF PGM

The prior generation module (PGM) derives a support-guided prior by comparing query features with foreground and background prototypes formed from the support image and mask.

A.2 DETAILED ARCHITECTURE OF PGM

  • PGM extracts support and query features with a frozen DINOv2 encoder, then uses the support mask to form foreground and background prototypes.Figure 5 depicts the PGM architecture.Source: §A.2 · p. 14, Figure 5 · p. 14
  • Normalized cosine similarities between query features and the two prototypes produce foreground and background similarity maps.Norm denotes min-max normalization.Source: §A.2 · p. 14
  • PGM subtracts the background map from the foreground map and applies ReLU to form the support-guided prior, averaging per-support priors in K-shot settings.The contrast emphasizes regions more similar to the support foreground and suppresses negative responses.Source: §A.2 · p. 14

A.3 VISUALIZATION EXAMPLES OF PMLLM AND PSUPP

The visualization contrasts support-guided and MLLM-derived priors and shows their complementary contributions to segmentation on PASCAL-5i.

A.3 VISUALIZATION EXAMPLES OF PMLLM AND PSUPP

  • The support-guided prior captures fine-grained visual correspondence but can respond to similar background regions and miss parts of the target.In Figure 6, each row presents the query image and support-guided prior heatmap before the MLLM-derived prior and final prediction.Source: §A.3 · p. 14, Figure 6 · p. 14
  • The MLLM-derived prior supplies coarse semantic localization that helps identify the target and suppress irrelevant regions.Figure 6 marks MLLM-derived prior regions in red; ground truth and predicted masks are blue and green, respectively.Source: §A.3 · p. 14, Figure 6 · p. 14
  • Combining the priors yields more complete and accurate segmentation results in the qualitative examples.The examples illustrate complementary detailed visual matching and high-level semantic guidance.Source: §A.3 · p. 14

A.4 ERROR BARS EVALUATION

The error-bar evaluation reports stable performance across five random seeds and four folds on PASCAL-5i and COCO-20i under the 1-shot setting.

A.4 ERROR BARS EVALUATION

  • The evaluation spans five selected random seeds and reports mean mIoU averaged over four folds.The selected seeds are 0, 1, 2, 3, and 321 (the default).Source: §A.4 · p. 14, Table 9 · p. 15
  • 86.8% mean mIoU with a 0.3 standard deviation is reported on PASCAL-5i across five seeds.The evaluation uses 1,000 testing episodes across four folds.Source: §A.4 · pp. 14–15, Table 9 · p. 15
  • 73.3% mean mIoU is reported on COCO-20i, where the standard deviation remains small despite the dataset's greater challenge.Fold-wise standard deviations are around 0.5 on PASCAL-5i; the passage describes COCO-20i variability as small without giving a numeric standard deviation.Source: §A.4 · pp. 14–15, Table 9 · p. 15

A.5 DIFFERENT NUMBER OF TESTING EPISODES

On COCO-20i, the reported mIoU changes only slightly across testing-episode counts from 1,000 to 20,000, with fold-wise variations described as consistent.

A.5 DIFFERENT NUMBER OF TESTING EPISODES

  • The evaluation varies COCO-20i testing episodes across 1,000, 4,000, 10,000, and 20,000, with random seed 321 fixed.Table 10 reports the mean as the average mIoU over four folds.Source: §A.5 · p. 15, Table 10 · p. 15
  • 0.8% separates the 74.3% mIoU with 1,000 testing episodes from the 73.5% average across all four episode settings.The comparison is on COCO-20i in the 1-shot setting.Source: §A.5 · p. 15, Table 10 · p. 15
  • Fold-wise variations remain consistent across episode settings, and the authors report stable performance under different testing conditions.Source: §A.5 · p. 15

A.6 QUALITATIVE ANALYSIS

Across qualitative and supplementary analyses, MK-FSS produces more complete masks in challenging scenes and benefits from linguistic cues. The analyses also report strong performance across MLLM choices, an optimal two-layer PCPG design, and gains from target-specific descriptions.

A.6 QUALITATIVE ANALYSIS

  • In challenging PASCAL-5i examples, MK-FSS produces more accurate, complete masks with clearer boundaries than representative methods.The comparison highlights cases involving occlusion and visually similar distractors, including bottles.Source: §A.6 · p. 15, Figure 7 · p. 16

A.8 IMPACT OF CROSS-ATTENTION DEPTH IN PCPG

  • The two-layer PCPG cross-attention design achieves the highest overall performance among the tested depths of one to three layers.The authors describe two layers as the optimal balance.Source: Table 11 · p. 16, §A.8 · p. 16

A.9 ANALYSIS OF IMPACT ACROSS MLLMS

  • 87.1% mean mIoU with Qwen3-VL-8B; MK-FSS also performs strongly across the evaluated MLLM choices.The framework’s performance further benefits from newer MLLMs with improved reasoning capabilities.Source: Table 12 · p. 16, §A.9 · p. 16

A.7 VISUALIZATION OF THE EFFECT OF LINGUISTIC CUES

  • Linguistic cues provide semantic priors that help visual features focus on relevant regions and disambiguate similar background objects.The examples include substantial appearance variation within the same object category, such as people and plants.Source: §A.7 · p. 16, Figure 8 · p. 17

A.10 EFFECTIVENESS OF OUR TARGET-SPECIFIC LANGUAGE DESCRIPTION

  • On every PASCAL-5i fold, target-specific descriptions generated per query image outperform ChatGPT-4.0 class-specific descriptions.The authors attribute the difference to image-specific descriptions capturing fine-grained, context-dependent details and providing more precise semantic cues.Source: Table 13 · p. 17, §A.10 · p. 17

Related papers

Loading 2609.28949v1…