PaperlayerView exact revision on arXiv

Source-linked AI summary

ProtoRAG: Prototype-Based Retrieval Augmentation for Few-Shot Fine-Grained Remote Sensing Object Detection

Jian Wang, Yuxiang Hong, Chufeng Zhou, Chao Pang, Xiaokang Zhang

arXiv:2609.05953v1cs.CV

TL;DR

Few-shot remote-sensing FGOD must distinguish similar subordinate categories from limited annotations, while coarse MLLM localization lacks explicit fine-grained visual evidence. ProtoRAG adds a learned prototype memory and uncertainty-guided reasoning, and reports gains over the strongest competitors across nine few-shot evaluations.

  • Problem

    Limited annotations must support both localization and fine-grained discrimination, while coarse MLLM localization lacks explicit visual references for distinguishing similar categories.

    Source: Introduction · p. 1

  • Method

    ProtoRAG retrieves category-specific visual prototypes after MLLM localization, learns discriminative and stable representations with DPSL, and invokes candidate-constrained MLLM comparison for ambiguous targets.

    Source: Introduction · p. 2, Discriminative Prototype Space Learning · p. 3

  • Results

    Across nine few-shot evaluations, ProtoRAG beats the strongest competing method by 14.80, 2.27, and 4.04 mAP50 points on MAR20, HRSC2016, and FAIR1M-2.0, respectively.

    Source: Overall Comparison · p. 5

  • Takeaways & Limitations

    Experiments on three challenging benchmarks support prototype retrieval as an effective approach for adapting MLLMs to remote-sensing FGOD.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Few-shot fine-grained object detection (FGOD) in remote sensing imagery is challenging because limited annotations must support both object localization and discrimination among visually similar subcategories. Although multimodal large language models (MLLMs) provide strong coarse object localization, they lack explicit visual evidence for reliable fine-grained recognition. To address this limitation, we propose ProtoRAG, a prototype-based retrieval-augmented framework that decouples coarse localization from fine-grained recognition by equipping MLLMs with an external object-level visual memory. To construct a reliable visual memory from limited support samples, we introduce Discriminative Prototype Space Learning (DPSL), which encourages discriminative and prototype-stable representations through supervised contrastive learning and prototype-consistency regularization. We further develop an uncertainty-guided candidate-constrained reasoning strategy that augments MLLMs with retrieved candidate-specific visual references and invokes multimodal reasoning only for ambiguous instances. Extensive experiments show that ProtoRAG consistently surpasses representative baselines in nine few-shot settings, outperforming the strongest baselines by 14.80, 2.27, and 4.04 mAP$_{50}$ on MAR20, HRSC2016, and FAIR1M-2.0, respectively.

Introduction

Few-shot remote-sensing FGOD requires distinguishing visually similar categories from limited object-level observations, a task that coarse MLLM localization alone does not solve. ProtoRAG adds an external prototype memory and retrieval-guided reasoning to support fine-grained recognition.

Introduction

  • Remote-sensing FGOD must localize objects and distinguish subordinate categories whose cues lie in subtle structures, despite small objects and varied imaging conditions.Remote-sensing imagery also brings orientation, scale, and background-clutter variation.Source: Introduction · p. 1
  • MLLMs can localize objects but may confuse similar variants because coarse semantic knowledge lacks explicit visual references for object-level structural comparison.Sparse annotations also make conventional detectors prone to overfitting, while few-shot detectors often transfer localization knowledge without explicitly learning fine-grained differences.Source: Introduction · p. 1, Figure 1 · p. 2
  • Support instances can provide explicit category references, but overlapping generic features and few-shot prototype estimates make memory quality vulnerable to support selection and imaging variation.The passage motivates memory that is both discriminative and robust to support diversity.Source: Introduction · pp. 1–2
  • ProtoRAG decouples coarse localization from fine-grained recognition by retrieving candidate-specific references, using direct retrieval for confident predictions and MLLM comparison for ambiguous cases.The contributions include DPSL for discriminative, prototype-stable representations and uncertainty-guided candidate-constrained reasoning.Source: Introduction · p. 2
  • Experiments report consistent improvements over conventional, few-shot, open-vocabulary, MLLM-based, and retrieval-based methods on MAR20, HRSC2016, and FAIR1M-2.0.The paper presents this as the first retrieval-augmented framework for few-shot remote-sensing FGOD.Source: Introduction · p. 2

Related Work

Prior few-shot remote-sensing FGOD methods use limited annotations for both localization and fine-grained recognition, while MLLM and prompt-based methods do not reliably capture subtle category structure. Existing retrieval approaches also remain limited in their task and feature-space scope.

Few-Shot Fine-Grained Object Detection

  • Few-shot FGOD methods reduce annotation needs but jointly learn localization and fine-grained recognition from sparse labels, limiting reliable fine-grained representations.Earlier FGOD work also uses part-aware modeling, global–local interaction, and multi-scale features, commonly under full supervision.Source: Few-Shot Fine-Grained Object Detection · p. 2

MLLM-Based Object Detection

  • MLLMs and prompted detectors can localize or extend detection to novel categories, but broadly aligned representations and generic-space retrieval limit fine-grained object detection.Prior retrieval methods mainly address image-level recognition or compare queries with prototypes or cached features in generic feature spaces.Source: MLLM-Based Object Detection · p. 2

Method

ProtoRAG separates MLLM-based coarse localization from few-shot fine-grained recognition through prototype learning and uncertainty-guided inference. During training, only DPSL's lightweight feature mapping is optimized; the visual encoder and MLLM remain frozen.

Method

  • ProtoRAG uses DPSL and visual-memory construction before inference combines coarse MLLM localization, object-level prototype retrieval, and candidate-constrained reasoning.Only DPSL's lightweight feature mapping is optimized; the visual encoder and MLLM stay frozen to retain pretrained representations while adapting under limited supervision.Source: Method · p. 2

Discriminative Prototype Space Learning

Generic visual features entangle similar fine-grained categories, making discriminative, stable prototypes difficult to construct from sparse support samples. DPSL adapts frozen DINOv3 features with complementary instance-discrimination and prototype-consistency objectives, then uses adapted support embeddings to build a prototype bank.

Discriminative Prototype Space Learning

  • An MLLM localizes query targets for candidate retrieval; low-entropy queries use direct prototype prediction, while high-entropy queries undergo candidate-constrained visual comparison.The pipeline retrieves candidate categories and support references from the object-level prototype memory.Source: Figure 2 · p. 3
  • DPSL combines instance-level discrimination with category-level prototype consistency to produce an adapted embedding space for prototype construction and retrieval.The objectives jointly enhance category discrimination and prototype stability.Source: Discriminative Prototype Space Learning · p. 3, Category-Level Prototype Consistency · p. 4

Category-Level Prototype Consistency

  • For eligible categories with at least 2m support instances, two disjoint subsets provide prototypes whose cosine distance is minimized.This consistency objective addresses prototype variation across support subsets.Source: Category-Level Prototype Consistency · pp. 3–4

Prototype Bank Construction

  • After training, class-mean adapted embeddings form a fixed support-only bank, with one cached crop per category and cosine-similarity Top-K retrieval.The cached crop supplies visual evidence when candidate-constrained reasoning is triggered.Source: Prototype Bank Construction · p. 4

Uncertainty-Guided Coarse-to-Fine Inference

ProtoRAG localizes objects coarsely, retrieves fine-grained prototypes, and uses retrieval uncertainty to decide whether to accept a prediction or invoke candidate-constrained MLLM reasoning. It outputs each localized target with its fine-grained category and prototype score.

Uncertainty-Guided Coarse-to-Fine Inference

  • Coarse-to-fine inference combines MLLM localization, prototype retrieval, and uncertainty-triggered candidate-constrained reasoning.The MLLM first localizes by coarse category; each target is then projected into the DPSL space and matched to the prototype bank.Source: Uncertainty-Guided Coarse-to-Fine… · p. 4

MLLM-Based Object Localization

  • The MLLM localizes target objects using only a super-category prompt, without fine-grained category names.Examples of super-categories include aircraft and ships.Source: MLLM-Based Object Localization · p. 4

Object-Level Prototype Retrieval

  • Cosine similarities to category prototypes rank fine-grained classes, with the Top-K categories retained as candidates.K defaults to 3; the Top-1 category is the direct prototype prediction.Source: Object-Level Prototype Retrieval · p. 4

Retrieval Uncertainty

  • Retrieval entropy measures ambiguity among candidates: low entropy indicates one category dominates, while high entropy indicates multiple plausible categories.Below the threshold, ProtoRAG assigns Top-1 directly; at or above it, the MLLM reasons over the target, scene context, and one cached support reference per candidate.Source: Retrieval Uncertainty · p. 4

Fine-Grained Detection Output

Experimental Setup

Experiments cover three remote-sensing datasets and evaluate end-to-end detection, retrieval, and coarse localization. The setup uses shared few-shot support splits and fixed model and inference settings.

Datasets

  • MAR20, HRSC2016, and FAIR1M-2.0 provide the evaluation datasets, with selected aircraft and ship categories.MAR20 contains 20 aircraft types; HRSC2016 contributes 14 ship categories, and FAIR1M-2.0 contributes aircraft and ship subcategories.Source: Datasets · p. 4

Implementation Details

  • Shared support splits use random seed 42 and K ∈ {3, 5, 10}, while ProtoRAG uses frozen DINOv3 ViT-L/16 and Qwen3-VL-8B-Instruct.DPSL trains for 40 epochs; inference uses K = 3 and triggers reasoning at H ≥ 0.60.Source: Datasets · p. 4, Implementation Details · p. 5

Evaluation Metrics

  • End-to-end detection is measured by mAP50, retrieval by Top-1 and Top-3 accuracy, and coarse localization by precision, recall, and F1 at IoU = 0.5.All methods use horizontal bounding boxes; Table 1 reports mAP50 (%) across the three datasets and zero-shot and few-shot settings.Source: Evaluation Metrics · p. 4, Table 1 · p. 5

Overall Comparison

ProtoRAG achieves the best performance across all nine few-shot evaluations, with gains over the strongest competing method on each dataset at K = 10. Results also show variation among detector, prompt-based, and generic retrieval approaches.

Overall Comparison

  • Standard and few-shot detectors generally benefit from more support images but remain substantially behind retrieval-based approaches, particularly on MAR20 and FAIR1M-2.0.Source: Overall Comparison · p. 5
  • VisTex-OVLM's nearly unchanged performance across shot settings indicates that generic image prompts do not readily capture domain-specific fine-grained distinctions.Source: Overall Comparison · p. 5
  • Controlled retrieval baselines outperform most detector-based methods, but their relative performance varies across datasets and shot settings.ProtoNet reaches 69.26 mAP50 on MAR20 under the 10-shot setting; generic-space retrieval does not consistently yield discriminative and stable prototypes.Source: Overall Comparison · p. 5
  • 14.80, 2.27, and 4.04 mAP50 points: ProtoRAG surpasses the strongest competing method on MAR20, HRSC2016, and FAIR1M-2.0, respectively, at K = 10.ProtoRAG achieves the best performance across all nine few-shot evaluations.Source: Overall Comparison · p. 5
  • ProtoRAG's gains over controlled retrieval baselines cannot be attributed solely to MLLM localization, and qualitative results show accurate fine-grained labels for similar localized targets.The paper attributes the gains to learning discriminative, stable prototypes and selectively reasoning over ambiguous candidates.Source: Overall Comparison · p. 5

Ablation Studies

The ablations assess ProtoRAG’s components, DPSL objectives and backbones, MLLM localization, and uncertainty-guided reasoning. They report gains from prototype learning and selective reasoning, alongside efficient adaptation across backbones.

Contribution of Individual Components

  • mAP50 rises from 6.34 with zero-shot prediction to 71.08 with prototype retrieval, 83.46 with DPSL, and 84.06 with uncertainty-guided reasoning.DPSL adds 12.38 points over prototype retrieval; reasoning is selectively invoked for uncertain targets.Source: Contribution of Individual Components · p. 6, Table 2 · p. 6

DPSL Training Objectives

  • DPSL’s supervised contrastive and prototype-consistency objectives achieve 73.05/88.15 and 84.92/94.29 Top-1/Top-3 accuracy, respectively; joint optimization improves retrieval further.The reported results attribute complementary effects to instance-level discrimination and category-level prototype stabilization.Source: DPSL Training Objectives · p. 6, Table 3 · p. 6

Training Objective Top-1 Acc. Top-3 Acc.

  • Table 3 reports retrieval accuracy as Top-1/Top-3 percentages, comparing individual DPSL objectives with their joint optimization.The objectives target instance-level discrimination and category-level prototype stabilization.Source: Table 3 · p. 6, DPSL Training Objectives · p. 6

Backbone Generalization and DPSL Efficiency

Impacts of MLLM Choices

  • Increasing coarse-detection F1 from 58.59 to 94.94 accompanies a rise in final mAP50 from 46.78 to 84.06.The passage highlights reliable localization as important for downstream fine-grained retrieval.Source: Impacts of MLLM Choices · p. 7, Table 5 · p. 7

Impacts of Uncertainty-Guided Reasoning

  • Reasoning trigger rates fall from 14.22% at K = 3 to 6.34% at K = 10, while reasoning improves mAP50 by 0.99, 0.80, and 0.60 points at K = 3, 5, and 10.The qualitative example shows retrieval entropy flagging an ambiguous detection that candidate-constrained reasoning corrects.Source: Table 6 · p. 7, Impacts of Uncertainty-Guided Reasoning · p. 7

Conclusion

ProtoRAG combines MLLM-based coarse localization with prototype retrieval and uncertainty-triggered reasoning for few-shot remote sensing detection. Experiments on three benchmarks show consistent improvement over representative baselines.

Conclusion

  • ProtoRAG decouples coarse localization from fine-grained recognition using task-specific prototype retrieval and DPSL-learned prototypes.DPSL learns discriminative and stable prototypes from limited support annotations; retrieval entropy triggers candidate-constrained reasoning for ambiguous targets.Source: Conclusion · p. 7
  • Experiments on three challenging benchmarks show consistent improvements over representative baselines.The conclusion presents prototype retrieval as an effective approach for adapting MLLMs to remote sensing fine-grained detection.Source: Conclusion · p. 7
  • The inference pipeline combines MLLM-based coarse localization, prototype retrieval, and selective reasoning for ambiguous targets.The framework is evaluated for few-shot fine-grained object detection in remote sensing imagery.Source: Conclusion · p. 7
Loading 2609.05953v1…