PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt-Free Universal Region Proposal Network

Qihong Tang, Changhan Liu, Shaofeng Zhang, Wenbin Li, Qi Fan, Yang Gao

arXiv:2603.17554v1cs.CV

TL;DR

Existing object proposal methods rely on external prompts, limiting use when categories or exemplars are unavailable. PF-RPN replaces those prompts with learnable visual embeddings and modules for proposal localization and selection. Across 19 datasets, it generalizes to unseen domains with limited training data and improves proposal quality without retraining or external prompts.

  • Problem

    External text, category, or exemplar-image prompts limit flexible potential-object localization, especially when target categories and exemplars are unavailable in unseen domains.

  • Method

    PF-RPN uses a learnable embedding with Sparse Image-Aware Adapter, Cascade Self-Prompt, and Centerness-Guided Query Selection modules to generate proposals from visual features without external prompts.

  • Results

    PF-RPN achieves consistent improvements across 19 datasets, including 6.0/7.5/6.6 AR improvement on CDFSOD with 100/300/900 candidate boxes.

  • Takeaways & Limitations

    The method can be optimized with 5% of COCO data and directly applied to downstream object-detection tasks without additional fine-tuning.

Abstract

from arXiv · show

Identifying potential objects is critical for object recognition and analysis across various computer vision applications. Existing methods typically localize potential objects by relying on exemplar images, predefined categories, or textual descriptions. However, their reliance on image and text prompts often limits flexibility, restricting adaptability in real-world scenarios. In this paper, we introduce a novel Prompt-Free Universal Region Proposal Network (PF-RPN), which identifies potential objects without relying on external prompts. First, the Sparse Image-Aware Adapter (SIA) module performs initial localization of potential objects using a learnable query embedding dynamically updated with visual features. Next, the Cascade Self-Prompt (CSP) module identifies the remaining potential objects by leveraging the self-prompted learnable embedding, autonomously aggregating informative visual features in a cascading manner. Finally, the Centerness-Guided Query Selection (CG-QS) module facilitates the selection of high-quality query embeddings using a centerness scoring network. Our method can be optimized with limited data (e.g., 5% of MS COCO data) and applied directly to various object detection application domains for identifying potential objects without fine-tuning, such as underwater object detection, industrial defect detection, and remote sensing image object detection. Experimental results across 19 datasets validate the effectiveness of our method. Code is available at https://github.com/tangqh03/PF-RPN.

1. Introduction

Existing region proposal methods depend on prompts or fail to identify potential objects in unseen domains, limiting open-world applications. PF-RPN addresses this with prompt-free visual feature aggregation and demonstrates generalization across domains without retraining.

  • Existing RPN methods often fail to identify potential target objects from unseen domains.
  • Prompt-free OVD methods can use VLM-generated descriptions, but this introduces considerable latency costs.
  • Prompt-based OVD methods rely on predefined categories or exemplar images, which are often unavailable in industrial defect and underwater detection.
  • PF-RPN uses a learnable embedding and visual features to localize potential objects without external prompts, exemplar images, or textual descriptions.The Sparse Image-Aware Adapter dynamically updates the embedding by selectively aggregating multi-level visual features.
  • The Cascade Self-Prompt module iteratively refines the query embedding with multi-scale visual context to identify challenging objects such as small or occluded objects.
  • 6.0/7.5/6.6 AR improvement on CDFSOD and 4.4/5.2/5.8 AR improvement on ODinW13 demonstrates PF-RPN’s cross-domain proposal gains with 100/300/900 candidate boxes, respectively.The method is trained with limited data and applied to unseen domains without retraining or external prompts.

2. Related Works

Existing prompt-free and open-vocabulary detectors reduce reliance on manual prompts but may require expensive captioners or miss rich multi-level visual cues. PF-RPN instead uses a learnable visual embedding with sparse feature adaptation, cascading refinement, and centerness-guided query selection to propose objects without text or visual prompts.

  • Related Works: Prompt-free and open-vocabulary detectors still rely on captioners, category names, or exemplar images for object localization.These requirements limit use when target categories or exemplar images are unavailable.
  • Method Overview: PF-RPN directly proposes potential objects across diverse domains without text or visual prompts.It uses a learnable embedding rather than manually provided prompts.
  • Sparse Image-Aware Adapter: SIA routes and fuses the most informative multi-level visual features into a learnable embedding through cross-attention.The adapter selects top-k feature levels and uses global and spatial features to update the embedding.
  • Cascade Self-Prompt: CSP iteratively refines the embedding from deep to shallow layers using similarity masks and masked average pooling.The cascade aggregates semantic and fine-grained structural information while suppressing background noise; the method uses three iterations.
  • Centerness-Guided Query Selection: CG-QS selects proposal queries by combining classification scores with predicted centerness scores.The module prioritizes embeddings near object centers, which the method associates with more accurate proposals.
  • Training Objective: An auxiliary router loss balances expert weights so multi-level feature maps are more fully exploited.The loss penalizes the standard deviation of expert weights to reduce load imbalance.

4. Experiments

Experiments evaluate PF-RPN with limited training data, across backbones, modules, detectors, and cross-domain benchmarks. Results and visualizations indicate consistent gains, progressive region discovery, improved object-focused representations, and stronger proposal localization.

  • Experimental setup: PF-RPN is trained on 5% of COCO and 5% of ImageNet, then evaluated on diverse ODinW13 and CD-FSOD cross-domain benchmarks without further fine-tuning.The evaluation includes domains such as wildlife photography, household objects, and aerial imagery.
  • Quantitative comparisons: PF-RPN improves average recall over Grounding DINO by 7.8/11.8/13.5 AR on CDFSOD and 4.4/5.2/5.8 AR on ODinW13 with 100/300/900 candidate boxes.It also surpasses YOLOE by 16.3/19.1/21.1 AR and Qwen2.5-VL-7B by 40.6/45.2/48.1 AR under the same candidate-box settings.
  • Detector integration: PF-RPN improves integrated detectors by 3.7 AP on COCO with DE-ViT and 5.5 AP on CD-FSOD with CD-ViTO.These results assess replacing existing RPN components while retaining the downstream detector settings.
  • Module analysis: Increasing Cascade Self-Prompt iterations progressively expands selected regions from partial coverage toward most potential object regions.The method uses selected visual features to update learnable embeddings between iterations.
  • Module analysis: The Sparse Image-Aware Adapter shifts the learnable embedding toward semantically relevant object regions and suppresses background activations.This produces a stronger prior for subsequent Cascade Self-Prompt refinement, although residual background activations motivate further refinement.
  • Module analysis: Centerness-Guided Query Selection favors center-located queries, while boundary queries more often produce localization errors.The visual comparison connects center-focused selection with more accurate proposal boxes.

5. Conclusion

PF-RPN addresses prompt dependence in arbitrary object proposal with a learnable embedding and three modules for visual adaptation, iterative refinement, and query selection. Experiments support its zero-shot cross-domain object-proposal performance without external prompts.

  • Conclusion: PF-RPN replaces externally supplied text or visual prompts with a learnable embedding serving as a proxy for text embeddings.The approach targets arbitrary potential-object proposal in open-world scenarios.
  • Conclusion: SIA, CSP, and CG-QS respectively improve visual embedding localization, iterative object-region discovery, and selection of appropriate queries.CG-QS combines centerness and classification scores for subsequent query selection.
  • Conclusion: Extensive experiments demonstrate PF-RPN’s superiority in zero-shot cross-domain object proposal.The conclusion presents the result as evidence for the method’s cross-domain generalization.

1. Ablation Study on k

The Sparse Image-Aware Adapter performs best with moderate sparsity: k = 2 balances contextual information against redundant feature maps on CD-FSOD.

  • Ablation Study on k: k = 2 yields the best overall performance for the Sparse Image-Aware Adapter on CD-FSOD.The paper adopts k = 2 as the default setting.
  • Ablation Study on k: Increasing k introduces redundant feature maps and slightly degrades performance, whereas too small a k limits available contextual information.The ablation identifies moderate sparsity as the best trade-off.

2. Ablation Study on Objective Loss

The centerness-loss weight λ controls the balance between center-focused query selection and regression quality, with λ = 5 producing the best performance on CD-FSOD.

  • Ablation Study on Objective Loss: λ = 5 achieves the best performance in the objective-loss ablation on CD-FSOD.λ controls the contribution of centerness loss to the overall loss.
  • Ablation Study on Objective Loss: Too-small λ fails to teach center-region query selection, while too-large λ lets centerness loss dominate optimization and harms regression performance.The result indicates that centerness supervision must be balanced against regression objectives.

3. Efficacy of Self-Prompt

Object-internal features localize objects more precisely than the learnable embedding, motivating multi-level visual feature updates to improve the embedding’s object localization.

  • Object-internal features focus on object regions with higher semantic consistency than the more diffused learnable embedding response.The comparison is shown in Fig. 8 through similarity maps for a 4th-level feature and the learnable embedding.
  • Multi-level feature maps update the learnable embedding using internal visual cues to enhance object localization.

4. Latency and Efficiency Analysis

PF-RPN supports efficient deployment: additional CSP iterations improve performance with negligible latency overhead, and integration with lightweight detectors preserves near-conventional RPN speeds.

  • Increasing CSP iterations from 1 to 3 yields consistent performance gains with only an approximately 4.6 ms increase in inference time.
  • PF-RPN can integrate with lightweight detectors as a real-time, high-performance RPN while preserving speeds comparable to conventional RPNs.Integration with YOLO-World achieves competitive performance at comparable inference speeds.

5. Analysis of False Positives

PF-RPN reduces false positives while improving proposal quality by assigning higher confidence to true object candidates and suppressing task-irrelevant regions.

  • PF-RPN achieves larger AP improvements and fewer false positives than existing RPNs under a 100-proposal constraint.

6. Dependence on Base Detectors

PF-RPN is extensible across base detectors and benefits consistently from stronger detector backbones, with performance improving as base-model capacity increases.

  • PF-RPN integrates with various base detectors and gains steady performance improvements as the base model becomes more capable.

7. Comparison with Previous Prompt-Free Methods

PF-RPN is compared with open-source prompt-free methods on both proposal quality and computational efficiency. Results are reported across CDFSOD and ODinW13, with Figure 9 visualizing trends across all 19 target datasets.

  • 7. Comparison with Previous Prompt-Free Methods: +13.0 AR100 over GenerateU on CD-FSOD, while using 95% less VRAM and nearly 20× faster inference.The comparison concerns representative open-source prompt-free methods.
  • 7. Comparison with Previous Prompt-Free Methods: PF-RPN is compared with GenerateU and Open-Det as representative open-source prompt-free methods.
  • 7. Comparison with Previous Prompt-Free Methods: CapDet and DetCLIPv3 are excluded because their official code is unavailable.
  • 7. Comparison with Previous Prompt-Free Methods: PF-RPN is faster than GDINO, primarily because it removes the computationally expensive text encoder.
  • 8. Detailed Experimental Results on All 19 Datasets: Performance metrics are reported across datasets from the CDFSOD and ODinW13 benchmarks.
  • 8. Detailed Experimental Results on All 19 Datasets: Figure 9 presents line-chart comparisons of detailed AR trends against existing baseline methods on all 19 target datasets.
Loading 2603.17554v1…