PaperlayerView exact revision on arXiv

Source-linked AI summary

MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoders

Abdulmalik Alquwayfili, Faisal AlMeshal, Jumanah Almajnouni, Huda Abdulhadi Alamri, Muhammad Kamran J Khan

arXiv:2609.27142v2cs.CV

TL;DR

Queries naming small objects in cluttered scenes can depend on localized evidence that frozen dual encoders underrepresent in a global image embedding. MINER augments frozen representations with regional embeddings and hubness-correcting rescoring, and improves retrieval across tested backbones and splits. Its analyses find that spatial coverage matters more than precise crop placement.

  • Problem

    A single global image embedding can underrepresent small or visually subordinate objects in cluttered scenes, limiting retrieval when queries name them.

    Source: §1 · p. 1

  • Method

    MINER combines a frozen dual encoder’s global image embedding with fixed regional embeddings and two-sided CSLS rescoring, without retraining.

    Source: §1 · pp. 2–3

  • Results

    MINER improves R@1 on every tested backbone and split, including gains of +5.28 on ROCS-COCO and +5.84 on ROCS-Flickr30K with SigLIP 2.

    Source: §5.2 · p. 10, §6 · p. 14

  • Takeaways & Limitations

    The gains from region augmentation are governed by spatial coverage rather than precise crop placement, making fixed crops sufficient among the tested alternatives.

    Source: §1 · p. 3, §5.3 · p. 12

Abstract

from arXiv · show

Text-to-image retrieval with frozen dual encoders degrades when the query names a small, visually subordinate object in a cluttered scene: a single global image embedding underrepresents the localized visual evidence. We present MINER, a training-free inference framework that augments a frozen dual encoder's global image embedding with a small bank of region-level embeddings and a hubness-correcting similarity rescoring, recovering visual evidence that global pooling underweights. To evaluate this setting, we introduce ROCS, a benchmark built from high-clutter subsets of Flickr30K and MS COCO whose images are re-captioned to name a single low-salience object. Experiments on CLIP, SigLIP, and SigLIP 2 show that MINER improves retrieval on every backbone, on ROCS and on the standard splits. Analyses show that these gains come primarily from broader spatial coverage rather than precise crop placement, revealing a simple and general way to recover localized evidence from frozen representations. Code: https://github.com/aalquwayfili/MINER. Dataset: https://huggingface.co/datasets/aalquwayfili/ROCS.

1. Introduction

Frozen dual encoders compress each image into one global vector, which can miss small, subordinate objects in cluttered scenes. MINER adds fixed regional embeddings and hubness-correcting rescoring, while ROCS evaluates retrieval with captions focused on such objects.

Background

  • Text-to-image retrieval ranks gallery images by match to a natural-language query, but a single global embedding can underrepresent localized evidence in densely populated scenes.Dual encoders independently map images and text into a shared embedding space for cosine-similarity retrieval.Source: §1 · p. 1

MINER: A lightweight training-free retrieval framework

  • MINER combines a frozen encoder’s global image representation with regional embeddings and two-sided CSLS rescoring, without retraining, query conditioning, or auxiliary inference models.Gallery images and crops are encoded once at indexing time and reused across queries.Source: §1 · p. 3

ROCS: A benchmark for retrieval in cluttered scenes

  • ROCS uses high-clutter Flickr30K and MS COCO images re-captioned to emphasize rare, low-attention, visually subordinate objects.The benchmark targets a failure mode that standard retrieval benchmarks do not stress.Source: §1 · pp. 1, 3

Design principles for training-free retrieval

  • MINER improves retrieval across CLIP, SigLIP, and SigLIP 2 on both ROCS and standard benchmarks; gains are driven more by spatial coverage than precise crop placement.Fixed crops match the tested attention-guided alternatives while retaining generalization across backbones.Source: §1 · p. 3

2. Related Work

Related work spans global dual-encoder retrieval, fine-grained alignment, retrieval benchmarks, multi-crop inference, and hubness correction. MINER is positioned as a query-agnostic, training-free approach for frozen encoders, while ROCS isolates whether small objects are represented at all.

2.1. Text-to-Image Retrieval

  • Dual encoders such as CLIP and ALIGN provide zero-shot retrieval, but their single global image embeddings can underrepresent small or subordinate objects.BLIP and SigLIP pursue representation quality and training efficiency through bootstrapped caption generation and sigmoid contrastive loss, respectively.Source: §2.1 · p. 3

2.2. Fine-Grained Vision–Language Alignment

  • Fine-grained methods use token-level or region-level alignment, while query-conditioned encoders and cross-attention rerankers may require per-query computation or re-encoding.MINER instead encodes each gallery image and its crops once at indexing time for reuse across queries.Source: §2.2 · p. 4

2.3. Retrieval Benchmarks

  • MSCOCO-FG and Flickr30K-FG refine full-scene caption granularity, whereas ROCS re-anchors captions on a single small, rare object.The benchmark directions are complementary: one stresses caption matching precision, the other whether a low-salience object is represented at all.Source: §2.3 · p. 4

2.4. Multi-Crop Inference Strategy

  • Fixed multi-crop inference has a long history, and content-aware crop placement still does not use the text query; MINER tests fixed and content-aware placement as alternatives.Its ablation finds no benefit from content-aware placement over fixed coverage.Source: §2.4 · p. 4

2.5. Hubness Correction in Cross-Modal Retrieval

  • Two-sided CSLS corrects hubness by subtracting query- and gallery-side neighborhood means, avoiding the held-out temperature tuning of Inverted Softmax.Its performance is reported as insensitive to neighborhood size k; QBNorm is evaluated as a drop-in alternative.Source: §2.5 · p. 5

3. Rare Objects in Cluttered Scenes Dataset

ROCS is curated from crowded COCO and Flickr30K scenes, filtered for small single-instance object classes, then re-captioned to name a rare object. The resulting benchmark increases object density and class diversity while focusing queries on visually subordinate targets.

3. Rare Objects in Cluttered Scenes Dataset

  • ROCS is a benchmark of crowded scenes with many object instances and underrepresented classes, constructed through an automated curation pipeline.The source images come from COCO and Flickr30K.Source: §3 · p. 5, §3.1 · p. 5

3.1. Object Detection

  • Selecting the top 10% of images by object count forms the High-Clutter Subset for subsequent processing.Figure 3 reports that object density rises by approximately 4× at the high-clutter stage on both splits.Source: §3.1 · pp. 5–6, Figure 3 · p. 6

3.2. Rare Object Selection

  • Rare classes appear exactly once per image, and instances with SAM3 mask area above 5% are discarded as visually dominant.The final ROCS subset retains high-clutter images with at least one rare instance after filtering; mask areas estimate visual footprint more tightly than bounding boxes.Source: §3.2 · pp. 6–7

3.3. VLM Captioning

  • ROCS captions describe the scene while explicitly naming a rare object, whose median bounding box covers 0.65% of the image and whose area is below 1% for 57% of instances.Class-aware Qwen3-VL prompting guides this re-captioning; final ROCS images have substantially more objects and broader class diversity than original splits.Source: §3.3 · pp. 7–8, §3.2 · p. 7

4. Methodology

MINER augments frozen dual encoders with fixed regional crops, fuses global and regional similarities, and applies two-sided CSLS rescoring. Its design uses broad spatial coverage rather than precise crop placement to recover localized evidence without retraining.

4. Methodology

  • MINER encodes each image globally and with fixed regional crops, fuses their similarities, then applies two-sided CSLS rescoring without training or per-encoder tuning.Gallery images and crops use the same frozen encoder.Source: §4 · p. 8, Figure 5 · p. 8

4.1. Region Candidates

  • Five fixed crops—the center and four corners, each at 60% scale—provide region candidates encoded through the same frozen backbone as the global image.Each crop covers 36% of the image; distributed coverage matters more than exact placement, and the fixed set matches a 3×3 grid and saliency-guided variants.Source: §4.1 · pp. 8–9

4.2. Region Fusion

  • The fused score blends global similarity with the strongest regional similarity, using α=0.4 by default.α values from 0.2 to 0.6 remain within 1 R@1 of the peak on ROCS-COCO; the crop excess can recover evidence diluted by global pooling for the correct image.Source: §4.2 · p. 9

4.3. Hubness Correction

  • Two-sided CSLS subtracts query-side and image-side nearest-neighbor mean similarities to penalize universal neighbors, with k=10 computed on the full test-split matrix.This makes the correction transductive within the evaluated split; performance is flat across k=5, 10, and 20, and the rescoring and region-augmentation gains are near-additive on ROCS-COCO.Source: §4.3 · pp. 9–10

5. Experiments

Experiments evaluate MINER across three frozen backbones and four retrieval splits, including ROCS benchmarks that emphasize a single rare object in cluttered scenes. MINER improves R@1 throughout, while ablations examine stage contributions, crop coverage, rescoring choices, and inference cost.

5.1. Experimental Setup

  • The evaluation covers CLIP L/14, SigLIP So/14, and SigLIP 2 So/16 on standard COCO 5K and Flickr30K plus ROCS-COCO and ROCS-Flickr30K.ROCS uses high-clutter subsets re-captioned to name a single rare or visually subordinate object; the default configuration uses five fixed crops, α=0.4, and k=10.Source: §5.1 · p. 10

5.2. Main Results

  • MINER improves R@1 on every backbone and split with no regressions, with larger gains on ROCS than standard splits and gains up to +9.66 for CLIP L/14.With SigLIP 2, gains are +5.28 on ROCS-COCO and +5.84 on ROCS-Flickr30K, versus +3.69 and +3.36 on the standard splits.Source: §5.2 · p. 10, Table 2 · p. 11

5.3. Ablation Study

  • On ROCS-COCO, rescoring alone adds +3.21 R@1 and crops alone +2.03, versus +5.28 together; the two-stage pattern holds across the three backbones and both ROCS splits.On ROCS-COCO with SigLIP 2, two-sided CSLS lowers k-occurrence skewness from 2.35 to 1.80 and images never retrieved from 1.6% to 0.4%.Source: §5.3 · pp. 11, 13
  • A fixed five-crop set matches a 3×3 grid and outperforms attention-placed crops, while random low-coverage crops lag; saliency-guided variants remain within 1 R@1 of fixed crops.A pooling probe finds random patch subsets largely preserve matched image–text cosine at equal patch count, whereas a contiguous quadrant degrades it; masking the named object causes rank-1 loss in 53% of corrected cases versus 16% for a same-size random mask.Source: §5.3 · pp. 12–13
  • On held-out queries, CSLS is the best untuned correction: NNN and DBNorm trail it on ROCS-COCO and edge it by 0.26 on ROCS-Flickr30K, while alternatives show temperature sensitivity or underperform.Across sweeps, α is within 1 R@1 of its peak across [0.2, 0.6], crop ratio is flat across [0.5, 0.7], recall rises with N to 5, and k is flat across {5, 10, 20}.Source: Table 3 · p. 13, §5.3 · pp. 13–14

5.4. Inference Cost and Scalability

  • At five crops, image encoding costs about 4.8× a global pass and gallery storage rises from 2.3 GB to 14 GB per million images; blending and rescoring add 0.32 ms per query.Shortlist rescoring at M=25 removes the crop index while remaining within 0.15 R@1 of the full gain; with a representative query bank, the recommended deployment setting is never more than 0.4 R@1 below transductive rescoring across tested backbones and splits.Source: Table 4 · p. 14, §5.4 · p. 14

6. Conclusion

The conclusion presents MINER as a training-free way to augment frozen dual encoders with fixed crops and hubness correction, alongside ROCS for cluttered rare-object retrieval. The appendix algorithm computes gallery embeddings once, then produces per-query rankings through fusion and CSLS rescoring.

6. Conclusion

  • MINER improves R@1 on every tested backbone and split, with the largest gains on rare-object queries and weaker encoders.The conclusion attributes the design to complementary near-additive contributions from crop-based localized evidence and rescoring that removes measurable hubness.Source: §6 · p. 14
  • ROCS is a benchmark of cluttered scenes whose captions each name a rare object, and MINER is presented as a drop-in for existing dual-encoder retrieval systems.The conclusion says crop augmentation is governed by spatial coverage rather than localization and that fixed crops match saliency-guided variants.Source: §6 · p. 14

Appendix A. Inference Algorithm

  • The inference algorithm outputs a ranking per query after region-global blending and CSLS rescoring, using frozen encoders and gallery embeddings computed once at indexing time.Only fusion and CSLS rescoring depend on the query.Source: Appendix A · p. 17

Appendix B. Full Hyperparameter Sweeps

The sweeps show broad hyperparameter optima across ROCS splits, supporting one default configuration without per-dataset tuning.

Appendix B. Full Hyperparameter Sweeps

  • Across ROCS splits, α stays within 1 R@1 of its peak across [0.2, 0.6], recall rises with N to 5, crop ratio is flat across [0.5, 0.7], and k is flat across {5, 10, 20}.The defaults are α=0.4, N=5, r=0.6, and k=10; only k=1 underperforms among the reported neighbourhood settings.Source: Appendix B · pp. 17–18, Figure 8 · p. 18

Appendix C. Patch-masking probe of the pooling layer

The patch-masking probe finds that SigLIP 2 preserves text alignment under distributed patch coverage but not under a contiguous localized subset. This supports spread-out crops over a single localized crop.

Appendix C. Patch-masking probe of the pooling layer

  • A small attention-positioned crop is contiguous by construction, so four spread-out corner crops are more reliable than one localized crop.This is the paper’s stated interpretation of the patch-masking pattern.Source: Appendix C · p. 18
  • At equal patch counts, random 25% of patches retain a mean cosine of 0.139, while the top-left contiguous 25% scores 0.085.Even retaining a random 25% scores 0.126; a contiguous center 50% scores 0.100.Source: Appendix C · p. 18, Table 5 · p. 19
  • Scattered subsets track the full-grid pooled embedding, whereas contiguous subsets do not.At equal patch counts, pretrained pooling preserves text alignment under distributed coverage but not under a localized subset.Source: Appendix C · p. 18, Table 5 · p. 19

Appendix D. Additional Retrieval Results

Additional retrieval results show MINER’s image-to-text gains on ROCS and its behavior across object sizes and method stages. The crop stage contributes more on ROCS, while rescoring is the larger single stage overall.

D.1. Image-to-Text Retrieval

  • MINER lifts image-to-text R@1 by +1.2 to +2.3 on every ROCS cell and is neutral on standard splits (−0.7 to +1.5).The standard-split result is attributed to Flickr30K already exceeding 96 and image queries matching dominant scenes.Source: Table 6 · p. 19, §D.1 · p. 19

D.2. Gain by Object Size

  • The gain spans rare-object sizes and peaks for objects covering 2 to 5% of the image, but is not distinguishable from zero for the smallest objects.The authors note this bounds what a 60%-scale crop can recover.Source: §D.2 · p. 19, Table 7 · p. 20

D.3. Stage Decomposition

  • Across backbones and splits, rescoring is the larger single stage, the two stages are near-additive, and crops contribute more on ROCS.The crop-stage share grows where queries name objects underweighted by the global embedding.Source: Table 8 · p. 20, §D.3 · p. 20

E.1. Inductive CSLS and Skewness Statistics

The appendix evaluates inductive CSLS and compares alternative hubness corrections. Inductive CSLS stays close to transductive CSLS; QBNorm and Sinkhorn are more sensitive to settings, while CSLS rebalances hub occurrences.

E.1. Inductive CSLS and Skewness Statistics

  • Inductive CSLS is within 0.9 R@1 of transductive on all twelve combinations, within 0.3 on nine, and higher on seven.The deployment variant estimates the gallery-side term from a historical query log and scores incoming queries independently.Source: §E.1 · pp. 20–21, Table 9 · p. 21

E.2. QBNorm Temperature Sweep

  • At β ∈[2, 3], QBNorm has a shallow optimum, but even its per-backbone best recovers less gain than CSLS at k=10.Its performance declines steadily toward the original β=20 default.Source: §E.2 · p. 21, Table 10 · p. 22

E.3. Sinkhorn Sensitivity

  • R@1 swings by 3 to 8 points across Sinkhorn temperatures; only τ=0.02 with one iteration beats CSLS on both splits.At convergence, τ=0.02 wins on ROCS-Flickr30K (+0.9) and loses on ROCS-COCO (−0.9).Source: §E.3 · p. 21, Table 11 · p. 22

E.4. Hub Statistics

  • Two-sided CSLS lowers occurrence skewness and shrinks the unreachable image set by four times, while reducing the worst hub only modestly.The correction rebalances the tail rather than removing individual hubs.Source: §E.4 · p. 22, Table 12 · p. 23

F.1. Candidate Shortlist Top-M Re-ranking

The appendix examines shortlist reranking, object-level attribution, crop fusion, gallery scaling, and budget accounting. Small shortlists recover full-gallery gains, while attribution and scaling analyses characterize when and how MINER’s crop contribution appears.

F.1. Candidate Shortlist Top-M Re-ranking

  • Shortlists of M=10–25 recover full-gallery MINER’s retrieval gain across all four benchmark splits; at M=500, performance matches full-gallery MINER.Crop encoding can therefore be delayed until inference without requiring a full regional index.Source: Table 13 · p. 23, §F.1 · p. 23

F.2. Object-Level Attribution

  • On corrected-baseline errors that MINER fixes, the winning crop contains the object in 68% of cases versus 29% chance; masking the object also tests its role in retrieval.For objects under 0.5% of the image, masking causes rank-1 loss in 31% versus 8% under the random-region control.Source: §F.2 · pp. 23–24, Table 14 · p. 24

F.3. Crop-Fusion Blending Rules

  • Max pooling is within 0.1 R@1 of the best crop-fusion rule, while mean pooling costs 1.1–1.6 R@1.The authors attribute the mean-pooling cost to averaging the rare-object evidence in one crop with four irrelevant crops.Source: Table 15 · p. 24, §F.3 · p. 24

F.4. Gallery Scaling

  • As the gallery diversifies to 11,232 images, the hubness share shrinks while the region gain holds; MINER stays ahead on all 24 cells.The region gain over the corrected baseline remains +1.5 to +2.4 at every tested size.Source: §F.4 · pp. 24–25, Table 16 · p. 25

F.5. Budget Accounting

  • Table 17 compares storage, compute, and ROCS retrieval across region-aware alternatives using the same frozen encoder with hubness correction.Rerankers requiring a cross-encoder or detector are outside this comparison’s setting.Source: Table 17 · p. 25, §F.5 · p. 25

Appendix G. ROCS Human Audit

The ROCS audit reports caption accuracy and retrieval on overlapping-caption queries, while subsequent appendices compare crop generators and saliency sources. The remaining material documents the prompts used for ROCS captioning and SAM3 detection.

Appendix G. ROCS Human Audit

  • 12% of audited captions contain a wrong claim, and 37% also fully fit another top-ranked image; on those queries R@1 is 19% for the global baseline and 16% for MINER, versus 59% and 65% on the rest.The audit covers 200 random queries, 100 per split.Source: Appendix G · p. 25, Table 18 · p. 26

Appendix H. Region Candidate Generators

  • All 60%-scale region generators land within 1.3 R@1 of one another, consistent with spatial coverage rather than precise localization driving the gains.The comparison includes fixed five-crop, 3×3 grid, random, attention-placed, and saliency-guided variants.Source: Appendix H · p. 26, Figure 10 · p. 28

Appendix I. Saliency-source comparison

  • Swapping the saliency source changes R@1 by at most 0.42 in any cell, despite visibly different attention maps.The comparison swaps the source placing the guided crop in the saliency-guided variant.Source: Appendix I · p. 26, Table 19 · p. 27

J.1. Qwen3-VL Captioning Prompts

  • ROCS captioning prompts substitute each image’s highlighted object category for the placeholder {rare class}.The prompts generate MS COCO–style captions during dataset curation.Source: §J.1 · p. 27

J.1.1. System Prompt

  • The system prompt requests one neutral, factual MS COCO-style sentence, roughly 10–18 words, naturally naming the highlighted object.It also discourages extra adjectives, lists, chained clauses, and meta phrases.Source: §J.1.1 · p. 27

J.1.2. User Prompt

  • The user prompt asks for one natural, neutral MS COCO-style sentence of roughly 8–16 words that clearly includes the labeled rare class.Source: §J.1.2 · p. 27

J.2. SAM3 Detection Prompts List

  • SAM3 receives the canonical 80 MS COCO class names as noun-phrase prompts, with six labels rephrased to disambiguate open-vocabulary detection.The remaining 74 prompts retain their canonical names.Source: §J.2 · pp. 27–28

Related papers

Loading 2609.27142v2…