PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Resolution: Object-to-Image Ratio Mismatch in Instance Retrieval

Boaz Meivar, Ofir Kedem, Amit Edenzon, Gal Chechik, Shai Avidan

arXiv:2610.11489v1cs.CVcs.IR

TL;DR

Visual instance retrieval struggles when an object occupies different fractions of query and gallery images, a factor often entangled with resolution loss. The paper isolates this mismatch and tests retrieval interventions, finding it accounts for most cross-distance degradation on many backbones and can be addressed on ILIAS 100M.

  • Problem

    Instance retrieval struggles when an object's fraction of the query and gallery differs, while scale changes also confound object-to-image ratio with resolution.

    Source: §1 · p. 2

  • Method

    The authors isolate object-to-image mismatch in controlled renders, then test query-side scale augmentation, OWLv2 crop reranking, and LoRA fine-tuning.

    Source: §1 · pp. 2–3, §5.5 · p. 9

  • Results

    Equalizing object-to-image ratio recovers more than 80% of cross-distance failure for 9 of 12 backbones, and training-free ILIAS 100M interventions achieve state-of-the-art performance.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    Object-to-image robustness is learnable, and query-side interventions can improve retrieval without training a new backbone or modifying the precomputed gallery index.

    Source: §1 · p. 3

Abstract

from arXiv · show

Visual instance retrieval often fails when the same object appears at different apparent sizes in the query and gallery. We show that the dominant cause is usually not resolution loss but object-to-image (O2I) ratio mismatch: the object occupies different fractions of the two images. On a controlled benchmark of 3,021 Objaverse objects rendered at five camera distances, more than 80% of the cross-distance degradation is attributable to O2I mismatch rather than resolution for 9 of 12 pretrained backbones; multi-scale architectures cut the resolution-only effect to single digits yet remain equally susceptible. The failure is also asymmetric: tight queries retrieve more reliably against wide gallery images than the reverse. Guided by this analysis, query-side scale augmentation and an OWLv2 crop reranker reach state of the art on ILIAS 100M (29.2 mAP@1000 before reranking, 42.0 after) without training or modifying the precomputed gallery index, and a LoRA fine-tune matches the query-side gains at a single forward pass, showing that O2I robustness is learnable.

1 INTRODUCTION

Visual instance retrieval can fail when the same object occupies different fractions of query and gallery images, a mismatch often conflated with resolution loss. The paper isolates this effect and shows that targeted interventions improve retrieval, including on ILIAS 100M.

1 INTRODUCTION

  • Retrieval scores show that a model fails but do not identify which visual factor caused the failure, while small targets are a longstanding retrieval difficulty.Common explanations for small-target failures emphasize clutter, limited resolution, or lost detail.Source: §1 · p. 1
  • O2I mismatch changes the image support forming a global descriptor, so identical objects can be compared under incompatible fractions of the input image.The paper distinguishes this from resolution loss, which concerns preserving or recovering object detail.Source: §1 · p. 2
  • For 9 of 12 tested backbones, more than 80% of cross-distance degradation is attributable to O2I mismatch rather than resolution loss.Multi-scale architectures reduce the resolution-only effect but remain susceptible to O2I mismatch.Source: §1 · p. 2
  • The failure is asymmetric: tight queries retrieve more reliably against wide gallery images than wide queries against tight gallery images.The paper reports this directional difference across its controlled analysis.Source: §1 · p. 3
  • On ILIAS 100M, query-side scale augmentation reaches 29.2 mAP@1000 before reranking, and OWLv2 crop reranking reaches 42.0 after, without training or modifying the gallery index.A LoRA fine-tune of DINOv3-L achieves similar gains to query-side test-time interventions at a single forward pass per image.Source: §1 · p. 3

2 RELATED WORK

Prior retrieval benchmarks measure real-world performance but entangle multiple visual factors, limiting their use for isolating O2I mismatch. Related work on robustness, scale-aware representations, and localized matching provides context for studying and addressing that factor.

2 RELATED WORK

  • Real-world benchmarks such as Oxford and Paris [Philbin et al., 2007; 2008] and ILIAS vary nuisance factors, making O2I-specific attribution difficult.ILIAS targets instance-level retrieval at scale with curated positives and hard distractors.Source: §2 · p. 3
  • Controlled robustness benchmarks include ImageNet-C and ObjectNet, but the cited literature typically studies classification rather than retrieval.Prior analyses also motivate concern that global representations need not be purely object-centric.Source: §2 · p. 3
  • Scale-aware architectures model scale directly, but are typically evaluated on classification or detection rather than image-to-image retrieval under controlled O2I mismatch.Detection and localization methods use localized readout, which is costly for web-scale first-stage retrieval but usable at reranking.Source: §2 · pp. 3–4
  • Patch-wise retrieval, query expansion [Chum et al., 2007; Arandjelović & Zisserman, 2012], and crop reranking exploit local structure or refine global rankings.The paper identifies O2I mismatch as a shared phenomenon behind gains from localized matching methods.Source: §2 · p. 4

3 CONTROLLED O2I BENCHMARK

The controlled benchmark varies camera distance while holding other visual factors largely fixed, enabling cross-distance comparisons across retrieval backbones. It tests whether retrieval changes with O2I ratio and whether architectures designed for scale variation resist that mismatch.

3.1 DATASET CONSTRUCTION

  • The synthetic benchmark renders 3,021 Objaverse objects at five camera distances, holding viewpoint, lighting, and neutral background fixed to isolate O2I variation.The dataset spans 120 categories and produces 15,105 images at 512 × 512 resolution.Source: §3 · p. 4, §3.1 · p. 4

3.2 CROSS-DISTANCE RETRIEVAL PROTOCOL

  • For each ordered pair of distinct distances, each object serves as a query against a gallery containing its target-distance render and other objects at all five distances.The query object's other-distance renders are masked to prevent trivial self-matches; ranks yield R@1, R@5, R@10, and MRR.Source: §3.2 · p. 4
  • All twelve backbones degrade with O2I mismatch, including multi-scale architectures, with 0.5–0.9 R@1 loss at the camera-distance-ratio extremes.Figure 2 plots cross-distance R@1 against dq/dt, with ratio 1 representing self-retrieval.Source: Figure 2 · p. 5

3.3 BACKBONES EVALUATED

  • The twelve backbones span self-supervised, vision-language, supervised, hierarchical or multi-scale, and 3D-reconstruction training paradigms.Including hierarchical and multi-scale architectures tests whether scale-variation design also confers O2I robustness.Source: §3.3 · p. 5

4 O2I ROBUSTNESS ANALYSIS

Across controlled retrieval tests, O2I mismatch explains most cross-distance degradation for most backbones, while resolution loss accounts for a smaller share. Fixed-pixel tests and directional comparisons further show that the effect is not simply resolution loss and is asymmetric.

4.1 ALL MODELS DEGRADE UNDER O2I MISMATCH

  • At a 3.3× camera-distance change, even the strongest backbone, SigLIP-2-L, retains only 0.126 R@1, and multi-scale models degrade along the same curve as standard ViTs.Retrieval peaks at self-distance and declines as the camera-distance ratio diverges in either direction.Source: §4.1 · p. 5

4.2 DISENTANGLING O2I FROM RESOLUTION

  • The decomposition compares raw cross-distance R@1 with O2I-matched crops upsampled to native input size; the remaining failure is attributed to resolution alone in this controlled setting.The raw failure is γ = 1 − α, the resolution-only residual is δ = 1 − β, and the resolution share is ε = δ/γ.Source: §4.2 · p. 6

4.3 DECOUPLING O2I FROM RESOLUTION WITH FIXED PIXELS

  • With identical object pixels pasted onto canvases from 256 to 1536 px, DINOv3-L self-similarity falls to 0.77 as O2I decreases from 1 to 1/36.Only the fraction of the image occupied by the object changes in this fixed-pixel test.Source: §4.3 · p. 6

4.2 DISENTANGLING O2I FROM RESOLUTION

  • For 9 of 12 backbones, resolution accounts for at most 20% of cross-distance degradation; multi-scale models retain large total drops despite resolution-only shares of 8–9%.For DINOv3-L, resolution accounts for 19% of the total drop; SigLIP-2-L and CLIP-L are exceptions with 56% and 41% resolution shares, respectively.Source: §4.2 · p. 6, Figure 3 · p. 7

4.4 CROSS-DISTANCE RETRIEVAL IS ASYMMETRIC IN O2I

  • Every backbone favors a wide-query-to-tight-gallery comparison over the reverse, with the reported difference ranging from +0.020 to +0.153 MRR.The effect spans self-supervised, vision-language, supervised, and hierarchical or multi-scale families.Source: §4.4 · pp. 6–7

5 CONFIRMING THE DIAGNOSIS ON ILIAS 100M

On ILIAS 100M, query-side scale interventions improve retrieval, and object-crop reranking and LoRA provide complementary routes to stronger results. Gains are concentrated in O2I-mismatched cases, supporting the diagnosis on a real benchmark.

5.1 APPROACH

  • Gray-pad averaging presents queries at five apparent sizes across a 4× range; MSA uses three scales and masks gray patches from CLS attention.Both retain the original query-vector dimension; MSA costs 3× query-time FLOPs.Source: §5.1 · pp. 7–8

5.5 RERANKING-STAGE CONFIRMATION: OBJECT-CENTRIC CROPS

  • 42.0 mAP@1000 versus 26.5 for DINOv3-L first stage; OWLv2 crop reranking sets ILIAS 100M reranking state of the art, 2.6 pp above 39.4.R@1 rises from 55.0 to 76.0 (+21.0 pp).Source: Table 1 · p. 8, §5.5 · p. 9

5.2 RESULTS

  • MSA raises R@1 from 0.550 to 0.592 and mAP@1000 from 0.265 to 0.292 on ILIAS 100M; all improvements over baseline are statistically significant (p < 0.001).The gains are +7.7% relative for R@1 and +10.2% for mAP@1000.Source: §5.2 · p. 8
  • MSA also improves mAP on Oxford and Paris Medium (+0.8 and +0.7 pp) and Hard (+1.6 and +1.3 pp), respectively.ILIAS is the primary benchmark because its queries and gallery positives have box annotations; Oxford and Paris annotate only queries.Source: §5.2 · p. 8

5.3 GAINS ARE CONCENTRATED WHERE O2I MISMATCH EXISTS

  • With a median query–positive O2I ratio of 3.25×, MSA gains most on moderate mismatches (+4.4 to +5.5 pp R@1) and less on extreme ratios above 10×.The scale grid bottoms out at 0.25×; the reverse-direction bin is slightly degraded.Source: §5.3 · p. 8
  • At equal five-view cost, GP gains +3.2 pp R@1 versus +0.8 pp for the best non-scale ensemble; αQE and query-side interventions stack to 31.4 mAP@1000.Applying augmentations on the gallery side hurts retrieval, while an oracle gallery-side zoom helps.Source: §5.3 · pp. 8–9

5.4 LEARNABLE ALTERNATIVE: LORA FINE-TUNE

  • LoRA reaches R@1 = 60.2 and mAP@1000 = 29.7, matching or exceeding MSA (59.2/29.2) with one forward pass per image.The adapters are trained to match the frozen baseline CLS feature from a canonical view while seeing a gray-padded, attention-masked, lower-O2I view.Source: §5.4 · p. 9

5.5 RERANKING-STAGE CONFIRMATION: OBJECT-CENTRIC CROPS

  • OWLv2 localizes regions in the top-1000 gallery candidates, then reranks each query–gallery pair by the maximum cosine similarity to the candidate’s region features.Detection and crop encoding are done once per gallery image and amortized across queries, with no training.Source: §5.5 · p. 9

6 DISCUSSION AND CONCLUSION

The paper concludes that O2I mismatch is a distinct, asymmetric retrieval failure that is not resolved by scale-invariant architectures. Training-free ILIAS interventions reach state of the art, while LoRA matches them at single-forward-pass cost.

6 DISCUSSION AND CONCLUSION

  • Training-free interventions achieve ILIAS 100M state of the art at 29.2 pre-rerank and 42.0 post-rerank mAP@1000; LoRA matches them at single-forward-pass cost.The authors identify full pretraining as a next step toward retrieval backbones that are O2I-invariant by construction.Source: §6 · p. 9
  • Equalizing O2I removes most cross-distance failure, but the failure is asymmetric and scale-invariant architectures do not transfer that invariance to O2I.The discussion frames O2I mismatch as an isolated failure mode of visual instance retrieval.Source: §6 · p. 9
  • The synthetic benchmark, released code, object identifiers, and category-selection procedure are specified as reproducibility artifacts.The statement also points to sections detailing the benchmark, checkpoints, interventions, reranker protocol, and significance testing.Source: REPRODUCIBILITY STATEMENT · pp. 9–10

A SYNTHETIC BENCHMARK: EXAMPLE RENDERINGS

The benchmark holds object identity and viewpoint fixed across rendered distances, varying the object's fraction of the image. The supplied passages describe its diversity and point to per-backbone and asymmetry analyses.

A SYNTHETIC BENCHMARK: EXAMPLE RENDERINGS

C DISENTANGLE DECOMPOSITION: PER-BACKBONE NUMBERS

D FIXED-PIXEL CANVAS EXPERIMENT: COSINE MATRIX

E PER-BACKBONE ASYMMETRY NUMBERS

F PER-BACKBONE SYMMETRY CURVES

  • Figure 7 plots MRR against mismatch magnitude, with dashed tight-query curves above solid reverse-direction curves across every retrieval backbone.At each x-position, the two curves represent equal mismatch magnitudes.Source: F PER-BACKBONE SYMMETRY CURVES · pp. 13–14

B DATASET DIVERSITY: CATEGORIES AND INSTANCES

G ADDITIONAL BACKBONE-COMPARISON RESULTS

Multi-scale models reduce some scale-related effects but remain vulnerable to O2I mismatch. Backbone comparisons also show O2I sensitivity with fixed object pixels and asymmetric retrieval across query–gallery directions.

G ADDITIONAL BACKBONE-COMPARISON RESULTS

  • At |∆d|=3.5, Swin, MViTv2, and FlexiViT reach R@1 of 0.073, 0.044, and 0.030, respectively; multi-scale designs mitigate but do not solve O2I mismatch.Swin-L leads at moderate gaps with overall mean R@1 of 0.677, while FlexiViT-L (0.604) and MViTv2-L (0.623) are no better than standard ViTs.Source: G ADDITIONAL BACKBONE-COMPARISON RESULTS · p. 14, Table 2 · p. 15
  • For 9 of 12 backbones, resolution accounts for at most 20% of cross-distance failure; DINOv3-L’s resolution-only residual is 0.040 of a 0.215 total drop.The remaining 81% of DINOv3-L’s drop vanishes after O2I is matched.Source: Table 2 · p. 15
  • With object pixels held byte-identical, the fixed-pixel canvas experiment varies only the fraction of the image occupied by the object.Table 3 reports mean cosine similarity between DINOv3-L features across canvas sizes for 3,021 objects.Source: Table 3 · p. 15
  • CLIP-L leads at small gaps, while SigLIP-2-L is superior at |∆d| ≥2.5; DUSt3R-L falls to R@1 ≤0.030 at large gaps.DINOv3-L outperforms DINOv2-G despite being 3.6× smaller; the paper cautions that training differences make the vision-language explanation difficult to isolate.Source: G ADDITIONAL BACKBONE-COMPARISON RESULTS · p. 15
  • Every backbone favors a tight query against a wide gallery over the reverse direction.Figure 6 plots mean MRR across the ten ordered cross-distance pairs in each direction.Source: Figure 6 · p. 16

H CROSS-DATASET VALIDATION: ROXFORD5K AND RPARIS6K

MSA improves results on Oxford and Paris, while query-side O2I matching improves ILIAS metrics and stacks with query expansion. Gallery-side tests support matching query and gallery O2I, and scale augmentation outperforms generic augmentations at equal cost.

H CROSS-DATASET VALIDATION: ROXFORD5K AND RPARIS6K

  • MSA improves Medium mAP by +0.8 on Oxford and +0.7 on Paris, with Hard-protocol gains of +1.6 and +1.3, respectively.Plain GP slightly hurts Oxford Easy (−2.4 pp); MSA’s attention masking limits that decline to −0.4 pp.Source: H CROSS-DATASET VALIDATION: ROXFORD5K… · p. 16

I QUERY EXPANSION COMPARISON: FULL TABLE

  • On ILIAS 100M, MSA + αQE reaches 31.4 mAP@1000, the best first-stage result; αQE alone raises mAP but degrades R@5 and R@10 without improving R@1.Query-side GP and MSA improve every metric, while αQE reorders retrieved results and cannot recover a wrong top match.Source: I QUERY EXPANSION COMPARISON: FULL TABLE · p. 16
  • Every backbone has higher MRR for tight-query/wide-gallery retrieval than for the reverse direction.The reported comparison averages MRR over ten ordered cross-distance pairs in each direction.Source: Table 4 · p. 17

J GALLERY-SIDE AUGMENTATION

  • On the 856K proxy gallery, gallery-side GP-5 or MSA-3 lowers R@1 by 12–13 pp and mAP by about 7 pp, while oracle-guided zooming raises R@1 from 0.55 to 0.75 and mAP from 0.26 to 0.45.The oracle uses ground-truth boxes for positives, unavailable in realistic retrieval; gallery-wide augmentation must touch every image, so the practical intervention is detector-guided shortlist reranking.Source: J GALLERY-SIDE AUGMENTATION · p. 17

K.1 SCALE AUGMENTATION VS. GENERIC TEST-TIME AUGMENTATION

  • GP provides a 3.2 pp R@1 gain, roughly three times the best non-scale alternative’s gain, at the same five-view query-time cost.Horizontal flip, color jitter, random crops, and mixed non-scale augmentation yield 0.3–1.0 pp R@1 improvements.Source: §K.1 · p. 18

K.2 ABLATION: ALTERNATIVE QUERY-SIDE STRATEGIES

Among query-side strategies, simple scale averaging is strong, while resolution-only changes are weaker than O2I-matching interventions. The accompanying probes examine selective pooling, attention masking, and crop-reranking mechanisms.

K.2 ABLATION: ALTERNATIVE QUERY-SIDE STRATEGIES

  • GP adds +3.2 pp R@1, and neither median nor consistency-weighted pooling significantly improves on it.Object-box cropping performs comparably to GP but requires a bounding box.Source: §K.2 · p. 18
  • MSA improves mAP on Oxford and Paris, with its largest gains on the Hard protocol where O2I mismatch is most severe.Table 7 reports results for DINOv3-L at 768 px.Source: Table 7 · p. 19
  • Resolution-only multi-resolution inference yields +0.7 pp R@1 and +0.5 pp mAP, weaker than GP/MSA because it changes resolution without changing O2I.MSA achieves its best results by preventing gray patches from contaminating the representation.Source: §K.2 · p. 19

L SELECTIVE PATCH POOLING AND ATTENTION-MASK ANALYSIS

  • The appendix probes selective patch pooling and decomposes OWLv2 crop-rerank gains into O2I and outside-bbox-context components.The described decomposition uses a pixel-preserving attention-mask intervention.Source: L SELECTIVE PATCH POOLING AND… · pp. 19–20

K.2 ABLATION: ALTERNATIVE QUERY-SIDE STRATEGIES

  • MSA achieves the best results on all metrics among the query-side strategies compared on ILIAS 100M.The methods are training-free, modify only the query representation, and leave the gallery unchanged.Source: §K.2 · p. 18, Table 10 · p. 20

L.1 SELECTIVE PATCH POOLING ON THE SYNTHETIC BENCHMARK

The appendix links O2I sensitivity to attention-mediated object encoding, separates crop-reranking gains into O2I and context contributions, and tests whether O2I robustness can be learned. It also reports evaluation procedures, a query-level stratification boundary, and implementation details.

L.1 SELECTIVE PATCH POOLING ON THE SYNTHETIC BENCHMARK

  • Object-box patch pooling yields R@1 = 0.793, below CLS at 0.845 but above all-patch mean pooling at 0.617; background-only patches reach 0.583.Excluding boundary-adjacent patches raises background-only R@1 to 0.620, supporting a role for long-range cross-attention; the authors note that multiple factors likely shape the mechanism.Source: §L.1 · p. 20
  • Masking attention to gray-pad patches raises mean R@1 from 0.920 to 0.931; the hardest asymmetric pair gains +14.5 pp.The authors suggest that preventing attention to uninformative gray patches recovers discriminative power.Source: §L.1 · p. 20

L.2 DECOMPOSING OWLV2 CROP-RERANK GAINS: O2I VS OUTSIDE-BBOX CONTEXT

  • On INSTRE, outside-bbox attention masking recovers 35% of the crop-oracle gain, while O2I resizing accounts for the remaining 65%; on ILIAS, the split is 49% and 51%.O2I remains the dominant single factor on both datasets; the mask changes no pixels and keeps patch positions fixed.Source: §L.2 · p. 21

M O2I-ROBUST LORA FINE-TUNING

  • On full ILIAS 100M, the LoRA-adapted backbone reaches R@1 = 60.2 and mAP@1000 = 29.7 with one forward pass per image.On the 856K proxy gallery, LoRA gains +3.5 pp R@1 and +2.1 pp mAP@1000 over vanilla DINOv3-L, and matches MSA’s mAP@1000 = 27.9.Source: M O2I-ROBUST LORA FINE-TUNING · p. 22

N ILIAS 100M RESULTS: FULL TABLE WITH COST BREAKDOWN AND p-VALUES

O SIGNIFICANCE TESTING METHODOLOGY

  • Recall significance uses McNemar’s test, while mAP@1000 uses a paired t-test on per-query average precision.The per-query average-precision-at-1000 values are aggregated across queries.Source: O SIGNIFICANCE TESTING METHODOLOGY · p. 23

P QUERY-POSITIVE PAIR STRATIFICATION

  • Query O2I bins use the query box area relative to the mean positive-gallery box area, smoothing within-query framing variability; pair-level bucketing could refine the analysis.The ILIAS positive set averages about five images per query.Source: P QUERY-POSITIVE PAIR STRATIFICATION · p. 23

Q BACKBONE IMPLEMENTATION DETAILS

  • The backbone comparison includes architectures chosen to test scale robustness, and records each model’s parameters and pooling mode.Models run in bfloat16 on one NVIDIA RTX A5000; pooling modes include CLS, pooled, timm CLS, and patch-token mean.Source: Q BACKBONE IMPLEMENTATION DETAILS · p. 24, Table 15 · p. 24
Loading 2610.11489v1…