PaperlayerView exact revision on arXiv

Source-linked AI summary

Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

Khang Nhat Hoang Vo, Belal Shoer, Abhirama Subramanyam Penamakuri

arXiv:2610.33798v1cs.CVcs.CL

TL;DR

Standard retrieval scores image-caption pairs in isolation, even when each candidate image has neighboring captions. This paper tests frozen-encoder scoring with candidate-side caption evidence and finds gains that depend on caption source, while leave-one-out can overstate the residual’s contribution.

  • Problem

    Dual encoders score each image-caption pair in isolation, leaving neighboring captions unused as evidence when a candidate image is scored.

    Source: §1 · p. 1

  • Method

    The paper fuses image similarity with similarity to each candidate image’s support-caption consensus and evaluates the scorer under a fixed support index.

    Source: §3.3 · p. 3, §6 · p. 9

  • Results

    Text-to-image R@1 improves by 3.4 to 16.8 points with same-source human captions, 1.9 to 7.2 with captioner-generated captions, and at most one point with captions from a different annotation source.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Under a fixed support index, the residual reduces to reweighting image and consensus similarity, and compositional gains come from text matching rather than better image-text grounding.

    Source: §6 · p. 9

    The evaluation assumes each database image has descriptive text and does not test naturally occurring metadata such as alt text, Wikipedia descriptions, or product titles.

    Source: Limitations · p. 9

Abstract

from arXiv · show

An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still judge every image-caption pair in isolation. We ask what these neighboring captions contribute when a frozen encoder consults them at inference time. Fusing image similarity with similarity to an image's own captions improves text-to-image R@1 by 3.4 to 16.8 points across nine frozen encoders when the captions come from the same annotators as the queries. The gain falls to 1.9 to 7.2 points when an off-the-shelf captioner writes them, and to at most one point when they come from a different annotation source; a paraphrase-trained text encoder strengthens the text side further. A natural refinement, scoring the part of the image that the captions leave unexplained, appears to add much more under a leave-one-out evaluation. We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half, and that under a fixed support index the residual reduces to a reweighting of what the captions already provide. On compositional benchmarks, the captions help by matching candidates to human descriptions of the image, without improving image-text grounding. The code is available at: https://github.com/nhatkhangcs/clip_consensus

1 Introduction

The introduction asks whether an image’s neighboring captions can improve frozen dual-encoder retrieval, since standard scoring ignores them. It reports gains that depend on caption source and warns that leave-one-out evaluation can exaggerate the residual’s contribution.

1 Introduction

  • Collections such as COCO and Flickr30K provide multiple captions per image, but standard benchmarks use them as queries or training targets rather than evidence when scoring candidate images.Dual encoders still represent each image with a single vector and score each image-caption pair in isolation.Source: §1 · p. 1
  • The proposed scorer averages each candidate’s support captions into a consensus direction, decomposes its image embedding into consensus and residual components, and fuses those similarities with image similarity.The paper studies how much this candidate-side text helps frozen encoders and how to evaluate it without leakage.Source: §1 · p. 2
  • Under leave-one-out evaluation, the full scorer beats tuned base-plus-consensus by 5.3 to 8.1 points in image-to-text R@1 on COCO, but by at most 0.7 points under a fixed support index.Size-matched leave-one-out recovers about half the gap, implicating both set size and query removal; support captions nevertheless help every evaluated encoder.Source: §1 · p. 2
  • The paper’s protocol evaluates candidate-side captions under a fixed support index, tunes scorers on the same objective, and uses image-clustered bootstrap intervals.The authors release the protocol as a standalone evaluation script.Source: §1 · p. 2
  • The residual’s apparent leave-one-out benefit is inflated because positive and negative candidates have differently constructed supports, while the residual reduces to candidate-dependent reweighting of image and consensus similarity.The paper reports that the residual yields at most a small text-to-image gain under the fixed support index.Source: §1 · p. 2
  • Text-to-image R@1 improves by 3.4 to 16.8 points with same-source human captions across nine frozen encoders, 1.9 to 7.2 with captioner-generated captions across six, and at most one point with different-source captions across four.A paraphrase-trained text encoder strengthens the text side; gains persist under corrected COCO positives, while compositional results reflect text matching rather than better image-text grounding.Source: §1 · p. 2

2 Related Work

Related work spans changes to dual-encoder training, candidate-side retrieval augmentation, inference-time score normalization, and retrieval benchmarks. This paper keeps encoders frozen and focuses on candidate scoring and its evaluation.

Image-text dual encoders

Candidate-side text and late fusion

  • Candidate-side augmentation has precedents in instance and text retrieval, while multimodal retrieval and caption reranking fuse or compare candidate-associated text; this work studies each image’s own captions.The paper’s focus is how much the fusion helps, which text encoder suits the text side, and how to evaluate it without leakage.Source: §2 · pp. 2–3

Inference-time score normalization and evaluation

  • Inference-time normalization offers metadata-free comparison methods, while compositional benchmarks test sensitivity to caption structure and can be solvable by blind text models.The paper compares normalization methods to separate gains requiring captions from general normalization gains, and pairs compositional results with blind baselines.Source: §2 · p. 3

3 Methodology

The methodology scores frozen image-text embeddings with candidate-side caption evidence. It defines a caption consensus and image residual, combines their similarities with the base score, and specifies support-construction protocols to prevent query leakage.

3 Methodology

  • The method uses a frozen dual encoder to encode each database image and its natural support captions, then combines base, consensus, and residual similarities.The paper analyzes what each score term contributes.Source: §3 · p. 3

3.1 Problem Setup

  • Standard retrieval ranks candidate images by cosine similarity to the query caption, ignoring the other captions associated with each image.The setup allows a query caption to belong to the candidate image’s caption set or not.Source: §3.1 · p. 3

3.2 Caption Consensus

  • The caption consensus is the mean direction of a candidate’s encoded support captions, summarizing content that multiple annotators mention rather than every image detail.The query is scored against this consensus direction.Source: §3.2 · p. 3

3.3 Image Residual

  • The image residual removes the consensus component from the image embedding and is intended to represent content that the shared caption direction does not capture.Because it is constructed from the image and consensus, its score is a function of the base and consensus scores and cannot independently isolate omitted content.Source: §3.3 · p. 3

3.4 Combined Score

  • The combined score weights base, consensus, and residual similarities; the image-to-text score also uses the query image’s own support captions to measure candidate-caption agreement.The base weight is fixed at 1, the other weights are selected on held-out labeled data, and the encoders remain unchanged.Source: §3.4 · pp. 3–4, Figure 2 · p. 4

3.5 Support Construction for Evaluation

  • Because a query caption usually belongs to its positive image’s caption set, including it among supports leaks the answer; leave-one-out removes it only for the positive, while size-matched leave-one-out also drops one support from each negative.The fixed support index holds out one caption per image across the database, so supports do not depend on the query; it is used for main results.Source: §3.5 · p. 4

4 Experiments

The experiments compare fixed-support retrieval, residual protocols, text-side choices, support sources, compositional benchmarks, and qualitative cases; scorers use the same tuning objective and fixed support index unless noted.

4.1 Retrieval with a Fixed Support Index

  • B+C exceeds Base by 4.9–12.0 points on COCO and 7.1–16.8 on Flickr30K under the fixed support index, with all intervals excluding zero.Weights are tuned on COCO only; Flickr30K serves as a held-out transfer setting.Source: Table 1 · p. 5, §4.1 · p. 5

4.2 Analysis of the Residual Term

  • Full leads separately tuned B+C by 5.3–8.1 I2T@1 points under standard LOO, versus 2.6–4.6 on COCO and 0.4–1.4 on Flickr30K under size-matched LOO.Under the fixed support index, the lead never exceeds 0.7 points; equalizing set size removes roughly half of the LOO inflation, with query-specific removal accounting for the remainder.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 The Text Side and Metadata-Free Alternatives

  • GTE-large fusion beats B+C by 2.6–4.3 T2I@1 and 3.1–7.6 I2T@1 points across encoders and datasets.QB-Norm gains 1.7–2.1 COCO T2I@1 points and slightly hurts Flickr30K; inverted and dual softmax are treated as transductive references.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Effect of Support Source and Quality

  • Generated supports improve every encoder in both directions by 1.9–7.2 T2I@1 and 1.9–7.3 I2T@1 points, recovering 42–59% of the COCO T2I gain from human supports.Cross-source human supports gain at most 1.0 point; the authors note that the captioner may transfer visual knowledge.Source: §4.4 · pp. 6–7, Table 4 · p. 7
  • Benefits rise steadily through four supports, while 60% mismatch makes clean-tuned gains −1.6 for SigLIP and −4.2 for OpenCLIP.Retuning keeps gains at or above Base at every mismatch level by shifting weight back to the image, motivating calibration to expected support quality.Source: Figure 4 · p. 7, §4.4 · p. 7

4.5 Compositional Benchmarks

  • On SugarCrepe, own-image captions yield 84–86.5% versus 61–67% for random-item captions, indicating text-to-text matching rather than improved image-text grounding.The paper notes that ARO cannot measure grounding here because a language model alone performs strongly on its tasks.Source: Table 5 · p. 8, §4.5 · p. 8

4.6 Qualitative Analysis

  • On COCO eval2500, Full with SigLIP ViT-B/16 corrects 1,069 Base errors but reverses 392; Base distractors often miss a specific queried detail.The reverse errors tend to involve generic queries that fit several images, consistent with some being false negatives under single-positive evaluation.Source: §4.6 · p. 8

5 Discussion and Recommendations

The discussion recommends query-independent support sets, tuned comparisons with simpler scorers, and deliberate choices about text encoders and support quality. It also cautions that support-source alignment and blind baselines affect how retrieval and compositional results should be interpreted.

Support-set construction

  • Leave-one-out gains of 5.3 to 8.1 points shrink to at most 0.7 points under a fixed support index, so make support sets independent of the query.Where leave-one-out cannot be avoided, a size-matched variant removes the set-size effect.Source: §5 · p. 8

Scorer comparisons

  • Retune simpler scorers against added terms: under the same protocol, the residual only reweights image and consensus similarities already used by base-plus-consensus.The residual appeared essential until base-plus-consensus was tuned under the same protocol and objective.Source: §5 · p. 8

Text-side choice

  • A paraphrase-trained sentence encoder over the same captions outperformed the dual encoders’ own text towers for all six encoders.The text encoder is a design choice, not one inherited from the image model.Source: §5 · p. 8

Support-source reporting

  • Support gains depend on query-support similarity: same-source human captions help most, generated captions recover part of the gain, and another annotation source adds almost nothing.Same-source results may overstate what to expect when deployment queries and metadata are written differently.Source: §5 · p. 8

Support-quality calibration

  • Weights tuned on clean supports can push retrieval below the image-only baseline with noisy supports, while weights retuned for the noise level kept it at or above Base.Fusion weights should match support quality expected at deployment.Source: §5 · pp. 8–9

Compositional evaluation

  • A language model without image access solves ARO-Order, so pair compositional results with blind and random-support references to clarify what gains measure.Reporting those references helps distinguish image grounding from text-based effects.Source: §5 · p. 9

6 Conclusion

The paper evaluates captions attached to database images as inference-time evidence for frozen dual encoders, using a fixed support index to avoid leakage. Caption fusion improves text-to-image retrieval most when supports and queries share a source, while the residual adds little and compositional gains reflect text matching rather than better grounding.

Conclusion

  • Under a fixed support index, fusing image and own-caption similarity improves text-to-image R@1 for every tested encoder; gains are largest for same-source captions, about half as large for generated captions, and close to zero across sources.A paraphrase-trained sentence encoder over the same captions outperforms each dual encoder’s text tower.Source: §6 · p. 9
  • The residual’s larger leave-one-out advantage reflects differently constructed positive and negative supports, while under a fixed index it only reweights image and consensus similarity.Set size and query removal each explain part of the leave-one-out gap.Source: §6 · p. 9
  • On compositional benchmarks, support captions help by matching candidates to human descriptions, not by improving image-text grounding; ARO can be solved without the image.The paper releases a fixed-support-index protocol for comparing candidate-side text.Source: §6 · p. 9

Limitations

The paper’s scope and protocols constrain how its retrieval gains should be interpreted. Its limitations include the descriptive-text setting and English caption domain, as well as diagnostic and computational details of the fixed-index experiments.

Limitations

  • The study assumes descriptive text for each database image and does not test naturally occurring metadata; its comparisons with Base measure the value of that text, not encoder improvement.Fusion weights are tuned on COCO and transfer to Flickr30K, but both datasets use short, crowd-sourced English captions under similar guidelines; other languages and domains are untested, and generated-caption gains may partly reflect the captioner’s visual knowledge.Source: Limitations · pp. 9–10

A.1 Protocol and Tuning Details

  • The protocol tunes fusion weights on COCO tune2500, reuses selected weights in specified experiments, and uses image-clustered bootstrap intervals because caption-level resampling can overstate significance.The encoders remain off the shelf, with no trainable parameters beyond fusion weights.Source: §A.1 · p. 13

A.2 Practical Cost

  • B+C and Full can each be indexed as one precomputed vector per image with Base’s index size; the reported million-image setup gives 1.54 GB storage and roughly 1.2 ms per query.The per-token BLIP-2 variant instead must store query-token residuals or recompute them at query time.Source: §A.2 · pp. 13–14

B.1 Image-to-Text Retrieval under the Fixed Index

  • Fixed-index image-to-text results have one correct caption per query rather than five and are therefore diagnostic, not comparable to standard 5-positive results.Every B+C-over-Base gain in Table 7 has a 95% confidence interval excluding zero.Source: §B.1 · p. 14, Table 7 · p. 14

B.2 False-Negative-Corrected Labels

  • B+C’s gain over Base persists on COCO with CxC and ECCV Caption positives, while ECCV Full-minus-B+C T2I differences of 0.31 to 1.71 points are not significant with 643 queries.The ECCV evaluation uses 3,060 image and 643 text queries within eval2500.Source: §B.2 · p. 14

B.3 Caption Position in Flickr30K

Caption position in Flickr30K affects query difficulty and fold variability, but B+C remains better than Base across positions. Using more support captions strengthens the gain, while a single support can be too noisy at weights tuned for four.

B.3 Caption Position in Flickr30K

  • B+C improves over Base at every Flickr30K caption position for every encoder, with the smallest gain at position 0; averaging all five folds removes positional bias.Position 0 averages 18.1 words, while position 4 averages 7.9 words; Base T2I@1 falls with position as shorter queries are harder.Source: §B.3 · p. 14, Table 9 · p. 14

B.4 Number of Support Captions

  • The gain grows with support count, but with one support B+C falls slightly below Base in COCO I2T for all four encoders and T2I for three SigLIP encoders.Weights were tuned for four supports, so the Full advantage at one support does not establish a residual-term benefit.Source: §B.4 · p. 14

C.1 Testing the LOO Mechanism

The appendix tests whether the residual’s apparent LOO advantage reflects useful caption evidence, then examines residual variants, text-side scoring, normalization, and alternative support sources.

C.1 Testing the LOO Mechanism

  • Under size-matched LOO, Full trails B+C in COCO T2I for four of six encoders; fixed indexing restores only a small advantage.The tables compare Full with B+C across standard LOO, size-matched LOO, and fixed-index supports.Source: §C.1 · pp. 14–15, Table 11 · p. 15

C.2 Random-Support Control under LOO

  • Random supports remove the LOO gain: it requires both positive-specific supports and captions that describe the image.The random-support control preserves the LOO asymmetry but replaces image-specific captions with random captions.Source: §C.2 · p. 15

C.3 Algebraic Form of the Residual Score

  • The residual makes the full score a candidate-dependent reweighting of image and consensus similarity, but its fixed-index scores nearly match tuned B+C.For SigLIP on COCO, Full yields 57.54 / 59.53 (I2T@1 / T2I@1), versus 57.53 / 59.50 for tuned B+C.Source: §C.3 · pp. 15–16

C.4 Alternative Residual Formulations

  • No alternative residual beats tuned B+C by more than about one point; image-agreement weighting lowers T2I in every column.V2’s gains reach at most 0.80, while V1’s largest gain (+1.02 COCO T2I for CLIP ViT-L/14) accompanies an I2T drop.Source: §C.4 · p. 16

C.5 Entity-Level Diagnostics

  • Consensus similarity tracks shared caption entities positively and caption-specific entities negatively; residual–query similarity correlates only weakly.The residual norm’s opposite pattern is not independent evidence because η = 1 − ρ^2 for unit vectors.Source: §C.5 · p. 16, Table 15 · p. 17

D.1 Text-Side Encoders; D.2 Inference-Time Normalization

  • Support consensus beats maximum-over-support scoring for every sentence encoder; QB-Norm uses a disjoint query bank, unlike transductive evaluated-query softmax methods.In image-to-text retrieval, QB-Norm changes B+C by −2.9 to +1.3 points on Flickr30K.Source: §D.1 · p. 16, §D.2 · p. 16, Table 16 · p. 17

E.1 Machine-Generated Supports

  • With generated supports, Full adds little over B+C; its gain is significant only for Flickr30K T2I with OpenCLIP and SigLIP.The supports are four Qwen2.5-VL-generated captions per image, and all five human captions serve as queries.Source: §E.1 · pp. 16–17

E.2 Cross-Source Supports from Localized Narratives

  • Cross-source gains shrink; with human-caption queries and a narrative support, Full beats B+C on T2I for all eight backbone–dataset pairs, while I2T is slightly lower.The T2I gains are +0.41 to +1.52, and six of eight confidence intervals exclude zero; the authors call this one setting among many.Source: §E.2 · p. 17

E.3 Robustness to Mismatched Supports

This section tests support mismatch and reports a stronger-model reranking check.

E.3 Robustness to Mismatched Supports

  • Accuracy falls monotonically as supports become mismatched; with fixed clean-support weights, B+C falls below Base at 60% mismatch in both directions, and Full does so in I2T.Retuning keeps both scorers at or above Base at every mismatch level by shrinking the support weight.Source: §E.3 · p. 17

F Stronger Models

  • BLIP-2 ViT-g’s reranked T2I@1 is 73.7 on the 2,500-image gallery, consistent with the published 68.3 on the full 5K gallery.The method uses ITC for initial scoring and rescoring of the top 16 candidates with ITM logit + ITC.Source: F Stronger Models · p. 17

G.1 Per-Encoder Results with Blind Baselines

The appendix checks blind and compositional references, additional benchmark constructions and encoders, fold variation, and fusion rules.

G.1 Per-Encoder Results with Blind Baselines

  • On ARO-Order, own-image captions reach 51–68% accuracy, versus about 20% for random-item captions.The per-encoder table compares candidate scoring against random-item captions, own-image captions, and B+C.Source: §G.1 · p. 17, Table 23 · p. 20

G.2 Benchmark Construction and Results on Additional Encoders

  • SugarCrepe positives matched one of the five support captions in 7,504 of 7,511 items; ARO-Order includes perturbations identical to the original in 18% of items.The appendix also restricts ARO VG-Relation and VG-Attribution to Visual Genome images overlapping COCO, covering 48% of each set.Source: §G.2 · p. 19

H.1 Overview

  • Additional encoders reproduce the main pattern: B+C improves T2I@1 by 3.4–13.2 points, while Full’s LOO I2T@1 advantage shrinks to at most 0.64 under the fixed index.For COCO LOO, Full leads B+C by 4.4–6.8 I2T@1 points; the three SigLIP encoders’ own captions alone outperform B+C on ARO.Source: §H.1 · pp. 19, 21, Table 25 · p. 20

H.2 Fold Variation and Confidence Intervals

  • The fixed-index R@1 tables report five-fold standard deviations and image-clustered bootstrap intervals for B+C’s gain over Base.These passages describe the uncertainty summaries but do not state their numerical values.Source: §H.2 · p. 21

H.3 Fusion Rules with the Encoder’s Text Tower

  • All three text-only variants fall well below B+C; maximum or softmax scoring over individual supports also trails consensus scoring.The fusion comparisons use the fixed protocol and tuned baselines; the text-only variants do not use the image.Source: §H.3 · p. 21

I Qualitative Results

The examples show both corrections and regressions from consensus–residual scoring, including cases where generic queries admit plausible alternatives.

I Qualitative Results

  • Across 12,500 COCO queries, Full moves 1,069 from incorrect to correct top-1 versus Base, while 392 move the other way.The illustrated queries are randomly sampled, not hand-picked.Source: I Qualitative Results · p. 21
  • In the correction examples, the correct image’s captions name details missing from the query’s image-similarity ranking, and consensus recovers them.Examples include a baseball query and a beige building query.Source: I Qualitative Results · p. 21
  • Generic queries can demote the correct image when a plausible distractor’s captions match just as well, creating apparent false negatives under single-positive evaluation.Figure 6 shows the correct image dropping from rank 1 to rank 2 while the new top-ranked image remains a plausible match.Source: I Qualitative Results · p. 21, Figure 6 · p. 23
Loading 2610.33798v1…