PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross Modal Retrieval with Querybank Normalisation

Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, Samuel Albanie

arXiv:2112.12777v3cs.CV

TL;DR

High-dimensional cross-modal embeddings can suffer from hubness, which degrades retrieval rankings. The paper introduces Querybank Normalisation, which uses a querybank to adjust similarities, and reports consistent gains across tasks, models, and benchmarks, including settings without concurrent test-query access.

  • Problem

    High-dimensional cross-modal embeddings can produce hubs that appear in disproportionately many nearest-neighbour sets and degrade retrieval rankings.

    Source: §1 · pp. 1–2

  • Method

    Querybank Normalisation uses a querybank to re-normalise query similarities and reduce gallery-hub influence without model fine-tuning.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    QB-Norm consistently improves strong baselines across a broad range of tasks, models, and benchmarks.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    The framework retains effectiveness without concurrent access to test queries, supporting its use when retrieval handles queries individually.

    Source: §1 · p. 2

    QB-Norm adds pre-computation costs, and adversarial querybank selection or substantial domain gaps can reduce its benefits.

    Source: §5 · p. 8

Abstract

from arXiv · show

Profiting from large-scale training datasets, advances in neural architecture design and efficient inference, joint embeddings have become the dominant approach for tackling cross-modal retrieval. In this work we first show that, despite their effectiveness, state-of-the-art joint embeddings suffer significantly from the longstanding "hubness problem" in which a small number of gallery embeddings form the nearest neighbours of many queries. Drawing inspiration from the NLP literature, we formulate a simple but effective framework called Querybank Normalisation (QB-Norm) that re-normalises query similarities to account for hubs in the embedding space. QB-Norm improves retrieval performance without requiring retraining. Differently from prior work, we show that QB-Norm works effectively without concurrent access to any test set queries. Within the QB-Norm framework, we also propose a novel similarity normalisation method, the Dynamic Inverted Softmax, that is significantly more robust than existing approaches. We showcase QB-Norm across a range of cross modal retrieval models and benchmarks where it consistently enhances strong baselines beyond the state of the art. Code is available at https://vladbogo.github.io/QB-Norm/.

1. Introduction

Cross-modal embeddings enable natural-language search across modalities, but their high-dimensional spaces can produce hubs that degrade retrieval rankings. The paper introduces Querybank Normalisation to reduce hub influence and proposes Dynamic Inverted Softmax as a more robust normalisation method.

1. Introduction

  • Natural-language queries offer a way to search image, audio, and video collections without using specialised database languages.Source: §1 · p. 1
  • High-dimensional cross-modal embeddings can produce hubs—vectors that appear among disproportionately many nearest-neighbour sets—and left unaddressed, hubs degrade search rankings.Source: §1 · pp. 1–2
  • Querybank Normalisation uses inference-time querybanks to reduce gallery-hub influence; the framework works without concurrent access to test queries, and Dynamic Inverted Softmax is more robust than prior methods.The authors also report that QB-Norm does not require model fine-tuning.Source: §1 · p. 2
  • QB-Norm is a non-parametric framework that improves retrieval without fine-tuning and is effective across a broad range of tasks, models, and benchmarks.Source: §1 · p. 2

2. Related work

Related work spans cross-modal embeddings and similarity-search techniques, while this paper focuses on hubness in cross-modal retrieval. It situates Querybank Normalisation alongside prior hubness methods and distinguishes its querybank approach from memory-bank uses in other tasks.

2. Related work

  • Cross-modal embedding research covers text-image, text-video, text-audio, and image-audio retrieval, while similarity-search work includes re-ranking, query expansion, compression, and approximate nearest-neighbour methods.Source: §2 · p. 2
  • Unlike memory banks used for general problem-solving, captioning, summarisation, self-supervised learning, or rare instances, Querybank Normalisation targets hubness mitigation.Source: §2 · p. 2
  • Figure 2 reports strong hubness across four text-video models on MSR-VTT and across four datasets using TT-CE+.Source: Figure 2 · p. 3
  • Prior hubness mitigation includes similarity rescaling, centroid-focused methods, and zero-shot mappings; some approaches scale quadratically, while the paper compares NLP-derived CSLS and Inverted Softmax with its method.The cited NLP approaches include CSLS [25] and Inverted Softmax [102]; general retrieval schemes are described in [28].Source: §2 · p. 3
  • Some prior approaches apply Inverted Softmax over the full set of test queries, requiring concurrent access that the authors call unrealistic for continuously operating retrieval systems.The paper demonstrates that concurrent access to test queries is not required.Source: §2 · p. 3
  • The authors report that prior techniques can damage retrieval for particular querybank selections, an issue their Dynamic Inverted Softmax addresses.Source: §2 · p. 3

3. Method

This section defines cross-modal retrieval with learned shared embeddings, documents hubness in text-video retrieval, and presents Querybank Normalisation (QB-Norm) as a querybank-based similarity-normalisation framework. It then introduces Dynamic Inverted Softmax (DIS), which conditionally applies inverted softmax to improve robustness when querybank coverage is poor.

3.1. Task definition

  • Cross-modal retrieval ranks gallery samples in a shared embedding space; this study focuses on natural-language-query tasks, where hubness mitigation has received limited attention.Training uses corresponding query-gallery pairs, while test queries and galleries are unseen during training.Source: §3.1 · pp. 3–4

3.2. Motivation

  • On MSR-VTT, a small number of videos are retrieved extremely often by every examined text-video method, including CLIP2Video.The authors report this hubness across methods with different combinations of video modalities, attention mechanisms, and large-scale pretraining.Source: §3.2 · p. 4

3.3. Querybank Normalisation

  • QB-Norm uses querybank probes to measure gallery-item hubness, then normalizes each query's gallery similarities using those probes.The framework comprises querybank construction and similarity normalisation, and targets practical approaches with at-most-linear gallery-size complexity.Source: §3.3 · p. 4

3.4. Design choices

  • DIS is significantly more robust than GC, CSLS, and IS, and does not harm performance with suboptimal querybank selection.It activates inverted softmax only when the retrieved nearest neighbor falls within a set of gallery vectors identified as potential hubs; the section also casts GC, CSLS, and IS as QB-Norm choices.Source: §3.4 · pp. 4–5

4. Experiments

The experiments test training-set querybanks, compare normalization strategies under different querybank conditions, and evaluate QB-NORM across retrieval tasks. They report improved retrieval across tested methods, benchmarks, and tasks.

4.1. Datasets and Evaluation Metrics

  • Experiments cover six text-video benchmarks, plus text-image, text-audio, and image-to-image retrieval; results report R@K and MdR averaged over three seeded runs.R@K is higher-is-better, while MdR is lower-is-better.Source: §4.1 · pp. 5–6

4.2. Querybank Normalisation

  • A 60K-sample training-set querybank performs comparably to a test-set querybank on MSR-VTT.The experiments therefore use training-set samples for remaining querybanks, avoiding concurrent access to test queries.Source: Table 1 · p. 6, §4.2 · p. 6
  • Performance rises with querybank size but saturates, while strong results are achievable with a few thousand random training samples.Figure 3 reports the size trend on the MSR-VTT validation split.Source: Figure 3 · p. 6, §4.2 · p. 6
  • DIS matches IS and CSLS with in-domain and close-domain querybanks, and is more robust with far-domain and adversarial querybanks.Far-domain sampling degraded performance below the unnormalised baseline for methods other than GC and DIS; adversarial selection harmed methods other than DIS.Source: §4.2 · p. 6, Table 2 · p. 7
  • DIS uses inverse temperature 20 and k = 1; the authors report the temperature worked consistently across datasets and k offered a performance-robustness trade-off.CLIP2Video is an exception, using inverse temperature 1.99−1.Source: §4.2 · p. 7
  • QB-NORM significantly reduces hubness, measured by skewness in the k-occurrences distribution, across four datasets.The reported analysis uses querybanks containing all training-set samples.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Comparison with other methods

  • QB-NORM brings significant improvement over all tested methods, benchmarks and tasks.The evaluation spans text-video, text-image, image-image, and text-audio retrieval.Source: §4.3 · p. 8, Table 10 · p. 8, Table 11 · p. 8

5. Limitations and societal impact

The paper notes computational costs and conditions that can reduce QB-NORM's benefits, alongside potential positive and harmful uses of cross-modal retrieval.

5. Limitations and societal impact

  • All QB-NORM normalisation techniques add pre-computation costs, and DIS adds a small further computational cost.Source: §5 · p. 8
  • Adversarial querybank selection and significant domain gaps can reduce the benefits of Querybank Normalisation.Source: §5 · p. 8
  • Cross-modal search can aid content discovery, but could also enable searching social media for signs of political dissent.Source: §5 · p. 8

6. Conclusions

The conclusion presents Querybank Normalisation as a framework for mitigating hubness, with Dynamic Inverted Softmax proposed for robust similarity normalisation across tasks, models, and benchmarks. The appendix outlines additional dataset details, ablations, comparisons, and analyses.

6. Conclusions

  • QB-NORM addresses hubness, and Dynamic Inverted Softmax provides robust similarity normalisation across a range of tasks, models, and benchmarks.The appendix describes further dataset details, ablations, comparisons, complexity analyses, and qualitative results.Source: §6 · p. 8

Appendix

  • The appendix covers ablations of the DIS hyperparameter, training-set querybanks with IS, and embedding dimensionality.It also outlines additional text-video results, normalisation complexity, comparison with CENT, skewness metrics, and further retrieval results.Source: Appendix · p. 14

A. Dataset details

The appendix describes datasets and evaluation setups for text-video, text-image, text-audio, and image-to-image retrieval. It also reports a DIS hyperparameter study that selects k = 1 as a trade-off across querybank domains.

A. Dataset details

A.1. Text-video retrieval

  • The text-video evaluation uses seven benchmarks, including MSR-VTT, MSVD, DiDeMo, LSMDC, VaTeX, QuerYD, and ActivityNet.Protocols vary by dataset; MSR-VTT uses both the official full test split and the 1k-A split, while VaTeX evaluation uses English annotations.Source: §A.1 · p. 14
  • QuerYD evaluation follows prior text-video work and does not use its temporal localization annotations; ActivityNet results use the val1 split.QuerYD has official train, validation, and test splits; ActivityNet uses paragraph video retrieval.Source: §A.1 · p. 14

A.3. Text-audio retrieval

  • Text-audio retrieval is evaluated on AudioCaps using the prior-work setup, with 49,291 training, 428 validation, and 816 test samples.Source: §A.3 · p. 14

A.4. Image-to-image retrieval

  • CUB-200-2011 uses the first 100 classes (5,863 images) for training and remaining classes (5,924) for testing, following prior work [97]; Stanford Online Products uses provided train/test splits of 59,551 and 60,502 images [97] [105].Source: §A.4 · pp. 14–15

B. The influence of the Top-k hyperparameter on DIS normalisation

  • For DIS, k = 1 offers a trade-off between performance with In Domain querybanks and robustness with Far Domain querybanks, and is used in all experiments.The study examines Top-k selection when constructing the gallery activation set.Source: B. The influence of the Top-k… · p. 15, Table 14 · p. 15

C. Can effective querybanks can be constructed from the training set with IS normalisation?

Training-set querybanks perform comparably to test-set querybanks for IS normalisation. The reported dimensionality analyses find similar QB-NORM gains across embedding sizes and a slight increase in gains with more video modalities.

C. Can effective querybanks can be constructed from the training set with IS normalisation?

  • A 60K-sample training-set querybank performs comparably to a test-set querybank with IS normalisation on MSR-VTT full split.The authors state that this supports the claim that QB-NORM need not access multiple test queries concurrently.Source: C. Can effective querybanks can be… · p. 15

D. The influence of embedding dimensionality on the effectiveness of QB-NORM

  • QB-NORM yields around the same gain across embedding sizes, with a slight increase in gain as the number of video modalities grows.The authors interpret this pattern as consistent with the hypothesis that intrinsic dimensionality, rather than shared embedding size, influences hubness.Source: D. The influence of embedding… · p. 15, Figure 5 · p. 15

E. Additional text-video retrieval results

Additional text-video comparisons report improvements from QB-NORM on MSR-VTT full split and ActivityNet. The appendix also discusses normalisation costs, CENT comparisons, and hubness measurements.

E. Additional text-video retrieval results

F. The computational complexity of normalisation strategies

  • DIS adds O(N) gallery-activation storage and an O(N) top-1 search at inference, while its IS normalisation step has constant-time overhead.The stated complexity assumes exact similarity searches and precomputed querybank-gallery similarities.Source: F. The computational complexity of… · pp. 16–17

G. Comparison to CENT

H. Hubness and Skewness

  • QB-NORM reduces skewness and the maximum number of times any video is retrieved, while making previously unretrieved videos retrievable.The skewness measure is based on the k-occurrence distribution, which counts query-neighbour occurrences for gallery embeddings.Source: Figure 6 · p. 16, H. Hubness and Skewness · p. 17

I. Additional ablations on other metrics

The additional ablations show that individual retrieval metrics follow the geometric-mean trend; the video-retrieval appendix also describes expert embeddings and qualitative examples.

I. Additional ablations on other metrics

  • Individual R1, R5, and R10 metrics reflect the querybank-size and inverse-temperature trends shown by their geometric mean.The reported individual-metric trends align with the main-paper results.Source: §I · p. 17

J. Video and text embeddings (experts) description used for video retrieval

  • This work uses pretrained TT-CE+ [27] and CE+ [73] weights with pretrained action, object, audio, and scene experts, including two action experts and two object-expert forms.The scene embedding has a 2208 dimension.Source: J. Video and text embeddings (experts)… · pp. 17, 19
  • The qualitative video-retrieval examples include two queries where QB-Norm retrieves the target video and one failure with a more related retrieved video.The third example remains a target-retrieval failure.Source: Figure 8 · p. 18
  • The action experts are Kinetics-trained I3D Action(KN) and IG-65m-trained 34-layer R(2+1)D Action(IG); object experts are ImageNet-trained SENet-154 Obj(IN) and Instagram-trained ResNext-101 Obj(IG).Action(KN) embeddings are extracted from frame clips at 25 fps and center-cropped to 224 pixels; both object embeddings are extracted at 25 fps.Source: J. Video and text embeddings (experts)… · p. 19
  • The audio expert uses YouTube-8m-trained VGGish, the scene expert uses Places365-pretrained DenseNet-161, speech uses Google Cloud transcription, and text uses fine-tuned GPT2-xl.The scene and final text embeddings have dimensions 2208 and 1600, respectively.Source: J. Video and text embeddings (experts)… · p. 19
  • CLIP2Video uses its provided model, CLIP4Clip is retrained per dataset, and the other retrieval models follow their official repository instructions.The appendix specifies pretrained weights and features for MMT-Oscar, RDML models, and audio retrieval.Source: J. Video and text embeddings (experts)… · p. 19

K. v2t performance metrics

The reverse video-to-text retrieval evaluation uses videos as queries to retrieve descriptions. With DIS and a querybank built from training-split videos, QB-Norm yields a striking performance boost.

K. v2t performance metrics

  • QB-Norm with DIS and a querybank of training-split videos yields a striking boost for video-to-text retrieval, where videos query descriptions.Table 21 reports the MSR-VTT full-split state-of-the-art comparison for this task; the supplied passages give no metric values.Source: K. v2t performance metrics · p. 19, Table 21 · p. 19

L. Qualitative results

Qualitative examples show that QB-Norm can retrieve missed target videos or produce more related results, though the examples reveal no consistent pattern among corrected or remaining failures.

L. Qualitative results

  • QB-Norm retrieves the correct target in some examples and produces a more query-related result in a failure case.The examples include gains for both shorter and longer, highly descriptive captions.Source: L. Qualitative results · p. 19
  • The corrected and remaining failure cases show no consistent, obvious qualitative trend across samples.The paper relates this observation to prior work suggesting hubness is a property of the distribution rather than individual samples.Source: L. Qualitative results · p. 19

Related papers

Loading 2112.12777v3…