Source-linked AI summary
Image-text Retrieval: A Survey on Recent Research and Development
Min Cao, Shiping Li, Juntao Li, Liqiang Nie, Min Zhang
TL;DR
Existing surveys provide less depth on ITR and largely leave cross-modal pre-training uncovered. This paper surveys ITR methods across four perspectives and compares representative approaches. It concludes that ITR has developed considerably, especially in the cross-modal pre-training era, while identifying less-explored issues.
Problem
Prior surveys address other multimodal tasks at the expense of ITR depth and largely omit the now-mainstream pre-training paradigm.
Source: §1 · p. 1Method
The paper organizes ITR methods around feature extraction, feature alignment, system efficiency, and cross-modal pre-training, and reviews datasets, metrics, and representative approaches.
Source: §1 · pp. 1–2Results
The survey concludes that ITR has developed considerably, especially with cross-modal pre-training, and that local-alignment approaches outperform global-alignment approaches overall.
Source: §5 · p. 6, §4.3 · p. 6Takeaways & Limitations
The review identifies data-related issues, including ambiguous or multiple image-text correspondences and underused scene text, as areas for further ITR development.
Source: §5 · p. 6Explorations of data multiplicity have considered training data while ignoring test data, leaving current results on such datasets controversial.
Source: §5 · p. 6
Abstract
from arXiv · showhide
In the past few years, cross-modal image-text retrieval (ITR) has experienced increased interest in the research community due to its excellent research value and broad real-world application. It is designed for the scenarios where the queries are from one modality and the retrieval galleries from another modality. This paper presents a comprehensive and up-to-date survey on the ITR approaches from four perspectives. By dissecting an ITR system into two processes: feature extraction and feature alignment, we summarize the recent advance of the ITR approaches from these two perspectives. On top of this, the efficiency-focused study on the ITR system is introduced as the third perspective. To keep pace with the times, we also provide a pioneering overview of the cross-modal pre-training ITR approaches as the fourth perspective. Finally, we outline the common benchmark datasets and valuation metric for ITR, and conduct the accuracy comparison among the representative ITR approaches. Some critical yet less studied issues are discussed at the end of the paper.
1 Introduction
The survey addresses the need for a deeper ITR-focused review that includes the increasingly prominent cross-modal pre-training paradigm. It organizes ITR research into four perspectives and reviews representative methods, datasets, metrics, and accuracy comparisons.
Survey scope
- Prior surveys covered other multimodal tasks, limiting ITR depth, and largely left pre-training untouched; this survey focuses on ITR and investigates pre-training.The paper also notes that pre-training had become mainstream.Source: §1 · p. 1
Feature extraction and alignment
- The taxonomy separates feature extraction into independent visual-semantic embedding, interactive cross-attention, and self-adaptive approaches, and alignment into global and fine-grained local approaches.The system is described as image/text processing branches followed by an integration module.Source: §1 · p. 1
System efficiency
- Efficiency-oriented methods use hash encoding, model compression, or coarse-to-fine fast-then-slow retrieval.Hashing binarizes floating-point features to reduce computational cost; compression emphasizes lightweight, low-energy running.Source: §1 · p. 2
Pre-training paradigm
- The survey also examines cross-modal pre-training, classifying approaches by model architecture, pre-training task, and pre-training data.It notes that large-scale pre-trained models can provide rich knowledge and encouraging retrieval performance without sophisticated retrieval engineering.Source: §1 · p. 2
Paper organization
- The paper outlines its coverage of the first three perspectives, pre-training methods, common datasets and metrics, representative-method accuracy comparisons, and future work.The first three perspectives are summarized in Section 2, pre-training in Section 3, and datasets, metrics, and comparisons in Section 4.Source: §1 · p. 2
2 Image-text Retrieval
The survey organizes image-text retrieval around feature extraction, feature alignment, and retrieval efficiency. It describes methods that trade off representational detail, accuracy, and computational cost.
2.1 Feature Extraction
- Feature extraction spans independent VSE, dense cross-attention, and self-adaptive routing; VSE enables offline gallery features, while cross-attention can improve accuracy at online pairwise cost.Self-adaptive methods route different pairs through different feature-extraction mechanisms.Source: §2.1 · p. 2
2.2 Feature Alignment
- Global alignment can miss fine detail; local and relation-aware methods model finer correspondences, while pretrain-then-finetune global approaches can perform satisfactorily [Jia et al., 2021].Local methods match image regions or patches with words; relation-aware methods use intra-modal relationships to support inter-modal alignment.Source: §2.2 · p. 3
2.3 Retrieval Efficiency
- Hashing reduces computation and storage but can degrade accuracy; compression targets bulky models, while fast-then-slow retrieval screens with VSE before CA to balance efficiency and accuracy.Even lightweight cross-attention models may remain slow because feature extraction executes quadratically.Source: §2.3 · pp. 3–4
3 Pre-training Image-text Retrieval
The survey reviews cross-modal pre-training for image-text retrieval through model architecture, pretext tasks, and data. It describes how these design choices shape visual feature processing and downstream retrieval.
Pre-training Image-text Retrieval
- Cross-modal pre-training for ITR grew rapidly from 2019, with research focusing on model architecture, pre-training tasks, and pre-training data.Most current approaches use transformer architectures as their building block.Source: §3 · p. 4
Model architecture
- Pre-training retrieval models use two-stream or one-stream transformers, but common image preprocessing can consume substantial computation and constrain visual expression through predefined vocabularies.The survey describes an end-to-end transformer framework as achieving a win-win between efficiency and accuracy.Source: §3 · p. 4
Pre-training task
- Pre-training tasks include coarse- and fine-grained image-text matching as well as masked language and vision modeling.Matching tasks compare global features or align patches, regions, words, and phrases; masked modeling predicts masked words or visual features or labels.Source: §3 · p. 4
Pre-training data
- Data strategies exploit intra- and cross-modal knowledge and expand pre-training sources with in-domain, non-paired single-modal, and newly collected large-scale data.The survey reports that larger-scale data collection usually brings outstanding performance on downstream cross-modal tasks, including ITR.Source: §3 · pp. 4–5
4 Datasets and Evaluation
The section describes common ITR datasets and R@K, then compares representative approaches, reporting accuracy gains over time and differences across feature extraction and alignment strategies.
4.1 Datasets
- COCO Captions contains 123,287 images with five human-generated captions each, while Flickr30K has 31,000 images with five descriptions per image.The section also describes their training, validation, and test splits and the COCO evaluation folds.Source: §4.1 · p. 5
4.2 Evaluation Metric
- R@K measures the proportion of correct matches among the top-K retrieved results.It is the most commonly used ITR evaluation metric.Source: §4.2 · p. 5
4.3 Accuracy Comparison
- Cross-attention approaches generally outperform visual-semantic-embedding approaches at R@1, except that large-scale pre-training can offset the latter’s weaker interaction modeling.SCAN marked a breakthrough over LTBNN at the time; DIME outperforms VSE and cross-attention approaches on Flickr30k but is inferior to SAN on COCO Captions.Source: §4.3 · p. 5
- Local-alignment-involved approaches outperform global-alignment-driven approaches overall, while ALIGN leads global-alignment approaches at R@1.The comparison also notes strong performance by local-alignment approaches including BEF.Source: §4.3 · pp. 5–6, Table 2 · p. 5
- From 2017 to 2021, ITR accuracy increased, with a large 2020 jump attributed to pre-training and a dramatic increase in training-data volume.The reported trend shows pre-training continuing that momentum.Source: §4.3 · p. 6, Figure 4 · p. 6
5 Conclusion and Future Works
The survey reviews ITR across four perspectives and concludes that the field has advanced considerably, especially with cross-modal pre-training. It identifies data, knowledge, and resource demands as areas for future work.
Conclusion
- The survey reviews feature extraction, feature alignment, system efficiency, and pre-training, and finds considerable ITR development, especially in the cross-modal pre-training era.It also summarizes common datasets and evaluation metrics and analyzes representative approaches quantitatively.Source: §5 · p. 6
Data
- Noisy and ambiguous image-text datasets constrain data-driven ITR; results on datasets such as COCO Captions remain controversial.The survey notes elusive descriptions and multiple plausible image-text correspondences, while prior multiplicity studies considered training data but ignored test data.Source: §5 · p. 6, Figure 5 · p. 6
Knowledge
- Commonsense knowledge derived from caption corpora is constrained by those corpora and may not fit ITR well; tailored knowledge and causal reasoning are proposed for future work.Source: §5 · p. 6
New paradigm
- Large-scale pretrain-then-finetune approaches have an accuracy advantage, but their need for substantial labeled fine-tuning data limits real-world applicability.The survey points to few-shot prompt-based tuning as a guide for a more resource-friendly pretrain-then-prompt paradigm.Source: §5 · p. 6