Source-linked AI summary
Learnable Pillar-based Re-ranking for Image-Text Retrieval
Leigang Qu, Meng Liu, Wenjie Wang, Zhedong Zheng, Liqiang Nie, Tat-Seng Chua
TL;DR
Image-text retrieval research has largely focused on pairwise matches, while higher-order neighbor relations are less explored and existing re-ranking methods face cross-modal challenges. LeaPRR represents entities using similarities to multimodal neighbor pillars and reasons over their neighborhood relations. Experiments on Flickr30K and MS-COCO support its effectiveness and superiority, and the authors report generalization and transferability.
Problem
Image-text retrieval methods focus on pairwise matches and leave higher-order neighbor relations among multiple samples underexplored; existing re-ranking approaches also face generalization, flexibility, sparsity, and asymmetry challenges.
Source: §1 · p. 1Method
LeaPRR uses final similarities to construct a multimodal pillar space and applies neighbor-aware graph reasoning and structure alignment to model cross-modal relations.
Source: §1 · p. 2, §3.2.1 · p. 4Results
Takeaways & Limitations
The authors describe the learnable re-ranking paradigm as plug-and-play and applicable to existing image-text retrieval backbones.
Source: §1 · p. 2The method is detailed under I2T based on the stated high consistency between I2T and T2I apart from retrieval direction.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Image-text retrieval aims to bridge the modality gap and retrieve cross-modal content based on semantic similarities. Prior work usually focuses on the pairwise relations (i.e., whether a data sample matches another) but ignores the higher-order neighbor relations (i.e., a matching structure among multiple data samples). Re-ranking, a popular post-processing practice, has revealed the superiority of capturing neighbor relations in single-modality retrieval tasks. However, it is ineffective to directly extend existing re-ranking algorithms to image-text retrieval. In this paper, we analyze the reason from four perspectives, i.e., generalization, flexibility, sparsity, and asymmetry, and propose a novel learnable pillar-based re-ranking paradigm. Concretely, we first select top-ranked intra- and inter-modal neighbors as pillars, and then reconstruct data samples with the neighbor relations between them and the pillars. In this way, each sample can be mapped into a multimodal pillar space only using similarities, ensuring generalization. After that, we design a neighbor-aware graph reasoning module to flexibly exploit the relations and excavate the sparse positive items within a neighborhood. We also present a structure alignment constraint to promote cross-modal collaboration and align the asymmetric modalities. On top of various base backbones, we carry out extensive experiments on two benchmark datasets, i.e., Flickr30K and MS-COCO, demonstrating the effectiveness, superiority, generalization, and transferability of our proposed re-ranking paradigm.
1 INTRODUCTION
Image-text retrieval models commonly focus on pairwise image–text matches, leaving higher-order neighbor relations unexplored. The paper argues that existing re-ranking methods do not transfer directly to this setting and introduces LeaPRR to address four challenges.
1 INTRODUCTION
- Existing image-text retrieval approaches focus on pairwise image–text matching, while higher-order relations among multiple samples remain unmodeled.The latter include query–neighbor and neighbor–neighbor relations.Source: §1 · p. 1, Figure 1 · p. 2
- Directly extending existing re-ranking methods to image-text retrieval faces generalization, flexibility, sparsity, and modality-asymmetry challenges.The cited example reports a 0.02% positive-item ratio for MS-COCO (5K), versus 2.3% for RParis6K.Source: §1 · pp. 1–2
- LeaPRR uses final similarities and top-ranked multimodal neighbors as pillars to represent entities in a pillar space.The similarities can be supplied by one- or two-tower frameworks, without intermediate content features.Source: §1 · p. 2
- Neighbor-aware graph reasoning models query–neighbor and neighbor–neighbor relations, while ranking constraints strengthen refined pillar features against sparse positives.A structure-alignment constraint transforms contextual structures across modalities to address asymmetry.Source: §1 · p. 2
- Experiments across base backbones and Flickr30K and MS-COCO validate LeaPRR's effectiveness and superiority; the paper also reports generalization and transferability.The introduction notes that αQE performance degrades as aggregated neighbors increase in image-text retrieval.Source: §1 · p. 2, Figure 2 · p. 2
2 RELATED WORK
Prior image-text retrieval work develops joint-embedding and fine-grained-interaction frameworks, while re-ranking research largely addresses single-modality retrieval. The section notes that these lines leave practical re-ranking and flexible multimodal neighbor modeling underexplored.
2.1 Image-Text Retrieval
- Two-tower methods embed images and texts jointly, offering parallel feature extraction and offline indexing but potentially coarser matching.The section contrasts these with one-tower methods that model fine-grained cross-modal interactions, at the cost of heavier computation.Source: §2.1 · pp. 2–3
- Most image-text retrieval work emphasizes joint embeddings or fine-grained alignments rather than practical post-processing re-ranking.Source: §2.1 · p. 3
2.2 Re-ranking
- Single-modality re-ranking includes query expansion, diffusion, and neighbor-based methods, with examples using nearest neighbors, manifolds, and affinity graphs.The section also mentions a k-reciprocal nearest-neighbor scheme designed for image-text retrieval.Source: §2.2 · p. 3
- Existing re-ranking methods depend on prior knowledge and lack flexibility to model query–neighbor and neighbor–neighbor relations, while overlooking ITR sparsity and asymmetry.Source: §2.2 · p. 3
3 METHODOLOGY
The methodology section lays out LeaPRR’s components, moving from the cross-modal re-ranking problem to pillar-based representation and graph reasoning, then to learning in pillar space.
3 METHODOLOGY
- The methodology first introduces the problem definition for cross-modal image-text retrieval re-ranking.Source: §3 · p. 3
- It then presents pillar-based representation and neighbor-aware graph reasoning.Source: §3 · p. 3
- The final methodological component is the learning strategy in pillar space.Source: §3 · p. 3
3.1 Problem Definition
The problem definition starts from a backbone-generated ranking and frames re-ranking as refining its top results. It considers both image-to-text and text-to-image retrieval.
3.1 Problem Definition
- A base image-text retrieval backbone computes query–item similarities, which are sorted to form the initial ranking list.The backbone may be one-tower or two-tower.Source: §3.1 · p. 3
- Re-ranking refines the initial list so semantically matching items move higher, focusing on the top K results.The section motivates this focus by noting users’ attention to top-ranked results in practical scenarios.Source: §3.1 · p. 3
- The task covers image-to-text and text-to-image retrieval, with the method described under I2T for both directions’ high consistency apart from retrieval direction.Source: §3.1 · p. 3
3.2 Pillar-based Representation
LeaPRR represents samples through their relations to selected multimodal pillars, using similarities rather than intermediate content features. This representation addresses the difficulty of supporting both one- and two-tower retrieval backbones.
3.2 Pillar-based Representation
- Similarity-only re-ranking avoids reliance on intermediate features, while relations to informative references offer an alternative way to represent each entity.The references should be related enough to help distinguish an item from others.Source: §3.2 · pp. 3–4
3.2.1 Multimodal Pillar Selection.
- For an image query, top-ranked texts become inter-modal pillars, while intra-modal database rankings supply additional image pillars.The paper uses equal numbers of intra- and inter-modal pillars for simplicity, though different counts are acceptable.Source: §3.2.1 · p. 4
3.2.2 Pillar-based Encoding.
- Pillar-based encoding represents the query and its top-K text neighbors using similarities to the selected pillars.The inter-modal similarity can come from any base backbone, while intra-modal similarities are computed with image and text encoders.Source: §3.2.2 · p. 4
3.3 Neighbor-aware Graph Reasoning
Neighbor-aware graph reasoning adaptively models query-neighbor and neighbor-neighbor relations in pillar space. It combines common-neighbor and learned affinities to propagate information and refine pillar-based representations.
3.3 Neighbor-aware Graph Reasoning
- Adaptive graph propagation captures both query-neighbor and neighbor-neighbor relations, addressing the limited flexibility of rule-based approaches.After pillar encoding, the query and its neighbors are treated as nodes connected by affinity scores.Source: §3.3 · pp. 4–5
3.3.1 Neighbor-based Affinity.
- Neighbor-based affinity links nodes according to shared top-C neighbors, suppresses noisy edges with a sparse factor, and is row-normalized.Neighbor sets are collected from intra- and inter-modal rankings.Source: §3.3.1 · p. 5
3.3.2 Learning-based Affinity.
- Learning-based affinity scores are computed from stacked pillar representations using fully connected layers and row-wise softmax.The resulting score represents a learned relation between each pair of nodes.Source: §3.3.2 · p. 5
3.3.3 Neighbor Propagation.
- The method averages neighbor-based and learned affinities for graph propagation, recomputing learned affinities after the first layer.The final graph-layer output is the refined pillar-based representation matrix.Source: §3.3.3 · p. 5
3.4 Learning from Pillar Space
The pillar-space model is optimized under image–text pairwise supervision with global, local, and cross-modal alignment constraints. These objectives combine global matching, local hard-positive structure, and mutual learning between retrieval directions.
3.4.1 Global Contrastive Loss.
- Global contrastive loss pulls queries toward positives and away from negatives using refined pillar-based representations.Similarity is cosine similarity between the refined query and neighbor representations.Source: §3.4.1 · p. 5
3.4.2 Local Triplet Loss.
- The local triplet objective focuses on the hardest positive, complementing contrastive learning while allowing variance among positives.Top-K negatives are already hard, so the method selects the hardest positive rather than mining the hardest negative.Source: §3.4.2 · p. 5
3.4.3 Mutual Modality Alignment.
- Mutual modality alignment replaces independent similarity averaging with two submodels that teach and promote each other.For positive sampling, the method uses pairwise ground truth and uniformly samples when multiple positives exist; it defines alignment loss with KL divergence.Source: §3.4.3 · p. 5
3.4 Learning from Pillar Space
- The model combines three constraints in both I2T and T2I settings to optimize learning from pillar space.The constraints are global contrastive, local triplet, and modality alignment losses.Source: §3.4 · p. 5, §3.4.3 · p. 6
4 EXPERIMENTS
Experiments evaluate LeaPRR on Flickr30K and MS-COCO and address baseline performance, component effects, and generalization across backbones and datasets. The protocol uses standard recall metrics and reports implementation and training settings.
4.1 Datasets
- Flickr30K and MS-COCO are the two evaluation datasets, with five human-annotated sentences per image in each.Flickr30K uses 29,783/1,000/1,000 train/validation/test images; MS-COCO uses 113,287/5,000/5,000 and the full 5K test setting.Source: §4.1 · p. 6
4.2 Experimental Settings
- The evaluation uses R@1, R@5, R@10, and rSum for image-to-text and text-to-image retrieval.The model is trained for 30 epochs with SGD, momentum 0.9, batch size 512, and learning rate 0.01; the validation checkpoint with highest rSum is tested.Source: §4.2.1 · p. 6, §4.2.2 · p. 6
4.2.1 Evaluation Protocols.
- The protocol evaluates recall at K values of 1, 5, and 10, with rSum aggregating recall across both retrieval directions.Higher R@K and rSum indicate better performance.Source: §4.2.1 · p. 6
4.2.2 Implementation Details.
- The implementation uses CAMERA’s pretrained visual and textual branches to calculate intra-modal similarities and collect intra-modal pillars.The cited implementation passage applies these branches across base backbones and experiments.Source: §4.2.2 · p. 6
4.3 Performance Comparison (RQ1)
LeaPRR improves re-ranking across two-tower and one-tower backbones, while existing baselines show uneven gains under sparse and asymmetric cross-modal neighborhoods.
4.3 Performance Comparison (RQ1)
- The comparison evaluates query-expansion, diffusion, and reciprocal-neighbor methods on VSE∞ and DIME∗, with query expansion restricted to the two-tower backbone.The baselines use published pretrained parameters; ensemble similarities are used where available.Source: §4.3 · pp. 6–7
- KRNN is the strongest cited baseline on most metrics, while diffusion yields only marginal rSum gains for VSE∞ and no consistent improvement for DIME∗.DFS improves rSum by 0.3% on Flickr30K and 0.1% on MS-COCO (5K) over VSE∞.Source: §4.3 · p. 7
- Query expansion can reduce VSE∞ performance, especially at R@5 and R@10, and database-side augmentation before expansion does not reverse the decline.The authors attribute the weakness to sparse positive items and cross-modal asymmetry.Source: §4.3 · p. 7
- 15.9% and 12.7%: LeaPRR improves rSum for DIME∗ on Flickr30K and MS-COCO (5K), respectively, outperforming baselines on most metrics across two backbones.The reported gains are attributed to flexible pillar representations and graph reasoning, alongside methods addressing sparsity and modality asymmetry.Source: §4.3 · p. 7
4.4 In-depth Analysis (RQ2 & RQ3)
Ablations and parameter studies examine pillar selection, graph reasoning, objectives, and neighbor counts; broader evaluations test generalization across backbones and transfer across backbones and datasets.
4.4.1 Ablation Study.
- Bottom-ranked or random pillars reduce performance versus the full model, while the full model also improves on variants using only intra-modal or inter-modal pillars.The bottom-ranked and random variants remain comparable to the DIME∗ backbone, indicating tolerance to poor pillars without performance collapse.Source: §4.4.1 · p. 7
- The contrastive and triplet objectives complement one another, the triplet objective appears to contribute more, and mutual modality alignment helps combine the two single submodels.Ensembles improve overall rSum, although some metrics, such as I2T R@5, may decline.Source: §4.4.1 · p. 8
4.4.2 Parameter Sensitivity.
- Pillar-count performance saturates at roughly L=32 or 64 for most metrics, while too few pillars underrepresent samples and too many risk adding noise.Almost all tested settings improve on the base backbone.Source: §4.4.2 · p. 8
- As neighbor count increases, rSum first improves, then saturates and declines; the t-i submodel saturates and deteriorates earlier than i-t.All tested neighbor settings improve overall performance over the base backbone; the authors relate earlier t-i decline to its larger negative-sample set.Source: §4.4.2 · p. 8
- Across five one- and two-tower architectures on Flickr30K and MS-COCO (5K), LeaPRR shows strong generalization and outperforms KRNN, especially on R@1.Table 4 covers SCAN∗, DIME∗, VSRN∗, CAMERA∗, and VSE∞.Source: §4.4.3 · p. 8, Table 4 · p. 8
4.4.4 Transferability Analysis.
- LeaPRR transfers across almost all tested backbones, and Flickr30K-to-MS-COCO transfer improves rSum by 11.0 versus 12.7 when learning on MS-COCO.The reported transfer improvement is described as highly comparable despite Flickr30K being one quarter the scale of MS-COCO.Source: §4.4.4 · p. 9, Figure 5 · p. 9, Table 5 · p. 9
5 CONCLUSION AND FUTURE WORK
The paper proposes a learnable re-ranking framework that models higher-order neighbor relations in image-text retrieval. Its experiments demonstrate effectiveness, superiority, generalization, and transferability; future work will explore external knowledge and transfer across downstream retrieval tasks.
5 CONCLUSION AND FUTURE WORK
- LeaPRR reformulates multimodal re-ranking using top-ranked neighbors as pillars and flexibly models relations among entities for one- and two-tower frameworks.The experiments support its effectiveness, superiority, generalization, and transferability.Source: §5 · p. 9
- Future work will incorporate abundant multimodal external knowledge and investigate transferability across a broader range of downstream retrieval tasks.Source: §5 · p. 9